2024-09-15T20:09:31.507 INFO:root:teuthology version: 1.2.2.dev2+g3752d38 2024-09-15T20:09:31.525 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-09-15T20:09:31.600 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/teuthology-2024-09-15_20:00:16-rados-main-distro-default-smithi/7905590 branch: main description: rados/standalone/{supported-random-distro$/{centos_latest} workloads/scrub} email: ceph-qa@ceph.com first_in_suite: false job_id: '7905590' kernel: kdb: 1 sha1: distro last_in_suite: false machine_type: smithi name: teuthology-2024-09-15_20:00:16-rados-main-distro-default-smithi no_nested_subset: false openstack: - volumes: count: 3 size: 10 os_type: centos os_version: 9.stream overrides: admin_socket: branch: main ceph: conf: mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 osd: debug ms: 1 debug osd: 20 flavor: default log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) sha1: fc3e0bff7468fd55478adbae7f75508b749124b6 ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} install: ceph: flavor: default sha1: fc3e0bff7468fd55478adbae7f75508b749124b6 workunit: branch: main sha1: 7498493279982dfe87f00616198a5967475f6169 owner: scheduled_teuthology@teuthology priority: 101 repo: https://git.ceph.com/ceph.git roles: - - mon.a - mgr.x - osd.0 - osd.1 - osd.2 - client.0 seed: 2855 sha1: fc3e0bff7468fd55478adbae7f75508b749124b6 sleep_before_teardown: 0 subset: 17936/100000 suite: rados suite_branch: main suite_path: /home/teuthworker/src/git.ceph.com_ceph_7498493279982dfe87f00616198a5967475f6169/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph.git suite_sha1: 7498493279982dfe87f00616198a5967475f6169 targets: smithi136.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBMboQ5EHmd/qK/XoAjAzR040KVaaYR9p3b1rzNxvvVc2wggKoMYZhMfsbPLStmPN3GfrAgekCbYl0azjsl+PS6k= tasks: - install: null - workunit: basedir: qa/standalone clients: all: - scrub teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: 3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a timestamp: 2024-09-15_20:00:16 tube: smithi user: teuthology verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.3401856 2024-09-15T20:09:31.600 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph_7498493279982dfe87f00616198a5967475f6169/qa; will attempt to use it 2024-09-15T20:09:31.601 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph_7498493279982dfe87f00616198a5967475f6169/qa/tasks 2024-09-15T20:09:31.601 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-09-15T20:09:31.602 INFO:teuthology.task.internal:Checking packages... 2024-09-15T20:09:31.622 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash 'fc3e0bff7468fd55478adbae7f75508b749124b6' 2024-09-15T20:09:31.622 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-09-15T20:09:31.622 INFO:teuthology.packaging:ref: None 2024-09-15T20:09:31.622 INFO:teuthology.packaging:tag: None 2024-09-15T20:09:31.622 INFO:teuthology.packaging:branch: main 2024-09-15T20:09:31.622 INFO:teuthology.packaging:sha1: fc3e0bff7468fd55478adbae7f75508b749124b6 2024-09-15T20:09:31.622 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=main 2024-09-15T20:09:31.843 INFO:teuthology.task.internal:Found packages for ceph version 19.3.0-4930.gfc3e0bff 2024-09-15T20:09:31.844 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-09-15T20:09:31.851 INFO:teuthology.task.internal:no buildpackages task found 2024-09-15T20:09:31.851 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-09-15T20:09:31.872 INFO:teuthology.task.internal:Saving configuration 2024-09-15T20:09:31.880 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-09-15T20:09:31.888 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-09-15T20:09:31.912 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi136.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-09-15_20:00:16-rados-main-distro-default-smithi/7905590', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9.stream', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-09-15 20:03:51.238402', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBMboQ5EHmd/qK/XoAjAzR040KVaaYR9p3b1rzNxvvVc2wggKoMYZhMfsbPLStmPN3GfrAgekCbYl0azjsl+PS6k='} 2024-09-15T20:09:31.912 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-09-15T20:09:31.920 INFO:teuthology.task.internal:roles: ubuntu@smithi136.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'osd.0', 'osd.1', 'osd.2', 'client.0'] 2024-09-15T20:09:31.920 INFO:teuthology.run_tasks:Running task console_log... 2024-09-15T20:09:31.955 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7fccd6e63ac0>, signals=[15]) 2024-09-15T20:09:31.955 INFO:teuthology.run_tasks:Running task internal.connect... 2024-09-15T20:09:31.962 INFO:teuthology.task.internal:Opening connections... 2024-09-15T20:09:31.962 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi136.front.sepia.ceph.com 2024-09-15T20:09:31.963 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi136.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-15T20:09:32.039 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-09-15T20:09:32.045 DEBUG:teuthology.orchestra.run.smithi136:> uname -m 2024-09-15T20:09:32.119 INFO:teuthology.orchestra.run.smithi136.stdout:x86_64 2024-09-15T20:09:32.119 DEBUG:teuthology.orchestra.run.smithi136:> cat /etc/os-release 2024-09-15T20:09:32.176 INFO:teuthology.orchestra.run.smithi136.stdout:NAME="CentOS Stream" 2024-09-15T20:09:32.176 INFO:teuthology.orchestra.run.smithi136.stdout:VERSION="9" 2024-09-15T20:09:32.176 INFO:teuthology.orchestra.run.smithi136.stdout:ID="centos" 2024-09-15T20:09:32.177 INFO:teuthology.orchestra.run.smithi136.stdout:ID_LIKE="rhel fedora" 2024-09-15T20:09:32.177 INFO:teuthology.orchestra.run.smithi136.stdout:VERSION_ID="9" 2024-09-15T20:09:32.177 INFO:teuthology.orchestra.run.smithi136.stdout:PLATFORM_ID="platform:el9" 2024-09-15T20:09:32.177 INFO:teuthology.orchestra.run.smithi136.stdout:PRETTY_NAME="CentOS Stream 9" 2024-09-15T20:09:32.177 INFO:teuthology.orchestra.run.smithi136.stdout:ANSI_COLOR="0;31" 2024-09-15T20:09:32.177 INFO:teuthology.orchestra.run.smithi136.stdout:LOGO="fedora-logo-icon" 2024-09-15T20:09:32.177 INFO:teuthology.orchestra.run.smithi136.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-09-15T20:09:32.177 INFO:teuthology.orchestra.run.smithi136.stdout:HOME_URL="https://centos.org/" 2024-09-15T20:09:32.177 INFO:teuthology.orchestra.run.smithi136.stdout:BUG_REPORT_URL="https://issues.redhat.com/" 2024-09-15T20:09:32.177 INFO:teuthology.orchestra.run.smithi136.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-09-15T20:09:32.177 INFO:teuthology.orchestra.run.smithi136.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-09-15T20:09:32.178 INFO:teuthology.lock.ops:Updating smithi136.front.sepia.ceph.com on lock server 2024-09-15T20:09:32.198 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-09-15T20:09:32.207 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-09-15T20:09:32.214 INFO:teuthology.task.internal:Checking for old test directory... 2024-09-15T20:09:32.214 DEBUG:teuthology.orchestra.run.smithi136:> test '!' -e /home/ubuntu/cephtest 2024-09-15T20:09:32.233 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-09-15T20:09:32.241 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-09-15T20:09:32.241 DEBUG:teuthology.orchestra.run.smithi136:> test -z $(ls -A /var/lib/ceph) 2024-09-15T20:09:32.302 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-09-15T20:09:32.332 INFO:teuthology.run_tasks:Running task kernel... 2024-09-15T20:09:32.346 INFO:teuthology.task.kernel:normalize config orig: {'kdb': 1, 'sha1': 'distro'} 2024-09-15T20:09:32.346 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': 1, 'sha1': 'distro'}}, timeout 300 2024-09-15T20:09:32.346 DEBUG:teuthology.orchestra.run.smithi136:> test -f /run/.containerenv -o -f /.dockerenv 2024-09-15T20:09:32.363 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-15T20:09:32.363 DEBUG:teuthology.orchestra.run.smithi136:> uname -r 2024-09-15T20:09:32.421 INFO:teuthology.orchestra.run.smithi136.stdout:5.14.0-503.el9.x86_64 2024-09-15T20:09:32.422 INFO:teuthology.task.kernel:Running kernel on smithi136: 5.14.0-503.el9.x86_64 2024-09-15T20:09:32.422 DEBUG:teuthology.orchestra.run.smithi136:> sudo yum install -y kernel 2024-09-15T20:09:36.377 INFO:teuthology.orchestra.run.smithi136.stdout:CentOS Stream 9 - BaseOS 3.8 MB/s | 8.2 MB 00:02 2024-09-15T20:09:39.661 INFO:teuthology.orchestra.run.smithi136.stdout:CentOS Stream 9 - AppStream 9.3 MB/s | 20 MB 00:02 2024-09-15T20:09:45.136 INFO:teuthology.orchestra.run.smithi136.stdout:CentOS Stream 9 - CRB 4.0 MB/s | 6.5 MB 00:01 2024-09-15T20:09:46.628 INFO:teuthology.orchestra.run.smithi136.stdout:CentOS Stream 9 - Extras packages 115 kB/s | 19 kB 00:00 2024-09-15T20:09:47.434 INFO:teuthology.orchestra.run.smithi136.stdout:Extra Packages for Enterprise Linux 33 MB/s | 23 MB 00:00 2024-09-15T20:09:52.798 INFO:teuthology.orchestra.run.smithi136.stdout:lab-extras 36 kB/s | 1.7 kB 00:00 2024-09-15T20:09:54.812 INFO:teuthology.orchestra.run.smithi136.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-09-15T20:09:54.813 INFO:teuthology.orchestra.run.smithi136.stdout:Package kernel-5.14.0-503.el9.x86_64 is already installed. 2024-09-15T20:09:54.922 INFO:teuthology.orchestra.run.smithi136.stdout:Dependencies resolved. 2024-09-15T20:09:54.927 INFO:teuthology.orchestra.run.smithi136.stdout:================================================================================ 2024-09-15T20:09:54.927 INFO:teuthology.orchestra.run.smithi136.stdout: Package Architecture Version Repository Size 2024-09-15T20:09:54.927 INFO:teuthology.orchestra.run.smithi136.stdout:================================================================================ 2024-09-15T20:09:54.927 INFO:teuthology.orchestra.run.smithi136.stdout:Installing: 2024-09-15T20:09:54.927 INFO:teuthology.orchestra.run.smithi136.stdout: kernel x86_64 5.14.0-505.el9 baseos 23 k 2024-09-15T20:09:54.927 INFO:teuthology.orchestra.run.smithi136.stdout:Installing dependencies: 2024-09-15T20:09:54.928 INFO:teuthology.orchestra.run.smithi136.stdout: kernel-core x86_64 5.14.0-505.el9 baseos 16 M 2024-09-15T20:09:54.928 INFO:teuthology.orchestra.run.smithi136.stdout: kernel-modules x86_64 5.14.0-505.el9 baseos 35 M 2024-09-15T20:09:54.928 INFO:teuthology.orchestra.run.smithi136.stdout: kernel-modules-core x86_64 5.14.0-505.el9 baseos 29 M 2024-09-15T20:09:54.928 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-09-15T20:09:54.928 INFO:teuthology.orchestra.run.smithi136.stdout:Transaction Summary 2024-09-15T20:09:54.928 INFO:teuthology.orchestra.run.smithi136.stdout:================================================================================ 2024-09-15T20:09:54.928 INFO:teuthology.orchestra.run.smithi136.stdout:Install 4 Packages 2024-09-15T20:09:54.928 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-09-15T20:09:54.929 INFO:teuthology.orchestra.run.smithi136.stdout:Total download size: 79 M 2024-09-15T20:09:54.929 INFO:teuthology.orchestra.run.smithi136.stdout:Installed size: 126 M 2024-09-15T20:09:54.929 INFO:teuthology.orchestra.run.smithi136.stdout:Downloading Packages: 2024-09-15T20:09:55.232 INFO:teuthology.orchestra.run.smithi136.stdout:(1/4): kernel-5.14.0-505.el9.x86_64.rpm 81 kB/s | 23 kB 00:00 2024-09-15T20:09:56.807 INFO:teuthology.orchestra.run.smithi136.stdout:(2/4): kernel-modules-5.14.0-505.el9.x86_64.rpm 19 MB/s | 35 MB 00:01 2024-09-15T20:09:57.582 INFO:teuthology.orchestra.run.smithi136.stdout:(3/4): kernel-core-5.14.0-505.el9.x86_64.rpm 5.9 MB/s | 16 MB 00:02 2024-09-15T20:10:03.848 INFO:teuthology.orchestra.run.smithi136.stdout:(4/4): kernel-modules-core-5.14.0-505.el9.x86_6 3.3 MB/s | 29 MB 00:08 2024-09-15T20:10:03.848 INFO:teuthology.orchestra.run.smithi136.stdout:-------------------------------------------------------------------------------- 2024-09-15T20:10:03.849 INFO:teuthology.orchestra.run.smithi136.stdout:Total 8.8 MB/s | 79 MB 00:08 2024-09-15T20:10:04.537 INFO:teuthology.orchestra.run.smithi136.stdout:Running transaction check 2024-09-15T20:10:04.891 INFO:teuthology.orchestra.run.smithi136.stdout:Transaction check succeeded. 2024-09-15T20:10:04.892 INFO:teuthology.orchestra.run.smithi136.stdout:Running transaction test 2024-09-15T20:10:05.880 INFO:teuthology.orchestra.run.smithi136.stdout:Transaction test succeeded. 2024-09-15T20:10:05.880 INFO:teuthology.orchestra.run.smithi136.stdout:Running transaction 2024-09-15T20:10:08.320 INFO:teuthology.orchestra.run.smithi136.stdout: Preparing : 1/1 2024-09-15T20:10:09.071 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : kernel-modules-core-5.14.0-505.el9.x86_64 1/4 2024-09-15T20:10:09.164 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : kernel-core-5.14.0-505.el9.x86_64 2/4 2024-09-15T20:10:11.968 INFO:teuthology.orchestra.run.smithi136.stdout: Running scriptlet: kernel-core-5.14.0-505.el9.x86_64 2/4 2024-09-15T20:10:12.089 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : kernel-modules-5.14.0-505.el9.x86_64 3/4 2024-09-15T20:10:16.922 INFO:teuthology.orchestra.run.smithi136.stdout: Running scriptlet: kernel-modules-5.14.0-505.el9.x86_64 3/4 2024-09-15T20:10:17.039 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : kernel-5.14.0-505.el9.x86_64 4/4 2024-09-15T20:10:21.884 INFO:teuthology.orchestra.run.smithi136.stdout: Running scriptlet: kernel-modules-core-5.14.0-505.el9.x86_64 4/4 2024-09-15T20:11:00.328 INFO:teuthology.orchestra.run.smithi136.stdout: Running scriptlet: kernel-core-5.14.0-505.el9.x86_64 4/4 2024-09-15T20:11:00.369 INFO:teuthology.orchestra.run.smithi136.stdout: Running scriptlet: kernel-modules-5.14.0-505.el9.x86_64 4/4 2024-09-15T20:11:00.691 INFO:teuthology.orchestra.run.smithi136.stdout: Running scriptlet: kernel-5.14.0-505.el9.x86_64 4/4 2024-09-15T20:11:00.691 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : kernel-5.14.0-505.el9.x86_64 1/4 2024-09-15T20:11:00.691 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : kernel-core-5.14.0-505.el9.x86_64 2/4 2024-09-15T20:11:00.691 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : kernel-modules-5.14.0-505.el9.x86_64 3/4 2024-09-15T20:11:01.525 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : kernel-modules-core-5.14.0-505.el9.x86_64 4/4 2024-09-15T20:11:01.525 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-09-15T20:11:01.525 INFO:teuthology.orchestra.run.smithi136.stdout:Installed: 2024-09-15T20:11:01.525 INFO:teuthology.orchestra.run.smithi136.stdout: kernel-5.14.0-505.el9.x86_64 2024-09-15T20:11:01.525 INFO:teuthology.orchestra.run.smithi136.stdout: kernel-core-5.14.0-505.el9.x86_64 2024-09-15T20:11:01.525 INFO:teuthology.orchestra.run.smithi136.stdout: kernel-modules-5.14.0-505.el9.x86_64 2024-09-15T20:11:01.525 INFO:teuthology.orchestra.run.smithi136.stdout: kernel-modules-core-5.14.0-505.el9.x86_64 2024-09-15T20:11:01.525 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-09-15T20:11:01.525 INFO:teuthology.orchestra.run.smithi136.stdout:Complete! 2024-09-15T20:11:01.905 DEBUG:teuthology.orchestra.run.smithi136:> rpm -q kernel | sort -rV | head -n 1 2024-09-15T20:11:01.996 INFO:teuthology.orchestra.run.smithi136.stdout:kernel-5.14.0-505.el9.x86_64 2024-09-15T20:11:01.996 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-505.el9.x86_64 2024-09-15T20:11:01.996 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-503.el9.x86_64 Expected: 5.14.0-505.el9.x86_64 2024-09-15T20:11:01.996 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-09-15T20:11:01.996 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-09-15T20:11:01.996 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2024-09-15T20:11:01.997 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi136.front.sepia.ceph.com, path=None, version=distro) 2024-09-15T20:11:01.997 DEBUG:teuthology.orchestra.run.smithi136:> rpm -q kernel | sort -rV | head -n 1 2024-09-15T20:11:02.056 INFO:teuthology.orchestra.run.smithi136.stdout:kernel-5.14.0-505.el9.x86_64 2024-09-15T20:11:02.056 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-505.el9.x86_64 2024-09-15T20:11:02.057 DEBUG:teuthology.orchestra.run.smithi136:> sudo rpm -qi grub2-tools 2024-09-15T20:11:02.088 INFO:teuthology.orchestra.run.smithi136.stdout:Name : grub2-tools 2024-09-15T20:11:02.088 INFO:teuthology.orchestra.run.smithi136.stdout:Epoch : 1 2024-09-15T20:11:02.088 INFO:teuthology.orchestra.run.smithi136.stdout:Version : 2.06 2024-09-15T20:11:02.088 INFO:teuthology.orchestra.run.smithi136.stdout:Release : 82.el9 2024-09-15T20:11:02.088 INFO:teuthology.orchestra.run.smithi136.stdout:Architecture: x86_64 2024-09-15T20:11:02.088 INFO:teuthology.orchestra.run.smithi136.stdout:Install Date: Wed 28 Aug 2024 09:06:27 PM UTC 2024-09-15T20:11:02.088 INFO:teuthology.orchestra.run.smithi136.stdout:Group : Unspecified 2024-09-15T20:11:02.088 INFO:teuthology.orchestra.run.smithi136.stdout:Size : 8274670 2024-09-15T20:11:02.088 INFO:teuthology.orchestra.run.smithi136.stdout:License : GPLv3+ 2024-09-15T20:11:02.088 INFO:teuthology.orchestra.run.smithi136.stdout:Signature : RSA/SHA256, Mon 01 Jul 2024 07:46:42 AM UTC, Key ID 05b555b38483c65d 2024-09-15T20:11:02.088 INFO:teuthology.orchestra.run.smithi136.stdout:Source RPM : grub2-2.06-82.el9.src.rpm 2024-09-15T20:11:02.088 INFO:teuthology.orchestra.run.smithi136.stdout:Build Date : Fri 28 Jun 2024 09:24:44 AM UTC 2024-09-15T20:11:02.088 INFO:teuthology.orchestra.run.smithi136.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-09-15T20:11:02.089 INFO:teuthology.orchestra.run.smithi136.stdout:Packager : builder@centos.org 2024-09-15T20:11:02.089 INFO:teuthology.orchestra.run.smithi136.stdout:Vendor : CentOS 2024-09-15T20:11:02.089 INFO:teuthology.orchestra.run.smithi136.stdout:URL : http://www.gnu.org/software/grub/ 2024-09-15T20:11:02.089 INFO:teuthology.orchestra.run.smithi136.stdout:Summary : Support tools for GRUB. 2024-09-15T20:11:02.089 INFO:teuthology.orchestra.run.smithi136.stdout:Description : 2024-09-15T20:11:02.089 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-09-15T20:11:02.089 INFO:teuthology.orchestra.run.smithi136.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-09-15T20:11:02.089 INFO:teuthology.orchestra.run.smithi136.stdout:customizable bootloader with modular architecture. It supports a rich 2024-09-15T20:11:02.089 INFO:teuthology.orchestra.run.smithi136.stdout:variety of kernel formats, file systems, computer architectures and 2024-09-15T20:11:02.089 INFO:teuthology.orchestra.run.smithi136.stdout:hardware devices. 2024-09-15T20:11:02.090 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-09-15T20:11:02.090 INFO:teuthology.orchestra.run.smithi136.stdout:This subpackage provides tools for support of all platforms. 2024-09-15T20:11:02.090 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-09-15T20:11:02.090 INFO:teuthology.task.kernel:Updating grub on smithi136 to boot 5.14.0-505.el9.x86_64 2024-09-15T20:11:02.091 DEBUG:teuthology.orchestra.run.smithi136:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-09-15T20:11:02.777 INFO:teuthology.orchestra.run.smithi136.stderr:Generating grub configuration file ... 2024-09-15T20:11:04.076 INFO:teuthology.orchestra.run.smithi136.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-09-15T20:11:04.111 INFO:teuthology.orchestra.run.smithi136.stderr:done 2024-09-15T20:11:04.114 DEBUG:teuthology.orchestra.run.smithi136:> mktemp 2024-09-15T20:11:04.130 INFO:teuthology.orchestra.run.smithi136.stdout:/tmp/tmp.SVMbKNN5F6 2024-09-15T20:11:04.130 DEBUG:teuthology.orchestra.run.smithi136:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.SVMbKNN5F6 2024-09-15T20:11:04.197 DEBUG:teuthology.orchestra.run.smithi136:> sudo chmod 0666 /tmp/tmp.SVMbKNN5F6 2024-09-15T20:11:04.338 DEBUG:teuthology.orchestra.remote:smithi136:/tmp/tmp.SVMbKNN5F6 is 6KB 2024-09-15T20:11:04.387 DEBUG:teuthology.orchestra.run.smithi136:> rm -fr /tmp/tmp.SVMbKNN5F6 2024-09-15T20:11:04.402 DEBUG:teuthology.orchestra.run.smithi136:> sudo /bin/ls /boot/loader/entries || true 2024-09-15T20:11:04.471 INFO:teuthology.orchestra.run.smithi136.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-0-rescue.conf 2024-09-15T20:11:04.471 INFO:teuthology.orchestra.run.smithi136.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-239.el9.x86_64.conf 2024-09-15T20:11:04.472 INFO:teuthology.orchestra.run.smithi136.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-503.el9.x86_64.conf 2024-09-15T20:11:04.472 INFO:teuthology.orchestra.run.smithi136.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-505.el9.x86_64.conf 2024-09-15T20:11:04.473 DEBUG:teuthology.orchestra.run.smithi136:> sudo grub2-set-default fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-505.el9.x86_64 2024-09-15T20:11:04.626 DEBUG:teuthology.orchestra.run.smithi136:> sudo shutdown -r now 2024-09-15T20:11:34.657 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2024-09-15T20:11:34.657 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi136.front.sepia.ceph.com' 2024-09-15T20:11:34.658 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi136.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-15T20:11:53.021 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.136 2024-09-15T20:12:02.025 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi136.front.sepia.ceph.com' 2024-09-15T20:12:02.025 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi136.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-15T20:12:05.085 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.136 2024-09-15T20:12:17.097 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi136.front.sepia.ceph.com' 2024-09-15T20:12:17.097 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi136.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-15T20:12:35.517 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.136 2024-09-15T20:12:50.529 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi136.front.sepia.ceph.com' 2024-09-15T20:12:50.529 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi136.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-15T20:12:50.908 DEBUG:teuthology.orchestra.run.smithi136:> true 2024-09-15T20:12:51.301 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi136.front.sepia.ceph.com' 2024-09-15T20:12:51.301 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "5.14.0-505.el9.x86_64"... 2024-09-15T20:12:51.302 DEBUG:teuthology.orchestra.run.smithi136:> uname -r 2024-09-15T20:12:51.316 INFO:teuthology.orchestra.run.smithi136.stdout:5.14.0-505.el9.x86_64 2024-09-15T20:12:51.316 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-505.el9.x86_64 vs 5.14.0-505.el9.x86_64 2024-09-15T20:12:51.316 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-09-15T20:12:51.316 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-09-15T20:12:52.317 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2024-09-15T20:12:52.318 DEBUG:teuthology.orchestra.run.smithi136:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-09-15T20:12:52.436 INFO:teuthology.orchestra.run.smithi136.stdout:ttyS1 2024-09-15T20:12:52.462 DEBUG:teuthology.parallel:result is None 2024-09-15T20:12:52.462 INFO:teuthology.run_tasks:Running task internal.base... 2024-09-15T20:12:52.469 INFO:teuthology.task.internal:Creating test directory... 2024-09-15T20:12:52.469 DEBUG:teuthology.orchestra.run.smithi136:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-09-15T20:12:52.511 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-09-15T20:12:52.518 INFO:teuthology.run_tasks:Running task internal.archive... 2024-09-15T20:12:52.524 INFO:teuthology.task.internal:Creating archive directory... 2024-09-15T20:12:52.525 DEBUG:teuthology.orchestra.run.smithi136:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-09-15T20:12:52.615 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-09-15T20:12:52.623 INFO:teuthology.task.internal:Enabling coredump saving... 2024-09-15T20:12:52.623 DEBUG:teuthology.orchestra.run.smithi136:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-09-15T20:12:52.773 INFO:teuthology.orchestra.run.smithi136.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T20:12:52.793 INFO:teuthology.orchestra.run.smithi136.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T20:12:52.797 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-09-15T20:12:52.805 INFO:teuthology.task.internal:Configuring sudo... 2024-09-15T20:12:52.805 DEBUG:teuthology.orchestra.run.smithi136:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-09-15T20:12:52.879 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-09-15T20:12:52.888 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-09-15T20:12:52.888 DEBUG:teuthology.orchestra.run.smithi136:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-09-15T20:12:52.937 DEBUG:teuthology.orchestra.run.smithi136:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-15T20:12:53.002 DEBUG:teuthology.orchestra.run.smithi136:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-15T20:12:53.080 DEBUG:teuthology.orchestra.run.smithi136:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-15T20:12:53.144 DEBUG:teuthology.orchestra.run.smithi136:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-15T20:12:53.217 DEBUG:teuthology.orchestra.run.smithi136:> set -ex 2024-09-15T20:12:53.217 DEBUG:teuthology.orchestra.run.smithi136:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-09-15T20:12:53.276 DEBUG:teuthology.orchestra.run.smithi136:> sudo service rsyslog restart 2024-09-15T20:12:53.405 INFO:teuthology.orchestra.run.smithi136.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-09-15T20:12:53.739 INFO:teuthology.run_tasks:Running task internal.timer... 2024-09-15T20:12:53.748 INFO:teuthology.task.internal:Starting timer... 2024-09-15T20:12:53.748 INFO:teuthology.run_tasks:Running task pcp... 2024-09-15T20:12:53.758 INFO:teuthology.run_tasks:Running task selinux... 2024-09-15T20:12:53.766 DEBUG:teuthology.orchestra.run.smithi136:> sudo service auditd rotate 2024-09-15T20:12:53.897 INFO:teuthology.orchestra.run.smithi136.stdout:Rotating logs: 2024-09-15T20:12:53.899 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-09-15T20:12:53.899 DEBUG:teuthology.orchestra.run.smithi136:> /usr/sbin/getenforce 2024-09-15T20:12:53.962 INFO:teuthology.orchestra.run.smithi136.stdout:Permissive 2024-09-15T20:12:53.962 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi136.front.sepia.ceph.com': 'permissive'} 2024-09-15T20:12:53.963 DEBUG:teuthology.orchestra.run.smithi136:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 2024-09-15T20:12:53.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-15T20:12:53.987 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-09-15T20:12:53.987 DEBUG:teuthology.orchestra.run.smithi136:> sudo /usr/sbin/setenforce permissive 2024-09-15T20:12:54.060 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-09-15T20:12:54.069 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-09-15T20:12:54.080 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-09-15T20:12:54.081 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi136.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-09-15T20:16:37.523 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi136.front.sepia.ceph.com')] 2024-09-15T20:16:37.524 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi136.front.sepia.ceph.com' 2024-09-15T20:16:37.525 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi136.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-15T20:16:37.609 DEBUG:teuthology.orchestra.run.smithi136:> true 2024-09-15T20:16:37.685 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi136.front.sepia.ceph.com' 2024-09-15T20:16:37.686 INFO:teuthology.run_tasks:Running task clock... 2024-09-15T20:16:37.696 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-09-15T20:16:37.696 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-09-15T20:16:37.697 DEBUG:teuthology.orchestra.run.smithi136:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-15T20:16:37.757 INFO:teuthology.orchestra.run.smithi136.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-09-15T20:16:37.772 INFO:teuthology.orchestra.run.smithi136.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-09-15T20:16:37.797 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: ntpd: command not found 2024-09-15T20:16:37.808 INFO:teuthology.orchestra.run.smithi136.stdout:506 Cannot talk to daemon 2024-09-15T20:16:37.822 INFO:teuthology.orchestra.run.smithi136.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-09-15T20:16:37.836 INFO:teuthology.orchestra.run.smithi136.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-09-15T20:16:37.889 INFO:teuthology.orchestra.run.smithi136.stderr:bash: line 1: ntpq: command not found 2024-09-15T20:16:37.893 INFO:teuthology.orchestra.run.smithi136.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-09-15T20:16:37.893 INFO:teuthology.orchestra.run.smithi136.stdout:=============================================================================== 2024-09-15T20:16:37.893 INFO:teuthology.orchestra.run.smithi136.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-15T20:16:37.893 INFO:teuthology.orchestra.run.smithi136.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-15T20:16:37.893 INFO:teuthology.orchestra.run.smithi136.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-15T20:16:37.893 INFO:teuthology.orchestra.run.smithi136.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-15T20:16:37.894 INFO:teuthology.run_tasks:Running task install... 2024-09-15T20:16:37.902 DEBUG:teuthology.task.install:project ceph 2024-09-15T20:16:37.902 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': 'fc3e0bff7468fd55478adbae7f75508b749124b6'}} 2024-09-15T20:16:37.902 DEBUG:teuthology.task.install:config {'flavor': 'default', 'sha1': 'fc3e0bff7468fd55478adbae7f75508b749124b6'} 2024-09-15T20:16:37.902 INFO:teuthology.task.install:Using flavor: default 2024-09-15T20:16:37.907 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-09-15T20:16:37.907 INFO:teuthology.task.install:extra packages: [] 2024-09-15T20:16:37.908 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'fc3e0bff7468fd55478adbae7f75508b749124b6', 'tag': None, 'wait_for_package': False} 2024-09-15T20:16:37.908 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=fc3e0bff7468fd55478adbae7f75508b749124b6 2024-09-15T20:16:38.064 INFO:teuthology.task.install.rpm:Pulling from https://2.chacra.ceph.com/r/ceph/main/fc3e0bff7468fd55478adbae7f75508b749124b6/centos/9/flavors/default/ 2024-09-15T20:16:38.065 INFO:teuthology.task.install.rpm:Package version is 19.3.0-4930.gfc3e0bff 2024-09-15T20:16:38.177 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://2.chacra.ceph.com/r/ceph/main/fc3e0bff7468fd55478adbae7f75508b749124b6/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://2.chacra.ceph.com/r/ceph/main/fc3e0bff7468fd55478adbae7f75508b749124b6/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://2.chacra.ceph.com/r/ceph/main/fc3e0bff7468fd55478adbae7f75508b749124b6/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-09-15T20:16:38.177 DEBUG:teuthology.orchestra.run.smithi136:> set -ex 2024-09-15T20:16:38.177 DEBUG:teuthology.orchestra.run.smithi136:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-09-15T20:16:38.203 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-09-15T20:16:38.203 DEBUG:teuthology.orchestra.run.smithi136:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/fc3e0bff7468fd55478adbae7f75508b749124b6/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-09-15T20:16:38.270 DEBUG:teuthology.orchestra.run.smithi136:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-09-15T20:16:38.346 DEBUG:teuthology.orchestra.run.smithi136:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-09-15T20:16:38.411 INFO:teuthology.orchestra.run.smithi136.stdout:check_obsoletes = 1 2024-09-15T20:16:38.412 DEBUG:teuthology.orchestra.run.smithi136:> sudo yum clean all 2024-09-15T20:16:38.648 INFO:teuthology.orchestra.run.smithi136.stdout:45 files removed 2024-09-15T20:16:38.675 DEBUG:teuthology.orchestra.run.smithi136:> sudo yum -y install ceph-radosgw 2024-09-15T20:16:39.250 INFO:teuthology.orchestra.run.smithi136.stdout:ceph packages for x86_64 293 kB/s | 84 kB 00:00 2024-09-15T20:16:39.542 INFO:teuthology.orchestra.run.smithi136.stdout:ceph noarch packages 97 kB/s | 24 kB 00:00 2024-09-15T20:16:39.818 INFO:teuthology.orchestra.run.smithi136.stdout:ceph source packages 8.1 kB/s | 1.9 kB 00:00 2024-09-15T20:16:41.685 INFO:teuthology.orchestra.run.smithi136.stdout:CentOS Stream 9 - BaseOS 4.5 MB/s | 8.2 MB 00:01 2024-09-15T20:16:44.253 INFO:teuthology.orchestra.run.smithi136.stdout:CentOS Stream 9 - AppStream 14 MB/s | 20 MB 00:01 2024-09-15T20:16:50.319 INFO:teuthology.orchestra.run.smithi136.stdout:CentOS Stream 9 - CRB 2.9 MB/s | 6.5 MB 00:02 2024-09-15T20:16:51.778 INFO:teuthology.orchestra.run.smithi136.stdout:CentOS Stream 9 - Extras packages 88 kB/s | 19 kB 00:00 2024-09-15T20:16:52.674 INFO:teuthology.orchestra.run.smithi136.stdout:Extra Packages for Enterprise Linux 29 MB/s | 23 MB 00:00 2024-09-15T20:16:58.080 INFO:teuthology.orchestra.run.smithi136.stdout:lab-extras 41 kB/s | 1.7 kB 00:00 2024-09-15T20:17:00.464 INFO:teuthology.orchestra.run.smithi136.stdout:Dependencies resolved. 2024-09-15T20:17:00.475 INFO:teuthology.orchestra.run.smithi136.stdout:================================================================================ 2024-09-15T20:17:00.475 INFO:teuthology.orchestra.run.smithi136.stdout: Package Arch Version Repository Size 2024-09-15T20:17:00.475 INFO:teuthology.orchestra.run.smithi136.stdout:================================================================================ 2024-09-15T20:17:00.475 INFO:teuthology.orchestra.run.smithi136.stdout:Installing: 2024-09-15T20:17:00.476 INFO:teuthology.orchestra.run.smithi136.stdout: ceph-radosgw x86_64 2:19.3.0-4930.gfc3e0bff.el9 ceph 11 M 2024-09-15T20:17:00.476 INFO:teuthology.orchestra.run.smithi136.stdout:Upgrading: 2024-09-15T20:17:00.476 INFO:teuthology.orchestra.run.smithi136.stdout: librados2 x86_64 2:19.3.0-4930.gfc3e0bff.el9 ceph 3.5 M 2024-09-15T20:17:00.476 INFO:teuthology.orchestra.run.smithi136.stdout: librbd1 x86_64 2:19.3.0-4930.gfc3e0bff.el9 ceph 3.2 M 2024-09-15T20:17:00.476 INFO:teuthology.orchestra.run.smithi136.stdout:Installing dependencies: 2024-09-15T20:17:00.476 INFO:teuthology.orchestra.run.smithi136.stdout: boost-program-options x86_64 1.75.0-8.el9 appstream 107 k 2024-09-15T20:17:00.476 INFO:teuthology.orchestra.run.smithi136.stdout: ceph-base x86_64 2:19.3.0-4930.gfc3e0bff.el9 ceph 5.5 M 2024-09-15T20:17:00.476 INFO:teuthology.orchestra.run.smithi136.stdout: ceph-common x86_64 2:19.3.0-4930.gfc3e0bff.el9 ceph 22 M 2024-09-15T20:17:00.476 INFO:teuthology.orchestra.run.smithi136.stdout: ceph-selinux x86_64 2:19.3.0-4930.gfc3e0bff.el9 ceph 25 k 2024-09-15T20:17:00.476 INFO:teuthology.orchestra.run.smithi136.stdout: gperftools-libs x86_64 2.9.1-3.el9 epel 308 k 2024-09-15T20:17:00.476 INFO:teuthology.orchestra.run.smithi136.stdout: libarrow x86_64 9.0.0-11.el9 epel 4.4 M 2024-09-15T20:17:00.476 INFO:teuthology.orchestra.run.smithi136.stdout: libarrow-doc noarch 9.0.0-11.el9 epel 26 k 2024-09-15T20:17:00.477 INFO:teuthology.orchestra.run.smithi136.stdout: libcephfs2 x86_64 2:19.3.0-4930.gfc3e0bff.el9 ceph 755 k 2024-09-15T20:17:00.477 INFO:teuthology.orchestra.run.smithi136.stdout: libnbd x86_64 1.20.2-2.el9 appstream 171 k 2024-09-15T20:17:00.477 INFO:teuthology.orchestra.run.smithi136.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-09-15T20:17:00.477 INFO:teuthology.orchestra.run.smithi136.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-09-15T20:17:00.477 INFO:teuthology.orchestra.run.smithi136.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-09-15T20:17:00.477 INFO:teuthology.orchestra.run.smithi136.stdout: libradosstriper1 x86_64 2:19.3.0-4930.gfc3e0bff.el9 ceph 508 k 2024-09-15T20:17:00.477 INFO:teuthology.orchestra.run.smithi136.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-09-15T20:17:00.477 INFO:teuthology.orchestra.run.smithi136.stdout: librgw2 x86_64 2:19.3.0-4930.gfc3e0bff.el9 ceph 5.5 M 2024-09-15T20:17:00.477 INFO:teuthology.orchestra.run.smithi136.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-09-15T20:17:00.477 INFO:teuthology.orchestra.run.smithi136.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-09-15T20:17:00.477 INFO:teuthology.orchestra.run.smithi136.stdout: parquet-libs x86_64 9.0.0-11.el9 epel 839 k 2024-09-15T20:17:00.477 INFO:teuthology.orchestra.run.smithi136.stdout: python3-ceph-argparse x86_64 2:19.3.0-4930.gfc3e0bff.el9 ceph 45 k 2024-09-15T20:17:00.477 INFO:teuthology.orchestra.run.smithi136.stdout: python3-ceph-common x86_64 2:19.3.0-4930.gfc3e0bff.el9 ceph 143 k 2024-09-15T20:17:00.478 INFO:teuthology.orchestra.run.smithi136.stdout: python3-cephfs x86_64 2:19.3.0-4930.gfc3e0bff.el9 ceph 161 k 2024-09-15T20:17:00.478 INFO:teuthology.orchestra.run.smithi136.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-09-15T20:17:00.478 INFO:teuthology.orchestra.run.smithi136.stdout: python3-rados x86_64 2:19.3.0-4930.gfc3e0bff.el9 ceph 320 k 2024-09-15T20:17:00.478 INFO:teuthology.orchestra.run.smithi136.stdout: python3-rbd x86_64 2:19.3.0-4930.gfc3e0bff.el9 ceph 305 k 2024-09-15T20:17:00.478 INFO:teuthology.orchestra.run.smithi136.stdout: python3-rgw x86_64 2:19.3.0-4930.gfc3e0bff.el9 ceph 99 k 2024-09-15T20:17:00.478 INFO:teuthology.orchestra.run.smithi136.stdout: qatlib x86_64 24.02.0-1.el9 appstream 221 k 2024-09-15T20:17:00.478 INFO:teuthology.orchestra.run.smithi136.stdout: qatzip-libs x86_64 1.2.0-1.el9 appstream 47 k 2024-09-15T20:17:00.478 INFO:teuthology.orchestra.run.smithi136.stdout: re2 x86_64 1:20211101-20.el9 epel 191 k 2024-09-15T20:17:00.478 INFO:teuthology.orchestra.run.smithi136.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-09-15T20:17:00.478 INFO:teuthology.orchestra.run.smithi136.stdout:Installing weak dependencies: 2024-09-15T20:17:00.478 INFO:teuthology.orchestra.run.smithi136.stdout: qatlib-service x86_64 24.02.0-1.el9 appstream 36 k 2024-09-15T20:17:00.478 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-09-15T20:17:00.478 INFO:teuthology.orchestra.run.smithi136.stdout:Transaction Summary 2024-09-15T20:17:00.479 INFO:teuthology.orchestra.run.smithi136.stdout:================================================================================ 2024-09-15T20:17:00.479 INFO:teuthology.orchestra.run.smithi136.stdout:Install 31 Packages 2024-09-15T20:17:00.479 INFO:teuthology.orchestra.run.smithi136.stdout:Upgrade 2 Packages 2024-09-15T20:17:00.479 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-09-15T20:17:00.479 INFO:teuthology.orchestra.run.smithi136.stdout:Total download size: 62 M 2024-09-15T20:17:00.479 INFO:teuthology.orchestra.run.smithi136.stdout:Downloading Packages: 2024-09-15T20:17:01.031 INFO:teuthology.orchestra.run.smithi136.stdout:(1/33): ceph-base-19.3.0-4930.gfc3e0bff.el9.x86 13 MB/s | 5.5 MB 00:00 2024-09-15T20:17:01.081 INFO:teuthology.orchestra.run.smithi136.stdout:(2/33): ceph-selinux-19.3.0-4930.gfc3e0bff.el9. 505 kB/s | 25 kB 00:00 2024-09-15T20:17:01.157 INFO:teuthology.orchestra.run.smithi136.stdout:(3/33): libcephfs2-19.3.0-4930.gfc3e0bff.el9.x8 9.9 MB/s | 755 kB 00:00 2024-09-15T20:17:01.288 INFO:teuthology.orchestra.run.smithi136.stdout:(4/33): libradosstriper1-19.3.0-4930.gfc3e0bff. 3.8 MB/s | 508 kB 00:00 2024-09-15T20:17:01.607 INFO:teuthology.orchestra.run.smithi136.stdout:(5/33): ceph-radosgw-19.3.0-4930.gfc3e0bff.el9. 11 MB/s | 11 MB 00:00 2024-09-15T20:17:01.691 INFO:teuthology.orchestra.run.smithi136.stdout:(6/33): librgw2-19.3.0-4930.gfc3e0bff.el9.x86_6 14 MB/s | 5.5 MB 00:00 2024-09-15T20:17:01.716 INFO:teuthology.orchestra.run.smithi136.stdout:(7/33): python3-ceph-argparse-19.3.0-4930.gfc3e 416 kB/s | 45 kB 00:00 2024-09-15T20:17:01.741 INFO:teuthology.orchestra.run.smithi136.stdout:(8/33): python3-ceph-common-19.3.0-4930.gfc3e0b 2.8 MB/s | 143 kB 00:00 2024-09-15T20:17:01.767 INFO:teuthology.orchestra.run.smithi136.stdout:(9/33): python3-cephfs-19.3.0-4930.gfc3e0bff.el 3.1 MB/s | 161 kB 00:00 2024-09-15T20:17:01.792 INFO:teuthology.orchestra.run.smithi136.stdout:(10/33): python3-rados-19.3.0-4930.gfc3e0bff.el 6.2 MB/s | 320 kB 00:00 2024-09-15T20:17:01.817 INFO:teuthology.orchestra.run.smithi136.stdout:(11/33): python3-rbd-19.3.0-4930.gfc3e0bff.el9. 5.9 MB/s | 305 kB 00:00 2024-09-15T20:17:01.843 INFO:teuthology.orchestra.run.smithi136.stdout:(12/33): python3-rgw-19.3.0-4930.gfc3e0bff.el9. 1.9 MB/s | 99 kB 00:00 2024-09-15T20:17:02.060 INFO:teuthology.orchestra.run.smithi136.stdout:(13/33): ceph-common-19.3.0-4930.gfc3e0bff.el9. 15 MB/s | 22 MB 00:01 2024-09-15T20:17:02.168 INFO:teuthology.orchestra.run.smithi136.stdout:(14/33): boost-program-options-1.75.0-8.el9.x86 305 kB/s | 107 kB 00:00 2024-09-15T20:17:02.193 INFO:teuthology.orchestra.run.smithi136.stdout:(15/33): libnbd-1.20.2-2.el9.x86_64.rpm 486 kB/s | 171 kB 00:00 2024-09-15T20:17:02.219 INFO:teuthology.orchestra.run.smithi136.stdout:(16/33): librabbitmq-0.11.0-7.el9.x86_64.rpm 900 kB/s | 45 kB 00:00 2024-09-15T20:17:02.244 INFO:teuthology.orchestra.run.smithi136.stdout:(17/33): libpmemobj-1.12.1-1.el9.x86_64.rpm 871 kB/s | 160 kB 00:00 2024-09-15T20:17:02.294 INFO:teuthology.orchestra.run.smithi136.stdout:(18/33): python3-prettytable-0.7.2-27.el9.noarc 843 kB/s | 42 kB 00:00 2024-09-15T20:17:02.344 INFO:teuthology.orchestra.run.smithi136.stdout:(19/33): lttng-ust-2.12.0-6.el9.x86_64.rpm 2.3 MB/s | 292 kB 00:00 2024-09-15T20:17:02.386 INFO:teuthology.orchestra.run.smithi136.stdout:(20/33): librdkafka-1.6.1-102.el9.x86_64.rpm 3.4 MB/s | 662 kB 00:00 2024-09-15T20:17:02.412 INFO:teuthology.orchestra.run.smithi136.stdout:(21/33): qatlib-24.02.0-1.el9.x86_64.rpm 1.8 MB/s | 221 kB 00:00 2024-09-15T20:17:02.437 INFO:teuthology.orchestra.run.smithi136.stdout:(22/33): qatlib-service-24.02.0-1.el9.x86_64.rp 389 kB/s | 36 kB 00:00 2024-09-15T20:17:02.462 INFO:teuthology.orchestra.run.smithi136.stdout:(23/33): qatzip-libs-1.2.0-1.el9.x86_64.rpm 624 kB/s | 47 kB 00:00 2024-09-15T20:17:02.513 INFO:teuthology.orchestra.run.smithi136.stdout:(24/33): gperftools-libs-2.9.1-3.el9.x86_64.rpm 3.0 MB/s | 308 kB 00:00 2024-09-15T20:17:02.539 INFO:teuthology.orchestra.run.smithi136.stdout:(25/33): liboath-2.6.7-2.el9.x86_64.rpm 2.0 MB/s | 49 kB 00:00 2024-09-15T20:17:02.563 INFO:teuthology.orchestra.run.smithi136.stdout:(26/33): libarrow-doc-9.0.0-11.el9.noarch.rpm 300 kB/s | 26 kB 00:00 2024-09-15T20:17:02.589 INFO:teuthology.orchestra.run.smithi136.stdout:(27/33): libunwind-1.6.2-1.el9.x86_64.rpm 1.3 MB/s | 67 kB 00:00 2024-09-15T20:17:02.614 INFO:teuthology.orchestra.run.smithi136.stdout:(28/33): re2-20211101-20.el9.x86_64.rpm 7.5 MB/s | 191 kB 00:00 2024-09-15T20:17:02.656 INFO:teuthology.orchestra.run.smithi136.stdout:(29/33): parquet-libs-9.0.0-11.el9.x86_64.rpm 8.9 MB/s | 839 kB 00:00 2024-09-15T20:17:02.706 INFO:teuthology.orchestra.run.smithi136.stdout:(30/33): thrift-0.15.0-2.el9.x86_64.rpm 17 MB/s | 1.6 MB 00:00 2024-09-15T20:17:02.781 INFO:teuthology.orchestra.run.smithi136.stdout:(31/33): libarrow-9.0.0-11.el9.x86_64.rpm 13 MB/s | 4.4 MB 00:00 2024-09-15T20:17:02.923 INFO:teuthology.orchestra.run.smithi136.stdout:(32/33): librados2-19.3.0-4930.gfc3e0bff.el9.x8 13 MB/s | 3.5 MB 00:00 2024-09-15T20:17:02.973 INFO:teuthology.orchestra.run.smithi136.stdout:(33/33): librbd1-19.3.0-4930.gfc3e0bff.el9.x86_ 12 MB/s | 3.2 MB 00:00 2024-09-15T20:17:02.978 INFO:teuthology.orchestra.run.smithi136.stdout:-------------------------------------------------------------------------------- 2024-09-15T20:17:02.978 INFO:teuthology.orchestra.run.smithi136.stdout:Total 25 MB/s | 62 MB 00:02 2024-09-15T20:17:03.089 INFO:teuthology.orchestra.run.smithi136.stdout:Running transaction check 2024-09-15T20:17:03.126 INFO:teuthology.orchestra.run.smithi136.stdout:Transaction check succeeded. 2024-09-15T20:17:03.126 INFO:teuthology.orchestra.run.smithi136.stdout:Running transaction test 2024-09-15T20:17:03.833 INFO:teuthology.orchestra.run.smithi136.stdout:Transaction test succeeded. 2024-09-15T20:17:03.833 INFO:teuthology.orchestra.run.smithi136.stdout:Running transaction 2024-09-15T20:17:05.403 INFO:teuthology.orchestra.run.smithi136.stdout: Preparing : 1/1 2024-09-15T20:17:05.493 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/35 2024-09-15T20:17:05.529 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 2/35 2024-09-15T20:17:05.554 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 3/35 2024-09-15T20:17:05.576 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 4/35 2024-09-15T20:17:05.598 INFO:teuthology.orchestra.run.smithi136.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 4/35 2024-09-15T20:17:05.797 INFO:teuthology.orchestra.run.smithi136.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 5/35 2024-09-15T20:17:05.848 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 5/35 2024-09-15T20:17:06.175 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : thrift-0.15.0-2.el9.x86_64 6/35 2024-09-15T20:17:06.272 INFO:teuthology.orchestra.run.smithi136.stdout: Upgrading : librados2-2:19.3.0-4930.gfc3e0bff.el9.x86_64 7/35 2024-09-15T20:17:06.365 INFO:teuthology.orchestra.run.smithi136.stdout: Running scriptlet: librados2-2:19.3.0-4930.gfc3e0bff.el9.x86_64 7/35 2024-09-15T20:17:06.461 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-rados-2:19.3.0-4930.gfc3e0bff.el9.x86_64 8/35 2024-09-15T20:17:06.507 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : libcephfs2-2:19.3.0-4930.gfc3e0bff.el9.x86_64 9/35 2024-09-15T20:17:06.562 INFO:teuthology.orchestra.run.smithi136.stdout: Running scriptlet: libcephfs2-2:19.3.0-4930.gfc3e0bff.el9.x86_64 9/35 2024-09-15T20:17:06.590 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : liboath-2.6.7-2.el9.x86_64 10/35 2024-09-15T20:17:06.643 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-ceph-argparse-2:19.3.0-4930.gfc3e0bff.el9. 11/35 2024-09-15T20:17:06.712 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-cephfs-2:19.3.0-4930.gfc3e0bff.el9.x86_64 12/35 2024-09-15T20:17:06.752 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : libradosstriper1-2:19.3.0-4930.gfc3e0bff.el9.x86_6 13/35 2024-09-15T20:17:06.807 INFO:teuthology.orchestra.run.smithi136.stdout: Running scriptlet: libradosstriper1-2:19.3.0-4930.gfc3e0bff.el9.x86_6 13/35 2024-09-15T20:17:06.841 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 14/35 2024-09-15T20:17:06.957 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : re2-1:20211101-20.el9.x86_64 15/35 2024-09-15T20:17:06.993 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : libarrow-9.0.0-11.el9.x86_64 16/35 2024-09-15T20:17:07.034 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 17/35 2024-09-15T20:17:07.074 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : gperftools-libs-2.9.1-3.el9.x86_64 18/35 2024-09-15T20:17:07.103 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : libarrow-doc-9.0.0-11.el9.noarch 19/35 2024-09-15T20:17:07.140 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 20/35 2024-09-15T20:17:07.174 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 21/35 2024-09-15T20:17:07.524 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : libnbd-1.20.2-2.el9.x86_64 22/35 2024-09-15T20:17:07.566 INFO:teuthology.orchestra.run.smithi136.stdout: Upgrading : librbd1-2:19.3.0-4930.gfc3e0bff.el9.x86_64 23/35 2024-09-15T20:17:07.659 INFO:teuthology.orchestra.run.smithi136.stdout: Running scriptlet: librbd1-2:19.3.0-4930.gfc3e0bff.el9.x86_64 23/35 2024-09-15T20:17:07.688 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-rbd-2:19.3.0-4930.gfc3e0bff.el9.x86_64 24/35 2024-09-15T20:17:07.733 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 25/35 2024-09-15T20:17:08.251 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : parquet-libs-9.0.0-11.el9.x86_64 26/35 2024-09-15T20:17:08.277 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : librgw2-2:19.3.0-4930.gfc3e0bff.el9.x86_64 27/35 2024-09-15T20:17:08.349 INFO:teuthology.orchestra.run.smithi136.stdout: Running scriptlet: librgw2-2:19.3.0-4930.gfc3e0bff.el9.x86_64 27/35 2024-09-15T20:17:08.432 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-rgw-2:19.3.0-4930.gfc3e0bff.el9.x86_64 28/35 2024-09-15T20:17:08.480 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-ceph-common-2:19.3.0-4930.gfc3e0bff.el9.x8 29/35 2024-09-15T20:17:10.822 INFO:teuthology.orchestra.run.smithi136.stdout: Running scriptlet: ceph-common-2:19.3.0-4930.gfc3e0bff.el9.x86_64 30/35 2024-09-15T20:17:10.868 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : ceph-common-2:19.3.0-4930.gfc3e0bff.el9.x86_64 30/35 2024-09-15T20:17:11.466 INFO:teuthology.orchestra.run.smithi136.stdout: Running scriptlet: ceph-common-2:19.3.0-4930.gfc3e0bff.el9.x86_64 30/35 2024-09-15T20:17:11.514 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : ceph-base-2:19.3.0-4930.gfc3e0bff.el9.x86_64 31/35 2024-09-15T20:17:11.605 INFO:teuthology.orchestra.run.smithi136.stdout: Running scriptlet: ceph-base-2:19.3.0-4930.gfc3e0bff.el9.x86_64 31/35 2024-09-15T20:17:11.606 INFO:teuthology.orchestra.run.smithi136.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-09-15T20:17:11.606 INFO:teuthology.orchestra.run.smithi136.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-09-15T20:17:11.606 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-09-15T20:17:11.648 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : ceph-selinux-2:19.3.0-4930.gfc3e0bff.el9.x86_64 32/35 2024-09-15T20:17:21.677 INFO:teuthology.orchestra.run.smithi136.stdout: Running scriptlet: ceph-selinux-2:19.3.0-4930.gfc3e0bff.el9.x86_64 32/35 2024-09-15T20:17:21.678 INFO:teuthology.orchestra.run.smithi136.stdout:skipping the directory /sys 2024-09-15T20:17:21.678 INFO:teuthology.orchestra.run.smithi136.stdout:skipping the directory /proc 2024-09-15T20:17:21.678 INFO:teuthology.orchestra.run.smithi136.stdout:skipping the directory /mnt 2024-09-15T20:17:21.678 INFO:teuthology.orchestra.run.smithi136.stdout:skipping the directory /var/tmp 2024-09-15T20:17:21.678 INFO:teuthology.orchestra.run.smithi136.stdout:skipping the directory /home 2024-09-15T20:17:21.678 INFO:teuthology.orchestra.run.smithi136.stdout:skipping the directory /root 2024-09-15T20:17:21.678 INFO:teuthology.orchestra.run.smithi136.stdout:skipping the directory /tmp 2024-09-15T20:17:21.678 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-09-15T20:17:22.696 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : ceph-radosgw-2:19.3.0-4930.gfc3e0bff.el9.x86_64 33/35 2024-09-15T20:17:22.722 INFO:teuthology.orchestra.run.smithi136.stdout: Running scriptlet: ceph-radosgw-2:19.3.0-4930.gfc3e0bff.el9.x86_64 33/35 2024-09-15T20:17:22.722 INFO:teuthology.orchestra.run.smithi136.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-15T20:17:22.722 INFO:teuthology.orchestra.run.smithi136.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-09-15T20:17:22.722 INFO:teuthology.orchestra.run.smithi136.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-09-15T20:17:22.722 INFO:teuthology.orchestra.run.smithi136.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-09-15T20:17:22.722 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-09-15T20:17:22.724 INFO:teuthology.orchestra.run.smithi136.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 34/35 2024-09-15T20:17:22.807 INFO:teuthology.orchestra.run.smithi136.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 34/35 2024-09-15T20:17:22.807 INFO:teuthology.orchestra.run.smithi136.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 35/35 2024-09-15T20:17:24.013 INFO:teuthology.orchestra.run.smithi136.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 35/35 2024-09-15T20:17:24.014 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : ceph-base-2:19.3.0-4930.gfc3e0bff.el9.x86_64 1/35 2024-09-15T20:17:24.014 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : ceph-common-2:19.3.0-4930.gfc3e0bff.el9.x86_64 2/35 2024-09-15T20:17:24.014 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : ceph-radosgw-2:19.3.0-4930.gfc3e0bff.el9.x86_64 3/35 2024-09-15T20:17:24.014 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : ceph-selinux-2:19.3.0-4930.gfc3e0bff.el9.x86_64 4/35 2024-09-15T20:17:24.014 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : libcephfs2-2:19.3.0-4930.gfc3e0bff.el9.x86_64 5/35 2024-09-15T20:17:24.014 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : libradosstriper1-2:19.3.0-4930.gfc3e0bff.el9.x86_6 6/35 2024-09-15T20:17:24.014 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : librgw2-2:19.3.0-4930.gfc3e0bff.el9.x86_64 7/35 2024-09-15T20:17:24.015 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-ceph-argparse-2:19.3.0-4930.gfc3e0bff.el9. 8/35 2024-09-15T20:17:24.015 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-ceph-common-2:19.3.0-4930.gfc3e0bff.el9.x8 9/35 2024-09-15T20:17:24.015 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-cephfs-2:19.3.0-4930.gfc3e0bff.el9.x86_64 10/35 2024-09-15T20:17:24.015 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-rados-2:19.3.0-4930.gfc3e0bff.el9.x86_64 11/35 2024-09-15T20:17:24.015 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-rbd-2:19.3.0-4930.gfc3e0bff.el9.x86_64 12/35 2024-09-15T20:17:24.015 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-rgw-2:19.3.0-4930.gfc3e0bff.el9.x86_64 13/35 2024-09-15T20:17:24.015 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/35 2024-09-15T20:17:24.015 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : libnbd-1.20.2-2.el9.x86_64 15/35 2024-09-15T20:17:24.015 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 16/35 2024-09-15T20:17:24.015 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 17/35 2024-09-15T20:17:24.015 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 18/35 2024-09-15T20:17:24.015 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 19/35 2024-09-15T20:17:24.015 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 20/35 2024-09-15T20:17:24.016 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 21/35 2024-09-15T20:17:24.016 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 22/35 2024-09-15T20:17:24.016 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 23/35 2024-09-15T20:17:24.016 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : gperftools-libs-2.9.1-3.el9.x86_64 24/35 2024-09-15T20:17:24.016 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : libarrow-9.0.0-11.el9.x86_64 25/35 2024-09-15T20:17:24.016 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : libarrow-doc-9.0.0-11.el9.noarch 26/35 2024-09-15T20:17:24.016 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 27/35 2024-09-15T20:17:24.016 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 28/35 2024-09-15T20:17:24.016 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : parquet-libs-9.0.0-11.el9.x86_64 29/35 2024-09-15T20:17:24.016 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : re2-1:20211101-20.el9.x86_64 30/35 2024-09-15T20:17:24.016 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 31/35 2024-09-15T20:17:24.016 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : librados2-2:19.3.0-4930.gfc3e0bff.el9.x86_64 32/35 2024-09-15T20:17:24.016 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 33/35 2024-09-15T20:17:24.017 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : librbd1-2:19.3.0-4930.gfc3e0bff.el9.x86_64 34/35 2024-09-15T20:17:24.913 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 35/35 2024-09-15T20:17:24.913 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-09-15T20:17:24.914 INFO:teuthology.orchestra.run.smithi136.stdout:Upgraded: 2024-09-15T20:17:24.914 INFO:teuthology.orchestra.run.smithi136.stdout: librados2-2:19.3.0-4930.gfc3e0bff.el9.x86_64 2024-09-15T20:17:24.914 INFO:teuthology.orchestra.run.smithi136.stdout: librbd1-2:19.3.0-4930.gfc3e0bff.el9.x86_64 2024-09-15T20:17:24.914 INFO:teuthology.orchestra.run.smithi136.stdout:Installed: 2024-09-15T20:17:24.914 INFO:teuthology.orchestra.run.smithi136.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-09-15T20:17:24.914 INFO:teuthology.orchestra.run.smithi136.stdout: ceph-base-2:19.3.0-4930.gfc3e0bff.el9.x86_64 2024-09-15T20:17:24.914 INFO:teuthology.orchestra.run.smithi136.stdout: ceph-common-2:19.3.0-4930.gfc3e0bff.el9.x86_64 2024-09-15T20:17:24.914 INFO:teuthology.orchestra.run.smithi136.stdout: ceph-radosgw-2:19.3.0-4930.gfc3e0bff.el9.x86_64 2024-09-15T20:17:24.914 INFO:teuthology.orchestra.run.smithi136.stdout: ceph-selinux-2:19.3.0-4930.gfc3e0bff.el9.x86_64 2024-09-15T20:17:24.914 INFO:teuthology.orchestra.run.smithi136.stdout: gperftools-libs-2.9.1-3.el9.x86_64 2024-09-15T20:17:24.914 INFO:teuthology.orchestra.run.smithi136.stdout: libarrow-9.0.0-11.el9.x86_64 2024-09-15T20:17:24.915 INFO:teuthology.orchestra.run.smithi136.stdout: libarrow-doc-9.0.0-11.el9.noarch 2024-09-15T20:17:24.915 INFO:teuthology.orchestra.run.smithi136.stdout: libcephfs2-2:19.3.0-4930.gfc3e0bff.el9.x86_64 2024-09-15T20:17:24.915 INFO:teuthology.orchestra.run.smithi136.stdout: libnbd-1.20.2-2.el9.x86_64 2024-09-15T20:17:24.915 INFO:teuthology.orchestra.run.smithi136.stdout: liboath-2.6.7-2.el9.x86_64 2024-09-15T20:17:24.915 INFO:teuthology.orchestra.run.smithi136.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-09-15T20:17:24.915 INFO:teuthology.orchestra.run.smithi136.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-09-15T20:17:24.915 INFO:teuthology.orchestra.run.smithi136.stdout: libradosstriper1-2:19.3.0-4930.gfc3e0bff.el9.x86_64 2024-09-15T20:17:24.915 INFO:teuthology.orchestra.run.smithi136.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-09-15T20:17:24.915 INFO:teuthology.orchestra.run.smithi136.stdout: librgw2-2:19.3.0-4930.gfc3e0bff.el9.x86_64 2024-09-15T20:17:24.915 INFO:teuthology.orchestra.run.smithi136.stdout: libunwind-1.6.2-1.el9.x86_64 2024-09-15T20:17:24.915 INFO:teuthology.orchestra.run.smithi136.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-09-15T20:17:24.915 INFO:teuthology.orchestra.run.smithi136.stdout: parquet-libs-9.0.0-11.el9.x86_64 2024-09-15T20:17:24.915 INFO:teuthology.orchestra.run.smithi136.stdout: python3-ceph-argparse-2:19.3.0-4930.gfc3e0bff.el9.x86_64 2024-09-15T20:17:24.916 INFO:teuthology.orchestra.run.smithi136.stdout: python3-ceph-common-2:19.3.0-4930.gfc3e0bff.el9.x86_64 2024-09-15T20:17:24.916 INFO:teuthology.orchestra.run.smithi136.stdout: python3-cephfs-2:19.3.0-4930.gfc3e0bff.el9.x86_64 2024-09-15T20:17:24.916 INFO:teuthology.orchestra.run.smithi136.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-09-15T20:17:24.916 INFO:teuthology.orchestra.run.smithi136.stdout: python3-rados-2:19.3.0-4930.gfc3e0bff.el9.x86_64 2024-09-15T20:17:24.916 INFO:teuthology.orchestra.run.smithi136.stdout: python3-rbd-2:19.3.0-4930.gfc3e0bff.el9.x86_64 2024-09-15T20:17:24.916 INFO:teuthology.orchestra.run.smithi136.stdout: python3-rgw-2:19.3.0-4930.gfc3e0bff.el9.x86_64 2024-09-15T20:17:24.916 INFO:teuthology.orchestra.run.smithi136.stdout: qatlib-24.02.0-1.el9.x86_64 2024-09-15T20:17:24.916 INFO:teuthology.orchestra.run.smithi136.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-09-15T20:17:24.917 INFO:teuthology.orchestra.run.smithi136.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-09-15T20:17:24.917 INFO:teuthology.orchestra.run.smithi136.stdout: re2-1:20211101-20.el9.x86_64 2024-09-15T20:17:24.917 INFO:teuthology.orchestra.run.smithi136.stdout: thrift-0.15.0-2.el9.x86_64 2024-09-15T20:17:24.917 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-09-15T20:17:24.917 INFO:teuthology.orchestra.run.smithi136.stdout:Complete! 2024-09-15T20:17:25.270 DEBUG:teuthology.orchestra.run.smithi136:> sudo yum -y install ceph-test 2024-09-15T20:17:25.848 INFO:teuthology.orchestra.run.smithi136.stdout:Last metadata expiration check: 0:00:27 ago on Sun 15 Sep 2024 08:16:58 PM UTC. 2024-09-15T20:17:25.983 INFO:teuthology.orchestra.run.smithi136.stdout:Dependencies resolved. 2024-09-15T20:17:25.984 INFO:teuthology.orchestra.run.smithi136.stdout:================================================================================ 2024-09-15T20:17:25.984 INFO:teuthology.orchestra.run.smithi136.stdout: Package Arch Version Repository Size 2024-09-15T20:17:25.984 INFO:teuthology.orchestra.run.smithi136.stdout:================================================================================ 2024-09-15T20:17:25.984 INFO:teuthology.orchestra.run.smithi136.stdout:Installing: 2024-09-15T20:17:25.984 INFO:teuthology.orchestra.run.smithi136.stdout: ceph-test x86_64 2:19.3.0-4930.gfc3e0bff.el9 ceph 46 M 2024-09-15T20:17:25.984 INFO:teuthology.orchestra.run.smithi136.stdout:Installing dependencies: 2024-09-15T20:17:25.984 INFO:teuthology.orchestra.run.smithi136.stdout: libcephsqlite x86_64 2:19.3.0-4930.gfc3e0bff.el9 ceph 161 k 2024-09-15T20:17:25.984 INFO:teuthology.orchestra.run.smithi136.stdout: socat x86_64 1.7.4.1-6.el9 appstream 304 k 2024-09-15T20:17:25.984 INFO:teuthology.orchestra.run.smithi136.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-09-15T20:17:25.984 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-09-15T20:17:25.984 INFO:teuthology.orchestra.run.smithi136.stdout:Transaction Summary 2024-09-15T20:17:25.984 INFO:teuthology.orchestra.run.smithi136.stdout:================================================================================ 2024-09-15T20:17:25.985 INFO:teuthology.orchestra.run.smithi136.stdout:Install 4 Packages 2024-09-15T20:17:25.985 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-09-15T20:17:25.985 INFO:teuthology.orchestra.run.smithi136.stdout:Total download size: 46 M 2024-09-15T20:17:25.985 INFO:teuthology.orchestra.run.smithi136.stdout:Installed size: 201 M 2024-09-15T20:17:25.985 INFO:teuthology.orchestra.run.smithi136.stdout:Downloading Packages: 2024-09-15T20:17:26.238 INFO:teuthology.orchestra.run.smithi136.stdout:(1/4): socat-1.7.4.1-6.el9.x86_64.rpm 2.4 MB/s | 304 kB 00:00 2024-09-15T20:17:26.280 INFO:teuthology.orchestra.run.smithi136.stdout:(2/4): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.5 MB/s | 64 kB 00:00 2024-09-15T20:17:26.347 INFO:teuthology.orchestra.run.smithi136.stdout:(3/4): libcephsqlite-19.3.0-4930.gfc3e0bff.el9. 691 kB/s | 161 kB 00:00 2024-09-15T20:17:27.547 INFO:teuthology.orchestra.run.smithi136.stdout:(4/4): ceph-test-19.3.0-4930.gfc3e0bff.el9.x86_ 32 MB/s | 46 MB 00:01 2024-09-15T20:17:27.549 INFO:teuthology.orchestra.run.smithi136.stdout:-------------------------------------------------------------------------------- 2024-09-15T20:17:27.549 INFO:teuthology.orchestra.run.smithi136.stdout:Total 29 MB/s | 46 MB 00:01 2024-09-15T20:17:27.577 INFO:teuthology.orchestra.run.smithi136.stdout:Running transaction check 2024-09-15T20:17:27.597 INFO:teuthology.orchestra.run.smithi136.stdout:Transaction check succeeded. 2024-09-15T20:17:27.597 INFO:teuthology.orchestra.run.smithi136.stdout:Running transaction test 2024-09-15T20:17:27.961 INFO:teuthology.orchestra.run.smithi136.stdout:Transaction test succeeded. 2024-09-15T20:17:27.961 INFO:teuthology.orchestra.run.smithi136.stdout:Running transaction 2024-09-15T20:17:28.477 INFO:teuthology.orchestra.run.smithi136.stdout: Preparing : 1/1 2024-09-15T20:17:28.557 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 1/4 2024-09-15T20:17:28.604 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : socat-1.7.4.1-6.el9.x86_64 2/4 2024-09-15T20:17:28.632 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : libcephsqlite-2:19.3.0-4930.gfc3e0bff.el9.x86_64 3/4 2024-09-15T20:17:33.043 INFO:teuthology.orchestra.run.smithi136.stdout: Running scriptlet: libcephsqlite-2:19.3.0-4930.gfc3e0bff.el9.x86_64 3/4 2024-09-15T20:17:33.123 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : ceph-test-2:19.3.0-4930.gfc3e0bff.el9.x86_64 4/4 2024-09-15T20:17:35.377 INFO:teuthology.orchestra.run.smithi136.stdout: Running scriptlet: ceph-test-2:19.3.0-4930.gfc3e0bff.el9.x86_64 4/4 2024-09-15T20:17:35.377 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : ceph-test-2:19.3.0-4930.gfc3e0bff.el9.x86_64 1/4 2024-09-15T20:17:35.377 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : libcephsqlite-2:19.3.0-4930.gfc3e0bff.el9.x86_64 2/4 2024-09-15T20:17:35.378 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : socat-1.7.4.1-6.el9.x86_64 3/4 2024-09-15T20:17:35.710 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 4/4 2024-09-15T20:17:35.710 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-09-15T20:17:35.710 INFO:teuthology.orchestra.run.smithi136.stdout:Installed: 2024-09-15T20:17:35.710 INFO:teuthology.orchestra.run.smithi136.stdout: ceph-test-2:19.3.0-4930.gfc3e0bff.el9.x86_64 2024-09-15T20:17:35.711 INFO:teuthology.orchestra.run.smithi136.stdout: libcephsqlite-2:19.3.0-4930.gfc3e0bff.el9.x86_64 2024-09-15T20:17:35.711 INFO:teuthology.orchestra.run.smithi136.stdout: socat-1.7.4.1-6.el9.x86_64 2024-09-15T20:17:35.711 INFO:teuthology.orchestra.run.smithi136.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-09-15T20:17:35.711 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-09-15T20:17:35.711 INFO:teuthology.orchestra.run.smithi136.stdout:Complete! 2024-09-15T20:17:35.983 DEBUG:teuthology.orchestra.run.smithi136:> sudo yum -y install ceph 2024-09-15T20:17:36.566 INFO:teuthology.orchestra.run.smithi136.stdout:Last metadata expiration check: 0:00:38 ago on Sun 15 Sep 2024 08:16:58 PM UTC. 2024-09-15T20:17:36.716 INFO:teuthology.orchestra.run.smithi136.stdout:Dependencies resolved. 2024-09-15T20:17:36.720 INFO:teuthology.orchestra.run.smithi136.stdout:================================================================================ 2024-09-15T20:17:36.720 INFO:teuthology.orchestra.run.smithi136.stdout: Package Arch Version Repository Size 2024-09-15T20:17:36.720 INFO:teuthology.orchestra.run.smithi136.stdout:================================================================================ 2024-09-15T20:17:36.720 INFO:teuthology.orchestra.run.smithi136.stdout:Installing: 2024-09-15T20:17:36.720 INFO:teuthology.orchestra.run.smithi136.stdout: ceph x86_64 2:19.3.0-4930.gfc3e0bff.el9 ceph 6.6 k 2024-09-15T20:17:36.720 INFO:teuthology.orchestra.run.smithi136.stdout:Installing dependencies: 2024-09-15T20:17:36.720 INFO:teuthology.orchestra.run.smithi136.stdout: ceph-mds x86_64 2:19.3.0-4930.gfc3e0bff.el9 ceph 2.4 M 2024-09-15T20:17:36.720 INFO:teuthology.orchestra.run.smithi136.stdout: ceph-mgr x86_64 2:19.3.0-4930.gfc3e0bff.el9 ceph 1.5 M 2024-09-15T20:17:36.720 INFO:teuthology.orchestra.run.smithi136.stdout: ceph-mgr-modules-core noarch 2:19.3.0-4930.gfc3e0bff.el9 ceph-noarch 291 k 2024-09-15T20:17:36.720 INFO:teuthology.orchestra.run.smithi136.stdout: ceph-mon x86_64 2:19.3.0-4930.gfc3e0bff.el9 ceph 4.8 M 2024-09-15T20:17:36.720 INFO:teuthology.orchestra.run.smithi136.stdout: ceph-osd x86_64 2:19.3.0-4930.gfc3e0bff.el9 ceph 17 M 2024-09-15T20:17:36.720 INFO:teuthology.orchestra.run.smithi136.stdout: lua x86_64 5.4.4-4.el9 appstream 188 k 2024-09-15T20:17:36.721 INFO:teuthology.orchestra.run.smithi136.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-09-15T20:17:36.721 INFO:teuthology.orchestra.run.smithi136.stdout: luarocks noarch 3.9.2-5.el9 epel 151 k 2024-09-15T20:17:36.721 INFO:teuthology.orchestra.run.smithi136.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-09-15T20:17:36.721 INFO:teuthology.orchestra.run.smithi136.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-09-15T20:17:36.721 INFO:teuthology.orchestra.run.smithi136.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-09-15T20:17:36.721 INFO:teuthology.orchestra.run.smithi136.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-09-15T20:17:36.721 INFO:teuthology.orchestra.run.smithi136.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-09-15T20:17:36.721 INFO:teuthology.orchestra.run.smithi136.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-09-15T20:17:36.721 INFO:teuthology.orchestra.run.smithi136.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-09-15T20:17:36.721 INFO:teuthology.orchestra.run.smithi136.stdout: python3-jaraco-collections 2024-09-15T20:17:36.721 INFO:teuthology.orchestra.run.smithi136.stdout: noarch 3.0.0-8.el9 epel 23 k 2024-09-15T20:17:36.721 INFO:teuthology.orchestra.run.smithi136.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-09-15T20:17:36.721 INFO:teuthology.orchestra.run.smithi136.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-09-15T20:17:36.721 INFO:teuthology.orchestra.run.smithi136.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-09-15T20:17:36.722 INFO:teuthology.orchestra.run.smithi136.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-09-15T20:17:36.722 INFO:teuthology.orchestra.run.smithi136.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-09-15T20:17:36.722 INFO:teuthology.orchestra.run.smithi136.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-09-15T20:17:36.722 INFO:teuthology.orchestra.run.smithi136.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2024-09-15T20:17:36.722 INFO:teuthology.orchestra.run.smithi136.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-09-15T20:17:36.722 INFO:teuthology.orchestra.run.smithi136.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-09-15T20:17:36.722 INFO:teuthology.orchestra.run.smithi136.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-09-15T20:17:36.722 INFO:teuthology.orchestra.run.smithi136.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-09-15T20:17:36.722 INFO:teuthology.orchestra.run.smithi136.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-09-15T20:17:36.722 INFO:teuthology.orchestra.run.smithi136.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-09-15T20:17:36.722 INFO:teuthology.orchestra.run.smithi136.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-09-15T20:17:36.722 INFO:teuthology.orchestra.run.smithi136.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-09-15T20:17:36.722 INFO:teuthology.orchestra.run.smithi136.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-09-15T20:17:36.722 INFO:teuthology.orchestra.run.smithi136.stdout: python3-urllib3 noarch 1.26.5-6.el9 baseos 215 k 2024-09-15T20:17:36.723 INFO:teuthology.orchestra.run.smithi136.stdout: python3-webob noarch 1.8.8-2.el9 epel 230 k 2024-09-15T20:17:36.723 INFO:teuthology.orchestra.run.smithi136.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-09-15T20:17:36.723 INFO:teuthology.orchestra.run.smithi136.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-09-15T20:17:36.723 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-09-15T20:17:36.723 INFO:teuthology.orchestra.run.smithi136.stdout:Transaction Summary 2024-09-15T20:17:36.723 INFO:teuthology.orchestra.run.smithi136.stdout:================================================================================ 2024-09-15T20:17:36.723 INFO:teuthology.orchestra.run.smithi136.stdout:Install 37 Packages 2024-09-15T20:17:36.723 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-09-15T20:17:36.724 INFO:teuthology.orchestra.run.smithi136.stdout:Total download size: 31 M 2024-09-15T20:17:36.724 INFO:teuthology.orchestra.run.smithi136.stdout:Installed size: 109 M 2024-09-15T20:17:36.724 INFO:teuthology.orchestra.run.smithi136.stdout:Downloading Packages: 2024-09-15T20:17:37.449 INFO:teuthology.orchestra.run.smithi136.stdout:(1/37): ceph-19.3.0-4930.gfc3e0bff.el9.x86_64.r 56 kB/s | 6.6 kB 00:00 2024-09-15T20:17:37.659 INFO:teuthology.orchestra.run.smithi136.stdout:(2/37): ceph-mds-19.3.0-4930.gfc3e0bff.el9.x86_ 7.3 MB/s | 2.4 MB 00:00 2024-09-15T20:17:37.699 INFO:teuthology.orchestra.run.smithi136.stdout:(3/37): ceph-mgr-19.3.0-4930.gfc3e0bff.el9.x86_ 4.2 MB/s | 1.5 MB 00:00 2024-09-15T20:17:37.833 INFO:teuthology.orchestra.run.smithi136.stdout:(4/37): ceph-mon-19.3.0-4930.gfc3e0bff.el9.x86_ 12 MB/s | 4.8 MB 00:00 2024-09-15T20:17:37.858 INFO:teuthology.orchestra.run.smithi136.stdout:(5/37): ceph-mgr-modules-core-19.3.0-4930.gfc3e 1.8 MB/s | 291 kB 00:00 2024-09-15T20:17:37.975 INFO:teuthology.orchestra.run.smithi136.stdout:(6/37): python3-cffi-1.14.5-5.el9.x86_64.rpm 1.7 MB/s | 253 kB 00:00 2024-09-15T20:17:38.018 INFO:teuthology.orchestra.run.smithi136.stdout:(7/37): python3-ply-3.11-14.el9.noarch.rpm 2.5 MB/s | 106 kB 00:00 2024-09-15T20:17:38.059 INFO:teuthology.orchestra.run.smithi136.stdout:(8/37): python3-pycparser-2.20-6.el9.noarch.rpm 3.2 MB/s | 135 kB 00:00 2024-09-15T20:17:38.235 INFO:teuthology.orchestra.run.smithi136.stdout:(9/37): ceph-osd-19.3.0-4930.gfc3e0bff.el9.x86_ 30 MB/s | 17 MB 00:00 2024-09-15T20:17:38.268 INFO:teuthology.orchestra.run.smithi136.stdout:(10/37): python3-cryptography-36.0.1-4.el9.x86_ 3.0 MB/s | 1.2 MB 00:00 2024-09-15T20:17:38.294 INFO:teuthology.orchestra.run.smithi136.stdout:(11/37): python3-requests-2.25.1-8.el9.noarch.r 533 kB/s | 125 kB 00:00 2024-09-15T20:17:38.319 INFO:teuthology.orchestra.run.smithi136.stdout:(12/37): python3-urllib3-1.26.5-6.el9.noarch.rp 2.5 MB/s | 215 kB 00:00 2024-09-15T20:17:38.386 INFO:teuthology.orchestra.run.smithi136.stdout:(13/37): python3-markupsafe-1.1.1-12.el9.x86_64 522 kB/s | 35 kB 00:00 2024-09-15T20:17:38.411 INFO:teuthology.orchestra.run.smithi136.stdout:(14/37): python3-mako-1.1.4-6.el9.noarch.rpm 1.4 MB/s | 172 kB 00:00 2024-09-15T20:17:38.436 INFO:teuthology.orchestra.run.smithi136.stdout:(15/37): lua-5.4.4-4.el9.x86_64.rpm 1.1 MB/s | 188 kB 00:00 2024-09-15T20:17:38.461 INFO:teuthology.orchestra.run.smithi136.stdout:(16/37): python3-packaging-20.9-5.el9.noarch.rp 1.0 MB/s | 77 kB 00:00 2024-09-15T20:17:38.487 INFO:teuthology.orchestra.run.smithi136.stdout:(17/37): python3-pytz-2021.1-5.el9.noarch.rpm 676 kB/s | 51 kB 00:00 2024-09-15T20:17:38.512 INFO:teuthology.orchestra.run.smithi136.stdout:(18/37): python3-toml-0.10.2-6.el9.noarch.rpm 551 kB/s | 42 kB 00:00 2024-09-15T20:17:38.554 INFO:teuthology.orchestra.run.smithi136.stdout:(19/37): luarocks-3.9.2-5.el9.noarch.rpm 2.2 MB/s | 151 kB 00:00 2024-09-15T20:17:38.583 INFO:teuthology.orchestra.run.smithi136.stdout:(20/37): python3-cheroot-8.6.0-4.el9.noarch.rpm 6.3 MB/s | 172 kB 00:00 2024-09-15T20:17:38.608 INFO:teuthology.orchestra.run.smithi136.stdout:(21/37): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 454 kB/s | 43 kB 00:00 2024-09-15T20:17:38.634 INFO:teuthology.orchestra.run.smithi136.stdout:(22/37): lua-devel-5.4.4-4.el9.x86_64.rpm 152 kB/s | 22 kB 00:00 2024-09-15T20:17:38.659 INFO:teuthology.orchestra.run.smithi136.stdout:(23/37): python3-jaraco-8.2.1-3.el9.noarch.rpm 213 kB/s | 11 kB 00:00 2024-09-15T20:17:38.692 INFO:teuthology.orchestra.run.smithi136.stdout:(24/37): python3-jaraco-collections-3.0.0-8.el9 882 kB/s | 23 kB 00:00 2024-09-15T20:17:38.718 INFO:teuthology.orchestra.run.smithi136.stdout:(25/37): python3-jaraco-functools-3.5.0-2.el9.n 834 kB/s | 19 kB 00:00 2024-09-15T20:17:38.743 INFO:teuthology.orchestra.run.smithi136.stdout:(26/37): python3-jaraco-classes-3.2.1-5.el9.noa 161 kB/s | 18 kB 00:00 2024-09-15T20:17:38.768 INFO:teuthology.orchestra.run.smithi136.stdout:(27/37): python3-jaraco-text-3.2.0-6.el9.noarch 390 kB/s | 20 kB 00:00 2024-09-15T20:17:38.794 INFO:teuthology.orchestra.run.smithi136.stdout:(28/37): python3-logutils-0.3.5-21.el9.noarch.r 924 kB/s | 46 kB 00:00 2024-09-15T20:17:38.821 INFO:teuthology.orchestra.run.smithi136.stdout:(29/37): python3-cherrypy-18.6.1-2.el9.noarch.r 1.5 MB/s | 358 kB 00:00 2024-09-15T20:17:38.844 INFO:teuthology.orchestra.run.smithi136.stdout:(30/37): python3-more-itertools-8.12.0-2.el9.no 1.0 MB/s | 79 kB 00:00 2024-09-15T20:17:38.861 INFO:teuthology.orchestra.run.smithi136.stdout:(31/37): python3-portend-3.1.0-2.el9.noarch.rpm 394 kB/s | 16 kB 00:00 2024-09-15T20:17:38.878 INFO:teuthology.orchestra.run.smithi136.stdout:(32/37): python3-tempora-5.0.0-2.el9.noarch.rpm 2.1 MB/s | 36 kB 00:00 2024-09-15T20:17:38.904 INFO:teuthology.orchestra.run.smithi136.stdout:(33/37): python3-pyOpenSSL-21.0.0-1.el9.noarch. 1.5 MB/s | 90 kB 00:00 2024-09-15T20:17:38.929 INFO:teuthology.orchestra.run.smithi136.stdout:(34/37): python3-pecan-1.4.2-3.el9.noarch.rpm 2.0 MB/s | 272 kB 00:00 2024-09-15T20:17:38.946 INFO:teuthology.orchestra.run.smithi136.stdout:(35/37): python3-zc-lockfile-2.0-10.el9.noarch. 1.2 MB/s | 20 kB 00:00 2024-09-15T20:17:38.971 INFO:teuthology.orchestra.run.smithi136.stdout:(36/37): python3-webob-1.8.8-2.el9.noarch.rpm 2.4 MB/s | 230 kB 00:00 2024-09-15T20:17:38.996 INFO:teuthology.orchestra.run.smithi136.stdout:(37/37): python3-werkzeug-2.0.3-3.el9.1.noarch. 4.5 MB/s | 427 kB 00:00 2024-09-15T20:17:39.001 INFO:teuthology.orchestra.run.smithi136.stdout:-------------------------------------------------------------------------------- 2024-09-15T20:17:39.001 INFO:teuthology.orchestra.run.smithi136.stdout:Total 13 MB/s | 31 MB 00:02 2024-09-15T20:17:39.158 INFO:teuthology.orchestra.run.smithi136.stdout:Running transaction check 2024-09-15T20:17:39.197 INFO:teuthology.orchestra.run.smithi136.stdout:Transaction check succeeded. 2024-09-15T20:17:39.198 INFO:teuthology.orchestra.run.smithi136.stdout:Running transaction test 2024-09-15T20:17:39.669 INFO:teuthology.orchestra.run.smithi136.stdout:Transaction test succeeded. 2024-09-15T20:17:39.669 INFO:teuthology.orchestra.run.smithi136.stdout:Running transaction 2024-09-15T20:17:41.025 INFO:teuthology.orchestra.run.smithi136.stdout: Preparing : 1/1 2024-09-15T20:17:41.074 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/37 2024-09-15T20:17:41.107 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/37 2024-09-15T20:17:41.135 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : lua-5.4.4-4.el9.x86_64 3/37 2024-09-15T20:17:41.206 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 4/37 2024-09-15T20:17:41.250 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : luarocks-3.9.2-5.el9.noarch 5/37 2024-09-15T20:17:41.283 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 6/37 2024-09-15T20:17:41.376 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 7/37 2024-09-15T20:17:41.453 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 8/37 2024-09-15T20:17:41.495 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-webob-1.8.8-2.el9.noarch 9/37 2024-09-15T20:17:41.529 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 10/37 2024-09-15T20:17:41.560 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 11/37 2024-09-15T20:17:41.594 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 12/37 2024-09-15T20:17:41.627 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 13/37 2024-09-15T20:17:41.668 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 14/37 2024-09-15T20:17:41.712 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 15/37 2024-09-15T20:17:41.754 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 16/37 2024-09-15T20:17:41.803 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 17/37 2024-09-15T20:17:41.843 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-packaging-20.9-5.el9.noarch 18/37 2024-09-15T20:17:41.917 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 19/37 2024-09-15T20:17:42.057 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 20/37 2024-09-15T20:17:42.141 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 21/37 2024-09-15T20:17:42.194 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-urllib3-1.26.5-6.el9.noarch 22/37 2024-09-15T20:17:42.233 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 23/37 2024-09-15T20:17:42.290 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-ply-3.11-14.el9.noarch 24/37 2024-09-15T20:17:42.349 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 25/37 2024-09-15T20:17:42.535 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 26/37 2024-09-15T20:17:42.579 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 27/37 2024-09-15T20:17:42.660 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 28/37 2024-09-15T20:17:42.747 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 29/37 2024-09-15T20:17:42.800 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 30/37 2024-09-15T20:17:42.941 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 31/37 2024-09-15T20:17:43.104 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : ceph-mgr-modules-core-2:19.3.0-4930.gfc3e0bff.el9. 32/37 2024-09-15T20:17:43.131 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : ceph-mgr-2:19.3.0-4930.gfc3e0bff.el9.x86_64 33/37 2024-09-15T20:17:43.158 INFO:teuthology.orchestra.run.smithi136.stdout: Running scriptlet: ceph-mgr-2:19.3.0-4930.gfc3e0bff.el9.x86_64 33/37 2024-09-15T20:17:43.158 INFO:teuthology.orchestra.run.smithi136.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-15T20:17:43.158 INFO:teuthology.orchestra.run.smithi136.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-09-15T20:17:43.158 INFO:teuthology.orchestra.run.smithi136.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-09-15T20:17:43.158 INFO:teuthology.orchestra.run.smithi136.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-09-15T20:17:43.158 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-09-15T20:17:44.640 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : ceph-osd-2:19.3.0-4930.gfc3e0bff.el9.x86_64 34/37 2024-09-15T20:17:44.667 INFO:teuthology.orchestra.run.smithi136.stdout: Running scriptlet: ceph-osd-2:19.3.0-4930.gfc3e0bff.el9.x86_64 34/37 2024-09-15T20:17:44.667 INFO:teuthology.orchestra.run.smithi136.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-15T20:17:44.667 INFO:teuthology.orchestra.run.smithi136.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-09-15T20:17:44.667 INFO:teuthology.orchestra.run.smithi136.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-09-15T20:17:44.667 INFO:teuthology.orchestra.run.smithi136.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-09-15T20:17:44.668 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-09-15T20:17:45.124 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : ceph-mon-2:19.3.0-4930.gfc3e0bff.el9.x86_64 35/37 2024-09-15T20:17:45.148 INFO:teuthology.orchestra.run.smithi136.stdout: Running scriptlet: ceph-mon-2:19.3.0-4930.gfc3e0bff.el9.x86_64 35/37 2024-09-15T20:17:45.148 INFO:teuthology.orchestra.run.smithi136.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-15T20:17:45.148 INFO:teuthology.orchestra.run.smithi136.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-09-15T20:17:45.149 INFO:teuthology.orchestra.run.smithi136.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-09-15T20:17:45.149 INFO:teuthology.orchestra.run.smithi136.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-09-15T20:17:45.149 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-09-15T20:17:45.374 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : ceph-mds-2:19.3.0-4930.gfc3e0bff.el9.x86_64 36/37 2024-09-15T20:17:45.398 INFO:teuthology.orchestra.run.smithi136.stdout: Running scriptlet: ceph-mds-2:19.3.0-4930.gfc3e0bff.el9.x86_64 36/37 2024-09-15T20:17:45.398 INFO:teuthology.orchestra.run.smithi136.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-15T20:17:45.398 INFO:teuthology.orchestra.run.smithi136.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-09-15T20:17:45.399 INFO:teuthology.orchestra.run.smithi136.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-09-15T20:17:45.399 INFO:teuthology.orchestra.run.smithi136.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-09-15T20:17:45.399 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-09-15T20:17:45.463 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : ceph-2:19.3.0-4930.gfc3e0bff.el9.x86_64 37/37 2024-09-15T20:17:47.233 INFO:teuthology.orchestra.run.smithi136.stdout: Running scriptlet: ceph-2:19.3.0-4930.gfc3e0bff.el9.x86_64 37/37 2024-09-15T20:17:47.233 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : ceph-2:19.3.0-4930.gfc3e0bff.el9.x86_64 1/37 2024-09-15T20:17:47.234 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : ceph-mds-2:19.3.0-4930.gfc3e0bff.el9.x86_64 2/37 2024-09-15T20:17:47.234 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : ceph-mgr-2:19.3.0-4930.gfc3e0bff.el9.x86_64 3/37 2024-09-15T20:17:47.234 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : ceph-mon-2:19.3.0-4930.gfc3e0bff.el9.x86_64 4/37 2024-09-15T20:17:47.234 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : ceph-osd-2:19.3.0-4930.gfc3e0bff.el9.x86_64 5/37 2024-09-15T20:17:47.234 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : ceph-mgr-modules-core-2:19.3.0-4930.gfc3e0bff.el9. 6/37 2024-09-15T20:17:47.234 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 7/37 2024-09-15T20:17:47.234 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 8/37 2024-09-15T20:17:47.234 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-ply-3.11-14.el9.noarch 9/37 2024-09-15T20:17:47.234 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 10/37 2024-09-15T20:17:47.234 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 11/37 2024-09-15T20:17:47.234 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-urllib3-1.26.5-6.el9.noarch 12/37 2024-09-15T20:17:47.234 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : lua-5.4.4-4.el9.x86_64 13/37 2024-09-15T20:17:47.234 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 14/37 2024-09-15T20:17:47.237 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 15/37 2024-09-15T20:17:47.237 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 16/37 2024-09-15T20:17:47.237 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 17/37 2024-09-15T20:17:47.237 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 18/37 2024-09-15T20:17:47.238 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 19/37 2024-09-15T20:17:47.238 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : luarocks-3.9.2-5.el9.noarch 20/37 2024-09-15T20:17:47.238 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 21/37 2024-09-15T20:17:47.238 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 22/37 2024-09-15T20:17:47.238 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 23/37 2024-09-15T20:17:47.238 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 24/37 2024-09-15T20:17:47.238 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 25/37 2024-09-15T20:17:47.238 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 26/37 2024-09-15T20:17:47.238 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 27/37 2024-09-15T20:17:47.238 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 28/37 2024-09-15T20:17:47.238 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 29/37 2024-09-15T20:17:47.238 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 30/37 2024-09-15T20:17:47.238 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 31/37 2024-09-15T20:17:47.238 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 32/37 2024-09-15T20:17:47.239 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 33/37 2024-09-15T20:17:47.239 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 34/37 2024-09-15T20:17:47.239 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-webob-1.8.8-2.el9.noarch 35/37 2024-09-15T20:17:47.239 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 36/37 2024-09-15T20:17:48.196 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 37/37 2024-09-15T20:17:48.196 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-09-15T20:17:48.196 INFO:teuthology.orchestra.run.smithi136.stdout:Installed: 2024-09-15T20:17:48.196 INFO:teuthology.orchestra.run.smithi136.stdout: ceph-2:19.3.0-4930.gfc3e0bff.el9.x86_64 2024-09-15T20:17:48.196 INFO:teuthology.orchestra.run.smithi136.stdout: ceph-mds-2:19.3.0-4930.gfc3e0bff.el9.x86_64 2024-09-15T20:17:48.197 INFO:teuthology.orchestra.run.smithi136.stdout: ceph-mgr-2:19.3.0-4930.gfc3e0bff.el9.x86_64 2024-09-15T20:17:48.197 INFO:teuthology.orchestra.run.smithi136.stdout: ceph-mgr-modules-core-2:19.3.0-4930.gfc3e0bff.el9.noarch 2024-09-15T20:17:48.197 INFO:teuthology.orchestra.run.smithi136.stdout: ceph-mon-2:19.3.0-4930.gfc3e0bff.el9.x86_64 2024-09-15T20:17:48.197 INFO:teuthology.orchestra.run.smithi136.stdout: ceph-osd-2:19.3.0-4930.gfc3e0bff.el9.x86_64 2024-09-15T20:17:48.197 INFO:teuthology.orchestra.run.smithi136.stdout: lua-5.4.4-4.el9.x86_64 2024-09-15T20:17:48.197 INFO:teuthology.orchestra.run.smithi136.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-09-15T20:17:48.197 INFO:teuthology.orchestra.run.smithi136.stdout: luarocks-3.9.2-5.el9.noarch 2024-09-15T20:17:48.197 INFO:teuthology.orchestra.run.smithi136.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-09-15T20:17:48.197 INFO:teuthology.orchestra.run.smithi136.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-09-15T20:17:48.197 INFO:teuthology.orchestra.run.smithi136.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-09-15T20:17:48.197 INFO:teuthology.orchestra.run.smithi136.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-09-15T20:17:48.197 INFO:teuthology.orchestra.run.smithi136.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-09-15T20:17:48.197 INFO:teuthology.orchestra.run.smithi136.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-09-15T20:17:48.198 INFO:teuthology.orchestra.run.smithi136.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-09-15T20:17:48.198 INFO:teuthology.orchestra.run.smithi136.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-09-15T20:17:48.198 INFO:teuthology.orchestra.run.smithi136.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-09-15T20:17:48.198 INFO:teuthology.orchestra.run.smithi136.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-09-15T20:17:48.198 INFO:teuthology.orchestra.run.smithi136.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-09-15T20:17:48.198 INFO:teuthology.orchestra.run.smithi136.stdout: python3-mako-1.1.4-6.el9.noarch 2024-09-15T20:17:48.198 INFO:teuthology.orchestra.run.smithi136.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-09-15T20:17:48.198 INFO:teuthology.orchestra.run.smithi136.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-09-15T20:17:48.198 INFO:teuthology.orchestra.run.smithi136.stdout: python3-packaging-20.9-5.el9.noarch 2024-09-15T20:17:48.198 INFO:teuthology.orchestra.run.smithi136.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-09-15T20:17:48.198 INFO:teuthology.orchestra.run.smithi136.stdout: python3-ply-3.11-14.el9.noarch 2024-09-15T20:17:48.198 INFO:teuthology.orchestra.run.smithi136.stdout: python3-portend-3.1.0-2.el9.noarch 2024-09-15T20:17:48.198 INFO:teuthology.orchestra.run.smithi136.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-09-15T20:17:48.198 INFO:teuthology.orchestra.run.smithi136.stdout: python3-pycparser-2.20-6.el9.noarch 2024-09-15T20:17:48.198 INFO:teuthology.orchestra.run.smithi136.stdout: python3-pytz-2021.1-5.el9.noarch 2024-09-15T20:17:48.199 INFO:teuthology.orchestra.run.smithi136.stdout: python3-requests-2.25.1-8.el9.noarch 2024-09-15T20:17:48.199 INFO:teuthology.orchestra.run.smithi136.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-09-15T20:17:48.199 INFO:teuthology.orchestra.run.smithi136.stdout: python3-toml-0.10.2-6.el9.noarch 2024-09-15T20:17:48.199 INFO:teuthology.orchestra.run.smithi136.stdout: python3-urllib3-1.26.5-6.el9.noarch 2024-09-15T20:17:48.199 INFO:teuthology.orchestra.run.smithi136.stdout: python3-webob-1.8.8-2.el9.noarch 2024-09-15T20:17:48.199 INFO:teuthology.orchestra.run.smithi136.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-09-15T20:17:48.199 INFO:teuthology.orchestra.run.smithi136.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-09-15T20:17:48.199 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-09-15T20:17:48.199 INFO:teuthology.orchestra.run.smithi136.stdout:Complete! 2024-09-15T20:17:48.542 DEBUG:teuthology.orchestra.run.smithi136:> sudo yum -y install ceph-base 2024-09-15T20:17:49.118 INFO:teuthology.orchestra.run.smithi136.stdout:Last metadata expiration check: 0:00:51 ago on Sun 15 Sep 2024 08:16:58 PM UTC. 2024-09-15T20:17:49.207 INFO:teuthology.orchestra.run.smithi136.stdout:Package ceph-base-2:19.3.0-4930.gfc3e0bff.el9.x86_64 is already installed. 2024-09-15T20:17:49.249 INFO:teuthology.orchestra.run.smithi136.stdout:Dependencies resolved. 2024-09-15T20:17:49.250 INFO:teuthology.orchestra.run.smithi136.stdout:Nothing to do. 2024-09-15T20:17:49.250 INFO:teuthology.orchestra.run.smithi136.stdout:Complete! 2024-09-15T20:17:49.328 DEBUG:teuthology.orchestra.run.smithi136:> sudo yum -y install cephadm 2024-09-15T20:17:49.892 INFO:teuthology.orchestra.run.smithi136.stdout:Last metadata expiration check: 0:00:51 ago on Sun 15 Sep 2024 08:16:58 PM UTC. 2024-09-15T20:17:50.027 INFO:teuthology.orchestra.run.smithi136.stdout:Dependencies resolved. 2024-09-15T20:17:50.027 INFO:teuthology.orchestra.run.smithi136.stdout:================================================================================ 2024-09-15T20:17:50.028 INFO:teuthology.orchestra.run.smithi136.stdout: Package Arch Version Repository Size 2024-09-15T20:17:50.028 INFO:teuthology.orchestra.run.smithi136.stdout:================================================================================ 2024-09-15T20:17:50.028 INFO:teuthology.orchestra.run.smithi136.stdout:Installing: 2024-09-15T20:17:50.028 INFO:teuthology.orchestra.run.smithi136.stdout: cephadm noarch 2:19.3.0-4930.gfc3e0bff.el9 ceph-noarch 784 k 2024-09-15T20:17:50.028 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-09-15T20:17:50.028 INFO:teuthology.orchestra.run.smithi136.stdout:Transaction Summary 2024-09-15T20:17:50.028 INFO:teuthology.orchestra.run.smithi136.stdout:================================================================================ 2024-09-15T20:17:50.028 INFO:teuthology.orchestra.run.smithi136.stdout:Install 1 Package 2024-09-15T20:17:50.028 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-09-15T20:17:50.028 INFO:teuthology.orchestra.run.smithi136.stdout:Total download size: 784 k 2024-09-15T20:17:50.029 INFO:teuthology.orchestra.run.smithi136.stdout:Installed size: 791 k 2024-09-15T20:17:50.029 INFO:teuthology.orchestra.run.smithi136.stdout:Downloading Packages: 2024-09-15T20:17:50.316 INFO:teuthology.orchestra.run.smithi136.stdout:cephadm-19.3.0-4930.gfc3e0bff.el9.noarch.rpm 2.7 MB/s | 784 kB 00:00 2024-09-15T20:17:50.317 INFO:teuthology.orchestra.run.smithi136.stdout:-------------------------------------------------------------------------------- 2024-09-15T20:17:50.317 INFO:teuthology.orchestra.run.smithi136.stdout:Total 2.6 MB/s | 784 kB 00:00 2024-09-15T20:17:50.317 INFO:teuthology.orchestra.run.smithi136.stdout:Running transaction check 2024-09-15T20:17:50.324 INFO:teuthology.orchestra.run.smithi136.stdout:Transaction check succeeded. 2024-09-15T20:17:50.325 INFO:teuthology.orchestra.run.smithi136.stdout:Running transaction test 2024-09-15T20:17:50.329 INFO:teuthology.orchestra.run.smithi136.stdout:Transaction test succeeded. 2024-09-15T20:17:50.329 INFO:teuthology.orchestra.run.smithi136.stdout:Running transaction 2024-09-15T20:17:50.404 INFO:teuthology.orchestra.run.smithi136.stdout: Preparing : 1/1 2024-09-15T20:17:50.869 INFO:teuthology.orchestra.run.smithi136.stdout: Running scriptlet: cephadm-2:19.3.0-4930.gfc3e0bff.el9.noarch 1/1 2024-09-15T20:17:50.962 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : cephadm-2:19.3.0-4930.gfc3e0bff.el9.noarch 1/1 2024-09-15T20:17:51.344 INFO:teuthology.orchestra.run.smithi136.stdout: Running scriptlet: cephadm-2:19.3.0-4930.gfc3e0bff.el9.noarch 1/1 2024-09-15T20:17:51.564 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : cephadm-2:19.3.0-4930.gfc3e0bff.el9.noarch 1/1 2024-09-15T20:17:51.564 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-09-15T20:17:51.564 INFO:teuthology.orchestra.run.smithi136.stdout:Installed: 2024-09-15T20:17:51.564 INFO:teuthology.orchestra.run.smithi136.stdout: cephadm-2:19.3.0-4930.gfc3e0bff.el9.noarch 2024-09-15T20:17:51.564 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-09-15T20:17:51.564 INFO:teuthology.orchestra.run.smithi136.stdout:Complete! 2024-09-15T20:17:51.769 DEBUG:teuthology.orchestra.run.smithi136:> sudo yum -y install ceph-immutable-object-cache 2024-09-15T20:17:52.346 INFO:teuthology.orchestra.run.smithi136.stdout:Last metadata expiration check: 0:00:54 ago on Sun 15 Sep 2024 08:16:58 PM UTC. 2024-09-15T20:17:52.484 INFO:teuthology.orchestra.run.smithi136.stdout:Dependencies resolved. 2024-09-15T20:17:52.484 INFO:teuthology.orchestra.run.smithi136.stdout:================================================================================ 2024-09-15T20:17:52.485 INFO:teuthology.orchestra.run.smithi136.stdout: Package Arch Version Repo Size 2024-09-15T20:17:52.485 INFO:teuthology.orchestra.run.smithi136.stdout:================================================================================ 2024-09-15T20:17:52.485 INFO:teuthology.orchestra.run.smithi136.stdout:Installing: 2024-09-15T20:17:52.485 INFO:teuthology.orchestra.run.smithi136.stdout: ceph-immutable-object-cache x86_64 2:19.3.0-4930.gfc3e0bff.el9 ceph 145 k 2024-09-15T20:17:52.485 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-09-15T20:17:52.485 INFO:teuthology.orchestra.run.smithi136.stdout:Transaction Summary 2024-09-15T20:17:52.485 INFO:teuthology.orchestra.run.smithi136.stdout:================================================================================ 2024-09-15T20:17:52.485 INFO:teuthology.orchestra.run.smithi136.stdout:Install 1 Package 2024-09-15T20:17:52.485 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-09-15T20:17:52.485 INFO:teuthology.orchestra.run.smithi136.stdout:Total download size: 145 k 2024-09-15T20:17:52.485 INFO:teuthology.orchestra.run.smithi136.stdout:Installed size: 432 k 2024-09-15T20:17:52.485 INFO:teuthology.orchestra.run.smithi136.stdout:Downloading Packages: 2024-09-15T20:17:52.698 INFO:teuthology.orchestra.run.smithi136.stdout:ceph-immutable-object-cache-19.3.0-4930.gfc3e0b 681 kB/s | 145 kB 00:00 2024-09-15T20:17:52.699 INFO:teuthology.orchestra.run.smithi136.stdout:-------------------------------------------------------------------------------- 2024-09-15T20:17:52.699 INFO:teuthology.orchestra.run.smithi136.stdout:Total 676 kB/s | 145 kB 00:00 2024-09-15T20:17:52.699 INFO:teuthology.orchestra.run.smithi136.stdout:Running transaction check 2024-09-15T20:17:52.707 INFO:teuthology.orchestra.run.smithi136.stdout:Transaction check succeeded. 2024-09-15T20:17:52.707 INFO:teuthology.orchestra.run.smithi136.stdout:Running transaction test 2024-09-15T20:17:52.764 INFO:teuthology.orchestra.run.smithi136.stdout:Transaction test succeeded. 2024-09-15T20:17:52.764 INFO:teuthology.orchestra.run.smithi136.stdout:Running transaction 2024-09-15T20:17:52.913 INFO:teuthology.orchestra.run.smithi136.stdout: Preparing : 1/1 2024-09-15T20:17:53.008 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : ceph-immutable-object-cache-2:19.3.0-4930.gfc3e0bff. 1/1 2024-09-15T20:17:53.032 INFO:teuthology.orchestra.run.smithi136.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.3.0-4930.gfc3e0bff. 1/1 2024-09-15T20:17:53.032 INFO:teuthology.orchestra.run.smithi136.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-15T20:17:53.033 INFO:teuthology.orchestra.run.smithi136.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-09-15T20:17:53.033 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-09-15T20:17:53.949 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : ceph-immutable-object-cache-2:19.3.0-4930.gfc3e0bff. 1/1 2024-09-15T20:17:53.949 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-09-15T20:17:53.950 INFO:teuthology.orchestra.run.smithi136.stdout:Installed: 2024-09-15T20:17:53.950 INFO:teuthology.orchestra.run.smithi136.stdout: ceph-immutable-object-cache-2:19.3.0-4930.gfc3e0bff.el9.x86_64 2024-09-15T20:17:53.950 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-09-15T20:17:53.950 INFO:teuthology.orchestra.run.smithi136.stdout:Complete! 2024-09-15T20:17:54.176 DEBUG:teuthology.orchestra.run.smithi136:> sudo yum -y install ceph-mgr 2024-09-15T20:17:54.744 INFO:teuthology.orchestra.run.smithi136.stdout:Last metadata expiration check: 0:00:56 ago on Sun 15 Sep 2024 08:16:58 PM UTC. 2024-09-15T20:17:54.837 INFO:teuthology.orchestra.run.smithi136.stdout:Package ceph-mgr-2:19.3.0-4930.gfc3e0bff.el9.x86_64 is already installed. 2024-09-15T20:17:54.881 INFO:teuthology.orchestra.run.smithi136.stdout:Dependencies resolved. 2024-09-15T20:17:54.882 INFO:teuthology.orchestra.run.smithi136.stdout:Nothing to do. 2024-09-15T20:17:54.882 INFO:teuthology.orchestra.run.smithi136.stdout:Complete! 2024-09-15T20:17:54.957 DEBUG:teuthology.orchestra.run.smithi136:> sudo yum -y install ceph-mgr-dashboard 2024-09-15T20:17:55.526 INFO:teuthology.orchestra.run.smithi136.stdout:Last metadata expiration check: 0:00:57 ago on Sun 15 Sep 2024 08:16:58 PM UTC. 2024-09-15T20:17:55.654 INFO:teuthology.orchestra.run.smithi136.stdout:Dependencies resolved. 2024-09-15T20:17:55.655 INFO:teuthology.orchestra.run.smithi136.stdout:================================================================================ 2024-09-15T20:17:55.655 INFO:teuthology.orchestra.run.smithi136.stdout: Package Arch Version Repository Size 2024-09-15T20:17:55.655 INFO:teuthology.orchestra.run.smithi136.stdout:================================================================================ 2024-09-15T20:17:55.655 INFO:teuthology.orchestra.run.smithi136.stdout:Installing: 2024-09-15T20:17:55.656 INFO:teuthology.orchestra.run.smithi136.stdout: ceph-mgr-dashboard noarch 2:19.3.0-4930.gfc3e0bff.el9 ceph-noarch 14 M 2024-09-15T20:17:55.656 INFO:teuthology.orchestra.run.smithi136.stdout:Installing dependencies: 2024-09-15T20:17:55.656 INFO:teuthology.orchestra.run.smithi136.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-09-15T20:17:55.656 INFO:teuthology.orchestra.run.smithi136.stdout: ceph-grafana-dashboards noarch 2:19.3.0-4930.gfc3e0bff.el9 ceph-noarch 34 k 2024-09-15T20:17:55.656 INFO:teuthology.orchestra.run.smithi136.stdout: ceph-prometheus-alerts noarch 2:19.3.0-4930.gfc3e0bff.el9 ceph-noarch 16 k 2024-09-15T20:17:55.656 INFO:teuthology.orchestra.run.smithi136.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-09-15T20:17:55.656 INFO:teuthology.orchestra.run.smithi136.stdout: protobuf x86_64 3.14.0-13.el9 appstream 1.0 M 2024-09-15T20:17:55.656 INFO:teuthology.orchestra.run.smithi136.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-09-15T20:17:55.656 INFO:teuthology.orchestra.run.smithi136.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-09-15T20:17:55.656 INFO:teuthology.orchestra.run.smithi136.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-09-15T20:17:55.656 INFO:teuthology.orchestra.run.smithi136.stdout: python3-protobuf noarch 3.14.0-13.el9 appstream 269 k 2024-09-15T20:17:55.656 INFO:teuthology.orchestra.run.smithi136.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-09-15T20:17:55.656 INFO:teuthology.orchestra.run.smithi136.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-09-15T20:17:55.657 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-09-15T20:17:55.657 INFO:teuthology.orchestra.run.smithi136.stdout:Transaction Summary 2024-09-15T20:17:55.657 INFO:teuthology.orchestra.run.smithi136.stdout:================================================================================ 2024-09-15T20:17:55.657 INFO:teuthology.orchestra.run.smithi136.stdout:Install 12 Packages 2024-09-15T20:17:55.657 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-09-15T20:17:55.657 INFO:teuthology.orchestra.run.smithi136.stdout:Total download size: 19 M 2024-09-15T20:17:55.657 INFO:teuthology.orchestra.run.smithi136.stdout:Installed size: 159 M 2024-09-15T20:17:55.658 INFO:teuthology.orchestra.run.smithi136.stdout:Downloading Packages: 2024-09-15T20:17:56.394 INFO:teuthology.orchestra.run.smithi136.stdout:(1/12): ceph-grafana-dashboards-19.3.0-4930.gfc 240 kB/s | 34 kB 00:00 2024-09-15T20:17:56.419 INFO:teuthology.orchestra.run.smithi136.stdout:(2/12): ceph-prometheus-alerts-19.3.0-4930.gfc3 98 kB/s | 16 kB 00:00 2024-09-15T20:17:56.645 INFO:teuthology.orchestra.run.smithi136.stdout:(3/12): python3-protobuf-3.14.0-13.el9.noarch.r 1.2 MB/s | 269 kB 00:00 2024-09-15T20:17:56.920 INFO:teuthology.orchestra.run.smithi136.stdout:(4/12): ceph-mgr-dashboard-19.3.0-4930.gfc3e0bf 21 MB/s | 14 MB 00:00 2024-09-15T20:17:56.945 INFO:teuthology.orchestra.run.smithi136.stdout:(5/12): protobuf-compiler-3.14.0-13.el9.x86_64. 2.8 MB/s | 863 kB 00:00 2024-09-15T20:17:56.988 INFO:teuthology.orchestra.run.smithi136.stdout:(6/12): protobuf-3.14.0-13.el9.x86_64.rpm 1.7 MB/s | 1.0 MB 00:00 2024-09-15T20:17:57.037 INFO:teuthology.orchestra.run.smithi136.stdout:(7/12): grpc-data-1.46.7-10.el9.noarch.rpm 212 kB/s | 19 kB 00:00 2024-09-15T20:17:57.071 INFO:teuthology.orchestra.run.smithi136.stdout:(8/12): abseil-cpp-20211102.0-3.el9.x86_64.rpm 3.6 MB/s | 548 kB 00:00 2024-09-15T20:17:57.096 INFO:teuthology.orchestra.run.smithi136.stdout:(9/12): python3-repoze-lru-0.7-16.el9.noarch.rp 1.2 MB/s | 31 kB 00:00 2024-09-15T20:17:57.122 INFO:teuthology.orchestra.run.smithi136.stdout:(10/12): python3-grpcio-tools-1.46.7-10.el9.x86 1.7 MB/s | 144 kB 00:00 2024-09-15T20:17:57.147 INFO:teuthology.orchestra.run.smithi136.stdout:(11/12): python3-routes-2.5.1-5.el9.noarch.rpm 3.7 MB/s | 188 kB 00:00 2024-09-15T20:17:57.214 INFO:teuthology.orchestra.run.smithi136.stdout:(12/12): python3-grpcio-1.46.7-10.el9.x86_64.rp 9.6 MB/s | 2.0 MB 00:00 2024-09-15T20:17:57.218 INFO:teuthology.orchestra.run.smithi136.stdout:-------------------------------------------------------------------------------- 2024-09-15T20:17:57.218 INFO:teuthology.orchestra.run.smithi136.stdout:Total 12 MB/s | 19 MB 00:01 2024-09-15T20:17:57.266 INFO:teuthology.orchestra.run.smithi136.stdout:Running transaction check 2024-09-15T20:17:57.285 INFO:teuthology.orchestra.run.smithi136.stdout:Transaction check succeeded. 2024-09-15T20:17:57.285 INFO:teuthology.orchestra.run.smithi136.stdout:Running transaction test 2024-09-15T20:17:57.572 INFO:teuthology.orchestra.run.smithi136.stdout:Transaction test succeeded. 2024-09-15T20:17:57.572 INFO:teuthology.orchestra.run.smithi136.stdout:Running transaction 2024-09-15T20:17:58.166 INFO:teuthology.orchestra.run.smithi136.stdout: Preparing : 1/1 2024-09-15T20:17:58.239 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 1/12 2024-09-15T20:17:58.277 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 2/12 2024-09-15T20:17:58.329 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 3/12 2024-09-15T20:17:58.360 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 4/12 2024-09-15T20:17:58.453 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 5/12 2024-09-15T20:17:58.577 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 6/12 2024-09-15T20:17:58.671 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 7/12 2024-09-15T20:17:58.724 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 8/12 2024-09-15T20:17:58.755 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 9/12 2024-09-15T20:17:58.784 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : ceph-prometheus-alerts-2:19.3.0-4930.gfc3e0bff.el9 10/12 2024-09-15T20:18:01.943 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : ceph-grafana-dashboards-2:19.3.0-4930.gfc3e0bff.el 11/12 2024-09-15T20:18:02.006 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : ceph-mgr-dashboard-2:19.3.0-4930.gfc3e0bff.el9.noa 12/12 2024-09-15T20:18:03.935 INFO:teuthology.orchestra.run.smithi136.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.3.0-4930.gfc3e0bff.el9.noa 12/12 2024-09-15T20:18:03.935 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : ceph-grafana-dashboards-2:19.3.0-4930.gfc3e0bff.el 1/12 2024-09-15T20:18:03.935 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : ceph-mgr-dashboard-2:19.3.0-4930.gfc3e0bff.el9.noa 2/12 2024-09-15T20:18:03.935 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : ceph-prometheus-alerts-2:19.3.0-4930.gfc3e0bff.el9 3/12 2024-09-15T20:18:03.935 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 4/12 2024-09-15T20:18:03.935 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 5/12 2024-09-15T20:18:03.935 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 6/12 2024-09-15T20:18:03.935 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 7/12 2024-09-15T20:18:03.935 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 8/12 2024-09-15T20:18:03.936 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 9/12 2024-09-15T20:18:03.936 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 10/12 2024-09-15T20:18:03.936 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 11/12 2024-09-15T20:18:04.271 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 12/12 2024-09-15T20:18:04.271 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-09-15T20:18:04.271 INFO:teuthology.orchestra.run.smithi136.stdout:Installed: 2024-09-15T20:18:04.271 INFO:teuthology.orchestra.run.smithi136.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-09-15T20:18:04.271 INFO:teuthology.orchestra.run.smithi136.stdout: ceph-grafana-dashboards-2:19.3.0-4930.gfc3e0bff.el9.noarch 2024-09-15T20:18:04.271 INFO:teuthology.orchestra.run.smithi136.stdout: ceph-mgr-dashboard-2:19.3.0-4930.gfc3e0bff.el9.noarch 2024-09-15T20:18:04.272 INFO:teuthology.orchestra.run.smithi136.stdout: ceph-prometheus-alerts-2:19.3.0-4930.gfc3e0bff.el9.noarch 2024-09-15T20:18:04.272 INFO:teuthology.orchestra.run.smithi136.stdout: grpc-data-1.46.7-10.el9.noarch 2024-09-15T20:18:04.272 INFO:teuthology.orchestra.run.smithi136.stdout: protobuf-3.14.0-13.el9.x86_64 2024-09-15T20:18:04.272 INFO:teuthology.orchestra.run.smithi136.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-09-15T20:18:04.272 INFO:teuthology.orchestra.run.smithi136.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-09-15T20:18:04.272 INFO:teuthology.orchestra.run.smithi136.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-09-15T20:18:04.272 INFO:teuthology.orchestra.run.smithi136.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-09-15T20:18:04.272 INFO:teuthology.orchestra.run.smithi136.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-09-15T20:18:04.272 INFO:teuthology.orchestra.run.smithi136.stdout: python3-routes-2.5.1-5.el9.noarch 2024-09-15T20:18:04.272 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-09-15T20:18:04.272 INFO:teuthology.orchestra.run.smithi136.stdout:Complete! 2024-09-15T20:18:04.545 DEBUG:teuthology.orchestra.run.smithi136:> sudo yum -y install ceph-mgr-diskprediction-local 2024-09-15T20:18:05.110 INFO:teuthology.orchestra.run.smithi136.stdout:Last metadata expiration check: 0:01:07 ago on Sun 15 Sep 2024 08:16:58 PM UTC. 2024-09-15T20:18:05.243 INFO:teuthology.orchestra.run.smithi136.stdout:Dependencies resolved. 2024-09-15T20:18:05.245 INFO:teuthology.orchestra.run.smithi136.stdout:======================================================================================= 2024-09-15T20:18:05.245 INFO:teuthology.orchestra.run.smithi136.stdout: Package Arch Version Repository Size 2024-09-15T20:18:05.245 INFO:teuthology.orchestra.run.smithi136.stdout:======================================================================================= 2024-09-15T20:18:05.245 INFO:teuthology.orchestra.run.smithi136.stdout:Installing: 2024-09-15T20:18:05.245 INFO:teuthology.orchestra.run.smithi136.stdout: ceph-mgr-diskprediction-local noarch 2:19.3.0-4930.gfc3e0bff.el9 ceph-noarch 7.4 M 2024-09-15T20:18:05.245 INFO:teuthology.orchestra.run.smithi136.stdout:Installing dependencies: 2024-09-15T20:18:05.245 INFO:teuthology.orchestra.run.smithi136.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-09-15T20:18:05.245 INFO:teuthology.orchestra.run.smithi136.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-09-15T20:18:05.245 INFO:teuthology.orchestra.run.smithi136.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-09-15T20:18:05.245 INFO:teuthology.orchestra.run.smithi136.stdout: libgfortran x86_64 11.5.0-2.el9 baseos 801 k 2024-09-15T20:18:05.245 INFO:teuthology.orchestra.run.smithi136.stdout: libquadmath x86_64 11.5.0-2.el9 baseos 191 k 2024-09-15T20:18:05.245 INFO:teuthology.orchestra.run.smithi136.stdout: openblas x86_64 0.3.26-2.el9 appstream 38 k 2024-09-15T20:18:05.246 INFO:teuthology.orchestra.run.smithi136.stdout: openblas-openmp x86_64 0.3.26-2.el9 appstream 4.9 M 2024-09-15T20:18:05.246 INFO:teuthology.orchestra.run.smithi136.stdout: python3-devel x86_64 3.9.19-8.el9 appstream 245 k 2024-09-15T20:18:05.246 INFO:teuthology.orchestra.run.smithi136.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-09-15T20:18:05.246 INFO:teuthology.orchestra.run.smithi136.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-09-15T20:18:05.246 INFO:teuthology.orchestra.run.smithi136.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-09-15T20:18:05.246 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-09-15T20:18:05.246 INFO:teuthology.orchestra.run.smithi136.stdout:Transaction Summary 2024-09-15T20:18:05.246 INFO:teuthology.orchestra.run.smithi136.stdout:======================================================================================= 2024-09-15T20:18:05.246 INFO:teuthology.orchestra.run.smithi136.stdout:Install 12 Packages 2024-09-15T20:18:05.246 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-09-15T20:18:05.247 INFO:teuthology.orchestra.run.smithi136.stdout:Total download size: 38 M 2024-09-15T20:18:05.247 INFO:teuthology.orchestra.run.smithi136.stdout:Installed size: 212 M 2024-09-15T20:18:05.247 INFO:teuthology.orchestra.run.smithi136.stdout:Downloading Packages: 2024-09-15T20:18:05.981 INFO:teuthology.orchestra.run.smithi136.stdout:(1/12): libquadmath-11.5.0-2.el9.x86_64.rpm 1.1 MB/s | 191 kB 00:00 2024-09-15T20:18:06.021 INFO:teuthology.orchestra.run.smithi136.stdout:(2/12): libgfortran-11.5.0-2.el9.x86_64.rpm 3.6 MB/s | 801 kB 00:00 2024-09-15T20:18:06.122 INFO:teuthology.orchestra.run.smithi136.stdout:(3/12): flexiblas-3.0.4-8.el9.x86_64.rpm 231 kB/s | 33 kB 00:00 2024-09-15T20:18:06.172 INFO:teuthology.orchestra.run.smithi136.stdout:(4/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 364 kB/s | 18 kB 00:00 2024-09-15T20:18:06.222 INFO:teuthology.orchestra.run.smithi136.stdout:(5/12): openblas-0.3.26-2.el9.x86_64.rpm 768 kB/s | 38 kB 00:00 2024-09-15T20:18:06.314 INFO:teuthology.orchestra.run.smithi136.stdout:(6/12): ceph-mgr-diskprediction-local-19.3.0-49 14 MB/s | 7.4 MB 00:00 2024-09-15T20:18:06.506 INFO:teuthology.orchestra.run.smithi136.stdout:(7/12): python3-devel-3.9.19-8.el9.x86_64.rpm 1.2 MB/s | 245 kB 00:00 2024-09-15T20:18:06.623 INFO:teuthology.orchestra.run.smithi136.stdout:(8/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 5.0 MB/s | 3.0 MB 00:00 2024-09-15T20:18:06.700 INFO:teuthology.orchestra.run.smithi136.stdout:(9/12): python3-numpy-f2py-1.20.1-5.el9.x86_64. 3.3 MB/s | 250 kB 00:00 2024-09-15T20:18:06.917 INFO:teuthology.orchestra.run.smithi136.stdout:(10/12): openblas-openmp-0.3.26-2.el9.x86_64.rp 7.1 MB/s | 4.9 MB 00:00 2024-09-15T20:18:07.182 INFO:teuthology.orchestra.run.smithi136.stdout:(11/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 7.5 MB/s | 5.1 MB 00:00 2024-09-15T20:18:07.691 INFO:teuthology.orchestra.run.smithi136.stdout:(12/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 16 MB/s | 16 MB 00:00 2024-09-15T20:18:07.692 INFO:teuthology.orchestra.run.smithi136.stdout:-------------------------------------------------------------------------------- 2024-09-15T20:18:07.692 INFO:teuthology.orchestra.run.smithi136.stdout:Total 16 MB/s | 38 MB 00:02 2024-09-15T20:18:08.048 INFO:teuthology.orchestra.run.smithi136.stdout:Running transaction check 2024-09-15T20:18:08.075 INFO:teuthology.orchestra.run.smithi136.stdout:Transaction check succeeded. 2024-09-15T20:18:08.075 INFO:teuthology.orchestra.run.smithi136.stdout:Running transaction test 2024-09-15T20:18:08.506 INFO:teuthology.orchestra.run.smithi136.stdout:Transaction test succeeded. 2024-09-15T20:18:08.506 INFO:teuthology.orchestra.run.smithi136.stdout:Running transaction 2024-09-15T20:18:09.301 INFO:teuthology.orchestra.run.smithi136.stdout: Preparing : 1/1 2024-09-15T20:18:09.368 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 1/12 2024-09-15T20:18:09.405 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : libquadmath-11.5.0-2.el9.x86_64 2/12 2024-09-15T20:18:09.496 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : libgfortran-11.5.0-2.el9.x86_64 3/12 2024-09-15T20:18:09.532 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-devel-3.9.19-8.el9.x86_64 4/12 2024-09-15T20:18:09.717 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : openblas-0.3.26-2.el9.x86_64 5/12 2024-09-15T20:18:09.747 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : openblas-openmp-0.3.26-2.el9.x86_64 6/12 2024-09-15T20:18:09.832 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-09-15T20:18:10.493 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-09-15T20:18:10.612 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-09-15T20:18:11.846 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-09-15T20:18:12.872 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-09-15T20:18:12.907 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : ceph-mgr-diskprediction-local-2:19.3.0-4930.gfc3e0 12/12 2024-09-15T20:18:15.543 INFO:teuthology.orchestra.run.smithi136.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.3.0-4930.gfc3e0 12/12 2024-09-15T20:18:15.543 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.3.0-4930.gfc3e0 1/12 2024-09-15T20:18:15.543 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : libgfortran-11.5.0-2.el9.x86_64 2/12 2024-09-15T20:18:15.543 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : libquadmath-11.5.0-2.el9.x86_64 3/12 2024-09-15T20:18:15.543 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 4/12 2024-09-15T20:18:15.544 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 5/12 2024-09-15T20:18:15.545 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 6/12 2024-09-15T20:18:15.545 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : openblas-0.3.26-2.el9.x86_64 7/12 2024-09-15T20:18:15.545 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : openblas-openmp-0.3.26-2.el9.x86_64 8/12 2024-09-15T20:18:15.545 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-devel-3.9.19-8.el9.x86_64 9/12 2024-09-15T20:18:15.545 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 10/12 2024-09-15T20:18:15.545 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 11/12 2024-09-15T20:18:15.938 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 12/12 2024-09-15T20:18:15.938 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-09-15T20:18:15.938 INFO:teuthology.orchestra.run.smithi136.stdout:Installed: 2024-09-15T20:18:15.938 INFO:teuthology.orchestra.run.smithi136.stdout: ceph-mgr-diskprediction-local-2:19.3.0-4930.gfc3e0bff.el9.noarch 2024-09-15T20:18:15.938 INFO:teuthology.orchestra.run.smithi136.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-09-15T20:18:15.938 INFO:teuthology.orchestra.run.smithi136.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-09-15T20:18:15.938 INFO:teuthology.orchestra.run.smithi136.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-09-15T20:18:15.938 INFO:teuthology.orchestra.run.smithi136.stdout: libgfortran-11.5.0-2.el9.x86_64 2024-09-15T20:18:15.938 INFO:teuthology.orchestra.run.smithi136.stdout: libquadmath-11.5.0-2.el9.x86_64 2024-09-15T20:18:15.938 INFO:teuthology.orchestra.run.smithi136.stdout: openblas-0.3.26-2.el9.x86_64 2024-09-15T20:18:15.938 INFO:teuthology.orchestra.run.smithi136.stdout: openblas-openmp-0.3.26-2.el9.x86_64 2024-09-15T20:18:15.938 INFO:teuthology.orchestra.run.smithi136.stdout: python3-devel-3.9.19-8.el9.x86_64 2024-09-15T20:18:15.939 INFO:teuthology.orchestra.run.smithi136.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-09-15T20:18:15.939 INFO:teuthology.orchestra.run.smithi136.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-09-15T20:18:15.939 INFO:teuthology.orchestra.run.smithi136.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-09-15T20:18:15.939 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-09-15T20:18:15.939 INFO:teuthology.orchestra.run.smithi136.stdout:Complete! 2024-09-15T20:18:16.398 DEBUG:teuthology.orchestra.run.smithi136:> sudo yum -y install ceph-mgr-rook 2024-09-15T20:18:16.972 INFO:teuthology.orchestra.run.smithi136.stdout:Last metadata expiration check: 0:01:18 ago on Sun 15 Sep 2024 08:16:58 PM UTC. 2024-09-15T20:18:17.109 INFO:teuthology.orchestra.run.smithi136.stdout:Dependencies resolved. 2024-09-15T20:18:17.109 INFO:teuthology.orchestra.run.smithi136.stdout:================================================================================ 2024-09-15T20:18:17.109 INFO:teuthology.orchestra.run.smithi136.stdout: Package Arch Version Repository Size 2024-09-15T20:18:17.109 INFO:teuthology.orchestra.run.smithi136.stdout:================================================================================ 2024-09-15T20:18:17.109 INFO:teuthology.orchestra.run.smithi136.stdout:Installing: 2024-09-15T20:18:17.110 INFO:teuthology.orchestra.run.smithi136.stdout: ceph-mgr-rook noarch 2:19.3.0-4930.gfc3e0bff.el9 ceph-noarch 49 k 2024-09-15T20:18:17.110 INFO:teuthology.orchestra.run.smithi136.stdout:Installing dependencies: 2024-09-15T20:18:17.110 INFO:teuthology.orchestra.run.smithi136.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-09-15T20:18:17.110 INFO:teuthology.orchestra.run.smithi136.stdout: python3-certifi noarch 2023.05.07-4.el9 epel 14 k 2024-09-15T20:18:17.110 INFO:teuthology.orchestra.run.smithi136.stdout: python3-google-auth noarch 1:2.34.0-1.el9 epel 220 k 2024-09-15T20:18:17.110 INFO:teuthology.orchestra.run.smithi136.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-09-15T20:18:17.110 INFO:teuthology.orchestra.run.smithi136.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-09-15T20:18:17.110 INFO:teuthology.orchestra.run.smithi136.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-09-15T20:18:17.110 INFO:teuthology.orchestra.run.smithi136.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-09-15T20:18:17.110 INFO:teuthology.orchestra.run.smithi136.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-09-15T20:18:17.110 INFO:teuthology.orchestra.run.smithi136.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-09-15T20:18:17.110 INFO:teuthology.orchestra.run.smithi136.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-09-15T20:18:17.110 INFO:teuthology.orchestra.run.smithi136.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-09-15T20:18:17.110 INFO:teuthology.orchestra.run.smithi136.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-09-15T20:18:17.111 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-09-15T20:18:17.111 INFO:teuthology.orchestra.run.smithi136.stdout:Transaction Summary 2024-09-15T20:18:17.111 INFO:teuthology.orchestra.run.smithi136.stdout:================================================================================ 2024-09-15T20:18:17.111 INFO:teuthology.orchestra.run.smithi136.stdout:Install 13 Packages 2024-09-15T20:18:17.111 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-09-15T20:18:17.111 INFO:teuthology.orchestra.run.smithi136.stdout:Total download size: 2.2 M 2024-09-15T20:18:17.111 INFO:teuthology.orchestra.run.smithi136.stdout:Installed size: 26 M 2024-09-15T20:18:17.111 INFO:teuthology.orchestra.run.smithi136.stdout:Downloading Packages: 2024-09-15T20:18:17.275 INFO:teuthology.orchestra.run.smithi136.stdout:(1/13): python3-jsonpointer-2.0-4.el9.noarch.rp 233 kB/s | 19 kB 00:00 2024-09-15T20:18:17.300 INFO:teuthology.orchestra.run.smithi136.stdout:(2/13): python3-jsonpatch-1.21-16.el9.noarch.rp 239 kB/s | 26 kB 00:00 2024-09-15T20:18:17.367 INFO:teuthology.orchestra.run.smithi136.stdout:(3/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 2.4 MB/s | 222 kB 00:00 2024-09-15T20:18:17.393 INFO:teuthology.orchestra.run.smithi136.stdout:(4/13): ceph-mgr-rook-19.3.0-4930.gfc3e0bff.el9 247 kB/s | 49 kB 00:00 2024-09-15T20:18:17.418 INFO:teuthology.orchestra.run.smithi136.stdout:(5/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.3 MB/s | 159 kB 00:00 2024-09-15T20:18:17.452 INFO:teuthology.orchestra.run.smithi136.stdout:(6/13): python3-pyasn1-modules-0.4.8-6.el9.noar 3.2 MB/s | 279 kB 00:00 2024-09-15T20:18:17.488 INFO:teuthology.orchestra.run.smithi136.stdout:(7/13): python3-cachetools-4.2.4-1.el9.noarch.r 480 kB/s | 32 kB 00:00 2024-09-15T20:18:17.511 INFO:teuthology.orchestra.run.smithi136.stdout:(8/13): python3-certifi-2023.05.07-4.el9.noarch 248 kB/s | 14 kB 00:00 2024-09-15T20:18:17.545 INFO:teuthology.orchestra.run.smithi136.stdout:(9/13): python3-requests-oauthlib-1.3.0-12.el9. 354 kB/s | 54 kB 00:00 2024-09-15T20:18:17.578 INFO:teuthology.orchestra.run.smithi136.stdout:(10/13): python3-google-auth-2.34.0-1.el9.noarc 2.4 MB/s | 220 kB 00:00 2024-09-15T20:18:17.611 INFO:teuthology.orchestra.run.smithi136.stdout:(11/13): python3-kubernetes-26.1.0-2.el9.noarch 10 MB/s | 1.0 MB 00:00 2024-09-15T20:18:17.628 INFO:teuthology.orchestra.run.smithi136.stdout:(12/13): python3-websocket-client-1.2.3-2.el9.n 1.8 MB/s | 90 kB 00:00 2024-09-15T20:18:17.645 INFO:teuthology.orchestra.run.smithi136.stdout:(13/13): python3-rsa-4.9-2.el9.noarch.rpm 587 kB/s | 59 kB 00:00 2024-09-15T20:18:17.648 INFO:teuthology.orchestra.run.smithi136.stdout:-------------------------------------------------------------------------------- 2024-09-15T20:18:17.648 INFO:teuthology.orchestra.run.smithi136.stdout:Total 4.1 MB/s | 2.2 MB 00:00 2024-09-15T20:18:17.711 INFO:teuthology.orchestra.run.smithi136.stdout:Running transaction check 2024-09-15T20:18:17.726 INFO:teuthology.orchestra.run.smithi136.stdout:Transaction check succeeded. 2024-09-15T20:18:17.726 INFO:teuthology.orchestra.run.smithi136.stdout:Running transaction test 2024-09-15T20:18:17.816 INFO:teuthology.orchestra.run.smithi136.stdout:Transaction test succeeded. 2024-09-15T20:18:17.816 INFO:teuthology.orchestra.run.smithi136.stdout:Running transaction 2024-09-15T20:18:18.321 INFO:teuthology.orchestra.run.smithi136.stdout: Preparing : 1/1 2024-09-15T20:18:18.499 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-09-15T20:18:18.546 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-09-15T20:18:18.609 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-09-15T20:18:18.639 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-09-15T20:18:18.680 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-certifi-2023.05.07-4.el9.noarch 5/13 2024-09-15T20:18:18.781 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-09-15T20:18:18.892 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-google-auth-1:2.34.0-1.el9.noarch 7/13 2024-09-15T20:18:18.949 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-09-15T20:18:19.665 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-09-15T20:18:19.722 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-09-15T20:18:19.755 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-09-15T20:18:19.804 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-09-15T20:18:19.825 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : ceph-mgr-rook-2:19.3.0-4930.gfc3e0bff.el9.noarch 13/13 2024-09-15T20:18:20.627 INFO:teuthology.orchestra.run.smithi136.stdout: Running scriptlet: ceph-mgr-rook-2:19.3.0-4930.gfc3e0bff.el9.noarch 13/13 2024-09-15T20:18:20.627 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : ceph-mgr-rook-2:19.3.0-4930.gfc3e0bff.el9.noarch 1/13 2024-09-15T20:18:20.627 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-09-15T20:18:20.627 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-09-15T20:18:20.627 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-09-15T20:18:20.627 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-09-15T20:18:20.627 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-09-15T20:18:20.627 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-09-15T20:18:20.628 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-09-15T20:18:20.628 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-certifi-2023.05.07-4.el9.noarch 9/13 2024-09-15T20:18:20.628 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-google-auth-1:2.34.0-1.el9.noarch 10/13 2024-09-15T20:18:20.628 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-09-15T20:18:20.628 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-09-15T20:18:20.967 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-09-15T20:18:20.967 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-09-15T20:18:20.967 INFO:teuthology.orchestra.run.smithi136.stdout:Installed: 2024-09-15T20:18:20.967 INFO:teuthology.orchestra.run.smithi136.stdout: ceph-mgr-rook-2:19.3.0-4930.gfc3e0bff.el9.noarch 2024-09-15T20:18:20.967 INFO:teuthology.orchestra.run.smithi136.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-09-15T20:18:20.967 INFO:teuthology.orchestra.run.smithi136.stdout: python3-certifi-2023.05.07-4.el9.noarch 2024-09-15T20:18:20.968 INFO:teuthology.orchestra.run.smithi136.stdout: python3-google-auth-1:2.34.0-1.el9.noarch 2024-09-15T20:18:20.968 INFO:teuthology.orchestra.run.smithi136.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-09-15T20:18:20.968 INFO:teuthology.orchestra.run.smithi136.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-09-15T20:18:20.968 INFO:teuthology.orchestra.run.smithi136.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-09-15T20:18:20.968 INFO:teuthology.orchestra.run.smithi136.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-09-15T20:18:20.968 INFO:teuthology.orchestra.run.smithi136.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-09-15T20:18:20.968 INFO:teuthology.orchestra.run.smithi136.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-09-15T20:18:20.968 INFO:teuthology.orchestra.run.smithi136.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-09-15T20:18:20.968 INFO:teuthology.orchestra.run.smithi136.stdout: python3-rsa-4.9-2.el9.noarch 2024-09-15T20:18:20.968 INFO:teuthology.orchestra.run.smithi136.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-09-15T20:18:20.968 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-09-15T20:18:20.968 INFO:teuthology.orchestra.run.smithi136.stdout:Complete! 2024-09-15T20:18:21.260 DEBUG:teuthology.orchestra.run.smithi136:> sudo yum -y install ceph-mgr-cephadm 2024-09-15T20:18:21.828 INFO:teuthology.orchestra.run.smithi136.stdout:Last metadata expiration check: 0:01:23 ago on Sun 15 Sep 2024 08:16:58 PM UTC. 2024-09-15T20:18:21.963 INFO:teuthology.orchestra.run.smithi136.stdout:Dependencies resolved. 2024-09-15T20:18:21.964 INFO:teuthology.orchestra.run.smithi136.stdout:================================================================================ 2024-09-15T20:18:21.964 INFO:teuthology.orchestra.run.smithi136.stdout: Package Arch Version Repository Size 2024-09-15T20:18:21.964 INFO:teuthology.orchestra.run.smithi136.stdout:================================================================================ 2024-09-15T20:18:21.964 INFO:teuthology.orchestra.run.smithi136.stdout:Installing: 2024-09-15T20:18:21.965 INFO:teuthology.orchestra.run.smithi136.stdout: ceph-mgr-cephadm noarch 2:19.3.0-4930.gfc3e0bff.el9 ceph-noarch 154 k 2024-09-15T20:18:21.965 INFO:teuthology.orchestra.run.smithi136.stdout:Installing dependencies: 2024-09-15T20:18:21.965 INFO:teuthology.orchestra.run.smithi136.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-09-15T20:18:21.965 INFO:teuthology.orchestra.run.smithi136.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-09-15T20:18:21.965 INFO:teuthology.orchestra.run.smithi136.stdout: python3-jinja2 noarch 2.11.3-6.el9 appstream 248 k 2024-09-15T20:18:21.965 INFO:teuthology.orchestra.run.smithi136.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-09-15T20:18:21.965 INFO:teuthology.orchestra.run.smithi136.stdout: python3-typing-extensions noarch 4.12.2-2.el9 epel 74 k 2024-09-15T20:18:21.965 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-09-15T20:18:21.965 INFO:teuthology.orchestra.run.smithi136.stdout:Transaction Summary 2024-09-15T20:18:21.965 INFO:teuthology.orchestra.run.smithi136.stdout:================================================================================ 2024-09-15T20:18:21.965 INFO:teuthology.orchestra.run.smithi136.stdout:Install 6 Packages 2024-09-15T20:18:21.965 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-09-15T20:18:21.966 INFO:teuthology.orchestra.run.smithi136.stdout:Total download size: 7.0 M 2024-09-15T20:18:21.966 INFO:teuthology.orchestra.run.smithi136.stdout:Installed size: 33 M 2024-09-15T20:18:21.966 INFO:teuthology.orchestra.run.smithi136.stdout:Downloading Packages: 2024-09-15T20:18:22.565 INFO:teuthology.orchestra.run.smithi136.stdout:(1/6): python3-jinja2-2.11.3-6.el9.noarch.rpm 1.7 MB/s | 248 kB 00:00 2024-09-15T20:18:22.620 INFO:teuthology.orchestra.run.smithi136.stdout:(2/6): python3-asyncssh-2.13.2-5.el9.noarch.rpm 11 MB/s | 548 kB 00:00 2024-09-15T20:18:22.640 INFO:teuthology.orchestra.run.smithi136.stdout:(3/6): ceph-mgr-cephadm-19.3.0-4930.gfc3e0bff.e 695 kB/s | 154 kB 00:00 2024-09-15T20:18:22.665 INFO:teuthology.orchestra.run.smithi136.stdout:(4/6): python3-natsort-7.1.1-5.el9.noarch.rpm 1.1 MB/s | 58 kB 00:00 2024-09-15T20:18:22.710 INFO:teuthology.orchestra.run.smithi136.stdout:(5/6): python3-typing-extensions-4.12.2-2.el9.n 1.1 MB/s | 74 kB 00:00 2024-09-15T20:18:22.791 INFO:teuthology.orchestra.run.smithi136.stdout:(6/6): python3-babel-2.9.1-2.el9.noarch.rpm 16 MB/s | 6.0 MB 00:00 2024-09-15T20:18:22.793 INFO:teuthology.orchestra.run.smithi136.stdout:-------------------------------------------------------------------------------- 2024-09-15T20:18:22.794 INFO:teuthology.orchestra.run.smithi136.stdout:Total 8.5 MB/s | 7.0 MB 00:00 2024-09-15T20:18:22.865 INFO:teuthology.orchestra.run.smithi136.stdout:Running transaction check 2024-09-15T20:18:22.876 INFO:teuthology.orchestra.run.smithi136.stdout:Transaction check succeeded. 2024-09-15T20:18:22.876 INFO:teuthology.orchestra.run.smithi136.stdout:Running transaction test 2024-09-15T20:18:22.980 INFO:teuthology.orchestra.run.smithi136.stdout:Transaction test succeeded. 2024-09-15T20:18:22.980 INFO:teuthology.orchestra.run.smithi136.stdout:Running transaction 2024-09-15T20:18:23.282 INFO:teuthology.orchestra.run.smithi136.stdout: Preparing : 1/1 2024-09-15T20:18:23.461 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-typing-extensions-4.12.2-2.el9.noarch 1/6 2024-09-15T20:18:23.510 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 2/6 2024-09-15T20:18:24.045 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-09-15T20:18:24.129 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-09-15T20:18:24.197 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-jinja2-2.11.3-6.el9.noarch 5/6 2024-09-15T20:18:24.222 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : ceph-mgr-cephadm-2:19.3.0-4930.gfc3e0bff.el9.noarch 6/6 2024-09-15T20:18:25.035 INFO:teuthology.orchestra.run.smithi136.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.3.0-4930.gfc3e0bff.el9.noarch 6/6 2024-09-15T20:18:25.035 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : ceph-mgr-cephadm-2:19.3.0-4930.gfc3e0bff.el9.noarch 1/6 2024-09-15T20:18:25.035 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-09-15T20:18:25.035 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-jinja2-2.11.3-6.el9.noarch 3/6 2024-09-15T20:18:25.035 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 4/6 2024-09-15T20:18:25.035 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-09-15T20:18:25.348 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-typing-extensions-4.12.2-2.el9.noarch 6/6 2024-09-15T20:18:25.348 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-09-15T20:18:25.348 INFO:teuthology.orchestra.run.smithi136.stdout:Installed: 2024-09-15T20:18:25.348 INFO:teuthology.orchestra.run.smithi136.stdout: ceph-mgr-cephadm-2:19.3.0-4930.gfc3e0bff.el9.noarch 2024-09-15T20:18:25.349 INFO:teuthology.orchestra.run.smithi136.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-09-15T20:18:25.349 INFO:teuthology.orchestra.run.smithi136.stdout: python3-babel-2.9.1-2.el9.noarch 2024-09-15T20:18:25.349 INFO:teuthology.orchestra.run.smithi136.stdout: python3-jinja2-2.11.3-6.el9.noarch 2024-09-15T20:18:25.349 INFO:teuthology.orchestra.run.smithi136.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-09-15T20:18:25.349 INFO:teuthology.orchestra.run.smithi136.stdout: python3-typing-extensions-4.12.2-2.el9.noarch 2024-09-15T20:18:25.349 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-09-15T20:18:25.349 INFO:teuthology.orchestra.run.smithi136.stdout:Complete! 2024-09-15T20:18:25.589 DEBUG:teuthology.orchestra.run.smithi136:> sudo yum -y install ceph-fuse 2024-09-15T20:18:26.162 INFO:teuthology.orchestra.run.smithi136.stdout:Last metadata expiration check: 0:01:28 ago on Sun 15 Sep 2024 08:16:58 PM UTC. 2024-09-15T20:18:26.298 INFO:teuthology.orchestra.run.smithi136.stdout:Dependencies resolved. 2024-09-15T20:18:26.299 INFO:teuthology.orchestra.run.smithi136.stdout:================================================================================ 2024-09-15T20:18:26.299 INFO:teuthology.orchestra.run.smithi136.stdout: Package Arch Version Repository Size 2024-09-15T20:18:26.299 INFO:teuthology.orchestra.run.smithi136.stdout:================================================================================ 2024-09-15T20:18:26.300 INFO:teuthology.orchestra.run.smithi136.stdout:Installing: 2024-09-15T20:18:26.300 INFO:teuthology.orchestra.run.smithi136.stdout: ceph-fuse x86_64 2:19.3.0-4930.gfc3e0bff.el9 ceph 900 k 2024-09-15T20:18:26.300 INFO:teuthology.orchestra.run.smithi136.stdout:Installing dependencies: 2024-09-15T20:18:26.300 INFO:teuthology.orchestra.run.smithi136.stdout: fuse x86_64 2.9.9-16.el9 baseos 80 k 2024-09-15T20:18:26.300 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-09-15T20:18:26.300 INFO:teuthology.orchestra.run.smithi136.stdout:Transaction Summary 2024-09-15T20:18:26.301 INFO:teuthology.orchestra.run.smithi136.stdout:================================================================================ 2024-09-15T20:18:26.301 INFO:teuthology.orchestra.run.smithi136.stdout:Install 2 Packages 2024-09-15T20:18:26.301 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-09-15T20:18:26.301 INFO:teuthology.orchestra.run.smithi136.stdout:Total download size: 980 k 2024-09-15T20:18:26.301 INFO:teuthology.orchestra.run.smithi136.stdout:Installed size: 2.9 M 2024-09-15T20:18:26.301 INFO:teuthology.orchestra.run.smithi136.stdout:Downloading Packages: 2024-09-15T20:18:26.510 INFO:teuthology.orchestra.run.smithi136.stdout:(1/2): fuse-2.9.9-16.el9.x86_64.rpm 723 kB/s | 80 kB 00:00 2024-09-15T20:18:26.686 INFO:teuthology.orchestra.run.smithi136.stdout:(2/2): ceph-fuse-19.3.0-4930.gfc3e0bff.el9.x86_ 3.1 MB/s | 900 kB 00:00 2024-09-15T20:18:26.686 INFO:teuthology.orchestra.run.smithi136.stdout:-------------------------------------------------------------------------------- 2024-09-15T20:18:26.686 INFO:teuthology.orchestra.run.smithi136.stdout:Total 2.5 MB/s | 980 kB 00:00 2024-09-15T20:18:26.698 INFO:teuthology.orchestra.run.smithi136.stdout:Running transaction check 2024-09-15T20:18:26.721 INFO:teuthology.orchestra.run.smithi136.stdout:Transaction check succeeded. 2024-09-15T20:18:26.721 INFO:teuthology.orchestra.run.smithi136.stdout:Running transaction test 2024-09-15T20:18:26.803 INFO:teuthology.orchestra.run.smithi136.stdout:Transaction test succeeded. 2024-09-15T20:18:26.804 INFO:teuthology.orchestra.run.smithi136.stdout:Running transaction 2024-09-15T20:18:26.983 INFO:teuthology.orchestra.run.smithi136.stdout: Preparing : 1/1 2024-09-15T20:18:27.122 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : fuse-2.9.9-16.el9.x86_64 1/2 2024-09-15T20:18:27.202 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : ceph-fuse-2:19.3.0-4930.gfc3e0bff.el9.x86_64 2/2 2024-09-15T20:18:27.871 INFO:teuthology.orchestra.run.smithi136.stdout: Running scriptlet: ceph-fuse-2:19.3.0-4930.gfc3e0bff.el9.x86_64 2/2 2024-09-15T20:18:27.871 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : ceph-fuse-2:19.3.0-4930.gfc3e0bff.el9.x86_64 1/2 2024-09-15T20:18:28.123 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : fuse-2.9.9-16.el9.x86_64 2/2 2024-09-15T20:18:28.123 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-09-15T20:18:28.123 INFO:teuthology.orchestra.run.smithi136.stdout:Installed: 2024-09-15T20:18:28.123 INFO:teuthology.orchestra.run.smithi136.stdout: ceph-fuse-2:19.3.0-4930.gfc3e0bff.el9.x86_64 fuse-2.9.9-16.el9.x86_64 2024-09-15T20:18:28.123 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-09-15T20:18:28.123 INFO:teuthology.orchestra.run.smithi136.stdout:Complete! 2024-09-15T20:18:28.416 DEBUG:teuthology.orchestra.run.smithi136:> sudo yum -y install ceph-volume 2024-09-15T20:18:28.992 INFO:teuthology.orchestra.run.smithi136.stdout:Last metadata expiration check: 0:01:30 ago on Sun 15 Sep 2024 08:16:58 PM UTC. 2024-09-15T20:18:29.132 INFO:teuthology.orchestra.run.smithi136.stdout:Dependencies resolved. 2024-09-15T20:18:29.133 INFO:teuthology.orchestra.run.smithi136.stdout:================================================================================ 2024-09-15T20:18:29.133 INFO:teuthology.orchestra.run.smithi136.stdout: Package Arch Version Repository Size 2024-09-15T20:18:29.133 INFO:teuthology.orchestra.run.smithi136.stdout:================================================================================ 2024-09-15T20:18:29.133 INFO:teuthology.orchestra.run.smithi136.stdout:Installing: 2024-09-15T20:18:29.133 INFO:teuthology.orchestra.run.smithi136.stdout: ceph-volume noarch 2:19.3.0-4930.gfc3e0bff.el9 ceph-noarch 285 k 2024-09-15T20:18:29.133 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-09-15T20:18:29.134 INFO:teuthology.orchestra.run.smithi136.stdout:Transaction Summary 2024-09-15T20:18:29.134 INFO:teuthology.orchestra.run.smithi136.stdout:================================================================================ 2024-09-15T20:18:29.134 INFO:teuthology.orchestra.run.smithi136.stdout:Install 1 Package 2024-09-15T20:18:29.134 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-09-15T20:18:29.134 INFO:teuthology.orchestra.run.smithi136.stdout:Total download size: 285 k 2024-09-15T20:18:29.134 INFO:teuthology.orchestra.run.smithi136.stdout:Installed size: 1.3 M 2024-09-15T20:18:29.134 INFO:teuthology.orchestra.run.smithi136.stdout:Downloading Packages: 2024-09-15T20:18:29.366 INFO:teuthology.orchestra.run.smithi136.stdout:ceph-volume-19.3.0-4930.gfc3e0bff.el9.noarch.rp 1.2 MB/s | 285 kB 00:00 2024-09-15T20:18:29.367 INFO:teuthology.orchestra.run.smithi136.stdout:-------------------------------------------------------------------------------- 2024-09-15T20:18:29.367 INFO:teuthology.orchestra.run.smithi136.stdout:Total 1.2 MB/s | 285 kB 00:00 2024-09-15T20:18:29.367 INFO:teuthology.orchestra.run.smithi136.stdout:Running transaction check 2024-09-15T20:18:29.375 INFO:teuthology.orchestra.run.smithi136.stdout:Transaction check succeeded. 2024-09-15T20:18:29.375 INFO:teuthology.orchestra.run.smithi136.stdout:Running transaction test 2024-09-15T20:18:29.404 INFO:teuthology.orchestra.run.smithi136.stdout:Transaction test succeeded. 2024-09-15T20:18:29.404 INFO:teuthology.orchestra.run.smithi136.stdout:Running transaction 2024-09-15T20:18:29.626 INFO:teuthology.orchestra.run.smithi136.stdout: Preparing : 1/1 2024-09-15T20:18:29.693 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : ceph-volume-2:19.3.0-4930.gfc3e0bff.el9.noarch 1/1 2024-09-15T20:18:29.709 INFO:teuthology.orchestra.run.smithi136.stdout: Running scriptlet: ceph-volume-2:19.3.0-4930.gfc3e0bff.el9.noarch 1/1 2024-09-15T20:18:29.710 INFO:teuthology.orchestra.run.smithi136.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-15T20:18:29.710 INFO:teuthology.orchestra.run.smithi136.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-09-15T20:18:29.710 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-09-15T20:18:30.681 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : ceph-volume-2:19.3.0-4930.gfc3e0bff.el9.noarch 1/1 2024-09-15T20:18:30.681 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-09-15T20:18:30.681 INFO:teuthology.orchestra.run.smithi136.stdout:Installed: 2024-09-15T20:18:30.681 INFO:teuthology.orchestra.run.smithi136.stdout: ceph-volume-2:19.3.0-4930.gfc3e0bff.el9.noarch 2024-09-15T20:18:30.681 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-09-15T20:18:30.682 INFO:teuthology.orchestra.run.smithi136.stdout:Complete! 2024-09-15T20:18:31.015 DEBUG:teuthology.orchestra.run.smithi136:> sudo yum -y install librados-devel 2024-09-15T20:18:31.607 INFO:teuthology.orchestra.run.smithi136.stdout:Last metadata expiration check: 0:01:33 ago on Sun 15 Sep 2024 08:16:58 PM UTC. 2024-09-15T20:18:31.745 INFO:teuthology.orchestra.run.smithi136.stdout:Dependencies resolved. 2024-09-15T20:18:31.745 INFO:teuthology.orchestra.run.smithi136.stdout:================================================================================ 2024-09-15T20:18:31.745 INFO:teuthology.orchestra.run.smithi136.stdout: Package Arch Version Repo Size 2024-09-15T20:18:31.745 INFO:teuthology.orchestra.run.smithi136.stdout:================================================================================ 2024-09-15T20:18:31.745 INFO:teuthology.orchestra.run.smithi136.stdout:Installing: 2024-09-15T20:18:31.745 INFO:teuthology.orchestra.run.smithi136.stdout: librados-devel x86_64 2:19.3.0-4930.gfc3e0bff.el9 ceph 127 k 2024-09-15T20:18:31.745 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-09-15T20:18:31.745 INFO:teuthology.orchestra.run.smithi136.stdout:Transaction Summary 2024-09-15T20:18:31.745 INFO:teuthology.orchestra.run.smithi136.stdout:================================================================================ 2024-09-15T20:18:31.746 INFO:teuthology.orchestra.run.smithi136.stdout:Install 1 Package 2024-09-15T20:18:31.746 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-09-15T20:18:31.746 INFO:teuthology.orchestra.run.smithi136.stdout:Total download size: 127 k 2024-09-15T20:18:31.746 INFO:teuthology.orchestra.run.smithi136.stdout:Installed size: 456 k 2024-09-15T20:18:31.746 INFO:teuthology.orchestra.run.smithi136.stdout:Downloading Packages: 2024-09-15T20:18:31.955 INFO:teuthology.orchestra.run.smithi136.stdout:librados-devel-19.3.0-4930.gfc3e0bff.el9.x86_64 600 kB/s | 127 kB 00:00 2024-09-15T20:18:31.956 INFO:teuthology.orchestra.run.smithi136.stdout:-------------------------------------------------------------------------------- 2024-09-15T20:18:31.956 INFO:teuthology.orchestra.run.smithi136.stdout:Total 596 kB/s | 127 kB 00:00 2024-09-15T20:18:31.956 INFO:teuthology.orchestra.run.smithi136.stdout:Running transaction check 2024-09-15T20:18:31.965 INFO:teuthology.orchestra.run.smithi136.stdout:Transaction check succeeded. 2024-09-15T20:18:31.966 INFO:teuthology.orchestra.run.smithi136.stdout:Running transaction test 2024-09-15T20:18:32.025 INFO:teuthology.orchestra.run.smithi136.stdout:Transaction test succeeded. 2024-09-15T20:18:32.025 INFO:teuthology.orchestra.run.smithi136.stdout:Running transaction 2024-09-15T20:18:32.174 INFO:teuthology.orchestra.run.smithi136.stdout: Preparing : 1/1 2024-09-15T20:18:32.270 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : librados-devel-2:19.3.0-4930.gfc3e0bff.el9.x86_64 1/1 2024-09-15T20:18:32.721 INFO:teuthology.orchestra.run.smithi136.stdout: Running scriptlet: librados-devel-2:19.3.0-4930.gfc3e0bff.el9.x86_64 1/1 2024-09-15T20:18:32.974 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : librados-devel-2:19.3.0-4930.gfc3e0bff.el9.x86_64 1/1 2024-09-15T20:18:32.974 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-09-15T20:18:32.974 INFO:teuthology.orchestra.run.smithi136.stdout:Installed: 2024-09-15T20:18:32.974 INFO:teuthology.orchestra.run.smithi136.stdout: librados-devel-2:19.3.0-4930.gfc3e0bff.el9.x86_64 2024-09-15T20:18:32.974 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-09-15T20:18:32.974 INFO:teuthology.orchestra.run.smithi136.stdout:Complete! 2024-09-15T20:18:33.303 DEBUG:teuthology.orchestra.run.smithi136:> sudo yum -y install libcephfs2 2024-09-15T20:18:33.891 INFO:teuthology.orchestra.run.smithi136.stdout:Last metadata expiration check: 0:01:35 ago on Sun 15 Sep 2024 08:16:58 PM UTC. 2024-09-15T20:18:33.989 INFO:teuthology.orchestra.run.smithi136.stdout:Package libcephfs2-2:19.3.0-4930.gfc3e0bff.el9.x86_64 is already installed. 2024-09-15T20:18:34.030 INFO:teuthology.orchestra.run.smithi136.stdout:Dependencies resolved. 2024-09-15T20:18:34.031 INFO:teuthology.orchestra.run.smithi136.stdout:Nothing to do. 2024-09-15T20:18:34.031 INFO:teuthology.orchestra.run.smithi136.stdout:Complete! 2024-09-15T20:18:34.103 DEBUG:teuthology.orchestra.run.smithi136:> sudo yum -y install libcephfs-devel 2024-09-15T20:18:34.677 INFO:teuthology.orchestra.run.smithi136.stdout:Last metadata expiration check: 0:01:36 ago on Sun 15 Sep 2024 08:16:58 PM UTC. 2024-09-15T20:18:34.810 INFO:teuthology.orchestra.run.smithi136.stdout:Dependencies resolved. 2024-09-15T20:18:34.811 INFO:teuthology.orchestra.run.smithi136.stdout:================================================================================ 2024-09-15T20:18:34.811 INFO:teuthology.orchestra.run.smithi136.stdout: Package Arch Version Repo Size 2024-09-15T20:18:34.811 INFO:teuthology.orchestra.run.smithi136.stdout:================================================================================ 2024-09-15T20:18:34.811 INFO:teuthology.orchestra.run.smithi136.stdout:Installing: 2024-09-15T20:18:34.811 INFO:teuthology.orchestra.run.smithi136.stdout: libcephfs-devel x86_64 2:19.3.0-4930.gfc3e0bff.el9 ceph 32 k 2024-09-15T20:18:34.811 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-09-15T20:18:34.811 INFO:teuthology.orchestra.run.smithi136.stdout:Transaction Summary 2024-09-15T20:18:34.811 INFO:teuthology.orchestra.run.smithi136.stdout:================================================================================ 2024-09-15T20:18:34.811 INFO:teuthology.orchestra.run.smithi136.stdout:Install 1 Package 2024-09-15T20:18:34.811 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-09-15T20:18:34.812 INFO:teuthology.orchestra.run.smithi136.stdout:Total download size: 32 k 2024-09-15T20:18:34.812 INFO:teuthology.orchestra.run.smithi136.stdout:Installed size: 142 k 2024-09-15T20:18:34.812 INFO:teuthology.orchestra.run.smithi136.stdout:Downloading Packages: 2024-09-15T20:18:34.951 INFO:teuthology.orchestra.run.smithi136.stdout:libcephfs-devel-19.3.0-4930.gfc3e0bff.el9.x86_6 229 kB/s | 32 kB 00:00 2024-09-15T20:18:34.951 INFO:teuthology.orchestra.run.smithi136.stdout:-------------------------------------------------------------------------------- 2024-09-15T20:18:34.951 INFO:teuthology.orchestra.run.smithi136.stdout:Total 227 kB/s | 32 kB 00:00 2024-09-15T20:18:34.951 INFO:teuthology.orchestra.run.smithi136.stdout:Running transaction check 2024-09-15T20:18:34.955 INFO:teuthology.orchestra.run.smithi136.stdout:Transaction check succeeded. 2024-09-15T20:18:34.955 INFO:teuthology.orchestra.run.smithi136.stdout:Running transaction test 2024-09-15T20:18:34.972 INFO:teuthology.orchestra.run.smithi136.stdout:Transaction test succeeded. 2024-09-15T20:18:34.972 INFO:teuthology.orchestra.run.smithi136.stdout:Running transaction 2024-09-15T20:18:35.054 INFO:teuthology.orchestra.run.smithi136.stdout: Preparing : 1/1 2024-09-15T20:18:35.149 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : libcephfs-devel-2:19.3.0-4930.gfc3e0bff.el9.x86_64 1/1 2024-09-15T20:18:35.511 INFO:teuthology.orchestra.run.smithi136.stdout: Running scriptlet: libcephfs-devel-2:19.3.0-4930.gfc3e0bff.el9.x86_64 1/1 2024-09-15T20:18:35.762 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : libcephfs-devel-2:19.3.0-4930.gfc3e0bff.el9.x86_64 1/1 2024-09-15T20:18:35.762 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-09-15T20:18:35.762 INFO:teuthology.orchestra.run.smithi136.stdout:Installed: 2024-09-15T20:18:35.762 INFO:teuthology.orchestra.run.smithi136.stdout: libcephfs-devel-2:19.3.0-4930.gfc3e0bff.el9.x86_64 2024-09-15T20:18:35.762 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-09-15T20:18:35.762 INFO:teuthology.orchestra.run.smithi136.stdout:Complete! 2024-09-15T20:18:35.956 DEBUG:teuthology.orchestra.run.smithi136:> sudo yum -y install librados2 2024-09-15T20:18:36.526 INFO:teuthology.orchestra.run.smithi136.stdout:Last metadata expiration check: 0:01:38 ago on Sun 15 Sep 2024 08:16:58 PM UTC. 2024-09-15T20:18:36.617 INFO:teuthology.orchestra.run.smithi136.stdout:Package librados2-2:19.3.0-4930.gfc3e0bff.el9.x86_64 is already installed. 2024-09-15T20:18:36.660 INFO:teuthology.orchestra.run.smithi136.stdout:Dependencies resolved. 2024-09-15T20:18:36.661 INFO:teuthology.orchestra.run.smithi136.stdout:Nothing to do. 2024-09-15T20:18:36.661 INFO:teuthology.orchestra.run.smithi136.stdout:Complete! 2024-09-15T20:18:36.724 DEBUG:teuthology.orchestra.run.smithi136:> sudo yum -y install librbd1 2024-09-15T20:18:37.289 INFO:teuthology.orchestra.run.smithi136.stdout:Last metadata expiration check: 0:01:39 ago on Sun 15 Sep 2024 08:16:58 PM UTC. 2024-09-15T20:18:37.381 INFO:teuthology.orchestra.run.smithi136.stdout:Package librbd1-2:19.3.0-4930.gfc3e0bff.el9.x86_64 is already installed. 2024-09-15T20:18:37.430 INFO:teuthology.orchestra.run.smithi136.stdout:Dependencies resolved. 2024-09-15T20:18:37.430 INFO:teuthology.orchestra.run.smithi136.stdout:Nothing to do. 2024-09-15T20:18:37.430 INFO:teuthology.orchestra.run.smithi136.stdout:Complete! 2024-09-15T20:18:37.497 DEBUG:teuthology.orchestra.run.smithi136:> sudo yum -y install python3-rados 2024-09-15T20:18:38.068 INFO:teuthology.orchestra.run.smithi136.stdout:Last metadata expiration check: 0:01:40 ago on Sun 15 Sep 2024 08:16:58 PM UTC. 2024-09-15T20:18:38.159 INFO:teuthology.orchestra.run.smithi136.stdout:Package python3-rados-2:19.3.0-4930.gfc3e0bff.el9.x86_64 is already installed. 2024-09-15T20:18:38.202 INFO:teuthology.orchestra.run.smithi136.stdout:Dependencies resolved. 2024-09-15T20:18:38.208 INFO:teuthology.orchestra.run.smithi136.stdout:Nothing to do. 2024-09-15T20:18:38.208 INFO:teuthology.orchestra.run.smithi136.stdout:Complete! 2024-09-15T20:18:38.280 DEBUG:teuthology.orchestra.run.smithi136:> sudo yum -y install python3-rgw 2024-09-15T20:18:38.845 INFO:teuthology.orchestra.run.smithi136.stdout:Last metadata expiration check: 0:01:40 ago on Sun 15 Sep 2024 08:16:58 PM UTC. 2024-09-15T20:18:38.935 INFO:teuthology.orchestra.run.smithi136.stdout:Package python3-rgw-2:19.3.0-4930.gfc3e0bff.el9.x86_64 is already installed. 2024-09-15T20:18:38.978 INFO:teuthology.orchestra.run.smithi136.stdout:Dependencies resolved. 2024-09-15T20:18:38.979 INFO:teuthology.orchestra.run.smithi136.stdout:Nothing to do. 2024-09-15T20:18:38.979 INFO:teuthology.orchestra.run.smithi136.stdout:Complete! 2024-09-15T20:18:39.054 DEBUG:teuthology.orchestra.run.smithi136:> sudo yum -y install python3-cephfs 2024-09-15T20:18:39.627 INFO:teuthology.orchestra.run.smithi136.stdout:Last metadata expiration check: 0:01:41 ago on Sun 15 Sep 2024 08:16:58 PM UTC. 2024-09-15T20:18:39.721 INFO:teuthology.orchestra.run.smithi136.stdout:Package python3-cephfs-2:19.3.0-4930.gfc3e0bff.el9.x86_64 is already installed. 2024-09-15T20:18:39.760 INFO:teuthology.orchestra.run.smithi136.stdout:Dependencies resolved. 2024-09-15T20:18:39.761 INFO:teuthology.orchestra.run.smithi136.stdout:Nothing to do. 2024-09-15T20:18:39.761 INFO:teuthology.orchestra.run.smithi136.stdout:Complete! 2024-09-15T20:18:39.835 DEBUG:teuthology.orchestra.run.smithi136:> sudo yum -y install python3-rbd 2024-09-15T20:18:40.403 INFO:teuthology.orchestra.run.smithi136.stdout:Last metadata expiration check: 0:01:42 ago on Sun 15 Sep 2024 08:16:58 PM UTC. 2024-09-15T20:18:40.495 INFO:teuthology.orchestra.run.smithi136.stdout:Package python3-rbd-2:19.3.0-4930.gfc3e0bff.el9.x86_64 is already installed. 2024-09-15T20:18:40.539 INFO:teuthology.orchestra.run.smithi136.stdout:Dependencies resolved. 2024-09-15T20:18:40.540 INFO:teuthology.orchestra.run.smithi136.stdout:Nothing to do. 2024-09-15T20:18:40.540 INFO:teuthology.orchestra.run.smithi136.stdout:Complete! 2024-09-15T20:18:40.610 DEBUG:teuthology.orchestra.run.smithi136:> sudo yum -y install rbd-fuse 2024-09-15T20:18:41.178 INFO:teuthology.orchestra.run.smithi136.stdout:Last metadata expiration check: 0:01:43 ago on Sun 15 Sep 2024 08:16:58 PM UTC. 2024-09-15T20:18:41.312 INFO:teuthology.orchestra.run.smithi136.stdout:Dependencies resolved. 2024-09-15T20:18:41.313 INFO:teuthology.orchestra.run.smithi136.stdout:================================================================================ 2024-09-15T20:18:41.313 INFO:teuthology.orchestra.run.smithi136.stdout: Package Architecture Version Repository Size 2024-09-15T20:18:41.313 INFO:teuthology.orchestra.run.smithi136.stdout:================================================================================ 2024-09-15T20:18:41.313 INFO:teuthology.orchestra.run.smithi136.stdout:Installing: 2024-09-15T20:18:41.313 INFO:teuthology.orchestra.run.smithi136.stdout: rbd-fuse x86_64 2:19.3.0-4930.gfc3e0bff.el9 ceph 85 k 2024-09-15T20:18:41.313 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-09-15T20:18:41.313 INFO:teuthology.orchestra.run.smithi136.stdout:Transaction Summary 2024-09-15T20:18:41.314 INFO:teuthology.orchestra.run.smithi136.stdout:================================================================================ 2024-09-15T20:18:41.314 INFO:teuthology.orchestra.run.smithi136.stdout:Install 1 Package 2024-09-15T20:18:41.314 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-09-15T20:18:41.314 INFO:teuthology.orchestra.run.smithi136.stdout:Total download size: 85 k 2024-09-15T20:18:41.314 INFO:teuthology.orchestra.run.smithi136.stdout:Installed size: 231 k 2024-09-15T20:18:41.314 INFO:teuthology.orchestra.run.smithi136.stdout:Downloading Packages: 2024-09-15T20:18:41.511 INFO:teuthology.orchestra.run.smithi136.stdout:rbd-fuse-19.3.0-4930.gfc3e0bff.el9.x86_64.rpm 431 kB/s | 85 kB 00:00 2024-09-15T20:18:41.512 INFO:teuthology.orchestra.run.smithi136.stdout:-------------------------------------------------------------------------------- 2024-09-15T20:18:41.512 INFO:teuthology.orchestra.run.smithi136.stdout:Total 429 kB/s | 85 kB 00:00 2024-09-15T20:18:41.512 INFO:teuthology.orchestra.run.smithi136.stdout:Running transaction check 2024-09-15T20:18:41.521 INFO:teuthology.orchestra.run.smithi136.stdout:Transaction check succeeded. 2024-09-15T20:18:41.521 INFO:teuthology.orchestra.run.smithi136.stdout:Running transaction test 2024-09-15T20:18:41.582 INFO:teuthology.orchestra.run.smithi136.stdout:Transaction test succeeded. 2024-09-15T20:18:41.582 INFO:teuthology.orchestra.run.smithi136.stdout:Running transaction 2024-09-15T20:18:41.740 INFO:teuthology.orchestra.run.smithi136.stdout: Preparing : 1/1 2024-09-15T20:18:41.825 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : rbd-fuse-2:19.3.0-4930.gfc3e0bff.el9.x86_64 1/1 2024-09-15T20:18:42.195 INFO:teuthology.orchestra.run.smithi136.stdout: Running scriptlet: rbd-fuse-2:19.3.0-4930.gfc3e0bff.el9.x86_64 1/1 2024-09-15T20:18:42.454 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : rbd-fuse-2:19.3.0-4930.gfc3e0bff.el9.x86_64 1/1 2024-09-15T20:18:42.454 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-09-15T20:18:42.454 INFO:teuthology.orchestra.run.smithi136.stdout:Installed: 2024-09-15T20:18:42.454 INFO:teuthology.orchestra.run.smithi136.stdout: rbd-fuse-2:19.3.0-4930.gfc3e0bff.el9.x86_64 2024-09-15T20:18:42.454 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-09-15T20:18:42.454 INFO:teuthology.orchestra.run.smithi136.stdout:Complete! 2024-09-15T20:18:42.714 DEBUG:teuthology.orchestra.run.smithi136:> sudo yum -y install rbd-mirror 2024-09-15T20:18:43.296 INFO:teuthology.orchestra.run.smithi136.stdout:Last metadata expiration check: 0:01:45 ago on Sun 15 Sep 2024 08:16:58 PM UTC. 2024-09-15T20:18:43.434 INFO:teuthology.orchestra.run.smithi136.stdout:Dependencies resolved. 2024-09-15T20:18:43.434 INFO:teuthology.orchestra.run.smithi136.stdout:================================================================================ 2024-09-15T20:18:43.434 INFO:teuthology.orchestra.run.smithi136.stdout: Package Arch Version Repo Size 2024-09-15T20:18:43.434 INFO:teuthology.orchestra.run.smithi136.stdout:================================================================================ 2024-09-15T20:18:43.434 INFO:teuthology.orchestra.run.smithi136.stdout:Installing: 2024-09-15T20:18:43.434 INFO:teuthology.orchestra.run.smithi136.stdout: rbd-mirror x86_64 2:19.3.0-4930.gfc3e0bff.el9 ceph 3.2 M 2024-09-15T20:18:43.434 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-09-15T20:18:43.435 INFO:teuthology.orchestra.run.smithi136.stdout:Transaction Summary 2024-09-15T20:18:43.435 INFO:teuthology.orchestra.run.smithi136.stdout:================================================================================ 2024-09-15T20:18:43.435 INFO:teuthology.orchestra.run.smithi136.stdout:Install 1 Package 2024-09-15T20:18:43.435 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-09-15T20:18:43.435 INFO:teuthology.orchestra.run.smithi136.stdout:Total download size: 3.2 M 2024-09-15T20:18:43.435 INFO:teuthology.orchestra.run.smithi136.stdout:Installed size: 14 M 2024-09-15T20:18:43.435 INFO:teuthology.orchestra.run.smithi136.stdout:Downloading Packages: 2024-09-15T20:18:44.784 INFO:teuthology.orchestra.run.smithi136.stdout:rbd-mirror-19.3.0-4930.gfc3e0bff.el9.x86_64.rpm 2.4 MB/s | 3.2 MB 00:01 2024-09-15T20:18:44.785 INFO:teuthology.orchestra.run.smithi136.stdout:-------------------------------------------------------------------------------- 2024-09-15T20:18:44.785 INFO:teuthology.orchestra.run.smithi136.stdout:Total 2.4 MB/s | 3.2 MB 00:01 2024-09-15T20:18:44.785 INFO:teuthology.orchestra.run.smithi136.stdout:Running transaction check 2024-09-15T20:18:44.796 INFO:teuthology.orchestra.run.smithi136.stdout:Transaction check succeeded. 2024-09-15T20:18:44.796 INFO:teuthology.orchestra.run.smithi136.stdout:Running transaction test 2024-09-15T20:18:44.879 INFO:teuthology.orchestra.run.smithi136.stdout:Transaction test succeeded. 2024-09-15T20:18:44.879 INFO:teuthology.orchestra.run.smithi136.stdout:Running transaction 2024-09-15T20:18:45.324 INFO:teuthology.orchestra.run.smithi136.stdout: Preparing : 1/1 2024-09-15T20:18:45.392 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : rbd-mirror-2:19.3.0-4930.gfc3e0bff.el9.x86_64 1/1 2024-09-15T20:18:45.421 INFO:teuthology.orchestra.run.smithi136.stdout: Running scriptlet: rbd-mirror-2:19.3.0-4930.gfc3e0bff.el9.x86_64 1/1 2024-09-15T20:18:45.421 INFO:teuthology.orchestra.run.smithi136.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-15T20:18:45.421 INFO:teuthology.orchestra.run.smithi136.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-09-15T20:18:45.421 INFO:teuthology.orchestra.run.smithi136.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-09-15T20:18:45.421 INFO:teuthology.orchestra.run.smithi136.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-09-15T20:18:45.421 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-09-15T20:18:46.486 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : rbd-mirror-2:19.3.0-4930.gfc3e0bff.el9.x86_64 1/1 2024-09-15T20:18:46.486 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-09-15T20:18:46.486 INFO:teuthology.orchestra.run.smithi136.stdout:Installed: 2024-09-15T20:18:46.486 INFO:teuthology.orchestra.run.smithi136.stdout: rbd-mirror-2:19.3.0-4930.gfc3e0bff.el9.x86_64 2024-09-15T20:18:46.486 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-09-15T20:18:46.486 INFO:teuthology.orchestra.run.smithi136.stdout:Complete! 2024-09-15T20:18:46.711 DEBUG:teuthology.orchestra.run.smithi136:> sudo yum -y install rbd-nbd 2024-09-15T20:18:47.289 INFO:teuthology.orchestra.run.smithi136.stdout:Last metadata expiration check: 0:01:49 ago on Sun 15 Sep 2024 08:16:58 PM UTC. 2024-09-15T20:18:47.423 INFO:teuthology.orchestra.run.smithi136.stdout:Dependencies resolved. 2024-09-15T20:18:47.423 INFO:teuthology.orchestra.run.smithi136.stdout:================================================================================ 2024-09-15T20:18:47.423 INFO:teuthology.orchestra.run.smithi136.stdout: Package Architecture Version Repository Size 2024-09-15T20:18:47.423 INFO:teuthology.orchestra.run.smithi136.stdout:================================================================================ 2024-09-15T20:18:47.423 INFO:teuthology.orchestra.run.smithi136.stdout:Installing: 2024-09-15T20:18:47.423 INFO:teuthology.orchestra.run.smithi136.stdout: rbd-nbd x86_64 2:19.3.0-4930.gfc3e0bff.el9 ceph 170 k 2024-09-15T20:18:47.423 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-09-15T20:18:47.423 INFO:teuthology.orchestra.run.smithi136.stdout:Transaction Summary 2024-09-15T20:18:47.424 INFO:teuthology.orchestra.run.smithi136.stdout:================================================================================ 2024-09-15T20:18:47.424 INFO:teuthology.orchestra.run.smithi136.stdout:Install 1 Package 2024-09-15T20:18:47.424 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-09-15T20:18:47.424 INFO:teuthology.orchestra.run.smithi136.stdout:Total download size: 170 k 2024-09-15T20:18:47.424 INFO:teuthology.orchestra.run.smithi136.stdout:Installed size: 490 k 2024-09-15T20:18:47.424 INFO:teuthology.orchestra.run.smithi136.stdout:Downloading Packages: 2024-09-15T20:18:47.654 INFO:teuthology.orchestra.run.smithi136.stdout:rbd-nbd-19.3.0-4930.gfc3e0bff.el9.x86_64.rpm 743 kB/s | 170 kB 00:00 2024-09-15T20:18:47.654 INFO:teuthology.orchestra.run.smithi136.stdout:-------------------------------------------------------------------------------- 2024-09-15T20:18:47.654 INFO:teuthology.orchestra.run.smithi136.stdout:Total 738 kB/s | 170 kB 00:00 2024-09-15T20:18:47.655 INFO:teuthology.orchestra.run.smithi136.stdout:Running transaction check 2024-09-15T20:18:47.662 INFO:teuthology.orchestra.run.smithi136.stdout:Transaction check succeeded. 2024-09-15T20:18:47.662 INFO:teuthology.orchestra.run.smithi136.stdout:Running transaction test 2024-09-15T20:18:47.726 INFO:teuthology.orchestra.run.smithi136.stdout:Transaction test succeeded. 2024-09-15T20:18:47.726 INFO:teuthology.orchestra.run.smithi136.stdout:Running transaction 2024-09-15T20:18:47.890 INFO:teuthology.orchestra.run.smithi136.stdout: Preparing : 1/1 2024-09-15T20:18:48.001 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : rbd-nbd-2:19.3.0-4930.gfc3e0bff.el9.x86_64 1/1 2024-09-15T20:18:48.421 INFO:teuthology.orchestra.run.smithi136.stdout: Running scriptlet: rbd-nbd-2:19.3.0-4930.gfc3e0bff.el9.x86_64 1/1 2024-09-15T20:18:48.711 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : rbd-nbd-2:19.3.0-4930.gfc3e0bff.el9.x86_64 1/1 2024-09-15T20:18:48.711 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-09-15T20:18:48.711 INFO:teuthology.orchestra.run.smithi136.stdout:Installed: 2024-09-15T20:18:48.712 INFO:teuthology.orchestra.run.smithi136.stdout: rbd-nbd-2:19.3.0-4930.gfc3e0bff.el9.x86_64 2024-09-15T20:18:48.712 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-09-15T20:18:48.712 INFO:teuthology.orchestra.run.smithi136.stdout:Complete! 2024-09-15T20:18:48.965 DEBUG:teuthology.parallel:result is None 2024-09-15T20:18:48.966 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=fc3e0bff7468fd55478adbae7f75508b749124b6 2024-09-15T20:18:49.112 DEBUG:teuthology.orchestra.run.smithi136:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-09-15T20:18:49.133 INFO:teuthology.orchestra.run.smithi136.stdout:19.3.0-4930.gfc3e0bff.el9 2024-09-15T20:18:49.133 INFO:teuthology.packaging:The installed version of ceph is 19.3.0-4930.gfc3e0bff.el9 2024-09-15T20:18:49.133 INFO:teuthology.task.install:The correct ceph version 19.3.0-4930.gfc3e0bff is installed. 2024-09-15T20:18:49.135 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-09-15T20:18:49.135 DEBUG:teuthology.orchestra.run.smithi136:> set -ex 2024-09-15T20:18:49.135 DEBUG:teuthology.orchestra.run.smithi136:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-09-15T20:18:49.200 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-09-15T20:18:49.201 DEBUG:teuthology.orchestra.run.smithi136:> set -ex 2024-09-15T20:18:49.201 DEBUG:teuthology.orchestra.run.smithi136:> sudo dd of=/usr/bin/daemon-helper 2024-09-15T20:18:49.264 DEBUG:teuthology.orchestra.run.smithi136:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-09-15T20:18:49.329 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-09-15T20:18:49.329 DEBUG:teuthology.orchestra.run.smithi136:> set -ex 2024-09-15T20:18:49.329 DEBUG:teuthology.orchestra.run.smithi136:> sudo dd of=/usr/bin/adjust-ulimits 2024-09-15T20:18:49.395 DEBUG:teuthology.orchestra.run.smithi136:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-09-15T20:18:49.459 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-09-15T20:18:49.459 DEBUG:teuthology.orchestra.run.smithi136:> set -ex 2024-09-15T20:18:49.459 DEBUG:teuthology.orchestra.run.smithi136:> sudo dd of=/usr/bin/stdin-killer 2024-09-15T20:18:49.523 DEBUG:teuthology.orchestra.run.smithi136:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-09-15T20:18:49.590 INFO:teuthology.run_tasks:Running task workunit... 2024-09-15T20:18:49.605 INFO:tasks.workunit:Pulling workunits from ref 7498493279982dfe87f00616198a5967475f6169 2024-09-15T20:18:49.606 INFO:tasks.workunit:Making a separate scratch dir for every client... 2024-09-15T20:18:49.606 INFO:tasks.workunit:timeout=3h 2024-09-15T20:18:49.606 INFO:tasks.workunit:cleanup=True 2024-09-15T20:18:49.606 DEBUG:teuthology.orchestra.run.smithi136:> stat -- /home/ubuntu/cephtest/mnt.0 2024-09-15T20:18:49.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-15T20:18:49.647 INFO:teuthology.orchestra.run.smithi136.stderr:stat: cannot statx '/home/ubuntu/cephtest/mnt.0': No such file or directory 2024-09-15T20:18:49.647 DEBUG:teuthology.orchestra.run.smithi136:> mkdir -- /home/ubuntu/cephtest/mnt.0 2024-09-15T20:18:49.703 INFO:tasks.workunit:Created dir /home/ubuntu/cephtest/mnt.0 2024-09-15T20:18:49.703 DEBUG:teuthology.orchestra.run.smithi136:> cd -- /home/ubuntu/cephtest/mnt.0 && mkdir -- client.0 2024-09-15T20:18:49.761 DEBUG:teuthology.orchestra.run.smithi136:> rm -rf /home/ubuntu/cephtest/clone.client.0 && git clone https://git.ceph.com/ceph.git /home/ubuntu/cephtest/clone.client.0 && cd /home/ubuntu/cephtest/clone.client.0 && git checkout 7498493279982dfe87f00616198a5967475f6169 2024-09-15T20:18:49.824 INFO:tasks.workunit.client.0.smithi136.stderr:Cloning into '/home/ubuntu/cephtest/clone.client.0'... 2024-09-15T20:19:44.918 INFO:tasks.workunit.client.0.smithi136.stderr:Note: switching to '7498493279982dfe87f00616198a5967475f6169'. 2024-09-15T20:19:44.919 INFO:tasks.workunit.client.0.smithi136.stderr: 2024-09-15T20:19:44.919 INFO:tasks.workunit.client.0.smithi136.stderr:You are in 'detached HEAD' state. You can look around, make experimental 2024-09-15T20:19:44.919 INFO:tasks.workunit.client.0.smithi136.stderr:changes and commit them, and you can discard any commits you make in this 2024-09-15T20:19:44.919 INFO:tasks.workunit.client.0.smithi136.stderr:state without impacting any branches by switching back to a branch. 2024-09-15T20:19:44.919 INFO:tasks.workunit.client.0.smithi136.stderr: 2024-09-15T20:19:44.919 INFO:tasks.workunit.client.0.smithi136.stderr:If you want to create a new branch to retain commits you create, you may 2024-09-15T20:19:44.919 INFO:tasks.workunit.client.0.smithi136.stderr:do so (now or later) by using -c with the switch command. Example: 2024-09-15T20:19:44.919 INFO:tasks.workunit.client.0.smithi136.stderr: 2024-09-15T20:19:44.919 INFO:tasks.workunit.client.0.smithi136.stderr: git switch -c 2024-09-15T20:19:44.919 INFO:tasks.workunit.client.0.smithi136.stderr: 2024-09-15T20:19:44.920 INFO:tasks.workunit.client.0.smithi136.stderr:Or undo this operation with: 2024-09-15T20:19:44.920 INFO:tasks.workunit.client.0.smithi136.stderr: 2024-09-15T20:19:44.920 INFO:tasks.workunit.client.0.smithi136.stderr: git switch - 2024-09-15T20:19:44.920 INFO:tasks.workunit.client.0.smithi136.stderr: 2024-09-15T20:19:44.920 INFO:tasks.workunit.client.0.smithi136.stderr:Turn off this advice by setting config variable advice.detachedHead to false 2024-09-15T20:19:44.920 INFO:tasks.workunit.client.0.smithi136.stderr: 2024-09-15T20:19:44.920 INFO:tasks.workunit.client.0.smithi136.stderr:HEAD is now at 74984932799 Merge pull request #59797 from zdover23/wip-doc-2024-09-15-install-install-storage-cluster 2024-09-15T20:19:44.920 DEBUG:teuthology.orchestra.run.smithi136:> cd -- /home/ubuntu/cephtest/clone.client.0/qa/standalone && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\0' >/home/ubuntu/cephtest/workunits.list.client.0 2024-09-15T20:19:44.975 DEBUG:teuthology.orchestra.run.smithi136:> set -ex 2024-09-15T20:19:44.975 DEBUG:teuthology.orchestra.run.smithi136:> dd if=/home/ubuntu/cephtest/workunits.list.client.0 of=/dev/stdout 2024-09-15T20:19:45.031 INFO:tasks.workunit:Running workunits matching scrub on client.0... 2024-09-15T20:19:45.032 INFO:tasks.workunit:Running workunit scrub/osd-mapper.sh... 2024-09-15T20:19:45.032 DEBUG:teuthology.orchestra.run.smithi136:workunit test scrub/osd-mapper.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=7498493279982dfe87f00616198a5967475f6169 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh 2024-09-15T20:19:45.100 INFO:tasks.workunit.client.0.smithi136.stderr:stty: 'standard input': Inappropriate ioctl for device 2024-09-15T20:19:45.106 INFO:tasks.workunit.client.0.smithi136.stderr:+ PS4='${BASH_SOURCE[0]}:$LINENO: ${FUNCNAME[0]}: ' 2024-09-15T20:19:45.106 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2315: main: export PATH=.:/home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin 2024-09-15T20:19:45.106 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2315: main: PATH=.:/home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin 2024-09-15T20:19:45.106 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2316: main: export PYTHONWARNINGS=ignore 2024-09-15T20:19:45.106 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2316: main: PYTHONWARNINGS=ignore 2024-09-15T20:19:45.106 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2317: main: export CEPH_CONF=/dev/null 2024-09-15T20:19:45.106 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2317: main: CEPH_CONF=/dev/null 2024-09-15T20:19:45.106 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2318: main: unset CEPH_ARGS 2024-09-15T20:19:45.106 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2320: main: local code 2024-09-15T20:19:45.106 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2321: main: run td/osd-mapper 2024-09-15T20:19:45.106 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:11: run: local dir=td/osd-mapper 2024-09-15T20:19:45.106 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:12: run: shift 2024-09-15T20:19:45.106 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:14: run: export CEPH_MON=127.0.0.1:7144 2024-09-15T20:19:45.107 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:14: run: CEPH_MON=127.0.0.1:7144 2024-09-15T20:19:45.107 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:15: run: export CEPH_ARGS 2024-09-15T20:19:45.107 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:16: run: uuidgen 2024-09-15T20:19:45.115 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:16: run: CEPH_ARGS+='--fsid=e9f4cc96-8f95-4971-84e5-d64acd70df0f --auth-supported=none ' 2024-09-15T20:19:45.115 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:17: run: CEPH_ARGS+='--mon-host=127.0.0.1:7144 ' 2024-09-15T20:19:45.115 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:19: run: export -n CEPH_CLI_TEST_DUP_COMMAND 2024-09-15T20:19:45.116 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:20: run: set 2024-09-15T20:19:45.116 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:20: run: sed -n -e 's/^\(TEST_[0-9a-z_]*\) .*/\1/p' 2024-09-15T20:19:45.120 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:20: run: local funcs=TEST_truncated_sna_record 2024-09-15T20:19:45.120 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:21: run: for func in $funcs 2024-09-15T20:19:45.120 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:22: run: setup td/osd-mapper 2024-09-15T20:19:45.120 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/osd-mapper 2024-09-15T20:19:45.120 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/osd-mapper 2024-09-15T20:19:45.120 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/osd-mapper 2024-09-15T20:19:45.120 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-15T20:19:45.120 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/osd-mapper KILL 2024-09-15T20:19:45.120 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-15T20:19:45.121 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-15T20:19:45.121 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-15T20:19:45.121 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-15T20:19:45.121 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-15T20:19:45.121 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-15T20:19:45.121 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-15T20:19:45.122 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-15T20:19:45.122 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-15T20:19:45.122 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-15T20:19:45.122 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-15T20:19:45.122 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-15T20:19:45.123 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T20:19:45.123 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-15T20:19:45.123 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-15T20:19:45.123 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T20:19:45.123 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-15T20:19:45.126 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-15T20:19:45.126 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/osd-mapper 2024-09-15T20:19:45.126 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-15T20:19:45.126 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T20:19:45.126 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.27314 2024-09-15T20:19:45.126 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.27314 2024-09-15T20:19:45.126 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-15T20:19:45.126 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-15T20:19:45.126 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/osd-mapper 2024-09-15T20:19:45.127 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2024-09-15T20:19:45.127 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T20:19:45.128 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.27314 2024-09-15T20:19:45.128 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.27314 2024-09-15T20:19:45.134 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2024-09-15T20:19:45.140 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 1024 -le 1024 ']' 2024-09-15T20:19:45.142 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:127: setup: ulimit -n 4096 2024-09-15T20:19:45.143 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2024-09-15T20:19:45.143 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/osd-mapper 1' TERM HUP INT 2024-09-15T20:19:45.143 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:23: run: TEST_truncated_sna_record td/osd-mapper 2024-09-15T20:19:45.143 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:45: TEST_truncated_sna_record: local dir=td/osd-mapper 2024-09-15T20:19:45.143 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:46: TEST_truncated_sna_record: cluster_conf=(['osds_num']='3' ['pgs_in_pool']='4' ['pool_name']='test') 2024-09-15T20:19:45.143 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:46: TEST_truncated_sna_record: local -A cluster_conf 2024-09-15T20:19:45.143 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:52: TEST_truncated_sna_record: local extr_dbg=3 2024-09-15T20:19:45.143 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:53: TEST_truncated_sna_record: (( extr_dbg > 1 )) 2024-09-15T20:19:45.143 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:53: TEST_truncated_sna_record: echo 'Dir: td/osd-mapper' 2024-09-15T20:19:45.143 INFO:tasks.workunit.client.0.smithi136.stdout:Dir: td/osd-mapper 2024-09-15T20:19:45.145 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:54: TEST_truncated_sna_record: standard_scrub_cluster td/osd-mapper cluster_conf 2024-09-15T20:19:45.146 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:229: standard_scrub_cluster: local dir=td/osd-mapper 2024-09-15T20:19:45.146 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:230: standard_scrub_cluster: local -n args=cluster_conf 2024-09-15T20:19:45.146 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:232: standard_scrub_cluster: local OSDS=3 2024-09-15T20:19:45.146 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:233: standard_scrub_cluster: local pg_num=4 2024-09-15T20:19:45.146 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:234: standard_scrub_cluster: local poolname=test 2024-09-15T20:19:45.146 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:235: standard_scrub_cluster: args['pool_name']=test 2024-09-15T20:19:45.147 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:236: standard_scrub_cluster: local extra_pars= 2024-09-15T20:19:45.147 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:237: standard_scrub_cluster: local debug_msg=dbg 2024-09-15T20:19:45.147 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:240: standard_scrub_cluster: local saved_echo_flag=x 2024-09-15T20:19:45.147 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:241: standard_scrub_cluster: set +x 2024-09-15T20:19:46.774 INFO:tasks.workunit.client.0.smithi136.stdout:add osd0 130dbee2-9dfc-4ede-9d5f-ae08c2872427 2024-09-15T20:19:46.992 INFO:tasks.workunit.client.0.smithi136.stdout:0 2024-09-15T20:19:47.086 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:19:47.085+0000 7f7bd7ab0640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T20:19:47.119 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:19:47.118+0000 7f7bd7ab0640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T20:19:47.120 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:19:47.119+0000 7f7bd7ab0640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T20:19:47.121 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:19:47.119+0000 7f7bd7ab0640 -1 bdev(0x558d34c61c00 td/osd-mapper/0/block) open stat got: (1) Operation not permitted 2024-09-15T20:19:47.121 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:19:47.120+0000 7f7bd7ab0640 -1 bluestore(td/osd-mapper/0) _read_fsid unparsable uuid 2024-09-15T20:19:53.112 INFO:tasks.workunit.client.0.smithi136.stdout:adding osd0 key to auth repository 2024-09-15T20:19:53.466 INFO:tasks.workunit.client.0.smithi136.stdout:start osd.0 2024-09-15T20:19:53.498 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:19:53.496+0000 7fdf2de3e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T20:19:53.500 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:19:53.498+0000 7fdf2de3e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T20:19:53.501 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:19:53.499+0000 7fdf2de3e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T20:19:53.741 INFO:tasks.workunit.client.0.smithi136.stdout:0 2024-09-15T20:19:54.615 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:19:54.612+0000 7fdf2de3e640 -1 Falling back to public interface 2024-09-15T20:19:55.015 INFO:tasks.workunit.client.0.smithi136.stdout:1 2024-09-15T20:19:56.298 INFO:tasks.workunit.client.0.smithi136.stdout:2 2024-09-15T20:19:57.083 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:19:57.081+0000 7fdf2de3e640 -1 osd.0 0 log_to_monitors true 2024-09-15T20:19:57.581 INFO:tasks.workunit.client.0.smithi136.stdout:3 2024-09-15T20:19:58.971 INFO:tasks.workunit.client.0.smithi136.stdout:4 2024-09-15T20:20:00.255 INFO:tasks.workunit.client.0.smithi136.stdout:5 2024-09-15T20:20:01.548 INFO:tasks.workunit.client.0.smithi136.stdout:6 2024-09-15T20:20:02.831 INFO:tasks.workunit.client.0.smithi136.stdout:7 2024-09-15T20:20:04.119 INFO:tasks.workunit.client.0.smithi136.stdout:8 2024-09-15T20:20:05.403 INFO:tasks.workunit.client.0.smithi136.stdout:9 2024-09-15T20:20:06.792 INFO:tasks.workunit.client.0.smithi136.stdout:10 2024-09-15T20:20:08.077 INFO:tasks.workunit.client.0.smithi136.stdout:11 2024-09-15T20:20:09.453 INFO:tasks.workunit.client.0.smithi136.stdout:12 2024-09-15T20:20:10.790 INFO:tasks.workunit.client.0.smithi136.stdout:13 2024-09-15T20:20:12.249 INFO:tasks.workunit.client.0.smithi136.stdout:14 2024-09-15T20:20:14.113 INFO:tasks.workunit.client.0.smithi136.stdout:15 2024-09-15T20:20:15.681 INFO:tasks.workunit.client.0.smithi136.stdout:16 2024-09-15T20:20:16.963 INFO:tasks.workunit.client.0.smithi136.stdout:17 2024-09-15T20:20:18.727 INFO:tasks.workunit.client.0.smithi136.stdout:18 2024-09-15T20:20:20.015 INFO:tasks.workunit.client.0.smithi136.stdout:19 2024-09-15T20:20:21.399 INFO:tasks.workunit.client.0.smithi136.stdout:20 2024-09-15T20:20:21.893 INFO:tasks.workunit.client.0.smithi136.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/598587094,v1:127.0.0.1:6803/598587094] [v2:127.0.0.1:6804/598587094,v1:127.0.0.1:6805/598587094] exists,up 130dbee2-9dfc-4ede-9d5f-ae08c2872427 2024-09-15T20:20:21.896 INFO:tasks.workunit.client.0.smithi136.stdout:add osd1 aa3a94d1-c180-456f-8f8b-c75b0a4a1477 2024-09-15T20:20:22.382 INFO:tasks.workunit.client.0.smithi136.stdout:1 2024-09-15T20:20:22.428 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:20:22.427+0000 7fcdd2018640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T20:20:22.468 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:20:22.467+0000 7fcdd2018640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T20:20:22.469 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:20:22.468+0000 7fcdd2018640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T20:20:22.470 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:20:22.468+0000 7fcdd2018640 -1 bdev(0x55a621857c00 td/osd-mapper/1/block) open stat got: (1) Operation not permitted 2024-09-15T20:20:22.470 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:20:22.468+0000 7fcdd2018640 -1 bluestore(td/osd-mapper/1) _read_fsid unparsable uuid 2024-09-15T20:20:28.146 INFO:tasks.workunit.client.0.smithi136.stdout:adding osd1 key to auth repository 2024-09-15T20:20:28.508 INFO:tasks.workunit.client.0.smithi136.stdout:start osd.1 2024-09-15T20:20:28.540 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:20:28.537+0000 7f77baa11640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T20:20:28.549 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:20:28.539+0000 7f77baa11640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T20:20:28.549 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:20:28.540+0000 7f77baa11640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T20:20:28.782 INFO:tasks.workunit.client.0.smithi136.stdout:0 2024-09-15T20:20:29.898 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:20:29.896+0000 7f77baa11640 -1 Falling back to public interface 2024-09-15T20:20:30.056 INFO:tasks.workunit.client.0.smithi136.stdout:1 2024-09-15T20:20:31.342 INFO:tasks.workunit.client.0.smithi136.stdout:2 2024-09-15T20:20:32.622 INFO:tasks.workunit.client.0.smithi136.stdout:3 2024-09-15T20:20:32.802 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:20:32.801+0000 7f77baa11640 -1 osd.1 0 log_to_monitors true 2024-09-15T20:20:33.991 INFO:tasks.workunit.client.0.smithi136.stdout:4 2024-09-15T20:20:35.275 INFO:tasks.workunit.client.0.smithi136.stdout:5 2024-09-15T20:20:36.563 INFO:tasks.workunit.client.0.smithi136.stdout:6 2024-09-15T20:20:37.902 INFO:tasks.workunit.client.0.smithi136.stdout:7 2024-09-15T20:20:39.187 INFO:tasks.workunit.client.0.smithi136.stdout:8 2024-09-15T20:20:40.563 INFO:tasks.workunit.client.0.smithi136.stdout:9 2024-09-15T20:20:41.953 INFO:tasks.workunit.client.0.smithi136.stdout:10 2024-09-15T20:20:43.242 INFO:tasks.workunit.client.0.smithi136.stdout:11 2024-09-15T20:20:44.633 INFO:tasks.workunit.client.0.smithi136.stdout:12 2024-09-15T20:20:46.471 INFO:tasks.workunit.client.0.smithi136.stdout:13 2024-09-15T20:20:48.130 INFO:tasks.workunit.client.0.smithi136.stdout:14 2024-09-15T20:20:49.832 INFO:tasks.workunit.client.0.smithi136.stdout:15 2024-09-15T20:20:51.287 INFO:tasks.workunit.client.0.smithi136.stdout:16 2024-09-15T20:20:52.686 INFO:tasks.workunit.client.0.smithi136.stdout:17 2024-09-15T20:20:54.105 INFO:tasks.workunit.client.0.smithi136.stdout:18 2024-09-15T20:20:55.609 INFO:tasks.workunit.client.0.smithi136.stdout:19 2024-09-15T20:20:55.925 INFO:tasks.workunit.client.0.smithi136.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/2572783983,v1:127.0.0.1:6811/2572783983] [v2:127.0.0.1:6812/2572783983,v1:127.0.0.1:6813/2572783983] exists,up aa3a94d1-c180-456f-8f8b-c75b0a4a1477 2024-09-15T20:20:55.929 INFO:tasks.workunit.client.0.smithi136.stdout:add osd2 dfb017d7-8565-48db-adc5-cdca8ed8863d 2024-09-15T20:20:56.310 INFO:tasks.workunit.client.0.smithi136.stdout:2 2024-09-15T20:20:56.358 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:20:56.354+0000 7eff4920f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T20:20:56.394 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:20:56.392+0000 7eff4920f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T20:20:56.395 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:20:56.393+0000 7eff4920f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T20:20:56.395 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:20:56.394+0000 7eff4920f640 -1 bdev(0x55717081fc00 td/osd-mapper/2/block) open stat got: (1) Operation not permitted 2024-09-15T20:20:56.395 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:20:56.394+0000 7eff4920f640 -1 bluestore(td/osd-mapper/2) _read_fsid unparsable uuid 2024-09-15T20:21:02.309 INFO:tasks.workunit.client.0.smithi136.stdout:adding osd2 key to auth repository 2024-09-15T20:21:02.667 INFO:tasks.workunit.client.0.smithi136.stdout:start osd.2 2024-09-15T20:21:02.698 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:21:02.696+0000 7f52ff9c5640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T20:21:02.700 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:21:02.698+0000 7f52ff9c5640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T20:21:02.701 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:21:02.699+0000 7f52ff9c5640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T20:21:02.988 INFO:tasks.workunit.client.0.smithi136.stdout:0 2024-09-15T20:21:04.264 INFO:tasks.workunit.client.0.smithi136.stdout:1 2024-09-15T20:21:04.541 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:21:04.540+0000 7f52ff9c5640 -1 Falling back to public interface 2024-09-15T20:21:05.548 INFO:tasks.workunit.client.0.smithi136.stdout:2 2024-09-15T20:21:06.830 INFO:tasks.workunit.client.0.smithi136.stdout:3 2024-09-15T20:21:07.302 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:21:07.300+0000 7f52ff9c5640 -1 osd.2 0 log_to_monitors true 2024-09-15T20:21:08.186 INFO:tasks.workunit.client.0.smithi136.stdout:4 2024-09-15T20:21:09.557 INFO:tasks.workunit.client.0.smithi136.stdout:5 2024-09-15T20:21:10.842 INFO:tasks.workunit.client.0.smithi136.stdout:6 2024-09-15T20:21:12.177 INFO:tasks.workunit.client.0.smithi136.stdout:7 2024-09-15T20:21:13.464 INFO:tasks.workunit.client.0.smithi136.stdout:8 2024-09-15T20:21:14.751 INFO:tasks.workunit.client.0.smithi136.stdout:9 2024-09-15T20:21:16.162 INFO:tasks.workunit.client.0.smithi136.stdout:10 2024-09-15T20:21:17.449 INFO:tasks.workunit.client.0.smithi136.stdout:11 2024-09-15T20:21:18.734 INFO:tasks.workunit.client.0.smithi136.stdout:12 2024-09-15T20:21:20.150 INFO:tasks.workunit.client.0.smithi136.stdout:13 2024-09-15T20:21:21.890 INFO:tasks.workunit.client.0.smithi136.stdout:14 2024-09-15T20:21:23.297 INFO:tasks.workunit.client.0.smithi136.stdout:15 2024-09-15T20:21:25.112 INFO:tasks.workunit.client.0.smithi136.stdout:16 2024-09-15T20:21:26.587 INFO:tasks.workunit.client.0.smithi136.stdout:17 2024-09-15T20:21:28.015 INFO:tasks.workunit.client.0.smithi136.stdout:18 2024-09-15T20:21:29.591 INFO:tasks.workunit.client.0.smithi136.stdout:19 2024-09-15T20:21:31.038 INFO:tasks.workunit.client.0.smithi136.stdout:20 2024-09-15T20:21:31.323 INFO:tasks.workunit.client.0.smithi136.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/1073590276,v1:127.0.0.1:6819/1073590276] [v2:127.0.0.1:6820/1073590276,v1:127.0.0.1:6821/1073590276] exists,up dfb017d7-8565-48db-adc5-cdca8ed8863d 2024-09-15T20:21:31.706 INFO:tasks.workunit.client.0.smithi136.stderr:pool 'test' created 2024-09-15T20:21:33.618 INFO:tasks.workunit.client.0.smithi136.stdout:waiting osd.0 seq 21474836497 2024-09-15T20:21:35.180 INFO:tasks.workunit.client.0.smithi136.stdout:waiting osd.1 seq 42949672975 2024-09-15T20:21:35.460 INFO:tasks.workunit.client.0.smithi136.stdout:waiting osd.2 seq 64424509446 2024-09-15T20:21:37.985 INFO:tasks.workunit.client.0.smithi136.stdout:standard_scrub_cluster: dbg: test pool is test 1 2024-09-15T20:21:37.985 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:55: TEST_truncated_sna_record: ceph tell 'osd.*' config set osd_stats_update_period_not_scrubbing 1 2024-09-15T20:21:38.074 INFO:tasks.workunit.client.0.smithi136.stdout:osd.0: { 2024-09-15T20:21:38.075 INFO:tasks.workunit.client.0.smithi136.stdout: "success": "osd_stats_update_period_not_scrubbing = '' (not observed, change may require restart) " 2024-09-15T20:21:38.075 INFO:tasks.workunit.client.0.smithi136.stdout:} 2024-09-15T20:21:38.088 INFO:tasks.workunit.client.0.smithi136.stdout:osd.1: { 2024-09-15T20:21:38.088 INFO:tasks.workunit.client.0.smithi136.stdout: "success": "osd_stats_update_period_not_scrubbing = '' (not observed, change may require restart) " 2024-09-15T20:21:38.088 INFO:tasks.workunit.client.0.smithi136.stdout:} 2024-09-15T20:21:38.101 INFO:tasks.workunit.client.0.smithi136.stdout:osd.2: { 2024-09-15T20:21:38.101 INFO:tasks.workunit.client.0.smithi136.stdout: "success": "osd_stats_update_period_not_scrubbing = '' (not observed, change may require restart) " 2024-09-15T20:21:38.101 INFO:tasks.workunit.client.0.smithi136.stdout:} 2024-09-15T20:21:38.110 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:56: TEST_truncated_sna_record: ceph tell 'osd.*' config set osd_stats_update_period_scrubbing 1 2024-09-15T20:21:38.210 INFO:tasks.workunit.client.0.smithi136.stdout:osd.0: { 2024-09-15T20:21:38.211 INFO:tasks.workunit.client.0.smithi136.stdout: "success": "osd_stats_update_period_scrubbing = '' (not observed, change may require restart) " 2024-09-15T20:21:38.211 INFO:tasks.workunit.client.0.smithi136.stdout:} 2024-09-15T20:21:38.224 INFO:tasks.workunit.client.0.smithi136.stdout:osd.1: { 2024-09-15T20:21:38.224 INFO:tasks.workunit.client.0.smithi136.stdout: "success": "osd_stats_update_period_scrubbing = '' (not observed, change may require restart) " 2024-09-15T20:21:38.225 INFO:tasks.workunit.client.0.smithi136.stdout:} 2024-09-15T20:21:38.239 INFO:tasks.workunit.client.0.smithi136.stdout:osd.2: { 2024-09-15T20:21:38.239 INFO:tasks.workunit.client.0.smithi136.stdout: "success": "osd_stats_update_period_scrubbing = '' (not observed, change may require restart) " 2024-09-15T20:21:38.239 INFO:tasks.workunit.client.0.smithi136.stdout:} 2024-09-15T20:21:38.246 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:58: TEST_truncated_sna_record: local osdn=3 2024-09-15T20:21:38.247 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:59: TEST_truncated_sna_record: local poolid=1 2024-09-15T20:21:38.247 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:60: TEST_truncated_sna_record: local poolname=test 2024-09-15T20:21:38.247 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:61: TEST_truncated_sna_record: local objname=objxxx 2024-09-15T20:21:38.247 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:64: TEST_truncated_sna_record: make_a_clone test objxxx snap01 snap02 2024-09-15T20:21:38.247 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:32: make_a_clone: local saved_echo_flag=x 2024-09-15T20:21:38.247 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:33: make_a_clone: set +x 2024-09-15T20:21:38.615 INFO:tasks.workunit.client.0.smithi136.stdout:created pool test snap snap01 2024-09-15T20:21:38.866 INFO:tasks.workunit.client.0.smithi136.stdout:created pool test snap snap02 2024-09-15T20:21:38.868 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:65: TEST_truncated_sna_record: make_a_clone test objxxx snap13 2024-09-15T20:21:38.869 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:32: make_a_clone: local saved_echo_flag=x 2024-09-15T20:21:38.869 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:33: make_a_clone: set +x 2024-09-15T20:21:39.335 INFO:tasks.workunit.client.0.smithi136.stdout:created pool test snap snap13 2024-09-15T20:21:39.337 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:66: TEST_truncated_sna_record: make_a_clone test objxxx snap24 snap25 2024-09-15T20:21:39.337 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:32: make_a_clone: local saved_echo_flag=x 2024-09-15T20:21:39.337 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:33: make_a_clone: set +x 2024-09-15T20:21:39.781 INFO:tasks.workunit.client.0.smithi136.stdout:created pool test snap snap24 2024-09-15T20:21:40.039 INFO:tasks.workunit.client.0.smithi136.stdout:created pool test snap snap25 2024-09-15T20:21:40.041 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:67: TEST_truncated_sna_record: echo 8556 2024-09-15T20:21:40.041 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:67: TEST_truncated_sna_record: rados -p test put objxxx - 2024-09-15T20:21:40.377 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:70: TEST_truncated_sna_record: ceph --format=json-pretty osd map test objxxx 2024-09-15T20:21:40.377 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:70: TEST_truncated_sna_record: jq -r .pgid 2024-09-15T20:21:40.668 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:70: TEST_truncated_sna_record: local pgid=1.3 2024-09-15T20:21:40.668 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:71: TEST_truncated_sna_record: ceph --format=json-pretty osd map test objxxx 2024-09-15T20:21:40.669 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:71: TEST_truncated_sna_record: jq -r '.up[0]' 2024-09-15T20:21:40.963 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:71: TEST_truncated_sna_record: local osd=1 2024-09-15T20:21:40.963 INFO:tasks.workunit.client.0.smithi136.stdout:pgid is 1.3 (primary: osd.1) 2024-09-15T20:21:40.964 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:72: TEST_truncated_sna_record: echo 'pgid is 1.3 (primary: osd.1)' 2024-09-15T20:21:40.964 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:74: TEST_truncated_sna_record: set_query_debug 1.3 2024-09-15T20:21:40.964 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:306: set_query_debug: local pgid=1.3 2024-09-15T20:21:40.964 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:308: set_query_debug: ceph pg dump pgs_brief 2024-09-15T20:21:40.964 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:308: set_query_debug: awk -v 'pg=^1.3' -n -e '$0 ~ pg { print(gensub(/[^0-9]*([0-9]+).*/,"\\1","g",$5)); }' 2024-09-15T20:21:41.222 INFO:tasks.workunit.client.0.smithi136.stderr:dumped pgs_brief 2024-09-15T20:21:41.234 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:308: set_query_debug: local prim_osd=1 2024-09-15T20:21:41.234 INFO:tasks.workunit.client.0.smithi136.stdout:Setting scrub debug data. Primary for 1.3 is 1 2024-09-15T20:21:41.235 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:310: set_query_debug: echo 'Setting scrub debug data. Primary for 1.3 is 1' 2024-09-15T20:21:41.236 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:311: set_query_debug: get_asok_path osd.1 2024-09-15T20:21:41.236 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.1 2024-09-15T20:21:41.236 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.1 ']' 2024-09-15T20:21:41.236 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-15T20:21:41.236 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T20:21:41.236 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.27314 2024-09-15T20:21:41.236 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.27314/ceph-osd.1.asok 2024-09-15T20:21:41.236 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:311: set_query_debug: CEPH_ARGS= 2024-09-15T20:21:41.236 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:311: set_query_debug: ceph --format=json daemon /tmp/ceph-asok.27314/ceph-osd.1.asok scrubdebug 1.3 set sessions 2024-09-15T20:21:41.323 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:77: TEST_truncated_sna_record: (( extr_dbg >= 1 )) 2024-09-15T20:21:41.323 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:77: TEST_truncated_sna_record: rados --format json-pretty -p test listsnaps objxxx 2024-09-15T20:21:41.353 INFO:tasks.workunit.client.0.smithi136.stdout:{"success":true}{ 2024-09-15T20:21:41.353 INFO:tasks.workunit.client.0.smithi136.stdout: "name": "objxxx", 2024-09-15T20:21:41.353 INFO:tasks.workunit.client.0.smithi136.stdout: "clones": [ 2024-09-15T20:21:41.353 INFO:tasks.workunit.client.0.smithi136.stdout: { 2024-09-15T20:21:41.353 INFO:tasks.workunit.client.0.smithi136.stdout: "id": 2, 2024-09-15T20:21:41.353 INFO:tasks.workunit.client.0.smithi136.stdout: "snapshots": [ 2024-09-15T20:21:41.353 INFO:tasks.workunit.client.0.smithi136.stdout: { 2024-09-15T20:21:41.353 INFO:tasks.workunit.client.0.smithi136.stdout: "id": 1, 2024-09-15T20:21:41.354 INFO:tasks.workunit.client.0.smithi136.stdout: "name": "snap01" 2024-09-15T20:21:41.354 INFO:tasks.workunit.client.0.smithi136.stdout: }, 2024-09-15T20:21:41.354 INFO:tasks.workunit.client.0.smithi136.stdout: { 2024-09-15T20:21:41.354 INFO:tasks.workunit.client.0.smithi136.stdout: "id": 2, 2024-09-15T20:21:41.354 INFO:tasks.workunit.client.0.smithi136.stdout: "name": "snap02" 2024-09-15T20:21:41.354 INFO:tasks.workunit.client.0.smithi136.stdout: } 2024-09-15T20:21:41.354 INFO:tasks.workunit.client.0.smithi136.stdout: ], 2024-09-15T20:21:41.354 INFO:tasks.workunit.client.0.smithi136.stdout: "size": 6, 2024-09-15T20:21:41.354 INFO:tasks.workunit.client.0.smithi136.stdout: "overlaps": [] 2024-09-15T20:21:41.354 INFO:tasks.workunit.client.0.smithi136.stdout: }, 2024-09-15T20:21:41.354 INFO:tasks.workunit.client.0.smithi136.stdout: { 2024-09-15T20:21:41.354 INFO:tasks.workunit.client.0.smithi136.stdout: "id": 3, 2024-09-15T20:21:41.354 INFO:tasks.workunit.client.0.smithi136.stdout: "snapshots": [ 2024-09-15T20:21:41.354 INFO:tasks.workunit.client.0.smithi136.stdout: { 2024-09-15T20:21:41.354 INFO:tasks.workunit.client.0.smithi136.stdout: "id": 3, 2024-09-15T20:21:41.355 INFO:tasks.workunit.client.0.smithi136.stdout: "name": "snap13" 2024-09-15T20:21:41.355 INFO:tasks.workunit.client.0.smithi136.stdout: } 2024-09-15T20:21:41.355 INFO:tasks.workunit.client.0.smithi136.stdout: ], 2024-09-15T20:21:41.355 INFO:tasks.workunit.client.0.smithi136.stdout: "size": 5, 2024-09-15T20:21:41.355 INFO:tasks.workunit.client.0.smithi136.stdout: "overlaps": [] 2024-09-15T20:21:41.355 INFO:tasks.workunit.client.0.smithi136.stdout: }, 2024-09-15T20:21:41.355 INFO:tasks.workunit.client.0.smithi136.stdout: { 2024-09-15T20:21:41.355 INFO:tasks.workunit.client.0.smithi136.stdout: "id": 5, 2024-09-15T20:21:41.355 INFO:tasks.workunit.client.0.smithi136.stdout: "snapshots": [ 2024-09-15T20:21:41.355 INFO:tasks.workunit.client.0.smithi136.stdout: { 2024-09-15T20:21:41.355 INFO:tasks.workunit.client.0.smithi136.stdout: "id": 4, 2024-09-15T20:21:41.355 INFO:tasks.workunit.client.0.smithi136.stdout: "name": "snap24" 2024-09-15T20:21:41.355 INFO:tasks.workunit.client.0.smithi136.stdout: }, 2024-09-15T20:21:41.356 INFO:tasks.workunit.client.0.smithi136.stdout: { 2024-09-15T20:21:41.356 INFO:tasks.workunit.client.0.smithi136.stdout: "id": 5, 2024-09-15T20:21:41.357 INFO:tasks.workunit.client.0.smithi136.stdout: "name": "snap25" 2024-09-15T20:21:41.357 INFO:tasks.workunit.client.0.smithi136.stdout: } 2024-09-15T20:21:41.357 INFO:tasks.workunit.client.0.smithi136.stdout: ], 2024-09-15T20:21:41.357 INFO:tasks.workunit.client.0.smithi136.stdout: "size": 5, 2024-09-15T20:21:41.357 INFO:tasks.workunit.client.0.smithi136.stdout: "overlaps": [] 2024-09-15T20:21:41.357 INFO:tasks.workunit.client.0.smithi136.stdout: }, 2024-09-15T20:21:41.357 INFO:tasks.workunit.client.0.smithi136.stdout: { 2024-09-15T20:21:41.357 INFO:tasks.workunit.client.0.smithi136.stdout: "id": "head", 2024-09-15T20:21:41.357 INFO:tasks.workunit.client.0.smithi136.stdout: "snapshots": [], 2024-09-15T20:21:41.357 INFO:tasks.workunit.client.0.smithi136.stdout: "size": 5 2024-09-15T20:21:41.357 INFO:tasks.workunit.client.0.smithi136.stdout: } 2024-09-15T20:21:41.357 INFO:tasks.workunit.client.0.smithi136.stdout: ] 2024-09-15T20:21:41.357 INFO:tasks.workunit.client.0.smithi136.stdout:} 2024-09-15T20:21:41.358 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:80: TEST_truncated_sna_record: ceph pg 1.3 deep-scrub 2024-09-15T20:21:41.447 INFO:tasks.workunit.client.0.smithi136.stdout:{ 2024-09-15T20:21:41.447 INFO:tasks.workunit.client.0.smithi136.stdout: "deep": true, 2024-09-15T20:21:41.447 INFO:tasks.workunit.client.0.smithi136.stdout: "must": true, 2024-09-15T20:21:41.447 INFO:tasks.workunit.client.0.smithi136.stdout: "stamp": "0.000000" 2024-09-15T20:21:41.447 INFO:tasks.workunit.client.0.smithi136.stdout:} 2024-09-15T20:21:41.455 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:84: TEST_truncated_sna_record: sleep 3 2024-09-15T20:21:44.458 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:85: TEST_truncated_sna_record: ceph pg dump pgs 2024-09-15T20:21:44.724 INFO:tasks.workunit.client.0.smithi136.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-15T20:21:44.724 INFO:tasks.workunit.client.0.smithi136.stdout:1.3 4 0 0 0 0 21 0 0 7 0 7 active+clean 2024-09-15T20:21:33.156815+0000 28'7 28:38 [1,2,0] 1 [1,2,0] 1 0'0 2024-09-15T20:21:31.637829+0000 0'0 2024-09-15T20:21:31.637829+0000 0 0 periodic scrub scheduled @ 2024-09-16T20:21:31.637829+0000 0 0 2024-09-15T20:21:44.725 INFO:tasks.workunit.client.0.smithi136.stdout:1.2 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-09-15T20:21:33.157577+0000 0'0 28:34 [0,1,2] 0 [0,1,2] 0 0'0 2024-09-15T20:21:31.637829+0000 0'0 2024-09-15T20:21:31.637829+0000 0 0 periodic scrub scheduled @ 2024-09-16T20:21:31.637829+0000 0 0 2024-09-15T20:21:44.725 INFO:tasks.workunit.client.0.smithi136.stdout:1.1 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-09-15T20:21:32.561441+0000 0'0 28:34 [2,0,1] 2 [2,0,1] 2 0'0 2024-09-15T20:21:31.637829+0000 0'0 2024-09-15T20:21:31.637829+0000 0 0 periodic scrub scheduled @ 2024-09-16T20:21:31.637829+0000 0 0 2024-09-15T20:21:44.725 INFO:tasks.workunit.client.0.smithi136.stdout:1.0 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-09-15T20:21:33.157547+0000 0'0 28:34 [1,0,2] 1 [1,0,2] 1 0'0 2024-09-15T20:21:31.637829+0000 0'0 2024-09-15T20:21:31.637829+0000 0 0 periodic scrub scheduled @ 2024-09-16T20:21:31.637829+0000 0 0 2024-09-15T20:21:44.725 INFO:tasks.workunit.client.0.smithi136.stdout: 2024-09-15T20:21:44.725 INFO:tasks.workunit.client.0.smithi136.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-15T20:21:44.725 INFO:tasks.workunit.client.0.smithi136.stderr:dumped pgs 2024-09-15T20:21:44.735 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:86: TEST_truncated_sna_record: grep -a -q -- 'event: --^^^^---- ScrubFinished' td/osd-mapper/osd.1.log 2024-09-15T20:21:44.742 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:90: TEST_truncated_sna_record: ceph pg dump pgs 2024-09-15T20:21:44.994 INFO:tasks.workunit.client.0.smithi136.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-15T20:21:44.995 INFO:tasks.workunit.client.0.smithi136.stdout:1.3 4 0 0 0 0 21 0 0 7 0 7 active+clean 2024-09-15T20:21:33.156815+0000 28'7 28:38 [1,2,0] 1 [1,2,0] 1 0'0 2024-09-15T20:21:31.637829+0000 0'0 2024-09-15T20:21:31.637829+0000 0 0 periodic scrub scheduled @ 2024-09-16T20:21:31.637829+0000 0 0 2024-09-15T20:21:44.995 INFO:tasks.workunit.client.0.smithi136.stdout:1.2 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-09-15T20:21:33.157577+0000 0'0 28:34 [0,1,2] 0 [0,1,2] 0 0'0 2024-09-15T20:21:31.637829+0000 0'0 2024-09-15T20:21:31.637829+0000 0 0 periodic scrub scheduled @ 2024-09-16T20:21:31.637829+0000 0 0 2024-09-15T20:21:44.995 INFO:tasks.workunit.client.0.smithi136.stdout:1.1 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-09-15T20:21:32.561441+0000 0'0 28:34 [2,0,1] 2 [2,0,1] 2 0'0 2024-09-15T20:21:31.637829+0000 0'0 2024-09-15T20:21:31.637829+0000 0 0 periodic scrub scheduled @ 2024-09-16T20:21:31.637829+0000 0 0 2024-09-15T20:21:44.995 INFO:tasks.workunit.client.0.smithi136.stdout:1.0 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-09-15T20:21:33.157547+0000 0'0 28:34 [1,0,2] 1 [1,0,2] 1 0'0 2024-09-15T20:21:31.637829+0000 0'0 2024-09-15T20:21:31.637829+0000 0 0 periodic scrub scheduled @ 2024-09-16T20:21:31.637829+0000 0 0 2024-09-15T20:21:44.995 INFO:tasks.workunit.client.0.smithi136.stdout: 2024-09-15T20:21:44.995 INFO:tasks.workunit.client.0.smithi136.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-15T20:21:44.995 INFO:tasks.workunit.client.0.smithi136.stderr:dumped pgs 2024-09-15T20:21:45.005 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:91: TEST_truncated_sna_record: ceph osd set noscrub 2024-09-15T20:21:45.369 INFO:tasks.workunit.client.0.smithi136.stderr:noscrub is set 2024-09-15T20:21:45.381 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:92: TEST_truncated_sna_record: ceph osd set nodeep-scrub 2024-09-15T20:21:45.905 INFO:tasks.workunit.client.0.smithi136.stderr:nodeep-scrub is set 2024-09-15T20:21:45.917 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:93: TEST_truncated_sna_record: sleep 5 2024-09-15T20:21:50.925 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:94: TEST_truncated_sna_record: grep -a -q -v ERR td/osd-mapper/osd.1.log 2024-09-15T20:21:50.925 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:97: TEST_truncated_sna_record: kill_daemons td/osd-mapper TERM osd 2024-09-15T20:21:50.925 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-15T20:21:50.926 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-15T20:21:50.926 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-15T20:21:50.926 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-15T20:21:50.926 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-15T20:21:51.230 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-15T20:21:51.230 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:99: TEST_truncated_sna_record: (( extr_dbg >= 2 )) 2024-09-15T20:21:51.230 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:99: TEST_truncated_sna_record: ceph-kvstore-tool bluestore-kv td/osd-mapper/0 dump p 2024-09-15T20:21:51.259 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:21:51.253+0000 7f67adc2f980 1 bdev(0x56061eb17c00 td/osd-mapper/0/block) open path td/osd-mapper/0/block 2024-09-15T20:21:51.259 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:21:51.253+0000 7f67adc2f980 0 bdev(0x56061eb17c00 td/osd-mapper/0/block) ioctl(F_SET_FILE_RW_HINT) on td/osd-mapper/0/block failed: (22) Invalid argument 2024-09-15T20:21:51.259 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:21:51.253+0000 7f67adc2f980 1 bdev(0x56061eb17c00 td/osd-mapper/0/block) open size 107374182400 (0x1900000000, 100 GiB) block_size 4096 (4 KiB) rotational device, discard supported 2024-09-15T20:21:51.301 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:21:51.299+0000 7f67adc2f980 1 bdev(0x56061eb17c00 td/osd-mapper/0/block) close 2024-09-15T20:21:51.517 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:21:51.515+0000 7f67adc2f980 1 bdev(0x56061eb17c00 td/osd-mapper/0/block) open path td/osd-mapper/0/block 2024-09-15T20:21:51.517 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:21:51.515+0000 7f67adc2f980 0 bdev(0x56061eb17c00 td/osd-mapper/0/block) ioctl(F_SET_FILE_RW_HINT) on td/osd-mapper/0/block failed: (22) Invalid argument 2024-09-15T20:21:51.517 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:21:51.515+0000 7f67adc2f980 1 bdev(0x56061eb17c00 td/osd-mapper/0/block) open size 107374182400 (0x1900000000, 100 GiB) block_size 4096 (4 KiB) rotational device, discard supported 2024-09-15T20:21:51.518 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:21:51.516+0000 7f67adc2f980 1 bluestore(td/osd-mapper/0) _set_cache_sizes cache_size 1073741824 meta 0.45 kv 0.45 kv_onode 0.04 data 0.06 2024-09-15T20:21:51.518 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:21:51.516+0000 7f67adc2f980 1 bdev(0x56061ec92400 td/osd-mapper/0/block) open path td/osd-mapper/0/block 2024-09-15T20:21:51.519 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:21:51.516+0000 7f67adc2f980 0 bdev(0x56061ec92400 td/osd-mapper/0/block) ioctl(F_SET_FILE_RW_HINT) on td/osd-mapper/0/block failed: (22) Invalid argument 2024-09-15T20:21:51.519 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:21:51.517+0000 7f67adc2f980 1 bdev(0x56061ec92400 td/osd-mapper/0/block) open size 107374182400 (0x1900000000, 100 GiB) block_size 4096 (4 KiB) rotational device, discard supported 2024-09-15T20:21:51.519 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:21:51.517+0000 7f67adc2f980 1 bluefs add_block_device bdev 1 path td/osd-mapper/0/block size 100 GiB 2024-09-15T20:21:51.519 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:21:51.517+0000 7f67adc2f980 1 bluefs mount 2024-09-15T20:21:51.519 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:21:51.517+0000 7f67adc2f980 1 bluefs _init_alloc shared, id 1, capacity 0x1900000000, block size 0x10000 2024-09-15T20:21:51.541 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:21:51.539+0000 7f67adc2f980 1 bluefs mount shared_bdev_used = 0 2024-09-15T20:21:51.542 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:21:51.539+0000 7f67adc2f980 1 bluestore(td/osd-mapper/0) _prepare_db_environment set db_paths to db,102005473280 db.slow,102005473280 2024-09-15T20:21:51.578 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:21:51.576+0000 7f67adc2f980 1 bluestore(td/osd-mapper/0) _open_db opened rocksdb path db options compression=kLZ4Compression,max_write_buffer_number=64,min_write_buffer_number_to_merge=6,compaction_style=kCompactionStyleLevel,write_buffer_size=16777216,max_background_jobs=4,level0_file_num_compaction_trigger=8,max_bytes_for_level_base=1073741824,max_bytes_for_level_multiplier=8,compaction_readahead_size=2MB,max_total_wal_size=1073741824,writable_file_max_buffer_size=0 2024-09-15T20:21:51.579 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:21:51.576+0000 7f67adc2f980 1 bluestore(td/osd-mapper/0) _open_super_meta old nid_max 2050 2024-09-15T20:21:51.579 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:21:51.576+0000 7f67adc2f980 1 bluestore(td/osd-mapper/0) _open_super_meta old blobid_max 20480 2024-09-15T20:21:51.579 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:21:51.576+0000 7f67adc2f980 1 bluestore(td/osd-mapper/0) _open_super_meta ondisk_format 4 compat_ondisk_format 3 2024-09-15T20:21:51.579 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:21:51.576+0000 7f67adc2f980 1 bluestore(td/osd-mapper/0) _open_super_meta min_alloc_size 0x1000 2024-09-15T20:21:51.579 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:21:51.577+0000 7f67adc2f980 1 freelist init 2024-09-15T20:21:51.579 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:21:51.577+0000 7f67adc2f980 1 freelist _read_cfg 2024-09-15T20:21:51.579 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:21:51.577+0000 7f67adc2f980 1 bluestore(td/osd-mapper/0) _open_fm effective freelist_type = bitmap, freelist_alloc_size = 0x1000, min_alloc_size = 0x1000 2024-09-15T20:21:51.580 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:21:51.577+0000 7f67adc2f980 1 bluestore(td/osd-mapper/0) _init_alloc loaded 100 GiB in 26 extents, allocator type hybrid, capacity 0x1900000000, block size 0x1000, free 0x18fffbc000, fragmentation 9.53677e-07 2024-09-15T20:21:51.580 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:21:51.577+0000 7f67adc2f980 1 bluefs umount 2024-09-15T20:21:51.580 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:21:51.577+0000 7f67adc2f980 1 bdev(0x56061ec92400 td/osd-mapper/0/block) close 2024-09-15T20:21:51.781 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:21:51.779+0000 7f67adc2f980 1 bdev(0x56061ec92400 td/osd-mapper/0/block) open path td/osd-mapper/0/block 2024-09-15T20:21:51.781 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:21:51.779+0000 7f67adc2f980 0 bdev(0x56061ec92400 td/osd-mapper/0/block) ioctl(F_SET_FILE_RW_HINT) on td/osd-mapper/0/block failed: (22) Invalid argument 2024-09-15T20:21:51.781 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:21:51.779+0000 7f67adc2f980 1 bdev(0x56061ec92400 td/osd-mapper/0/block) open size 107374182400 (0x1900000000, 100 GiB) block_size 4096 (4 KiB) rotational device, discard supported 2024-09-15T20:21:51.781 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:21:51.779+0000 7f67adc2f980 1 bluefs add_block_device bdev 1 path td/osd-mapper/0/block size 100 GiB 2024-09-15T20:21:51.781 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:21:51.779+0000 7f67adc2f980 1 bluefs mount 2024-09-15T20:21:51.790 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:21:51.788+0000 7f67adc2f980 1 bluefs _init_alloc shared, id 1, capacity 0x1900000000, block size 0x10000 2024-09-15T20:21:51.799 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:21:51.797+0000 7f67adc2f980 1 bluefs mount shared_bdev_used = 27459584 2024-09-15T20:21:51.799 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:21:51.797+0000 7f67adc2f980 1 bluestore(td/osd-mapper/0) _prepare_db_environment set db_paths to db,102005473280 db.slow,102005473280 2024-09-15T20:21:51.832 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:21:51.829+0000 7f67adc2f980 1 bluestore(td/osd-mapper/0) _open_db opened rocksdb path db options compression=kLZ4Compression,max_write_buffer_number=64,min_write_buffer_number_to_merge=6,compaction_style=kCompactionStyleLevel,write_buffer_size=16777216,max_background_jobs=4,level0_file_num_compaction_trigger=8,max_bytes_for_level_base=1073741824,max_bytes_for_level_multiplier=8,compaction_readahead_size=2MB,max_total_wal_size=1073741824,writable_file_max_buffer_size=0 2024-09-15T20:21:51.834 INFO:tasks.workunit.client.0.smithi136.stdout:p %00%00%00%00%00%00%00%00%7b%3fC%c4%00%00%00%00%00%00%00%01.osd_superblock 2024-09-15T20:21:51.834 INFO:tasks.workunit.client.0.smithi136.stdout:00000000 0b 05 58 02 00 00 e9 f4 cc 96 8f 95 49 71 84 e5 |..X.........Iq..| 2024-09-15T20:21:51.834 INFO:tasks.workunit.client.0.smithi136.stdout:00000010 d6 4a cd 70 df 0f 00 00 00 00 1e 00 00 00 00 00 |.J.p............| 2024-09-15T20:21:51.834 INFO:tasks.workunit.client.0.smithi136.stdout:00000020 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................| 2024-09-15T20:21:51.834 INFO:tasks.workunit.client.0.smithi136.stdout:* 2024-09-15T20:21:51.834 INFO:tasks.workunit.client.0.smithi136.stdout:00000040 00 00 00 00 00 00 fe ff 03 00 00 00 00 00 11 00 |................| 2024-09-15T20:21:51.834 INFO:tasks.workunit.client.0.smithi136.stdout:00000050 00 00 01 00 00 00 00 00 00 00 1a 00 00 00 69 6e |..............in| 2024-09-15T20:21:51.834 INFO:tasks.workunit.client.0.smithi136.stdout:00000060 69 74 69 61 6c 20 66 65 61 74 75 72 65 20 73 65 |itial feature se| 2024-09-15T20:21:51.835 INFO:tasks.workunit.client.0.smithi136.stdout:00000070 74 28 7e 76 2e 31 38 29 02 00 00 00 00 00 00 00 |t(~v.18)........| 2024-09-15T20:21:51.835 INFO:tasks.workunit.client.0.smithi136.stdout:00000080 0d 00 00 00 70 67 69 6e 66 6f 20 6f 62 6a 65 63 |....pginfo objec| 2024-09-15T20:21:51.835 INFO:tasks.workunit.client.0.smithi136.stdout:00000090 74 03 00 00 00 00 00 00 00 0e 00 00 00 6f 62 6a |t............obj| 2024-09-15T20:21:51.835 INFO:tasks.workunit.client.0.smithi136.stdout:000000a0 65 63 74 20 6c 6f 63 61 74 6f 72 04 00 00 00 00 |ect locator.....| 2024-09-15T20:21:51.835 INFO:tasks.workunit.client.0.smithi136.stdout:000000b0 00 00 00 10 00 00 00 6c 61 73 74 5f 65 70 6f 63 |.......last_epoc| 2024-09-15T20:21:51.835 INFO:tasks.workunit.client.0.smithi136.stdout:000000c0 68 5f 63 6c 65 61 6e 05 00 00 00 00 00 00 00 0a |h_clean.........| 2024-09-15T20:21:51.835 INFO:tasks.workunit.client.0.smithi136.stdout:000000d0 00 00 00 63 61 74 65 67 6f 72 69 65 73 06 00 00 |...categories...| 2024-09-15T20:21:51.835 INFO:tasks.workunit.client.0.smithi136.stdout:000000e0 00 00 00 00 00 0b 00 00 00 68 6f 62 6a 65 63 74 |.........hobject| 2024-09-15T20:21:51.835 INFO:tasks.workunit.client.0.smithi136.stdout:000000f0 70 6f 6f 6c 07 00 00 00 00 00 00 00 07 00 00 00 |pool............| 2024-09-15T20:21:51.835 INFO:tasks.workunit.client.0.smithi136.stdout:00000100 62 69 67 69 6e 66 6f 08 00 00 00 00 00 00 00 0b |biginfo.........| 2024-09-15T20:21:51.835 INFO:tasks.workunit.client.0.smithi136.stdout:00000110 00 00 00 6c 65 76 65 6c 64 62 69 6e 66 6f 09 00 |...leveldbinfo..| 2024-09-15T20:21:51.836 INFO:tasks.workunit.client.0.smithi136.stdout:00000120 00 00 00 00 00 00 0a 00 00 00 6c 65 76 65 6c 64 |..........leveld| 2024-09-15T20:21:51.836 INFO:tasks.workunit.client.0.smithi136.stdout:00000130 62 6c 6f 67 0a 00 00 00 00 00 00 00 0a 00 00 00 |blog............| 2024-09-15T20:21:51.836 INFO:tasks.workunit.client.0.smithi136.stdout:00000140 73 6e 61 70 6d 61 70 70 65 72 0b 00 00 00 00 00 |snapmapper......| 2024-09-15T20:21:51.836 INFO:tasks.workunit.client.0.smithi136.stdout:00000150 00 00 0f 00 00 00 73 68 61 72 64 65 64 20 6f 62 |......sharded ob| 2024-09-15T20:21:51.836 INFO:tasks.workunit.client.0.smithi136.stdout:00000160 6a 65 63 74 73 0c 00 00 00 00 00 00 00 11 00 00 |jects...........| 2024-09-15T20:21:51.836 INFO:tasks.workunit.client.0.smithi136.stdout:00000170 00 74 72 61 6e 73 61 63 74 69 6f 6e 20 68 69 6e |.transaction hin| 2024-09-15T20:21:51.836 INFO:tasks.workunit.client.0.smithi136.stdout:00000180 74 73 0d 00 00 00 00 00 00 00 0e 00 00 00 70 67 |ts............pg| 2024-09-15T20:21:51.836 INFO:tasks.workunit.client.0.smithi136.stdout:00000190 20 6d 65 74 61 20 6f 62 6a 65 63 74 0e 00 00 00 | meta object....| 2024-09-15T20:21:51.836 INFO:tasks.workunit.client.0.smithi136.stdout:000001a0 00 00 00 00 14 00 00 00 65 78 70 6c 69 63 69 74 |........explicit| 2024-09-15T20:21:51.836 INFO:tasks.workunit.client.0.smithi136.stdout:000001b0 20 6d 69 73 73 69 6e 67 20 73 65 74 0f 00 00 00 | missing set....| 2024-09-15T20:21:51.857 INFO:tasks.workunit.client.0.smithi136.stdout:000001c0 00 00 00 00 10 00 00 00 66 61 73 74 69 6e 66 6f |........fastinfo| 2024-09-15T20:21:51.857 INFO:tasks.workunit.client.0.smithi136.stdout:000001d0 20 70 67 20 61 74 74 72 10 00 00 00 00 00 00 00 | pg attr........| 2024-09-15T20:21:51.857 INFO:tasks.workunit.client.0.smithi136.stdout:000001e0 16 00 00 00 64 65 6c 65 74 65 73 20 69 6e 20 6d |....deletes in m| 2024-09-15T20:21:51.857 INFO:tasks.workunit.client.0.smithi136.stdout:000001f0 69 73 73 69 6e 67 20 73 65 74 11 00 00 00 00 00 |issing set......| 2024-09-15T20:21:51.857 INFO:tasks.workunit.client.0.smithi136.stdout:00000200 00 00 1c 00 00 00 6e 65 77 20 73 6e 61 70 6d 61 |......new snapma| 2024-09-15T20:21:51.857 INFO:tasks.workunit.client.0.smithi136.stdout:00000210 70 70 65 72 20 6b 65 79 20 73 74 72 75 63 74 75 |pper key structu| 2024-09-15T20:21:51.857 INFO:tasks.workunit.client.0.smithi136.stdout:00000220 72 65 1e 00 00 00 05 00 00 00 13 0d be e2 9d fc |re..............| 2024-09-15T20:21:51.857 INFO:tasks.workunit.client.0.smithi136.stdout:00000230 4e de 9d 5f ae 08 c2 87 24 27 00 00 00 00 00 00 |N.._....$'......| 2024-09-15T20:21:51.858 INFO:tasks.workunit.client.0.smithi136.stdout:00000240 00 00 1e 00 00 00 84 41 e7 66 57 b0 a3 13 01 00 |.......A.fW.....| 2024-09-15T20:21:51.858 INFO:tasks.workunit.client.0.smithi136.stdout:00000250 00 00 01 00 00 00 01 00 00 00 1e 00 00 00 |..............| 2024-09-15T20:21:51.858 INFO:tasks.workunit.client.0.smithi136.stdout:0000025e 2024-09-15T20:21:51.858 INFO:tasks.workunit.client.0.smithi136.stdout: 2024-09-15T20:21:51.858 INFO:tasks.workunit.client.0.smithi136.stdout:p %00%00%00%00%00%00%00%00%7b%3fC%c4%00%00%00%00%00%00%00%01~ 2024-09-15T20:21:51.858 INFO:tasks.workunit.client.0.smithi136.stdout: 2024-09-15T20:21:51.858 INFO:tasks.workunit.client.0.smithi136.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.OBJ_0000000000000001.FA570D58.2.objxxx.. 2024-09-15T20:21:51.858 INFO:tasks.workunit.client.0.smithi136.stdout:00000000 01 01 41 00 00 00 04 03 27 00 00 00 00 00 00 00 |..A.....'.......| 2024-09-15T20:21:51.858 INFO:tasks.workunit.client.0.smithi136.stdout:00000010 06 00 00 00 6f 62 6a 78 78 78 02 00 00 00 00 00 |....objxxx......| 2024-09-15T20:21:51.859 INFO:tasks.workunit.client.0.smithi136.stdout:00000020 00 00 af 75 d0 85 00 00 00 00 00 01 00 00 00 00 |...u............| 2024-09-15T20:21:51.859 INFO:tasks.workunit.client.0.smithi136.stdout:00000030 00 00 00 02 00 00 00 01 00 00 00 00 00 00 00 02 |................| 2024-09-15T20:21:51.859 INFO:tasks.workunit.client.0.smithi136.stdout:00000040 00 00 00 00 00 00 00 |.......| 2024-09-15T20:21:51.859 INFO:tasks.workunit.client.0.smithi136.stdout:00000047 2024-09-15T20:21:51.859 INFO:tasks.workunit.client.0.smithi136.stdout: 2024-09-15T20:21:51.859 INFO:tasks.workunit.client.0.smithi136.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.OBJ_0000000000000001.FA570D58.3.objxxx.. 2024-09-15T20:21:51.859 INFO:tasks.workunit.client.0.smithi136.stdout:00000000 01 01 39 00 00 00 04 03 27 00 00 00 00 00 00 00 |..9.....'.......| 2024-09-15T20:21:51.859 INFO:tasks.workunit.client.0.smithi136.stdout:00000010 06 00 00 00 6f 62 6a 78 78 78 03 00 00 00 00 00 |....objxxx......| 2024-09-15T20:21:51.859 INFO:tasks.workunit.client.0.smithi136.stdout:00000020 00 00 af 75 d0 85 00 00 00 00 00 01 00 00 00 00 |...u............| 2024-09-15T20:21:51.860 INFO:tasks.workunit.client.0.smithi136.stdout:00000030 00 00 00 01 00 00 00 03 00 00 00 00 00 00 00 |...............| 2024-09-15T20:21:51.860 INFO:tasks.workunit.client.0.smithi136.stdout:0000003f 2024-09-15T20:21:51.866 INFO:tasks.workunit.client.0.smithi136.stdout: 2024-09-15T20:21:51.866 INFO:tasks.workunit.client.0.smithi136.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.OBJ_0000000000000001.FA570D58.5.objxxx.. 2024-09-15T20:21:51.866 INFO:tasks.workunit.client.0.smithi136.stdout:00000000 01 01 41 00 00 00 04 03 27 00 00 00 00 00 00 00 |..A.....'.......| 2024-09-15T20:21:51.866 INFO:tasks.workunit.client.0.smithi136.stdout:00000010 06 00 00 00 6f 62 6a 78 78 78 05 00 00 00 00 00 |....objxxx......| 2024-09-15T20:21:51.866 INFO:tasks.workunit.client.0.smithi136.stdout:00000020 00 00 af 75 d0 85 00 00 00 00 00 01 00 00 00 00 |...u............| 2024-09-15T20:21:51.866 INFO:tasks.workunit.client.0.smithi136.stdout:00000030 00 00 00 02 00 00 00 04 00 00 00 00 00 00 00 05 |................| 2024-09-15T20:21:51.866 INFO:tasks.workunit.client.0.smithi136.stdout:00000040 00 00 00 00 00 00 00 |.......| 2024-09-15T20:21:51.866 INFO:tasks.workunit.client.0.smithi136.stdout:00000047 2024-09-15T20:21:51.867 INFO:tasks.workunit.client.0.smithi136.stdout: 2024-09-15T20:21:51.867 INFO:tasks.workunit.client.0.smithi136.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.FA570D58.2.objxxx.. 2024-09-15T20:21:51.867 INFO:tasks.workunit.client.0.smithi136.stdout:00000000 01 01 35 00 00 00 01 00 00 00 00 00 00 00 04 03 |..5.............| 2024-09-15T20:21:51.867 INFO:tasks.workunit.client.0.smithi136.stdout:00000010 27 00 00 00 00 00 00 00 06 00 00 00 6f 62 6a 78 |'...........objx| 2024-09-15T20:21:51.867 INFO:tasks.workunit.client.0.smithi136.stdout:00000020 78 78 02 00 00 00 00 00 00 00 af 75 d0 85 00 00 |xx.........u....| 2024-09-15T20:21:51.867 INFO:tasks.workunit.client.0.smithi136.stdout:00000030 00 00 00 01 00 00 00 00 00 00 00 |...........| 2024-09-15T20:21:51.867 INFO:tasks.workunit.client.0.smithi136.stdout:0000003b 2024-09-15T20:21:51.867 INFO:tasks.workunit.client.0.smithi136.stdout: 2024-09-15T20:21:51.868 INFO:tasks.workunit.client.0.smithi136.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000002_0000000000000001.FA570D58.2.objxxx.. 2024-09-15T20:21:51.868 INFO:tasks.workunit.client.0.smithi136.stdout:00000000 01 01 35 00 00 00 02 00 00 00 00 00 00 00 04 03 |..5.............| 2024-09-15T20:21:51.868 INFO:tasks.workunit.client.0.smithi136.stdout:00000010 27 00 00 00 00 00 00 00 06 00 00 00 6f 62 6a 78 |'...........objx| 2024-09-15T20:21:51.868 INFO:tasks.workunit.client.0.smithi136.stdout:00000020 78 78 02 00 00 00 00 00 00 00 af 75 d0 85 00 00 |xx.........u....| 2024-09-15T20:21:51.868 INFO:tasks.workunit.client.0.smithi136.stdout:00000030 00 00 00 01 00 00 00 00 00 00 00 |...........| 2024-09-15T20:21:51.868 INFO:tasks.workunit.client.0.smithi136.stdout:0000003b 2024-09-15T20:21:51.869 INFO:tasks.workunit.client.0.smithi136.stdout: 2024-09-15T20:21:51.869 INFO:tasks.workunit.client.0.smithi136.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.FA570D58.3.objxxx.. 2024-09-15T20:21:51.869 INFO:tasks.workunit.client.0.smithi136.stdout:00000000 01 01 35 00 00 00 03 00 00 00 00 00 00 00 04 03 |..5.............| 2024-09-15T20:21:51.869 INFO:tasks.workunit.client.0.smithi136.stdout:00000010 27 00 00 00 00 00 00 00 06 00 00 00 6f 62 6a 78 |'...........objx| 2024-09-15T20:21:51.869 INFO:tasks.workunit.client.0.smithi136.stdout:00000020 78 78 03 00 00 00 00 00 00 00 af 75 d0 85 00 00 |xx.........u....| 2024-09-15T20:21:51.869 INFO:tasks.workunit.client.0.smithi136.stdout:00000030 00 00 00 01 00 00 00 00 00 00 00 |...........| 2024-09-15T20:21:51.873 INFO:tasks.workunit.client.0.smithi136.stdout:0000003b 2024-09-15T20:21:51.873 INFO:tasks.workunit.client.0.smithi136.stdout: 2024-09-15T20:21:51.873 INFO:tasks.workunit.client.0.smithi136.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000004_0000000000000001.FA570D58.5.objxxx.. 2024-09-15T20:21:51.873 INFO:tasks.workunit.client.0.smithi136.stdout:00000000 01 01 35 00 00 00 04 00 00 00 00 00 00 00 04 03 |..5.............| 2024-09-15T20:21:51.873 INFO:tasks.workunit.client.0.smithi136.stdout:00000010 27 00 00 00 00 00 00 00 06 00 00 00 6f 62 6a 78 |'...........objx| 2024-09-15T20:21:51.873 INFO:tasks.workunit.client.0.smithi136.stdout:00000020 78 78 05 00 00 00 00 00 00 00 af 75 d0 85 00 00 |xx.........u....| 2024-09-15T20:21:51.873 INFO:tasks.workunit.client.0.smithi136.stdout:00000030 00 00 00 01 00 00 00 00 00 00 00 |...........| 2024-09-15T20:21:51.873 INFO:tasks.workunit.client.0.smithi136.stdout:0000003b 2024-09-15T20:21:51.874 INFO:tasks.workunit.client.0.smithi136.stdout: 2024-09-15T20:21:51.878 INFO:tasks.workunit.client.0.smithi136.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000005_0000000000000001.FA570D58.5.objxxx.. 2024-09-15T20:21:51.878 INFO:tasks.workunit.client.0.smithi136.stdout:00000000 01 01 35 00 00 00 05 00 00 00 00 00 00 00 04 03 |..5.............| 2024-09-15T20:21:51.878 INFO:tasks.workunit.client.0.smithi136.stdout:00000010 27 00 00 00 00 00 00 00 06 00 00 00 6f 62 6a 78 |'...........objx| 2024-09-15T20:21:51.878 INFO:tasks.workunit.client.0.smithi136.stdout:00000020 78 78 05 00 00 00 00 00 00 00 af 75 d0 85 00 00 |xx.........u....| 2024-09-15T20:21:51.881 INFO:tasks.workunit.client.0.smithi136.stdout:00000030 00 00 00 01 00 00 00 00 00 00 00 |...........| 2024-09-15T20:21:51.881 INFO:tasks.workunit.client.0.smithi136.stdout:0000003b 2024-09-15T20:21:51.881 INFO:tasks.workunit.client.0.smithi136.stdout: 2024-09-15T20:21:51.881 INFO:tasks.workunit.client.0.smithi136.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02~ 2024-09-15T20:21:51.881 INFO:tasks.workunit.client.0.smithi136.stdout: 2024-09-15T20:21:51.881 INFO:tasks.workunit.client.0.smithi136.stdout:p %00%00%00%00%00%00%00%00%c1%a3%fcn%00%00%00%00%00%00%04%03~ 2024-09-15T20:21:51.881 INFO:tasks.workunit.client.0.smithi136.stdout: 2024-09-15T20:21:51.881 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:21:51.829+0000 7f67adc2f980 1 bluefs umount 2024-09-15T20:21:51.882 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:21:51.829+0000 7f67adc2f980 1 bdev(0x56061ec92400 td/osd-mapper/0/block) close 2024-09-15T20:21:52.050 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:21:52.049+0000 7f67adc2f980 1 freelist shutdown 2024-09-15T20:21:52.051 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:21:52.049+0000 7f67adc2f980 1 bdev(0x56061eb17c00 td/osd-mapper/0/block) close 2024-09-15T20:21:52.283 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:100: TEST_truncated_sna_record: (( extr_dbg >= 2 )) 2024-09-15T20:21:52.283 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:100: TEST_truncated_sna_record: ceph-kvstore-tool bluestore-kv td/osd-mapper/2 dump p 2024-09-15T20:21:52.283 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:100: TEST_truncated_sna_record: grep -a SNA_ 2024-09-15T20:21:52.310 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:21:52.306+0000 7f30c28d2980 1 bdev(0x55d9131d3c00 td/osd-mapper/2/block) open path td/osd-mapper/2/block 2024-09-15T20:21:52.311 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:21:52.306+0000 7f30c28d2980 0 bdev(0x55d9131d3c00 td/osd-mapper/2/block) ioctl(F_SET_FILE_RW_HINT) on td/osd-mapper/2/block failed: (22) Invalid argument 2024-09-15T20:21:52.311 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:21:52.307+0000 7f30c28d2980 1 bdev(0x55d9131d3c00 td/osd-mapper/2/block) open size 107374182400 (0x1900000000, 100 GiB) block_size 4096 (4 KiB) rotational device, discard supported 2024-09-15T20:21:52.330 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:21:52.327+0000 7f30c28d2980 1 bdev(0x55d9131d3c00 td/osd-mapper/2/block) close 2024-09-15T20:21:52.571 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:21:52.569+0000 7f30c28d2980 1 bdev(0x55d9131d3c00 td/osd-mapper/2/block) open path td/osd-mapper/2/block 2024-09-15T20:21:52.571 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:21:52.569+0000 7f30c28d2980 0 bdev(0x55d9131d3c00 td/osd-mapper/2/block) ioctl(F_SET_FILE_RW_HINT) on td/osd-mapper/2/block failed: (22) Invalid argument 2024-09-15T20:21:52.571 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:21:52.569+0000 7f30c28d2980 1 bdev(0x55d9131d3c00 td/osd-mapper/2/block) open size 107374182400 (0x1900000000, 100 GiB) block_size 4096 (4 KiB) rotational device, discard supported 2024-09-15T20:21:52.581 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:21:52.579+0000 7f30c28d2980 1 bluestore(td/osd-mapper/2) _set_cache_sizes cache_size 1073741824 meta 0.45 kv 0.45 kv_onode 0.04 data 0.06 2024-09-15T20:21:52.583 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:21:52.579+0000 7f30c28d2980 1 bdev(0x55d91334e400 td/osd-mapper/2/block) open path td/osd-mapper/2/block 2024-09-15T20:21:52.583 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:21:52.579+0000 7f30c28d2980 0 bdev(0x55d91334e400 td/osd-mapper/2/block) ioctl(F_SET_FILE_RW_HINT) on td/osd-mapper/2/block failed: (22) Invalid argument 2024-09-15T20:21:52.583 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:21:52.579+0000 7f30c28d2980 1 bdev(0x55d91334e400 td/osd-mapper/2/block) open size 107374182400 (0x1900000000, 100 GiB) block_size 4096 (4 KiB) rotational device, discard supported 2024-09-15T20:21:52.583 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:21:52.579+0000 7f30c28d2980 1 bluefs add_block_device bdev 1 path td/osd-mapper/2/block size 100 GiB 2024-09-15T20:21:52.583 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:21:52.579+0000 7f30c28d2980 1 bluefs mount 2024-09-15T20:21:52.583 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:21:52.580+0000 7f30c28d2980 1 bluefs _init_alloc shared, id 1, capacity 0x1900000000, block size 0x10000 2024-09-15T20:21:52.589 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:21:52.587+0000 7f30c28d2980 1 bluefs mount shared_bdev_used = 0 2024-09-15T20:21:52.589 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:21:52.587+0000 7f30c28d2980 1 bluestore(td/osd-mapper/2) _prepare_db_environment set db_paths to db,102005473280 db.slow,102005473280 2024-09-15T20:21:52.630 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:21:52.624+0000 7f30c28d2980 1 bluestore(td/osd-mapper/2) _open_db opened rocksdb path db options compression=kLZ4Compression,max_write_buffer_number=64,min_write_buffer_number_to_merge=6,compaction_style=kCompactionStyleLevel,write_buffer_size=16777216,max_background_jobs=4,level0_file_num_compaction_trigger=8,max_bytes_for_level_base=1073741824,max_bytes_for_level_multiplier=8,compaction_readahead_size=2MB,max_total_wal_size=1073741824,writable_file_max_buffer_size=0 2024-09-15T20:21:52.630 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:21:52.624+0000 7f30c28d2980 1 bluestore(td/osd-mapper/2) _open_super_meta old nid_max 2050 2024-09-15T20:21:52.631 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:21:52.624+0000 7f30c28d2980 1 bluestore(td/osd-mapper/2) _open_super_meta old blobid_max 20480 2024-09-15T20:21:52.631 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:21:52.624+0000 7f30c28d2980 1 bluestore(td/osd-mapper/2) _open_super_meta ondisk_format 4 compat_ondisk_format 3 2024-09-15T20:21:52.631 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:21:52.624+0000 7f30c28d2980 1 bluestore(td/osd-mapper/2) _open_super_meta min_alloc_size 0x1000 2024-09-15T20:21:52.631 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:21:52.624+0000 7f30c28d2980 1 freelist init 2024-09-15T20:21:52.631 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:21:52.624+0000 7f30c28d2980 1 freelist _read_cfg 2024-09-15T20:21:52.631 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:21:52.624+0000 7f30c28d2980 1 bluestore(td/osd-mapper/2) _open_fm effective freelist_type = bitmap, freelist_alloc_size = 0x1000, min_alloc_size = 0x1000 2024-09-15T20:21:52.632 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:21:52.625+0000 7f30c28d2980 1 bluestore(td/osd-mapper/2) _init_alloc loaded 100 GiB in 17 extents, allocator type hybrid, capacity 0x1900000000, block size 0x1000, free 0x18fffbc000, fragmentation 6.10353e-07 2024-09-15T20:21:52.632 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:21:52.625+0000 7f30c28d2980 1 bluefs umount 2024-09-15T20:21:52.632 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:21:52.625+0000 7f30c28d2980 1 bdev(0x55d91334e400 td/osd-mapper/2/block) close 2024-09-15T20:21:52.844 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:21:52.841+0000 7f30c28d2980 1 bdev(0x55d91334e400 td/osd-mapper/2/block) open path td/osd-mapper/2/block 2024-09-15T20:21:52.844 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:21:52.841+0000 7f30c28d2980 0 bdev(0x55d91334e400 td/osd-mapper/2/block) ioctl(F_SET_FILE_RW_HINT) on td/osd-mapper/2/block failed: (22) Invalid argument 2024-09-15T20:21:52.844 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:21:52.841+0000 7f30c28d2980 1 bdev(0x55d91334e400 td/osd-mapper/2/block) open size 107374182400 (0x1900000000, 100 GiB) block_size 4096 (4 KiB) rotational device, discard supported 2024-09-15T20:21:52.844 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:21:52.841+0000 7f30c28d2980 1 bluefs add_block_device bdev 1 path td/osd-mapper/2/block size 100 GiB 2024-09-15T20:21:52.844 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:21:52.841+0000 7f30c28d2980 1 bluefs mount 2024-09-15T20:21:52.846 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:21:52.844+0000 7f30c28d2980 1 bluefs _init_alloc shared, id 1, capacity 0x1900000000, block size 0x10000 2024-09-15T20:21:52.853 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:21:52.850+0000 7f30c28d2980 1 bluefs mount shared_bdev_used = 27459584 2024-09-15T20:21:52.854 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:21:52.850+0000 7f30c28d2980 1 bluestore(td/osd-mapper/2) _prepare_db_environment set db_paths to db,102005473280 db.slow,102005473280 2024-09-15T20:21:52.883 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:21:52.880+0000 7f30c28d2980 1 bluestore(td/osd-mapper/2) _open_db opened rocksdb path db options compression=kLZ4Compression,max_write_buffer_number=64,min_write_buffer_number_to_merge=6,compaction_style=kCompactionStyleLevel,write_buffer_size=16777216,max_background_jobs=4,level0_file_num_compaction_trigger=8,max_bytes_for_level_base=1073741824,max_bytes_for_level_multiplier=8,compaction_readahead_size=2MB,max_total_wal_size=1073741824,writable_file_max_buffer_size=0 2024-09-15T20:21:52.883 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:21:52.880+0000 7f30c28d2980 1 bluefs umount 2024-09-15T20:21:52.883 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:21:52.880+0000 7f30c28d2980 1 bdev(0x55d91334e400 td/osd-mapper/2/block) close 2024-09-15T20:21:53.107 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:21:53.105+0000 7f30c28d2980 1 freelist shutdown 2024-09-15T20:21:53.107 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:21:53.105+0000 7f30c28d2980 1 bdev(0x55d9131d3c00 td/osd-mapper/2/block) close 2024-09-15T20:21:53.335 INFO:tasks.workunit.client.0.smithi136.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.FA570D58.2.objxxx.. 2024-09-15T20:21:53.336 INFO:tasks.workunit.client.0.smithi136.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000002_0000000000000001.FA570D58.2.objxxx.. 2024-09-15T20:21:53.336 INFO:tasks.workunit.client.0.smithi136.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.FA570D58.3.objxxx.. 2024-09-15T20:21:53.336 INFO:tasks.workunit.client.0.smithi136.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000004_0000000000000001.FA570D58.5.objxxx.. 2024-09-15T20:21:53.336 INFO:tasks.workunit.client.0.smithi136.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000005_0000000000000001.FA570D58.5.objxxx.. 2024-09-15T20:21:53.337 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:101: TEST_truncated_sna_record: (( extr_dbg >= 2 )) 2024-09-15T20:21:53.337 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:101: TEST_truncated_sna_record: grep -a SNA_ /tmp/oo2.dump 2024-09-15T20:21:53.337 INFO:tasks.workunit.client.0.smithi136.stderr:grep: /tmp/oo2.dump: No such file or directory 2024-09-15T20:21:53.338 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:102: TEST_truncated_sna_record: (( extr_dbg >= 2 )) 2024-09-15T20:21:53.338 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:102: TEST_truncated_sna_record: ceph-kvstore-tool bluestore-kv td/osd-mapper/2 dump p 2024-09-15T20:21:53.941 INFO:tasks.workunit.client.0.smithi136.stdout:p %00%00%00%00%00%00%00%00%7b%3fC%c4%00%00%00%00%00%00%00%01.osd_superblock 2024-09-15T20:21:53.941 INFO:tasks.workunit.client.0.smithi136.stdout:00000000 0b 05 58 02 00 00 e9 f4 cc 96 8f 95 49 71 84 e5 |..X.........Iq..| 2024-09-15T20:21:53.941 INFO:tasks.workunit.client.0.smithi136.stdout:00000010 d6 4a cd 70 df 0f 02 00 00 00 1e 00 00 00 00 00 |.J.p............| 2024-09-15T20:21:53.941 INFO:tasks.workunit.client.0.smithi136.stdout:00000020 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................| 2024-09-15T20:21:53.941 INFO:tasks.workunit.client.0.smithi136.stdout:* 2024-09-15T20:21:53.941 INFO:tasks.workunit.client.0.smithi136.stdout:00000040 00 00 00 00 00 00 fe ff 03 00 00 00 00 00 11 00 |................| 2024-09-15T20:21:53.942 INFO:tasks.workunit.client.0.smithi136.stdout:00000050 00 00 01 00 00 00 00 00 00 00 1a 00 00 00 69 6e |..............in| 2024-09-15T20:21:53.942 INFO:tasks.workunit.client.0.smithi136.stdout:00000060 69 74 69 61 6c 20 66 65 61 74 75 72 65 20 73 65 |itial feature se| 2024-09-15T20:21:53.942 INFO:tasks.workunit.client.0.smithi136.stdout:00000070 74 28 7e 76 2e 31 38 29 02 00 00 00 00 00 00 00 |t(~v.18)........| 2024-09-15T20:21:53.942 INFO:tasks.workunit.client.0.smithi136.stdout:00000080 0d 00 00 00 70 67 69 6e 66 6f 20 6f 62 6a 65 63 |....pginfo objec| 2024-09-15T20:21:53.942 INFO:tasks.workunit.client.0.smithi136.stdout:00000090 74 03 00 00 00 00 00 00 00 0e 00 00 00 6f 62 6a |t............obj| 2024-09-15T20:21:53.942 INFO:tasks.workunit.client.0.smithi136.stdout:000000a0 65 63 74 20 6c 6f 63 61 74 6f 72 04 00 00 00 00 |ect locator.....| 2024-09-15T20:21:53.942 INFO:tasks.workunit.client.0.smithi136.stdout:000000b0 00 00 00 10 00 00 00 6c 61 73 74 5f 65 70 6f 63 |.......last_epoc| 2024-09-15T20:21:53.942 INFO:tasks.workunit.client.0.smithi136.stdout:000000c0 68 5f 63 6c 65 61 6e 05 00 00 00 00 00 00 00 0a |h_clean.........| 2024-09-15T20:21:53.942 INFO:tasks.workunit.client.0.smithi136.stdout:000000d0 00 00 00 63 61 74 65 67 6f 72 69 65 73 06 00 00 |...categories...| 2024-09-15T20:21:53.942 INFO:tasks.workunit.client.0.smithi136.stdout:000000e0 00 00 00 00 00 0b 00 00 00 68 6f 62 6a 65 63 74 |.........hobject| 2024-09-15T20:21:53.942 INFO:tasks.workunit.client.0.smithi136.stdout:000000f0 70 6f 6f 6c 07 00 00 00 00 00 00 00 07 00 00 00 |pool............| 2024-09-15T20:21:53.942 INFO:tasks.workunit.client.0.smithi136.stdout:00000100 62 69 67 69 6e 66 6f 08 00 00 00 00 00 00 00 0b |biginfo.........| 2024-09-15T20:21:53.942 INFO:tasks.workunit.client.0.smithi136.stdout:00000110 00 00 00 6c 65 76 65 6c 64 62 69 6e 66 6f 09 00 |...leveldbinfo..| 2024-09-15T20:21:53.944 INFO:tasks.workunit.client.0.smithi136.stdout:00000120 00 00 00 00 00 00 0a 00 00 00 6c 65 76 65 6c 64 |..........leveld| 2024-09-15T20:21:53.944 INFO:tasks.workunit.client.0.smithi136.stdout:00000130 62 6c 6f 67 0a 00 00 00 00 00 00 00 0a 00 00 00 |blog............| 2024-09-15T20:21:53.944 INFO:tasks.workunit.client.0.smithi136.stdout:00000140 73 6e 61 70 6d 61 70 70 65 72 0b 00 00 00 00 00 |snapmapper......| 2024-09-15T20:21:53.944 INFO:tasks.workunit.client.0.smithi136.stdout:00000150 00 00 0f 00 00 00 73 68 61 72 64 65 64 20 6f 62 |......sharded ob| 2024-09-15T20:21:53.944 INFO:tasks.workunit.client.0.smithi136.stdout:00000160 6a 65 63 74 73 0c 00 00 00 00 00 00 00 11 00 00 |jects...........| 2024-09-15T20:21:53.944 INFO:tasks.workunit.client.0.smithi136.stdout:00000170 00 74 72 61 6e 73 61 63 74 69 6f 6e 20 68 69 6e |.transaction hin| 2024-09-15T20:21:53.944 INFO:tasks.workunit.client.0.smithi136.stdout:00000180 74 73 0d 00 00 00 00 00 00 00 0e 00 00 00 70 67 |ts............pg| 2024-09-15T20:21:53.945 INFO:tasks.workunit.client.0.smithi136.stdout:00000190 20 6d 65 74 61 20 6f 62 6a 65 63 74 0e 00 00 00 | meta object....| 2024-09-15T20:21:53.945 INFO:tasks.workunit.client.0.smithi136.stdout:000001a0 00 00 00 00 14 00 00 00 65 78 70 6c 69 63 69 74 |........explicit| 2024-09-15T20:21:53.945 INFO:tasks.workunit.client.0.smithi136.stdout:000001b0 20 6d 69 73 73 69 6e 67 20 73 65 74 0f 00 00 00 | missing set....| 2024-09-15T20:21:53.945 INFO:tasks.workunit.client.0.smithi136.stdout:000001c0 00 00 00 00 10 00 00 00 66 61 73 74 69 6e 66 6f |........fastinfo| 2024-09-15T20:21:53.945 INFO:tasks.workunit.client.0.smithi136.stdout:000001d0 20 70 67 20 61 74 74 72 10 00 00 00 00 00 00 00 | pg attr........| 2024-09-15T20:21:53.945 INFO:tasks.workunit.client.0.smithi136.stdout:000001e0 16 00 00 00 64 65 6c 65 74 65 73 20 69 6e 20 6d |....deletes in m| 2024-09-15T20:21:53.945 INFO:tasks.workunit.client.0.smithi136.stdout:000001f0 69 73 73 69 6e 67 20 73 65 74 11 00 00 00 00 00 |issing set......| 2024-09-15T20:21:53.945 INFO:tasks.workunit.client.0.smithi136.stdout:00000200 00 00 1c 00 00 00 6e 65 77 20 73 6e 61 70 6d 61 |......new snapma| 2024-09-15T20:21:53.945 INFO:tasks.workunit.client.0.smithi136.stdout:00000210 70 70 65 72 20 6b 65 79 20 73 74 72 75 63 74 75 |pper key structu| 2024-09-15T20:21:53.945 INFO:tasks.workunit.client.0.smithi136.stdout:00000220 72 65 1e 00 00 00 0f 00 00 00 df b0 17 d7 85 65 |re.............e| 2024-09-15T20:21:53.945 INFO:tasks.workunit.client.0.smithi136.stdout:00000230 48 db ad c5 cd ca 8e d8 86 3d 00 00 00 00 00 00 |H........=......| 2024-09-15T20:21:53.945 INFO:tasks.workunit.client.0.smithi136.stdout:00000240 00 00 1e 00 00 00 c9 41 e7 66 eb e9 1f 26 01 00 |.......A.f...&..| 2024-09-15T20:21:53.945 INFO:tasks.workunit.client.0.smithi136.stdout:00000250 00 00 01 00 00 00 01 00 00 00 1e 00 00 00 |..............| 2024-09-15T20:21:53.946 INFO:tasks.workunit.client.0.smithi136.stdout:0000025e 2024-09-15T20:21:53.946 INFO:tasks.workunit.client.0.smithi136.stdout: 2024-09-15T20:21:53.946 INFO:tasks.workunit.client.0.smithi136.stdout:p %00%00%00%00%00%00%00%00%7b%3fC%c4%00%00%00%00%00%00%00%01~ 2024-09-15T20:21:53.946 INFO:tasks.workunit.client.0.smithi136.stdout: 2024-09-15T20:21:53.946 INFO:tasks.workunit.client.0.smithi136.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.OBJ_0000000000000001.FA570D58.2.objxxx.. 2024-09-15T20:21:53.946 INFO:tasks.workunit.client.0.smithi136.stdout:00000000 01 01 41 00 00 00 04 03 27 00 00 00 00 00 00 00 |..A.....'.......| 2024-09-15T20:21:53.946 INFO:tasks.workunit.client.0.smithi136.stdout:00000010 06 00 00 00 6f 62 6a 78 78 78 02 00 00 00 00 00 |....objxxx......| 2024-09-15T20:21:53.946 INFO:tasks.workunit.client.0.smithi136.stdout:00000020 00 00 af 75 d0 85 00 00 00 00 00 01 00 00 00 00 |...u............| 2024-09-15T20:21:53.946 INFO:tasks.workunit.client.0.smithi136.stdout:00000030 00 00 00 02 00 00 00 01 00 00 00 00 00 00 00 02 |................| 2024-09-15T20:21:53.946 INFO:tasks.workunit.client.0.smithi136.stdout:00000040 00 00 00 00 00 00 00 |.......| 2024-09-15T20:21:53.946 INFO:tasks.workunit.client.0.smithi136.stdout:00000047 2024-09-15T20:21:53.946 INFO:tasks.workunit.client.0.smithi136.stdout: 2024-09-15T20:21:53.946 INFO:tasks.workunit.client.0.smithi136.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.OBJ_0000000000000001.FA570D58.3.objxxx.. 2024-09-15T20:21:53.946 INFO:tasks.workunit.client.0.smithi136.stdout:00000000 01 01 39 00 00 00 04 03 27 00 00 00 00 00 00 00 |..9.....'.......| 2024-09-15T20:21:53.947 INFO:tasks.workunit.client.0.smithi136.stdout:00000010 06 00 00 00 6f 62 6a 78 78 78 03 00 00 00 00 00 |....objxxx......| 2024-09-15T20:21:53.947 INFO:tasks.workunit.client.0.smithi136.stdout:00000020 00 00 af 75 d0 85 00 00 00 00 00 01 00 00 00 00 |...u............| 2024-09-15T20:21:53.947 INFO:tasks.workunit.client.0.smithi136.stdout:00000030 00 00 00 01 00 00 00 03 00 00 00 00 00 00 00 |...............| 2024-09-15T20:21:53.947 INFO:tasks.workunit.client.0.smithi136.stdout:0000003f 2024-09-15T20:21:53.947 INFO:tasks.workunit.client.0.smithi136.stdout: 2024-09-15T20:21:53.947 INFO:tasks.workunit.client.0.smithi136.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.OBJ_0000000000000001.FA570D58.5.objxxx.. 2024-09-15T20:21:53.947 INFO:tasks.workunit.client.0.smithi136.stdout:00000000 01 01 41 00 00 00 04 03 27 00 00 00 00 00 00 00 |..A.....'.......| 2024-09-15T20:21:53.947 INFO:tasks.workunit.client.0.smithi136.stdout:00000010 06 00 00 00 6f 62 6a 78 78 78 05 00 00 00 00 00 |....objxxx......| 2024-09-15T20:21:53.947 INFO:tasks.workunit.client.0.smithi136.stdout:00000020 00 00 af 75 d0 85 00 00 00 00 00 01 00 00 00 00 |...u............| 2024-09-15T20:21:53.947 INFO:tasks.workunit.client.0.smithi136.stdout:00000030 00 00 00 02 00 00 00 04 00 00 00 00 00 00 00 05 |................| 2024-09-15T20:21:53.948 INFO:tasks.workunit.client.0.smithi136.stdout:00000040 00 00 00 00 00 00 00 |.......| 2024-09-15T20:21:53.948 INFO:tasks.workunit.client.0.smithi136.stdout:00000047 2024-09-15T20:21:53.948 INFO:tasks.workunit.client.0.smithi136.stdout: 2024-09-15T20:21:53.948 INFO:tasks.workunit.client.0.smithi136.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.FA570D58.2.objxxx.. 2024-09-15T20:21:53.948 INFO:tasks.workunit.client.0.smithi136.stdout:00000000 01 01 35 00 00 00 01 00 00 00 00 00 00 00 04 03 |..5.............| 2024-09-15T20:21:53.948 INFO:tasks.workunit.client.0.smithi136.stdout:00000010 27 00 00 00 00 00 00 00 06 00 00 00 6f 62 6a 78 |'...........objx| 2024-09-15T20:21:53.948 INFO:tasks.workunit.client.0.smithi136.stdout:00000020 78 78 02 00 00 00 00 00 00 00 af 75 d0 85 00 00 |xx.........u....| 2024-09-15T20:21:53.949 INFO:tasks.workunit.client.0.smithi136.stdout:00000030 00 00 00 01 00 00 00 00 00 00 00 |...........| 2024-09-15T20:21:53.949 INFO:tasks.workunit.client.0.smithi136.stdout:0000003b 2024-09-15T20:21:53.949 INFO:tasks.workunit.client.0.smithi136.stdout: 2024-09-15T20:21:53.949 INFO:tasks.workunit.client.0.smithi136.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000002_0000000000000001.FA570D58.2.objxxx.. 2024-09-15T20:21:53.949 INFO:tasks.workunit.client.0.smithi136.stdout:00000000 01 01 35 00 00 00 02 00 00 00 00 00 00 00 04 03 |..5.............| 2024-09-15T20:21:53.949 INFO:tasks.workunit.client.0.smithi136.stdout:00000010 27 00 00 00 00 00 00 00 06 00 00 00 6f 62 6a 78 |'...........objx| 2024-09-15T20:21:53.949 INFO:tasks.workunit.client.0.smithi136.stdout:00000020 78 78 02 00 00 00 00 00 00 00 af 75 d0 85 00 00 |xx.........u....| 2024-09-15T20:21:53.949 INFO:tasks.workunit.client.0.smithi136.stdout:00000030 00 00 00 01 00 00 00 00 00 00 00 |...........| 2024-09-15T20:21:53.949 INFO:tasks.workunit.client.0.smithi136.stdout:0000003b 2024-09-15T20:21:53.949 INFO:tasks.workunit.client.0.smithi136.stdout: 2024-09-15T20:21:53.949 INFO:tasks.workunit.client.0.smithi136.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.FA570D58.3.objxxx.. 2024-09-15T20:21:53.949 INFO:tasks.workunit.client.0.smithi136.stdout:00000000 01 01 35 00 00 00 03 00 00 00 00 00 00 00 04 03 |..5.............| 2024-09-15T20:21:53.949 INFO:tasks.workunit.client.0.smithi136.stdout:00000010 27 00 00 00 00 00 00 00 06 00 00 00 6f 62 6a 78 |'...........objx| 2024-09-15T20:21:53.949 INFO:tasks.workunit.client.0.smithi136.stdout:00000020 78 78 03 00 00 00 00 00 00 00 af 75 d0 85 00 00 |xx.........u....| 2024-09-15T20:21:53.950 INFO:tasks.workunit.client.0.smithi136.stdout:00000030 00 00 00 01 00 00 00 00 00 00 00 |...........| 2024-09-15T20:21:53.950 INFO:tasks.workunit.client.0.smithi136.stdout:0000003b 2024-09-15T20:21:53.950 INFO:tasks.workunit.client.0.smithi136.stdout: 2024-09-15T20:21:53.950 INFO:tasks.workunit.client.0.smithi136.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000004_0000000000000001.FA570D58.5.objxxx.. 2024-09-15T20:21:53.950 INFO:tasks.workunit.client.0.smithi136.stdout:00000000 01 01 35 00 00 00 04 00 00 00 00 00 00 00 04 03 |..5.............| 2024-09-15T20:21:53.950 INFO:tasks.workunit.client.0.smithi136.stdout:00000010 27 00 00 00 00 00 00 00 06 00 00 00 6f 62 6a 78 |'...........objx| 2024-09-15T20:21:53.950 INFO:tasks.workunit.client.0.smithi136.stdout:00000020 78 78 05 00 00 00 00 00 00 00 af 75 d0 85 00 00 |xx.........u....| 2024-09-15T20:21:53.950 INFO:tasks.workunit.client.0.smithi136.stdout:00000030 00 00 00 01 00 00 00 00 00 00 00 |...........| 2024-09-15T20:21:53.950 INFO:tasks.workunit.client.0.smithi136.stdout:0000003b 2024-09-15T20:21:53.950 INFO:tasks.workunit.client.0.smithi136.stdout: 2024-09-15T20:21:53.951 INFO:tasks.workunit.client.0.smithi136.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000005_0000000000000001.FA570D58.5.objxxx.. 2024-09-15T20:21:53.951 INFO:tasks.workunit.client.0.smithi136.stdout:00000000 01 01 35 00 00 00 05 00 00 00 00 00 00 00 04 03 |..5.............| 2024-09-15T20:21:53.951 INFO:tasks.workunit.client.0.smithi136.stdout:00000010 27 00 00 00 00 00 00 00 06 00 00 00 6f 62 6a 78 |'...........objx| 2024-09-15T20:21:53.952 INFO:tasks.workunit.client.0.smithi136.stdout:00000020 78 78 05 00 00 00 00 00 00 00 af 75 d0 85 00 00 |xx.........u....| 2024-09-15T20:21:53.952 INFO:tasks.workunit.client.0.smithi136.stdout:00000030 00 00 00 01 00 00 00 00 00 00 00 |...........| 2024-09-15T20:21:53.952 INFO:tasks.workunit.client.0.smithi136.stdout:0000003b 2024-09-15T20:21:53.952 INFO:tasks.workunit.client.0.smithi136.stdout: 2024-09-15T20:21:53.952 INFO:tasks.workunit.client.0.smithi136.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02~ 2024-09-15T20:21:53.952 INFO:tasks.workunit.client.0.smithi136.stdout: 2024-09-15T20:21:53.952 INFO:tasks.workunit.client.0.smithi136.stdout:p %00%00%00%00%00%00%00%00%c1%a3%fcn%00%00%00%00%00%00%04%03~ 2024-09-15T20:21:53.953 INFO:tasks.workunit.client.0.smithi136.stdout: 2024-09-15T20:21:54.396 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:104: TEST_truncated_sna_record: ceph-kvstore-tool bluestore-kv td/osd-mapper/1 dump p 2024-09-15T20:21:54.396 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:104: TEST_truncated_sna_record: grep -a -e 'SNA_[0-9]_000000000000000[0-9]_000000000000000' 2024-09-15T20:21:54.397 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:104: TEST_truncated_sna_record: awk -e '{print $2;}' 2024-09-15T20:21:54.397 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:104: TEST_truncated_sna_record: wc -l 2024-09-15T20:21:55.455 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:104: TEST_truncated_sna_record: local num_sna_b4=5 2024-09-15T20:21:55.456 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:106: TEST_truncated_sna_record: expr 3 - 1 2024-09-15T20:21:55.457 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:106: TEST_truncated_sna_record: seq 0 2 2024-09-15T20:21:55.458 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:106: TEST_truncated_sna_record: for sdn in $(seq 0 $(expr $osdn - 1)) 2024-09-15T20:21:55.458 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:108: TEST_truncated_sna_record: kvdir=td/osd-mapper/0 2024-09-15T20:21:55.458 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:109: TEST_truncated_sna_record: echo 'corrupting the SnapMapper DB of osd.0 (db: td/osd-mapper/0)' 2024-09-15T20:21:55.458 INFO:tasks.workunit.client.0.smithi136.stdout:corrupting the SnapMapper DB of osd.0 (db: td/osd-mapper/0) 2024-09-15T20:21:55.459 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:110: TEST_truncated_sna_record: (( extr_dbg >= 3 )) 2024-09-15T20:21:55.459 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:110: TEST_truncated_sna_record: ceph-kvstore-tool bluestore-kv td/osd-mapper/0 dump p 2024-09-15T20:21:55.483 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:21:55.482+0000 7f1e3d4f1980 1 bdev(0x55d448937c00 td/osd-mapper/0/block) open path td/osd-mapper/0/block 2024-09-15T20:21:55.484 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:21:55.482+0000 7f1e3d4f1980 0 bdev(0x55d448937c00 td/osd-mapper/0/block) ioctl(F_SET_FILE_RW_HINT) on td/osd-mapper/0/block failed: (22) Invalid argument 2024-09-15T20:21:55.484 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:21:55.482+0000 7f1e3d4f1980 1 bdev(0x55d448937c00 td/osd-mapper/0/block) open size 107374182400 (0x1900000000, 100 GiB) block_size 4096 (4 KiB) rotational device, discard supported 2024-09-15T20:21:55.490 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:21:55.488+0000 7f1e3d4f1980 1 bdev(0x55d448937c00 td/osd-mapper/0/block) close 2024-09-15T20:21:55.746 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:21:55.744+0000 7f1e3d4f1980 1 bdev(0x55d448937c00 td/osd-mapper/0/block) open path td/osd-mapper/0/block 2024-09-15T20:21:55.746 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:21:55.744+0000 7f1e3d4f1980 0 bdev(0x55d448937c00 td/osd-mapper/0/block) ioctl(F_SET_FILE_RW_HINT) on td/osd-mapper/0/block failed: (22) Invalid argument 2024-09-15T20:21:55.746 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:21:55.744+0000 7f1e3d4f1980 1 bdev(0x55d448937c00 td/osd-mapper/0/block) open size 107374182400 (0x1900000000, 100 GiB) block_size 4096 (4 KiB) rotational device, discard supported 2024-09-15T20:21:55.748 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:21:55.745+0000 7f1e3d4f1980 1 bluestore(td/osd-mapper/0) _set_cache_sizes cache_size 1073741824 meta 0.45 kv 0.45 kv_onode 0.04 data 0.06 2024-09-15T20:21:55.748 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:21:55.745+0000 7f1e3d4f1980 1 bdev(0x55d448ab2400 td/osd-mapper/0/block) open path td/osd-mapper/0/block 2024-09-15T20:21:55.748 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:21:55.745+0000 7f1e3d4f1980 0 bdev(0x55d448ab2400 td/osd-mapper/0/block) ioctl(F_SET_FILE_RW_HINT) on td/osd-mapper/0/block failed: (22) Invalid argument 2024-09-15T20:21:55.748 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:21:55.745+0000 7f1e3d4f1980 1 bdev(0x55d448ab2400 td/osd-mapper/0/block) open size 107374182400 (0x1900000000, 100 GiB) block_size 4096 (4 KiB) rotational device, discard supported 2024-09-15T20:21:55.748 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:21:55.745+0000 7f1e3d4f1980 1 bluefs add_block_device bdev 1 path td/osd-mapper/0/block size 100 GiB 2024-09-15T20:21:55.748 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:21:55.745+0000 7f1e3d4f1980 1 bluefs mount 2024-09-15T20:21:55.749 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:21:55.746+0000 7f1e3d4f1980 1 bluefs _init_alloc shared, id 1, capacity 0x1900000000, block size 0x10000 2024-09-15T20:21:55.756 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:21:55.753+0000 7f1e3d4f1980 1 bluefs mount shared_bdev_used = 0 2024-09-15T20:21:55.757 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:21:55.753+0000 7f1e3d4f1980 1 bluestore(td/osd-mapper/0) _prepare_db_environment set db_paths to db,102005473280 db.slow,102005473280 2024-09-15T20:21:55.793 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:21:55.791+0000 7f1e3d4f1980 1 bluestore(td/osd-mapper/0) _open_db opened rocksdb path db options compression=kLZ4Compression,max_write_buffer_number=64,min_write_buffer_number_to_merge=6,compaction_style=kCompactionStyleLevel,write_buffer_size=16777216,max_background_jobs=4,level0_file_num_compaction_trigger=8,max_bytes_for_level_base=1073741824,max_bytes_for_level_multiplier=8,compaction_readahead_size=2MB,max_total_wal_size=1073741824,writable_file_max_buffer_size=0 2024-09-15T20:21:55.793 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:21:55.791+0000 7f1e3d4f1980 1 bluestore(td/osd-mapper/0) _open_super_meta old nid_max 2050 2024-09-15T20:21:55.793 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:21:55.791+0000 7f1e3d4f1980 1 bluestore(td/osd-mapper/0) _open_super_meta old blobid_max 20480 2024-09-15T20:21:55.793 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:21:55.791+0000 7f1e3d4f1980 1 bluestore(td/osd-mapper/0) _open_super_meta ondisk_format 4 compat_ondisk_format 3 2024-09-15T20:21:55.794 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:21:55.791+0000 7f1e3d4f1980 1 bluestore(td/osd-mapper/0) _open_super_meta min_alloc_size 0x1000 2024-09-15T20:21:55.794 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:21:55.791+0000 7f1e3d4f1980 1 freelist init 2024-09-15T20:21:55.794 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:21:55.791+0000 7f1e3d4f1980 1 freelist _read_cfg 2024-09-15T20:21:55.794 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:21:55.791+0000 7f1e3d4f1980 1 bluestore(td/osd-mapper/0) _open_fm effective freelist_type = bitmap, freelist_alloc_size = 0x1000, min_alloc_size = 0x1000 2024-09-15T20:21:55.794 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:21:55.792+0000 7f1e3d4f1980 1 bluestore(td/osd-mapper/0) _init_alloc loaded 100 GiB in 26 extents, allocator type hybrid, capacity 0x1900000000, block size 0x1000, free 0x18fffbc000, fragmentation 9.53677e-07 2024-09-15T20:21:55.794 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:21:55.792+0000 7f1e3d4f1980 1 bluefs umount 2024-09-15T20:21:55.795 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:21:55.792+0000 7f1e3d4f1980 1 bdev(0x55d448ab2400 td/osd-mapper/0/block) close 2024-09-15T20:21:56.010 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:21:56.008+0000 7f1e3d4f1980 1 bdev(0x55d448ab2400 td/osd-mapper/0/block) open path td/osd-mapper/0/block 2024-09-15T20:21:56.010 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:21:56.008+0000 7f1e3d4f1980 0 bdev(0x55d448ab2400 td/osd-mapper/0/block) ioctl(F_SET_FILE_RW_HINT) on td/osd-mapper/0/block failed: (22) Invalid argument 2024-09-15T20:21:56.010 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:21:56.008+0000 7f1e3d4f1980 1 bdev(0x55d448ab2400 td/osd-mapper/0/block) open size 107374182400 (0x1900000000, 100 GiB) block_size 4096 (4 KiB) rotational device, discard supported 2024-09-15T20:21:56.010 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:21:56.008+0000 7f1e3d4f1980 1 bluefs add_block_device bdev 1 path td/osd-mapper/0/block size 100 GiB 2024-09-15T20:21:56.010 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:21:56.008+0000 7f1e3d4f1980 1 bluefs mount 2024-09-15T20:21:56.014 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:21:56.012+0000 7f1e3d4f1980 1 bluefs _init_alloc shared, id 1, capacity 0x1900000000, block size 0x10000 2024-09-15T20:21:56.026 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:21:56.021+0000 7f1e3d4f1980 1 bluefs mount shared_bdev_used = 27459584 2024-09-15T20:21:56.027 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:21:56.021+0000 7f1e3d4f1980 1 bluestore(td/osd-mapper/0) _prepare_db_environment set db_paths to db,102005473280 db.slow,102005473280 2024-09-15T20:21:56.055 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:21:56.053+0000 7f1e3d4f1980 1 bluestore(td/osd-mapper/0) _open_db opened rocksdb path db options compression=kLZ4Compression,max_write_buffer_number=64,min_write_buffer_number_to_merge=6,compaction_style=kCompactionStyleLevel,write_buffer_size=16777216,max_background_jobs=4,level0_file_num_compaction_trigger=8,max_bytes_for_level_base=1073741824,max_bytes_for_level_multiplier=8,compaction_readahead_size=2MB,max_total_wal_size=1073741824,writable_file_max_buffer_size=0 2024-09-15T20:21:56.059 INFO:tasks.workunit.client.0.smithi136.stdout:p %00%00%00%00%00%00%00%00%7b%3fC%c4%00%00%00%00%00%00%00%01.osd_superblock 2024-09-15T20:21:56.059 INFO:tasks.workunit.client.0.smithi136.stdout:00000000 0b 05 58 02 00 00 e9 f4 cc 96 8f 95 49 71 84 e5 |..X.........Iq..| 2024-09-15T20:21:56.059 INFO:tasks.workunit.client.0.smithi136.stdout:00000010 d6 4a cd 70 df 0f 00 00 00 00 1e 00 00 00 00 00 |.J.p............| 2024-09-15T20:21:56.060 INFO:tasks.workunit.client.0.smithi136.stdout:00000020 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................| 2024-09-15T20:21:56.060 INFO:tasks.workunit.client.0.smithi136.stdout:* 2024-09-15T20:21:56.060 INFO:tasks.workunit.client.0.smithi136.stdout:00000040 00 00 00 00 00 00 fe ff 03 00 00 00 00 00 11 00 |................| 2024-09-15T20:21:56.060 INFO:tasks.workunit.client.0.smithi136.stdout:00000050 00 00 01 00 00 00 00 00 00 00 1a 00 00 00 69 6e |..............in| 2024-09-15T20:21:56.060 INFO:tasks.workunit.client.0.smithi136.stdout:00000060 69 74 69 61 6c 20 66 65 61 74 75 72 65 20 73 65 |itial feature se| 2024-09-15T20:21:56.060 INFO:tasks.workunit.client.0.smithi136.stdout:00000070 74 28 7e 76 2e 31 38 29 02 00 00 00 00 00 00 00 |t(~v.18)........| 2024-09-15T20:21:56.060 INFO:tasks.workunit.client.0.smithi136.stdout:00000080 0d 00 00 00 70 67 69 6e 66 6f 20 6f 62 6a 65 63 |....pginfo objec| 2024-09-15T20:21:56.060 INFO:tasks.workunit.client.0.smithi136.stdout:00000090 74 03 00 00 00 00 00 00 00 0e 00 00 00 6f 62 6a |t............obj| 2024-09-15T20:21:56.060 INFO:tasks.workunit.client.0.smithi136.stdout:000000a0 65 63 74 20 6c 6f 63 61 74 6f 72 04 00 00 00 00 |ect locator.....| 2024-09-15T20:21:56.060 INFO:tasks.workunit.client.0.smithi136.stdout:000000b0 00 00 00 10 00 00 00 6c 61 73 74 5f 65 70 6f 63 |.......last_epoc| 2024-09-15T20:21:56.060 INFO:tasks.workunit.client.0.smithi136.stdout:000000c0 68 5f 63 6c 65 61 6e 05 00 00 00 00 00 00 00 0a |h_clean.........| 2024-09-15T20:21:56.061 INFO:tasks.workunit.client.0.smithi136.stdout:000000d0 00 00 00 63 61 74 65 67 6f 72 69 65 73 06 00 00 |...categories...| 2024-09-15T20:21:56.061 INFO:tasks.workunit.client.0.smithi136.stdout:000000e0 00 00 00 00 00 0b 00 00 00 68 6f 62 6a 65 63 74 |.........hobject| 2024-09-15T20:21:56.061 INFO:tasks.workunit.client.0.smithi136.stdout:000000f0 70 6f 6f 6c 07 00 00 00 00 00 00 00 07 00 00 00 |pool............| 2024-09-15T20:21:56.061 INFO:tasks.workunit.client.0.smithi136.stdout:00000100 62 69 67 69 6e 66 6f 08 00 00 00 00 00 00 00 0b |biginfo.........| 2024-09-15T20:21:56.061 INFO:tasks.workunit.client.0.smithi136.stdout:00000110 00 00 00 6c 65 76 65 6c 64 62 69 6e 66 6f 09 00 |...leveldbinfo..| 2024-09-15T20:21:56.061 INFO:tasks.workunit.client.0.smithi136.stdout:00000120 00 00 00 00 00 00 0a 00 00 00 6c 65 76 65 6c 64 |..........leveld| 2024-09-15T20:21:56.061 INFO:tasks.workunit.client.0.smithi136.stdout:00000130 62 6c 6f 67 0a 00 00 00 00 00 00 00 0a 00 00 00 |blog............| 2024-09-15T20:21:56.061 INFO:tasks.workunit.client.0.smithi136.stdout:00000140 73 6e 61 70 6d 61 70 70 65 72 0b 00 00 00 00 00 |snapmapper......| 2024-09-15T20:21:56.061 INFO:tasks.workunit.client.0.smithi136.stdout:00000150 00 00 0f 00 00 00 73 68 61 72 64 65 64 20 6f 62 |......sharded ob| 2024-09-15T20:21:56.061 INFO:tasks.workunit.client.0.smithi136.stdout:00000160 6a 65 63 74 73 0c 00 00 00 00 00 00 00 11 00 00 |jects...........| 2024-09-15T20:21:56.061 INFO:tasks.workunit.client.0.smithi136.stdout:00000170 00 74 72 61 6e 73 61 63 74 69 6f 6e 20 68 69 6e |.transaction hin| 2024-09-15T20:21:56.061 INFO:tasks.workunit.client.0.smithi136.stdout:00000180 74 73 0d 00 00 00 00 00 00 00 0e 00 00 00 70 67 |ts............pg| 2024-09-15T20:21:56.062 INFO:tasks.workunit.client.0.smithi136.stdout:00000190 20 6d 65 74 61 20 6f 62 6a 65 63 74 0e 00 00 00 | meta object....| 2024-09-15T20:21:56.062 INFO:tasks.workunit.client.0.smithi136.stdout:000001a0 00 00 00 00 14 00 00 00 65 78 70 6c 69 63 69 74 |........explicit| 2024-09-15T20:21:56.062 INFO:tasks.workunit.client.0.smithi136.stdout:000001b0 20 6d 69 73 73 69 6e 67 20 73 65 74 0f 00 00 00 | missing set....| 2024-09-15T20:21:56.062 INFO:tasks.workunit.client.0.smithi136.stdout:000001c0 00 00 00 00 10 00 00 00 66 61 73 74 69 6e 66 6f |........fastinfo| 2024-09-15T20:21:56.062 INFO:tasks.workunit.client.0.smithi136.stdout:000001d0 20 70 67 20 61 74 74 72 10 00 00 00 00 00 00 00 | pg attr........| 2024-09-15T20:21:56.062 INFO:tasks.workunit.client.0.smithi136.stdout:000001e0 16 00 00 00 64 65 6c 65 74 65 73 20 69 6e 20 6d |....deletes in m| 2024-09-15T20:21:56.062 INFO:tasks.workunit.client.0.smithi136.stdout:000001f0 69 73 73 69 6e 67 20 73 65 74 11 00 00 00 00 00 |issing set......| 2024-09-15T20:21:56.062 INFO:tasks.workunit.client.0.smithi136.stdout:00000200 00 00 1c 00 00 00 6e 65 77 20 73 6e 61 70 6d 61 |......new snapma| 2024-09-15T20:21:56.062 INFO:tasks.workunit.client.0.smithi136.stdout:00000210 70 70 65 72 20 6b 65 79 20 73 74 72 75 63 74 75 |pper key structu| 2024-09-15T20:21:56.062 INFO:tasks.workunit.client.0.smithi136.stdout:00000220 72 65 1e 00 00 00 05 00 00 00 13 0d be e2 9d fc |re..............| 2024-09-15T20:21:56.062 INFO:tasks.workunit.client.0.smithi136.stdout:00000230 4e de 9d 5f ae 08 c2 87 24 27 00 00 00 00 00 00 |N.._....$'......| 2024-09-15T20:21:56.062 INFO:tasks.workunit.client.0.smithi136.stdout:00000240 00 00 1e 00 00 00 84 41 e7 66 57 b0 a3 13 01 00 |.......A.fW.....| 2024-09-15T20:21:56.063 INFO:tasks.workunit.client.0.smithi136.stdout:00000250 00 00 01 00 00 00 01 00 00 00 1e 00 00 00 |..............| 2024-09-15T20:21:56.063 INFO:tasks.workunit.client.0.smithi136.stdout:0000025e 2024-09-15T20:21:56.063 INFO:tasks.workunit.client.0.smithi136.stdout: 2024-09-15T20:21:56.063 INFO:tasks.workunit.client.0.smithi136.stdout:p %00%00%00%00%00%00%00%00%7b%3fC%c4%00%00%00%00%00%00%00%01~ 2024-09-15T20:21:56.063 INFO:tasks.workunit.client.0.smithi136.stdout: 2024-09-15T20:21:56.063 INFO:tasks.workunit.client.0.smithi136.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.OBJ_0000000000000001.FA570D58.2.objxxx.. 2024-09-15T20:21:56.063 INFO:tasks.workunit.client.0.smithi136.stdout:00000000 01 01 41 00 00 00 04 03 27 00 00 00 00 00 00 00 |..A.....'.......| 2024-09-15T20:21:56.063 INFO:tasks.workunit.client.0.smithi136.stdout:00000010 06 00 00 00 6f 62 6a 78 78 78 02 00 00 00 00 00 |....objxxx......| 2024-09-15T20:21:56.063 INFO:tasks.workunit.client.0.smithi136.stdout:00000020 00 00 af 75 d0 85 00 00 00 00 00 01 00 00 00 00 |...u............| 2024-09-15T20:21:56.063 INFO:tasks.workunit.client.0.smithi136.stdout:00000030 00 00 00 02 00 00 00 01 00 00 00 00 00 00 00 02 |................| 2024-09-15T20:21:56.063 INFO:tasks.workunit.client.0.smithi136.stdout:00000040 00 00 00 00 00 00 00 |.......| 2024-09-15T20:21:56.063 INFO:tasks.workunit.client.0.smithi136.stdout:00000047 2024-09-15T20:21:56.064 INFO:tasks.workunit.client.0.smithi136.stdout: 2024-09-15T20:21:56.064 INFO:tasks.workunit.client.0.smithi136.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.OBJ_0000000000000001.FA570D58.3.objxxx.. 2024-09-15T20:21:56.064 INFO:tasks.workunit.client.0.smithi136.stdout:00000000 01 01 39 00 00 00 04 03 27 00 00 00 00 00 00 00 |..9.....'.......| 2024-09-15T20:21:56.064 INFO:tasks.workunit.client.0.smithi136.stdout:00000010 06 00 00 00 6f 62 6a 78 78 78 03 00 00 00 00 00 |....objxxx......| 2024-09-15T20:21:56.064 INFO:tasks.workunit.client.0.smithi136.stdout:00000020 00 00 af 75 d0 85 00 00 00 00 00 01 00 00 00 00 |...u............| 2024-09-15T20:21:56.064 INFO:tasks.workunit.client.0.smithi136.stdout:00000030 00 00 00 01 00 00 00 03 00 00 00 00 00 00 00 |...............| 2024-09-15T20:21:56.064 INFO:tasks.workunit.client.0.smithi136.stdout:0000003f 2024-09-15T20:21:56.064 INFO:tasks.workunit.client.0.smithi136.stdout: 2024-09-15T20:21:56.064 INFO:tasks.workunit.client.0.smithi136.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.OBJ_0000000000000001.FA570D58.5.objxxx.. 2024-09-15T20:21:56.064 INFO:tasks.workunit.client.0.smithi136.stdout:00000000 01 01 41 00 00 00 04 03 27 00 00 00 00 00 00 00 |..A.....'.......| 2024-09-15T20:21:56.064 INFO:tasks.workunit.client.0.smithi136.stdout:00000010 06 00 00 00 6f 62 6a 78 78 78 05 00 00 00 00 00 |....objxxx......| 2024-09-15T20:21:56.064 INFO:tasks.workunit.client.0.smithi136.stdout:00000020 00 00 af 75 d0 85 00 00 00 00 00 01 00 00 00 00 |...u............| 2024-09-15T20:21:56.065 INFO:tasks.workunit.client.0.smithi136.stdout:00000030 00 00 00 02 00 00 00 04 00 00 00 00 00 00 00 05 |................| 2024-09-15T20:21:56.065 INFO:tasks.workunit.client.0.smithi136.stdout:00000040 00 00 00 00 00 00 00 |.......| 2024-09-15T20:21:56.065 INFO:tasks.workunit.client.0.smithi136.stdout:00000047 2024-09-15T20:21:56.065 INFO:tasks.workunit.client.0.smithi136.stdout: 2024-09-15T20:21:56.065 INFO:tasks.workunit.client.0.smithi136.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.FA570D58.2.objxxx.. 2024-09-15T20:21:56.065 INFO:tasks.workunit.client.0.smithi136.stdout:00000000 01 01 35 00 00 00 01 00 00 00 00 00 00 00 04 03 |..5.............| 2024-09-15T20:21:56.065 INFO:tasks.workunit.client.0.smithi136.stdout:00000010 27 00 00 00 00 00 00 00 06 00 00 00 6f 62 6a 78 |'...........objx| 2024-09-15T20:21:56.065 INFO:tasks.workunit.client.0.smithi136.stdout:00000020 78 78 02 00 00 00 00 00 00 00 af 75 d0 85 00 00 |xx.........u....| 2024-09-15T20:21:56.065 INFO:tasks.workunit.client.0.smithi136.stdout:00000030 00 00 00 01 00 00 00 00 00 00 00 |...........| 2024-09-15T20:21:56.065 INFO:tasks.workunit.client.0.smithi136.stdout:0000003b 2024-09-15T20:21:56.065 INFO:tasks.workunit.client.0.smithi136.stdout: 2024-09-15T20:21:56.065 INFO:tasks.workunit.client.0.smithi136.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000002_0000000000000001.FA570D58.2.objxxx.. 2024-09-15T20:21:56.066 INFO:tasks.workunit.client.0.smithi136.stdout:00000000 01 01 35 00 00 00 02 00 00 00 00 00 00 00 04 03 |..5.............| 2024-09-15T20:21:56.066 INFO:tasks.workunit.client.0.smithi136.stdout:00000010 27 00 00 00 00 00 00 00 06 00 00 00 6f 62 6a 78 |'...........objx| 2024-09-15T20:21:56.066 INFO:tasks.workunit.client.0.smithi136.stdout:00000020 78 78 02 00 00 00 00 00 00 00 af 75 d0 85 00 00 |xx.........u....| 2024-09-15T20:21:56.066 INFO:tasks.workunit.client.0.smithi136.stdout:00000030 00 00 00 01 00 00 00 00 00 00 00 |...........| 2024-09-15T20:21:56.066 INFO:tasks.workunit.client.0.smithi136.stdout:0000003b 2024-09-15T20:21:56.066 INFO:tasks.workunit.client.0.smithi136.stdout: 2024-09-15T20:21:56.066 INFO:tasks.workunit.client.0.smithi136.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.FA570D58.3.objxxx.. 2024-09-15T20:21:56.066 INFO:tasks.workunit.client.0.smithi136.stdout:00000000 01 01 35 00 00 00 03 00 00 00 00 00 00 00 04 03 |..5.............| 2024-09-15T20:21:56.066 INFO:tasks.workunit.client.0.smithi136.stdout:00000010 27 00 00 00 00 00 00 00 06 00 00 00 6f 62 6a 78 |'...........objx| 2024-09-15T20:21:56.066 INFO:tasks.workunit.client.0.smithi136.stdout:00000020 78 78 03 00 00 00 00 00 00 00 af 75 d0 85 00 00 |xx.........u....| 2024-09-15T20:21:56.067 INFO:tasks.workunit.client.0.smithi136.stdout:00000030 00 00 00 01 00 00 00 00 00 00 00 |...........| 2024-09-15T20:21:56.067 INFO:tasks.workunit.client.0.smithi136.stdout:0000003b 2024-09-15T20:21:56.067 INFO:tasks.workunit.client.0.smithi136.stdout: 2024-09-15T20:21:56.067 INFO:tasks.workunit.client.0.smithi136.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000004_0000000000000001.FA570D58.5.objxxx.. 2024-09-15T20:21:56.067 INFO:tasks.workunit.client.0.smithi136.stdout:00000000 01 01 35 00 00 00 04 00 00 00 00 00 00 00 04 03 |..5.............| 2024-09-15T20:21:56.067 INFO:tasks.workunit.client.0.smithi136.stdout:00000010 27 00 00 00 00 00 00 00 06 00 00 00 6f 62 6a 78 |'...........objx| 2024-09-15T20:21:56.067 INFO:tasks.workunit.client.0.smithi136.stdout:00000020 78 78 05 00 00 00 00 00 00 00 af 75 d0 85 00 00 |xx.........u....| 2024-09-15T20:21:56.067 INFO:tasks.workunit.client.0.smithi136.stdout:00000030 00 00 00 01 00 00 00 00 00 00 00 |...........| 2024-09-15T20:21:56.067 INFO:tasks.workunit.client.0.smithi136.stdout:0000003b 2024-09-15T20:21:56.068 INFO:tasks.workunit.client.0.smithi136.stdout: 2024-09-15T20:21:56.068 INFO:tasks.workunit.client.0.smithi136.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000005_0000000000000001.FA570D58.5.objxxx.. 2024-09-15T20:21:56.068 INFO:tasks.workunit.client.0.smithi136.stdout:00000000 01 01 35 00 00 00 05 00 00 00 00 00 00 00 04 03 |..5.............| 2024-09-15T20:21:56.068 INFO:tasks.workunit.client.0.smithi136.stdout:00000010 27 00 00 00 00 00 00 00 06 00 00 00 6f 62 6a 78 |'...........objx| 2024-09-15T20:21:56.068 INFO:tasks.workunit.client.0.smithi136.stdout:00000020 78 78 05 00 00 00 00 00 00 00 af 75 d0 85 00 00 |xx.........u....| 2024-09-15T20:21:56.068 INFO:tasks.workunit.client.0.smithi136.stdout:00000030 00 00 00 01 00 00 00 00 00 00 00 |...........| 2024-09-15T20:21:56.068 INFO:tasks.workunit.client.0.smithi136.stdout:0000003b 2024-09-15T20:21:56.069 INFO:tasks.workunit.client.0.smithi136.stdout: 2024-09-15T20:21:56.069 INFO:tasks.workunit.client.0.smithi136.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02~ 2024-09-15T20:21:56.069 INFO:tasks.workunit.client.0.smithi136.stdout: 2024-09-15T20:21:56.069 INFO:tasks.workunit.client.0.smithi136.stdout:p %00%00%00%00%00%00%00%00%c1%a3%fcn%00%00%00%00%00%00%04%03~ 2024-09-15T20:21:56.069 INFO:tasks.workunit.client.0.smithi136.stdout: 2024-09-15T20:21:56.069 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:21:56.054+0000 7f1e3d4f1980 1 bluefs umount 2024-09-15T20:21:56.069 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:21:56.054+0000 7f1e3d4f1980 1 bdev(0x55d448ab2400 td/osd-mapper/0/block) close 2024-09-15T20:21:56.274 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:21:56.273+0000 7f1e3d4f1980 1 freelist shutdown 2024-09-15T20:21:56.275 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:21:56.273+0000 7f1e3d4f1980 1 bdev(0x55d448937c00 td/osd-mapper/0/block) close 2024-09-15T20:21:56.513 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:114: TEST_truncated_sna_record: ceph-kvstore-tool bluestore-kv td/osd-mapper/0 dump p 2024-09-15T20:21:56.513 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:114: TEST_truncated_sna_record: grep -a -e 'SNA_[0-9]_0000000000000003_000000000000000' 2024-09-15T20:21:56.513 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:114: TEST_truncated_sna_record: awk -e '{print $2;}' 2024-09-15T20:21:57.578 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:114: TEST_truncated_sna_record: KY=%00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.FA570D58.3.objxxx.. 2024-09-15T20:21:57.578 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:115: TEST_truncated_sna_record: (( extr_dbg >= 1 )) 2024-09-15T20:21:57.578 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:115: TEST_truncated_sna_record: echo 'SNA key: %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.FA570D58.3.objxxx..' 2024-09-15T20:21:57.578 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:115: TEST_truncated_sna_record: cat -v 2024-09-15T20:21:57.579 INFO:tasks.workunit.client.0.smithi136.stdout:SNA key: %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.FA570D58.3.objxxx.. 2024-09-15T20:21:57.580 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:117: TEST_truncated_sna_record: mktemp -p /tmp --suffix=_the_val 2024-09-15T20:21:57.580 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:117: TEST_truncated_sna_record: tmp_fn1=/tmp/tmp.vYWP8IfLxP_the_val 2024-09-15T20:21:57.580 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:118: TEST_truncated_sna_record: (( extr_dbg >= 1 )) 2024-09-15T20:21:57.580 INFO:tasks.workunit.client.0.smithi136.stdout:Value dumped in: /tmp/tmp.vYWP8IfLxP_the_val 2024-09-15T20:21:57.600 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:118: TEST_truncated_sna_record: echo 'Value dumped in: /tmp/tmp.vYWP8IfLxP_the_val' 2024-09-15T20:21:57.606 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:119: TEST_truncated_sna_record: ceph-kvstore-tool bluestore-kv td/osd-mapper/0 get p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.FA570D58.3.objxxx.. out /tmp/tmp.vYWP8IfLxP_the_val 2024-09-15T20:21:58.187 INFO:tasks.workunit.client.0.smithi136.stdout:(p, %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.FA570D58.3.objxxx..) 2024-09-15T20:21:58.640 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:120: TEST_truncated_sna_record: (( extr_dbg >= 2 )) 2024-09-15T20:21:58.640 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:120: TEST_truncated_sna_record: od -xc /tmp/tmp.vYWP8IfLxP_the_val 2024-09-15T20:21:58.665 INFO:tasks.workunit.client.0.smithi136.stdout:0000000 0101 0035 0000 0003 0000 0000 0000 0304 2024-09-15T20:21:58.665 INFO:tasks.workunit.client.0.smithi136.stdout: 001 001 5 \0 \0 \0 003 \0 \0 \0 \0 \0 \0 \0 004 003 2024-09-15T20:21:58.665 INFO:tasks.workunit.client.0.smithi136.stdout:0000020 0027 0000 0000 0000 0006 0000 626f 786a 2024-09-15T20:21:58.665 INFO:tasks.workunit.client.0.smithi136.stdout: ' \0 \0 \0 \0 \0 \0 \0 006 \0 \0 \0 o b j x 2024-09-15T20:21:58.665 INFO:tasks.workunit.client.0.smithi136.stdout:0000040 7878 0003 0000 0000 0000 75af 85d0 0000 2024-09-15T20:21:58.665 INFO:tasks.workunit.client.0.smithi136.stdout: x x 003 \0 \0 \0 \0 \0 \0 \0 257 u 320 205 \0 \0 2024-09-15T20:21:58.665 INFO:tasks.workunit.client.0.smithi136.stdout:0000060 0000 0100 0000 0000 0000 0000 2024-09-15T20:21:58.665 INFO:tasks.workunit.client.0.smithi136.stdout: \0 \0 \0 001 \0 \0 \0 \0 \0 \0 \0 2024-09-15T20:21:58.665 INFO:tasks.workunit.client.0.smithi136.stdout:0000073 2024-09-15T20:21:58.666 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:122: TEST_truncated_sna_record: NKY=%00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_000000 2024-09-15T20:21:58.666 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:123: TEST_truncated_sna_record: ceph-kvstore-tool bluestore-kv td/osd-mapper/0 rm p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.FA570D58.3.objxxx.. 2024-09-15T20:22:00.476 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:124: TEST_truncated_sna_record: ceph-kvstore-tool bluestore-kv td/osd-mapper/0 set p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_000000 in /tmp/tmp.vYWP8IfLxP_the_val 2024-09-15T20:22:01.783 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:126: TEST_truncated_sna_record: (( extr_dbg >= 1 )) 2024-09-15T20:22:01.783 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:106: TEST_truncated_sna_record: for sdn in $(seq 0 $(expr $osdn - 1)) 2024-09-15T20:22:01.784 INFO:tasks.workunit.client.0.smithi136.stdout:corrupting the SnapMapper DB of osd.1 (db: td/osd-mapper/1) 2024-09-15T20:22:01.784 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:108: TEST_truncated_sna_record: kvdir=td/osd-mapper/1 2024-09-15T20:22:01.784 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:109: TEST_truncated_sna_record: echo 'corrupting the SnapMapper DB of osd.1 (db: td/osd-mapper/1)' 2024-09-15T20:22:01.785 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:110: TEST_truncated_sna_record: (( extr_dbg >= 3 )) 2024-09-15T20:22:01.785 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:110: TEST_truncated_sna_record: ceph-kvstore-tool bluestore-kv td/osd-mapper/1 dump p 2024-09-15T20:22:01.810 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:22:01.808+0000 7f8ca6b0e980 1 bdev(0x55740ebd1c00 td/osd-mapper/1/block) open path td/osd-mapper/1/block 2024-09-15T20:22:01.810 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:22:01.808+0000 7f8ca6b0e980 0 bdev(0x55740ebd1c00 td/osd-mapper/1/block) ioctl(F_SET_FILE_RW_HINT) on td/osd-mapper/1/block failed: (22) Invalid argument 2024-09-15T20:22:01.811 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:22:01.808+0000 7f8ca6b0e980 1 bdev(0x55740ebd1c00 td/osd-mapper/1/block) open size 107374182400 (0x1900000000, 100 GiB) block_size 4096 (4 KiB) rotational device, discard supported 2024-09-15T20:22:01.811 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:22:01.810+0000 7f8ca6b0e980 1 bdev(0x55740ebd1c00 td/osd-mapper/1/block) close 2024-09-15T20:22:02.076 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:22:02.074+0000 7f8ca6b0e980 1 bdev(0x55740ebd1c00 td/osd-mapper/1/block) open path td/osd-mapper/1/block 2024-09-15T20:22:02.076 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:22:02.074+0000 7f8ca6b0e980 0 bdev(0x55740ebd1c00 td/osd-mapper/1/block) ioctl(F_SET_FILE_RW_HINT) on td/osd-mapper/1/block failed: (22) Invalid argument 2024-09-15T20:22:02.077 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:22:02.074+0000 7f8ca6b0e980 1 bdev(0x55740ebd1c00 td/osd-mapper/1/block) open size 107374182400 (0x1900000000, 100 GiB) block_size 4096 (4 KiB) rotational device, discard supported 2024-09-15T20:22:02.084 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:22:02.081+0000 7f8ca6b0e980 1 bluestore(td/osd-mapper/1) _set_cache_sizes cache_size 1073741824 meta 0.45 kv 0.45 kv_onode 0.04 data 0.06 2024-09-15T20:22:02.084 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:22:02.082+0000 7f8ca6b0e980 1 bdev(0x55740ed4c400 td/osd-mapper/1/block) open path td/osd-mapper/1/block 2024-09-15T20:22:02.084 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:22:02.082+0000 7f8ca6b0e980 0 bdev(0x55740ed4c400 td/osd-mapper/1/block) ioctl(F_SET_FILE_RW_HINT) on td/osd-mapper/1/block failed: (22) Invalid argument 2024-09-15T20:22:02.084 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:22:02.082+0000 7f8ca6b0e980 1 bdev(0x55740ed4c400 td/osd-mapper/1/block) open size 107374182400 (0x1900000000, 100 GiB) block_size 4096 (4 KiB) rotational device, discard supported 2024-09-15T20:22:02.084 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:22:02.082+0000 7f8ca6b0e980 1 bluefs add_block_device bdev 1 path td/osd-mapper/1/block size 100 GiB 2024-09-15T20:22:02.084 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:22:02.082+0000 7f8ca6b0e980 1 bluefs mount 2024-09-15T20:22:02.084 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:22:02.082+0000 7f8ca6b0e980 1 bluefs _init_alloc shared, id 1, capacity 0x1900000000, block size 0x10000 2024-09-15T20:22:02.091 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:22:02.089+0000 7f8ca6b0e980 1 bluefs mount shared_bdev_used = 0 2024-09-15T20:22:02.091 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:22:02.089+0000 7f8ca6b0e980 1 bluestore(td/osd-mapper/1) _prepare_db_environment set db_paths to db,102005473280 db.slow,102005473280 2024-09-15T20:22:02.129 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:22:02.127+0000 7f8ca6b0e980 1 bluestore(td/osd-mapper/1) _open_db opened rocksdb path db options compression=kLZ4Compression,max_write_buffer_number=64,min_write_buffer_number_to_merge=6,compaction_style=kCompactionStyleLevel,write_buffer_size=16777216,max_background_jobs=4,level0_file_num_compaction_trigger=8,max_bytes_for_level_base=1073741824,max_bytes_for_level_multiplier=8,compaction_readahead_size=2MB,max_total_wal_size=1073741824,writable_file_max_buffer_size=0 2024-09-15T20:22:02.129 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:22:02.127+0000 7f8ca6b0e980 1 bluestore(td/osd-mapper/1) _open_super_meta old nid_max 2050 2024-09-15T20:22:02.129 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:22:02.127+0000 7f8ca6b0e980 1 bluestore(td/osd-mapper/1) _open_super_meta old blobid_max 20480 2024-09-15T20:22:02.129 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:22:02.127+0000 7f8ca6b0e980 1 bluestore(td/osd-mapper/1) _open_super_meta ondisk_format 4 compat_ondisk_format 3 2024-09-15T20:22:02.129 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:22:02.127+0000 7f8ca6b0e980 1 bluestore(td/osd-mapper/1) _open_super_meta min_alloc_size 0x1000 2024-09-15T20:22:02.129 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:22:02.127+0000 7f8ca6b0e980 1 freelist init 2024-09-15T20:22:02.129 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:22:02.127+0000 7f8ca6b0e980 1 freelist _read_cfg 2024-09-15T20:22:02.130 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:22:02.127+0000 7f8ca6b0e980 1 bluestore(td/osd-mapper/1) _open_fm effective freelist_type = bitmap, freelist_alloc_size = 0x1000, min_alloc_size = 0x1000 2024-09-15T20:22:02.130 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:22:02.127+0000 7f8ca6b0e980 1 bluestore(td/osd-mapper/1) _init_alloc loaded 100 GiB in 19 extents, allocator type hybrid, capacity 0x1900000000, block size 0x1000, free 0x18fffbc000, fragmentation 6.86647e-07 2024-09-15T20:22:02.130 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:22:02.128+0000 7f8ca6b0e980 1 bluefs umount 2024-09-15T20:22:02.130 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:22:02.128+0000 7f8ca6b0e980 1 bdev(0x55740ed4c400 td/osd-mapper/1/block) close 2024-09-15T20:22:02.346 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:22:02.344+0000 7f8ca6b0e980 1 bdev(0x55740ed4c400 td/osd-mapper/1/block) open path td/osd-mapper/1/block 2024-09-15T20:22:02.346 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:22:02.344+0000 7f8ca6b0e980 0 bdev(0x55740ed4c400 td/osd-mapper/1/block) ioctl(F_SET_FILE_RW_HINT) on td/osd-mapper/1/block failed: (22) Invalid argument 2024-09-15T20:22:02.346 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:22:02.344+0000 7f8ca6b0e980 1 bdev(0x55740ed4c400 td/osd-mapper/1/block) open size 107374182400 (0x1900000000, 100 GiB) block_size 4096 (4 KiB) rotational device, discard supported 2024-09-15T20:22:02.346 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:22:02.344+0000 7f8ca6b0e980 1 bluefs add_block_device bdev 1 path td/osd-mapper/1/block size 100 GiB 2024-09-15T20:22:02.346 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:22:02.344+0000 7f8ca6b0e980 1 bluefs mount 2024-09-15T20:22:02.346 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:22:02.344+0000 7f8ca6b0e980 1 bluefs _init_alloc shared, id 1, capacity 0x1900000000, block size 0x10000 2024-09-15T20:22:02.352 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:22:02.350+0000 7f8ca6b0e980 1 bluefs mount shared_bdev_used = 27459584 2024-09-15T20:22:02.353 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:22:02.350+0000 7f8ca6b0e980 1 bluestore(td/osd-mapper/1) _prepare_db_environment set db_paths to db,102005473280 db.slow,102005473280 2024-09-15T20:22:02.383 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:22:02.381+0000 7f8ca6b0e980 1 bluestore(td/osd-mapper/1) _open_db opened rocksdb path db options compression=kLZ4Compression,max_write_buffer_number=64,min_write_buffer_number_to_merge=6,compaction_style=kCompactionStyleLevel,write_buffer_size=16777216,max_background_jobs=4,level0_file_num_compaction_trigger=8,max_bytes_for_level_base=1073741824,max_bytes_for_level_multiplier=8,compaction_readahead_size=2MB,max_total_wal_size=1073741824,writable_file_max_buffer_size=0 2024-09-15T20:22:02.385 INFO:tasks.workunit.client.0.smithi136.stdout:p %00%00%00%00%00%00%00%00%7b%3fC%c4%00%00%00%00%00%00%00%01.osd_superblock 2024-09-15T20:22:02.385 INFO:tasks.workunit.client.0.smithi136.stdout:00000000 0b 05 58 02 00 00 e9 f4 cc 96 8f 95 49 71 84 e5 |..X.........Iq..| 2024-09-15T20:22:02.385 INFO:tasks.workunit.client.0.smithi136.stdout:00000010 d6 4a cd 70 df 0f 01 00 00 00 1e 00 00 00 00 00 |.J.p............| 2024-09-15T20:22:02.385 INFO:tasks.workunit.client.0.smithi136.stdout:00000020 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................| 2024-09-15T20:22:02.385 INFO:tasks.workunit.client.0.smithi136.stdout:* 2024-09-15T20:22:02.385 INFO:tasks.workunit.client.0.smithi136.stdout:00000040 00 00 00 00 00 00 fe ff 03 00 00 00 00 00 11 00 |................| 2024-09-15T20:22:02.385 INFO:tasks.workunit.client.0.smithi136.stdout:00000050 00 00 01 00 00 00 00 00 00 00 1a 00 00 00 69 6e |..............in| 2024-09-15T20:22:02.385 INFO:tasks.workunit.client.0.smithi136.stdout:00000060 69 74 69 61 6c 20 66 65 61 74 75 72 65 20 73 65 |itial feature se| 2024-09-15T20:22:02.386 INFO:tasks.workunit.client.0.smithi136.stdout:00000070 74 28 7e 76 2e 31 38 29 02 00 00 00 00 00 00 00 |t(~v.18)........| 2024-09-15T20:22:02.386 INFO:tasks.workunit.client.0.smithi136.stdout:00000080 0d 00 00 00 70 67 69 6e 66 6f 20 6f 62 6a 65 63 |....pginfo objec| 2024-09-15T20:22:02.386 INFO:tasks.workunit.client.0.smithi136.stdout:00000090 74 03 00 00 00 00 00 00 00 0e 00 00 00 6f 62 6a |t............obj| 2024-09-15T20:22:02.386 INFO:tasks.workunit.client.0.smithi136.stdout:000000a0 65 63 74 20 6c 6f 63 61 74 6f 72 04 00 00 00 00 |ect locator.....| 2024-09-15T20:22:02.386 INFO:tasks.workunit.client.0.smithi136.stdout:000000b0 00 00 00 10 00 00 00 6c 61 73 74 5f 65 70 6f 63 |.......last_epoc| 2024-09-15T20:22:02.386 INFO:tasks.workunit.client.0.smithi136.stdout:000000c0 68 5f 63 6c 65 61 6e 05 00 00 00 00 00 00 00 0a |h_clean.........| 2024-09-15T20:22:02.386 INFO:tasks.workunit.client.0.smithi136.stdout:000000d0 00 00 00 63 61 74 65 67 6f 72 69 65 73 06 00 00 |...categories...| 2024-09-15T20:22:02.386 INFO:tasks.workunit.client.0.smithi136.stdout:000000e0 00 00 00 00 00 0b 00 00 00 68 6f 62 6a 65 63 74 |.........hobject| 2024-09-15T20:22:02.386 INFO:tasks.workunit.client.0.smithi136.stdout:000000f0 70 6f 6f 6c 07 00 00 00 00 00 00 00 07 00 00 00 |pool............| 2024-09-15T20:22:02.386 INFO:tasks.workunit.client.0.smithi136.stdout:00000100 62 69 67 69 6e 66 6f 08 00 00 00 00 00 00 00 0b |biginfo.........| 2024-09-15T20:22:02.386 INFO:tasks.workunit.client.0.smithi136.stdout:00000110 00 00 00 6c 65 76 65 6c 64 62 69 6e 66 6f 09 00 |...leveldbinfo..| 2024-09-15T20:22:02.386 INFO:tasks.workunit.client.0.smithi136.stdout:00000120 00 00 00 00 00 00 0a 00 00 00 6c 65 76 65 6c 64 |..........leveld| 2024-09-15T20:22:02.388 INFO:tasks.workunit.client.0.smithi136.stdout:00000130 62 6c 6f 67 0a 00 00 00 00 00 00 00 0a 00 00 00 |blog............| 2024-09-15T20:22:02.388 INFO:tasks.workunit.client.0.smithi136.stdout:00000140 73 6e 61 70 6d 61 70 70 65 72 0b 00 00 00 00 00 |snapmapper......| 2024-09-15T20:22:02.389 INFO:tasks.workunit.client.0.smithi136.stdout:00000150 00 00 0f 00 00 00 73 68 61 72 64 65 64 20 6f 62 |......sharded ob| 2024-09-15T20:22:02.389 INFO:tasks.workunit.client.0.smithi136.stdout:00000160 6a 65 63 74 73 0c 00 00 00 00 00 00 00 11 00 00 |jects...........| 2024-09-15T20:22:02.389 INFO:tasks.workunit.client.0.smithi136.stdout:00000170 00 74 72 61 6e 73 61 63 74 69 6f 6e 20 68 69 6e |.transaction hin| 2024-09-15T20:22:02.389 INFO:tasks.workunit.client.0.smithi136.stdout:00000180 74 73 0d 00 00 00 00 00 00 00 0e 00 00 00 70 67 |ts............pg| 2024-09-15T20:22:02.389 INFO:tasks.workunit.client.0.smithi136.stdout:00000190 20 6d 65 74 61 20 6f 62 6a 65 63 74 0e 00 00 00 | meta object....| 2024-09-15T20:22:02.389 INFO:tasks.workunit.client.0.smithi136.stdout:000001a0 00 00 00 00 14 00 00 00 65 78 70 6c 69 63 69 74 |........explicit| 2024-09-15T20:22:02.389 INFO:tasks.workunit.client.0.smithi136.stdout:000001b0 20 6d 69 73 73 69 6e 67 20 73 65 74 0f 00 00 00 | missing set....| 2024-09-15T20:22:02.390 INFO:tasks.workunit.client.0.smithi136.stdout:000001c0 00 00 00 00 10 00 00 00 66 61 73 74 69 6e 66 6f |........fastinfo| 2024-09-15T20:22:02.390 INFO:tasks.workunit.client.0.smithi136.stdout:000001d0 20 70 67 20 61 74 74 72 10 00 00 00 00 00 00 00 | pg attr........| 2024-09-15T20:22:02.390 INFO:tasks.workunit.client.0.smithi136.stdout:000001e0 16 00 00 00 64 65 6c 65 74 65 73 20 69 6e 20 6d |....deletes in m| 2024-09-15T20:22:02.390 INFO:tasks.workunit.client.0.smithi136.stdout:000001f0 69 73 73 69 6e 67 20 73 65 74 11 00 00 00 00 00 |issing set......| 2024-09-15T20:22:02.390 INFO:tasks.workunit.client.0.smithi136.stdout:00000200 00 00 1c 00 00 00 6e 65 77 20 73 6e 61 70 6d 61 |......new snapma| 2024-09-15T20:22:02.390 INFO:tasks.workunit.client.0.smithi136.stdout:00000210 70 70 65 72 20 6b 65 79 20 73 74 72 75 63 74 75 |pper key structu| 2024-09-15T20:22:02.390 INFO:tasks.workunit.client.0.smithi136.stdout:00000220 72 65 1e 00 00 00 0a 00 00 00 aa 3a 94 d1 c1 80 |re.........:....| 2024-09-15T20:22:02.390 INFO:tasks.workunit.client.0.smithi136.stdout:00000230 45 6f 8f 8b c7 5b 0a 4a 14 77 00 00 00 00 00 00 |Eo...[.J.w......| 2024-09-15T20:22:02.390 INFO:tasks.workunit.client.0.smithi136.stdout:00000240 00 00 1e 00 00 00 a7 41 e7 66 7e 36 30 03 01 00 |.......A.f~60...| 2024-09-15T20:22:02.390 INFO:tasks.workunit.client.0.smithi136.stdout:00000250 00 00 01 00 00 00 01 00 00 00 1e 00 00 00 |..............| 2024-09-15T20:22:02.390 INFO:tasks.workunit.client.0.smithi136.stdout:0000025e 2024-09-15T20:22:02.390 INFO:tasks.workunit.client.0.smithi136.stdout: 2024-09-15T20:22:02.390 INFO:tasks.workunit.client.0.smithi136.stdout:p %00%00%00%00%00%00%00%00%7b%3fC%c4%00%00%00%00%00%00%00%01~ 2024-09-15T20:22:02.390 INFO:tasks.workunit.client.0.smithi136.stdout: 2024-09-15T20:22:02.391 INFO:tasks.workunit.client.0.smithi136.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.OBJ_0000000000000001.FA570D58.2.objxxx.. 2024-09-15T20:22:02.391 INFO:tasks.workunit.client.0.smithi136.stdout:00000000 01 01 41 00 00 00 04 03 27 00 00 00 00 00 00 00 |..A.....'.......| 2024-09-15T20:22:02.391 INFO:tasks.workunit.client.0.smithi136.stdout:00000010 06 00 00 00 6f 62 6a 78 78 78 02 00 00 00 00 00 |....objxxx......| 2024-09-15T20:22:02.391 INFO:tasks.workunit.client.0.smithi136.stdout:00000020 00 00 af 75 d0 85 00 00 00 00 00 01 00 00 00 00 |...u............| 2024-09-15T20:22:02.391 INFO:tasks.workunit.client.0.smithi136.stdout:00000030 00 00 00 02 00 00 00 01 00 00 00 00 00 00 00 02 |................| 2024-09-15T20:22:02.391 INFO:tasks.workunit.client.0.smithi136.stdout:00000040 00 00 00 00 00 00 00 |.......| 2024-09-15T20:22:02.391 INFO:tasks.workunit.client.0.smithi136.stdout:00000047 2024-09-15T20:22:02.391 INFO:tasks.workunit.client.0.smithi136.stdout: 2024-09-15T20:22:02.391 INFO:tasks.workunit.client.0.smithi136.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.OBJ_0000000000000001.FA570D58.3.objxxx.. 2024-09-15T20:22:02.391 INFO:tasks.workunit.client.0.smithi136.stdout:00000000 01 01 39 00 00 00 04 03 27 00 00 00 00 00 00 00 |..9.....'.......| 2024-09-15T20:22:02.391 INFO:tasks.workunit.client.0.smithi136.stdout:00000010 06 00 00 00 6f 62 6a 78 78 78 03 00 00 00 00 00 |....objxxx......| 2024-09-15T20:22:02.391 INFO:tasks.workunit.client.0.smithi136.stdout:00000020 00 00 af 75 d0 85 00 00 00 00 00 01 00 00 00 00 |...u............| 2024-09-15T20:22:02.392 INFO:tasks.workunit.client.0.smithi136.stdout:00000030 00 00 00 01 00 00 00 03 00 00 00 00 00 00 00 |...............| 2024-09-15T20:22:02.392 INFO:tasks.workunit.client.0.smithi136.stdout:0000003f 2024-09-15T20:22:02.392 INFO:tasks.workunit.client.0.smithi136.stdout: 2024-09-15T20:22:02.392 INFO:tasks.workunit.client.0.smithi136.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.OBJ_0000000000000001.FA570D58.5.objxxx.. 2024-09-15T20:22:02.392 INFO:tasks.workunit.client.0.smithi136.stdout:00000000 01 01 41 00 00 00 04 03 27 00 00 00 00 00 00 00 |..A.....'.......| 2024-09-15T20:22:02.392 INFO:tasks.workunit.client.0.smithi136.stdout:00000010 06 00 00 00 6f 62 6a 78 78 78 05 00 00 00 00 00 |....objxxx......| 2024-09-15T20:22:02.392 INFO:tasks.workunit.client.0.smithi136.stdout:00000020 00 00 af 75 d0 85 00 00 00 00 00 01 00 00 00 00 |...u............| 2024-09-15T20:22:02.392 INFO:tasks.workunit.client.0.smithi136.stdout:00000030 00 00 00 02 00 00 00 04 00 00 00 00 00 00 00 05 |................| 2024-09-15T20:22:02.392 INFO:tasks.workunit.client.0.smithi136.stdout:00000040 00 00 00 00 00 00 00 |.......| 2024-09-15T20:22:02.392 INFO:tasks.workunit.client.0.smithi136.stdout:00000047 2024-09-15T20:22:02.392 INFO:tasks.workunit.client.0.smithi136.stdout: 2024-09-15T20:22:02.392 INFO:tasks.workunit.client.0.smithi136.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.FA570D58.2.objxxx.. 2024-09-15T20:22:02.392 INFO:tasks.workunit.client.0.smithi136.stdout:00000000 01 01 35 00 00 00 01 00 00 00 00 00 00 00 04 03 |..5.............| 2024-09-15T20:22:02.393 INFO:tasks.workunit.client.0.smithi136.stdout:00000010 27 00 00 00 00 00 00 00 06 00 00 00 6f 62 6a 78 |'...........objx| 2024-09-15T20:22:02.393 INFO:tasks.workunit.client.0.smithi136.stdout:00000020 78 78 02 00 00 00 00 00 00 00 af 75 d0 85 00 00 |xx.........u....| 2024-09-15T20:22:02.393 INFO:tasks.workunit.client.0.smithi136.stdout:00000030 00 00 00 01 00 00 00 00 00 00 00 |...........| 2024-09-15T20:22:02.393 INFO:tasks.workunit.client.0.smithi136.stdout:0000003b 2024-09-15T20:22:02.393 INFO:tasks.workunit.client.0.smithi136.stdout: 2024-09-15T20:22:02.393 INFO:tasks.workunit.client.0.smithi136.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000002_0000000000000001.FA570D58.2.objxxx.. 2024-09-15T20:22:02.393 INFO:tasks.workunit.client.0.smithi136.stdout:00000000 01 01 35 00 00 00 02 00 00 00 00 00 00 00 04 03 |..5.............| 2024-09-15T20:22:02.393 INFO:tasks.workunit.client.0.smithi136.stdout:00000010 27 00 00 00 00 00 00 00 06 00 00 00 6f 62 6a 78 |'...........objx| 2024-09-15T20:22:02.393 INFO:tasks.workunit.client.0.smithi136.stdout:00000020 78 78 02 00 00 00 00 00 00 00 af 75 d0 85 00 00 |xx.........u....| 2024-09-15T20:22:02.393 INFO:tasks.workunit.client.0.smithi136.stdout:00000030 00 00 00 01 00 00 00 00 00 00 00 |...........| 2024-09-15T20:22:02.393 INFO:tasks.workunit.client.0.smithi136.stdout:0000003b 2024-09-15T20:22:02.394 INFO:tasks.workunit.client.0.smithi136.stdout: 2024-09-15T20:22:02.394 INFO:tasks.workunit.client.0.smithi136.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.FA570D58.3.objxxx.. 2024-09-15T20:22:02.394 INFO:tasks.workunit.client.0.smithi136.stdout:00000000 01 01 35 00 00 00 03 00 00 00 00 00 00 00 04 03 |..5.............| 2024-09-15T20:22:02.394 INFO:tasks.workunit.client.0.smithi136.stdout:00000010 27 00 00 00 00 00 00 00 06 00 00 00 6f 62 6a 78 |'...........objx| 2024-09-15T20:22:02.394 INFO:tasks.workunit.client.0.smithi136.stdout:00000020 78 78 03 00 00 00 00 00 00 00 af 75 d0 85 00 00 |xx.........u....| 2024-09-15T20:22:02.394 INFO:tasks.workunit.client.0.smithi136.stdout:00000030 00 00 00 01 00 00 00 00 00 00 00 |...........| 2024-09-15T20:22:02.394 INFO:tasks.workunit.client.0.smithi136.stdout:0000003b 2024-09-15T20:22:02.394 INFO:tasks.workunit.client.0.smithi136.stdout: 2024-09-15T20:22:02.394 INFO:tasks.workunit.client.0.smithi136.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000004_0000000000000001.FA570D58.5.objxxx.. 2024-09-15T20:22:02.395 INFO:tasks.workunit.client.0.smithi136.stdout:00000000 01 01 35 00 00 00 04 00 00 00 00 00 00 00 04 03 |..5.............| 2024-09-15T20:22:02.395 INFO:tasks.workunit.client.0.smithi136.stdout:00000010 27 00 00 00 00 00 00 00 06 00 00 00 6f 62 6a 78 |'...........objx| 2024-09-15T20:22:02.395 INFO:tasks.workunit.client.0.smithi136.stdout:00000020 78 78 05 00 00 00 00 00 00 00 af 75 d0 85 00 00 |xx.........u....| 2024-09-15T20:22:02.395 INFO:tasks.workunit.client.0.smithi136.stdout:00000030 00 00 00 01 00 00 00 00 00 00 00 |...........| 2024-09-15T20:22:02.395 INFO:tasks.workunit.client.0.smithi136.stdout:0000003b 2024-09-15T20:22:02.395 INFO:tasks.workunit.client.0.smithi136.stdout: 2024-09-15T20:22:02.395 INFO:tasks.workunit.client.0.smithi136.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000005_0000000000000001.FA570D58.5.objxxx.. 2024-09-15T20:22:02.395 INFO:tasks.workunit.client.0.smithi136.stdout:00000000 01 01 35 00 00 00 05 00 00 00 00 00 00 00 04 03 |..5.............| 2024-09-15T20:22:02.395 INFO:tasks.workunit.client.0.smithi136.stdout:00000010 27 00 00 00 00 00 00 00 06 00 00 00 6f 62 6a 78 |'...........objx| 2024-09-15T20:22:02.395 INFO:tasks.workunit.client.0.smithi136.stdout:00000020 78 78 05 00 00 00 00 00 00 00 af 75 d0 85 00 00 |xx.........u....| 2024-09-15T20:22:02.395 INFO:tasks.workunit.client.0.smithi136.stdout:00000030 00 00 00 01 00 00 00 00 00 00 00 |...........| 2024-09-15T20:22:02.395 INFO:tasks.workunit.client.0.smithi136.stdout:0000003b 2024-09-15T20:22:02.395 INFO:tasks.workunit.client.0.smithi136.stdout: 2024-09-15T20:22:02.395 INFO:tasks.workunit.client.0.smithi136.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02~ 2024-09-15T20:22:02.396 INFO:tasks.workunit.client.0.smithi136.stdout: 2024-09-15T20:22:02.396 INFO:tasks.workunit.client.0.smithi136.stdout:p %00%00%00%00%00%00%00%00%c1%a3%fcn%00%00%00%00%00%00%04%03~ 2024-09-15T20:22:02.396 INFO:tasks.workunit.client.0.smithi136.stdout: 2024-09-15T20:22:02.396 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:22:02.381+0000 7f8ca6b0e980 1 bluefs umount 2024-09-15T20:22:02.396 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:22:02.381+0000 7f8ca6b0e980 1 bdev(0x55740ed4c400 td/osd-mapper/1/block) close 2024-09-15T20:22:02.607 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:22:02.606+0000 7f8ca6b0e980 1 freelist shutdown 2024-09-15T20:22:02.608 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:22:02.606+0000 7f8ca6b0e980 1 bdev(0x55740ebd1c00 td/osd-mapper/1/block) close 2024-09-15T20:22:02.847 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:114: TEST_truncated_sna_record: ceph-kvstore-tool bluestore-kv td/osd-mapper/1 dump p 2024-09-15T20:22:02.847 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:114: TEST_truncated_sna_record: grep -a -e 'SNA_[0-9]_0000000000000003_000000000000000' 2024-09-15T20:22:02.847 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:114: TEST_truncated_sna_record: awk -e '{print $2;}' 2024-09-15T20:22:03.899 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:114: TEST_truncated_sna_record: KY=%00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.FA570D58.3.objxxx.. 2024-09-15T20:22:03.899 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:115: TEST_truncated_sna_record: (( extr_dbg >= 1 )) 2024-09-15T20:22:03.899 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:115: TEST_truncated_sna_record: echo 'SNA key: %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.FA570D58.3.objxxx..' 2024-09-15T20:22:03.900 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:115: TEST_truncated_sna_record: cat -v 2024-09-15T20:22:03.900 INFO:tasks.workunit.client.0.smithi136.stdout:SNA key: %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.FA570D58.3.objxxx.. 2024-09-15T20:22:03.900 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:117: TEST_truncated_sna_record: mktemp -p /tmp --suffix=_the_val 2024-09-15T20:22:03.906 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:117: TEST_truncated_sna_record: tmp_fn1=/tmp/tmp.AUiveHT4vh_the_val 2024-09-15T20:22:03.906 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:118: TEST_truncated_sna_record: (( extr_dbg >= 1 )) 2024-09-15T20:22:03.906 INFO:tasks.workunit.client.0.smithi136.stdout:Value dumped in: /tmp/tmp.AUiveHT4vh_the_val 2024-09-15T20:22:03.907 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:118: TEST_truncated_sna_record: echo 'Value dumped in: /tmp/tmp.AUiveHT4vh_the_val' 2024-09-15T20:22:03.907 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:119: TEST_truncated_sna_record: ceph-kvstore-tool bluestore-kv td/osd-mapper/1 get p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.FA570D58.3.objxxx.. out /tmp/tmp.AUiveHT4vh_the_val 2024-09-15T20:22:04.498 INFO:tasks.workunit.client.0.smithi136.stdout:(p, %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.FA570D58.3.objxxx..) 2024-09-15T20:22:04.961 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:120: TEST_truncated_sna_record: (( extr_dbg >= 2 )) 2024-09-15T20:22:04.961 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:120: TEST_truncated_sna_record: od -xc /tmp/tmp.AUiveHT4vh_the_val 2024-09-15T20:22:04.962 INFO:tasks.workunit.client.0.smithi136.stdout:0000000 0101 0035 0000 0003 0000 0000 0000 0304 2024-09-15T20:22:04.962 INFO:tasks.workunit.client.0.smithi136.stdout: 001 001 5 \0 \0 \0 003 \0 \0 \0 \0 \0 \0 \0 004 003 2024-09-15T20:22:04.962 INFO:tasks.workunit.client.0.smithi136.stdout:0000020 0027 0000 0000 0000 0006 0000 626f 786a 2024-09-15T20:22:04.962 INFO:tasks.workunit.client.0.smithi136.stdout: ' \0 \0 \0 \0 \0 \0 \0 006 \0 \0 \0 o b j x 2024-09-15T20:22:04.962 INFO:tasks.workunit.client.0.smithi136.stdout:0000040 7878 0003 0000 0000 0000 75af 85d0 0000 2024-09-15T20:22:04.962 INFO:tasks.workunit.client.0.smithi136.stdout: x x 003 \0 \0 \0 \0 \0 \0 \0 257 u 320 205 \0 \0 2024-09-15T20:22:04.962 INFO:tasks.workunit.client.0.smithi136.stdout:0000060 0000 0100 0000 0000 0000 0000 2024-09-15T20:22:04.962 INFO:tasks.workunit.client.0.smithi136.stdout: \0 \0 \0 001 \0 \0 \0 \0 \0 \0 \0 2024-09-15T20:22:04.962 INFO:tasks.workunit.client.0.smithi136.stdout:0000073 2024-09-15T20:22:04.963 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:122: TEST_truncated_sna_record: NKY=%00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_000000 2024-09-15T20:22:04.963 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:123: TEST_truncated_sna_record: ceph-kvstore-tool bluestore-kv td/osd-mapper/1 rm p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.FA570D58.3.objxxx.. 2024-09-15T20:22:06.771 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:124: TEST_truncated_sna_record: ceph-kvstore-tool bluestore-kv td/osd-mapper/1 set p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_000000 in /tmp/tmp.AUiveHT4vh_the_val 2024-09-15T20:22:08.085 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:126: TEST_truncated_sna_record: (( extr_dbg >= 1 )) 2024-09-15T20:22:08.085 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:106: TEST_truncated_sna_record: for sdn in $(seq 0 $(expr $osdn - 1)) 2024-09-15T20:22:08.085 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:108: TEST_truncated_sna_record: kvdir=td/osd-mapper/2 2024-09-15T20:22:08.086 INFO:tasks.workunit.client.0.smithi136.stdout:corrupting the SnapMapper DB of osd.2 (db: td/osd-mapper/2) 2024-09-15T20:22:08.086 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:109: TEST_truncated_sna_record: echo 'corrupting the SnapMapper DB of osd.2 (db: td/osd-mapper/2)' 2024-09-15T20:22:08.086 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:110: TEST_truncated_sna_record: (( extr_dbg >= 3 )) 2024-09-15T20:22:08.086 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:110: TEST_truncated_sna_record: ceph-kvstore-tool bluestore-kv td/osd-mapper/2 dump p 2024-09-15T20:22:08.110 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:22:08.108+0000 7f3ef351e980 1 bdev(0x561b381d1c00 td/osd-mapper/2/block) open path td/osd-mapper/2/block 2024-09-15T20:22:08.110 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:22:08.108+0000 7f3ef351e980 0 bdev(0x561b381d1c00 td/osd-mapper/2/block) ioctl(F_SET_FILE_RW_HINT) on td/osd-mapper/2/block failed: (22) Invalid argument 2024-09-15T20:22:08.110 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:22:08.108+0000 7f3ef351e980 1 bdev(0x561b381d1c00 td/osd-mapper/2/block) open size 107374182400 (0x1900000000, 100 GiB) block_size 4096 (4 KiB) rotational device, discard supported 2024-09-15T20:22:08.125 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:22:08.123+0000 7f3ef351e980 1 bdev(0x561b381d1c00 td/osd-mapper/2/block) close 2024-09-15T20:22:08.377 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:22:08.375+0000 7f3ef351e980 1 bdev(0x561b381d1c00 td/osd-mapper/2/block) open path td/osd-mapper/2/block 2024-09-15T20:22:08.377 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:22:08.375+0000 7f3ef351e980 0 bdev(0x561b381d1c00 td/osd-mapper/2/block) ioctl(F_SET_FILE_RW_HINT) on td/osd-mapper/2/block failed: (22) Invalid argument 2024-09-15T20:22:08.377 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:22:08.375+0000 7f3ef351e980 1 bdev(0x561b381d1c00 td/osd-mapper/2/block) open size 107374182400 (0x1900000000, 100 GiB) block_size 4096 (4 KiB) rotational device, discard supported 2024-09-15T20:22:08.378 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:22:08.376+0000 7f3ef351e980 1 bluestore(td/osd-mapper/2) _set_cache_sizes cache_size 1073741824 meta 0.45 kv 0.45 kv_onode 0.04 data 0.06 2024-09-15T20:22:08.378 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:22:08.376+0000 7f3ef351e980 1 bdev(0x561b3834c400 td/osd-mapper/2/block) open path td/osd-mapper/2/block 2024-09-15T20:22:08.379 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:22:08.376+0000 7f3ef351e980 0 bdev(0x561b3834c400 td/osd-mapper/2/block) ioctl(F_SET_FILE_RW_HINT) on td/osd-mapper/2/block failed: (22) Invalid argument 2024-09-15T20:22:08.379 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:22:08.377+0000 7f3ef351e980 1 bdev(0x561b3834c400 td/osd-mapper/2/block) open size 107374182400 (0x1900000000, 100 GiB) block_size 4096 (4 KiB) rotational device, discard supported 2024-09-15T20:22:08.379 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:22:08.377+0000 7f3ef351e980 1 bluefs add_block_device bdev 1 path td/osd-mapper/2/block size 100 GiB 2024-09-15T20:22:08.379 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:22:08.377+0000 7f3ef351e980 1 bluefs mount 2024-09-15T20:22:08.379 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:22:08.377+0000 7f3ef351e980 1 bluefs _init_alloc shared, id 1, capacity 0x1900000000, block size 0x10000 2024-09-15T20:22:08.394 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:22:08.392+0000 7f3ef351e980 1 bluefs mount shared_bdev_used = 0 2024-09-15T20:22:08.394 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:22:08.392+0000 7f3ef351e980 1 bluestore(td/osd-mapper/2) _prepare_db_environment set db_paths to db,102005473280 db.slow,102005473280 2024-09-15T20:22:08.431 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:22:08.429+0000 7f3ef351e980 1 bluestore(td/osd-mapper/2) _open_db opened rocksdb path db options compression=kLZ4Compression,max_write_buffer_number=64,min_write_buffer_number_to_merge=6,compaction_style=kCompactionStyleLevel,write_buffer_size=16777216,max_background_jobs=4,level0_file_num_compaction_trigger=8,max_bytes_for_level_base=1073741824,max_bytes_for_level_multiplier=8,compaction_readahead_size=2MB,max_total_wal_size=1073741824,writable_file_max_buffer_size=0 2024-09-15T20:22:08.431 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:22:08.429+0000 7f3ef351e980 1 bluestore(td/osd-mapper/2) _open_super_meta old nid_max 2050 2024-09-15T20:22:08.432 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:22:08.429+0000 7f3ef351e980 1 bluestore(td/osd-mapper/2) _open_super_meta old blobid_max 20480 2024-09-15T20:22:08.432 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:22:08.429+0000 7f3ef351e980 1 bluestore(td/osd-mapper/2) _open_super_meta ondisk_format 4 compat_ondisk_format 3 2024-09-15T20:22:08.432 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:22:08.429+0000 7f3ef351e980 1 bluestore(td/osd-mapper/2) _open_super_meta min_alloc_size 0x1000 2024-09-15T20:22:08.432 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:22:08.429+0000 7f3ef351e980 1 freelist init 2024-09-15T20:22:08.432 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:22:08.429+0000 7f3ef351e980 1 freelist _read_cfg 2024-09-15T20:22:08.432 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:22:08.429+0000 7f3ef351e980 1 bluestore(td/osd-mapper/2) _open_fm effective freelist_type = bitmap, freelist_alloc_size = 0x1000, min_alloc_size = 0x1000 2024-09-15T20:22:08.433 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:22:08.430+0000 7f3ef351e980 1 bluestore(td/osd-mapper/2) _init_alloc loaded 100 GiB in 17 extents, allocator type hybrid, capacity 0x1900000000, block size 0x1000, free 0x18fffbc000, fragmentation 6.10353e-07 2024-09-15T20:22:08.433 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:22:08.430+0000 7f3ef351e980 1 bluefs umount 2024-09-15T20:22:08.433 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:22:08.430+0000 7f3ef351e980 1 bdev(0x561b3834c400 td/osd-mapper/2/block) close 2024-09-15T20:22:08.641 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:22:08.639+0000 7f3ef351e980 1 bdev(0x561b3834c400 td/osd-mapper/2/block) open path td/osd-mapper/2/block 2024-09-15T20:22:08.641 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:22:08.639+0000 7f3ef351e980 0 bdev(0x561b3834c400 td/osd-mapper/2/block) ioctl(F_SET_FILE_RW_HINT) on td/osd-mapper/2/block failed: (22) Invalid argument 2024-09-15T20:22:08.641 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:22:08.639+0000 7f3ef351e980 1 bdev(0x561b3834c400 td/osd-mapper/2/block) open size 107374182400 (0x1900000000, 100 GiB) block_size 4096 (4 KiB) rotational device, discard supported 2024-09-15T20:22:08.641 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:22:08.639+0000 7f3ef351e980 1 bluefs add_block_device bdev 1 path td/osd-mapper/2/block size 100 GiB 2024-09-15T20:22:08.641 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:22:08.639+0000 7f3ef351e980 1 bluefs mount 2024-09-15T20:22:08.649 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:22:08.647+0000 7f3ef351e980 1 bluefs _init_alloc shared, id 1, capacity 0x1900000000, block size 0x10000 2024-09-15T20:22:08.669 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:22:08.664+0000 7f3ef351e980 1 bluefs mount shared_bdev_used = 27459584 2024-09-15T20:22:08.669 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:22:08.664+0000 7f3ef351e980 1 bluestore(td/osd-mapper/2) _prepare_db_environment set db_paths to db,102005473280 db.slow,102005473280 2024-09-15T20:22:08.698 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:22:08.695+0000 7f3ef351e980 1 bluestore(td/osd-mapper/2) _open_db opened rocksdb path db options compression=kLZ4Compression,max_write_buffer_number=64,min_write_buffer_number_to_merge=6,compaction_style=kCompactionStyleLevel,write_buffer_size=16777216,max_background_jobs=4,level0_file_num_compaction_trigger=8,max_bytes_for_level_base=1073741824,max_bytes_for_level_multiplier=8,compaction_readahead_size=2MB,max_total_wal_size=1073741824,writable_file_max_buffer_size=0 2024-09-15T20:22:08.699 INFO:tasks.workunit.client.0.smithi136.stdout:p %00%00%00%00%00%00%00%00%7b%3fC%c4%00%00%00%00%00%00%00%01.osd_superblock 2024-09-15T20:22:08.699 INFO:tasks.workunit.client.0.smithi136.stdout:00000000 0b 05 58 02 00 00 e9 f4 cc 96 8f 95 49 71 84 e5 |..X.........Iq..| 2024-09-15T20:22:08.700 INFO:tasks.workunit.client.0.smithi136.stdout:00000010 d6 4a cd 70 df 0f 02 00 00 00 1e 00 00 00 00 00 |.J.p............| 2024-09-15T20:22:08.700 INFO:tasks.workunit.client.0.smithi136.stdout:00000020 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................| 2024-09-15T20:22:08.700 INFO:tasks.workunit.client.0.smithi136.stdout:* 2024-09-15T20:22:08.700 INFO:tasks.workunit.client.0.smithi136.stdout:00000040 00 00 00 00 00 00 fe ff 03 00 00 00 00 00 11 00 |................| 2024-09-15T20:22:08.700 INFO:tasks.workunit.client.0.smithi136.stdout:00000050 00 00 01 00 00 00 00 00 00 00 1a 00 00 00 69 6e |..............in| 2024-09-15T20:22:08.700 INFO:tasks.workunit.client.0.smithi136.stdout:00000060 69 74 69 61 6c 20 66 65 61 74 75 72 65 20 73 65 |itial feature se| 2024-09-15T20:22:08.700 INFO:tasks.workunit.client.0.smithi136.stdout:00000070 74 28 7e 76 2e 31 38 29 02 00 00 00 00 00 00 00 |t(~v.18)........| 2024-09-15T20:22:08.700 INFO:tasks.workunit.client.0.smithi136.stdout:00000080 0d 00 00 00 70 67 69 6e 66 6f 20 6f 62 6a 65 63 |....pginfo objec| 2024-09-15T20:22:08.700 INFO:tasks.workunit.client.0.smithi136.stdout:00000090 74 03 00 00 00 00 00 00 00 0e 00 00 00 6f 62 6a |t............obj| 2024-09-15T20:22:08.700 INFO:tasks.workunit.client.0.smithi136.stdout:000000a0 65 63 74 20 6c 6f 63 61 74 6f 72 04 00 00 00 00 |ect locator.....| 2024-09-15T20:22:08.700 INFO:tasks.workunit.client.0.smithi136.stdout:000000b0 00 00 00 10 00 00 00 6c 61 73 74 5f 65 70 6f 63 |.......last_epoc| 2024-09-15T20:22:08.700 INFO:tasks.workunit.client.0.smithi136.stdout:000000c0 68 5f 63 6c 65 61 6e 05 00 00 00 00 00 00 00 0a |h_clean.........| 2024-09-15T20:22:08.700 INFO:tasks.workunit.client.0.smithi136.stdout:000000d0 00 00 00 63 61 74 65 67 6f 72 69 65 73 06 00 00 |...categories...| 2024-09-15T20:22:08.701 INFO:tasks.workunit.client.0.smithi136.stdout:000000e0 00 00 00 00 00 0b 00 00 00 68 6f 62 6a 65 63 74 |.........hobject| 2024-09-15T20:22:08.701 INFO:tasks.workunit.client.0.smithi136.stdout:000000f0 70 6f 6f 6c 07 00 00 00 00 00 00 00 07 00 00 00 |pool............| 2024-09-15T20:22:08.701 INFO:tasks.workunit.client.0.smithi136.stdout:00000100 62 69 67 69 6e 66 6f 08 00 00 00 00 00 00 00 0b |biginfo.........| 2024-09-15T20:22:08.701 INFO:tasks.workunit.client.0.smithi136.stdout:00000110 00 00 00 6c 65 76 65 6c 64 62 69 6e 66 6f 09 00 |...leveldbinfo..| 2024-09-15T20:22:08.701 INFO:tasks.workunit.client.0.smithi136.stdout:00000120 00 00 00 00 00 00 0a 00 00 00 6c 65 76 65 6c 64 |..........leveld| 2024-09-15T20:22:08.701 INFO:tasks.workunit.client.0.smithi136.stdout:00000130 62 6c 6f 67 0a 00 00 00 00 00 00 00 0a 00 00 00 |blog............| 2024-09-15T20:22:08.701 INFO:tasks.workunit.client.0.smithi136.stdout:00000140 73 6e 61 70 6d 61 70 70 65 72 0b 00 00 00 00 00 |snapmapper......| 2024-09-15T20:22:08.701 INFO:tasks.workunit.client.0.smithi136.stdout:00000150 00 00 0f 00 00 00 73 68 61 72 64 65 64 20 6f 62 |......sharded ob| 2024-09-15T20:22:08.701 INFO:tasks.workunit.client.0.smithi136.stdout:00000160 6a 65 63 74 73 0c 00 00 00 00 00 00 00 11 00 00 |jects...........| 2024-09-15T20:22:08.701 INFO:tasks.workunit.client.0.smithi136.stdout:00000170 00 74 72 61 6e 73 61 63 74 69 6f 6e 20 68 69 6e |.transaction hin| 2024-09-15T20:22:08.701 INFO:tasks.workunit.client.0.smithi136.stdout:00000180 74 73 0d 00 00 00 00 00 00 00 0e 00 00 00 70 67 |ts............pg| 2024-09-15T20:22:08.701 INFO:tasks.workunit.client.0.smithi136.stdout:00000190 20 6d 65 74 61 20 6f 62 6a 65 63 74 0e 00 00 00 | meta object....| 2024-09-15T20:22:08.701 INFO:tasks.workunit.client.0.smithi136.stdout:000001a0 00 00 00 00 14 00 00 00 65 78 70 6c 69 63 69 74 |........explicit| 2024-09-15T20:22:08.701 INFO:tasks.workunit.client.0.smithi136.stdout:000001b0 20 6d 69 73 73 69 6e 67 20 73 65 74 0f 00 00 00 | missing set....| 2024-09-15T20:22:08.702 INFO:tasks.workunit.client.0.smithi136.stdout:000001c0 00 00 00 00 10 00 00 00 66 61 73 74 69 6e 66 6f |........fastinfo| 2024-09-15T20:22:08.702 INFO:tasks.workunit.client.0.smithi136.stdout:000001d0 20 70 67 20 61 74 74 72 10 00 00 00 00 00 00 00 | pg attr........| 2024-09-15T20:22:08.702 INFO:tasks.workunit.client.0.smithi136.stdout:000001e0 16 00 00 00 64 65 6c 65 74 65 73 20 69 6e 20 6d |....deletes in m| 2024-09-15T20:22:08.702 INFO:tasks.workunit.client.0.smithi136.stdout:000001f0 69 73 73 69 6e 67 20 73 65 74 11 00 00 00 00 00 |issing set......| 2024-09-15T20:22:08.702 INFO:tasks.workunit.client.0.smithi136.stdout:00000200 00 00 1c 00 00 00 6e 65 77 20 73 6e 61 70 6d 61 |......new snapma| 2024-09-15T20:22:08.702 INFO:tasks.workunit.client.0.smithi136.stdout:00000210 70 70 65 72 20 6b 65 79 20 73 74 72 75 63 74 75 |pper key structu| 2024-09-15T20:22:08.702 INFO:tasks.workunit.client.0.smithi136.stdout:00000220 72 65 1e 00 00 00 0f 00 00 00 df b0 17 d7 85 65 |re.............e| 2024-09-15T20:22:08.702 INFO:tasks.workunit.client.0.smithi136.stdout:00000230 48 db ad c5 cd ca 8e d8 86 3d 00 00 00 00 00 00 |H........=......| 2024-09-15T20:22:08.702 INFO:tasks.workunit.client.0.smithi136.stdout:00000240 00 00 1e 00 00 00 c9 41 e7 66 eb e9 1f 26 01 00 |.......A.f...&..| 2024-09-15T20:22:08.702 INFO:tasks.workunit.client.0.smithi136.stdout:00000250 00 00 01 00 00 00 01 00 00 00 1e 00 00 00 |..............| 2024-09-15T20:22:08.702 INFO:tasks.workunit.client.0.smithi136.stdout:0000025e 2024-09-15T20:22:08.702 INFO:tasks.workunit.client.0.smithi136.stdout: 2024-09-15T20:22:08.702 INFO:tasks.workunit.client.0.smithi136.stdout:p %00%00%00%00%00%00%00%00%7b%3fC%c4%00%00%00%00%00%00%00%01~ 2024-09-15T20:22:08.702 INFO:tasks.workunit.client.0.smithi136.stdout: 2024-09-15T20:22:08.703 INFO:tasks.workunit.client.0.smithi136.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.OBJ_0000000000000001.FA570D58.2.objxxx.. 2024-09-15T20:22:08.703 INFO:tasks.workunit.client.0.smithi136.stdout:00000000 01 01 41 00 00 00 04 03 27 00 00 00 00 00 00 00 |..A.....'.......| 2024-09-15T20:22:08.703 INFO:tasks.workunit.client.0.smithi136.stdout:00000010 06 00 00 00 6f 62 6a 78 78 78 02 00 00 00 00 00 |....objxxx......| 2024-09-15T20:22:08.703 INFO:tasks.workunit.client.0.smithi136.stdout:00000020 00 00 af 75 d0 85 00 00 00 00 00 01 00 00 00 00 |...u............| 2024-09-15T20:22:08.703 INFO:tasks.workunit.client.0.smithi136.stdout:00000030 00 00 00 02 00 00 00 01 00 00 00 00 00 00 00 02 |................| 2024-09-15T20:22:08.703 INFO:tasks.workunit.client.0.smithi136.stdout:00000040 00 00 00 00 00 00 00 |.......| 2024-09-15T20:22:08.703 INFO:tasks.workunit.client.0.smithi136.stdout:00000047 2024-09-15T20:22:08.703 INFO:tasks.workunit.client.0.smithi136.stdout: 2024-09-15T20:22:08.703 INFO:tasks.workunit.client.0.smithi136.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.OBJ_0000000000000001.FA570D58.3.objxxx.. 2024-09-15T20:22:08.703 INFO:tasks.workunit.client.0.smithi136.stdout:00000000 01 01 39 00 00 00 04 03 27 00 00 00 00 00 00 00 |..9.....'.......| 2024-09-15T20:22:08.703 INFO:tasks.workunit.client.0.smithi136.stdout:00000010 06 00 00 00 6f 62 6a 78 78 78 03 00 00 00 00 00 |....objxxx......| 2024-09-15T20:22:08.703 INFO:tasks.workunit.client.0.smithi136.stdout:00000020 00 00 af 75 d0 85 00 00 00 00 00 01 00 00 00 00 |...u............| 2024-09-15T20:22:08.703 INFO:tasks.workunit.client.0.smithi136.stdout:00000030 00 00 00 01 00 00 00 03 00 00 00 00 00 00 00 |...............| 2024-09-15T20:22:08.703 INFO:tasks.workunit.client.0.smithi136.stdout:0000003f 2024-09-15T20:22:08.704 INFO:tasks.workunit.client.0.smithi136.stdout: 2024-09-15T20:22:08.704 INFO:tasks.workunit.client.0.smithi136.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.OBJ_0000000000000001.FA570D58.5.objxxx.. 2024-09-15T20:22:08.704 INFO:tasks.workunit.client.0.smithi136.stdout:00000000 01 01 41 00 00 00 04 03 27 00 00 00 00 00 00 00 |..A.....'.......| 2024-09-15T20:22:08.704 INFO:tasks.workunit.client.0.smithi136.stdout:00000010 06 00 00 00 6f 62 6a 78 78 78 05 00 00 00 00 00 |....objxxx......| 2024-09-15T20:22:08.704 INFO:tasks.workunit.client.0.smithi136.stdout:00000020 00 00 af 75 d0 85 00 00 00 00 00 01 00 00 00 00 |...u............| 2024-09-15T20:22:08.704 INFO:tasks.workunit.client.0.smithi136.stdout:00000030 00 00 00 02 00 00 00 04 00 00 00 00 00 00 00 05 |................| 2024-09-15T20:22:08.704 INFO:tasks.workunit.client.0.smithi136.stdout:00000040 00 00 00 00 00 00 00 |.......| 2024-09-15T20:22:08.704 INFO:tasks.workunit.client.0.smithi136.stdout:00000047 2024-09-15T20:22:08.704 INFO:tasks.workunit.client.0.smithi136.stdout: 2024-09-15T20:22:08.704 INFO:tasks.workunit.client.0.smithi136.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.FA570D58.2.objxxx.. 2024-09-15T20:22:08.704 INFO:tasks.workunit.client.0.smithi136.stdout:00000000 01 01 35 00 00 00 01 00 00 00 00 00 00 00 04 03 |..5.............| 2024-09-15T20:22:08.704 INFO:tasks.workunit.client.0.smithi136.stdout:00000010 27 00 00 00 00 00 00 00 06 00 00 00 6f 62 6a 78 |'...........objx| 2024-09-15T20:22:08.704 INFO:tasks.workunit.client.0.smithi136.stdout:00000020 78 78 02 00 00 00 00 00 00 00 af 75 d0 85 00 00 |xx.........u....| 2024-09-15T20:22:08.705 INFO:tasks.workunit.client.0.smithi136.stdout:00000030 00 00 00 01 00 00 00 00 00 00 00 |...........| 2024-09-15T20:22:08.705 INFO:tasks.workunit.client.0.smithi136.stdout:0000003b 2024-09-15T20:22:08.705 INFO:tasks.workunit.client.0.smithi136.stdout: 2024-09-15T20:22:08.705 INFO:tasks.workunit.client.0.smithi136.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000002_0000000000000001.FA570D58.2.objxxx.. 2024-09-15T20:22:08.705 INFO:tasks.workunit.client.0.smithi136.stdout:00000000 01 01 35 00 00 00 02 00 00 00 00 00 00 00 04 03 |..5.............| 2024-09-15T20:22:08.705 INFO:tasks.workunit.client.0.smithi136.stdout:00000010 27 00 00 00 00 00 00 00 06 00 00 00 6f 62 6a 78 |'...........objx| 2024-09-15T20:22:08.705 INFO:tasks.workunit.client.0.smithi136.stdout:00000020 78 78 02 00 00 00 00 00 00 00 af 75 d0 85 00 00 |xx.........u....| 2024-09-15T20:22:08.705 INFO:tasks.workunit.client.0.smithi136.stdout:00000030 00 00 00 01 00 00 00 00 00 00 00 |...........| 2024-09-15T20:22:08.706 INFO:tasks.workunit.client.0.smithi136.stdout:0000003b 2024-09-15T20:22:08.706 INFO:tasks.workunit.client.0.smithi136.stdout: 2024-09-15T20:22:08.707 INFO:tasks.workunit.client.0.smithi136.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.FA570D58.3.objxxx.. 2024-09-15T20:22:08.707 INFO:tasks.workunit.client.0.smithi136.stdout:00000000 01 01 35 00 00 00 03 00 00 00 00 00 00 00 04 03 |..5.............| 2024-09-15T20:22:08.707 INFO:tasks.workunit.client.0.smithi136.stdout:00000010 27 00 00 00 00 00 00 00 06 00 00 00 6f 62 6a 78 |'...........objx| 2024-09-15T20:22:08.707 INFO:tasks.workunit.client.0.smithi136.stdout:00000020 78 78 03 00 00 00 00 00 00 00 af 75 d0 85 00 00 |xx.........u....| 2024-09-15T20:22:08.707 INFO:tasks.workunit.client.0.smithi136.stdout:00000030 00 00 00 01 00 00 00 00 00 00 00 |...........| 2024-09-15T20:22:08.707 INFO:tasks.workunit.client.0.smithi136.stdout:0000003b 2024-09-15T20:22:08.707 INFO:tasks.workunit.client.0.smithi136.stdout: 2024-09-15T20:22:08.707 INFO:tasks.workunit.client.0.smithi136.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000004_0000000000000001.FA570D58.5.objxxx.. 2024-09-15T20:22:08.708 INFO:tasks.workunit.client.0.smithi136.stdout:00000000 01 01 35 00 00 00 04 00 00 00 00 00 00 00 04 03 |..5.............| 2024-09-15T20:22:08.708 INFO:tasks.workunit.client.0.smithi136.stdout:00000010 27 00 00 00 00 00 00 00 06 00 00 00 6f 62 6a 78 |'...........objx| 2024-09-15T20:22:08.708 INFO:tasks.workunit.client.0.smithi136.stdout:00000020 78 78 05 00 00 00 00 00 00 00 af 75 d0 85 00 00 |xx.........u....| 2024-09-15T20:22:08.708 INFO:tasks.workunit.client.0.smithi136.stdout:00000030 00 00 00 01 00 00 00 00 00 00 00 |...........| 2024-09-15T20:22:08.708 INFO:tasks.workunit.client.0.smithi136.stdout:0000003b 2024-09-15T20:22:08.708 INFO:tasks.workunit.client.0.smithi136.stdout: 2024-09-15T20:22:08.708 INFO:tasks.workunit.client.0.smithi136.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000005_0000000000000001.FA570D58.5.objxxx.. 2024-09-15T20:22:08.708 INFO:tasks.workunit.client.0.smithi136.stdout:00000000 01 01 35 00 00 00 05 00 00 00 00 00 00 00 04 03 |..5.............| 2024-09-15T20:22:08.708 INFO:tasks.workunit.client.0.smithi136.stdout:00000010 27 00 00 00 00 00 00 00 06 00 00 00 6f 62 6a 78 |'...........objx| 2024-09-15T20:22:08.709 INFO:tasks.workunit.client.0.smithi136.stdout:00000020 78 78 05 00 00 00 00 00 00 00 af 75 d0 85 00 00 |xx.........u....| 2024-09-15T20:22:08.709 INFO:tasks.workunit.client.0.smithi136.stdout:00000030 00 00 00 01 00 00 00 00 00 00 00 |...........| 2024-09-15T20:22:08.709 INFO:tasks.workunit.client.0.smithi136.stdout:0000003b 2024-09-15T20:22:08.709 INFO:tasks.workunit.client.0.smithi136.stdout: 2024-09-15T20:22:08.709 INFO:tasks.workunit.client.0.smithi136.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02~ 2024-09-15T20:22:08.710 INFO:tasks.workunit.client.0.smithi136.stdout: 2024-09-15T20:22:08.710 INFO:tasks.workunit.client.0.smithi136.stdout:p %00%00%00%00%00%00%00%00%c1%a3%fcn%00%00%00%00%00%00%04%03~ 2024-09-15T20:22:08.710 INFO:tasks.workunit.client.0.smithi136.stdout: 2024-09-15T20:22:08.711 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:22:08.696+0000 7f3ef351e980 1 bluefs umount 2024-09-15T20:22:08.711 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:22:08.696+0000 7f3ef351e980 1 bdev(0x561b3834c400 td/osd-mapper/2/block) close 2024-09-15T20:22:08.909 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:22:08.905+0000 7f3ef351e980 1 freelist shutdown 2024-09-15T20:22:08.909 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:22:08.905+0000 7f3ef351e980 1 bdev(0x561b381d1c00 td/osd-mapper/2/block) close 2024-09-15T20:22:09.144 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:114: TEST_truncated_sna_record: ceph-kvstore-tool bluestore-kv td/osd-mapper/2 dump p 2024-09-15T20:22:09.144 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:114: TEST_truncated_sna_record: grep -a -e 'SNA_[0-9]_0000000000000003_000000000000000' 2024-09-15T20:22:09.144 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:114: TEST_truncated_sna_record: awk -e '{print $2;}' 2024-09-15T20:22:10.197 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:114: TEST_truncated_sna_record: KY=%00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.FA570D58.3.objxxx.. 2024-09-15T20:22:10.197 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:115: TEST_truncated_sna_record: (( extr_dbg >= 1 )) 2024-09-15T20:22:10.198 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:115: TEST_truncated_sna_record: echo 'SNA key: %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.FA570D58.3.objxxx..' 2024-09-15T20:22:10.198 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:115: TEST_truncated_sna_record: cat -v 2024-09-15T20:22:10.198 INFO:tasks.workunit.client.0.smithi136.stdout:SNA key: %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.FA570D58.3.objxxx.. 2024-09-15T20:22:10.199 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:117: TEST_truncated_sna_record: mktemp -p /tmp --suffix=_the_val 2024-09-15T20:22:10.200 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:117: TEST_truncated_sna_record: tmp_fn1=/tmp/tmp.p8PoY8LW0L_the_val 2024-09-15T20:22:10.200 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:118: TEST_truncated_sna_record: (( extr_dbg >= 1 )) 2024-09-15T20:22:10.200 INFO:tasks.workunit.client.0.smithi136.stdout:Value dumped in: /tmp/tmp.p8PoY8LW0L_the_val 2024-09-15T20:22:10.200 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:118: TEST_truncated_sna_record: echo 'Value dumped in: /tmp/tmp.p8PoY8LW0L_the_val' 2024-09-15T20:22:10.200 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:119: TEST_truncated_sna_record: ceph-kvstore-tool bluestore-kv td/osd-mapper/2 get p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.FA570D58.3.objxxx.. out /tmp/tmp.p8PoY8LW0L_the_val 2024-09-15T20:22:10.802 INFO:tasks.workunit.client.0.smithi136.stdout:(p, %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.FA570D58.3.objxxx..) 2024-09-15T20:22:11.262 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:120: TEST_truncated_sna_record: (( extr_dbg >= 2 )) 2024-09-15T20:22:11.263 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:120: TEST_truncated_sna_record: od -xc /tmp/tmp.p8PoY8LW0L_the_val 2024-09-15T20:22:11.263 INFO:tasks.workunit.client.0.smithi136.stdout:0000000 0101 0035 0000 0003 0000 0000 0000 0304 2024-09-15T20:22:11.263 INFO:tasks.workunit.client.0.smithi136.stdout: 001 001 5 \0 \0 \0 003 \0 \0 \0 \0 \0 \0 \0 004 003 2024-09-15T20:22:11.263 INFO:tasks.workunit.client.0.smithi136.stdout:0000020 0027 0000 0000 0000 0006 0000 626f 786a 2024-09-15T20:22:11.263 INFO:tasks.workunit.client.0.smithi136.stdout: ' \0 \0 \0 \0 \0 \0 \0 006 \0 \0 \0 o b j x 2024-09-15T20:22:11.264 INFO:tasks.workunit.client.0.smithi136.stdout:0000040 7878 0003 0000 0000 0000 75af 85d0 0000 2024-09-15T20:22:11.264 INFO:tasks.workunit.client.0.smithi136.stdout: x x 003 \0 \0 \0 \0 \0 \0 \0 257 u 320 205 \0 \0 2024-09-15T20:22:11.264 INFO:tasks.workunit.client.0.smithi136.stdout:0000060 0000 0100 0000 0000 0000 0000 2024-09-15T20:22:11.264 INFO:tasks.workunit.client.0.smithi136.stdout: \0 \0 \0 001 \0 \0 \0 \0 \0 \0 \0 2024-09-15T20:22:11.264 INFO:tasks.workunit.client.0.smithi136.stdout:0000073 2024-09-15T20:22:11.264 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:122: TEST_truncated_sna_record: NKY=%00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_000000 2024-09-15T20:22:11.264 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:123: TEST_truncated_sna_record: ceph-kvstore-tool bluestore-kv td/osd-mapper/2 rm p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.FA570D58.3.objxxx.. 2024-09-15T20:22:13.070 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:124: TEST_truncated_sna_record: ceph-kvstore-tool bluestore-kv td/osd-mapper/2 set p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_000000 in /tmp/tmp.p8PoY8LW0L_the_val 2024-09-15T20:22:14.380 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:126: TEST_truncated_sna_record: (( extr_dbg >= 1 )) 2024-09-15T20:22:14.380 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:129: TEST_truncated_sna_record: orig_osd_args=' --osd_deep_scrub_randomize_ratio=0 --osd_scrub_interval_randomize_ratio=0 --osd_scrub_backoff_ratio=0.0 --osd_pool_default_pg_autoscale_mode=off --osd_pg_stat_report_interval_max_seconds=1 --osd_pg_stat_report_interval_max_epochs=1 --osd_scrub_retry_after_noscrub=5 --osd_scrub_retry_pg_state=5 --osd_scrub_retry_delay=3 ' 2024-09-15T20:22:14.380 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:130: TEST_truncated_sna_record: echo --osd_deep_scrub_randomize_ratio=0 --osd_scrub_interval_randomize_ratio=0 --osd_scrub_backoff_ratio=0.0 --osd_pool_default_pg_autoscale_mode=off --osd_pg_stat_report_interval_max_seconds=1 --osd_pg_stat_report_interval_max_epochs=1 --osd_scrub_retry_after_noscrub=5 --osd_scrub_retry_pg_state=5 --osd_scrub_retry_delay=3 2024-09-15T20:22:14.381 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:130: TEST_truncated_sna_record: orig_osd_args=' --osd_deep_scrub_randomize_ratio=0 --osd_scrub_interval_randomize_ratio=0 --osd_scrub_backoff_ratio=0.0 --osd_pool_default_pg_autoscale_mode=off --osd_pg_stat_report_interval_max_seconds=1 --osd_pg_stat_report_interval_max_epochs=1 --osd_scrub_retry_after_noscrub=5 --osd_scrub_retry_pg_state=5 --osd_scrub_retry_delay=3' 2024-09-15T20:22:14.381 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:131: TEST_truncated_sna_record: (( extr_dbg >= 2 )) 2024-09-15T20:22:14.381 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:131: TEST_truncated_sna_record: echo 'Copied OSD args: / --osd_deep_scrub_randomize_ratio=0 --osd_scrub_interval_randomize_ratio=0 --osd_scrub_backoff_ratio=0.0 --osd_pool_default_pg_autoscale_mode=off --osd_pg_stat_report_interval_max_seconds=1 --osd_pg_stat_report_interval_max_epochs=1 --osd_scrub_retry_after_noscrub=5 --osd_scrub_retry_pg_state=5 --osd_scrub_retry_delay=3/ /--osd_deep_scrub_randomize_ratio=0 --osd_scrub_interval_randomize_ratio=0 --osd_scrub_backoff_ratio=0.0 --osd_pool_default_pg_autoscale_mode=off --osd_pg_stat_report_interval_max_seconds=1 --osd_pg_stat_report_interval_max_epochs=1 --osd_scrub_retry_after_noscrub=5 --osd_scrub_retry_pg_state=5 --osd_scrub_retry_delay=3/' 2024-09-15T20:22:14.381 INFO:tasks.workunit.client.0.smithi136.stdout:Copied OSD args: / --osd_deep_scrub_randomize_ratio=0 --osd_scrub_interval_randomize_ratio=0 --osd_scrub_backoff_ratio=0.0 --osd_pool_default_pg_autoscale_mode=off --osd_pg_stat_report_interval_max_seconds=1 --osd_pg_stat_report_interval_max_epochs=1 --osd_scrub_retry_after_noscrub=5 --osd_scrub_retry_pg_state=5 --osd_scrub_retry_delay=3/ /--osd_deep_scrub_randomize_ratio=0 --osd_scrub_interval_randomize_ratio=0 --osd_scrub_backoff_ratio=0.0 --osd_pool_default_pg_autoscale_mode=off --osd_pg_stat_report_interval_max_seconds=1 --osd_pg_stat_report_interval_max_epochs=1 --osd_scrub_retry_after_noscrub=5 --osd_scrub_retry_pg_state=5 --osd_scrub_retry_delay=3/ 2024-09-15T20:22:14.381 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:132: TEST_truncated_sna_record: expr 3 - 1 2024-09-15T20:22:14.382 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:132: TEST_truncated_sna_record: seq 0 2 2024-09-15T20:22:14.386 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:132: TEST_truncated_sna_record: for sdn in $(seq 0 $(expr $osdn - 1)) 2024-09-15T20:22:14.386 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:134: TEST_truncated_sna_record: CEPH_ARGS='--fsid=e9f4cc96-8f95-4971-84e5-d64acd70df0f --auth-supported=none --mon-host=127.0.0.1:7144 --osd_deep_scrub_randomize_ratio=0 --osd_scrub_interval_randomize_ratio=0 --osd_scrub_backoff_ratio=0.0 --osd_pool_default_pg_autoscale_mode=off --osd_pg_stat_report_interval_max_seconds=1 --osd_pg_stat_report_interval_max_epochs=1 --osd_scrub_retry_after_noscrub=5 --osd_scrub_retry_pg_state=5 --osd_scrub_retry_delay=3' 2024-09-15T20:22:14.386 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:134: TEST_truncated_sna_record: activate_osd td/osd-mapper 0 2024-09-15T20:22:14.386 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-mapper 2024-09-15T20:22:14.386 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-15T20:22:14.386 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=0 2024-09-15T20:22:14.386 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-15T20:22:14.387 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-mapper/0 2024-09-15T20:22:14.387 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=e9f4cc96-8f95-4971-84e5-d64acd70df0f --auth-supported=none --mon-host=127.0.0.1:7144 --osd_deep_scrub_randomize_ratio=0 --osd_scrub_interval_randomize_ratio=0 --osd_scrub_backoff_ratio=0.0 --osd_pool_default_pg_autoscale_mode=off --osd_pg_stat_report_interval_max_seconds=1 --osd_pg_stat_report_interval_max_epochs=1 --osd_scrub_retry_after_noscrub=5 --osd_scrub_retry_pg_state=5 --osd_scrub_retry_delay=3' 2024-09-15T20:22:14.387 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T20:22:14.387 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T20:22:14.387 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T20:22:14.387 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-mapper/0' 2024-09-15T20:22:14.387 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-mapper/0/journal' 2024-09-15T20:22:14.387 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-15T20:22:14.387 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-15T20:22:14.388 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-mapper' 2024-09-15T20:22:14.388 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-15T20:22:14.388 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-15T20:22:14.388 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-15T20:22:14.388 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-15T20:22:14.388 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T20:22:14.388 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.27314 2024-09-15T20:22:14.388 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.27314/$cluster-$name.asok' 2024-09-15T20:22:14.388 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.27314/$cluster-$name.asok' 2024-09-15T20:22:14.388 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-15T20:22:14.388 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-mapper/$name.log' 2024-09-15T20:22:14.388 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-mapper/$name.pid' 2024-09-15T20:22:14.388 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T20:22:14.388 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T20:22:14.389 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T20:22:14.389 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T20:22:14.389 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-15T20:22:14.389 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-15T20:22:14.389 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-mapper/0 2024-09-15T20:22:14.389 INFO:tasks.workunit.client.0.smithi136.stdout:start osd.0 2024-09-15T20:22:14.390 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.0 2024-09-15T20:22:14.390 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 0 --fsid=e9f4cc96-8f95-4971-84e5-d64acd70df0f --auth-supported=none --mon-host=127.0.0.1:7144 --osd_deep_scrub_randomize_ratio=0 --osd_scrub_interval_randomize_ratio=0 --osd_scrub_backoff_ratio=0.0 --osd_pool_default_pg_autoscale_mode=off --osd_pg_stat_report_interval_max_seconds=1 --osd_pg_stat_report_interval_max_epochs=1 --osd_scrub_retry_after_noscrub=5 --osd_scrub_retry_pg_state=5 --osd_scrub_retry_delay=3 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-mapper/0 --osd-journal=td/osd-mapper/0/journal --chdir= --run-dir=td/osd-mapper '--admin-socket=/tmp/ceph-asok.27314/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-mapper/$name.log' '--pid-file=td/osd-mapper/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T20:22:14.390 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-mapper/0/whoami 2024-09-15T20:22:14.390 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 0 = 0 ']' 2024-09-15T20:22:14.390 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-15T20:22:14.390 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-15T20:22:14.390 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-15T20:22:14.435 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:22:14.433+0000 7f33c920d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T20:22:14.437 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:22:14.435+0000 7f33c920d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T20:22:14.438 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:22:14.437+0000 7f33c920d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T20:22:14.756 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 0 2024-09-15T20:22:14.756 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-15T20:22:14.756 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2024-09-15T20:22:14.756 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-15T20:22:14.756 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-15T20:22:14.756 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:22:14.757 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-15T20:22:14.757 INFO:tasks.workunit.client.0.smithi136.stdout:0 2024-09-15T20:22:14.758 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:22:14.758 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T20:22:15.068 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:22:15.537 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:22:15.535+0000 7f33c920d640 -1 Falling back to public interface 2024-09-15T20:22:16.070 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:22:16.070 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:22:16.070 INFO:tasks.workunit.client.0.smithi136.stdout:1 2024-09-15T20:22:16.071 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-15T20:22:16.071 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:22:16.071 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T20:22:16.375 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:22:17.376 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:22:17.376 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:22:17.376 INFO:tasks.workunit.client.0.smithi136.stdout:2 2024-09-15T20:22:17.377 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-15T20:22:17.377 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:22:17.377 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T20:22:17.690 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:22:18.164 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:22:18.163+0000 7f33c920d640 -1 osd.0 30 log_to_monitors true 2024-09-15T20:22:18.689 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:22:18.689 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:22:18.690 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-15T20:22:18.690 INFO:tasks.workunit.client.0.smithi136.stdout:3 2024-09-15T20:22:18.690 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:22:18.690 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T20:22:18.993 INFO:tasks.workunit.client.0.smithi136.stdout:osd.0 up in weight 1 up_from 33 up_thru 33 down_at 31 last_clean_interval [5,30) [v2:127.0.0.1:6802/491900280,v1:127.0.0.1:6803/491900280] [v2:127.0.0.1:6804/491900280,v1:127.0.0.1:6805/491900280] exists,up 130dbee2-9dfc-4ede-9d5f-ae08c2872427 2024-09-15T20:22:18.997 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-15T20:22:18.997 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-15T20:22:18.997 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-15T20:22:18.997 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:132: TEST_truncated_sna_record: for sdn in $(seq 0 $(expr $osdn - 1)) 2024-09-15T20:22:18.997 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:134: TEST_truncated_sna_record: CEPH_ARGS='--fsid=e9f4cc96-8f95-4971-84e5-d64acd70df0f --auth-supported=none --mon-host=127.0.0.1:7144 --osd_deep_scrub_randomize_ratio=0 --osd_scrub_interval_randomize_ratio=0 --osd_scrub_backoff_ratio=0.0 --osd_pool_default_pg_autoscale_mode=off --osd_pg_stat_report_interval_max_seconds=1 --osd_pg_stat_report_interval_max_epochs=1 --osd_scrub_retry_after_noscrub=5 --osd_scrub_retry_pg_state=5 --osd_scrub_retry_delay=3' 2024-09-15T20:22:18.997 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:134: TEST_truncated_sna_record: activate_osd td/osd-mapper 1 2024-09-15T20:22:18.997 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-mapper 2024-09-15T20:22:18.997 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-15T20:22:18.997 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=1 2024-09-15T20:22:18.998 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-15T20:22:18.998 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-mapper/1 2024-09-15T20:22:18.998 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=e9f4cc96-8f95-4971-84e5-d64acd70df0f --auth-supported=none --mon-host=127.0.0.1:7144 --osd_deep_scrub_randomize_ratio=0 --osd_scrub_interval_randomize_ratio=0 --osd_scrub_backoff_ratio=0.0 --osd_pool_default_pg_autoscale_mode=off --osd_pg_stat_report_interval_max_seconds=1 --osd_pg_stat_report_interval_max_epochs=1 --osd_scrub_retry_after_noscrub=5 --osd_scrub_retry_pg_state=5 --osd_scrub_retry_delay=3' 2024-09-15T20:22:18.998 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T20:22:18.998 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T20:22:18.998 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T20:22:18.998 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-mapper/1' 2024-09-15T20:22:18.998 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-mapper/1/journal' 2024-09-15T20:22:18.998 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-15T20:22:18.998 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-15T20:22:18.998 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-mapper' 2024-09-15T20:22:18.998 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-15T20:22:18.999 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-15T20:22:18.999 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-15T20:22:18.999 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-15T20:22:18.999 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T20:22:18.999 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.27314 2024-09-15T20:22:18.999 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.27314/$cluster-$name.asok' 2024-09-15T20:22:18.999 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.27314/$cluster-$name.asok' 2024-09-15T20:22:18.999 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-15T20:22:18.999 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-mapper/$name.log' 2024-09-15T20:22:18.999 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-mapper/$name.pid' 2024-09-15T20:22:18.999 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T20:22:18.999 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T20:22:19.000 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T20:22:19.000 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T20:22:19.000 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-15T20:22:19.000 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-15T20:22:19.000 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-mapper/1 2024-09-15T20:22:19.000 INFO:tasks.workunit.client.0.smithi136.stdout:start osd.1 2024-09-15T20:22:19.001 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.1 2024-09-15T20:22:19.001 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 1 --fsid=e9f4cc96-8f95-4971-84e5-d64acd70df0f --auth-supported=none --mon-host=127.0.0.1:7144 --osd_deep_scrub_randomize_ratio=0 --osd_scrub_interval_randomize_ratio=0 --osd_scrub_backoff_ratio=0.0 --osd_pool_default_pg_autoscale_mode=off --osd_pg_stat_report_interval_max_seconds=1 --osd_pg_stat_report_interval_max_epochs=1 --osd_scrub_retry_after_noscrub=5 --osd_scrub_retry_pg_state=5 --osd_scrub_retry_delay=3 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-mapper/1 --osd-journal=td/osd-mapper/1/journal --chdir= --run-dir=td/osd-mapper '--admin-socket=/tmp/ceph-asok.27314/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-mapper/$name.log' '--pid-file=td/osd-mapper/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T20:22:19.001 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-mapper/1/whoami 2024-09-15T20:22:19.001 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 1 = 1 ']' 2024-09-15T20:22:19.001 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-15T20:22:19.002 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-15T20:22:19.002 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-15T20:22:19.040 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:22:19.038+0000 7f16a2a16640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T20:22:19.072 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:22:19.070+0000 7f16a2a16640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T20:22:19.073 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:22:19.071+0000 7f16a2a16640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T20:22:19.295 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 1 2024-09-15T20:22:19.295 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-15T20:22:19.295 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2024-09-15T20:22:19.295 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-15T20:22:19.295 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-15T20:22:19.295 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:22:19.296 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-15T20:22:19.296 INFO:tasks.workunit.client.0.smithi136.stdout:0 2024-09-15T20:22:19.296 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:22:19.296 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T20:22:19.589 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:22:20.590 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:22:20.590 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:22:20.591 INFO:tasks.workunit.client.0.smithi136.stdout:1 2024-09-15T20:22:20.591 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-15T20:22:20.591 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:22:20.591 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T20:22:20.686 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:22:20.685+0000 7f16a2a16640 -1 Falling back to public interface 2024-09-15T20:22:20.901 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:22:21.903 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:22:21.903 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:22:21.903 INFO:tasks.workunit.client.0.smithi136.stdout:2 2024-09-15T20:22:21.904 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-15T20:22:21.904 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:22:21.904 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T20:22:22.207 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:22:23.208 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:22:23.208 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:22:23.208 INFO:tasks.workunit.client.0.smithi136.stdout:3 2024-09-15T20:22:23.209 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-15T20:22:23.209 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:22:23.209 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T20:22:23.513 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:22:23.543 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:22:23.540+0000 7f16a2a16640 -1 osd.1 30 log_to_monitors true 2024-09-15T20:22:24.515 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:22:24.515 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:22:24.515 INFO:tasks.workunit.client.0.smithi136.stdout:4 2024-09-15T20:22:24.516 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-15T20:22:24.516 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:22:24.516 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T20:22:24.815 INFO:tasks.workunit.client.0.smithi136.stdout:osd.1 up in weight 1 up_from 36 up_thru 36 down_at 31 last_clean_interval [10,30) [v2:127.0.0.1:6810/122981293,v1:127.0.0.1:6811/122981293] [v2:127.0.0.1:6812/122981293,v1:127.0.0.1:6813/122981293] exists,up aa3a94d1-c180-456f-8f8b-c75b0a4a1477 2024-09-15T20:22:24.818 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-15T20:22:24.818 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-15T20:22:24.819 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-15T20:22:24.819 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:132: TEST_truncated_sna_record: for sdn in $(seq 0 $(expr $osdn - 1)) 2024-09-15T20:22:24.819 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:134: TEST_truncated_sna_record: CEPH_ARGS='--fsid=e9f4cc96-8f95-4971-84e5-d64acd70df0f --auth-supported=none --mon-host=127.0.0.1:7144 --osd_deep_scrub_randomize_ratio=0 --osd_scrub_interval_randomize_ratio=0 --osd_scrub_backoff_ratio=0.0 --osd_pool_default_pg_autoscale_mode=off --osd_pg_stat_report_interval_max_seconds=1 --osd_pg_stat_report_interval_max_epochs=1 --osd_scrub_retry_after_noscrub=5 --osd_scrub_retry_pg_state=5 --osd_scrub_retry_delay=3' 2024-09-15T20:22:24.819 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:134: TEST_truncated_sna_record: activate_osd td/osd-mapper 2 2024-09-15T20:22:24.820 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-mapper 2024-09-15T20:22:24.820 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-15T20:22:24.820 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=2 2024-09-15T20:22:24.820 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-15T20:22:24.820 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-mapper/2 2024-09-15T20:22:24.820 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=e9f4cc96-8f95-4971-84e5-d64acd70df0f --auth-supported=none --mon-host=127.0.0.1:7144 --osd_deep_scrub_randomize_ratio=0 --osd_scrub_interval_randomize_ratio=0 --osd_scrub_backoff_ratio=0.0 --osd_pool_default_pg_autoscale_mode=off --osd_pg_stat_report_interval_max_seconds=1 --osd_pg_stat_report_interval_max_epochs=1 --osd_scrub_retry_after_noscrub=5 --osd_scrub_retry_pg_state=5 --osd_scrub_retry_delay=3' 2024-09-15T20:22:24.821 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T20:22:24.821 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T20:22:24.821 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T20:22:24.821 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-mapper/2' 2024-09-15T20:22:24.821 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-mapper/2/journal' 2024-09-15T20:22:24.821 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-15T20:22:24.821 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-15T20:22:24.821 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-mapper' 2024-09-15T20:22:24.821 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-15T20:22:24.821 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-15T20:22:24.821 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-15T20:22:24.822 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-15T20:22:24.822 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T20:22:24.822 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.27314 2024-09-15T20:22:24.822 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.27314/$cluster-$name.asok' 2024-09-15T20:22:24.822 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.27314/$cluster-$name.asok' 2024-09-15T20:22:24.822 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-15T20:22:24.822 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-mapper/$name.log' 2024-09-15T20:22:24.822 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-mapper/$name.pid' 2024-09-15T20:22:24.822 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T20:22:24.822 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T20:22:24.822 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T20:22:24.823 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T20:22:24.823 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-15T20:22:24.823 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-15T20:22:24.823 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-mapper/2 2024-09-15T20:22:24.823 INFO:tasks.workunit.client.0.smithi136.stdout:start osd.2 2024-09-15T20:22:24.824 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.2 2024-09-15T20:22:24.824 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 2 --fsid=e9f4cc96-8f95-4971-84e5-d64acd70df0f --auth-supported=none --mon-host=127.0.0.1:7144 --osd_deep_scrub_randomize_ratio=0 --osd_scrub_interval_randomize_ratio=0 --osd_scrub_backoff_ratio=0.0 --osd_pool_default_pg_autoscale_mode=off --osd_pg_stat_report_interval_max_seconds=1 --osd_pg_stat_report_interval_max_epochs=1 --osd_scrub_retry_after_noscrub=5 --osd_scrub_retry_pg_state=5 --osd_scrub_retry_delay=3 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-mapper/2 --osd-journal=td/osd-mapper/2/journal --chdir= --run-dir=td/osd-mapper '--admin-socket=/tmp/ceph-asok.27314/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-mapper/$name.log' '--pid-file=td/osd-mapper/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T20:22:24.824 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-mapper/2/whoami 2024-09-15T20:22:24.824 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 2 = 2 ']' 2024-09-15T20:22:24.824 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-15T20:22:24.824 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-15T20:22:24.825 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-15T20:22:24.862 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:22:24.859+0000 7f3ee9a0b640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T20:22:24.892 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:22:24.889+0000 7f3ee9a0b640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T20:22:24.893 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:22:24.890+0000 7f3ee9a0b640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T20:22:25.137 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 2 2024-09-15T20:22:25.141 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-15T20:22:25.141 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=2 2024-09-15T20:22:25.141 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-15T20:22:25.141 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-15T20:22:25.142 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:22:25.142 INFO:tasks.workunit.client.0.smithi136.stdout:0 2024-09-15T20:22:25.143 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-15T20:22:25.143 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:22:25.143 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T20:22:25.431 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:22:26.249 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:22:26.246+0000 7f3ee9a0b640 -1 Falling back to public interface 2024-09-15T20:22:26.432 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:22:26.432 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:22:26.432 INFO:tasks.workunit.client.0.smithi136.stdout:1 2024-09-15T20:22:26.432 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-15T20:22:26.433 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:22:26.433 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T20:22:26.728 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:22:27.730 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:22:27.730 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:22:27.730 INFO:tasks.workunit.client.0.smithi136.stdout:2 2024-09-15T20:22:27.731 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-15T20:22:27.731 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:22:27.731 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T20:22:28.027 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:22:28.905 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:22:28.903+0000 7f3ee9a0b640 -1 osd.2 30 log_to_monitors true 2024-09-15T20:22:29.028 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:22:29.028 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:22:29.029 INFO:tasks.workunit.client.0.smithi136.stdout:3 2024-09-15T20:22:29.029 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-15T20:22:29.029 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:22:29.029 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T20:22:29.418 INFO:tasks.workunit.client.0.smithi136.stdout:osd.2 up in weight 1 up_from 40 up_thru 17 down_at 31 last_clean_interval [15,30) [v2:127.0.0.1:6818/466760004,v1:127.0.0.1:6819/466760004] [v2:127.0.0.1:6820/466760004,v1:127.0.0.1:6821/466760004] exists,up dfb017d7-8565-48db-adc5-cdca8ed8863d 2024-09-15T20:22:29.419 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-15T20:22:29.419 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-15T20:22:29.419 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-15T20:22:29.419 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:136: TEST_truncated_sna_record: sleep 1 2024-09-15T20:22:30.415 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:138: TEST_truncated_sna_record: expr 3 - 1 2024-09-15T20:22:30.416 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:138: TEST_truncated_sna_record: seq 0 2 2024-09-15T20:22:30.416 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:138: TEST_truncated_sna_record: for sdn in $(seq 0 $(expr $osdn - 1)) 2024-09-15T20:22:30.417 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:140: TEST_truncated_sna_record: timeout 60 ceph tell osd.0 version 2024-09-15T20:22:30.524 INFO:tasks.workunit.client.0.smithi136.stdout:{ 2024-09-15T20:22:30.524 INFO:tasks.workunit.client.0.smithi136.stdout: "version": "19.3.0-4930-gfc3e0bff", 2024-09-15T20:22:30.524 INFO:tasks.workunit.client.0.smithi136.stdout: "release": "squid", 2024-09-15T20:22:30.524 INFO:tasks.workunit.client.0.smithi136.stdout: "release_type": "dev" 2024-09-15T20:22:30.524 INFO:tasks.workunit.client.0.smithi136.stdout:} 2024-09-15T20:22:30.534 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:138: TEST_truncated_sna_record: for sdn in $(seq 0 $(expr $osdn - 1)) 2024-09-15T20:22:30.534 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:140: TEST_truncated_sna_record: timeout 60 ceph tell osd.1 version 2024-09-15T20:22:30.631 INFO:tasks.workunit.client.0.smithi136.stdout:{ 2024-09-15T20:22:30.631 INFO:tasks.workunit.client.0.smithi136.stdout: "version": "19.3.0-4930-gfc3e0bff", 2024-09-15T20:22:30.631 INFO:tasks.workunit.client.0.smithi136.stdout: "release": "squid", 2024-09-15T20:22:30.631 INFO:tasks.workunit.client.0.smithi136.stdout: "release_type": "dev" 2024-09-15T20:22:30.631 INFO:tasks.workunit.client.0.smithi136.stdout:} 2024-09-15T20:22:30.639 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:138: TEST_truncated_sna_record: for sdn in $(seq 0 $(expr $osdn - 1)) 2024-09-15T20:22:30.639 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:140: TEST_truncated_sna_record: timeout 60 ceph tell osd.2 version 2024-09-15T20:22:30.788 INFO:tasks.workunit.client.0.smithi136.stdout:{ 2024-09-15T20:22:30.788 INFO:tasks.workunit.client.0.smithi136.stdout: "version": "19.3.0-4930-gfc3e0bff", 2024-09-15T20:22:30.788 INFO:tasks.workunit.client.0.smithi136.stdout: "release": "squid", 2024-09-15T20:22:30.788 INFO:tasks.workunit.client.0.smithi136.stdout: "release_type": "dev" 2024-09-15T20:22:30.788 INFO:tasks.workunit.client.0.smithi136.stdout:} 2024-09-15T20:22:30.797 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:142: TEST_truncated_sna_record: rados --format json-pretty -p test listsnaps objxxx 2024-09-15T20:22:30.826 INFO:tasks.workunit.client.0.smithi136.stdout:{ 2024-09-15T20:22:30.826 INFO:tasks.workunit.client.0.smithi136.stdout: "name": "objxxx", 2024-09-15T20:22:30.826 INFO:tasks.workunit.client.0.smithi136.stdout: "clones": [ 2024-09-15T20:22:30.826 INFO:tasks.workunit.client.0.smithi136.stdout: { 2024-09-15T20:22:30.826 INFO:tasks.workunit.client.0.smithi136.stdout: "id": 2, 2024-09-15T20:22:30.827 INFO:tasks.workunit.client.0.smithi136.stdout: "snapshots": [ 2024-09-15T20:22:30.827 INFO:tasks.workunit.client.0.smithi136.stdout: { 2024-09-15T20:22:30.827 INFO:tasks.workunit.client.0.smithi136.stdout: "id": 1, 2024-09-15T20:22:30.827 INFO:tasks.workunit.client.0.smithi136.stdout: "name": "snap01" 2024-09-15T20:22:30.827 INFO:tasks.workunit.client.0.smithi136.stdout: }, 2024-09-15T20:22:30.827 INFO:tasks.workunit.client.0.smithi136.stdout: { 2024-09-15T20:22:30.828 INFO:tasks.workunit.client.0.smithi136.stdout: "id": 2, 2024-09-15T20:22:30.828 INFO:tasks.workunit.client.0.smithi136.stdout: "name": "snap02" 2024-09-15T20:22:30.828 INFO:tasks.workunit.client.0.smithi136.stdout: } 2024-09-15T20:22:30.828 INFO:tasks.workunit.client.0.smithi136.stdout: ], 2024-09-15T20:22:30.828 INFO:tasks.workunit.client.0.smithi136.stdout: "size": 6, 2024-09-15T20:22:30.828 INFO:tasks.workunit.client.0.smithi136.stdout: "overlaps": [] 2024-09-15T20:22:30.828 INFO:tasks.workunit.client.0.smithi136.stdout: }, 2024-09-15T20:22:30.828 INFO:tasks.workunit.client.0.smithi136.stdout: { 2024-09-15T20:22:30.828 INFO:tasks.workunit.client.0.smithi136.stdout: "id": 3, 2024-09-15T20:22:30.828 INFO:tasks.workunit.client.0.smithi136.stdout: "snapshots": [ 2024-09-15T20:22:30.828 INFO:tasks.workunit.client.0.smithi136.stdout: { 2024-09-15T20:22:30.828 INFO:tasks.workunit.client.0.smithi136.stdout: "id": 3, 2024-09-15T20:22:30.828 INFO:tasks.workunit.client.0.smithi136.stdout: "name": "snap13" 2024-09-15T20:22:30.829 INFO:tasks.workunit.client.0.smithi136.stdout: } 2024-09-15T20:22:30.829 INFO:tasks.workunit.client.0.smithi136.stdout: ], 2024-09-15T20:22:30.829 INFO:tasks.workunit.client.0.smithi136.stdout: "size": 5, 2024-09-15T20:22:30.829 INFO:tasks.workunit.client.0.smithi136.stdout: "overlaps": [] 2024-09-15T20:22:30.829 INFO:tasks.workunit.client.0.smithi136.stdout: }, 2024-09-15T20:22:30.829 INFO:tasks.workunit.client.0.smithi136.stdout: { 2024-09-15T20:22:30.829 INFO:tasks.workunit.client.0.smithi136.stdout: "id": 5, 2024-09-15T20:22:30.829 INFO:tasks.workunit.client.0.smithi136.stdout: "snapshots": [ 2024-09-15T20:22:30.829 INFO:tasks.workunit.client.0.smithi136.stdout: { 2024-09-15T20:22:30.829 INFO:tasks.workunit.client.0.smithi136.stdout: "id": 4, 2024-09-15T20:22:30.829 INFO:tasks.workunit.client.0.smithi136.stdout: "name": "snap24" 2024-09-15T20:22:30.829 INFO:tasks.workunit.client.0.smithi136.stdout: }, 2024-09-15T20:22:30.829 INFO:tasks.workunit.client.0.smithi136.stdout: { 2024-09-15T20:22:30.829 INFO:tasks.workunit.client.0.smithi136.stdout: "id": 5, 2024-09-15T20:22:30.829 INFO:tasks.workunit.client.0.smithi136.stdout: "name": "snap25" 2024-09-15T20:22:30.830 INFO:tasks.workunit.client.0.smithi136.stdout: } 2024-09-15T20:22:30.830 INFO:tasks.workunit.client.0.smithi136.stdout: ], 2024-09-15T20:22:30.830 INFO:tasks.workunit.client.0.smithi136.stdout: "size": 5, 2024-09-15T20:22:30.830 INFO:tasks.workunit.client.0.smithi136.stdout: "overlaps": [] 2024-09-15T20:22:30.830 INFO:tasks.workunit.client.0.smithi136.stdout: }, 2024-09-15T20:22:30.830 INFO:tasks.workunit.client.0.smithi136.stdout: { 2024-09-15T20:22:30.830 INFO:tasks.workunit.client.0.smithi136.stdout: "id": "head", 2024-09-15T20:22:30.830 INFO:tasks.workunit.client.0.smithi136.stdout: "snapshots": [], 2024-09-15T20:22:30.830 INFO:tasks.workunit.client.0.smithi136.stdout: "size": 5 2024-09-15T20:22:30.830 INFO:tasks.workunit.client.0.smithi136.stdout: } 2024-09-15T20:22:30.830 INFO:tasks.workunit.client.0.smithi136.stdout: ] 2024-09-15T20:22:30.830 INFO:tasks.workunit.client.0.smithi136.stdout:} 2024-09-15T20:22:30.831 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:145: TEST_truncated_sna_record: ceph osd unset nodeep-scrub 2024-09-15T20:22:31.209 INFO:tasks.workunit.client.0.smithi136.stderr:nodeep-scrub is unset 2024-09-15T20:22:31.221 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:146: TEST_truncated_sna_record: ceph osd unset noscrub 2024-09-15T20:22:31.730 INFO:tasks.workunit.client.0.smithi136.stderr:noscrub is unset 2024-09-15T20:22:31.743 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:149: TEST_truncated_sna_record: ceph --format=json-pretty osd map test objxxx 2024-09-15T20:22:31.743 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:149: TEST_truncated_sna_record: jq -r '.up[0]' 2024-09-15T20:22:32.037 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:149: TEST_truncated_sna_record: local cur_prim=1 2024-09-15T20:22:32.037 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:150: TEST_truncated_sna_record: ceph pg dump pgs 2024-09-15T20:22:32.338 INFO:tasks.workunit.client.0.smithi136.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-15T20:22:32.338 INFO:tasks.workunit.client.0.smithi136.stdout:1.3 4 0 4 0 0 21 0 0 7 0 7 active+undersized+degraded 2024-09-15T20:22:24.789684+0000 28'7 38:57 [1,0] 1 [1,0] 1 28'7 2024-09-15T20:21:42.101714+0000 28'7 2024-09-15T20:21:42.101714+0000 0 1 periodic scrub scheduled @ 2024-09-16T20:21:42.101714+0000 4 0 2024-09-15T20:22:32.338 INFO:tasks.workunit.client.0.smithi136.stdout:1.2 0 0 0 0 0 0 0 0 0 0 0 active+undersized 2024-09-15T20:22:24.864867+0000 0'0 38:31 [0,1] 0 [0,1] 0 0'0 2024-09-15T20:21:31.637829+0000 0'0 2024-09-15T20:21:31.637829+0000 0 0 periodic scrub scheduled @ 2024-09-16T20:21:31.637829+0000 0 0 2024-09-15T20:22:32.338 INFO:tasks.workunit.client.0.smithi136.stdout:1.1 0 0 0 0 0 0 0 0 0 0 0 active+undersized 2024-09-15T20:22:24.864873+0000 0'0 38:20 [0,1] 0 [0,1] 0 0'0 2024-09-15T20:21:31.637829+0000 0'0 2024-09-15T20:21:31.637829+0000 0 0 periodic scrub scheduled @ 2024-09-16T20:21:31.637829+0000 0 0 2024-09-15T20:22:32.338 INFO:tasks.workunit.client.0.smithi136.stdout:1.0 0 0 0 0 0 0 0 0 0 0 0 active+undersized 2024-09-15T20:22:24.789644+0000 0'0 38:22 [1,0] 1 [1,0] 1 0'0 2024-09-15T20:21:31.637829+0000 0'0 2024-09-15T20:21:31.637829+0000 0 0 periodic scrub scheduled @ 2024-09-16T20:21:31.637829+0000 0 0 2024-09-15T20:22:32.338 INFO:tasks.workunit.client.0.smithi136.stdout: 2024-09-15T20:22:32.339 INFO:tasks.workunit.client.0.smithi136.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-15T20:22:32.339 INFO:tasks.workunit.client.0.smithi136.stderr:dumped pgs 2024-09-15T20:22:32.349 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:151: TEST_truncated_sna_record: sleep 2 2024-09-15T20:22:34.350 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:152: TEST_truncated_sna_record: ceph pg 1.3 deep-scrub 2024-09-15T20:22:34.455 INFO:tasks.workunit.client.0.smithi136.stdout:{ 2024-09-15T20:22:34.455 INFO:tasks.workunit.client.0.smithi136.stdout: "deep": true, 2024-09-15T20:22:34.455 INFO:tasks.workunit.client.0.smithi136.stdout: "must": true, 2024-09-15T20:22:34.455 INFO:tasks.workunit.client.0.smithi136.stdout: "stamp": "0.000000" 2024-09-15T20:22:34.455 INFO:tasks.workunit.client.0.smithi136.stdout:} 2024-09-15T20:22:34.464 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:153: TEST_truncated_sna_record: sleep 5 2024-09-15T20:22:39.466 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:154: TEST_truncated_sna_record: ceph pg dump pgs 2024-09-15T20:22:39.733 INFO:tasks.workunit.client.0.smithi136.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-15T20:22:39.733 INFO:tasks.workunit.client.0.smithi136.stdout:1.3 4 0 0 0 0 21 0 0 7 0 7 active+clean 2024-09-15T20:22:30.555987+0000 28'7 44:76 [1,2,0] 1 [1,2,0] 1 28'7 2024-09-15T20:21:42.101714+0000 28'7 2024-09-15T20:21:42.101714+0000 0 1 periodic scrub scheduled @ 2024-09-16T20:21:42.101714+0000 4 0 2024-09-15T20:22:39.733 INFO:tasks.workunit.client.0.smithi136.stdout:1.2 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-09-15T20:22:30.011491+0000 0'0 44:49 [0,1,2] 0 [0,1,2] 0 0'0 2024-09-15T20:21:31.637829+0000 0'0 2024-09-15T20:21:31.637829+0000 0 0 periodic scrub scheduled @ 2024-09-16T20:21:31.637829+0000 0 0 2024-09-15T20:22:39.733 INFO:tasks.workunit.client.0.smithi136.stdout:1.1 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-09-15T20:22:30.455433+0000 0'0 44:28 [2,0,1] 2 [2,0,1] 2 0'0 2024-09-15T20:21:31.637829+0000 0'0 2024-09-15T20:21:31.637829+0000 0 0 periodic scrub scheduled @ 2024-09-16T20:21:31.637829+0000 0 0 2024-09-15T20:22:39.733 INFO:tasks.workunit.client.0.smithi136.stdout:1.0 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-09-15T20:22:30.555925+0000 0'0 44:40 [1,0,2] 1 [1,0,2] 1 0'0 2024-09-15T20:21:31.637829+0000 0'0 2024-09-15T20:21:31.637829+0000 0 0 periodic scrub scheduled @ 2024-09-16T20:21:31.637829+0000 0 0 2024-09-15T20:22:39.733 INFO:tasks.workunit.client.0.smithi136.stdout: 2024-09-15T20:22:39.733 INFO:tasks.workunit.client.0.smithi136.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-15T20:22:39.734 INFO:tasks.workunit.client.0.smithi136.stderr:dumped pgs 2024-09-15T20:22:39.744 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:155: TEST_truncated_sna_record: (( extr_dbg >= 1 )) 2024-09-15T20:22:39.744 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:155: TEST_truncated_sna_record: grep -a ERR td/osd-mapper/osd.1.log 2024-09-15T20:22:39.746 INFO:tasks.workunit.client.0.smithi136.stdout:2024-09-15T20:22:34.574+0000 7f1684c33640 -1 log_channel(cluster) log [ERR] : osd.1 found snap mapper error on pg 1.3 oid 1:f5ae0ba1:::objxxx:3 snaps in mapper: {}, oi: {3} ...repaired 2024-09-15T20:22:39.746 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:156: TEST_truncated_sna_record: grep -a -q ERR td/osd-mapper/osd.1.log 2024-09-15T20:22:39.748 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:160: TEST_truncated_sna_record: grep -a ERR td/osd-mapper/osd.1.log 2024-09-15T20:22:39.748 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:160: TEST_truncated_sna_record: wc -l 2024-09-15T20:22:39.750 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:160: TEST_truncated_sna_record: local prev_err_cnt=1 2024-09-15T20:22:39.750 INFO:tasks.workunit.client.0.smithi136.stdout:prev count: 1 2024-09-15T20:22:39.750 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:161: TEST_truncated_sna_record: echo 'prev count: 1' 2024-09-15T20:22:39.751 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:164: TEST_truncated_sna_record: ceph pg 1.3 deep-scrub 2024-09-15T20:22:39.846 INFO:tasks.workunit.client.0.smithi136.stdout:{ 2024-09-15T20:22:39.846 INFO:tasks.workunit.client.0.smithi136.stdout: "deep": true, 2024-09-15T20:22:39.846 INFO:tasks.workunit.client.0.smithi136.stdout: "must": true, 2024-09-15T20:22:39.846 INFO:tasks.workunit.client.0.smithi136.stdout: "stamp": "0.000000" 2024-09-15T20:22:39.846 INFO:tasks.workunit.client.0.smithi136.stdout:} 2024-09-15T20:22:39.853 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:165: TEST_truncated_sna_record: sleep 5 2024-09-15T20:22:44.854 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:166: TEST_truncated_sna_record: ceph pg dump pgs 2024-09-15T20:22:45.123 INFO:tasks.workunit.client.0.smithi136.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-15T20:22:45.123 INFO:tasks.workunit.client.0.smithi136.stdout:1.3 4 0 0 0 0 21 0 0 7 0 7 active+clean 2024-09-15T20:22:40.660807+0000 28'7 44:92 [1,2,0] 1 [1,2,0] 1 28'7 2024-09-15T20:22:40.660750+0000 28'7 2024-09-15T20:22:40.660750+0000 0 1 periodic scrub scheduled @ 2024-09-16T20:22:40.660750+0000 4 0 2024-09-15T20:22:45.123 INFO:tasks.workunit.client.0.smithi136.stdout:1.2 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-09-15T20:22:30.011491+0000 0'0 44:49 [0,1,2] 0 [0,1,2] 0 0'0 2024-09-15T20:21:31.637829+0000 0'0 2024-09-15T20:21:31.637829+0000 0 0 periodic scrub scheduled @ 2024-09-16T20:21:31.637829+0000 0 0 2024-09-15T20:22:45.123 INFO:tasks.workunit.client.0.smithi136.stdout:1.1 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-09-15T20:22:30.455433+0000 0'0 44:28 [2,0,1] 2 [2,0,1] 2 0'0 2024-09-15T20:21:31.637829+0000 0'0 2024-09-15T20:21:31.637829+0000 0 0 periodic scrub scheduled @ 2024-09-16T20:21:31.637829+0000 0 0 2024-09-15T20:22:45.123 INFO:tasks.workunit.client.0.smithi136.stdout:1.0 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-09-15T20:22:30.555925+0000 0'0 44:40 [1,0,2] 1 [1,0,2] 1 0'0 2024-09-15T20:21:31.637829+0000 0'0 2024-09-15T20:21:31.637829+0000 0 0 periodic scrub scheduled @ 2024-09-16T20:21:31.637829+0000 0 0 2024-09-15T20:22:45.123 INFO:tasks.workunit.client.0.smithi136.stdout: 2024-09-15T20:22:45.123 INFO:tasks.workunit.client.0.smithi136.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-15T20:22:45.124 INFO:tasks.workunit.client.0.smithi136.stderr:dumped pgs 2024-09-15T20:22:45.133 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:167: TEST_truncated_sna_record: (( extr_dbg >= 1 )) 2024-09-15T20:22:45.134 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:167: TEST_truncated_sna_record: grep -a ERR td/osd-mapper/osd.1.log 2024-09-15T20:22:45.135 INFO:tasks.workunit.client.0.smithi136.stdout:2024-09-15T20:22:34.574+0000 7f1684c33640 -1 log_channel(cluster) log [ERR] : osd.1 found snap mapper error on pg 1.3 oid 1:f5ae0ba1:::objxxx:3 snaps in mapper: {}, oi: {3} ...repaired 2024-09-15T20:22:45.135 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:168: TEST_truncated_sna_record: grep -a ERR td/osd-mapper/osd.1.log 2024-09-15T20:22:45.136 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:168: TEST_truncated_sna_record: wc -l 2024-09-15T20:22:45.137 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:168: TEST_truncated_sna_record: local current_err_cnt=1 2024-09-15T20:22:45.137 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:169: TEST_truncated_sna_record: (( extr_dbg >= 1 )) 2024-09-15T20:22:45.138 INFO:tasks.workunit.client.0.smithi136.stdout:current count: 1 2024-09-15T20:22:45.138 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:169: TEST_truncated_sna_record: echo 'current count: 1' 2024-09-15T20:22:45.138 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:170: TEST_truncated_sna_record: (( current_err_cnt == prev_err_cnt )) 2024-09-15T20:22:45.139 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:171: TEST_truncated_sna_record: kill_daemons td/osd-mapper TERM osd 2024-09-15T20:22:45.139 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-15T20:22:45.139 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-15T20:22:45.139 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-15T20:22:45.139 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-15T20:22:45.139 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-15T20:22:45.444 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-15T20:22:45.445 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:172: TEST_truncated_sna_record: kvdir=td/osd-mapper/1 2024-09-15T20:22:45.445 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:173: TEST_truncated_sna_record: (( extr_dbg >= 2 )) 2024-09-15T20:22:45.445 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:173: TEST_truncated_sna_record: ceph-kvstore-tool bluestore-kv td/osd-mapper/1 dump p 2024-09-15T20:22:45.445 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:173: TEST_truncated_sna_record: grep -a -e 'SNA_[0-9]_' 2024-09-15T20:22:45.445 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:174: TEST_truncated_sna_record: awk -e '{print $2;}' 2024-09-15T20:22:46.493 INFO:tasks.workunit.client.0.smithi136.stdout:%00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.FA570D58.2.objxxx.. 2024-09-15T20:22:46.493 INFO:tasks.workunit.client.0.smithi136.stdout:%00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000002_0000000000000001.FA570D58.2.objxxx.. 2024-09-15T20:22:46.493 INFO:tasks.workunit.client.0.smithi136.stdout:%00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_000000 2024-09-15T20:22:46.493 INFO:tasks.workunit.client.0.smithi136.stdout:%00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.FA570D58.3.objxxx.. 2024-09-15T20:22:46.493 INFO:tasks.workunit.client.0.smithi136.stdout:%00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000004_0000000000000001.FA570D58.5.objxxx.. 2024-09-15T20:22:46.493 INFO:tasks.workunit.client.0.smithi136.stdout:%00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000005_0000000000000001.FA570D58.5.objxxx.. 2024-09-15T20:22:46.494 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:176: TEST_truncated_sna_record: ceph-kvstore-tool bluestore-kv td/osd-mapper/1 dump p 2024-09-15T20:22:46.494 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:176: TEST_truncated_sna_record: grep -a -e 'SNA_[0-9]_000000000000000[0-9]_000000000000000' 2024-09-15T20:22:46.494 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:176: TEST_truncated_sna_record: awk -e '{print $2;}' 2024-09-15T20:22:46.494 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:176: TEST_truncated_sna_record: wc -l 2024-09-15T20:22:47.549 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:176: TEST_truncated_sna_record: local num_sna_full=5 2024-09-15T20:22:47.549 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:177: TEST_truncated_sna_record: (( num_sna_full == num_sna_b4 )) 2024-09-15T20:22:47.549 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:178: TEST_truncated_sna_record: return 0 2024-09-15T20:22:47.549 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:24: run: teardown td/osd-mapper 2024-09-15T20:22:47.549 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/osd-mapper 2024-09-15T20:22:47.549 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-15T20:22:47.549 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/osd-mapper KILL 2024-09-15T20:22:47.549 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-15T20:22:47.550 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-15T20:22:47.550 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-15T20:22:47.550 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-15T20:22:47.550 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-15T20:22:47.659 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-15T20:22:47.659 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-15T20:22:47.659 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-15T20:22:47.660 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-15T20:22:47.661 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-15T20:22:47.661 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-15T20:22:47.661 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-15T20:22:47.663 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T20:22:47.663 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-15T20:22:47.663 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-15T20:22:47.663 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T20:22:47.664 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-15T20:22:47.665 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-15T20:22:47.665 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/osd-mapper 2024-09-15T20:22:47.682 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-15T20:22:47.682 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T20:22:47.682 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.27314 2024-09-15T20:22:47.682 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.27314 2024-09-15T20:22:47.684 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-15T20:22:47.684 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-15T20:22:47.684 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2322: main: code=0 2024-09-15T20:22:47.684 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2326: main: teardown td/osd-mapper 0 2024-09-15T20:22:47.684 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/osd-mapper 2024-09-15T20:22:47.684 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs=0 2024-09-15T20:22:47.684 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/osd-mapper KILL 2024-09-15T20:22:47.684 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-15T20:22:47.685 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-15T20:22:47.685 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-15T20:22:47.685 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-15T20:22:47.685 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-15T20:22:47.685 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-15T20:22:47.686 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-15T20:22:47.699 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-15T20:22:47.699 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-15T20:22:47.699 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-15T20:22:47.699 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-15T20:22:47.699 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-15T20:22:47.699 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T20:22:47.699 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-15T20:22:47.699 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-15T20:22:47.699 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T20:22:47.700 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-15T20:22:47.700 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o 0 = 1 ']' 2024-09-15T20:22:47.700 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/osd-mapper 2024-09-15T20:22:47.700 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-15T20:22:47.700 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T20:22:47.700 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.27314 2024-09-15T20:22:47.700 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.27314 2024-09-15T20:22:47.700 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-15T20:22:47.700 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-15T20:22:47.700 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2327: main: return 0 2024-09-15T20:22:47.700 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-09-15T20:22:47.701 DEBUG:teuthology.orchestra.run.smithi136:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-09-15T20:22:47.767 INFO:tasks.workunit:Running workunit scrub/osd-recovery-scrub.sh... 2024-09-15T20:22:47.768 DEBUG:teuthology.orchestra.run.smithi136:workunit test scrub/osd-recovery-scrub.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=7498493279982dfe87f00616198a5967475f6169 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh 2024-09-15T20:22:47.835 INFO:tasks.workunit.client.0.smithi136.stderr:stty: 'standard input': Inappropriate ioctl for device 2024-09-15T20:22:47.841 INFO:tasks.workunit.client.0.smithi136.stderr:+ PS4='${BASH_SOURCE[0]}:$LINENO: ${FUNCNAME[0]}: ' 2024-09-15T20:22:47.841 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2315: main: export PATH=.:/home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin 2024-09-15T20:22:47.841 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2315: main: PATH=.:/home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin 2024-09-15T20:22:47.841 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2316: main: export PYTHONWARNINGS=ignore 2024-09-15T20:22:47.841 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2316: main: PYTHONWARNINGS=ignore 2024-09-15T20:22:47.841 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2317: main: export CEPH_CONF=/dev/null 2024-09-15T20:22:47.842 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2317: main: CEPH_CONF=/dev/null 2024-09-15T20:22:47.842 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2318: main: unset CEPH_ARGS 2024-09-15T20:22:47.842 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2320: main: local code 2024-09-15T20:22:47.842 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2321: main: run td/osd-recovery-scrub 2024-09-15T20:22:47.842 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:20: run: local dir=td/osd-recovery-scrub 2024-09-15T20:22:47.842 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:21: run: shift 2024-09-15T20:22:47.842 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:23: run: export CEPH_MON=127.0.0.1:7124 2024-09-15T20:22:47.842 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:23: run: CEPH_MON=127.0.0.1:7124 2024-09-15T20:22:47.842 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:24: run: export CEPH_ARGS 2024-09-15T20:22:47.842 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:25: run: uuidgen 2024-09-15T20:22:47.842 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:25: run: CEPH_ARGS+='--fsid=a80d5a97-0ed0-4dbf-8fa5-41ae5f0552a3 --auth-supported=none ' 2024-09-15T20:22:47.842 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:26: run: CEPH_ARGS+='--mon-host=127.0.0.1:7124 ' 2024-09-15T20:22:47.842 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:27: run: CEPH_ARGS+='--osd-op-queue=wpq ' 2024-09-15T20:22:47.843 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:29: run: export -n CEPH_CLI_TEST_DUP_COMMAND 2024-09-15T20:22:47.845 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:30: run: set 2024-09-15T20:22:47.845 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:30: run: sed -n -e 's/^\(TEST_[0-9a-z_]*\) .*/\1/p' 2024-09-15T20:22:47.845 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:30: run: local 'funcs=TEST_recovery_scrub_1 2024-09-15T20:22:47.845 INFO:tasks.workunit.client.0.smithi136.stderr:TEST_recovery_scrub_2' 2024-09-15T20:22:47.846 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:31: run: for func in $funcs 2024-09-15T20:22:47.846 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:32: run: TEST_recovery_scrub_1 td/osd-recovery-scrub 2024-09-15T20:22:47.846 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:39: TEST_recovery_scrub_1: local dir=td/osd-recovery-scrub 2024-09-15T20:22:47.846 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:40: TEST_recovery_scrub_1: local poolname=test 2024-09-15T20:22:47.846 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:42: TEST_recovery_scrub_1: TESTDATA=testdata.43761 2024-09-15T20:22:47.846 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:43: TEST_recovery_scrub_1: OSDS=4 2024-09-15T20:22:47.846 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:44: TEST_recovery_scrub_1: PGS=1 2024-09-15T20:22:47.846 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:45: TEST_recovery_scrub_1: OBJECTS=100 2024-09-15T20:22:47.846 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:46: TEST_recovery_scrub_1: ERRORS=0 2024-09-15T20:22:47.846 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:48: TEST_recovery_scrub_1: setup td/osd-recovery-scrub 2024-09-15T20:22:47.846 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/osd-recovery-scrub 2024-09-15T20:22:47.846 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/osd-recovery-scrub 2024-09-15T20:22:47.846 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/osd-recovery-scrub 2024-09-15T20:22:47.847 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-15T20:22:47.847 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/osd-recovery-scrub KILL 2024-09-15T20:22:47.847 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-15T20:22:47.847 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-15T20:22:47.847 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-15T20:22:47.847 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-15T20:22:47.847 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-15T20:22:47.847 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-15T20:22:47.847 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-15T20:22:47.848 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-15T20:22:47.848 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-15T20:22:47.848 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-15T20:22:47.848 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-15T20:22:47.848 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-15T20:22:47.849 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T20:22:47.850 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-15T20:22:47.850 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-15T20:22:47.850 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T20:22:47.850 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-15T20:22:47.853 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-15T20:22:47.853 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/osd-recovery-scrub 2024-09-15T20:22:47.853 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-15T20:22:47.853 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T20:22:47.853 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.43761 2024-09-15T20:22:47.854 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.43761 2024-09-15T20:22:47.854 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-15T20:22:47.854 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-15T20:22:47.854 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/osd-recovery-scrub 2024-09-15T20:22:47.854 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2024-09-15T20:22:47.854 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T20:22:47.855 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.43761 2024-09-15T20:22:47.855 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.43761 2024-09-15T20:22:47.857 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2024-09-15T20:22:47.857 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 1024 -le 1024 ']' 2024-09-15T20:22:47.857 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:127: setup: ulimit -n 4096 2024-09-15T20:22:47.857 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2024-09-15T20:22:47.857 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/osd-recovery-scrub 1' TERM HUP INT 2024-09-15T20:22:47.857 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:49: TEST_recovery_scrub_1: run_mon td/osd-recovery-scrub a --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-09-15T20:22:47.857 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/osd-recovery-scrub 2024-09-15T20:22:47.857 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2024-09-15T20:22:47.857 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2024-09-15T20:22:47.857 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2024-09-15T20:22:47.857 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/osd-recovery-scrub/a 2024-09-15T20:22:47.857 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-recovery-scrub/a --run-dir=td/osd-recovery-scrub --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-09-15T20:22:48.319 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2024-09-15T20:22:48.319 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-15T20:22:48.319 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-15T20:22:48.319 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-15T20:22:48.319 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T20:22:48.319 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.43761 2024-09-15T20:22:48.319 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.43761/$cluster-$name.asok' 2024-09-15T20:22:48.319 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-recovery-scrub/a '--log-file=td/osd-recovery-scrub/$name.log' '--admin-socket=/tmp/ceph-asok.43761/$cluster-$name.asok' --mon-cluster-log-file=td/osd-recovery-scrub/log --run-dir=td/osd-recovery-scrub '--pid-file=td/osd-recovery-scrub/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-09-15T20:22:48.519 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2024-09-15T20:22:48.521 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2024-09-15T20:22:48.521 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-15T20:22:48.521 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-15T20:22:48.521 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2024-09-15T20:22:48.521 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2024-09-15T20:22:48.521 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-15T20:22:48.521 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-15T20:22:48.521 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-15T20:22:48.522 INFO:tasks.workunit.client.0.smithi136.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-15T20:22:48.522 INFO:tasks.workunit.client.0.smithi136.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T20:22:48.529 INFO:tasks.workunit.client.0.smithi136.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.43761 2024-09-15T20:22:48.529 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.43761/ceph-mon.a.asok 2024-09-15T20:22:48.530 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-15T20:22:48.530 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.43761/ceph-mon.a.asok config get fsid 2024-09-15T20:22:48.602 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2024-09-15T20:22:48.602 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-15T20:22:48.603 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-15T20:22:48.603 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2024-09-15T20:22:48.603 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2024-09-15T20:22:48.603 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-15T20:22:48.603 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-15T20:22:48.603 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-15T20:22:48.603 INFO:tasks.workunit.client.0.smithi136.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-15T20:22:48.603 INFO:tasks.workunit.client.0.smithi136.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T20:22:48.603 INFO:tasks.workunit.client.0.smithi136.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.43761 2024-09-15T20:22:48.603 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.43761/ceph-mon.a.asok 2024-09-15T20:22:48.603 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-15T20:22:48.603 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.43761/ceph-mon.a.asok config get mon_host 2024-09-15T20:22:48.680 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:50: TEST_recovery_scrub_1: run_mgr td/osd-recovery-scrub x 2024-09-15T20:22:48.680 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: run_mgr: local dir=td/osd-recovery-scrub 2024-09-15T20:22:48.680 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: run_mgr: shift 2024-09-15T20:22:48.680 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: run_mgr: local id=x 2024-09-15T20:22:48.680 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: run_mgr: shift 2024-09-15T20:22:48.680 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:548: run_mgr: local data=td/osd-recovery-scrub/x 2024-09-15T20:22:48.680 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:550: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-15T20:22:49.496 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: get_asok_path 2024-09-15T20:22:49.496 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-15T20:22:49.496 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-15T20:22:49.496 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-15T20:22:49.496 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T20:22:49.496 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.43761 2024-09-15T20:22:49.496 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.43761/$cluster-$name.asok' 2024-09-15T20:22:49.496 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-15T20:22:49.497 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-recovery-scrub/x '--log-file=td/osd-recovery-scrub/$name.log' '--admin-socket=/tmp/ceph-asok.43761/$cluster-$name.asok' --run-dir=td/osd-recovery-scrub '--pid-file=td/osd-recovery-scrub/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-15T20:22:49.531 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:51: TEST_recovery_scrub_1: local 'ceph_osd_args=--osd-scrub-interval-randomize-ratio=0 ' 2024-09-15T20:22:49.531 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:52: TEST_recovery_scrub_1: ceph_osd_args+='--osd_scrub_backoff_ratio=0 ' 2024-09-15T20:22:49.531 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:53: TEST_recovery_scrub_1: ceph_osd_args+='--osd_stats_update_period_not_scrubbing=3 ' 2024-09-15T20:22:49.531 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:54: TEST_recovery_scrub_1: ceph_osd_args+=--osd_stats_update_period_scrubbing=2 2024-09-15T20:22:49.531 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:55: TEST_recovery_scrub_1: expr 4 - 1 2024-09-15T20:22:49.531 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:55: TEST_recovery_scrub_1: seq 0 3 2024-09-15T20:22:49.531 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:55: TEST_recovery_scrub_1: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-09-15T20:22:49.531 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:57: TEST_recovery_scrub_1: run_osd td/osd-recovery-scrub 0 --osd_scrub_during_recovery=false 2024-09-15T20:22:49.531 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-recovery-scrub 2024-09-15T20:22:49.532 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-15T20:22:49.532 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=0 2024-09-15T20:22:49.532 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-15T20:22:49.532 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-recovery-scrub/0 2024-09-15T20:22:49.532 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=a80d5a97-0ed0-4dbf-8fa5-41ae5f0552a3 --auth-supported=none --mon-host=127.0.0.1:7124 --osd-op-queue=wpq ' 2024-09-15T20:22:49.532 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T20:22:49.532 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T20:22:49.532 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T20:22:49.532 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-recovery-scrub/0' 2024-09-15T20:22:49.532 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-scrub/0/journal' 2024-09-15T20:22:49.532 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-15T20:22:49.532 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-15T20:22:49.533 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-recovery-scrub' 2024-09-15T20:22:49.533 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-15T20:22:49.533 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-15T20:22:49.533 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-15T20:22:49.533 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-15T20:22:49.533 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T20:22:49.533 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.43761 2024-09-15T20:22:49.533 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.43761/$cluster-$name.asok' 2024-09-15T20:22:49.533 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.43761/$cluster-$name.asok' 2024-09-15T20:22:49.533 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-15T20:22:49.533 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-15T20:22:49.533 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-15T20:22:49.533 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-recovery-scrub/$name.log' 2024-09-15T20:22:49.534 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-recovery-scrub/$name.pid' 2024-09-15T20:22:49.534 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T20:22:49.534 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T20:22:49.534 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T20:22:49.534 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T20:22:49.534 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-15T20:22:49.534 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=--osd_scrub_during_recovery=false 2024-09-15T20:22:49.534 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-recovery-scrub/0 2024-09-15T20:22:49.534 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-15T20:22:49.534 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=8807c1df-6e7e-47fb-94e8-222239bafe07 2024-09-15T20:22:49.535 INFO:tasks.workunit.client.0.smithi136.stdout:add osd0 8807c1df-6e7e-47fb-94e8-222239bafe07 2024-09-15T20:22:49.535 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd0 8807c1df-6e7e-47fb-94e8-222239bafe07' 2024-09-15T20:22:49.535 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-15T20:22:49.549 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQAZQudm90mVIBAAWQbd44Vd8K10jPh7H/fGbw== 2024-09-15T20:22:49.549 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQAZQudm90mVIBAAWQbd44Vd8K10jPh7H/fGbw=="}' 2024-09-15T20:22:49.549 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 8807c1df-6e7e-47fb-94e8-222239bafe07 -i td/osd-recovery-scrub/0/new.json 2024-09-15T20:22:49.753 INFO:tasks.workunit.client.0.smithi136.stdout:0 2024-09-15T20:22:49.762 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-recovery-scrub/0/new.json 2024-09-15T20:22:49.763 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 0 --fsid=a80d5a97-0ed0-4dbf-8fa5-41ae5f0552a3 --auth-supported=none --mon-host=127.0.0.1:7124 --osd-op-queue=wpq --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-scrub/0 --osd-journal=td/osd-recovery-scrub/0/journal --chdir= --run-dir=td/osd-recovery-scrub '--admin-socket=/tmp/ceph-asok.43761/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-scrub/$name.log' '--pid-file=td/osd-recovery-scrub/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_scrub_during_recovery=false --mkfs --key AQAZQudm90mVIBAAWQbd44Vd8K10jPh7H/fGbw== --osd-uuid 8807c1df-6e7e-47fb-94e8-222239bafe07 2024-09-15T20:22:49.793 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:22:49.791+0000 7f24d0087640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T20:22:49.803 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:22:49.801+0000 7f24d0087640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T20:22:49.804 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:22:49.803+0000 7f24d0087640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T20:22:49.805 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:22:49.803+0000 7f24d0087640 -1 bdev(0x55cbe682bc00 td/osd-recovery-scrub/0/block) open stat got: (1) Operation not permitted 2024-09-15T20:22:49.805 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:22:49.803+0000 7f24d0087640 -1 bluestore(td/osd-recovery-scrub/0) _read_fsid unparsable uuid 2024-09-15T20:22:55.793 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-recovery-scrub/0/keyring 2024-09-15T20:22:55.794 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-15T20:22:55.794 INFO:tasks.workunit.client.0.smithi136.stdout:adding osd0 key to auth repository 2024-09-15T20:22:55.795 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd0 key to auth repository 2024-09-15T20:22:55.795 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-recovery-scrub/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-15T20:22:56.158 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.0 2024-09-15T20:22:56.159 INFO:tasks.workunit.client.0.smithi136.stdout:start osd.0 2024-09-15T20:22:56.159 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 0 --fsid=a80d5a97-0ed0-4dbf-8fa5-41ae5f0552a3 --auth-supported=none --mon-host=127.0.0.1:7124 --osd-op-queue=wpq --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-scrub/0 --osd-journal=td/osd-recovery-scrub/0/journal --chdir= --run-dir=td/osd-recovery-scrub '--admin-socket=/tmp/ceph-asok.43761/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-scrub/$name.log' '--pid-file=td/osd-recovery-scrub/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_scrub_during_recovery=false 2024-09-15T20:22:56.159 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-15T20:22:56.159 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-15T20:22:56.159 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-15T20:22:56.185 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:22:56.184+0000 7fe917c1f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T20:22:56.188 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:22:56.186+0000 7fe917c1f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T20:22:56.193 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:22:56.187+0000 7fe917c1f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T20:22:56.433 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 0 2024-09-15T20:22:56.433 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-15T20:22:56.433 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2024-09-15T20:22:56.433 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-15T20:22:56.434 INFO:tasks.workunit.client.0.smithi136.stdout:0 2024-09-15T20:22:56.434 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-15T20:22:56.434 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:22:56.434 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-15T20:22:56.434 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:22:56.434 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T20:22:56.709 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:22:57.554 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:22:57.552+0000 7fe917c1f640 -1 Falling back to public interface 2024-09-15T20:22:57.710 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:22:57.710 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:22:57.711 INFO:tasks.workunit.client.0.smithi136.stdout:1 2024-09-15T20:22:57.711 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-15T20:22:57.711 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:22:57.711 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T20:22:57.995 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:22:58.997 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:22:58.997 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:22:58.997 INFO:tasks.workunit.client.0.smithi136.stdout:2 2024-09-15T20:22:58.997 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-15T20:22:58.998 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:22:58.998 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T20:22:59.287 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:23:00.195 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:23:00.193+0000 7fe917c1f640 -1 osd.0 0 log_to_monitors true 2024-09-15T20:23:00.289 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:23:00.289 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:23:00.289 INFO:tasks.workunit.client.0.smithi136.stdout:3 2024-09-15T20:23:00.290 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-15T20:23:00.290 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:23:00.290 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T20:23:00.671 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:23:01.673 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:23:01.673 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:23:01.673 INFO:tasks.workunit.client.0.smithi136.stdout:4 2024-09-15T20:23:01.674 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-15T20:23:01.674 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:23:01.674 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T20:23:01.955 INFO:tasks.workunit.client.0.smithi136.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/3401025102,v1:127.0.0.1:6803/3401025102] [v2:127.0.0.1:6804/3401025102,v1:127.0.0.1:6805/3401025102] exists,up 8807c1df-6e7e-47fb-94e8-222239bafe07 2024-09-15T20:23:01.959 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-15T20:23:01.959 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-15T20:23:01.959 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-15T20:23:01.959 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:55: TEST_recovery_scrub_1: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-09-15T20:23:01.959 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:57: TEST_recovery_scrub_1: run_osd td/osd-recovery-scrub 1 --osd_scrub_during_recovery=false 2024-09-15T20:23:01.959 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-recovery-scrub 2024-09-15T20:23:01.959 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-15T20:23:01.959 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=1 2024-09-15T20:23:01.959 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-15T20:23:01.959 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-recovery-scrub/1 2024-09-15T20:23:01.959 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=a80d5a97-0ed0-4dbf-8fa5-41ae5f0552a3 --auth-supported=none --mon-host=127.0.0.1:7124 --osd-op-queue=wpq ' 2024-09-15T20:23:01.960 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T20:23:01.960 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T20:23:01.960 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T20:23:01.960 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-recovery-scrub/1' 2024-09-15T20:23:01.960 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-scrub/1/journal' 2024-09-15T20:23:01.960 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-15T20:23:01.960 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-15T20:23:01.960 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-recovery-scrub' 2024-09-15T20:23:01.960 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-15T20:23:01.960 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-15T20:23:01.960 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-15T20:23:01.960 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-15T20:23:01.960 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T20:23:01.960 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.43761 2024-09-15T20:23:01.961 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.43761/$cluster-$name.asok' 2024-09-15T20:23:01.961 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.43761/$cluster-$name.asok' 2024-09-15T20:23:01.961 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-15T20:23:01.961 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-15T20:23:01.961 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-15T20:23:01.961 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-recovery-scrub/$name.log' 2024-09-15T20:23:01.961 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-recovery-scrub/$name.pid' 2024-09-15T20:23:01.961 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T20:23:01.961 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T20:23:01.961 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T20:23:01.961 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T20:23:01.961 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-15T20:23:01.961 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=--osd_scrub_during_recovery=false 2024-09-15T20:23:01.961 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-recovery-scrub/1 2024-09-15T20:23:01.962 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-15T20:23:01.962 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=8683755b-aa72-4cb6-97fe-396bad87485b 2024-09-15T20:23:01.962 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd1 8683755b-aa72-4cb6-97fe-396bad87485b' 2024-09-15T20:23:01.962 INFO:tasks.workunit.client.0.smithi136.stdout:add osd1 8683755b-aa72-4cb6-97fe-396bad87485b 2024-09-15T20:23:01.962 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-15T20:23:01.977 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQAlQudmDtkiOhAACKsI+dy13bw/ruoys5Q5Jw== 2024-09-15T20:23:01.977 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQAlQudmDtkiOhAACKsI+dy13bw/ruoys5Q5Jw=="}' 2024-09-15T20:23:01.977 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 8683755b-aa72-4cb6-97fe-396bad87485b -i td/osd-recovery-scrub/1/new.json 2024-09-15T20:23:02.339 INFO:tasks.workunit.client.0.smithi136.stdout:1 2024-09-15T20:23:02.350 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-recovery-scrub/1/new.json 2024-09-15T20:23:02.351 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 1 --fsid=a80d5a97-0ed0-4dbf-8fa5-41ae5f0552a3 --auth-supported=none --mon-host=127.0.0.1:7124 --osd-op-queue=wpq --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-scrub/1 --osd-journal=td/osd-recovery-scrub/1/journal --chdir= --run-dir=td/osd-recovery-scrub '--admin-socket=/tmp/ceph-asok.43761/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-scrub/$name.log' '--pid-file=td/osd-recovery-scrub/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_scrub_during_recovery=false --mkfs --key AQAlQudmDtkiOhAACKsI+dy13bw/ruoys5Q5Jw== --osd-uuid 8683755b-aa72-4cb6-97fe-396bad87485b 2024-09-15T20:23:02.381 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:23:02.379+0000 7f063422a640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T20:23:02.431 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:23:02.429+0000 7f063422a640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T20:23:02.433 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:23:02.430+0000 7f063422a640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T20:23:02.433 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:23:02.431+0000 7f063422a640 -1 bdev(0x55be66bb3c00 td/osd-recovery-scrub/1/block) open stat got: (1) Operation not permitted 2024-09-15T20:23:02.434 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:23:02.431+0000 7f063422a640 -1 bluestore(td/osd-recovery-scrub/1) _read_fsid unparsable uuid 2024-09-15T20:23:07.910 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-recovery-scrub/1/keyring 2024-09-15T20:23:07.911 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-15T20:23:07.912 INFO:tasks.workunit.client.0.smithi136.stdout:adding osd1 key to auth repository 2024-09-15T20:23:07.912 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd1 key to auth repository 2024-09-15T20:23:07.912 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-recovery-scrub/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-15T20:23:08.273 INFO:tasks.workunit.client.0.smithi136.stdout:start osd.1 2024-09-15T20:23:08.274 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.1 2024-09-15T20:23:08.275 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 1 --fsid=a80d5a97-0ed0-4dbf-8fa5-41ae5f0552a3 --auth-supported=none --mon-host=127.0.0.1:7124 --osd-op-queue=wpq --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-scrub/1 --osd-journal=td/osd-recovery-scrub/1/journal --chdir= --run-dir=td/osd-recovery-scrub '--admin-socket=/tmp/ceph-asok.43761/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-scrub/$name.log' '--pid-file=td/osd-recovery-scrub/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_scrub_during_recovery=false 2024-09-15T20:23:08.275 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-15T20:23:08.275 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-15T20:23:08.275 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-15T20:23:08.301 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:23:08.299+0000 7f097f941640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T20:23:08.303 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:23:08.301+0000 7f097f941640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T20:23:08.304 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:23:08.302+0000 7f097f941640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T20:23:08.550 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 1 2024-09-15T20:23:08.550 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-15T20:23:08.550 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2024-09-15T20:23:08.550 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-15T20:23:08.550 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-15T20:23:08.550 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:23:08.550 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-15T20:23:08.551 INFO:tasks.workunit.client.0.smithi136.stdout:0 2024-09-15T20:23:08.551 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:23:08.551 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T20:23:08.825 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:23:09.674 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:23:09.671+0000 7f097f941640 -1 Falling back to public interface 2024-09-15T20:23:09.827 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:23:09.827 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:23:09.827 INFO:tasks.workunit.client.0.smithi136.stdout:1 2024-09-15T20:23:09.827 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-15T20:23:09.828 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:23:09.828 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T20:23:10.113 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:23:11.114 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:23:11.114 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:23:11.114 INFO:tasks.workunit.client.0.smithi136.stdout:2 2024-09-15T20:23:11.115 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-15T20:23:11.115 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:23:11.115 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T20:23:11.400 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:23:12.401 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:23:12.401 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:23:12.402 INFO:tasks.workunit.client.0.smithi136.stdout:3 2024-09-15T20:23:12.402 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-15T20:23:12.402 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:23:12.402 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T20:23:12.683 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:23:12.853 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:23:12.851+0000 7f097f941640 -1 osd.1 0 log_to_monitors true 2024-09-15T20:23:13.685 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:23:13.685 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:23:13.685 INFO:tasks.workunit.client.0.smithi136.stdout:4 2024-09-15T20:23:13.685 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-15T20:23:13.686 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:23:13.686 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T20:23:14.001 INFO:tasks.workunit.client.0.smithi136.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/82005435,v1:127.0.0.1:6811/82005435] [v2:127.0.0.1:6812/82005435,v1:127.0.0.1:6813/82005435] exists,up 8683755b-aa72-4cb6-97fe-396bad87485b 2024-09-15T20:23:14.005 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-15T20:23:14.005 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-15T20:23:14.005 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-15T20:23:14.006 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:55: TEST_recovery_scrub_1: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-09-15T20:23:14.006 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:57: TEST_recovery_scrub_1: run_osd td/osd-recovery-scrub 2 --osd_scrub_during_recovery=false 2024-09-15T20:23:14.006 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-recovery-scrub 2024-09-15T20:23:14.006 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-15T20:23:14.006 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=2 2024-09-15T20:23:14.006 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-15T20:23:14.006 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-recovery-scrub/2 2024-09-15T20:23:14.006 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=a80d5a97-0ed0-4dbf-8fa5-41ae5f0552a3 --auth-supported=none --mon-host=127.0.0.1:7124 --osd-op-queue=wpq ' 2024-09-15T20:23:14.006 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T20:23:14.006 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T20:23:14.006 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T20:23:14.007 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-recovery-scrub/2' 2024-09-15T20:23:14.007 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-scrub/2/journal' 2024-09-15T20:23:14.007 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-15T20:23:14.007 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-15T20:23:14.007 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-recovery-scrub' 2024-09-15T20:23:14.007 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-15T20:23:14.007 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-15T20:23:14.007 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-15T20:23:14.007 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-15T20:23:14.007 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T20:23:14.007 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.43761 2024-09-15T20:23:14.007 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.43761/$cluster-$name.asok' 2024-09-15T20:23:14.007 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.43761/$cluster-$name.asok' 2024-09-15T20:23:14.007 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-15T20:23:14.008 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-15T20:23:14.008 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-15T20:23:14.008 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-recovery-scrub/$name.log' 2024-09-15T20:23:14.008 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-recovery-scrub/$name.pid' 2024-09-15T20:23:14.008 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T20:23:14.008 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T20:23:14.008 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T20:23:14.008 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T20:23:14.008 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-15T20:23:14.008 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=--osd_scrub_during_recovery=false 2024-09-15T20:23:14.009 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-recovery-scrub/2 2024-09-15T20:23:14.009 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-15T20:23:14.009 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=c08ac09c-a387-42b1-9faf-b5527dd10a7c 2024-09-15T20:23:14.009 INFO:tasks.workunit.client.0.smithi136.stdout:add osd2 c08ac09c-a387-42b1-9faf-b5527dd10a7c 2024-09-15T20:23:14.009 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd2 c08ac09c-a387-42b1-9faf-b5527dd10a7c' 2024-09-15T20:23:14.009 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-15T20:23:14.024 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQAyQudmOhhNARAAf7XBpcXsxeMMSYM03QGuEQ== 2024-09-15T20:23:14.024 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQAyQudmOhhNARAAf7XBpcXsxeMMSYM03QGuEQ=="}' 2024-09-15T20:23:14.024 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new c08ac09c-a387-42b1-9faf-b5527dd10a7c -i td/osd-recovery-scrub/2/new.json 2024-09-15T20:23:14.375 INFO:tasks.workunit.client.0.smithi136.stdout:2 2024-09-15T20:23:14.386 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-recovery-scrub/2/new.json 2024-09-15T20:23:14.387 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 2 --fsid=a80d5a97-0ed0-4dbf-8fa5-41ae5f0552a3 --auth-supported=none --mon-host=127.0.0.1:7124 --osd-op-queue=wpq --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-scrub/2 --osd-journal=td/osd-recovery-scrub/2/journal --chdir= --run-dir=td/osd-recovery-scrub '--admin-socket=/tmp/ceph-asok.43761/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-scrub/$name.log' '--pid-file=td/osd-recovery-scrub/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_scrub_during_recovery=false --mkfs --key AQAyQudmOhhNARAAf7XBpcXsxeMMSYM03QGuEQ== --osd-uuid c08ac09c-a387-42b1-9faf-b5527dd10a7c 2024-09-15T20:23:14.416 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:23:14.414+0000 7f625c018640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T20:23:14.459 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:23:14.457+0000 7f625c018640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T20:23:14.460 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:23:14.459+0000 7f625c018640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T20:23:14.461 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:23:14.459+0000 7f625c018640 -1 bdev(0x55606c77fc00 td/osd-recovery-scrub/2/block) open stat got: (1) Operation not permitted 2024-09-15T20:23:14.461 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:23:14.459+0000 7f625c018640 -1 bluestore(td/osd-recovery-scrub/2) _read_fsid unparsable uuid 2024-09-15T20:23:20.516 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-recovery-scrub/2/keyring 2024-09-15T20:23:20.516 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-15T20:23:20.517 INFO:tasks.workunit.client.0.smithi136.stdout:adding osd2 key to auth repository 2024-09-15T20:23:20.517 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd2 key to auth repository 2024-09-15T20:23:20.517 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-recovery-scrub/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-15T20:23:20.874 INFO:tasks.workunit.client.0.smithi136.stdout:start osd.2 2024-09-15T20:23:20.874 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.2 2024-09-15T20:23:20.874 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 2 --fsid=a80d5a97-0ed0-4dbf-8fa5-41ae5f0552a3 --auth-supported=none --mon-host=127.0.0.1:7124 --osd-op-queue=wpq --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-scrub/2 --osd-journal=td/osd-recovery-scrub/2/journal --chdir= --run-dir=td/osd-recovery-scrub '--admin-socket=/tmp/ceph-asok.43761/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-scrub/$name.log' '--pid-file=td/osd-recovery-scrub/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_scrub_during_recovery=false 2024-09-15T20:23:20.875 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-15T20:23:20.875 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-15T20:23:20.875 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-15T20:23:20.902 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:23:20.900+0000 7fd43da0b640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T20:23:20.904 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:23:20.903+0000 7fd43da0b640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T20:23:20.905 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:23:20.904+0000 7fd43da0b640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T20:23:21.151 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 2 2024-09-15T20:23:21.152 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-15T20:23:21.152 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=2 2024-09-15T20:23:21.152 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-15T20:23:21.152 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-15T20:23:21.152 INFO:tasks.workunit.client.0.smithi136.stdout:0 2024-09-15T20:23:21.153 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:23:21.153 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-15T20:23:21.153 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:23:21.153 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T20:23:21.464 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:23:22.270 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:23:22.268+0000 7fd43da0b640 -1 Falling back to public interface 2024-09-15T20:23:22.465 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:23:22.466 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:23:22.466 INFO:tasks.workunit.client.0.smithi136.stdout:1 2024-09-15T20:23:22.466 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-15T20:23:22.466 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:23:22.466 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T20:23:22.743 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:23:23.745 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:23:23.745 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:23:23.745 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-15T20:23:23.745 INFO:tasks.workunit.client.0.smithi136.stdout:2 2024-09-15T20:23:23.745 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:23:23.746 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T20:23:24.024 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:23:24.917 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:23:24.915+0000 7fd43da0b640 -1 osd.2 0 log_to_monitors true 2024-09-15T20:23:25.026 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:23:25.026 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:23:25.026 INFO:tasks.workunit.client.0.smithi136.stdout:3 2024-09-15T20:23:25.027 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-15T20:23:25.027 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:23:25.027 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T20:23:25.402 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:23:26.403 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:23:26.404 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:23:26.404 INFO:tasks.workunit.client.0.smithi136.stdout:4 2024-09-15T20:23:26.404 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-15T20:23:26.405 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:23:26.405 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T20:23:26.688 INFO:tasks.workunit.client.0.smithi136.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/3535125569,v1:127.0.0.1:6819/3535125569] [v2:127.0.0.1:6820/3535125569,v1:127.0.0.1:6821/3535125569] exists,up c08ac09c-a387-42b1-9faf-b5527dd10a7c 2024-09-15T20:23:26.692 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-15T20:23:26.692 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-15T20:23:26.692 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-15T20:23:26.692 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:55: TEST_recovery_scrub_1: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-09-15T20:23:26.692 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:57: TEST_recovery_scrub_1: run_osd td/osd-recovery-scrub 3 --osd_scrub_during_recovery=false 2024-09-15T20:23:26.693 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-recovery-scrub 2024-09-15T20:23:26.693 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-15T20:23:26.693 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=3 2024-09-15T20:23:26.693 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-15T20:23:26.693 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-recovery-scrub/3 2024-09-15T20:23:26.693 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=a80d5a97-0ed0-4dbf-8fa5-41ae5f0552a3 --auth-supported=none --mon-host=127.0.0.1:7124 --osd-op-queue=wpq ' 2024-09-15T20:23:26.693 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T20:23:26.693 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T20:23:26.693 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T20:23:26.693 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-recovery-scrub/3' 2024-09-15T20:23:26.693 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-scrub/3/journal' 2024-09-15T20:23:26.694 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-15T20:23:26.694 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-15T20:23:26.694 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-recovery-scrub' 2024-09-15T20:23:26.694 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-15T20:23:26.694 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-15T20:23:26.694 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-15T20:23:26.694 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-15T20:23:26.694 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T20:23:26.694 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.43761 2024-09-15T20:23:26.694 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.43761/$cluster-$name.asok' 2024-09-15T20:23:26.694 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.43761/$cluster-$name.asok' 2024-09-15T20:23:26.694 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-15T20:23:26.695 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-15T20:23:26.695 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-15T20:23:26.695 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-recovery-scrub/$name.log' 2024-09-15T20:23:26.695 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-recovery-scrub/$name.pid' 2024-09-15T20:23:26.695 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T20:23:26.695 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T20:23:26.695 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T20:23:26.695 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T20:23:26.695 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-15T20:23:26.695 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=--osd_scrub_during_recovery=false 2024-09-15T20:23:26.695 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-recovery-scrub/3 2024-09-15T20:23:26.695 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-15T20:23:26.696 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=4554795b-4da1-44ed-aa67-d6029b043f04 2024-09-15T20:23:26.696 INFO:tasks.workunit.client.0.smithi136.stdout:add osd3 4554795b-4da1-44ed-aa67-d6029b043f04 2024-09-15T20:23:26.697 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd3 4554795b-4da1-44ed-aa67-d6029b043f04' 2024-09-15T20:23:26.702 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-15T20:23:26.711 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQA+Qudmu21AKhAAFrpQnf1/RFn7xajjmwPUQA== 2024-09-15T20:23:26.711 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQA+Qudmu21AKhAAFrpQnf1/RFn7xajjmwPUQA=="}' 2024-09-15T20:23:26.711 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 4554795b-4da1-44ed-aa67-d6029b043f04 -i td/osd-recovery-scrub/3/new.json 2024-09-15T20:23:27.071 INFO:tasks.workunit.client.0.smithi136.stdout:3 2024-09-15T20:23:27.083 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-recovery-scrub/3/new.json 2024-09-15T20:23:27.084 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 3 --fsid=a80d5a97-0ed0-4dbf-8fa5-41ae5f0552a3 --auth-supported=none --mon-host=127.0.0.1:7124 --osd-op-queue=wpq --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-scrub/3 --osd-journal=td/osd-recovery-scrub/3/journal --chdir= --run-dir=td/osd-recovery-scrub '--admin-socket=/tmp/ceph-asok.43761/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-scrub/$name.log' '--pid-file=td/osd-recovery-scrub/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_scrub_during_recovery=false --mkfs --key AQA+Qudmu21AKhAAFrpQnf1/RFn7xajjmwPUQA== --osd-uuid 4554795b-4da1-44ed-aa67-d6029b043f04 2024-09-15T20:23:27.114 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:23:27.112+0000 7f649a49c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T20:23:27.156 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:23:27.154+0000 7f649a49c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T20:23:27.157 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:23:27.155+0000 7f649a49c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T20:23:27.157 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:23:27.155+0000 7f649a49c640 -1 bdev(0x55ee5e397c00 td/osd-recovery-scrub/3/block) open stat got: (1) Operation not permitted 2024-09-15T20:23:27.157 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:23:27.156+0000 7f649a49c640 -1 bluestore(td/osd-recovery-scrub/3) _read_fsid unparsable uuid 2024-09-15T20:23:33.666 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-recovery-scrub/3/keyring 2024-09-15T20:23:33.666 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-15T20:23:33.667 INFO:tasks.workunit.client.0.smithi136.stdout:adding osd3 key to auth repository 2024-09-15T20:23:33.667 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd3 key to auth repository 2024-09-15T20:23:33.667 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-recovery-scrub/3/keyring auth add osd.3 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-15T20:23:34.024 INFO:tasks.workunit.client.0.smithi136.stdout:start osd.3 2024-09-15T20:23:34.025 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.3 2024-09-15T20:23:34.025 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 3 --fsid=a80d5a97-0ed0-4dbf-8fa5-41ae5f0552a3 --auth-supported=none --mon-host=127.0.0.1:7124 --osd-op-queue=wpq --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-scrub/3 --osd-journal=td/osd-recovery-scrub/3/journal --chdir= --run-dir=td/osd-recovery-scrub '--admin-socket=/tmp/ceph-asok.43761/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-scrub/$name.log' '--pid-file=td/osd-recovery-scrub/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_scrub_during_recovery=false 2024-09-15T20:23:34.025 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-15T20:23:34.025 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-15T20:23:34.025 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-15T20:23:34.052 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:23:34.050+0000 7f270caae640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T20:23:34.054 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:23:34.052+0000 7f270caae640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T20:23:34.055 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:23:34.054+0000 7f270caae640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T20:23:34.301 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 3 2024-09-15T20:23:34.301 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-15T20:23:34.301 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=3 2024-09-15T20:23:34.301 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-15T20:23:34.301 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-15T20:23:34.301 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:23:34.302 INFO:tasks.workunit.client.0.smithi136.stdout:0 2024-09-15T20:23:34.302 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-15T20:23:34.302 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:23:34.302 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-15T20:23:34.578 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:23:35.579 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:23:35.579 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:23:35.580 INFO:tasks.workunit.client.0.smithi136.stdout:1 2024-09-15T20:23:35.580 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-15T20:23:35.580 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:23:35.580 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-15T20:23:35.686 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:23:35.685+0000 7f270caae640 -1 Falling back to public interface 2024-09-15T20:23:35.870 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:23:36.871 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:23:36.871 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:23:36.872 INFO:tasks.workunit.client.0.smithi136.stdout:2 2024-09-15T20:23:36.872 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-15T20:23:36.872 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:23:36.872 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-15T20:23:37.158 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:23:38.159 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:23:38.159 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:23:38.159 INFO:tasks.workunit.client.0.smithi136.stdout:3 2024-09-15T20:23:38.160 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-15T20:23:38.160 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:23:38.160 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-15T20:23:38.290 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:23:38.288+0000 7f270caae640 -1 osd.3 0 log_to_monitors true 2024-09-15T20:23:38.498 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:23:39.500 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:23:39.500 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:23:39.500 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-15T20:23:39.500 INFO:tasks.workunit.client.0.smithi136.stdout:4 2024-09-15T20:23:39.501 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:23:39.501 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-15T20:23:39.784 INFO:tasks.workunit.client.0.smithi136.stdout:osd.3 up in weight 1 up_from 20 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6826/2355698142,v1:127.0.0.1:6827/2355698142] [v2:127.0.0.1:6828/2355698142,v1:127.0.0.1:6829/2355698142] exists,up 4554795b-4da1-44ed-aa67-d6029b043f04 2024-09-15T20:23:39.785 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-15T20:23:39.785 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-15T20:23:39.785 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-15T20:23:39.785 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:61: TEST_recovery_scrub_1: create_pool test 1 1 2024-09-15T20:23:39.785 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create test 1 1 2024-09-15T20:23:40.177 INFO:tasks.workunit.client.0.smithi136.stderr:pool 'test' created 2024-09-15T20:23:40.188 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2024-09-15T20:23:41.190 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:62: TEST_recovery_scrub_1: wait_for_clean 2024-09-15T20:23:41.190 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-15T20:23:41.190 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-15T20:23:41.190 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-15T20:23:41.191 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-15T20:23:41.191 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-15T20:23:41.191 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-15T20:23:41.191 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-15T20:23:41.191 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-15T20:23:41.191 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-15T20:23:41.280 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-15T20:23:41.281 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-15T20:23:41.281 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-15T20:23:41.281 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-15T20:23:41.281 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-15T20:23:41.281 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-15T20:23:41.569 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-15T20:23:41.569 INFO:tasks.workunit.client.0.smithi136.stderr:1 2024-09-15T20:23:41.569 INFO:tasks.workunit.client.0.smithi136.stderr:2 2024-09-15T20:23:41.569 INFO:tasks.workunit.client.0.smithi136.stderr:3' 2024-09-15T20:23:41.569 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-15T20:23:41.569 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T20:23:41.569 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-15T20:23:41.721 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836490 2024-09-15T20:23:41.721 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836490 2024-09-15T20:23:41.721 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836490' 2024-09-15T20:23:41.721 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T20:23:41.721 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-15T20:23:41.824 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=42949672967 2024-09-15T20:23:41.824 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 42949672967 2024-09-15T20:23:41.824 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836490 1-42949672967' 2024-09-15T20:23:41.824 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T20:23:41.824 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-15T20:23:41.926 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=64424509445 2024-09-15T20:23:41.926 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 64424509445 2024-09-15T20:23:41.926 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836490 1-42949672967 2-64424509445' 2024-09-15T20:23:41.926 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T20:23:41.927 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-15T20:23:42.028 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=85899345922 2024-09-15T20:23:42.028 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 85899345922 2024-09-15T20:23:42.029 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836490 1-42949672967 2-64424509445 3-85899345922' 2024-09-15T20:23:42.029 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T20:23:42.029 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836490 2024-09-15T20:23:42.029 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T20:23:42.030 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-15T20:23:42.030 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836490 2024-09-15T20:23:42.030 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T20:23:42.031 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836490 2024-09-15T20:23:42.031 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836490' 2024-09-15T20:23:42.032 INFO:tasks.workunit.client.0.smithi136.stdout:waiting osd.0 seq 21474836490 2024-09-15T20:23:42.032 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T20:23:42.308 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836488 -lt 21474836490 2024-09-15T20:23:42.308 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-15T20:23:43.309 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-15T20:23:43.309 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T20:23:43.597 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836490 -lt 21474836490 2024-09-15T20:23:43.597 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T20:23:43.597 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-42949672967 2024-09-15T20:23:43.597 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T20:23:43.598 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-15T20:23:43.599 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-42949672967 2024-09-15T20:23:43.599 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T20:23:43.600 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=42949672967 2024-09-15T20:23:43.600 INFO:tasks.workunit.client.0.smithi136.stdout:waiting osd.1 seq 42949672967 2024-09-15T20:23:43.600 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 42949672967' 2024-09-15T20:23:43.601 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-15T20:23:43.918 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 42949672967 -lt 42949672967 2024-09-15T20:23:43.918 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T20:23:43.919 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-64424509445 2024-09-15T20:23:43.919 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T20:23:43.920 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-15T20:23:43.921 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-64424509445 2024-09-15T20:23:43.921 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T20:23:43.922 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=64424509445 2024-09-15T20:23:43.922 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 64424509445' 2024-09-15T20:23:43.922 INFO:tasks.workunit.client.0.smithi136.stdout:waiting osd.2 seq 64424509445 2024-09-15T20:23:43.922 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-15T20:23:44.197 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 64424509445 -lt 64424509445 2024-09-15T20:23:44.198 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T20:23:44.199 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 3-85899345922 2024-09-15T20:23:44.199 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T20:23:44.199 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=3 2024-09-15T20:23:44.200 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 3-85899345922 2024-09-15T20:23:44.200 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T20:23:44.201 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=85899345922 2024-09-15T20:23:44.201 INFO:tasks.workunit.client.0.smithi136.stdout:waiting osd.3 seq 85899345922 2024-09-15T20:23:44.202 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.3 seq 85899345922' 2024-09-15T20:23:44.202 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-15T20:23:44.484 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 85899345922 -lt 85899345922 2024-09-15T20:23:44.485 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-15T20:23:44.485 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T20:23:44.485 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T20:23:44.832 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 1 == 0 2024-09-15T20:23:44.832 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-15T20:23:44.833 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-15T20:23:44.833 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-15T20:23:44.833 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T20:23:44.833 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T20:23:44.833 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T20:23:44.833 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T20:23:45.105 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=1 2024-09-15T20:23:45.106 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-15T20:23:45.106 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T20:23:45.106 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T20:23:45.448 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 = 1 2024-09-15T20:23:45.448 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-15T20:23:45.448 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-15T20:23:45.449 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:63: TEST_recovery_scrub_1: ceph osd dump 2024-09-15T20:23:45.449 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:63: TEST_recovery_scrub_1: grep '^pool.*['\'']test['\'']' 2024-09-15T20:23:45.449 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:63: TEST_recovery_scrub_1: awk '{ print $2 }' 2024-09-15T20:23:45.740 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:63: TEST_recovery_scrub_1: poolid=1 2024-09-15T20:23:45.740 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:65: TEST_recovery_scrub_1: ceph pg dump pgs 2024-09-15T20:23:46.002 INFO:tasks.workunit.client.0.smithi136.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-15T20:23:46.002 INFO:tasks.workunit.client.0.smithi136.stdout:1.0 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-09-15T20:23:40.763237+0000 0'0 24:12 [1] 1 [1] 1 0'0 2024-09-15T20:23:40.125794+0000 0'0 2024-09-15T20:23:40.125794+0000 0 0 periodic scrub scheduled @ 2024-09-16T20:42:26.632137+0000 0 0 2024-09-15T20:23:46.002 INFO:tasks.workunit.client.0.smithi136.stdout: 2024-09-15T20:23:46.002 INFO:tasks.workunit.client.0.smithi136.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-15T20:23:46.002 INFO:tasks.workunit.client.0.smithi136.stderr:dumped pgs 2024-09-15T20:23:46.014 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:67: TEST_recovery_scrub_1: dd if=/dev/urandom of=testdata.43761 bs=1M count=50 2024-09-15T20:23:46.153 INFO:tasks.workunit.client.0.smithi136.stderr:50+0 records in 2024-09-15T20:23:46.153 INFO:tasks.workunit.client.0.smithi136.stderr:50+0 records out 2024-09-15T20:23:46.154 INFO:tasks.workunit.client.0.smithi136.stderr:52428800 bytes (52 MB, 50 MiB) copied, 0.138587 s, 378 MB/s 2024-09-15T20:23:46.154 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: seq 1 100 2024-09-15T20:23:46.155 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-09-15T20:23:46.155 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj1 testdata.43761 2024-09-15T20:23:47.793 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-09-15T20:23:47.793 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj2 testdata.43761 2024-09-15T20:23:49.274 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-09-15T20:23:49.274 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj3 testdata.43761 2024-09-15T20:23:50.888 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-09-15T20:23:50.888 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj4 testdata.43761 2024-09-15T20:23:52.628 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-09-15T20:23:52.628 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj5 testdata.43761 2024-09-15T20:23:54.226 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-09-15T20:23:54.226 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj6 testdata.43761 2024-09-15T20:23:55.874 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-09-15T20:23:55.875 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj7 testdata.43761 2024-09-15T20:23:57.380 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-09-15T20:23:57.380 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj8 testdata.43761 2024-09-15T20:23:58.969 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-09-15T20:23:58.970 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj9 testdata.43761 2024-09-15T20:24:00.535 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-09-15T20:24:00.535 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj10 testdata.43761 2024-09-15T20:24:02.075 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-09-15T20:24:02.075 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj11 testdata.43761 2024-09-15T20:24:03.655 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-09-15T20:24:03.655 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj12 testdata.43761 2024-09-15T20:24:05.152 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-09-15T20:24:05.152 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj13 testdata.43761 2024-09-15T20:24:06.666 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-09-15T20:24:06.666 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj14 testdata.43761 2024-09-15T20:24:08.247 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-09-15T20:24:08.247 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj15 testdata.43761 2024-09-15T20:24:09.710 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-09-15T20:24:09.711 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj16 testdata.43761 2024-09-15T20:24:11.091 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-09-15T20:24:11.091 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj17 testdata.43761 2024-09-15T20:24:12.713 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-09-15T20:24:12.713 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj18 testdata.43761 2024-09-15T20:24:14.210 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-09-15T20:24:14.210 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj19 testdata.43761 2024-09-15T20:24:15.624 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-09-15T20:24:15.624 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj20 testdata.43761 2024-09-15T20:24:17.045 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-09-15T20:24:17.045 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj21 testdata.43761 2024-09-15T20:24:18.843 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-09-15T20:24:18.843 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj22 testdata.43761 2024-09-15T20:24:20.357 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-09-15T20:24:20.357 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj23 testdata.43761 2024-09-15T20:24:21.838 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-09-15T20:24:21.838 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj24 testdata.43761 2024-09-15T20:24:23.243 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-09-15T20:24:23.243 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj25 testdata.43761 2024-09-15T20:24:24.748 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-09-15T20:24:24.748 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj26 testdata.43761 2024-09-15T20:24:26.228 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-09-15T20:24:26.229 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj27 testdata.43761 2024-09-15T20:24:27.576 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-09-15T20:24:27.576 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj28 testdata.43761 2024-09-15T20:24:29.056 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-09-15T20:24:29.056 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj29 testdata.43761 2024-09-15T20:24:30.603 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-09-15T20:24:30.603 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj30 testdata.43761 2024-09-15T20:24:32.158 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-09-15T20:24:32.158 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj31 testdata.43761 2024-09-15T20:24:33.664 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-09-15T20:24:33.664 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj32 testdata.43761 2024-09-15T20:24:35.077 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-09-15T20:24:35.077 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj33 testdata.43761 2024-09-15T20:24:36.641 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-09-15T20:24:36.641 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj34 testdata.43761 2024-09-15T20:24:38.238 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-09-15T20:24:38.239 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj35 testdata.43761 2024-09-15T20:24:39.894 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-09-15T20:24:39.895 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj36 testdata.43761 2024-09-15T20:24:41.341 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-09-15T20:24:41.341 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj37 testdata.43761 2024-09-15T20:24:42.888 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-09-15T20:24:42.888 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj38 testdata.43761 2024-09-15T20:24:44.477 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-09-15T20:24:44.477 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj39 testdata.43761 2024-09-15T20:24:46.066 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-09-15T20:24:46.066 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj40 testdata.43761 2024-09-15T20:24:47.463 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-09-15T20:24:47.463 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj41 testdata.43761 2024-09-15T20:24:48.985 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-09-15T20:24:48.985 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj42 testdata.43761 2024-09-15T20:24:50.449 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-09-15T20:24:50.449 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj43 testdata.43761 2024-09-15T20:24:51.996 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-09-15T20:24:51.996 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj44 testdata.43761 2024-09-15T20:24:53.393 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-09-15T20:24:53.393 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj45 testdata.43761 2024-09-15T20:24:54.965 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-09-15T20:24:54.965 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj46 testdata.43761 2024-09-15T20:24:56.503 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-09-15T20:24:56.504 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj47 testdata.43761 2024-09-15T20:24:58.076 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-09-15T20:24:58.076 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj48 testdata.43761 2024-09-15T20:24:59.556 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-09-15T20:24:59.556 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj49 testdata.43761 2024-09-15T20:25:01.078 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-09-15T20:25:01.078 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj50 testdata.43761 2024-09-15T20:25:02.609 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-09-15T20:25:02.609 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj51 testdata.43761 2024-09-15T20:25:04.181 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-09-15T20:25:04.181 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj52 testdata.43761 2024-09-15T20:25:05.594 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-09-15T20:25:05.595 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj53 testdata.43761 2024-09-15T20:25:07.010 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-09-15T20:25:07.010 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj54 testdata.43761 2024-09-15T20:25:08.517 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-09-15T20:25:08.517 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj55 testdata.43761 2024-09-15T20:25:10.081 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-09-15T20:25:10.082 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj56 testdata.43761 2024-09-15T20:25:11.445 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-09-15T20:25:11.446 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj57 testdata.43761 2024-09-15T20:25:13.027 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-09-15T20:25:13.027 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj58 testdata.43761 2024-09-15T20:25:14.608 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-09-15T20:25:14.608 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj59 testdata.43761 2024-09-15T20:25:16.056 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-09-15T20:25:16.057 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj60 testdata.43761 2024-09-15T20:25:17.429 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-09-15T20:25:17.429 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj61 testdata.43761 2024-09-15T20:25:18.926 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-09-15T20:25:18.926 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj62 testdata.43761 2024-09-15T20:25:20.417 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-09-15T20:25:20.417 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj63 testdata.43761 2024-09-15T20:25:21.956 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-09-15T20:25:21.956 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj64 testdata.43761 2024-09-15T20:25:23.362 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-09-15T20:25:23.362 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj65 testdata.43761 2024-09-15T20:25:24.943 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-09-15T20:25:24.944 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj66 testdata.43761 2024-09-15T20:25:26.458 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-09-15T20:25:26.458 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj67 testdata.43761 2024-09-15T20:25:27.939 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-09-15T20:25:27.939 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj68 testdata.43761 2024-09-15T20:25:29.303 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-09-15T20:25:29.303 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj69 testdata.43761 2024-09-15T20:25:30.810 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-09-15T20:25:30.810 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj70 testdata.43761 2024-09-15T20:25:32.358 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-09-15T20:25:32.358 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj71 testdata.43761 2024-09-15T20:25:33.897 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-09-15T20:25:33.897 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj72 testdata.43761 2024-09-15T20:25:35.286 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-09-15T20:25:35.286 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj73 testdata.43761 2024-09-15T20:25:36.801 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-09-15T20:25:36.801 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj74 testdata.43761 2024-09-15T20:25:38.323 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-09-15T20:25:38.323 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj75 testdata.43761 2024-09-15T20:25:39.905 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-09-15T20:25:39.905 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj76 testdata.43761 2024-09-15T20:25:41.286 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-09-15T20:25:41.286 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj77 testdata.43761 2024-09-15T20:25:42.800 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-09-15T20:25:42.800 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj78 testdata.43761 2024-09-15T20:25:44.332 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-09-15T20:25:44.332 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj79 testdata.43761 2024-09-15T20:25:45.955 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-09-15T20:25:45.955 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj80 testdata.43761 2024-09-15T20:25:47.326 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-09-15T20:25:47.327 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj81 testdata.43761 2024-09-15T20:25:48.933 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-09-15T20:25:48.933 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj82 testdata.43761 2024-09-15T20:25:50.515 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-09-15T20:25:50.515 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj83 testdata.43761 2024-09-15T20:25:52.062 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-09-15T20:25:52.063 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj84 testdata.43761 2024-09-15T20:25:53.493 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-09-15T20:25:53.493 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj85 testdata.43761 2024-09-15T20:25:55.008 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-09-15T20:25:55.008 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj86 testdata.43761 2024-09-15T20:25:56.639 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-09-15T20:25:56.640 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj87 testdata.43761 2024-09-15T20:25:58.212 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-09-15T20:25:58.212 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj88 testdata.43761 2024-09-15T20:25:59.752 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-09-15T20:25:59.752 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj89 testdata.43761 2024-09-15T20:26:01.207 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-09-15T20:26:01.207 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj90 testdata.43761 2024-09-15T20:26:02.822 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-09-15T20:26:02.822 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj91 testdata.43761 2024-09-15T20:26:04.378 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-09-15T20:26:04.378 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj92 testdata.43761 2024-09-15T20:26:05.952 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-09-15T20:26:05.952 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj93 testdata.43761 2024-09-15T20:26:07.332 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-09-15T20:26:07.332 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj94 testdata.43761 2024-09-15T20:26:08.899 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-09-15T20:26:08.899 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj95 testdata.43761 2024-09-15T20:26:10.446 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-09-15T20:26:10.447 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj96 testdata.43761 2024-09-15T20:26:12.011 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-09-15T20:26:12.011 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj97 testdata.43761 2024-09-15T20:26:13.408 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-09-15T20:26:13.409 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj98 testdata.43761 2024-09-15T20:26:14.919 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-09-15T20:26:14.919 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj99 testdata.43761 2024-09-15T20:26:16.475 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-09-15T20:26:16.475 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj100 testdata.43761 2024-09-15T20:26:18.024 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:72: TEST_recovery_scrub_1: rm -f testdata.43761 2024-09-15T20:26:18.030 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:74: TEST_recovery_scrub_1: ceph osd pool set test size 4 2024-09-15T20:26:18.421 INFO:tasks.workunit.client.0.smithi136.stderr:set pool 1 size to 4 2024-09-15T20:26:18.432 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:77: TEST_recovery_scrub_1: set -o pipefail 2024-09-15T20:26:18.433 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:78: TEST_recovery_scrub_1: count=0 2024-09-15T20:26:18.433 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:79: TEST_recovery_scrub_1: true 2024-09-15T20:26:18.433 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:81: TEST_recovery_scrub_1: ceph --format json pg dump pgs 2024-09-15T20:26:18.433 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:82: TEST_recovery_scrub_1: jq '.pg_stats | [.[] | .state | contains("recovering")]' 2024-09-15T20:26:18.434 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:82: TEST_recovery_scrub_1: grep -q true 2024-09-15T20:26:18.696 INFO:tasks.workunit.client.0.smithi136.stderr:dumped pgs 2024-09-15T20:26:18.708 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:86: TEST_recovery_scrub_1: sleep 2 2024-09-15T20:26:20.709 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:87: TEST_recovery_scrub_1: test 0 -eq 10 2024-09-15T20:26:20.710 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:92: TEST_recovery_scrub_1: expr 0 + 1 2024-09-15T20:26:20.711 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:92: TEST_recovery_scrub_1: count=1 2024-09-15T20:26:20.711 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:79: TEST_recovery_scrub_1: true 2024-09-15T20:26:20.712 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:81: TEST_recovery_scrub_1: ceph --format json pg dump pgs 2024-09-15T20:26:20.712 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:82: TEST_recovery_scrub_1: jq '.pg_stats | [.[] | .state | contains("recovering")]' 2024-09-15T20:26:20.712 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:82: TEST_recovery_scrub_1: grep -q true 2024-09-15T20:26:20.980 INFO:tasks.workunit.client.0.smithi136.stderr:dumped pgs 2024-09-15T20:26:20.992 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:86: TEST_recovery_scrub_1: sleep 2 2024-09-15T20:26:22.993 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:87: TEST_recovery_scrub_1: test 1 -eq 10 2024-09-15T20:26:22.994 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:92: TEST_recovery_scrub_1: expr 1 + 1 2024-09-15T20:26:22.995 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:92: TEST_recovery_scrub_1: count=2 2024-09-15T20:26:22.995 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:79: TEST_recovery_scrub_1: true 2024-09-15T20:26:22.996 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:81: TEST_recovery_scrub_1: ceph --format json pg dump pgs 2024-09-15T20:26:22.996 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:82: TEST_recovery_scrub_1: jq '.pg_stats | [.[] | .state | contains("recovering")]' 2024-09-15T20:26:22.996 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:82: TEST_recovery_scrub_1: grep -q true 2024-09-15T20:26:23.270 INFO:tasks.workunit.client.0.smithi136.stderr:dumped pgs 2024-09-15T20:26:23.281 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:86: TEST_recovery_scrub_1: sleep 2 2024-09-15T20:26:25.282 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:87: TEST_recovery_scrub_1: test 2 -eq 10 2024-09-15T20:26:25.283 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:92: TEST_recovery_scrub_1: expr 2 + 1 2024-09-15T20:26:25.284 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:92: TEST_recovery_scrub_1: count=3 2024-09-15T20:26:25.284 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:79: TEST_recovery_scrub_1: true 2024-09-15T20:26:25.285 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:81: TEST_recovery_scrub_1: ceph --format json pg dump pgs 2024-09-15T20:26:25.285 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:82: TEST_recovery_scrub_1: jq '.pg_stats | [.[] | .state | contains("recovering")]' 2024-09-15T20:26:25.285 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:82: TEST_recovery_scrub_1: grep -q true 2024-09-15T20:26:25.561 INFO:tasks.workunit.client.0.smithi136.stderr:dumped pgs 2024-09-15T20:26:25.573 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:84: TEST_recovery_scrub_1: break 2024-09-15T20:26:25.573 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:94: TEST_recovery_scrub_1: set +o pipefail 2024-09-15T20:26:25.573 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:95: TEST_recovery_scrub_1: ceph pg dump pgs 2024-09-15T20:26:25.854 INFO:tasks.workunit.client.0.smithi136.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-15T20:26:25.854 INFO:tasks.workunit.client.0.smithi136.stdout:1.0 100 0 300 0 0 5242880000 0 0 1300 0 1300 active+recovering+undersized+degraded+remapped 2024-09-15T20:26:22.340260+0000 24'1300 27:1324 [1,0,2,3] 1 [1,0] 1 0'0 2024-09-15T20:23:40.125794+0000 0'0 2024-09-15T20:23:40.125794+0000 0 0 no scrub is scheduled 0 0 2024-09-15T20:26:25.854 INFO:tasks.workunit.client.0.smithi136.stdout: 2024-09-15T20:26:25.855 INFO:tasks.workunit.client.0.smithi136.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-15T20:26:25.855 INFO:tasks.workunit.client.0.smithi136.stderr:dumped pgs 2024-09-15T20:26:25.865 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:97: TEST_recovery_scrub_1: sleep 10 2024-09-15T20:26:35.867 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:99: TEST_recovery_scrub_1: kill_daemons td/osd-recovery-scrub 2024-09-15T20:26:35.868 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-15T20:26:35.868 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-15T20:26:35.868 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-15T20:26:35.868 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-15T20:26:35.868 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-15T20:26:41.185 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-15T20:26:41.185 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:101: TEST_recovery_scrub_1: declare -a err_strings 2024-09-15T20:26:41.185 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:102: TEST_recovery_scrub_1: err_strings[0]='recovery in progress.*scrubs' 2024-09-15T20:26:41.186 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:104: TEST_recovery_scrub_1: expr 4 - 1 2024-09-15T20:26:41.186 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:104: TEST_recovery_scrub_1: seq 0 3 2024-09-15T20:26:41.188 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:104: TEST_recovery_scrub_1: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-09-15T20:26:41.188 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:106: TEST_recovery_scrub_1: grep 'recovery in progress.*scrubs' td/osd-recovery-scrub/osd.0.log 2024-09-15T20:26:41.191 INFO:tasks.workunit.client.0.smithi136.stdout:2024-09-15T20:26:24.211+0000 7fe9105f9640 15 osd.0 osd-scrub:restrictions_on_scrubbing: recovery in progress. Operator-initiated scrubs only. 2024-09-15T20:26:41.191 INFO:tasks.workunit.client.0.smithi136.stdout:2024-09-15T20:26:25.202+0000 7fe9105f9640 15 osd.0 osd-scrub:restrictions_on_scrubbing: recovery in progress. Operator-initiated scrubs only. 2024-09-15T20:26:41.191 INFO:tasks.workunit.client.0.smithi136.stdout:2024-09-15T20:26:27.220+0000 7fe9105f9640 15 osd.0 osd-scrub:restrictions_on_scrubbing: recovery in progress. Operator-initiated scrubs only. 2024-09-15T20:26:41.191 INFO:tasks.workunit.client.0.smithi136.stdout:2024-09-15T20:26:29.185+0000 7fe9105f9640 15 osd.0 osd-scrub:restrictions_on_scrubbing: recovery in progress. Operator-initiated scrubs only. 2024-09-15T20:26:41.191 INFO:tasks.workunit.client.0.smithi136.stdout:2024-09-15T20:26:30.168+0000 7fe9105f9640 15 osd.0 osd-scrub:restrictions_on_scrubbing: recovery in progress. Operator-initiated scrubs only. 2024-09-15T20:26:41.191 INFO:tasks.workunit.client.0.smithi136.stdout:2024-09-15T20:26:33.191+0000 7fe9105f9640 15 osd.0 osd-scrub:restrictions_on_scrubbing: recovery in progress. Operator-initiated scrubs only. 2024-09-15T20:26:41.191 INFO:tasks.workunit.client.0.smithi136.stdout:2024-09-15T20:26:35.196+0000 7fe9105f9640 15 osd.0 osd-scrub:restrictions_on_scrubbing: recovery in progress. Operator-initiated scrubs only. 2024-09-15T20:26:41.191 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:104: TEST_recovery_scrub_1: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-09-15T20:26:41.192 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:106: TEST_recovery_scrub_1: grep 'recovery in progress.*scrubs' td/osd-recovery-scrub/osd.1.log 2024-09-15T20:26:41.205 INFO:tasks.workunit.client.0.smithi136.stdout:2024-09-15T20:26:26.886+0000 7f09788dd640 15 osd.1 osd-scrub:restrictions_on_scrubbing: recovery in progress. Operator-initiated scrubs only. 2024-09-15T20:26:41.206 INFO:tasks.workunit.client.0.smithi136.stdout:2024-09-15T20:26:27.898+0000 7f09788dd640 15 osd.1 osd-scrub:restrictions_on_scrubbing: recovery in progress. Operator-initiated scrubs only. 2024-09-15T20:26:41.206 INFO:tasks.workunit.client.0.smithi136.stdout:2024-09-15T20:26:29.906+0000 7f09788dd640 15 osd.1 osd-scrub:restrictions_on_scrubbing: recovery in progress. Operator-initiated scrubs only. 2024-09-15T20:26:41.206 INFO:tasks.workunit.client.0.smithi136.stdout:2024-09-15T20:26:34.937+0000 7f09788dd640 15 osd.1 osd-scrub:restrictions_on_scrubbing: recovery in progress. Operator-initiated scrubs only. 2024-09-15T20:26:41.206 INFO:tasks.workunit.client.0.smithi136.stdout:2024-09-15T20:26:35.918+0000 7f09788dd640 15 osd.1 osd-scrub:restrictions_on_scrubbing: recovery in progress. Operator-initiated scrubs only. 2024-09-15T20:26:41.206 INFO:tasks.workunit.client.0.smithi136.stdout:2024-09-15T20:26:38.957+0000 7f09788dd640 15 osd.1 osd-scrub:restrictions_on_scrubbing: recovery in progress. Operator-initiated scrubs only. 2024-09-15T20:26:41.206 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:104: TEST_recovery_scrub_1: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-09-15T20:26:41.206 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:106: TEST_recovery_scrub_1: grep 'recovery in progress.*scrubs' td/osd-recovery-scrub/osd.2.log 2024-09-15T20:26:41.209 INFO:tasks.workunit.client.0.smithi136.stdout:2024-09-15T20:26:23.960+0000 7fd436406640 15 osd.2 osd-scrub:restrictions_on_scrubbing: recovery in progress. Operator-initiated scrubs only. 2024-09-15T20:26:41.209 INFO:tasks.workunit.client.0.smithi136.stdout:2024-09-15T20:26:27.002+0000 7fd436406640 15 osd.2 osd-scrub:restrictions_on_scrubbing: recovery in progress. Operator-initiated scrubs only. 2024-09-15T20:26:41.209 INFO:tasks.workunit.client.0.smithi136.stdout:2024-09-15T20:26:28.996+0000 7fd436406640 15 osd.2 osd-scrub:restrictions_on_scrubbing: recovery in progress. Operator-initiated scrubs only. 2024-09-15T20:26:41.209 INFO:tasks.workunit.client.0.smithi136.stdout:2024-09-15T20:26:33.974+0000 7fd436406640 15 osd.2 osd-scrub:restrictions_on_scrubbing: recovery in progress. Operator-initiated scrubs only. 2024-09-15T20:26:41.209 INFO:tasks.workunit.client.0.smithi136.stdout:2024-09-15T20:26:37.927+0000 7fd436406640 15 osd.2 osd-scrub:restrictions_on_scrubbing: recovery in progress. Operator-initiated scrubs only. 2024-09-15T20:26:41.209 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:104: TEST_recovery_scrub_1: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-09-15T20:26:41.209 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:106: TEST_recovery_scrub_1: grep 'recovery in progress.*scrubs' td/osd-recovery-scrub/osd.3.log 2024-09-15T20:26:41.211 INFO:tasks.workunit.client.0.smithi136.stdout:2024-09-15T20:26:25.197+0000 7f2705a4a640 15 osd.3 osd-scrub:restrictions_on_scrubbing: recovery in progress. Operator-initiated scrubs only. 2024-09-15T20:26:41.212 INFO:tasks.workunit.client.0.smithi136.stdout:2024-09-15T20:26:26.245+0000 7f2705a4a640 15 osd.3 osd-scrub:restrictions_on_scrubbing: recovery in progress. Operator-initiated scrubs only. 2024-09-15T20:26:41.212 INFO:tasks.workunit.client.0.smithi136.stdout:2024-09-15T20:26:28.274+0000 7f2705a4a640 15 osd.3 osd-scrub:restrictions_on_scrubbing: recovery in progress. Operator-initiated scrubs only. 2024-09-15T20:26:41.212 INFO:tasks.workunit.client.0.smithi136.stdout:2024-09-15T20:26:29.306+0000 7f2705a4a640 15 osd.3 osd-scrub:restrictions_on_scrubbing: recovery in progress. Operator-initiated scrubs only. 2024-09-15T20:26:41.212 INFO:tasks.workunit.client.0.smithi136.stdout:2024-09-15T20:26:32.205+0000 7f2705a4a640 15 osd.3 osd-scrub:restrictions_on_scrubbing: recovery in progress. Operator-initiated scrubs only. 2024-09-15T20:26:41.212 INFO:tasks.workunit.client.0.smithi136.stdout:2024-09-15T20:26:33.210+0000 7f2705a4a640 15 osd.3 osd-scrub:restrictions_on_scrubbing: recovery in progress. Operator-initiated scrubs only. 2024-09-15T20:26:41.212 INFO:tasks.workunit.client.0.smithi136.stdout:2024-09-15T20:26:38.260+0000 7f2705a4a640 15 osd.3 osd-scrub:restrictions_on_scrubbing: recovery in progress. Operator-initiated scrubs only. 2024-09-15T20:26:41.212 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:108: TEST_recovery_scrub_1: for err_string in "${err_strings[@]}" 2024-09-15T20:26:41.213 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:110: TEST_recovery_scrub_1: found=false 2024-09-15T20:26:41.213 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:111: TEST_recovery_scrub_1: count=0 2024-09-15T20:26:41.213 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:112: TEST_recovery_scrub_1: expr 4 - 1 2024-09-15T20:26:41.213 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:112: TEST_recovery_scrub_1: seq 0 3 2024-09-15T20:26:41.214 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:112: TEST_recovery_scrub_1: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-09-15T20:26:41.214 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:114: TEST_recovery_scrub_1: grep -q 'recovery in progress.*scrubs' td/osd-recovery-scrub/osd.0.log 2024-09-15T20:26:41.217 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:116: TEST_recovery_scrub_1: found=true 2024-09-15T20:26:41.217 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:117: TEST_recovery_scrub_1: expr 0 + 1 2024-09-15T20:26:41.218 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:117: TEST_recovery_scrub_1: count=1 2024-09-15T20:26:41.218 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:112: TEST_recovery_scrub_1: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-09-15T20:26:41.218 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:114: TEST_recovery_scrub_1: grep -q 'recovery in progress.*scrubs' td/osd-recovery-scrub/osd.1.log 2024-09-15T20:26:41.232 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:116: TEST_recovery_scrub_1: found=true 2024-09-15T20:26:41.232 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:117: TEST_recovery_scrub_1: expr 1 + 1 2024-09-15T20:26:41.233 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:117: TEST_recovery_scrub_1: count=2 2024-09-15T20:26:41.234 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:112: TEST_recovery_scrub_1: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-09-15T20:26:41.234 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:114: TEST_recovery_scrub_1: grep -q 'recovery in progress.*scrubs' td/osd-recovery-scrub/osd.2.log 2024-09-15T20:26:41.236 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:116: TEST_recovery_scrub_1: found=true 2024-09-15T20:26:41.236 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:117: TEST_recovery_scrub_1: expr 2 + 1 2024-09-15T20:26:41.237 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:117: TEST_recovery_scrub_1: count=3 2024-09-15T20:26:41.237 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:112: TEST_recovery_scrub_1: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-09-15T20:26:41.237 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:114: TEST_recovery_scrub_1: grep -q 'recovery in progress.*scrubs' td/osd-recovery-scrub/osd.3.log 2024-09-15T20:26:41.239 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:116: TEST_recovery_scrub_1: found=true 2024-09-15T20:26:41.240 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:117: TEST_recovery_scrub_1: expr 3 + 1 2024-09-15T20:26:41.242 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:117: TEST_recovery_scrub_1: count=4 2024-09-15T20:26:41.242 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:120: TEST_recovery_scrub_1: '[' true = false ']' 2024-09-15T20:26:41.242 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:124: TEST_recovery_scrub_1: '[' 4 -eq 4 ']' 2024-09-15T20:26:41.242 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:127: TEST_recovery_scrub_1: teardown td/osd-recovery-scrub 2024-09-15T20:26:41.242 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/osd-recovery-scrub 2024-09-15T20:26:41.242 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-15T20:26:41.242 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/osd-recovery-scrub KILL 2024-09-15T20:26:41.242 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-15T20:26:41.242 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-15T20:26:41.242 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-15T20:26:41.242 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-15T20:26:41.243 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-15T20:26:41.248 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-15T20:26:41.248 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-15T20:26:41.249 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-15T20:26:41.249 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-15T20:26:41.250 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-15T20:26:41.250 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-15T20:26:41.250 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-15T20:26:41.251 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T20:26:41.251 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-15T20:26:41.252 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-15T20:26:41.252 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T20:26:41.252 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-15T20:26:41.253 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-15T20:26:41.253 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/osd-recovery-scrub 2024-09-15T20:26:41.276 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-15T20:26:41.276 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T20:26:41.276 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.43761 2024-09-15T20:26:41.276 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.43761 2024-09-15T20:26:41.276 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-15T20:26:41.277 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-15T20:26:41.277 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:129: TEST_recovery_scrub_1: '[' 0 '!=' 0 ']' 2024-09-15T20:26:41.277 INFO:tasks.workunit.client.0.smithi136.stdout:TEST PASSED 2024-09-15T20:26:41.278 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:135: TEST_recovery_scrub_1: echo 'TEST PASSED' 2024-09-15T20:26:41.279 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:136: TEST_recovery_scrub_1: return 0 2024-09-15T20:26:41.279 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:31: run: for func in $funcs 2024-09-15T20:26:41.279 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:32: run: TEST_recovery_scrub_2 td/osd-recovery-scrub 2024-09-15T20:26:41.279 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:242: TEST_recovery_scrub_2: local dir=td/osd-recovery-scrub 2024-09-15T20:26:41.279 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:243: TEST_recovery_scrub_2: local poolname=test 2024-09-15T20:26:41.279 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:244: TEST_recovery_scrub_2: return 0 2024-09-15T20:26:41.279 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2322: main: code=0 2024-09-15T20:26:41.279 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2326: main: teardown td/osd-recovery-scrub 0 2024-09-15T20:26:41.279 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/osd-recovery-scrub 2024-09-15T20:26:41.279 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs=0 2024-09-15T20:26:41.279 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/osd-recovery-scrub KILL 2024-09-15T20:26:41.279 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-15T20:26:41.280 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-15T20:26:41.280 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-15T20:26:41.280 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-15T20:26:41.280 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-15T20:26:41.281 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-15T20:26:41.281 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-15T20:26:41.281 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-15T20:26:41.281 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-15T20:26:41.282 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-15T20:26:41.282 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-15T20:26:41.282 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-15T20:26:41.283 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T20:26:41.283 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-15T20:26:41.283 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-15T20:26:41.283 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T20:26:41.283 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-15T20:26:41.284 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o 0 = 1 ']' 2024-09-15T20:26:41.285 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/osd-recovery-scrub 2024-09-15T20:26:41.286 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-15T20:26:41.286 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T20:26:41.286 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.43761 2024-09-15T20:26:41.286 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.43761 2024-09-15T20:26:41.287 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-15T20:26:41.287 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-15T20:26:41.287 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2327: main: return 0 2024-09-15T20:26:41.288 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-09-15T20:26:41.288 DEBUG:teuthology.orchestra.run.smithi136:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-09-15T20:26:41.353 INFO:tasks.workunit:Running workunit scrub/osd-scrub-dump.sh... 2024-09-15T20:26:41.354 DEBUG:teuthology.orchestra.run.smithi136:workunit test scrub/osd-scrub-dump.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=7498493279982dfe87f00616198a5967475f6169 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh 2024-09-15T20:26:41.415 INFO:tasks.workunit.client.0.smithi136.stderr:stty: 'standard input': Inappropriate ioctl for device 2024-09-15T20:26:41.420 INFO:tasks.workunit.client.0.smithi136.stderr:+ PS4='${BASH_SOURCE[0]}:$LINENO: ${FUNCNAME[0]}: ' 2024-09-15T20:26:41.420 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2315: main: export PATH=.:/home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin 2024-09-15T20:26:41.420 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2315: main: PATH=.:/home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin 2024-09-15T20:26:41.420 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2316: main: export PYTHONWARNINGS=ignore 2024-09-15T20:26:41.420 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2316: main: PYTHONWARNINGS=ignore 2024-09-15T20:26:41.420 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2317: main: export CEPH_CONF=/dev/null 2024-09-15T20:26:41.421 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2317: main: CEPH_CONF=/dev/null 2024-09-15T20:26:41.421 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2318: main: unset CEPH_ARGS 2024-09-15T20:26:41.421 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2320: main: local code 2024-09-15T20:26:41.421 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2321: main: run td/osd-scrub-dump 2024-09-15T20:26:41.421 INFO:tasks.workunit.client.0.smithi136.stdout:This test is disabled 2024-09-15T20:26:41.422 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:30: run: echo 'This test is disabled' 2024-09-15T20:26:41.422 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:31: run: return 0 2024-09-15T20:26:41.423 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2322: main: code=0 2024-09-15T20:26:41.423 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2326: main: teardown td/osd-scrub-dump 0 2024-09-15T20:26:41.423 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/osd-scrub-dump 2024-09-15T20:26:41.423 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs=0 2024-09-15T20:26:41.423 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/osd-scrub-dump KILL 2024-09-15T20:26:41.423 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-15T20:26:41.423 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-15T20:26:41.423 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-15T20:26:41.423 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-15T20:26:41.423 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-15T20:26:41.423 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-15T20:26:41.423 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-15T20:26:41.424 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-15T20:26:41.424 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-15T20:26:41.424 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-15T20:26:41.424 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-15T20:26:41.425 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-15T20:26:41.425 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T20:26:41.425 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-15T20:26:41.425 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-15T20:26:41.426 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T20:26:41.426 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-15T20:26:41.427 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o 0 = 1 ']' 2024-09-15T20:26:41.427 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/osd-scrub-dump 2024-09-15T20:26:41.428 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-15T20:26:41.429 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T20:26:41.429 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52547 2024-09-15T20:26:41.429 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.52547 2024-09-15T20:26:41.430 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-15T20:26:41.430 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-15T20:26:41.430 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2327: main: return 0 2024-09-15T20:26:41.430 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-09-15T20:26:41.430 DEBUG:teuthology.orchestra.run.smithi136:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-09-15T20:26:41.494 INFO:tasks.workunit:Running workunit scrub/osd-scrub-repair.sh... 2024-09-15T20:26:41.495 DEBUG:teuthology.orchestra.run.smithi136:workunit test scrub/osd-scrub-repair.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=7498493279982dfe87f00616198a5967475f6169 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh 2024-09-15T20:26:41.555 INFO:tasks.workunit.client.0.smithi136.stderr:+ source /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh 2024-09-15T20:26:41.555 INFO:tasks.workunit.client.0.smithi136.stderr:++ TIMEOUT=300 2024-09-15T20:26:41.555 INFO:tasks.workunit.client.0.smithi136.stderr:++ WAIT_FOR_CLEAN_TIMEOUT=90 2024-09-15T20:26:41.555 INFO:tasks.workunit.client.0.smithi136.stderr:++ MAX_TIMEOUT=15 2024-09-15T20:26:41.555 INFO:tasks.workunit.client.0.smithi136.stderr:++ PG_NUM=4 2024-09-15T20:26:41.556 INFO:tasks.workunit.client.0.smithi136.stderr:++ TMPDIR=/tmp 2024-09-15T20:26:41.556 INFO:tasks.workunit.client.0.smithi136.stderr:++ CEPH_BUILD_VIRTUALENV=/tmp 2024-09-15T20:26:41.556 INFO:tasks.workunit.client.0.smithi136.stderr:++ TESTDIR=/home/ubuntu/cephtest 2024-09-15T20:26:41.556 INFO:tasks.workunit.client.0.smithi136.stderr:+++ uname 2024-09-15T20:26:41.557 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' Linux = FreeBSD ']' 2024-09-15T20:26:41.557 INFO:tasks.workunit.client.0.smithi136.stderr:++ SED=sed 2024-09-15T20:26:41.557 INFO:tasks.workunit.client.0.smithi136.stderr:++ AWK=awk 2024-09-15T20:26:41.557 INFO:tasks.workunit.client.0.smithi136.stderr:+++ stty -a 2024-09-15T20:26:41.557 INFO:tasks.workunit.client.0.smithi136.stderr:+++ head -1 2024-09-15T20:26:41.557 INFO:tasks.workunit.client.0.smithi136.stderr:+++ sed -e 's/.*columns \([0-9]*\).*/\1/' 2024-09-15T20:26:41.557 INFO:tasks.workunit.client.0.smithi136.stderr:stty: 'standard input': Inappropriate ioctl for device 2024-09-15T20:26:41.558 INFO:tasks.workunit.client.0.smithi136.stderr:++ termwidth= 2024-09-15T20:26:41.558 INFO:tasks.workunit.client.0.smithi136.stderr:++ '[' -n '' -a '' '!=' 0 ']' 2024-09-15T20:26:41.558 INFO:tasks.workunit.client.0.smithi136.stderr:++ DIFFCOLOPTS='-y ' 2024-09-15T20:26:41.558 INFO:tasks.workunit.client.0.smithi136.stderr:++ KERNCORE=kernel.core_pattern 2024-09-15T20:26:41.558 INFO:tasks.workunit.client.0.smithi136.stderr:++ EXTRA_OPTS= 2024-09-15T20:26:41.560 INFO:tasks.workunit.client.0.smithi136.stderr:++ test '' = TESTS 2024-09-15T20:26:41.561 INFO:tasks.workunit.client.0.smithi136.stderr:+ source /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh 2024-09-15T20:26:41.561 INFO:tasks.workunit.client.0.smithi136.stderr:++ uname 2024-09-15T20:26:41.562 INFO:tasks.workunit.client.0.smithi136.stderr:+ '[' Linux = FreeBSD ']' 2024-09-15T20:26:41.562 INFO:tasks.workunit.client.0.smithi136.stderr:+ use_ec_overwrite=true 2024-09-15T20:26:41.562 INFO:tasks.workunit.client.0.smithi136.stderr:+ getjson=no 2024-09-15T20:26:41.562 INFO:tasks.workunit.client.0.smithi136.stderr:+ jqfilter='def walk(f): 2024-09-15T20:26:41.563 INFO:tasks.workunit.client.0.smithi136.stderr: . as $in 2024-09-15T20:26:41.563 INFO:tasks.workunit.client.0.smithi136.stderr: | if type == "object" then 2024-09-15T20:26:41.563 INFO:tasks.workunit.client.0.smithi136.stderr: reduce keys[] as $key 2024-09-15T20:26:41.563 INFO:tasks.workunit.client.0.smithi136.stderr: ( {}; . + { ($key): ($in[$key] | walk(f)) } ) | f 2024-09-15T20:26:41.563 INFO:tasks.workunit.client.0.smithi136.stderr: elif type == "array" then map( walk(f) ) | f 2024-09-15T20:26:41.563 INFO:tasks.workunit.client.0.smithi136.stderr: else f 2024-09-15T20:26:41.563 INFO:tasks.workunit.client.0.smithi136.stderr: end; 2024-09-15T20:26:41.563 INFO:tasks.workunit.client.0.smithi136.stderr:walk(if type == "object" then del(.mtime) else . end) 2024-09-15T20:26:41.563 INFO:tasks.workunit.client.0.smithi136.stderr:| walk(if type == "object" then del(.local_mtime) else . end) 2024-09-15T20:26:41.563 INFO:tasks.workunit.client.0.smithi136.stderr:| walk(if type == "object" then del(.last_reqid) else . end) 2024-09-15T20:26:41.563 INFO:tasks.workunit.client.0.smithi136.stderr:| walk(if type == "object" then del(.version) else . end) 2024-09-15T20:26:41.563 INFO:tasks.workunit.client.0.smithi136.stderr:| walk(if type == "object" then del(.prior_version) else . end)' 2024-09-15T20:26:41.563 INFO:tasks.workunit.client.0.smithi136.stderr:+ sortkeys='import json; import sys ; JSON=sys.stdin.read() ; ud = json.loads(JSON) ; print(json.dumps(ud, sort_keys=True, indent=2))' 2024-09-15T20:26:41.568 INFO:tasks.workunit.client.0.smithi136.stderr:+ main osd-scrub-repair 2024-09-15T20:26:41.568 INFO:tasks.workunit.client.0.smithi136.stderr:+ local dir=td/osd-scrub-repair 2024-09-15T20:26:41.568 INFO:tasks.workunit.client.0.smithi136.stderr:+ shift 2024-09-15T20:26:41.569 INFO:tasks.workunit.client.0.smithi136.stderr:+ shopt -s -o xtrace 2024-09-15T20:26:41.569 INFO:tasks.workunit.client.0.smithi136.stderr:+ PS4='${BASH_SOURCE[0]}:$LINENO: ${FUNCNAME[0]}: ' 2024-09-15T20:26:41.569 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2315: main: export PATH=.:/home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin 2024-09-15T20:26:41.569 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2315: main: PATH=.:/home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin 2024-09-15T20:26:41.569 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2316: main: export PYTHONWARNINGS=ignore 2024-09-15T20:26:41.570 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2316: main: PYTHONWARNINGS=ignore 2024-09-15T20:26:41.570 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2317: main: export CEPH_CONF=/dev/null 2024-09-15T20:26:41.570 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2317: main: CEPH_CONF=/dev/null 2024-09-15T20:26:41.570 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2318: main: unset CEPH_ARGS 2024-09-15T20:26:41.570 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2320: main: local code 2024-09-15T20:26:41.570 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2321: main: run td/osd-scrub-repair 2024-09-15T20:26:41.570 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:52: run: local dir=td/osd-scrub-repair 2024-09-15T20:26:41.570 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:53: run: shift 2024-09-15T20:26:41.570 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:55: run: export CEPH_MON=127.0.0.1:7107 2024-09-15T20:26:41.570 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:55: run: CEPH_MON=127.0.0.1:7107 2024-09-15T20:26:41.570 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:56: run: export CEPH_ARGS 2024-09-15T20:26:41.570 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:57: run: uuidgen 2024-09-15T20:26:41.570 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:57: run: CEPH_ARGS+='--fsid=ce4892c7-6c6d-4b4f-8e6f-07a3e11133e5 --auth-supported=none ' 2024-09-15T20:26:41.570 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:58: run: CEPH_ARGS+='--mon-host=127.0.0.1:7107 ' 2024-09-15T20:26:41.571 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:59: run: CEPH_ARGS+='--osd-skip-data-digest=false ' 2024-09-15T20:26:41.571 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:61: run: export -n CEPH_CLI_TEST_DUP_COMMAND 2024-09-15T20:26:41.571 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:62: run: set 2024-09-15T20:26:41.571 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:62: run: sed -n -e 's/^\(TEST_[0-9a-z_]*\) .*/\1/p' 2024-09-15T20:26:41.572 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:62: run: local 'funcs=TEST_allow_repair_during_recovery 2024-09-15T20:26:41.572 INFO:tasks.workunit.client.0.smithi136.stderr:TEST_auto_repair_bluestore_basic 2024-09-15T20:26:41.573 INFO:tasks.workunit.client.0.smithi136.stderr:TEST_auto_repair_bluestore_failed 2024-09-15T20:26:41.573 INFO:tasks.workunit.client.0.smithi136.stderr:TEST_auto_repair_bluestore_failed_norecov 2024-09-15T20:26:41.573 INFO:tasks.workunit.client.0.smithi136.stderr:TEST_auto_repair_bluestore_scrub 2024-09-15T20:26:41.573 INFO:tasks.workunit.client.0.smithi136.stderr:TEST_auto_repair_bluestore_tag 2024-09-15T20:26:41.573 INFO:tasks.workunit.client.0.smithi136.stderr:TEST_auto_repair_erasure_coded_appends 2024-09-15T20:26:41.573 INFO:tasks.workunit.client.0.smithi136.stderr:TEST_auto_repair_erasure_coded_overwrites 2024-09-15T20:26:41.573 INFO:tasks.workunit.client.0.smithi136.stderr:TEST_corrupt_and_repair_jerasure_appends 2024-09-15T20:26:41.573 INFO:tasks.workunit.client.0.smithi136.stderr:TEST_corrupt_and_repair_jerasure_overwrites 2024-09-15T20:26:41.573 INFO:tasks.workunit.client.0.smithi136.stderr:TEST_corrupt_and_repair_lrc_appends 2024-09-15T20:26:41.573 INFO:tasks.workunit.client.0.smithi136.stderr:TEST_corrupt_and_repair_lrc_overwrites 2024-09-15T20:26:41.573 INFO:tasks.workunit.client.0.smithi136.stderr:TEST_corrupt_and_repair_replicated 2024-09-15T20:26:41.573 INFO:tasks.workunit.client.0.smithi136.stderr:TEST_corrupt_scrub_erasure_appends 2024-09-15T20:26:41.573 INFO:tasks.workunit.client.0.smithi136.stderr:TEST_corrupt_scrub_erasure_overwrites 2024-09-15T20:26:41.574 INFO:tasks.workunit.client.0.smithi136.stderr:TEST_corrupt_scrub_replicated 2024-09-15T20:26:41.574 INFO:tasks.workunit.client.0.smithi136.stderr:TEST_corrupt_snapset_scrub_rep 2024-09-15T20:26:41.574 INFO:tasks.workunit.client.0.smithi136.stderr:TEST_list_missing_erasure_coded_appends 2024-09-15T20:26:41.574 INFO:tasks.workunit.client.0.smithi136.stderr:TEST_list_missing_erasure_coded_overwrites 2024-09-15T20:26:41.574 INFO:tasks.workunit.client.0.smithi136.stderr:TEST_periodic_scrub_replicated 2024-09-15T20:26:41.574 INFO:tasks.workunit.client.0.smithi136.stderr:TEST_repair_stats 2024-09-15T20:26:41.574 INFO:tasks.workunit.client.0.smithi136.stderr:TEST_repair_stats_ec 2024-09-15T20:26:41.574 INFO:tasks.workunit.client.0.smithi136.stderr:TEST_request_scrub_priority 2024-09-15T20:26:41.574 INFO:tasks.workunit.client.0.smithi136.stderr:TEST_scrub_warning 2024-09-15T20:26:41.574 INFO:tasks.workunit.client.0.smithi136.stderr:TEST_skip_non_repair_during_recovery 2024-09-15T20:26:41.574 INFO:tasks.workunit.client.0.smithi136.stderr:TEST_unfound_erasure_coded_appends 2024-09-15T20:26:41.574 INFO:tasks.workunit.client.0.smithi136.stderr:TEST_unfound_erasure_coded_overwrites' 2024-09-15T20:26:41.574 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:63: run: for func in $funcs 2024-09-15T20:26:41.574 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:64: run: setup td/osd-scrub-repair 2024-09-15T20:26:41.574 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/osd-scrub-repair 2024-09-15T20:26:41.575 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/osd-scrub-repair 2024-09-15T20:26:41.575 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/osd-scrub-repair 2024-09-15T20:26:41.575 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-15T20:26:41.575 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/osd-scrub-repair KILL 2024-09-15T20:26:41.575 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-15T20:26:41.575 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-15T20:26:41.575 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-15T20:26:41.575 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-15T20:26:41.575 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-15T20:26:41.576 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-15T20:26:41.576 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-15T20:26:41.576 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-15T20:26:41.576 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-15T20:26:41.576 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-15T20:26:41.576 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-15T20:26:41.577 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-15T20:26:41.577 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T20:26:41.577 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-15T20:26:41.577 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-15T20:26:41.577 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T20:26:41.578 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-15T20:26:41.579 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-15T20:26:41.579 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/osd-scrub-repair 2024-09-15T20:26:41.580 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-15T20:26:41.580 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T20:26:41.580 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52602 2024-09-15T20:26:41.580 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.52602 2024-09-15T20:26:41.581 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-15T20:26:41.581 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-15T20:26:41.581 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/osd-scrub-repair 2024-09-15T20:26:41.582 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2024-09-15T20:26:41.583 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T20:26:41.583 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52602 2024-09-15T20:26:41.583 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.52602 2024-09-15T20:26:41.585 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2024-09-15T20:26:41.585 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 1024 -le 1024 ']' 2024-09-15T20:26:41.585 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:127: setup: ulimit -n 4096 2024-09-15T20:26:41.585 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2024-09-15T20:26:41.585 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/osd-scrub-repair 1' TERM HUP INT 2024-09-15T20:26:41.585 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:65: run: TEST_allow_repair_during_recovery td/osd-scrub-repair 2024-09-15T20:26:41.585 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:114: TEST_allow_repair_during_recovery: local dir=td/osd-scrub-repair 2024-09-15T20:26:41.586 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:115: TEST_allow_repair_during_recovery: local poolname=rbd 2024-09-15T20:26:41.586 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:117: TEST_allow_repair_during_recovery: run_mon td/osd-scrub-repair a --osd_pool_default_size=2 2024-09-15T20:26:41.586 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/osd-scrub-repair 2024-09-15T20:26:41.586 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2024-09-15T20:26:41.586 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2024-09-15T20:26:41.586 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2024-09-15T20:26:41.586 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/osd-scrub-repair/a 2024-09-15T20:26:41.586 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-scrub-repair/a --run-dir=td/osd-scrub-repair --osd_pool_default_size=2 2024-09-15T20:26:42.049 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2024-09-15T20:26:42.049 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-15T20:26:42.049 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-15T20:26:42.049 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-15T20:26:42.049 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T20:26:42.049 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52602 2024-09-15T20:26:42.049 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.52602/$cluster-$name.asok' 2024-09-15T20:26:42.050 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-scrub-repair/a '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.52602/$cluster-$name.asok' --mon-cluster-log-file=td/osd-scrub-repair/log --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=2 2024-09-15T20:26:42.241 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2024-09-15T20:26:42.243 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2024-09-15T20:26:42.243 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-15T20:26:42.243 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-15T20:26:42.243 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2024-09-15T20:26:42.243 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2024-09-15T20:26:42.243 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-15T20:26:42.243 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-15T20:26:42.243 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-15T20:26:42.243 INFO:tasks.workunit.client.0.smithi136.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-15T20:26:42.243 INFO:tasks.workunit.client.0.smithi136.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T20:26:42.243 INFO:tasks.workunit.client.0.smithi136.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52602 2024-09-15T20:26:42.244 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.52602/ceph-mon.a.asok 2024-09-15T20:26:42.244 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-15T20:26:42.244 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.52602/ceph-mon.a.asok config get fsid 2024-09-15T20:26:42.322 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2024-09-15T20:26:42.323 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-15T20:26:42.323 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-15T20:26:42.323 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2024-09-15T20:26:42.323 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2024-09-15T20:26:42.323 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-15T20:26:42.323 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-15T20:26:42.323 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-15T20:26:42.323 INFO:tasks.workunit.client.0.smithi136.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-15T20:26:42.323 INFO:tasks.workunit.client.0.smithi136.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T20:26:42.323 INFO:tasks.workunit.client.0.smithi136.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52602 2024-09-15T20:26:42.323 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.52602/ceph-mon.a.asok 2024-09-15T20:26:42.324 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-15T20:26:42.324 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.52602/ceph-mon.a.asok config get mon_host 2024-09-15T20:26:42.402 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:118: TEST_allow_repair_during_recovery: run_mgr td/osd-scrub-repair x 2024-09-15T20:26:42.402 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: run_mgr: local dir=td/osd-scrub-repair 2024-09-15T20:26:42.402 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: run_mgr: shift 2024-09-15T20:26:42.402 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: run_mgr: local id=x 2024-09-15T20:26:42.402 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: run_mgr: shift 2024-09-15T20:26:42.402 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:548: run_mgr: local data=td/osd-scrub-repair/x 2024-09-15T20:26:42.402 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:550: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-15T20:26:43.217 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: get_asok_path 2024-09-15T20:26:43.218 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-15T20:26:43.218 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-15T20:26:43.218 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-15T20:26:43.218 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T20:26:43.218 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52602 2024-09-15T20:26:43.218 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.52602/$cluster-$name.asok' 2024-09-15T20:26:43.218 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-15T20:26:43.219 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-scrub-repair/x '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.52602/$cluster-$name.asok' --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-15T20:26:43.249 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:119: TEST_allow_repair_during_recovery: run_osd td/osd-scrub-repair 0 --osd_scrub_during_recovery=false --osd_repair_during_recovery=true --osd_debug_pretend_recovery_active=true 2024-09-15T20:26:43.249 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-scrub-repair 2024-09-15T20:26:43.249 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-15T20:26:43.249 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=0 2024-09-15T20:26:43.249 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-15T20:26:43.249 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-scrub-repair/0 2024-09-15T20:26:43.249 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=ce4892c7-6c6d-4b4f-8e6f-07a3e11133e5 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-15T20:26:43.249 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T20:26:43.249 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T20:26:43.249 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T20:26:43.249 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-09-15T20:26:43.249 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-09-15T20:26:43.250 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-15T20:26:43.250 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-15T20:26:43.250 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-15T20:26:43.250 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-15T20:26:43.250 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-15T20:26:43.250 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-15T20:26:43.250 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-15T20:26:43.251 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T20:26:43.251 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52602 2024-09-15T20:26:43.251 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.52602/$cluster-$name.asok' 2024-09-15T20:26:43.251 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.52602/$cluster-$name.asok' 2024-09-15T20:26:43.251 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-15T20:26:43.251 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-15T20:26:43.251 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-15T20:26:43.251 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-15T20:26:43.251 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-15T20:26:43.251 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T20:26:43.251 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T20:26:43.251 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T20:26:43.251 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T20:26:43.251 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-15T20:26:43.251 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+='--osd_scrub_during_recovery=false --osd_repair_during_recovery=true --osd_debug_pretend_recovery_active=true' 2024-09-15T20:26:43.252 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-scrub-repair/0 2024-09-15T20:26:43.252 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-15T20:26:43.252 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=e5ea25f0-4c51-4ae7-9fe9-7db46be48002 2024-09-15T20:26:43.252 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd0 e5ea25f0-4c51-4ae7-9fe9-7db46be48002' 2024-09-15T20:26:43.252 INFO:tasks.workunit.client.0.smithi136.stdout:add osd0 e5ea25f0-4c51-4ae7-9fe9-7db46be48002 2024-09-15T20:26:43.253 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-15T20:26:43.268 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQADQ+dmFojgDxAASDyD09u9uMiV7pWBYO0c9g== 2024-09-15T20:26:43.269 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQADQ+dmFojgDxAASDyD09u9uMiV7pWBYO0c9g=="}' 2024-09-15T20:26:43.269 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new e5ea25f0-4c51-4ae7-9fe9-7db46be48002 -i td/osd-scrub-repair/0/new.json 2024-09-15T20:26:43.474 INFO:tasks.workunit.client.0.smithi136.stdout:0 2024-09-15T20:26:43.483 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-scrub-repair/0/new.json 2024-09-15T20:26:43.485 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 0 --fsid=ce4892c7-6c6d-4b4f-8e6f-07a3e11133e5 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.52602/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_scrub_during_recovery=false --osd_repair_during_recovery=true --osd_debug_pretend_recovery_active=true --mkfs --key AQADQ+dmFojgDxAASDyD09u9uMiV7pWBYO0c9g== --osd-uuid e5ea25f0-4c51-4ae7-9fe9-7db46be48002 2024-09-15T20:26:43.516 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:26:43.514+0000 7fa8a54fe640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T20:26:43.525 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:26:43.523+0000 7fa8a54fe640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T20:26:43.526 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:26:43.524+0000 7fa8a54fe640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T20:26:43.527 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:26:43.525+0000 7fa8a54fe640 -1 bdev(0x55ec79b9dc00 td/osd-scrub-repair/0/block) open stat got: (1) Operation not permitted 2024-09-15T20:26:43.527 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:26:43.525+0000 7fa8a54fe640 -1 bluestore(td/osd-scrub-repair/0) _read_fsid unparsable uuid 2024-09-15T20:26:49.545 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-scrub-repair/0/keyring 2024-09-15T20:26:49.545 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-15T20:26:49.546 INFO:tasks.workunit.client.0.smithi136.stdout:adding osd0 key to auth repository 2024-09-15T20:26:49.546 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd0 key to auth repository 2024-09-15T20:26:49.546 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-scrub-repair/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-15T20:26:49.907 INFO:tasks.workunit.client.0.smithi136.stdout:start osd.0 2024-09-15T20:26:49.908 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.0 2024-09-15T20:26:49.908 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 0 --fsid=ce4892c7-6c6d-4b4f-8e6f-07a3e11133e5 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.52602/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_scrub_during_recovery=false --osd_repair_during_recovery=true --osd_debug_pretend_recovery_active=true 2024-09-15T20:26:49.908 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-15T20:26:49.908 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-15T20:26:49.908 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-15T20:26:49.937 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:26:49.935+0000 7f76bbe0c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T20:26:49.938 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:26:49.937+0000 7f76bbe0c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T20:26:49.939 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:26:49.938+0000 7f76bbe0c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T20:26:50.183 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 0 2024-09-15T20:26:50.184 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-15T20:26:50.184 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2024-09-15T20:26:50.184 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-15T20:26:50.184 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-15T20:26:50.184 INFO:tasks.workunit.client.0.smithi136.stdout:0 2024-09-15T20:26:50.185 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:26:50.185 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-15T20:26:50.185 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:26:50.185 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T20:26:50.459 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:26:51.461 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:26:51.461 INFO:tasks.workunit.client.0.smithi136.stdout:1 2024-09-15T20:26:51.462 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:26:51.462 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-15T20:26:51.462 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:26:51.462 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T20:26:51.549 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:26:51.547+0000 7f76bbe0c640 -1 Falling back to public interface 2024-09-15T20:26:51.750 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:26:52.751 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:26:52.751 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:26:52.751 INFO:tasks.workunit.client.0.smithi136.stdout:2 2024-09-15T20:26:52.752 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-15T20:26:52.752 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:26:52.752 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T20:26:53.040 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:26:53.943 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:26:53.941+0000 7f76bbe0c640 -1 osd.0 0 log_to_monitors true 2024-09-15T20:26:54.042 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:26:54.042 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:26:54.042 INFO:tasks.workunit.client.0.smithi136.stdout:3 2024-09-15T20:26:54.043 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-15T20:26:54.043 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:26:54.043 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T20:26:54.398 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:26:55.399 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:26:55.400 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:26:55.400 INFO:tasks.workunit.client.0.smithi136.stdout:4 2024-09-15T20:26:55.400 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-15T20:26:55.400 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:26:55.401 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T20:26:55.703 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:26:56.705 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:26:56.705 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:26:56.705 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-15T20:26:56.705 INFO:tasks.workunit.client.0.smithi136.stdout:5 2024-09-15T20:26:56.706 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:26:56.706 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T20:26:56.992 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:26:57.994 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:26:57.994 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:26:57.994 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-15T20:26:57.994 INFO:tasks.workunit.client.0.smithi136.stdout:6 2024-09-15T20:26:57.995 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:26:57.995 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T20:26:58.284 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:26:59.286 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:26:59.286 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:26:59.286 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-15T20:26:59.286 INFO:tasks.workunit.client.0.smithi136.stdout:7 2024-09-15T20:26:59.287 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:26:59.287 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T20:26:59.574 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:27:00.575 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:27:00.576 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:27:00.576 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-15T20:27:00.576 INFO:tasks.workunit.client.0.smithi136.stdout:8 2024-09-15T20:27:00.576 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:27:00.576 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T20:27:00.934 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:27:01.935 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:27:01.935 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:27:01.935 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-15T20:27:01.935 INFO:tasks.workunit.client.0.smithi136.stdout:9 2024-09-15T20:27:01.936 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:27:01.936 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T20:27:02.222 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:27:03.224 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:27:03.224 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:27:03.224 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-15T20:27:03.225 INFO:tasks.workunit.client.0.smithi136.stdout:10 2024-09-15T20:27:03.225 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:27:03.225 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T20:27:03.514 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:27:04.516 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:27:04.516 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:27:04.516 INFO:tasks.workunit.client.0.smithi136.stdout:11 2024-09-15T20:27:04.517 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-15T20:27:04.517 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:27:04.517 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T20:27:04.937 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:27:05.938 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:27:05.938 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:27:05.939 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-15T20:27:05.939 INFO:tasks.workunit.client.0.smithi136.stdout:12 2024-09-15T20:27:05.939 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:27:05.939 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T20:27:06.310 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:27:07.311 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:27:07.312 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:27:07.312 INFO:tasks.workunit.client.0.smithi136.stdout:13 2024-09-15T20:27:07.312 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-15T20:27:07.312 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:27:07.313 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T20:27:07.700 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:27:08.701 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:27:08.701 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:27:08.702 INFO:tasks.workunit.client.0.smithi136.stdout:14 2024-09-15T20:27:08.702 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-15T20:27:08.702 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:27:08.702 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T20:27:09.111 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:27:10.112 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:27:10.112 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:27:10.112 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-15T20:27:10.112 INFO:tasks.workunit.client.0.smithi136.stdout:15 2024-09-15T20:27:10.113 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:27:10.113 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T20:27:10.401 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:27:11.403 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:27:11.403 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:27:11.403 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-15T20:27:11.403 INFO:tasks.workunit.client.0.smithi136.stdout:16 2024-09-15T20:27:11.404 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:27:11.404 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T20:27:11.691 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:27:12.693 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:27:12.693 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:27:12.694 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-15T20:27:12.694 INFO:tasks.workunit.client.0.smithi136.stdout:17 2024-09-15T20:27:12.695 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:27:12.695 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T20:27:13.252 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:27:14.253 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:27:14.253 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:27:14.254 INFO:tasks.workunit.client.0.smithi136.stdout:18 2024-09-15T20:27:14.254 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 18 2024-09-15T20:27:14.254 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:27:14.254 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T20:27:14.829 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:27:15.830 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:27:15.830 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:27:15.830 INFO:tasks.workunit.client.0.smithi136.stdout:19 2024-09-15T20:27:15.831 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 19 2024-09-15T20:27:15.831 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:27:15.831 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T20:27:16.147 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:27:17.148 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:27:17.148 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:27:17.148 INFO:tasks.workunit.client.0.smithi136.stdout:20 2024-09-15T20:27:17.149 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 20 2024-09-15T20:27:17.149 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:27:17.149 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T20:27:17.450 INFO:tasks.workunit.client.0.smithi136.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/1402690408,v1:127.0.0.1:6803/1402690408] [v2:127.0.0.1:6804/1402690408,v1:127.0.0.1:6805/1402690408] exists,up e5ea25f0-4c51-4ae7-9fe9-7db46be48002 2024-09-15T20:27:17.454 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-15T20:27:17.454 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-15T20:27:17.454 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-15T20:27:17.455 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:122: TEST_allow_repair_during_recovery: run_osd td/osd-scrub-repair 1 --osd_scrub_during_recovery=false --osd_repair_during_recovery=true --osd_debug_pretend_recovery_active=true 2024-09-15T20:27:17.455 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-scrub-repair 2024-09-15T20:27:17.455 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-15T20:27:17.455 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=1 2024-09-15T20:27:17.455 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-15T20:27:17.455 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-scrub-repair/1 2024-09-15T20:27:17.455 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=ce4892c7-6c6d-4b4f-8e6f-07a3e11133e5 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-15T20:27:17.455 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T20:27:17.455 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T20:27:17.455 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T20:27:17.455 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-09-15T20:27:17.455 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-09-15T20:27:17.455 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-15T20:27:17.456 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-15T20:27:17.456 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-15T20:27:17.456 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-15T20:27:17.456 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-15T20:27:17.456 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-15T20:27:17.456 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-15T20:27:17.456 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T20:27:17.456 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52602 2024-09-15T20:27:17.456 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.52602/$cluster-$name.asok' 2024-09-15T20:27:17.456 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.52602/$cluster-$name.asok' 2024-09-15T20:27:17.456 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-15T20:27:17.456 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-15T20:27:17.456 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-15T20:27:17.457 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-15T20:27:17.457 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-15T20:27:17.457 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T20:27:17.457 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T20:27:17.457 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T20:27:17.457 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T20:27:17.457 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-15T20:27:17.457 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+='--osd_scrub_during_recovery=false --osd_repair_during_recovery=true --osd_debug_pretend_recovery_active=true' 2024-09-15T20:27:17.457 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-scrub-repair/1 2024-09-15T20:27:17.457 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-15T20:27:17.457 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=c14f43b7-3d6c-4725-9029-717d9f4f2838 2024-09-15T20:27:17.457 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd1 c14f43b7-3d6c-4725-9029-717d9f4f2838' 2024-09-15T20:27:17.458 INFO:tasks.workunit.client.0.smithi136.stdout:add osd1 c14f43b7-3d6c-4725-9029-717d9f4f2838 2024-09-15T20:27:17.458 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-15T20:27:17.472 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQAlQ+dmTc0LHBAAzkXLl9rl9iX9wd85MrNpGQ== 2024-09-15T20:27:17.472 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQAlQ+dmTc0LHBAAzkXLl9rl9iX9wd85MrNpGQ=="}' 2024-09-15T20:27:17.472 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new c14f43b7-3d6c-4725-9029-717d9f4f2838 -i td/osd-scrub-repair/1/new.json 2024-09-15T20:27:17.858 INFO:tasks.workunit.client.0.smithi136.stdout:1 2024-09-15T20:27:17.869 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-scrub-repair/1/new.json 2024-09-15T20:27:17.870 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 1 --fsid=ce4892c7-6c6d-4b4f-8e6f-07a3e11133e5 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.52602/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_scrub_during_recovery=false --osd_repair_during_recovery=true --osd_debug_pretend_recovery_active=true --mkfs --key AQAlQ+dmTc0LHBAAzkXLl9rl9iX9wd85MrNpGQ== --osd-uuid c14f43b7-3d6c-4725-9029-717d9f4f2838 2024-09-15T20:27:17.901 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:27:17.899+0000 7fb12ec18640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T20:27:17.942 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:27:17.940+0000 7fb12ec18640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T20:27:17.943 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:27:17.941+0000 7fb12ec18640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T20:27:17.944 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:27:17.942+0000 7fb12ec18640 -1 bdev(0x55708e6dbc00 td/osd-scrub-repair/1/block) open stat got: (1) Operation not permitted 2024-09-15T20:27:17.944 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:27:17.942+0000 7fb12ec18640 -1 bluestore(td/osd-scrub-repair/1) _read_fsid unparsable uuid 2024-09-15T20:27:23.982 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-scrub-repair/1/keyring 2024-09-15T20:27:23.982 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-15T20:27:23.983 INFO:tasks.workunit.client.0.smithi136.stdout:adding osd1 key to auth repository 2024-09-15T20:27:23.983 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd1 key to auth repository 2024-09-15T20:27:23.983 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-scrub-repair/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-15T20:27:24.343 INFO:tasks.workunit.client.0.smithi136.stdout:start osd.1 2024-09-15T20:27:24.344 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.1 2024-09-15T20:27:24.344 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 1 --fsid=ce4892c7-6c6d-4b4f-8e6f-07a3e11133e5 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.52602/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_scrub_during_recovery=false --osd_repair_during_recovery=true --osd_debug_pretend_recovery_active=true 2024-09-15T20:27:24.344 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-15T20:27:24.344 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-15T20:27:24.344 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-15T20:27:24.373 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:27:24.371+0000 7ff0bfc20640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T20:27:24.375 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:27:24.373+0000 7ff0bfc20640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T20:27:24.376 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:27:24.374+0000 7ff0bfc20640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T20:27:24.624 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 1 2024-09-15T20:27:24.624 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-15T20:27:24.624 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2024-09-15T20:27:24.624 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-15T20:27:24.625 INFO:tasks.workunit.client.0.smithi136.stdout:0 2024-09-15T20:27:24.625 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-15T20:27:24.625 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:27:24.625 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-15T20:27:24.625 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:27:24.625 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T20:27:24.899 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:27:25.498 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:27:25.496+0000 7ff0bfc20640 -1 Falling back to public interface 2024-09-15T20:27:25.900 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:27:25.900 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:27:25.901 INFO:tasks.workunit.client.0.smithi136.stdout:1 2024-09-15T20:27:25.901 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-15T20:27:25.901 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:27:25.901 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T20:27:26.179 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:27:27.180 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:27:27.180 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:27:27.181 INFO:tasks.workunit.client.0.smithi136.stdout:2 2024-09-15T20:27:27.181 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-15T20:27:27.181 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:27:27.181 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T20:27:27.471 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:27:28.114 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:27:28.112+0000 7ff0bfc20640 -1 osd.1 0 log_to_monitors true 2024-09-15T20:27:28.472 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:27:28.472 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:27:28.472 INFO:tasks.workunit.client.0.smithi136.stdout:3 2024-09-15T20:27:28.473 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-15T20:27:28.473 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:27:28.473 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T20:27:28.789 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:27:29.790 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:27:29.790 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:27:29.790 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-15T20:27:29.791 INFO:tasks.workunit.client.0.smithi136.stdout:4 2024-09-15T20:27:29.791 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:27:29.791 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T20:27:30.139 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:27:31.140 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:27:31.140 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:27:31.140 INFO:tasks.workunit.client.0.smithi136.stdout:5 2024-09-15T20:27:31.141 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-15T20:27:31.141 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:27:31.141 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T20:27:31.474 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:27:32.475 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:27:32.475 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:27:32.476 INFO:tasks.workunit.client.0.smithi136.stdout:6 2024-09-15T20:27:32.476 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-15T20:27:32.476 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:27:32.476 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T20:27:32.763 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:27:33.765 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:27:33.765 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:27:33.765 INFO:tasks.workunit.client.0.smithi136.stdout:7 2024-09-15T20:27:33.765 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-15T20:27:33.766 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:27:33.766 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T20:27:34.052 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:27:35.053 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:27:35.053 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:27:35.053 INFO:tasks.workunit.client.0.smithi136.stdout:8 2024-09-15T20:27:35.054 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-15T20:27:35.054 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:27:35.054 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T20:27:35.341 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:27:36.342 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:27:36.342 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:27:36.343 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-15T20:27:36.343 INFO:tasks.workunit.client.0.smithi136.stdout:9 2024-09-15T20:27:36.343 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:27:36.343 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T20:27:36.632 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:27:37.634 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:27:37.634 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:27:37.634 INFO:tasks.workunit.client.0.smithi136.stdout:10 2024-09-15T20:27:37.635 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-15T20:27:37.635 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:27:37.635 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T20:27:37.923 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:27:38.924 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:27:38.925 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:27:38.925 INFO:tasks.workunit.client.0.smithi136.stdout:11 2024-09-15T20:27:38.925 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-15T20:27:38.925 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:27:38.925 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T20:27:39.216 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:27:40.217 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:27:40.217 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:27:40.217 INFO:tasks.workunit.client.0.smithi136.stdout:12 2024-09-15T20:27:40.218 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-15T20:27:40.218 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:27:40.218 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T20:27:40.505 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:27:41.506 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:27:41.506 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:27:41.506 INFO:tasks.workunit.client.0.smithi136.stdout:13 2024-09-15T20:27:41.507 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-15T20:27:41.507 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:27:41.507 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T20:27:42.201 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:27:43.202 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:27:43.202 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:27:43.202 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-15T20:27:43.203 INFO:tasks.workunit.client.0.smithi136.stdout:14 2024-09-15T20:27:43.203 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:27:43.203 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T20:27:43.935 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:27:44.937 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:27:44.937 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:27:44.937 INFO:tasks.workunit.client.0.smithi136.stdout:15 2024-09-15T20:27:44.938 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-15T20:27:44.938 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:27:44.938 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T20:27:45.404 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:27:46.405 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:27:46.405 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:27:46.406 INFO:tasks.workunit.client.0.smithi136.stdout:16 2024-09-15T20:27:46.406 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-15T20:27:46.406 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:27:46.406 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T20:27:47.064 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:27:48.065 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:27:48.066 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:27:48.066 INFO:tasks.workunit.client.0.smithi136.stdout:17 2024-09-15T20:27:48.066 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-15T20:27:48.066 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:27:48.066 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T20:27:48.534 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:27:49.536 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:27:49.536 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:27:49.536 INFO:tasks.workunit.client.0.smithi136.stdout:18 2024-09-15T20:27:49.537 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 18 2024-09-15T20:27:49.537 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:27:49.537 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T20:27:50.536 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:27:51.537 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:27:51.537 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:27:51.537 INFO:tasks.workunit.client.0.smithi136.stdout:19 2024-09-15T20:27:51.538 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 19 2024-09-15T20:27:51.538 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:27:51.538 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T20:27:51.821 INFO:tasks.workunit.client.0.smithi136.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/2731127297,v1:127.0.0.1:6811/2731127297] [v2:127.0.0.1:6812/2731127297,v1:127.0.0.1:6813/2731127297] exists,up c14f43b7-3d6c-4725-9029-717d9f4f2838 2024-09-15T20:27:51.822 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-15T20:27:51.822 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-15T20:27:51.822 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-15T20:27:51.822 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:125: TEST_allow_repair_during_recovery: create_rbd_pool 2024-09-15T20:27:51.822 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:526: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2024-09-15T20:27:52.147 INFO:tasks.workunit.client.0.smithi136.stderr:pool 'rbd' does not exist 2024-09-15T20:27:52.159 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:527: create_rbd_pool: create_pool rbd 4 2024-09-15T20:27:52.159 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create rbd 4 2024-09-15T20:27:52.689 INFO:tasks.workunit.client.0.smithi136.stderr:pool 'rbd' created 2024-09-15T20:27:52.701 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2024-09-15T20:27:53.702 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:528: create_rbd_pool: rbd pool init rbd 2024-09-15T20:27:55.458 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:126: TEST_allow_repair_during_recovery: wait_for_clean 2024-09-15T20:27:55.459 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-15T20:27:55.459 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-15T20:27:55.459 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-15T20:27:55.460 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-15T20:27:55.460 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-15T20:27:55.460 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-15T20:27:55.460 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-15T20:27:55.460 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-15T20:27:55.460 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-15T20:27:55.557 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-15T20:27:55.557 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-15T20:27:55.558 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-15T20:27:55.558 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-15T20:27:55.558 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-15T20:27:55.558 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-15T20:27:55.835 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-15T20:27:55.835 INFO:tasks.workunit.client.0.smithi136.stderr:1' 2024-09-15T20:27:55.836 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-15T20:27:55.836 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T20:27:55.836 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-15T20:27:55.939 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836490 2024-09-15T20:27:55.939 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836490 2024-09-15T20:27:55.939 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836490' 2024-09-15T20:27:55.939 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T20:27:55.939 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-15T20:27:56.042 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=42949672966 2024-09-15T20:27:56.042 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 42949672966 2024-09-15T20:27:56.042 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836490 1-42949672966' 2024-09-15T20:27:56.042 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T20:27:56.043 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836490 2024-09-15T20:27:56.043 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T20:27:56.043 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-15T20:27:56.044 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836490 2024-09-15T20:27:56.044 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T20:27:56.045 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836490 2024-09-15T20:27:56.045 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836490' 2024-09-15T20:27:56.045 INFO:tasks.workunit.client.0.smithi136.stdout:waiting osd.0 seq 21474836490 2024-09-15T20:27:56.046 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T20:27:56.322 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836489 -lt 21474836490 2024-09-15T20:27:56.322 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-15T20:27:57.323 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-15T20:27:57.324 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T20:27:57.612 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836490 -lt 21474836490 2024-09-15T20:27:57.613 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T20:27:57.613 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-42949672966 2024-09-15T20:27:57.613 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T20:27:57.614 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-15T20:27:57.615 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-42949672966 2024-09-15T20:27:57.615 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T20:27:57.616 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=42949672966 2024-09-15T20:27:57.616 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 42949672966' 2024-09-15T20:27:57.616 INFO:tasks.workunit.client.0.smithi136.stdout:waiting osd.1 seq 42949672966 2024-09-15T20:27:57.616 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-15T20:27:57.895 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 42949672967 -lt 42949672966 2024-09-15T20:27:57.895 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-15T20:27:57.895 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T20:27:57.896 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T20:27:58.260 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 4 == 0 2024-09-15T20:27:58.260 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-15T20:27:58.261 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-15T20:27:58.261 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-15T20:27:58.261 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T20:27:58.261 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T20:27:58.262 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T20:27:58.262 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T20:27:58.532 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=4 2024-09-15T20:27:58.533 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-15T20:27:58.533 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T20:27:58.533 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T20:27:58.875 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 = 4 2024-09-15T20:27:58.875 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-15T20:27:58.875 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-15T20:27:58.875 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:128: TEST_allow_repair_during_recovery: add_something td/osd-scrub-repair rbd 2024-09-15T20:27:58.875 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-09-15T20:27:58.875 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=rbd 2024-09-15T20:27:58.875 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=SOMETHING 2024-09-15T20:27:58.876 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-09-15T20:27:58.876 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-09-15T20:27:58.876 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-09-15T20:27:59.267 INFO:tasks.workunit.client.0.smithi136.stderr:noscrub is set 2024-09-15T20:27:59.278 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-09-15T20:27:59.660 INFO:tasks.workunit.client.0.smithi136.stderr:nodeep-scrub is set 2024-09-15T20:27:59.672 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-09-15T20:27:59.672 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-09-15T20:27:59.672 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool rbd put SOMETHING td/osd-scrub-repair/ORIGINAL 2024-09-15T20:28:00.032 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:129: TEST_allow_repair_during_recovery: get_not_primary rbd SOMETHING 2024-09-15T20:28:00.032 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1220: get_not_primary: local poolname=rbd 2024-09-15T20:28:00.032 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1221: get_not_primary: local objectname=SOMETHING 2024-09-15T20:28:00.032 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1223: get_not_primary: get_primary rbd SOMETHING 2024-09-15T20:28:00.032 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1187: get_primary: local poolname=rbd 2024-09-15T20:28:00.033 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1188: get_primary: local objectname=SOMETHING 2024-09-15T20:28:00.033 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1190: get_primary: ceph --format json osd map rbd SOMETHING 2024-09-15T20:28:00.033 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1191: get_primary: jq .acting_primary 2024-09-15T20:28:00.311 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1223: get_not_primary: local primary=1 2024-09-15T20:28:00.311 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1224: get_not_primary: ceph --format json osd map rbd SOMETHING 2024-09-15T20:28:00.311 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1225: get_not_primary: jq '.acting | map(select (. != 1)) | .[0]' 2024-09-15T20:28:00.590 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:129: TEST_allow_repair_during_recovery: corrupt_and_repair_one td/osd-scrub-repair rbd 0 2024-09-15T20:28:00.591 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:226: corrupt_and_repair_one: local dir=td/osd-scrub-repair 2024-09-15T20:28:00.591 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:227: corrupt_and_repair_one: local poolname=rbd 2024-09-15T20:28:00.591 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:228: corrupt_and_repair_one: local osd=0 2024-09-15T20:28:00.591 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:233: corrupt_and_repair_one: objectstore_tool td/osd-scrub-repair 0 SOMETHING remove 2024-09-15T20:28:00.591 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2024-09-15T20:28:00.591 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2024-09-15T20:28:00.591 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=0 2024-09-15T20:28:00.591 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2024-09-15T20:28:00.591 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 SOMETHING remove 2024-09-15T20:28:00.591 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-09-15T20:28:00.591 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2024-09-15T20:28:00.591 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=0 2024-09-15T20:28:00.591 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2024-09-15T20:28:00.591 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2024-09-15T20:28:00.592 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-15T20:28:00.592 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-15T20:28:00.592 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-15T20:28:00.592 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-15T20:28:00.592 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-15T20:28:00.896 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-15T20:28:00.896 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 SOMETHING remove 2024-09-15T20:28:00.896 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-15T20:28:00.896 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-15T20:28:00.896 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=0 2024-09-15T20:28:00.896 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-15T20:28:00.896 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-09-15T20:28:00.897 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 SOMETHING remove 2024-09-15T20:28:02.638 INFO:tasks.workunit.client.0.smithi136.stdout:remove #1:eb822e21:::SOMETHING:head# 2024-09-15T20:28:03.155 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 2024-09-15T20:28:03.155 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2024-09-15T20:28:03.155 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-15T20:28:03.155 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=0 2024-09-15T20:28:03.155 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-15T20:28:03.155 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-09-15T20:28:03.156 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=ce4892c7-6c6d-4b4f-8e6f-07a3e11133e5 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-15T20:28:03.156 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T20:28:03.156 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T20:28:03.156 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T20:28:03.156 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-09-15T20:28:03.156 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-09-15T20:28:03.156 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-15T20:28:03.156 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-15T20:28:03.156 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-15T20:28:03.156 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-15T20:28:03.156 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-15T20:28:03.156 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-15T20:28:03.157 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-15T20:28:03.157 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T20:28:03.157 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52602 2024-09-15T20:28:03.157 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.52602/$cluster-$name.asok' 2024-09-15T20:28:03.157 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.52602/$cluster-$name.asok' 2024-09-15T20:28:03.157 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-15T20:28:03.157 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-15T20:28:03.157 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-15T20:28:03.157 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T20:28:03.157 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T20:28:03.157 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T20:28:03.157 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T20:28:03.157 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-15T20:28:03.158 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-15T20:28:03.158 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-09-15T20:28:03.159 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.0 2024-09-15T20:28:03.159 INFO:tasks.workunit.client.0.smithi136.stderr:start osd.0 2024-09-15T20:28:03.159 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 0 --fsid=ce4892c7-6c6d-4b4f-8e6f-07a3e11133e5 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.52602/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T20:28:03.159 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-09-15T20:28:03.159 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 0 = 0 ']' 2024-09-15T20:28:03.159 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-15T20:28:03.159 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-15T20:28:03.159 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-15T20:28:03.187 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:28:03.185+0000 7faae4cef640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T20:28:03.189 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:28:03.187+0000 7faae4cef640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T20:28:03.190 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:28:03.189+0000 7faae4cef640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T20:28:03.448 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 0 2024-09-15T20:28:03.448 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-15T20:28:03.448 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2024-09-15T20:28:03.448 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-15T20:28:03.448 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-15T20:28:03.449 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:28:03.449 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-15T20:28:03.449 INFO:tasks.workunit.client.0.smithi136.stderr:0 2024-09-15T20:28:03.449 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:28:03.449 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T20:28:03.730 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:28:04.732 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:28:04.732 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:28:04.733 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-15T20:28:04.733 INFO:tasks.workunit.client.0.smithi136.stderr:1 2024-09-15T20:28:04.733 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:28:04.733 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T20:28:05.018 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:28:05.045 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:28:05.043+0000 7faae4cef640 -1 Falling back to public interface 2024-09-15T20:28:06.020 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:28:06.020 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:28:06.020 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-15T20:28:06.020 INFO:tasks.workunit.client.0.smithi136.stderr:2 2024-09-15T20:28:06.020 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:28:06.020 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T20:28:06.300 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:28:07.301 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:28:07.301 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:28:07.301 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-15T20:28:07.302 INFO:tasks.workunit.client.0.smithi136.stderr:3 2024-09-15T20:28:07.302 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:28:07.302 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T20:28:07.581 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:28:08.005 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:28:08.003+0000 7faae4cef640 -1 osd.0 22 log_to_monitors true 2024-09-15T20:28:08.582 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:28:08.582 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:28:08.582 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-15T20:28:08.582 INFO:tasks.workunit.client.0.smithi136.stderr:4 2024-09-15T20:28:08.582 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:28:08.583 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T20:28:08.907 INFO:tasks.workunit.client.0.smithi136.stderr:osd.0 up in weight 1 up_from 26 up_thru 26 down_at 23 last_clean_interval [5,22) [v2:127.0.0.1:6802/1920843438,v1:127.0.0.1:6803/1920843438] [v2:127.0.0.1:6804/1920843438,v1:127.0.0.1:6805/1920843438] exists,up e5ea25f0-4c51-4ae7-9fe9-7db46be48002 2024-09-15T20:28:08.908 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-15T20:28:08.908 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-15T20:28:08.908 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-15T20:28:08.908 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2024-09-15T20:28:08.908 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-15T20:28:08.908 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-15T20:28:08.908 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-15T20:28:08.908 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-15T20:28:08.909 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-15T20:28:08.909 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-15T20:28:08.909 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-15T20:28:08.909 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-15T20:28:08.909 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-15T20:28:08.995 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-15T20:28:08.995 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-15T20:28:08.996 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-15T20:28:08.996 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-15T20:28:08.996 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-15T20:28:08.996 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-15T20:28:09.298 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-15T20:28:09.298 INFO:tasks.workunit.client.0.smithi136.stderr:1' 2024-09-15T20:28:09.298 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-15T20:28:09.298 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T20:28:09.298 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-15T20:28:09.420 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=111669149698 2024-09-15T20:28:09.420 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 111669149698 2024-09-15T20:28:09.420 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-111669149698' 2024-09-15T20:28:09.420 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T20:28:09.420 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-15T20:28:09.581 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=42949672970 2024-09-15T20:28:09.581 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 42949672970 2024-09-15T20:28:09.581 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-111669149698 1-42949672970' 2024-09-15T20:28:09.581 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T20:28:09.581 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-111669149698 2024-09-15T20:28:09.581 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T20:28:09.582 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-15T20:28:09.582 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-111669149698 2024-09-15T20:28:09.582 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T20:28:09.583 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=111669149698 2024-09-15T20:28:09.583 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 111669149698' 2024-09-15T20:28:09.583 INFO:tasks.workunit.client.0.smithi136.stderr:waiting osd.0 seq 111669149698 2024-09-15T20:28:09.584 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T20:28:09.886 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 1 -lt 111669149698 2024-09-15T20:28:09.886 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-15T20:28:10.887 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-15T20:28:10.887 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T20:28:11.175 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 1 -lt 111669149698 2024-09-15T20:28:11.175 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-15T20:28:12.176 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 299 -eq 0 ']' 2024-09-15T20:28:12.176 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T20:28:12.461 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 111669149698 -lt 111669149698 2024-09-15T20:28:12.461 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T20:28:12.461 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-42949672970 2024-09-15T20:28:12.462 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T20:28:12.462 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-15T20:28:12.463 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-42949672970 2024-09-15T20:28:12.463 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T20:28:12.464 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=42949672970 2024-09-15T20:28:12.464 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 42949672970' 2024-09-15T20:28:12.464 INFO:tasks.workunit.client.0.smithi136.stderr:waiting osd.1 seq 42949672970 2024-09-15T20:28:12.464 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-15T20:28:12.740 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 42949672970 -lt 42949672970 2024-09-15T20:28:12.741 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-15T20:28:12.741 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T20:28:12.741 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T20:28:13.090 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 4 == 0 2024-09-15T20:28:13.091 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-15T20:28:13.091 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-15T20:28:13.091 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-15T20:28:13.091 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T20:28:13.091 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T20:28:13.091 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T20:28:13.091 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T20:28:13.364 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=4 2024-09-15T20:28:13.364 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-15T20:28:13.365 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T20:28:13.365 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T20:28:13.710 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 = 4 2024-09-15T20:28:13.710 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-15T20:28:13.710 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-15T20:28:13.710 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:237: corrupt_and_repair_one: get_pg rbd SOMETHING 2024-09-15T20:28:13.710 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1084: get_pg: local poolname=rbd 2024-09-15T20:28:13.710 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1085: get_pg: local objectname=SOMETHING 2024-09-15T20:28:13.710 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1087: get_pg: ceph --format json osd map rbd SOMETHING 2024-09-15T20:28:13.710 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1087: get_pg: jq -r .pgid 2024-09-15T20:28:13.991 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:237: corrupt_and_repair_one: local pg=1.3 2024-09-15T20:28:13.991 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:238: corrupt_and_repair_one: repair 1.3 2024-09-15T20:28:13.991 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1846: repair: local pgid=1.3 2024-09-15T20:28:13.991 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1847: repair: get_last_scrub_stamp 1.3 2024-09-15T20:28:13.991 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.3 2024-09-15T20:28:13.991 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-15T20:28:13.992 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-15T20:28:13.992 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.3") | .last_scrub_stamp' 2024-09-15T20:28:14.263 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1847: repair: local last_scrub=2024-09-15T20:27:52.570974+0000 2024-09-15T20:28:14.263 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1848: repair: ceph pg repair 1.3 2024-09-15T20:28:14.533 INFO:tasks.workunit.client.0.smithi136.stderr:instructing pg 1.3 on osd.1 to repair 2024-09-15T20:28:14.545 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1849: repair: wait_for_scrub 1.3 2024-09-15T20:27:52.570974+0000 2024-09-15T20:28:14.546 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: local pgid=1.3 2024-09-15T20:28:14.546 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2018: wait_for_scrub: local last_scrub=2024-09-15T20:27:52.570974+0000 2024-09-15T20:28:14.546 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2019: wait_for_scrub: local sname=last_scrub_stamp 2024-09-15T20:28:14.546 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i=0 )) 2024-09-15T20:28:14.546 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-15T20:28:14.546 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.3 last_scrub_stamp 2024-09-15T20:28:14.546 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.3 2024-09-15T20:28:14.546 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-15T20:28:14.546 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-15T20:28:14.546 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.3") | .last_scrub_stamp' 2024-09-15T20:28:14.819 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-15T20:27:52.570974+0000 '>' 2024-09-15T20:27:52.570974+0000 2024-09-15T20:28:14.820 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-15T20:28:15.820 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-15T20:28:15.821 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-15T20:28:15.821 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.3 last_scrub_stamp 2024-09-15T20:28:15.821 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.3 2024-09-15T20:28:15.822 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-15T20:28:15.822 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-15T20:28:15.822 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.3") | .last_scrub_stamp' 2024-09-15T20:28:16.099 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-15T20:27:52.570974+0000 '>' 2024-09-15T20:27:52.570974+0000 2024-09-15T20:28:16.099 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-15T20:28:17.100 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-15T20:28:17.100 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-15T20:28:17.101 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.3 last_scrub_stamp 2024-09-15T20:28:17.101 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.3 2024-09-15T20:28:17.101 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-15T20:28:17.101 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-15T20:28:17.101 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.3") | .last_scrub_stamp' 2024-09-15T20:28:17.387 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-15T20:28:15.002030+0000 '>' 2024-09-15T20:27:52.570974+0000 2024-09-15T20:28:17.387 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2023: wait_for_scrub: return 0 2024-09-15T20:28:17.387 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1850: repair: sleep 2 2024-09-15T20:28:19.389 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:242: corrupt_and_repair_one: objectstore_tool td/osd-scrub-repair 1 SOMETHING list-attrs 2024-09-15T20:28:19.389 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2024-09-15T20:28:19.389 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2024-09-15T20:28:19.389 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=1 2024-09-15T20:28:19.389 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2024-09-15T20:28:19.389 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 SOMETHING list-attrs 2024-09-15T20:28:19.389 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-09-15T20:28:19.389 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2024-09-15T20:28:19.389 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=1 2024-09-15T20:28:19.390 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2024-09-15T20:28:19.390 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-09-15T20:28:19.390 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-15T20:28:19.390 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-15T20:28:19.390 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-15T20:28:19.390 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-15T20:28:19.390 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-15T20:28:19.697 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-15T20:28:19.697 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 SOMETHING list-attrs 2024-09-15T20:28:19.697 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-15T20:28:19.697 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-15T20:28:19.697 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=1 2024-09-15T20:28:19.697 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-15T20:28:19.698 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-09-15T20:28:19.698 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 SOMETHING list-attrs 2024-09-15T20:28:20.246 INFO:tasks.workunit.client.0.smithi136.stderr:Error getting attr on : 1.3_head,#-3:c0000000:::scrub_1.3:head#, (61) No data available 2024-09-15T20:28:20.247 INFO:tasks.workunit.client.0.smithi136.stdout:_ 2024-09-15T20:28:20.247 INFO:tasks.workunit.client.0.smithi136.stdout:snapset 2024-09-15T20:28:20.515 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2024-09-15T20:28:20.515 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2024-09-15T20:28:20.515 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-15T20:28:20.515 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=1 2024-09-15T20:28:20.515 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-15T20:28:20.515 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-09-15T20:28:20.515 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=ce4892c7-6c6d-4b4f-8e6f-07a3e11133e5 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-15T20:28:20.515 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T20:28:20.516 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T20:28:20.516 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T20:28:20.516 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-09-15T20:28:20.516 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-09-15T20:28:20.516 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-15T20:28:20.516 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-15T20:28:20.516 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-15T20:28:20.516 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-15T20:28:20.516 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-15T20:28:20.516 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-15T20:28:20.516 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-15T20:28:20.516 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T20:28:20.516 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52602 2024-09-15T20:28:20.516 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.52602/$cluster-$name.asok' 2024-09-15T20:28:20.517 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.52602/$cluster-$name.asok' 2024-09-15T20:28:20.517 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-15T20:28:20.517 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-15T20:28:20.517 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-15T20:28:20.517 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T20:28:20.517 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T20:28:20.517 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T20:28:20.517 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T20:28:20.517 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-15T20:28:20.517 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-15T20:28:20.517 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-09-15T20:28:20.517 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.1 2024-09-15T20:28:20.517 INFO:tasks.workunit.client.0.smithi136.stderr:start osd.1 2024-09-15T20:28:20.518 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 1 --fsid=ce4892c7-6c6d-4b4f-8e6f-07a3e11133e5 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.52602/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T20:28:20.518 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-09-15T20:28:20.518 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 1 = 1 ']' 2024-09-15T20:28:20.518 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-15T20:28:20.519 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-15T20:28:20.519 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-15T20:28:20.546 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:28:20.544+0000 7ff64a80f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T20:28:20.548 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:28:20.546+0000 7ff64a80f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T20:28:20.549 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:28:20.548+0000 7ff64a80f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T20:28:20.805 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 1 2024-09-15T20:28:20.806 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-15T20:28:20.806 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2024-09-15T20:28:20.806 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-15T20:28:20.806 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-15T20:28:20.806 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:28:20.806 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-15T20:28:20.806 INFO:tasks.workunit.client.0.smithi136.stderr:0 2024-09-15T20:28:20.806 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:28:20.806 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T20:28:21.085 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:28:21.921 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:28:21.919+0000 7ff64a80f640 -1 Falling back to public interface 2024-09-15T20:28:22.086 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:28:22.086 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:28:22.086 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-15T20:28:22.086 INFO:tasks.workunit.client.0.smithi136.stderr:1 2024-09-15T20:28:22.086 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:28:22.087 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T20:28:22.367 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:28:23.369 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:28:23.370 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:28:23.370 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-15T20:28:23.370 INFO:tasks.workunit.client.0.smithi136.stderr:2 2024-09-15T20:28:23.370 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:28:23.370 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T20:28:23.651 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:28:24.653 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:28:24.653 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:28:24.653 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-15T20:28:24.653 INFO:tasks.workunit.client.0.smithi136.stderr:3 2024-09-15T20:28:24.653 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:28:24.653 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T20:28:24.784 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:28:24.782+0000 7ff64a80f640 -1 osd.1 28 log_to_monitors true 2024-09-15T20:28:24.961 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:28:25.963 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:28:25.963 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:28:25.963 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-15T20:28:25.963 INFO:tasks.workunit.client.0.smithi136.stderr:4 2024-09-15T20:28:25.963 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:28:25.963 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T20:28:26.269 INFO:tasks.workunit.client.0.smithi136.stderr:osd.1 up in weight 1 up_from 32 up_thru 32 down_at 29 last_clean_interval [10,28) [v2:127.0.0.1:6810/3303816764,v1:127.0.0.1:6811/3303816764] [v2:127.0.0.1:6812/3303816764,v1:127.0.0.1:6813/3303816764] exists,up c14f43b7-3d6c-4725-9029-717d9f4f2838 2024-09-15T20:28:26.269 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-15T20:28:26.269 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-15T20:28:26.269 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-15T20:28:26.269 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2024-09-15T20:28:26.269 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-15T20:28:26.270 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-15T20:28:26.270 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-15T20:28:26.270 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-15T20:28:26.270 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-15T20:28:26.270 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-15T20:28:26.270 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-15T20:28:26.270 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-15T20:28:26.270 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-15T20:28:26.356 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-15T20:28:26.356 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-15T20:28:26.356 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-15T20:28:26.356 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-15T20:28:26.356 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-15T20:28:26.356 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-15T20:28:26.649 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-15T20:28:26.650 INFO:tasks.workunit.client.0.smithi136.stderr:1' 2024-09-15T20:28:26.650 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-15T20:28:26.650 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T20:28:26.650 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-15T20:28:26.752 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=111669149702 2024-09-15T20:28:26.752 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 111669149702 2024-09-15T20:28:26.752 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-111669149702' 2024-09-15T20:28:26.753 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T20:28:26.753 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-15T20:28:26.855 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=137438953474 2024-09-15T20:28:26.856 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 137438953474 2024-09-15T20:28:26.856 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-111669149702 1-137438953474' 2024-09-15T20:28:26.856 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T20:28:26.856 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-111669149702 2024-09-15T20:28:26.856 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T20:28:26.857 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-15T20:28:26.857 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-111669149702 2024-09-15T20:28:26.857 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T20:28:26.859 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=111669149702 2024-09-15T20:28:26.859 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 111669149702' 2024-09-15T20:28:26.859 INFO:tasks.workunit.client.0.smithi136.stderr:waiting osd.0 seq 111669149702 2024-09-15T20:28:26.859 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T20:28:27.132 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 111669149701 -lt 111669149702 2024-09-15T20:28:27.133 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-15T20:28:28.134 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-15T20:28:28.134 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T20:28:28.421 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 111669149702 -lt 111669149702 2024-09-15T20:28:28.421 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T20:28:28.421 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-137438953474 2024-09-15T20:28:28.421 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T20:28:28.422 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-15T20:28:28.423 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-137438953474 2024-09-15T20:28:28.423 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T20:28:28.424 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=137438953474 2024-09-15T20:28:28.424 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 137438953474' 2024-09-15T20:28:28.424 INFO:tasks.workunit.client.0.smithi136.stderr:waiting osd.1 seq 137438953474 2024-09-15T20:28:28.424 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-15T20:28:28.701 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 137438953474 -lt 137438953474 2024-09-15T20:28:28.701 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-15T20:28:28.701 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T20:28:28.701 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T20:28:29.048 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 4 == 0 2024-09-15T20:28:29.048 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-15T20:28:29.048 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-15T20:28:29.048 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-15T20:28:29.048 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T20:28:29.048 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T20:28:29.049 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T20:28:29.049 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T20:28:29.324 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=4 2024-09-15T20:28:29.325 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-15T20:28:29.325 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T20:28:29.325 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T20:28:29.671 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 = 4 2024-09-15T20:28:29.672 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-15T20:28:29.672 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-15T20:28:29.672 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:243: corrupt_and_repair_one: rados --pool rbd get SOMETHING td/osd-scrub-repair/COPY 2024-09-15T20:28:29.702 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:244: corrupt_and_repair_one: diff td/osd-scrub-repair/ORIGINAL td/osd-scrub-repair/COPY 2024-09-15T20:28:29.704 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:66: run: teardown td/osd-scrub-repair 2024-09-15T20:28:29.704 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/osd-scrub-repair 2024-09-15T20:28:29.704 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-15T20:28:29.704 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/osd-scrub-repair KILL 2024-09-15T20:28:29.704 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-15T20:28:29.704 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-15T20:28:29.704 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-15T20:28:29.704 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-15T20:28:29.704 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-15T20:28:29.810 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-15T20:28:29.810 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-15T20:28:29.811 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-15T20:28:29.811 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-15T20:28:29.813 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-15T20:28:29.813 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-15T20:28:29.813 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-15T20:28:29.813 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T20:28:29.814 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-15T20:28:29.814 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-15T20:28:29.814 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T20:28:29.815 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-15T20:28:29.816 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-15T20:28:29.816 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/osd-scrub-repair 2024-09-15T20:28:29.827 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-15T20:28:29.827 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T20:28:29.827 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52602 2024-09-15T20:28:29.827 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.52602 2024-09-15T20:28:29.829 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-15T20:28:29.829 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-15T20:28:29.829 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:63: run: for func in $funcs 2024-09-15T20:28:29.829 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:64: run: setup td/osd-scrub-repair 2024-09-15T20:28:29.829 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/osd-scrub-repair 2024-09-15T20:28:29.829 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/osd-scrub-repair 2024-09-15T20:28:29.829 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/osd-scrub-repair 2024-09-15T20:28:29.830 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-15T20:28:29.830 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/osd-scrub-repair KILL 2024-09-15T20:28:29.830 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-15T20:28:29.830 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-15T20:28:29.830 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-15T20:28:29.830 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-15T20:28:29.830 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-15T20:28:29.830 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-15T20:28:29.830 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-15T20:28:29.831 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-15T20:28:29.832 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-15T20:28:29.833 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-15T20:28:29.833 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-15T20:28:29.833 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-15T20:28:29.833 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T20:28:29.834 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-15T20:28:29.834 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-15T20:28:29.834 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T20:28:29.835 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-15T20:28:29.836 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-15T20:28:29.836 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/osd-scrub-repair 2024-09-15T20:28:29.837 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-15T20:28:29.837 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T20:28:29.837 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52602 2024-09-15T20:28:29.837 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.52602 2024-09-15T20:28:29.838 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-15T20:28:29.838 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-15T20:28:29.838 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/osd-scrub-repair 2024-09-15T20:28:29.839 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2024-09-15T20:28:29.839 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T20:28:29.839 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52602 2024-09-15T20:28:29.839 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.52602 2024-09-15T20:28:29.842 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2024-09-15T20:28:29.842 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 4096 -le 1024 ']' 2024-09-15T20:28:29.842 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2024-09-15T20:28:29.842 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/osd-scrub-repair 1' TERM HUP INT 2024-09-15T20:28:29.842 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:65: run: TEST_auto_repair_bluestore_basic td/osd-scrub-repair 2024-09-15T20:28:29.842 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:438: TEST_auto_repair_bluestore_basic: local dir=td/osd-scrub-repair 2024-09-15T20:28:29.842 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:439: TEST_auto_repair_bluestore_basic: cluster_conf=(['osds_num']='3' ['pgs_in_pool']='1' ['pool_name']='testpool' ['extras']=' --osd_scrub_auto_repair=true') 2024-09-15T20:28:29.842 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:439: TEST_auto_repair_bluestore_basic: local -A cluster_conf 2024-09-15T20:28:29.842 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:445: TEST_auto_repair_bluestore_basic: local extr_dbg=3 2024-09-15T20:28:29.843 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:446: TEST_auto_repair_bluestore_basic: standard_scrub_cluster td/osd-scrub-repair cluster_conf 2024-09-15T20:28:29.843 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:229: standard_scrub_cluster: local dir=td/osd-scrub-repair 2024-09-15T20:28:29.843 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:230: standard_scrub_cluster: local -n args=cluster_conf 2024-09-15T20:28:29.843 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:232: standard_scrub_cluster: local OSDS=3 2024-09-15T20:28:29.843 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:233: standard_scrub_cluster: local pg_num=1 2024-09-15T20:28:29.843 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:234: standard_scrub_cluster: local poolname=testpool 2024-09-15T20:28:29.843 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:235: standard_scrub_cluster: args['pool_name']=testpool 2024-09-15T20:28:29.843 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:236: standard_scrub_cluster: local 'extra_pars= --osd_scrub_auto_repair=true' 2024-09-15T20:28:29.843 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:237: standard_scrub_cluster: local debug_msg=dbg 2024-09-15T20:28:29.843 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:240: standard_scrub_cluster: local saved_echo_flag=x 2024-09-15T20:28:29.843 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:241: standard_scrub_cluster: set +x 2024-09-15T20:28:31.491 INFO:tasks.workunit.client.0.smithi136.stdout:add osd0 34eb470d-2432-4fae-bd6a-c5b147873211 2024-09-15T20:28:31.713 INFO:tasks.workunit.client.0.smithi136.stdout:0 2024-09-15T20:28:31.760 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:28:31.758+0000 7fe44d410640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T20:28:31.772 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:28:31.771+0000 7fe44d410640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T20:28:31.773 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:28:31.772+0000 7fe44d410640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T20:28:31.774 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:28:31.772+0000 7fe44d410640 -1 bdev(0x5640371ffc00 td/osd-scrub-repair/0/block) open stat got: (1) Operation not permitted 2024-09-15T20:28:31.774 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:28:31.772+0000 7fe44d410640 -1 bluestore(td/osd-scrub-repair/0) _read_fsid unparsable uuid 2024-09-15T20:28:37.511 INFO:tasks.workunit.client.0.smithi136.stdout:adding osd0 key to auth repository 2024-09-15T20:28:37.874 INFO:tasks.workunit.client.0.smithi136.stdout:start osd.0 2024-09-15T20:28:37.909 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:28:37.906+0000 7f5104c0f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T20:28:37.911 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:28:37.908+0000 7f5104c0f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T20:28:37.912 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:28:37.909+0000 7f5104c0f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T20:28:38.151 INFO:tasks.workunit.client.0.smithi136.stdout:0 2024-09-15T20:28:39.426 INFO:tasks.workunit.client.0.smithi136.stdout:1 2024-09-15T20:28:39.503 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:28:39.500+0000 7f5104c0f640 -1 Falling back to public interface 2024-09-15T20:28:40.719 INFO:tasks.workunit.client.0.smithi136.stdout:2 2024-09-15T20:28:42.004 INFO:tasks.workunit.client.0.smithi136.stdout:3 2024-09-15T20:28:42.496 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:28:42.494+0000 7f5104c0f640 -1 osd.0 0 log_to_monitors true 2024-09-15T20:28:43.290 INFO:tasks.workunit.client.0.smithi136.stdout:4 2024-09-15T20:28:44.581 INFO:tasks.workunit.client.0.smithi136.stdout:5 2024-09-15T20:28:45.874 INFO:tasks.workunit.client.0.smithi136.stdout:6 2024-09-15T20:28:47.162 INFO:tasks.workunit.client.0.smithi136.stdout:7 2024-09-15T20:28:48.547 INFO:tasks.workunit.client.0.smithi136.stdout:8 2024-09-15T20:28:49.837 INFO:tasks.workunit.client.0.smithi136.stdout:9 2024-09-15T20:28:51.156 INFO:tasks.workunit.client.0.smithi136.stdout:10 2024-09-15T20:28:52.501 INFO:tasks.workunit.client.0.smithi136.stdout:11 2024-09-15T20:28:53.790 INFO:tasks.workunit.client.0.smithi136.stdout:12 2024-09-15T20:28:55.266 INFO:tasks.workunit.client.0.smithi136.stdout:13 2024-09-15T20:28:56.851 INFO:tasks.workunit.client.0.smithi136.stdout:14 2024-09-15T20:28:58.508 INFO:tasks.workunit.client.0.smithi136.stdout:15 2024-09-15T20:29:00.023 INFO:tasks.workunit.client.0.smithi136.stdout:16 2024-09-15T20:29:01.356 INFO:tasks.workunit.client.0.smithi136.stdout:17 2024-09-15T20:29:02.726 INFO:tasks.workunit.client.0.smithi136.stdout:18 2024-09-15T20:29:04.197 INFO:tasks.workunit.client.0.smithi136.stdout:19 2024-09-15T20:29:05.573 INFO:tasks.workunit.client.0.smithi136.stdout:20 2024-09-15T20:29:05.870 INFO:tasks.workunit.client.0.smithi136.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/679146318,v1:127.0.0.1:6803/679146318] [v2:127.0.0.1:6804/679146318,v1:127.0.0.1:6805/679146318] exists,up 34eb470d-2432-4fae-bd6a-c5b147873211 2024-09-15T20:29:05.874 INFO:tasks.workunit.client.0.smithi136.stdout:add osd1 e2b0abc8-3dcb-42a5-8cc5-6a2e17284cdb 2024-09-15T20:29:06.224 INFO:tasks.workunit.client.0.smithi136.stdout:1 2024-09-15T20:29:06.273 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:29:06.271+0000 7efdee611640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T20:29:06.301 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:29:06.299+0000 7efdee611640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T20:29:06.302 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:29:06.300+0000 7efdee611640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T20:29:06.302 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:29:06.300+0000 7efdee611640 -1 bdev(0x55a46e9edc00 td/osd-scrub-repair/1/block) open stat got: (1) Operation not permitted 2024-09-15T20:29:06.302 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:29:06.301+0000 7efdee611640 -1 bluestore(td/osd-scrub-repair/1) _read_fsid unparsable uuid 2024-09-15T20:29:12.048 INFO:tasks.workunit.client.0.smithi136.stdout:adding osd1 key to auth repository 2024-09-15T20:29:12.407 INFO:tasks.workunit.client.0.smithi136.stdout:start osd.1 2024-09-15T20:29:12.441 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:29:12.439+0000 7ff90c7c5640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T20:29:12.443 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:29:12.441+0000 7ff90c7c5640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T20:29:12.444 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:29:12.442+0000 7ff90c7c5640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T20:29:12.684 INFO:tasks.workunit.client.0.smithi136.stdout:0 2024-09-15T20:29:13.808 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:29:13.806+0000 7ff90c7c5640 -1 Falling back to public interface 2024-09-15T20:29:13.960 INFO:tasks.workunit.client.0.smithi136.stdout:1 2024-09-15T20:29:15.246 INFO:tasks.workunit.client.0.smithi136.stdout:2 2024-09-15T20:29:16.416 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:29:16.414+0000 7ff90c7c5640 -1 osd.1 0 log_to_monitors true 2024-09-15T20:29:16.535 INFO:tasks.workunit.client.0.smithi136.stdout:3 2024-09-15T20:29:17.900 INFO:tasks.workunit.client.0.smithi136.stdout:4 2024-09-15T20:29:19.190 INFO:tasks.workunit.client.0.smithi136.stdout:5 2024-09-15T20:29:20.519 INFO:tasks.workunit.client.0.smithi136.stdout:6 2024-09-15T20:29:21.810 INFO:tasks.workunit.client.0.smithi136.stdout:7 2024-09-15T20:29:23.213 INFO:tasks.workunit.client.0.smithi136.stdout:8 2024-09-15T20:29:24.561 INFO:tasks.workunit.client.0.smithi136.stdout:9 2024-09-15T20:29:25.850 INFO:tasks.workunit.client.0.smithi136.stdout:10 2024-09-15T20:29:27.207 INFO:tasks.workunit.client.0.smithi136.stdout:11 2024-09-15T20:29:28.545 INFO:tasks.workunit.client.0.smithi136.stdout:12 2024-09-15T20:29:29.837 INFO:tasks.workunit.client.0.smithi136.stdout:13 2024-09-15T20:29:31.633 INFO:tasks.workunit.client.0.smithi136.stdout:14 2024-09-15T20:29:33.418 INFO:tasks.workunit.client.0.smithi136.stdout:15 2024-09-15T20:29:34.805 INFO:tasks.workunit.client.0.smithi136.stdout:16 2024-09-15T20:29:36.468 INFO:tasks.workunit.client.0.smithi136.stdout:17 2024-09-15T20:29:38.028 INFO:tasks.workunit.client.0.smithi136.stdout:18 2024-09-15T20:29:39.410 INFO:tasks.workunit.client.0.smithi136.stdout:19 2024-09-15T20:29:39.689 INFO:tasks.workunit.client.0.smithi136.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/3894067837,v1:127.0.0.1:6811/3894067837] [v2:127.0.0.1:6812/3894067837,v1:127.0.0.1:6813/3894067837] exists,up e2b0abc8-3dcb-42a5-8cc5-6a2e17284cdb 2024-09-15T20:29:39.693 INFO:tasks.workunit.client.0.smithi136.stdout:add osd2 fdb305dd-4832-4660-a03d-b0c47b2864bb 2024-09-15T20:29:40.038 INFO:tasks.workunit.client.0.smithi136.stdout:2 2024-09-15T20:29:40.086 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:29:40.084+0000 7fb7aea0c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T20:29:40.122 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:29:40.121+0000 7fb7aea0c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T20:29:40.123 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:29:40.122+0000 7fb7aea0c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T20:29:40.124 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:29:40.122+0000 7fb7aea0c640 -1 bdev(0x557ab881fc00 td/osd-scrub-repair/2/block) open stat got: (1) Operation not permitted 2024-09-15T20:29:40.124 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:29:40.122+0000 7fb7aea0c640 -1 bluestore(td/osd-scrub-repair/2) _read_fsid unparsable uuid 2024-09-15T20:29:46.451 INFO:tasks.workunit.client.0.smithi136.stdout:adding osd2 key to auth repository 2024-09-15T20:29:46.811 INFO:tasks.workunit.client.0.smithi136.stdout:start osd.2 2024-09-15T20:29:46.846 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:29:46.844+0000 7f6c5be11640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T20:29:46.848 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:29:46.846+0000 7f6c5be11640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T20:29:46.850 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:29:46.848+0000 7f6c5be11640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T20:29:47.089 INFO:tasks.workunit.client.0.smithi136.stdout:0 2024-09-15T20:29:48.219 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:29:48.217+0000 7f6c5be11640 -1 Falling back to public interface 2024-09-15T20:29:48.370 INFO:tasks.workunit.client.0.smithi136.stdout:1 2024-09-15T20:29:49.653 INFO:tasks.workunit.client.0.smithi136.stdout:2 2024-09-15T20:29:50.839 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:29:50.837+0000 7f6c5be11640 -1 osd.2 0 log_to_monitors true 2024-09-15T20:29:50.941 INFO:tasks.workunit.client.0.smithi136.stdout:3 2024-09-15T20:29:52.340 INFO:tasks.workunit.client.0.smithi136.stdout:4 2024-09-15T20:29:53.764 INFO:tasks.workunit.client.0.smithi136.stdout:5 2024-09-15T20:29:55.078 INFO:tasks.workunit.client.0.smithi136.stdout:6 2024-09-15T20:29:56.368 INFO:tasks.workunit.client.0.smithi136.stdout:7 2024-09-15T20:29:57.658 INFO:tasks.workunit.client.0.smithi136.stdout:8 2024-09-15T20:29:59.045 INFO:tasks.workunit.client.0.smithi136.stdout:9 2024-09-15T20:30:00.332 INFO:tasks.workunit.client.0.smithi136.stdout:10 2024-09-15T20:30:01.699 INFO:tasks.workunit.client.0.smithi136.stdout:11 2024-09-15T20:30:03.063 INFO:tasks.workunit.client.0.smithi136.stdout:12 2024-09-15T20:30:04.352 INFO:tasks.workunit.client.0.smithi136.stdout:13 2024-09-15T20:30:05.718 INFO:tasks.workunit.client.0.smithi136.stdout:14 2024-09-15T20:30:07.613 INFO:tasks.workunit.client.0.smithi136.stdout:15 2024-09-15T20:30:09.227 INFO:tasks.workunit.client.0.smithi136.stdout:16 2024-09-15T20:30:10.847 INFO:tasks.workunit.client.0.smithi136.stdout:17 2024-09-15T20:30:12.520 INFO:tasks.workunit.client.0.smithi136.stdout:18 2024-09-15T20:30:14.099 INFO:tasks.workunit.client.0.smithi136.stdout:19 2024-09-15T20:30:14.444 INFO:tasks.workunit.client.0.smithi136.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/344488065,v1:127.0.0.1:6819/344488065] [v2:127.0.0.1:6820/344488065,v1:127.0.0.1:6821/344488065] exists,up fdb305dd-4832-4660-a03d-b0c47b2864bb 2024-09-15T20:30:14.774 INFO:tasks.workunit.client.0.smithi136.stderr:pool 'testpool' created 2024-09-15T20:30:16.514 INFO:tasks.workunit.client.0.smithi136.stdout:waiting osd.0 seq 21474836499 2024-09-15T20:30:18.080 INFO:tasks.workunit.client.0.smithi136.stdout:waiting osd.1 seq 42949672977 2024-09-15T20:30:18.361 INFO:tasks.workunit.client.0.smithi136.stdout:waiting osd.2 seq 64424509448 2024-09-15T20:30:20.878 INFO:tasks.workunit.client.0.smithi136.stdout:standard_scrub_cluster: dbg: test pool is testpool 1 2024-09-15T20:30:20.879 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:447: TEST_auto_repair_bluestore_basic: local poolid=1 2024-09-15T20:30:20.879 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:448: TEST_auto_repair_bluestore_basic: local poolname=testpool 2024-09-15T20:30:20.879 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:450: TEST_auto_repair_bluestore_basic: ceph osd pool set testpool size 2 2024-09-15T20:30:21.247 INFO:tasks.workunit.client.0.smithi136.stderr:set pool 1 size to 2 2024-09-15T20:30:21.259 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:451: TEST_auto_repair_bluestore_basic: wait_for_clean 2024-09-15T20:30:21.259 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-15T20:30:21.259 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-15T20:30:21.259 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-15T20:30:21.259 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-15T20:30:21.260 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-15T20:30:21.260 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-15T20:30:21.260 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-15T20:30:21.260 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-15T20:30:21.260 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-15T20:30:21.348 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-15T20:30:21.348 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-15T20:30:21.348 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-15T20:30:21.348 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-15T20:30:21.348 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-15T20:30:21.349 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-15T20:30:21.737 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-15T20:30:21.737 INFO:tasks.workunit.client.0.smithi136.stderr:1 2024-09-15T20:30:21.737 INFO:tasks.workunit.client.0.smithi136.stderr:2' 2024-09-15T20:30:21.737 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-15T20:30:21.737 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T20:30:21.737 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-15T20:30:21.861 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836501 2024-09-15T20:30:21.861 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836501 2024-09-15T20:30:21.861 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836501' 2024-09-15T20:30:21.861 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T20:30:21.861 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-15T20:30:21.971 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=42949672979 2024-09-15T20:30:21.971 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 42949672979 2024-09-15T20:30:21.971 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836501 1-42949672979' 2024-09-15T20:30:21.971 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T20:30:21.971 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-15T20:30:22.105 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=64424509450 2024-09-15T20:30:22.105 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 64424509450 2024-09-15T20:30:22.105 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836501 1-42949672979 2-64424509450' 2024-09-15T20:30:22.105 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T20:30:22.106 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836501 2024-09-15T20:30:22.106 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T20:30:22.106 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-15T20:30:22.107 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836501 2024-09-15T20:30:22.107 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T20:30:22.108 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836501 2024-09-15T20:30:22.108 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836501' 2024-09-15T20:30:22.108 INFO:tasks.workunit.client.0.smithi136.stdout:waiting osd.0 seq 21474836501 2024-09-15T20:30:22.108 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T20:30:22.385 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836500 -lt 21474836501 2024-09-15T20:30:22.385 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-15T20:30:23.386 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-15T20:30:23.386 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T20:30:23.693 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836501 -lt 21474836501 2024-09-15T20:30:23.693 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T20:30:23.693 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-42949672979 2024-09-15T20:30:23.693 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T20:30:23.694 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-15T20:30:23.695 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-42949672979 2024-09-15T20:30:23.695 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T20:30:23.696 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=42949672979 2024-09-15T20:30:23.696 INFO:tasks.workunit.client.0.smithi136.stdout:waiting osd.1 seq 42949672979 2024-09-15T20:30:23.696 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 42949672979' 2024-09-15T20:30:23.696 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-15T20:30:23.973 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 42949672979 -lt 42949672979 2024-09-15T20:30:23.973 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T20:30:23.974 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-64424509450 2024-09-15T20:30:23.974 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T20:30:23.974 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-15T20:30:23.975 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-64424509450 2024-09-15T20:30:23.975 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T20:30:23.976 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=64424509450 2024-09-15T20:30:23.976 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 64424509450' 2024-09-15T20:30:23.976 INFO:tasks.workunit.client.0.smithi136.stdout:waiting osd.2 seq 64424509450 2024-09-15T20:30:23.977 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-15T20:30:24.252 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 64424509450 -lt 64424509450 2024-09-15T20:30:24.253 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-15T20:30:24.253 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T20:30:24.253 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T20:30:24.596 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 1 == 0 2024-09-15T20:30:24.596 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-15T20:30:24.596 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-15T20:30:24.596 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-15T20:30:24.596 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T20:30:24.596 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T20:30:24.596 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T20:30:24.596 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T20:30:24.867 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=0 2024-09-15T20:30:24.868 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-15T20:30:24.868 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T20:30:24.868 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T20:30:25.210 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 0 = 1 2024-09-15T20:30:25.211 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 0 '!=' -1 2024-09-15T20:30:25.211 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: loop=0 2024-09-15T20:30:25.211 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1652: wait_for_clean: num_active_clean=0 2024-09-15T20:30:25.211 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-15T20:30:25.211 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 0.1 2024-09-15T20:30:25.312 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-15T20:30:25.312 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-15T20:30:25.312 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-15T20:30:25.312 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-15T20:30:25.312 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T20:30:25.312 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T20:30:25.312 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T20:30:25.313 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T20:30:25.586 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=1 2024-09-15T20:30:25.587 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-15T20:30:25.587 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T20:30:25.587 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T20:30:25.932 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 = 1 2024-09-15T20:30:25.932 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-15T20:30:25.932 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-15T20:30:25.932 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:454: TEST_auto_repair_bluestore_basic: local payload=ABCDEF 2024-09-15T20:30:25.933 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:455: TEST_auto_repair_bluestore_basic: echo ABCDEF 2024-09-15T20:30:25.933 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:456: TEST_auto_repair_bluestore_basic: rados --pool testpool put SOMETHING td/osd-scrub-repair/ORIGINAL 2024-09-15T20:30:26.068 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:460: TEST_auto_repair_bluestore_basic: get_not_primary testpool SOMETHING 2024-09-15T20:30:26.068 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1220: get_not_primary: local poolname=testpool 2024-09-15T20:30:26.068 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1221: get_not_primary: local objectname=SOMETHING 2024-09-15T20:30:26.068 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1223: get_not_primary: get_primary testpool SOMETHING 2024-09-15T20:30:26.069 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1187: get_primary: local poolname=testpool 2024-09-15T20:30:26.069 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1188: get_primary: local objectname=SOMETHING 2024-09-15T20:30:26.069 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1190: get_primary: ceph --format json osd map testpool SOMETHING 2024-09-15T20:30:26.069 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1191: get_primary: jq .acting_primary 2024-09-15T20:30:26.349 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1223: get_not_primary: local primary=1 2024-09-15T20:30:26.349 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1224: get_not_primary: ceph --format json osd map testpool SOMETHING 2024-09-15T20:30:26.349 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1225: get_not_primary: jq '.acting | map(select (. != 1)) | .[0]' 2024-09-15T20:30:26.656 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:460: TEST_auto_repair_bluestore_basic: objectstore_tool td/osd-scrub-repair 0 SOMETHING remove 2024-09-15T20:30:26.657 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2024-09-15T20:30:26.657 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2024-09-15T20:30:26.657 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=0 2024-09-15T20:30:26.657 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2024-09-15T20:30:26.657 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 SOMETHING remove 2024-09-15T20:30:26.657 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-09-15T20:30:26.657 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2024-09-15T20:30:26.657 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=0 2024-09-15T20:30:26.657 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2024-09-15T20:30:26.657 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2024-09-15T20:30:26.657 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-15T20:30:26.657 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-15T20:30:26.658 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-15T20:30:26.658 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-15T20:30:26.658 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-15T20:30:26.963 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-15T20:30:26.963 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 SOMETHING remove 2024-09-15T20:30:26.963 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-15T20:30:26.963 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-15T20:30:26.963 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=0 2024-09-15T20:30:26.963 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-15T20:30:26.963 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-09-15T20:30:26.963 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 SOMETHING remove 2024-09-15T20:30:28.572 INFO:tasks.workunit.client.0.smithi136.stdout:remove #1:eb822e21:::SOMETHING:head# 2024-09-15T20:30:29.090 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 2024-09-15T20:30:29.090 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2024-09-15T20:30:29.090 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-15T20:30:29.090 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=0 2024-09-15T20:30:29.090 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-15T20:30:29.090 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-09-15T20:30:29.090 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=ce4892c7-6c6d-4b4f-8e6f-07a3e11133e5 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-15T20:30:29.091 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T20:30:29.091 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T20:30:29.091 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T20:30:29.091 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-09-15T20:30:29.091 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-09-15T20:30:29.091 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-15T20:30:29.091 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-15T20:30:29.091 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-15T20:30:29.091 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-15T20:30:29.091 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-15T20:30:29.091 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-15T20:30:29.091 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-15T20:30:29.092 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T20:30:29.092 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52602 2024-09-15T20:30:29.092 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.52602/$cluster-$name.asok' 2024-09-15T20:30:29.092 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.52602/$cluster-$name.asok' 2024-09-15T20:30:29.092 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-15T20:30:29.092 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-15T20:30:29.092 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-15T20:30:29.092 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T20:30:29.092 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T20:30:29.092 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T20:30:29.092 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T20:30:29.092 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-15T20:30:29.092 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-15T20:30:29.092 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-09-15T20:30:29.093 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.0 2024-09-15T20:30:29.093 INFO:tasks.workunit.client.0.smithi136.stderr:start osd.0 2024-09-15T20:30:29.093 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 0 --fsid=ce4892c7-6c6d-4b4f-8e6f-07a3e11133e5 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.52602/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T20:30:29.094 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-09-15T20:30:29.094 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 0 = 0 ']' 2024-09-15T20:30:29.094 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-15T20:30:29.094 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-15T20:30:29.094 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-15T20:30:29.121 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:30:29.119+0000 7ffb4a5a8640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T20:30:29.123 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:30:29.122+0000 7ffb4a5a8640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T20:30:29.125 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:30:29.123+0000 7ffb4a5a8640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T20:30:29.376 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 0 2024-09-15T20:30:29.376 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-15T20:30:29.376 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2024-09-15T20:30:29.377 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-15T20:30:29.377 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-15T20:30:29.377 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:30:29.377 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-15T20:30:29.377 INFO:tasks.workunit.client.0.smithi136.stderr:0 2024-09-15T20:30:29.377 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:30:29.377 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T20:30:29.670 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:30:30.238 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:30:30.236+0000 7ffb4a5a8640 -1 Falling back to public interface 2024-09-15T20:30:30.671 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:30:30.671 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:30:30.671 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-15T20:30:30.671 INFO:tasks.workunit.client.0.smithi136.stderr:1 2024-09-15T20:30:30.672 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:30:30.672 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T20:30:30.961 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:30:31.962 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:30:31.962 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:30:31.962 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-15T20:30:31.963 INFO:tasks.workunit.client.0.smithi136.stderr:2 2024-09-15T20:30:31.963 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:30:31.963 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T20:30:32.258 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:30:32.700 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:30:32.698+0000 7ffb4a5a8640 -1 osd.0 20 log_to_monitors true 2024-09-15T20:30:33.260 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:30:33.260 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:30:33.260 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-15T20:30:33.260 INFO:tasks.workunit.client.0.smithi136.stderr:3 2024-09-15T20:30:33.260 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:30:33.261 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T20:30:33.746 INFO:tasks.workunit.client.0.smithi136.stderr:osd.0 up in weight 1 up_from 24 up_thru 0 down_at 21 last_clean_interval [5,20) [v2:127.0.0.1:6802/842719307,v1:127.0.0.1:6803/842719307] [v2:127.0.0.1:6804/842719307,v1:127.0.0.1:6805/842719307] exists,up 34eb470d-2432-4fae-bd6a-c5b147873211 2024-09-15T20:30:33.746 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-15T20:30:33.746 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-15T20:30:33.746 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-15T20:30:33.747 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2024-09-15T20:30:33.747 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-15T20:30:33.747 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-15T20:30:33.747 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-15T20:30:33.747 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-15T20:30:33.748 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-15T20:30:33.748 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-15T20:30:33.748 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-15T20:30:33.748 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-15T20:30:33.748 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-15T20:30:33.835 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-15T20:30:33.836 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-15T20:30:33.836 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-15T20:30:33.836 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-15T20:30:33.836 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-15T20:30:33.836 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-15T20:30:34.116 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-15T20:30:34.116 INFO:tasks.workunit.client.0.smithi136.stderr:1 2024-09-15T20:30:34.116 INFO:tasks.workunit.client.0.smithi136.stderr:2' 2024-09-15T20:30:34.116 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-15T20:30:34.116 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T20:30:34.116 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-15T20:30:34.241 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=103079215106 2024-09-15T20:30:34.242 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 103079215106 2024-09-15T20:30:34.242 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-103079215106' 2024-09-15T20:30:34.242 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T20:30:34.242 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-15T20:30:34.343 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=42949672982 2024-09-15T20:30:34.343 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 42949672982 2024-09-15T20:30:34.344 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-103079215106 1-42949672982' 2024-09-15T20:30:34.344 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T20:30:34.344 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-15T20:30:34.446 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=64424509453 2024-09-15T20:30:34.446 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 64424509453 2024-09-15T20:30:34.446 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-103079215106 1-42949672982 2-64424509453' 2024-09-15T20:30:34.446 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T20:30:34.446 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-103079215106 2024-09-15T20:30:34.446 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T20:30:34.447 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-15T20:30:34.448 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-103079215106 2024-09-15T20:30:34.448 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T20:30:34.449 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=103079215106 2024-09-15T20:30:34.449 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 103079215106' 2024-09-15T20:30:34.449 INFO:tasks.workunit.client.0.smithi136.stderr:waiting osd.0 seq 103079215106 2024-09-15T20:30:34.449 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T20:30:34.724 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 1 -lt 103079215106 2024-09-15T20:30:34.725 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-15T20:30:35.726 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-15T20:30:35.726 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T20:30:36.014 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 103079215106 -lt 103079215106 2024-09-15T20:30:36.014 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T20:30:36.014 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-42949672982 2024-09-15T20:30:36.014 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T20:30:36.015 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-15T20:30:36.016 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-42949672982 2024-09-15T20:30:36.016 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T20:30:36.017 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=42949672982 2024-09-15T20:30:36.017 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 42949672982' 2024-09-15T20:30:36.017 INFO:tasks.workunit.client.0.smithi136.stderr:waiting osd.1 seq 42949672982 2024-09-15T20:30:36.017 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-15T20:30:36.289 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 42949672983 -lt 42949672982 2024-09-15T20:30:36.290 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T20:30:36.290 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-64424509453 2024-09-15T20:30:36.290 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T20:30:36.291 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-15T20:30:36.291 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-64424509453 2024-09-15T20:30:36.291 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T20:30:36.292 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=64424509453 2024-09-15T20:30:36.293 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 64424509453' 2024-09-15T20:30:36.293 INFO:tasks.workunit.client.0.smithi136.stderr:waiting osd.2 seq 64424509453 2024-09-15T20:30:36.293 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-15T20:30:36.567 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 64424509454 -lt 64424509453 2024-09-15T20:30:36.567 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-15T20:30:36.567 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T20:30:36.567 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T20:30:36.917 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 1 == 0 2024-09-15T20:30:36.917 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-15T20:30:36.917 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-15T20:30:36.917 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-15T20:30:36.917 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T20:30:36.917 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T20:30:36.918 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T20:30:36.918 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T20:30:37.190 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=1 2024-09-15T20:30:37.190 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-15T20:30:37.190 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T20:30:37.190 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T20:30:37.534 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 = 1 2024-09-15T20:30:37.534 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-15T20:30:37.534 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-15T20:30:37.534 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:461: TEST_auto_repair_bluestore_basic: ceph tell 'osd.*' config set osd_scrub_auto_repair true 2024-09-15T20:30:37.660 INFO:tasks.workunit.client.0.smithi136.stdout:osd.0: { 2024-09-15T20:30:37.661 INFO:tasks.workunit.client.0.smithi136.stdout: "success": "osd_delete_sleep = '' osd_delete_sleep_hdd = '' osd_delete_sleep_hybrid = '' osd_delete_sleep_ssd = '' osd_max_backfills = '' osd_recovery_max_active = '' osd_recovery_max_active_hdd = '' osd_recovery_max_active_ssd = '' osd_recovery_sleep = '' osd_recovery_sleep_hdd = '' osd_recovery_sleep_hybrid = '' osd_recovery_sleep_ssd = '' osd_scrub_auto_repair = '' (not observed, change may require restart) osd_scrub_sleep = '' osd_snap_trim_sleep = '' osd_snap_trim_sleep_hdd = '' osd_snap_trim_sleep_hybrid = '' osd_snap_trim_sleep_ssd = '' " 2024-09-15T20:30:37.661 INFO:tasks.workunit.client.0.smithi136.stdout:} 2024-09-15T20:30:37.674 INFO:tasks.workunit.client.0.smithi136.stdout:osd.1: { 2024-09-15T20:30:37.674 INFO:tasks.workunit.client.0.smithi136.stdout: "success": "osd_scrub_auto_repair = '' (not observed, change may require restart) " 2024-09-15T20:30:37.674 INFO:tasks.workunit.client.0.smithi136.stdout:} 2024-09-15T20:30:37.687 INFO:tasks.workunit.client.0.smithi136.stdout:osd.2: { 2024-09-15T20:30:37.687 INFO:tasks.workunit.client.0.smithi136.stdout: "success": "osd_scrub_auto_repair = '' (not observed, change may require restart) " 2024-09-15T20:30:37.687 INFO:tasks.workunit.client.0.smithi136.stdout:} 2024-09-15T20:30:37.696 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:463: TEST_auto_repair_bluestore_basic: get_pg testpool SOMETHING 2024-09-15T20:30:37.696 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1084: get_pg: local poolname=testpool 2024-09-15T20:30:37.696 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1085: get_pg: local objectname=SOMETHING 2024-09-15T20:30:37.696 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1087: get_pg: ceph --format json osd map testpool SOMETHING 2024-09-15T20:30:37.696 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1087: get_pg: jq -r .pgid 2024-09-15T20:30:37.985 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:463: TEST_auto_repair_bluestore_basic: local pgid=1.0 2024-09-15T20:30:37.986 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:464: TEST_auto_repair_bluestore_basic: get_primary testpool SOMETHING 2024-09-15T20:30:37.986 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1187: get_primary: local poolname=testpool 2024-09-15T20:30:37.986 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1188: get_primary: local objectname=SOMETHING 2024-09-15T20:30:37.986 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1190: get_primary: ceph --format json osd map testpool SOMETHING 2024-09-15T20:30:37.986 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1191: get_primary: jq .acting_primary 2024-09-15T20:30:38.263 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:464: TEST_auto_repair_bluestore_basic: local primary=1 2024-09-15T20:30:38.264 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:465: TEST_auto_repair_bluestore_basic: get_last_scrub_stamp 1.0 2024-09-15T20:30:38.264 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-15T20:30:38.264 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-15T20:30:38.264 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-15T20:30:38.264 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-15T20:30:38.538 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:465: TEST_auto_repair_bluestore_basic: local last_scrub_stamp=2024-09-15T20:30:14.723350+0000 2024-09-15T20:30:38.538 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:468: TEST_auto_repair_bluestore_basic: ceph tell 1.0 schedule-deep-scrub 2024-09-15T20:30:38.634 INFO:tasks.workunit.client.0.smithi136.stdout:{ 2024-09-15T20:30:38.635 INFO:tasks.workunit.client.0.smithi136.stdout: "deep": true, 2024-09-15T20:30:38.635 INFO:tasks.workunit.client.0.smithi136.stdout: "must": false, 2024-09-15T20:30:38.635 INFO:tasks.workunit.client.0.smithi136.stdout: "stamp": "2024-09-01T20:28:58.633647+0000" 2024-09-15T20:30:38.635 INFO:tasks.workunit.client.0.smithi136.stdout:} 2024-09-15T20:30:38.644 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:471: TEST_auto_repair_bluestore_basic: wait_for_scrub 1.0 2024-09-15T20:30:14.723350+0000 2024-09-15T20:30:38.645 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: local pgid=1.0 2024-09-15T20:30:38.645 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2018: wait_for_scrub: local last_scrub=2024-09-15T20:30:14.723350+0000 2024-09-15T20:30:38.645 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2019: wait_for_scrub: local sname=last_scrub_stamp 2024-09-15T20:30:38.645 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i=0 )) 2024-09-15T20:30:38.645 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-15T20:30:38.645 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-15T20:30:38.645 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-15T20:30:38.645 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-15T20:30:38.645 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-15T20:30:38.645 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-15T20:30:38.915 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-15T20:30:14.723350+0000 '>' 2024-09-15T20:30:14.723350+0000 2024-09-15T20:30:38.915 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-15T20:30:39.917 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-15T20:30:39.917 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-15T20:30:39.918 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-15T20:30:39.918 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-15T20:30:39.918 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-15T20:30:39.918 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-15T20:30:39.918 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-15T20:30:40.199 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-15T20:30:14.723350+0000 '>' 2024-09-15T20:30:14.723350+0000 2024-09-15T20:30:40.199 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-15T20:30:41.200 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-15T20:30:41.200 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-15T20:30:41.201 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-15T20:30:41.201 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-15T20:30:41.201 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-15T20:30:41.201 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-15T20:30:41.201 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-15T20:30:41.479 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-15T20:30:38.994001+0000 '>' 2024-09-15T20:30:14.723350+0000 2024-09-15T20:30:41.479 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2023: wait_for_scrub: return 0 2024-09-15T20:30:41.479 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:472: TEST_auto_repair_bluestore_basic: wait_for_clean 2024-09-15T20:30:41.479 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-15T20:30:41.479 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-15T20:30:41.479 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-15T20:30:41.479 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-15T20:30:41.480 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-15T20:30:41.480 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-15T20:30:41.480 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-15T20:30:41.480 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-15T20:30:41.480 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-15T20:30:41.568 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-15T20:30:41.568 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-15T20:30:41.568 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-15T20:30:41.568 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-15T20:30:41.569 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-15T20:30:41.569 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-15T20:30:41.850 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-15T20:30:41.850 INFO:tasks.workunit.client.0.smithi136.stderr:1 2024-09-15T20:30:41.850 INFO:tasks.workunit.client.0.smithi136.stderr:2' 2024-09-15T20:30:41.850 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-15T20:30:41.850 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T20:30:41.850 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-15T20:30:41.953 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=103079215108 2024-09-15T20:30:41.953 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 103079215108 2024-09-15T20:30:41.953 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-103079215108' 2024-09-15T20:30:41.954 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T20:30:41.954 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-15T20:30:42.056 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=42949672985 2024-09-15T20:30:42.056 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 42949672985 2024-09-15T20:30:42.057 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-103079215108 1-42949672985' 2024-09-15T20:30:42.057 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T20:30:42.057 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-15T20:30:42.159 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=64424509456 2024-09-15T20:30:42.159 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 64424509456 2024-09-15T20:30:42.159 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-103079215108 1-42949672985 2-64424509456' 2024-09-15T20:30:42.159 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T20:30:42.159 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-103079215108 2024-09-15T20:30:42.160 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T20:30:42.160 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-15T20:30:42.161 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-103079215108 2024-09-15T20:30:42.161 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T20:30:42.162 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=103079215108 2024-09-15T20:30:42.162 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 103079215108' 2024-09-15T20:30:42.162 INFO:tasks.workunit.client.0.smithi136.stdout:waiting osd.0 seq 103079215108 2024-09-15T20:30:42.162 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T20:30:42.436 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 103079215107 -lt 103079215108 2024-09-15T20:30:42.437 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-15T20:30:43.438 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-15T20:30:43.438 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T20:30:43.727 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 103079215109 -lt 103079215108 2024-09-15T20:30:43.727 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T20:30:43.728 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-42949672985 2024-09-15T20:30:43.728 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T20:30:43.728 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-15T20:30:43.729 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-42949672985 2024-09-15T20:30:43.729 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T20:30:43.730 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=42949672985 2024-09-15T20:30:43.730 INFO:tasks.workunit.client.0.smithi136.stdout:waiting osd.1 seq 42949672985 2024-09-15T20:30:43.731 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 42949672985' 2024-09-15T20:30:43.731 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-15T20:30:44.010 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 42949672985 -lt 42949672985 2024-09-15T20:30:44.010 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T20:30:44.011 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-64424509456 2024-09-15T20:30:44.011 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T20:30:44.011 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-15T20:30:44.012 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-64424509456 2024-09-15T20:30:44.012 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T20:30:44.013 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=64424509456 2024-09-15T20:30:44.013 INFO:tasks.workunit.client.0.smithi136.stdout:waiting osd.2 seq 64424509456 2024-09-15T20:30:44.014 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 64424509456' 2024-09-15T20:30:44.014 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-15T20:30:44.323 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 64424509456 -lt 64424509456 2024-09-15T20:30:44.324 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-15T20:30:44.324 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T20:30:44.324 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T20:30:44.664 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 1 == 0 2024-09-15T20:30:44.664 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-15T20:30:44.664 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-15T20:30:44.664 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-15T20:30:44.664 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T20:30:44.664 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T20:30:44.664 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T20:30:44.664 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T20:30:44.939 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=1 2024-09-15T20:30:44.939 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-15T20:30:44.939 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T20:30:44.940 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T20:30:45.277 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 = 1 2024-09-15T20:30:45.277 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-15T20:30:45.278 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-15T20:30:45.278 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:473: TEST_auto_repair_bluestore_basic: ceph pg dump pgs 2024-09-15T20:30:45.541 INFO:tasks.workunit.client.0.smithi136.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-15T20:30:45.541 INFO:tasks.workunit.client.0.smithi136.stdout:1.0 1 0 0 0 0 7 0 0 1 0 1 active+clean 2024-09-15T20:30:39.152573+0000 20'1 25:61 [1,0] 1 [1,0] 1 20'1 2024-09-15T20:30:38.994001+0000 20'1 2024-09-15T20:30:38.994001+0000 0 1 periodic scrub scheduled @ 2024-09-16T20:30:38.994001+0000 1 0 2024-09-15T20:30:45.541 INFO:tasks.workunit.client.0.smithi136.stdout: 2024-09-15T20:30:45.541 INFO:tasks.workunit.client.0.smithi136.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-15T20:30:45.542 INFO:tasks.workunit.client.0.smithi136.stderr:dumped pgs 2024-09-15T20:30:45.553 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:476: TEST_auto_repair_bluestore_basic: get_not_primary testpool SOMETHING 2024-09-15T20:30:45.553 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1220: get_not_primary: local poolname=testpool 2024-09-15T20:30:45.553 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1221: get_not_primary: local objectname=SOMETHING 2024-09-15T20:30:45.553 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1223: get_not_primary: get_primary testpool SOMETHING 2024-09-15T20:30:45.553 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1187: get_primary: local poolname=testpool 2024-09-15T20:30:45.553 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1188: get_primary: local objectname=SOMETHING 2024-09-15T20:30:45.554 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1190: get_primary: ceph --format json osd map testpool SOMETHING 2024-09-15T20:30:45.554 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1191: get_primary: jq .acting_primary 2024-09-15T20:30:45.847 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1223: get_not_primary: local primary=1 2024-09-15T20:30:45.847 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1224: get_not_primary: ceph --format json osd map testpool SOMETHING 2024-09-15T20:30:45.847 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1225: get_not_primary: jq '.acting | map(select (. != 1)) | .[0]' 2024-09-15T20:30:46.126 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:476: TEST_auto_repair_bluestore_basic: objectstore_tool td/osd-scrub-repair 0 SOMETHING list-attrs 2024-09-15T20:30:46.127 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2024-09-15T20:30:46.127 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2024-09-15T20:30:46.127 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=0 2024-09-15T20:30:46.127 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2024-09-15T20:30:46.127 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 SOMETHING list-attrs 2024-09-15T20:30:46.127 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-09-15T20:30:46.127 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2024-09-15T20:30:46.127 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=0 2024-09-15T20:30:46.127 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2024-09-15T20:30:46.127 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2024-09-15T20:30:46.127 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-15T20:30:46.127 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-15T20:30:46.127 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-15T20:30:46.128 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-15T20:30:46.128 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-15T20:30:46.432 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-15T20:30:46.432 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 SOMETHING list-attrs 2024-09-15T20:30:46.432 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-15T20:30:46.432 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-15T20:30:46.433 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=0 2024-09-15T20:30:46.433 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-15T20:30:46.433 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-09-15T20:30:46.433 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 SOMETHING list-attrs 2024-09-15T20:30:46.851 INFO:tasks.workunit.client.0.smithi136.stdout:_ 2024-09-15T20:30:46.851 INFO:tasks.workunit.client.0.smithi136.stdout:snapset 2024-09-15T20:30:47.119 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 2024-09-15T20:30:47.119 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2024-09-15T20:30:47.119 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-15T20:30:47.119 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=0 2024-09-15T20:30:47.119 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-15T20:30:47.119 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-09-15T20:30:47.119 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=ce4892c7-6c6d-4b4f-8e6f-07a3e11133e5 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-15T20:30:47.119 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T20:30:47.119 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T20:30:47.119 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T20:30:47.119 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-09-15T20:30:47.120 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-09-15T20:30:47.120 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-15T20:30:47.120 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-15T20:30:47.120 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-15T20:30:47.120 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-15T20:30:47.120 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-15T20:30:47.120 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-15T20:30:47.120 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-15T20:30:47.120 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T20:30:47.120 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52602 2024-09-15T20:30:47.120 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.52602/$cluster-$name.asok' 2024-09-15T20:30:47.120 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.52602/$cluster-$name.asok' 2024-09-15T20:30:47.120 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-15T20:30:47.121 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-15T20:30:47.121 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-15T20:30:47.121 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T20:30:47.121 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T20:30:47.121 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T20:30:47.121 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T20:30:47.121 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-15T20:30:47.121 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-15T20:30:47.121 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-09-15T20:30:47.121 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.0 2024-09-15T20:30:47.121 INFO:tasks.workunit.client.0.smithi136.stderr:start osd.0 2024-09-15T20:30:47.121 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 0 --fsid=ce4892c7-6c6d-4b4f-8e6f-07a3e11133e5 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.52602/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T20:30:47.122 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-09-15T20:30:47.122 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 0 = 0 ']' 2024-09-15T20:30:47.122 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-15T20:30:47.122 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-15T20:30:47.122 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-15T20:30:47.149 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:30:47.147+0000 7f89c73c5640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T20:30:47.151 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:30:47.149+0000 7f89c73c5640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T20:30:47.152 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:30:47.150+0000 7f89c73c5640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T20:30:47.405 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 0 2024-09-15T20:30:47.405 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-15T20:30:47.405 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2024-09-15T20:30:47.405 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-15T20:30:47.405 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-15T20:30:47.406 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:30:47.406 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-15T20:30:47.406 INFO:tasks.workunit.client.0.smithi136.stderr:0 2024-09-15T20:30:47.406 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:30:47.406 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T20:30:47.714 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:30:48.255 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:30:48.253+0000 7f89c73c5640 -1 Falling back to public interface 2024-09-15T20:30:48.716 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:30:48.716 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:30:48.716 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-15T20:30:48.716 INFO:tasks.workunit.client.0.smithi136.stderr:1 2024-09-15T20:30:48.716 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:30:48.716 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T20:30:49.003 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:30:50.004 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:30:50.004 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:30:50.005 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-15T20:30:50.005 INFO:tasks.workunit.client.0.smithi136.stderr:2 2024-09-15T20:30:50.005 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:30:50.005 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T20:30:50.306 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:30:51.130 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:30:51.128+0000 7f89c73c5640 -1 osd.0 25 log_to_monitors true 2024-09-15T20:30:51.308 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:30:51.308 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:30:51.308 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-15T20:30:51.308 INFO:tasks.workunit.client.0.smithi136.stderr:3 2024-09-15T20:30:51.308 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:30:51.308 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T20:30:51.655 INFO:tasks.workunit.client.0.smithi136.stderr:osd.0 up in weight 1 up_from 29 up_thru 0 down_at 26 last_clean_interval [24,25) [v2:127.0.0.1:6802/2467758974,v1:127.0.0.1:6803/2467758974] [v2:127.0.0.1:6804/2467758974,v1:127.0.0.1:6805/2467758974] exists,up 34eb470d-2432-4fae-bd6a-c5b147873211 2024-09-15T20:30:51.655 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-15T20:30:51.656 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-15T20:30:51.656 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-15T20:30:51.656 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2024-09-15T20:30:51.656 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-15T20:30:51.656 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-15T20:30:51.656 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-15T20:30:51.656 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-15T20:30:51.656 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-15T20:30:51.656 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-15T20:30:51.656 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-15T20:30:51.656 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-15T20:30:51.656 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-15T20:30:51.744 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-15T20:30:51.744 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-15T20:30:51.744 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-15T20:30:51.744 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-15T20:30:51.745 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-15T20:30:51.745 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-15T20:30:52.021 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-15T20:30:52.021 INFO:tasks.workunit.client.0.smithi136.stderr:1 2024-09-15T20:30:52.021 INFO:tasks.workunit.client.0.smithi136.stderr:2' 2024-09-15T20:30:52.021 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-15T20:30:52.021 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T20:30:52.021 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-15T20:30:52.131 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=124554051586 2024-09-15T20:30:52.131 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 124554051586 2024-09-15T20:30:52.131 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-124554051586' 2024-09-15T20:30:52.131 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T20:30:52.131 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-15T20:30:52.277 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=42949672988 2024-09-15T20:30:52.277 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 42949672988 2024-09-15T20:30:52.277 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-124554051586 1-42949672988' 2024-09-15T20:30:52.277 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T20:30:52.277 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-15T20:30:52.380 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=64424509459 2024-09-15T20:30:52.381 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 64424509459 2024-09-15T20:30:52.381 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-124554051586 1-42949672988 2-64424509459' 2024-09-15T20:30:52.381 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T20:30:52.381 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-124554051586 2024-09-15T20:30:52.381 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T20:30:52.382 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-15T20:30:52.382 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-124554051586 2024-09-15T20:30:52.382 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T20:30:52.383 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=124554051586 2024-09-15T20:30:52.384 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 124554051586' 2024-09-15T20:30:52.384 INFO:tasks.workunit.client.0.smithi136.stderr:waiting osd.0 seq 124554051586 2024-09-15T20:30:52.384 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T20:30:52.659 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 1 -lt 124554051586 2024-09-15T20:30:52.659 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-15T20:30:53.660 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-15T20:30:53.660 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T20:30:53.950 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 124554051586 -lt 124554051586 2024-09-15T20:30:53.950 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T20:30:53.950 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-42949672988 2024-09-15T20:30:53.951 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T20:30:53.951 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-15T20:30:53.952 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-42949672988 2024-09-15T20:30:53.952 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T20:30:53.953 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=42949672988 2024-09-15T20:30:53.953 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 42949672988' 2024-09-15T20:30:53.953 INFO:tasks.workunit.client.0.smithi136.stderr:waiting osd.1 seq 42949672988 2024-09-15T20:30:53.953 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-15T20:30:54.229 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 42949672988 -lt 42949672988 2024-09-15T20:30:54.230 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T20:30:54.230 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-64424509459 2024-09-15T20:30:54.230 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T20:30:54.231 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-15T20:30:54.231 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-64424509459 2024-09-15T20:30:54.231 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T20:30:54.233 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=64424509459 2024-09-15T20:30:54.233 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 64424509459' 2024-09-15T20:30:54.233 INFO:tasks.workunit.client.0.smithi136.stderr:waiting osd.2 seq 64424509459 2024-09-15T20:30:54.233 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-15T20:30:54.509 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 64424509459 -lt 64424509459 2024-09-15T20:30:54.509 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-15T20:30:54.510 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T20:30:54.510 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T20:30:54.854 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 1 == 0 2024-09-15T20:30:54.854 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-15T20:30:54.854 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-15T20:30:54.854 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-15T20:30:54.854 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T20:30:54.854 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T20:30:54.854 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T20:30:54.854 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T20:30:55.127 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=0 2024-09-15T20:30:55.127 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-15T20:30:55.127 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T20:30:55.127 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T20:30:55.470 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 0 = 1 2024-09-15T20:30:55.471 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 0 '!=' -1 2024-09-15T20:30:55.471 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: loop=0 2024-09-15T20:30:55.471 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1652: wait_for_clean: num_active_clean=0 2024-09-15T20:30:55.471 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-15T20:30:55.471 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 0.1 2024-09-15T20:30:55.572 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-15T20:30:55.572 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-15T20:30:55.572 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-15T20:30:55.572 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-15T20:30:55.572 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T20:30:55.572 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T20:30:55.572 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T20:30:55.572 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T20:30:55.846 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=1 2024-09-15T20:30:55.846 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-15T20:30:55.846 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T20:30:55.846 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T20:30:56.187 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 = 1 2024-09-15T20:30:56.188 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-15T20:30:56.188 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-15T20:30:56.188 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:477: TEST_auto_repair_bluestore_basic: get_not_primary testpool SOMETHING 2024-09-15T20:30:56.188 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1220: get_not_primary: local poolname=testpool 2024-09-15T20:30:56.188 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1221: get_not_primary: local objectname=SOMETHING 2024-09-15T20:30:56.188 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1223: get_not_primary: get_primary testpool SOMETHING 2024-09-15T20:30:56.188 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1187: get_primary: local poolname=testpool 2024-09-15T20:30:56.188 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1188: get_primary: local objectname=SOMETHING 2024-09-15T20:30:56.188 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1190: get_primary: ceph --format json osd map testpool SOMETHING 2024-09-15T20:30:56.188 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1191: get_primary: jq .acting_primary 2024-09-15T20:30:56.465 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1223: get_not_primary: local primary=1 2024-09-15T20:30:56.465 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1224: get_not_primary: ceph --format json osd map testpool SOMETHING 2024-09-15T20:30:56.465 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1225: get_not_primary: jq '.acting | map(select (. != 1)) | .[0]' 2024-09-15T20:30:56.777 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:477: TEST_auto_repair_bluestore_basic: objectstore_tool td/osd-scrub-repair 0 SOMETHING get-bytes td/osd-scrub-repair/COPY 2024-09-15T20:30:56.777 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2024-09-15T20:30:56.777 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2024-09-15T20:30:56.777 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=0 2024-09-15T20:30:56.777 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2024-09-15T20:30:56.777 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 SOMETHING get-bytes td/osd-scrub-repair/COPY 2024-09-15T20:30:56.777 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-09-15T20:30:56.777 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2024-09-15T20:30:56.777 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=0 2024-09-15T20:30:56.778 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2024-09-15T20:30:56.778 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2024-09-15T20:30:56.778 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-15T20:30:56.778 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-15T20:30:56.778 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-15T20:30:56.778 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-15T20:30:56.778 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-15T20:30:57.083 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-15T20:30:57.083 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 SOMETHING get-bytes td/osd-scrub-repair/COPY 2024-09-15T20:30:57.084 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-15T20:30:57.084 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-15T20:30:57.084 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=0 2024-09-15T20:30:57.084 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-15T20:30:57.084 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-09-15T20:30:57.084 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 SOMETHING get-bytes td/osd-scrub-repair/COPY 2024-09-15T20:30:57.835 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 2024-09-15T20:30:57.835 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2024-09-15T20:30:57.835 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-15T20:30:57.835 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=0 2024-09-15T20:30:57.835 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-15T20:30:57.835 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-09-15T20:30:57.835 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=ce4892c7-6c6d-4b4f-8e6f-07a3e11133e5 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-15T20:30:57.836 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T20:30:57.836 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T20:30:57.836 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T20:30:57.836 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-09-15T20:30:57.836 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-09-15T20:30:57.836 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-15T20:30:57.836 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-15T20:30:57.836 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-15T20:30:57.836 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-15T20:30:57.836 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-15T20:30:57.836 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-15T20:30:57.836 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-15T20:30:57.837 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T20:30:57.837 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52602 2024-09-15T20:30:57.837 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.52602/$cluster-$name.asok' 2024-09-15T20:30:57.837 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.52602/$cluster-$name.asok' 2024-09-15T20:30:57.837 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-15T20:30:57.837 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-15T20:30:57.837 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-15T20:30:57.837 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T20:30:57.837 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T20:30:57.837 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T20:30:57.837 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T20:30:57.837 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-15T20:30:57.837 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-15T20:30:57.837 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-09-15T20:30:57.838 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.0 2024-09-15T20:30:57.838 INFO:tasks.workunit.client.0.smithi136.stderr:start osd.0 2024-09-15T20:30:57.838 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 0 --fsid=ce4892c7-6c6d-4b4f-8e6f-07a3e11133e5 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.52602/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T20:30:57.838 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-09-15T20:30:57.838 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 0 = 0 ']' 2024-09-15T20:30:57.839 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-15T20:30:57.839 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-15T20:30:57.839 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-15T20:30:57.863 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:30:57.860+0000 7f6763b1b640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T20:30:57.866 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:30:57.862+0000 7f6763b1b640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T20:30:57.866 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:30:57.864+0000 7f6763b1b640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T20:30:58.118 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 0 2024-09-15T20:30:58.118 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-15T20:30:58.118 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2024-09-15T20:30:58.119 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-15T20:30:58.119 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-15T20:30:58.119 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:30:58.119 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-15T20:30:58.119 INFO:tasks.workunit.client.0.smithi136.stderr:0 2024-09-15T20:30:58.119 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:30:58.119 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T20:30:58.395 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:30:59.234 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:30:59.231+0000 7f6763b1b640 -1 Falling back to public interface 2024-09-15T20:30:59.397 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:30:59.397 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:30:59.397 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-15T20:30:59.397 INFO:tasks.workunit.client.0.smithi136.stderr:1 2024-09-15T20:30:59.397 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:30:59.397 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T20:30:59.712 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:31:00.714 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:31:00.714 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:31:00.714 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-15T20:31:00.714 INFO:tasks.workunit.client.0.smithi136.stderr:2 2024-09-15T20:31:00.714 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:31:00.714 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T20:31:01.003 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:31:02.004 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:31:02.005 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:31:02.005 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-15T20:31:02.005 INFO:tasks.workunit.client.0.smithi136.stderr:3 2024-09-15T20:31:02.005 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:31:02.005 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T20:31:02.019 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:31:02.017+0000 7f6763b1b640 -1 osd.0 30 log_to_monitors true 2024-09-15T20:31:02.321 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:31:03.323 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:31:03.323 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:31:03.323 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-15T20:31:03.323 INFO:tasks.workunit.client.0.smithi136.stderr:4 2024-09-15T20:31:03.323 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:31:03.323 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T20:31:03.610 INFO:tasks.workunit.client.0.smithi136.stderr:osd.0 up in weight 1 up_from 34 up_thru 0 down_at 31 last_clean_interval [29,30) [v2:127.0.0.1:6802/1855866252,v1:127.0.0.1:6803/1855866252] [v2:127.0.0.1:6804/1855866252,v1:127.0.0.1:6805/1855866252] exists,up 34eb470d-2432-4fae-bd6a-c5b147873211 2024-09-15T20:31:03.610 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-15T20:31:03.610 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-15T20:31:03.610 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-15T20:31:03.610 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2024-09-15T20:31:03.610 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-15T20:31:03.610 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-15T20:31:03.610 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-15T20:31:03.610 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-15T20:31:03.611 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-15T20:31:03.611 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-15T20:31:03.611 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-15T20:31:03.611 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-15T20:31:03.611 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-15T20:31:03.699 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-15T20:31:03.700 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-15T20:31:03.700 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-15T20:31:03.700 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-15T20:31:03.700 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-15T20:31:03.700 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-15T20:31:04.002 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-15T20:31:04.002 INFO:tasks.workunit.client.0.smithi136.stderr:1 2024-09-15T20:31:04.002 INFO:tasks.workunit.client.0.smithi136.stderr:2' 2024-09-15T20:31:04.002 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-15T20:31:04.002 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T20:31:04.002 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-15T20:31:04.104 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=146028888066 2024-09-15T20:31:04.105 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 146028888066 2024-09-15T20:31:04.105 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-146028888066' 2024-09-15T20:31:04.105 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T20:31:04.105 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-15T20:31:04.207 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=42949672991 2024-09-15T20:31:04.207 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 42949672991 2024-09-15T20:31:04.207 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-146028888066 1-42949672991' 2024-09-15T20:31:04.207 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T20:31:04.207 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-15T20:31:04.310 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=64424509462 2024-09-15T20:31:04.310 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 64424509462 2024-09-15T20:31:04.311 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-146028888066 1-42949672991 2-64424509462' 2024-09-15T20:31:04.311 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T20:31:04.311 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-146028888066 2024-09-15T20:31:04.311 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T20:31:04.311 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-15T20:31:04.312 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-146028888066 2024-09-15T20:31:04.312 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T20:31:04.313 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=146028888066 2024-09-15T20:31:04.313 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 146028888066' 2024-09-15T20:31:04.313 INFO:tasks.workunit.client.0.smithi136.stderr:waiting osd.0 seq 146028888066 2024-09-15T20:31:04.313 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T20:31:04.587 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 1 -lt 146028888066 2024-09-15T20:31:04.588 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-15T20:31:05.588 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-15T20:31:05.589 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T20:31:05.875 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 146028888066 -lt 146028888066 2024-09-15T20:31:05.875 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T20:31:05.876 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-42949672991 2024-09-15T20:31:05.876 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T20:31:05.877 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-15T20:31:05.877 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-42949672991 2024-09-15T20:31:05.877 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T20:31:05.878 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=42949672991 2024-09-15T20:31:05.878 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 42949672991' 2024-09-15T20:31:05.878 INFO:tasks.workunit.client.0.smithi136.stderr:waiting osd.1 seq 42949672991 2024-09-15T20:31:05.879 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-15T20:31:06.155 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 42949672992 -lt 42949672991 2024-09-15T20:31:06.155 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T20:31:06.155 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-64424509462 2024-09-15T20:31:06.156 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T20:31:06.156 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-15T20:31:06.157 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-64424509462 2024-09-15T20:31:06.157 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T20:31:06.160 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=64424509462 2024-09-15T20:31:06.161 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 64424509462' 2024-09-15T20:31:06.161 INFO:tasks.workunit.client.0.smithi136.stderr:waiting osd.2 seq 64424509462 2024-09-15T20:31:06.161 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-15T20:31:06.435 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 64424509463 -lt 64424509462 2024-09-15T20:31:06.435 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-15T20:31:06.435 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T20:31:06.436 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T20:31:06.779 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 1 == 0 2024-09-15T20:31:06.779 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-15T20:31:06.780 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-15T20:31:06.780 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-15T20:31:06.780 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T20:31:06.780 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T20:31:06.780 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T20:31:06.780 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T20:31:07.053 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=1 2024-09-15T20:31:07.053 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-15T20:31:07.054 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T20:31:07.054 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T20:31:07.403 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 = 1 2024-09-15T20:31:07.403 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-15T20:31:07.403 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-15T20:31:07.403 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:478: TEST_auto_repair_bluestore_basic: diff td/osd-scrub-repair/ORIGINAL td/osd-scrub-repair/COPY 2024-09-15T20:31:07.404 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:479: TEST_auto_repair_bluestore_basic: grep scrub_finish td/osd-scrub-repair/osd.1.log 2024-09-15T20:31:07.405 INFO:tasks.workunit.client.0.smithi136.stdout:2024-09-15T20:30:38.993+0000 7ff8f0737640 10 osd.1 pg_epoch: 25 pg[1.0( v 20'1 (0'0,20'1] local-lis/les=24/25 n=1 ec=16/16 lis/c=24/24 les/c/f=25/25/0 sis=24) [1,0] r=0 lpr=24 crt=20'1 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep [ 1.0: AUTO_REPAIR ] ] scrubber: scrub_finish before flags: AUTO_REPAIR. repair state: no-repair. deep_scrub_on_error: 0 2024-09-15T20:31:07.406 INFO:tasks.workunit.client.0.smithi136.stdout:2024-09-15T20:30:38.993+0000 7ff8f0737640 10 osd.1 pg_epoch: 25 pg[1.0( v 20'1 (0'0,20'1] local-lis/les=24/25 n=1 ec=16/16 lis/c=24/24 les/c/f=25/25/0 sis=24) [1,0] r=0 lpr=24 crt=20'1 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep [ 1.0: AUTO_REPAIR ] ] scrubber: _scrub_finish info stats: valid m_is_repair: 1 2024-09-15T20:31:07.406 INFO:tasks.workunit.client.0.smithi136.stdout:2024-09-15T20:30:38.993+0000 7ff8f0737640 15 osd.1 pg_epoch: 25 pg[1.0( v 20'1 (0'0,20'1] local-lis/les=24/25 n=1 ec=16/16 lis/c=24/24 les/c/f=25/25/0 sis=24) [1,0] r=0 lpr=24 crt=20'1 lcod 0'0 mlcod 0'0 active+scrubbing+deep+repair [ 1.0: AUTO_REPAIR ] mbc={255={(1+0)=1}}] scrubber: scrub_finish: 1 errors. 1 errors fixed 2024-09-15T20:31:07.406 INFO:tasks.workunit.client.0.smithi136.stdout:2024-09-15T20:30:38.993+0000 7ff8f0737640 20 osd.1 pg_epoch: 25 pg[1.0( v 20'1 (0'0,20'1] local-lis/les=24/25 n=1 ec=16/16 lis/c=24/24 les/c/f=25/25/0 sis=24) [1,0] r=0 lpr=24 crt=20'1 lcod 0'0 mlcod 0'0 active+scrubbing+deep+repair [ 1.0: AUTO_REPAIR ] mbc={255={(1+0)=1}}] scrubber: scrub_finish All may be fixed 2024-09-15T20:31:07.406 INFO:tasks.workunit.client.0.smithi136.stdout:2024-09-15T20:30:38.993+0000 7ff8f0737640 19 osd.1 pg_epoch: 25 pg[1.0( v 20'1 (0'0,20'1] local-lis/les=24/25 n=1 ec=16/16 lis/c=24/24 les/c/f=25/25/0 sis=24) [1,0] r=0 lpr=24 crt=20'1 lcod 0'0 mlcod 0'0 active+scrubbing+deep+repair [ 1.0: AUTO_REPAIR ] mbc={255={(1+0)=1}}] scrubber: scrub_finish shard 1 num_omap_bytes = 0 num_omap_keys = 0 2024-09-15T20:31:07.407 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:66: run: teardown td/osd-scrub-repair 2024-09-15T20:31:07.407 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/osd-scrub-repair 2024-09-15T20:31:07.407 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-15T20:31:07.407 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/osd-scrub-repair KILL 2024-09-15T20:31:07.407 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-15T20:31:07.407 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-15T20:31:07.407 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-15T20:31:07.407 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-15T20:31:07.407 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-15T20:31:07.513 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-15T20:31:07.514 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-15T20:31:07.514 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-15T20:31:07.515 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-15T20:31:07.516 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-15T20:31:07.516 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-15T20:31:07.516 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-15T20:31:07.517 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T20:31:07.517 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-15T20:31:07.517 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-15T20:31:07.518 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T20:31:07.518 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-15T20:31:07.519 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-15T20:31:07.519 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/osd-scrub-repair 2024-09-15T20:31:07.535 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-15T20:31:07.536 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T20:31:07.536 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52602 2024-09-15T20:31:07.536 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.52602 2024-09-15T20:31:07.537 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-15T20:31:07.537 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-15T20:31:07.538 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:63: run: for func in $funcs 2024-09-15T20:31:07.538 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:64: run: setup td/osd-scrub-repair 2024-09-15T20:31:07.538 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/osd-scrub-repair 2024-09-15T20:31:07.538 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/osd-scrub-repair 2024-09-15T20:31:07.538 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/osd-scrub-repair 2024-09-15T20:31:07.538 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-15T20:31:07.538 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/osd-scrub-repair KILL 2024-09-15T20:31:07.538 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-15T20:31:07.538 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-15T20:31:07.539 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-15T20:31:07.539 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-15T20:31:07.539 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-15T20:31:07.539 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-15T20:31:07.539 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-15T20:31:07.540 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-15T20:31:07.540 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-15T20:31:07.541 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-15T20:31:07.541 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-15T20:31:07.541 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-15T20:31:07.542 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T20:31:07.542 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-15T20:31:07.542 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-15T20:31:07.543 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T20:31:07.543 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-15T20:31:07.544 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-15T20:31:07.545 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/osd-scrub-repair 2024-09-15T20:31:07.545 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-15T20:31:07.545 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T20:31:07.546 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52602 2024-09-15T20:31:07.546 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.52602 2024-09-15T20:31:07.546 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-15T20:31:07.546 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-15T20:31:07.546 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/osd-scrub-repair 2024-09-15T20:31:07.548 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2024-09-15T20:31:07.548 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T20:31:07.548 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52602 2024-09-15T20:31:07.548 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.52602 2024-09-15T20:31:07.550 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2024-09-15T20:31:07.550 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 4096 -le 1024 ']' 2024-09-15T20:31:07.550 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2024-09-15T20:31:07.550 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/osd-scrub-repair 1' TERM HUP INT 2024-09-15T20:31:07.551 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:65: run: TEST_auto_repair_bluestore_failed td/osd-scrub-repair 2024-09-15T20:31:07.551 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:539: TEST_auto_repair_bluestore_failed: local dir=td/osd-scrub-repair 2024-09-15T20:31:07.551 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:540: TEST_auto_repair_bluestore_failed: local poolname=testpool 2024-09-15T20:31:07.551 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:543: TEST_auto_repair_bluestore_failed: run_mon td/osd-scrub-repair a 2024-09-15T20:31:07.551 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/osd-scrub-repair 2024-09-15T20:31:07.551 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2024-09-15T20:31:07.551 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2024-09-15T20:31:07.551 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2024-09-15T20:31:07.552 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/osd-scrub-repair/a 2024-09-15T20:31:07.552 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-scrub-repair/a --run-dir=td/osd-scrub-repair 2024-09-15T20:31:08.020 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2024-09-15T20:31:08.020 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-15T20:31:08.020 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-15T20:31:08.020 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-15T20:31:08.020 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T20:31:08.020 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52602 2024-09-15T20:31:08.020 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.52602/$cluster-$name.asok' 2024-09-15T20:31:08.020 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-scrub-repair/a '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.52602/$cluster-$name.asok' --mon-cluster-log-file=td/osd-scrub-repair/log --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-09-15T20:31:08.195 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2024-09-15T20:31:08.196 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2024-09-15T20:31:08.197 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-15T20:31:08.197 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-15T20:31:08.197 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2024-09-15T20:31:08.197 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2024-09-15T20:31:08.197 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-15T20:31:08.197 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-15T20:31:08.197 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-15T20:31:08.197 INFO:tasks.workunit.client.0.smithi136.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-15T20:31:08.197 INFO:tasks.workunit.client.0.smithi136.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T20:31:08.197 INFO:tasks.workunit.client.0.smithi136.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52602 2024-09-15T20:31:08.197 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.52602/ceph-mon.a.asok 2024-09-15T20:31:08.198 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-15T20:31:08.198 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.52602/ceph-mon.a.asok config get fsid 2024-09-15T20:31:08.276 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2024-09-15T20:31:08.276 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-15T20:31:08.276 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-15T20:31:08.276 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2024-09-15T20:31:08.276 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2024-09-15T20:31:08.276 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-15T20:31:08.276 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-15T20:31:08.276 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-15T20:31:08.276 INFO:tasks.workunit.client.0.smithi136.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-15T20:31:08.277 INFO:tasks.workunit.client.0.smithi136.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T20:31:08.277 INFO:tasks.workunit.client.0.smithi136.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52602 2024-09-15T20:31:08.277 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.52602/ceph-mon.a.asok 2024-09-15T20:31:08.277 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-15T20:31:08.277 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.52602/ceph-mon.a.asok config get mon_host 2024-09-15T20:31:08.355 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:544: TEST_auto_repair_bluestore_failed: run_mgr td/osd-scrub-repair x 2024-09-15T20:31:08.355 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: run_mgr: local dir=td/osd-scrub-repair 2024-09-15T20:31:08.355 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: run_mgr: shift 2024-09-15T20:31:08.355 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: run_mgr: local id=x 2024-09-15T20:31:08.355 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: run_mgr: shift 2024-09-15T20:31:08.355 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:548: run_mgr: local data=td/osd-scrub-repair/x 2024-09-15T20:31:08.355 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:550: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-15T20:31:09.191 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: get_asok_path 2024-09-15T20:31:09.191 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-15T20:31:09.191 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-15T20:31:09.192 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-15T20:31:09.192 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T20:31:09.192 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52602 2024-09-15T20:31:09.192 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.52602/$cluster-$name.asok' 2024-09-15T20:31:09.192 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-15T20:31:09.193 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-scrub-repair/x '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.52602/$cluster-$name.asok' --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-15T20:31:09.220 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:547: TEST_auto_repair_bluestore_failed: local 'ceph_osd_args=--osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0' 2024-09-15T20:31:09.221 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:548: TEST_auto_repair_bluestore_failed: seq 0 2 2024-09-15T20:31:09.226 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:548: TEST_auto_repair_bluestore_failed: for id in $(seq 0 2) 2024-09-15T20:31:09.226 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:549: TEST_auto_repair_bluestore_failed: run_osd td/osd-scrub-repair 0 --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-09-15T20:31:09.226 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-scrub-repair 2024-09-15T20:31:09.226 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-15T20:31:09.226 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=0 2024-09-15T20:31:09.226 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-15T20:31:09.227 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-scrub-repair/0 2024-09-15T20:31:09.227 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=ce4892c7-6c6d-4b4f-8e6f-07a3e11133e5 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-15T20:31:09.227 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T20:31:09.227 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T20:31:09.227 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T20:31:09.227 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-09-15T20:31:09.227 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-09-15T20:31:09.227 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-15T20:31:09.227 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-15T20:31:09.227 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-15T20:31:09.227 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-15T20:31:09.227 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-15T20:31:09.227 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-15T20:31:09.228 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-15T20:31:09.228 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T20:31:09.228 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52602 2024-09-15T20:31:09.228 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.52602/$cluster-$name.asok' 2024-09-15T20:31:09.228 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.52602/$cluster-$name.asok' 2024-09-15T20:31:09.228 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-15T20:31:09.228 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-15T20:31:09.228 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-15T20:31:09.228 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-15T20:31:09.228 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-15T20:31:09.228 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T20:31:09.228 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T20:31:09.229 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T20:31:09.229 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T20:31:09.229 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-15T20:31:09.229 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+='--osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0' 2024-09-15T20:31:09.229 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-scrub-repair/0 2024-09-15T20:31:09.229 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-15T20:31:09.229 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=555689c8-c81e-4042-90bc-288636f201a7 2024-09-15T20:31:09.229 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd0 555689c8-c81e-4042-90bc-288636f201a7' 2024-09-15T20:31:09.229 INFO:tasks.workunit.client.0.smithi136.stdout:add osd0 555689c8-c81e-4042-90bc-288636f201a7 2024-09-15T20:31:09.230 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-15T20:31:09.244 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQANROdmeuh0DhAAmtO0doi/JktXWvc8NZpQ4g== 2024-09-15T20:31:09.245 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQANROdmeuh0DhAAmtO0doi/JktXWvc8NZpQ4g=="}' 2024-09-15T20:31:09.245 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 555689c8-c81e-4042-90bc-288636f201a7 -i td/osd-scrub-repair/0/new.json 2024-09-15T20:31:09.446 INFO:tasks.workunit.client.0.smithi136.stdout:0 2024-09-15T20:31:09.456 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-scrub-repair/0/new.json 2024-09-15T20:31:09.457 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 0 --fsid=ce4892c7-6c6d-4b4f-8e6f-07a3e11133e5 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.52602/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --mkfs --key AQANROdmeuh0DhAAmtO0doi/JktXWvc8NZpQ4g== --osd-uuid 555689c8-c81e-4042-90bc-288636f201a7 2024-09-15T20:31:09.488 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:31:09.486+0000 7f7d151c3640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T20:31:09.497 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:31:09.495+0000 7f7d151c3640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T20:31:09.498 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:31:09.496+0000 7f7d151c3640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T20:31:09.498 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:31:09.497+0000 7f7d151c3640 -1 bdev(0x5555e0385c00 td/osd-scrub-repair/0/block) open stat got: (1) Operation not permitted 2024-09-15T20:31:09.499 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:31:09.497+0000 7f7d151c3640 -1 bluestore(td/osd-scrub-repair/0) _read_fsid unparsable uuid 2024-09-15T20:31:15.210 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-scrub-repair/0/keyring 2024-09-15T20:31:15.210 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-15T20:31:15.211 INFO:tasks.workunit.client.0.smithi136.stdout:adding osd0 key to auth repository 2024-09-15T20:31:15.212 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd0 key to auth repository 2024-09-15T20:31:15.212 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-scrub-repair/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-15T20:31:15.579 INFO:tasks.workunit.client.0.smithi136.stdout:start osd.0 2024-09-15T20:31:15.579 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.0 2024-09-15T20:31:15.580 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 0 --fsid=ce4892c7-6c6d-4b4f-8e6f-07a3e11133e5 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.52602/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-09-15T20:31:15.580 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-15T20:31:15.580 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-15T20:31:15.580 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-15T20:31:15.608 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:31:15.606+0000 7f4f32989640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T20:31:15.610 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:31:15.608+0000 7f4f32989640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T20:31:15.611 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:31:15.609+0000 7f4f32989640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T20:31:15.859 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 0 2024-09-15T20:31:15.859 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-15T20:31:15.859 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2024-09-15T20:31:15.859 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-15T20:31:15.859 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-15T20:31:15.859 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:31:15.859 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-15T20:31:15.859 INFO:tasks.workunit.client.0.smithi136.stdout:0 2024-09-15T20:31:15.860 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:31:15.860 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T20:31:16.133 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:31:16.725 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:31:16.724+0000 7f4f32989640 -1 Falling back to public interface 2024-09-15T20:31:17.135 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:31:17.135 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:31:17.135 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-15T20:31:17.135 INFO:tasks.workunit.client.0.smithi136.stdout:1 2024-09-15T20:31:17.136 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:31:17.136 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T20:31:17.411 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:31:18.413 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:31:18.413 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:31:18.413 INFO:tasks.workunit.client.0.smithi136.stdout:2 2024-09-15T20:31:18.414 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-15T20:31:18.414 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:31:18.414 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T20:31:18.691 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:31:19.209 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:31:19.207+0000 7f4f32989640 -1 osd.0 0 log_to_monitors true 2024-09-15T20:31:19.693 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:31:19.693 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:31:19.693 INFO:tasks.workunit.client.0.smithi136.stdout:3 2024-09-15T20:31:19.694 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-15T20:31:19.694 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:31:19.694 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T20:31:20.010 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:31:21.011 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:31:21.011 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:31:21.012 INFO:tasks.workunit.client.0.smithi136.stdout:4 2024-09-15T20:31:21.012 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-15T20:31:21.012 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:31:21.012 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T20:31:21.301 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:31:22.302 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:31:22.303 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:31:22.303 INFO:tasks.workunit.client.0.smithi136.stdout:5 2024-09-15T20:31:22.303 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-15T20:31:22.303 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:31:22.303 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T20:31:22.588 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:31:23.590 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:31:23.590 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:31:23.590 INFO:tasks.workunit.client.0.smithi136.stdout:6 2024-09-15T20:31:23.591 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-15T20:31:23.591 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:31:23.591 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T20:31:23.951 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:31:24.953 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:31:24.953 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:31:24.953 INFO:tasks.workunit.client.0.smithi136.stdout:7 2024-09-15T20:31:24.954 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-15T20:31:24.954 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:31:24.955 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T20:31:25.239 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:31:26.240 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:31:26.240 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:31:26.240 INFO:tasks.workunit.client.0.smithi136.stdout:8 2024-09-15T20:31:26.241 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-15T20:31:26.241 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:31:26.241 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T20:31:26.525 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:31:27.527 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:31:27.527 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:31:27.528 INFO:tasks.workunit.client.0.smithi136.stdout:9 2024-09-15T20:31:27.528 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-15T20:31:27.528 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:31:27.528 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T20:31:27.820 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:31:28.821 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:31:28.821 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:31:28.821 INFO:tasks.workunit.client.0.smithi136.stdout:10 2024-09-15T20:31:28.822 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-15T20:31:28.822 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:31:28.822 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T20:31:29.166 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:31:30.167 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:31:30.168 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:31:30.168 INFO:tasks.workunit.client.0.smithi136.stdout:11 2024-09-15T20:31:30.168 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-15T20:31:30.168 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:31:30.168 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T20:31:30.451 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:31:31.453 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:31:31.453 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:31:31.453 INFO:tasks.workunit.client.0.smithi136.stdout:12 2024-09-15T20:31:31.454 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-15T20:31:31.454 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:31:31.454 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T20:31:31.882 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:31:32.883 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:31:32.884 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:31:32.884 INFO:tasks.workunit.client.0.smithi136.stdout:13 2024-09-15T20:31:32.884 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-15T20:31:32.884 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:31:32.885 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T20:31:33.694 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:31:34.696 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:31:34.696 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:31:34.696 INFO:tasks.workunit.client.0.smithi136.stdout:14 2024-09-15T20:31:34.697 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-15T20:31:34.697 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:31:34.697 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T20:31:35.154 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:31:36.156 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:31:36.156 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:31:36.156 INFO:tasks.workunit.client.0.smithi136.stdout:15 2024-09-15T20:31:36.157 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-15T20:31:36.157 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:31:36.157 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T20:31:36.683 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:31:37.685 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:31:37.685 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:31:37.685 INFO:tasks.workunit.client.0.smithi136.stdout:16 2024-09-15T20:31:37.686 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-15T20:31:37.686 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:31:37.686 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T20:31:37.973 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:31:38.974 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:31:38.974 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:31:38.975 INFO:tasks.workunit.client.0.smithi136.stdout:17 2024-09-15T20:31:38.975 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-15T20:31:38.975 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:31:38.975 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T20:31:39.406 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:31:40.407 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:31:40.407 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:31:40.408 INFO:tasks.workunit.client.0.smithi136.stdout:18 2024-09-15T20:31:40.408 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 18 2024-09-15T20:31:40.408 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:31:40.408 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T20:31:41.180 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:31:42.181 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:31:42.182 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:31:42.182 INFO:tasks.workunit.client.0.smithi136.stdout:19 2024-09-15T20:31:42.182 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 19 2024-09-15T20:31:42.183 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:31:42.183 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T20:31:42.499 INFO:tasks.workunit.client.0.smithi136.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/4272646923,v1:127.0.0.1:6803/4272646923] [v2:127.0.0.1:6804/4272646923,v1:127.0.0.1:6805/4272646923] exists,up 555689c8-c81e-4042-90bc-288636f201a7 2024-09-15T20:31:42.503 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-15T20:31:42.503 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-15T20:31:42.503 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-15T20:31:42.503 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:548: TEST_auto_repair_bluestore_failed: for id in $(seq 0 2) 2024-09-15T20:31:42.504 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:549: TEST_auto_repair_bluestore_failed: run_osd td/osd-scrub-repair 1 --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-09-15T20:31:42.504 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-scrub-repair 2024-09-15T20:31:42.504 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-15T20:31:42.504 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=1 2024-09-15T20:31:42.504 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-15T20:31:42.504 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-scrub-repair/1 2024-09-15T20:31:42.504 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=ce4892c7-6c6d-4b4f-8e6f-07a3e11133e5 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-15T20:31:42.504 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T20:31:42.504 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T20:31:42.504 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T20:31:42.504 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-09-15T20:31:42.504 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-09-15T20:31:42.504 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-15T20:31:42.504 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-15T20:31:42.505 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-15T20:31:42.505 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-15T20:31:42.505 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-15T20:31:42.505 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-15T20:31:42.505 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-15T20:31:42.505 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T20:31:42.505 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52602 2024-09-15T20:31:42.505 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.52602/$cluster-$name.asok' 2024-09-15T20:31:42.505 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.52602/$cluster-$name.asok' 2024-09-15T20:31:42.505 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-15T20:31:42.505 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-15T20:31:42.505 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-15T20:31:42.505 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-15T20:31:42.505 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-15T20:31:42.506 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T20:31:42.506 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T20:31:42.506 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T20:31:42.506 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T20:31:42.506 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-15T20:31:42.506 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+='--osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0' 2024-09-15T20:31:42.506 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-scrub-repair/1 2024-09-15T20:31:42.506 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-15T20:31:42.506 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=a91f9a5c-e300-4dd1-9666-c0e98286384c 2024-09-15T20:31:42.506 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd1 a91f9a5c-e300-4dd1-9666-c0e98286384c' 2024-09-15T20:31:42.506 INFO:tasks.workunit.client.0.smithi136.stdout:add osd1 a91f9a5c-e300-4dd1-9666-c0e98286384c 2024-09-15T20:31:42.507 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-15T20:31:42.522 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQAuROdmslH/HhAAFIjg/wD7k/GjD9NmVgaxDQ== 2024-09-15T20:31:42.522 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQAuROdmslH/HhAAFIjg/wD7k/GjD9NmVgaxDQ=="}' 2024-09-15T20:31:42.522 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new a91f9a5c-e300-4dd1-9666-c0e98286384c -i td/osd-scrub-repair/1/new.json 2024-09-15T20:31:42.873 INFO:tasks.workunit.client.0.smithi136.stdout:1 2024-09-15T20:31:42.884 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-scrub-repair/1/new.json 2024-09-15T20:31:42.885 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 1 --fsid=ce4892c7-6c6d-4b4f-8e6f-07a3e11133e5 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.52602/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --mkfs --key AQAuROdmslH/HhAAFIjg/wD7k/GjD9NmVgaxDQ== --osd-uuid a91f9a5c-e300-4dd1-9666-c0e98286384c 2024-09-15T20:31:42.916 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:31:42.913+0000 7f470c61f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T20:31:42.968 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:31:42.965+0000 7f470c61f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T20:31:42.969 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:31:42.966+0000 7f470c61f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T20:31:42.969 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:31:42.967+0000 7f470c61f640 -1 bdev(0x55df311c5c00 td/osd-scrub-repair/1/block) open stat got: (1) Operation not permitted 2024-09-15T20:31:42.969 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:31:42.967+0000 7f470c61f640 -1 bluestore(td/osd-scrub-repair/1) _read_fsid unparsable uuid 2024-09-15T20:31:49.136 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-scrub-repair/1/keyring 2024-09-15T20:31:49.136 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-15T20:31:49.137 INFO:tasks.workunit.client.0.smithi136.stdout:adding osd1 key to auth repository 2024-09-15T20:31:49.137 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd1 key to auth repository 2024-09-15T20:31:49.138 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-scrub-repair/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-15T20:31:49.501 INFO:tasks.workunit.client.0.smithi136.stdout:start osd.1 2024-09-15T20:31:49.501 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.1 2024-09-15T20:31:49.501 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 1 --fsid=ce4892c7-6c6d-4b4f-8e6f-07a3e11133e5 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.52602/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-09-15T20:31:49.502 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-15T20:31:49.502 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-15T20:31:49.502 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-15T20:31:49.530 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:31:49.528+0000 7f8c35616640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T20:31:49.532 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:31:49.530+0000 7f8c35616640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T20:31:49.533 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:31:49.531+0000 7f8c35616640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T20:31:49.782 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 1 2024-09-15T20:31:49.783 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-15T20:31:49.783 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2024-09-15T20:31:49.783 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-15T20:31:49.783 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-15T20:31:49.783 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:31:49.783 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-15T20:31:49.783 INFO:tasks.workunit.client.0.smithi136.stdout:0 2024-09-15T20:31:49.784 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:31:49.784 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T20:31:50.058 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:31:51.059 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:31:51.060 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:31:51.060 INFO:tasks.workunit.client.0.smithi136.stdout:1 2024-09-15T20:31:51.060 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-15T20:31:51.061 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:31:51.061 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T20:31:51.179 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:31:51.177+0000 7f8c35616640 -1 Falling back to public interface 2024-09-15T20:31:51.383 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:31:52.384 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:31:52.384 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:31:52.385 INFO:tasks.workunit.client.0.smithi136.stdout:2 2024-09-15T20:31:52.385 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-15T20:31:52.385 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:31:52.385 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T20:31:52.665 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:31:53.667 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:31:53.667 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:31:53.667 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-15T20:31:53.667 INFO:tasks.workunit.client.0.smithi136.stdout:3 2024-09-15T20:31:53.668 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:31:53.668 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T20:31:53.946 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:31:54.162 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:31:54.160+0000 7f8c35616640 -1 osd.1 0 log_to_monitors true 2024-09-15T20:31:54.947 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:31:54.947 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:31:54.948 INFO:tasks.workunit.client.0.smithi136.stdout:4 2024-09-15T20:31:54.948 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-15T20:31:54.948 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:31:54.948 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T20:31:55.345 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:31:56.346 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:31:56.346 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:31:56.346 INFO:tasks.workunit.client.0.smithi136.stdout:5 2024-09-15T20:31:56.347 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-15T20:31:56.347 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:31:56.347 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T20:31:56.631 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:31:57.633 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:31:57.633 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:31:57.633 INFO:tasks.workunit.client.0.smithi136.stdout:6 2024-09-15T20:31:57.634 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-15T20:31:57.634 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:31:57.634 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T20:31:57.976 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:31:58.978 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:31:58.978 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:31:58.978 INFO:tasks.workunit.client.0.smithi136.stdout:7 2024-09-15T20:31:58.979 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-15T20:31:58.979 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:31:58.979 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T20:31:59.284 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:32:00.284 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:32:00.284 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:32:00.285 INFO:tasks.workunit.client.0.smithi136.stdout:8 2024-09-15T20:32:00.285 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-15T20:32:00.285 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:32:00.285 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T20:32:00.574 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:32:01.575 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:32:01.575 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:32:01.576 INFO:tasks.workunit.client.0.smithi136.stdout:9 2024-09-15T20:32:01.576 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-15T20:32:01.576 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:32:01.576 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T20:32:01.864 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:32:02.865 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:32:02.865 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:32:02.866 INFO:tasks.workunit.client.0.smithi136.stdout:10 2024-09-15T20:32:02.866 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-15T20:32:02.866 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:32:02.866 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T20:32:03.188 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:32:04.190 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:32:04.192 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:32:04.192 INFO:tasks.workunit.client.0.smithi136.stdout:11 2024-09-15T20:32:04.193 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-15T20:32:04.193 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:32:04.193 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T20:32:04.477 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:32:05.478 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:32:05.478 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:32:05.478 INFO:tasks.workunit.client.0.smithi136.stdout:12 2024-09-15T20:32:05.479 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-15T20:32:05.479 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:32:05.479 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T20:32:05.989 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:32:06.990 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:32:06.990 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:32:06.990 INFO:tasks.workunit.client.0.smithi136.stdout:13 2024-09-15T20:32:06.991 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-15T20:32:06.991 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:32:06.991 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T20:32:07.425 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:32:08.426 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:32:08.426 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:32:08.427 INFO:tasks.workunit.client.0.smithi136.stdout:14 2024-09-15T20:32:08.427 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-15T20:32:08.427 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:32:08.427 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T20:32:08.987 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:32:09.989 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:32:09.989 INFO:tasks.workunit.client.0.smithi136.stdout:15 2024-09-15T20:32:09.989 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:32:09.989 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-15T20:32:09.989 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:32:09.990 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T20:32:10.670 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:32:11.672 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:32:11.672 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:32:11.672 INFO:tasks.workunit.client.0.smithi136.stdout:16 2024-09-15T20:32:11.673 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-15T20:32:11.673 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:32:11.673 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T20:32:12.022 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:32:13.023 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:32:13.023 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:32:13.024 INFO:tasks.workunit.client.0.smithi136.stdout:17 2024-09-15T20:32:13.024 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-15T20:32:13.024 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:32:13.024 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T20:32:13.432 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:32:14.434 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:32:14.434 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:32:14.434 INFO:tasks.workunit.client.0.smithi136.stdout:18 2024-09-15T20:32:14.435 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 18 2024-09-15T20:32:14.435 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:32:14.435 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T20:32:14.978 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:32:15.980 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:32:15.980 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:32:15.980 INFO:tasks.workunit.client.0.smithi136.stdout:19 2024-09-15T20:32:15.981 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 19 2024-09-15T20:32:15.981 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:32:15.981 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T20:32:16.445 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:32:17.447 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:32:17.447 INFO:tasks.workunit.client.0.smithi136.stdout:20 2024-09-15T20:32:17.448 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:32:17.449 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 20 2024-09-15T20:32:17.449 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:32:17.449 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T20:32:17.759 INFO:tasks.workunit.client.0.smithi136.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/2904457361,v1:127.0.0.1:6811/2904457361] [v2:127.0.0.1:6812/2904457361,v1:127.0.0.1:6813/2904457361] exists,up a91f9a5c-e300-4dd1-9666-c0e98286384c 2024-09-15T20:32:17.763 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-15T20:32:17.763 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-15T20:32:17.763 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-15T20:32:17.763 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:548: TEST_auto_repair_bluestore_failed: for id in $(seq 0 2) 2024-09-15T20:32:17.763 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:549: TEST_auto_repair_bluestore_failed: run_osd td/osd-scrub-repair 2 --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-09-15T20:32:17.764 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-scrub-repair 2024-09-15T20:32:17.764 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-15T20:32:17.764 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=2 2024-09-15T20:32:17.764 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-15T20:32:17.764 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-scrub-repair/2 2024-09-15T20:32:17.764 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=ce4892c7-6c6d-4b4f-8e6f-07a3e11133e5 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-15T20:32:17.764 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T20:32:17.764 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T20:32:17.764 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T20:32:17.764 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/2' 2024-09-15T20:32:17.764 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/2/journal' 2024-09-15T20:32:17.764 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-15T20:32:17.764 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-15T20:32:17.765 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-15T20:32:17.765 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-15T20:32:17.765 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-15T20:32:17.765 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-15T20:32:17.765 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-15T20:32:17.765 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T20:32:17.765 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52602 2024-09-15T20:32:17.765 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.52602/$cluster-$name.asok' 2024-09-15T20:32:17.765 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.52602/$cluster-$name.asok' 2024-09-15T20:32:17.765 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-15T20:32:17.765 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-15T20:32:17.765 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-15T20:32:17.765 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-15T20:32:17.766 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-15T20:32:17.766 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T20:32:17.766 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T20:32:17.766 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T20:32:17.766 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T20:32:17.766 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-15T20:32:17.766 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+='--osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0' 2024-09-15T20:32:17.766 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-scrub-repair/2 2024-09-15T20:32:17.766 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-15T20:32:17.766 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=fe16efee-434f-4883-a7dc-86cab48dad8a 2024-09-15T20:32:17.766 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd2 fe16efee-434f-4883-a7dc-86cab48dad8a' 2024-09-15T20:32:17.766 INFO:tasks.workunit.client.0.smithi136.stdout:add osd2 fe16efee-434f-4883-a7dc-86cab48dad8a 2024-09-15T20:32:17.767 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-15T20:32:17.783 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQBRROdmNFKPLhAAsNCwgX4+hSFjG0mVhUrzYw== 2024-09-15T20:32:17.783 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQBRROdmNFKPLhAAsNCwgX4+hSFjG0mVhUrzYw=="}' 2024-09-15T20:32:17.783 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new fe16efee-434f-4883-a7dc-86cab48dad8a -i td/osd-scrub-repair/2/new.json 2024-09-15T20:32:18.132 INFO:tasks.workunit.client.0.smithi136.stdout:2 2024-09-15T20:32:18.143 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-scrub-repair/2/new.json 2024-09-15T20:32:18.144 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 2 --fsid=ce4892c7-6c6d-4b4f-8e6f-07a3e11133e5 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.52602/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --mkfs --key AQBRROdmNFKPLhAAsNCwgX4+hSFjG0mVhUrzYw== --osd-uuid fe16efee-434f-4883-a7dc-86cab48dad8a 2024-09-15T20:32:18.176 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:32:18.174+0000 7fb40af4d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T20:32:18.217 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:32:18.215+0000 7fb40af4d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T20:32:18.218 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:32:18.216+0000 7fb40af4d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T20:32:18.218 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:32:18.216+0000 7fb40af4d640 -1 bdev(0x562e9e417c00 td/osd-scrub-repair/2/block) open stat got: (1) Operation not permitted 2024-09-15T20:32:18.218 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:32:18.216+0000 7fb40af4d640 -1 bluestore(td/osd-scrub-repair/2) _read_fsid unparsable uuid 2024-09-15T20:32:24.296 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-scrub-repair/2/keyring 2024-09-15T20:32:24.296 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-15T20:32:24.297 INFO:tasks.workunit.client.0.smithi136.stdout:adding osd2 key to auth repository 2024-09-15T20:32:24.297 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd2 key to auth repository 2024-09-15T20:32:24.297 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-scrub-repair/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-15T20:32:24.656 INFO:tasks.workunit.client.0.smithi136.stdout:start osd.2 2024-09-15T20:32:24.657 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.2 2024-09-15T20:32:24.657 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 2 --fsid=ce4892c7-6c6d-4b4f-8e6f-07a3e11133e5 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.52602/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-09-15T20:32:24.657 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-15T20:32:24.658 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-15T20:32:24.658 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-15T20:32:24.685 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:32:24.683+0000 7f7219010640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T20:32:24.687 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:32:24.685+0000 7f7219010640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T20:32:24.688 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:32:24.687+0000 7f7219010640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T20:32:24.933 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 2 2024-09-15T20:32:24.933 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-15T20:32:24.933 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=2 2024-09-15T20:32:24.933 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-15T20:32:24.933 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-15T20:32:24.933 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:32:24.933 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-15T20:32:24.934 INFO:tasks.workunit.client.0.smithi136.stdout:0 2024-09-15T20:32:24.934 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:32:24.934 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T20:32:25.224 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:32:26.225 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:32:26.225 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:32:26.225 INFO:tasks.workunit.client.0.smithi136.stdout:1 2024-09-15T20:32:26.226 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-15T20:32:26.226 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:32:26.226 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T20:32:26.513 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:32:26.545 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:32:26.542+0000 7f7219010640 -1 Falling back to public interface 2024-09-15T20:32:27.515 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:32:27.515 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:32:27.515 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-15T20:32:27.515 INFO:tasks.workunit.client.0.smithi136.stdout:2 2024-09-15T20:32:27.516 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:32:27.516 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T20:32:27.801 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:32:28.803 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:32:28.803 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:32:28.804 INFO:tasks.workunit.client.0.smithi136.stdout:3 2024-09-15T20:32:28.804 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-15T20:32:28.804 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:32:28.804 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T20:32:29.091 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:32:29.664 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:32:29.661+0000 7f7219010640 -1 osd.2 0 log_to_monitors true 2024-09-15T20:32:30.093 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:32:30.093 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:32:30.093 INFO:tasks.workunit.client.0.smithi136.stdout:4 2024-09-15T20:32:30.093 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-15T20:32:30.094 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:32:30.094 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T20:32:30.379 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:32:31.381 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:32:31.381 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:32:31.381 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-15T20:32:31.382 INFO:tasks.workunit.client.0.smithi136.stdout:5 2024-09-15T20:32:31.382 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:32:31.382 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T20:32:31.669 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:32:32.671 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:32:32.671 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:32:32.671 INFO:tasks.workunit.client.0.smithi136.stdout:6 2024-09-15T20:32:32.671 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-15T20:32:32.672 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:32:32.672 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T20:32:32.958 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:32:33.960 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:32:33.960 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:32:33.960 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-15T20:32:33.960 INFO:tasks.workunit.client.0.smithi136.stdout:7 2024-09-15T20:32:33.961 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:32:33.961 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T20:32:34.324 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:32:35.325 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:32:35.325 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:32:35.326 INFO:tasks.workunit.client.0.smithi136.stdout:8 2024-09-15T20:32:35.326 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-15T20:32:35.326 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:32:35.326 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T20:32:35.613 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:32:36.614 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:32:36.614 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:32:36.615 INFO:tasks.workunit.client.0.smithi136.stdout:9 2024-09-15T20:32:36.615 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-15T20:32:36.615 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:32:36.615 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T20:32:36.903 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:32:37.904 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:32:37.904 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:32:37.905 INFO:tasks.workunit.client.0.smithi136.stdout:10 2024-09-15T20:32:37.905 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-15T20:32:37.905 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:32:37.905 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T20:32:38.185 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:32:39.187 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:32:39.187 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:32:39.187 INFO:tasks.workunit.client.0.smithi136.stdout:11 2024-09-15T20:32:39.188 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-15T20:32:39.188 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:32:39.188 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T20:32:39.471 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:32:40.472 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:32:40.473 INFO:tasks.workunit.client.0.smithi136.stdout:12 2024-09-15T20:32:40.473 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:32:40.473 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-15T20:32:40.473 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:32:40.473 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T20:32:40.757 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:32:41.759 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:32:41.759 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:32:41.759 INFO:tasks.workunit.client.0.smithi136.stdout:13 2024-09-15T20:32:41.759 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-15T20:32:41.760 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:32:41.760 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T20:32:42.463 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:32:43.464 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:32:43.465 INFO:tasks.workunit.client.0.smithi136.stdout:14 2024-09-15T20:32:43.465 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:32:43.465 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-15T20:32:43.465 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:32:43.465 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T20:32:43.897 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:32:44.898 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:32:44.898 INFO:tasks.workunit.client.0.smithi136.stdout:15 2024-09-15T20:32:44.899 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:32:44.899 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-15T20:32:44.899 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:32:44.899 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T20:32:45.364 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:32:46.365 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:32:46.365 INFO:tasks.workunit.client.0.smithi136.stdout:16 2024-09-15T20:32:46.366 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:32:46.366 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-15T20:32:46.366 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:32:46.366 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T20:32:46.879 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:32:47.880 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:32:47.880 INFO:tasks.workunit.client.0.smithi136.stdout:17 2024-09-15T20:32:47.881 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:32:47.881 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-15T20:32:47.881 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:32:47.881 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T20:32:48.594 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:32:49.595 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:32:49.596 INFO:tasks.workunit.client.0.smithi136.stdout:18 2024-09-15T20:32:49.596 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:32:49.596 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 18 2024-09-15T20:32:49.597 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:32:49.597 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T20:32:49.888 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:32:50.889 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:32:50.890 INFO:tasks.workunit.client.0.smithi136.stdout:19 2024-09-15T20:32:50.890 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:32:50.890 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 19 2024-09-15T20:32:50.890 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:32:50.890 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T20:32:51.179 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:32:52.181 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:32:52.181 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:32:52.181 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 20 2024-09-15T20:32:52.181 INFO:tasks.workunit.client.0.smithi136.stdout:20 2024-09-15T20:32:52.182 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:32:52.182 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T20:32:52.470 INFO:tasks.workunit.client.0.smithi136.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/2507433563,v1:127.0.0.1:6819/2507433563] [v2:127.0.0.1:6820/2507433563,v1:127.0.0.1:6821/2507433563] exists,up fe16efee-434f-4883-a7dc-86cab48dad8a 2024-09-15T20:32:52.471 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-15T20:32:52.471 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-15T20:32:52.471 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-15T20:32:52.471 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:552: TEST_auto_repair_bluestore_failed: create_pool testpool 1 1 2024-09-15T20:32:52.471 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create testpool 1 1 2024-09-15T20:32:52.995 INFO:tasks.workunit.client.0.smithi136.stderr:pool 'testpool' created 2024-09-15T20:32:53.007 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2024-09-15T20:32:54.008 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:553: TEST_auto_repair_bluestore_failed: ceph osd pool set testpool size 2 2024-09-15T20:32:54.384 INFO:tasks.workunit.client.0.smithi136.stderr:set pool 1 size to 2 2024-09-15T20:32:54.396 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:554: TEST_auto_repair_bluestore_failed: wait_for_clean 2024-09-15T20:32:54.396 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-15T20:32:54.396 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-15T20:32:54.396 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-15T20:32:54.396 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-15T20:32:54.397 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-15T20:32:54.397 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-15T20:32:54.397 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-15T20:32:54.397 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-15T20:32:54.397 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-15T20:32:54.486 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-15T20:32:54.486 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-15T20:32:54.486 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-15T20:32:54.486 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-15T20:32:54.486 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-15T20:32:54.487 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-15T20:32:54.773 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-15T20:32:54.773 INFO:tasks.workunit.client.0.smithi136.stderr:1 2024-09-15T20:32:54.773 INFO:tasks.workunit.client.0.smithi136.stderr:2' 2024-09-15T20:32:54.773 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-15T20:32:54.773 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T20:32:54.774 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-15T20:32:54.913 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836499 2024-09-15T20:32:54.914 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836499 2024-09-15T20:32:54.914 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836499' 2024-09-15T20:32:54.914 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T20:32:54.914 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-15T20:32:55.015 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=42949672970 2024-09-15T20:32:55.015 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 42949672970 2024-09-15T20:32:55.015 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836499 1-42949672970' 2024-09-15T20:32:55.016 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T20:32:55.016 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-15T20:32:55.120 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=64424509445 2024-09-15T20:32:55.120 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 64424509445 2024-09-15T20:32:55.120 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836499 1-42949672970 2-64424509445' 2024-09-15T20:32:55.120 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T20:32:55.120 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836499 2024-09-15T20:32:55.121 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T20:32:55.121 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-15T20:32:55.122 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836499 2024-09-15T20:32:55.122 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T20:32:55.123 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836499 2024-09-15T20:32:55.123 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836499' 2024-09-15T20:32:55.123 INFO:tasks.workunit.client.0.smithi136.stdout:waiting osd.0 seq 21474836499 2024-09-15T20:32:55.123 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T20:32:55.421 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836499 -lt 21474836499 2024-09-15T20:32:55.421 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T20:32:55.421 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-42949672970 2024-09-15T20:32:55.421 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T20:32:55.421 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-15T20:32:55.422 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-42949672970 2024-09-15T20:32:55.422 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T20:32:55.423 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=42949672970 2024-09-15T20:32:55.423 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 42949672970' 2024-09-15T20:32:55.423 INFO:tasks.workunit.client.0.smithi136.stdout:waiting osd.1 seq 42949672970 2024-09-15T20:32:55.423 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-15T20:32:55.731 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 42949672970 -lt 42949672970 2024-09-15T20:32:55.731 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T20:32:55.731 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-64424509445 2024-09-15T20:32:55.731 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T20:32:55.732 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-15T20:32:55.733 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-64424509445 2024-09-15T20:32:55.733 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T20:32:55.734 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=64424509445 2024-09-15T20:32:55.734 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 64424509445' 2024-09-15T20:32:55.734 INFO:tasks.workunit.client.0.smithi136.stdout:waiting osd.2 seq 64424509445 2024-09-15T20:32:55.734 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-15T20:32:56.100 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 64424509445 -lt 64424509445 2024-09-15T20:32:56.101 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-15T20:32:56.101 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T20:32:56.101 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T20:32:56.582 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 1 == 0 2024-09-15T20:32:56.582 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-15T20:32:56.582 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-15T20:32:56.582 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-15T20:32:56.582 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T20:32:56.582 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T20:32:56.582 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T20:32:56.582 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T20:32:56.855 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=0 2024-09-15T20:32:56.855 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-15T20:32:56.855 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T20:32:56.855 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T20:32:57.202 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 0 = 1 2024-09-15T20:32:57.202 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 0 '!=' -1 2024-09-15T20:32:57.202 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: loop=0 2024-09-15T20:32:57.202 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1652: wait_for_clean: num_active_clean=0 2024-09-15T20:32:57.202 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-15T20:32:57.202 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 0.1 2024-09-15T20:32:57.303 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-15T20:32:57.303 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-15T20:32:57.303 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-15T20:32:57.304 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-15T20:32:57.304 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T20:32:57.304 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T20:32:57.304 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T20:32:57.304 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T20:32:57.573 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=0 2024-09-15T20:32:57.573 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-15T20:32:57.573 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T20:32:57.574 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T20:32:57.919 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 0 = 1 2024-09-15T20:32:57.920 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 0 '!=' 0 2024-09-15T20:32:57.920 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: get_is_making_recovery_progress 2024-09-15T20:32:57.920 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2024-09-15T20:32:57.920 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-15T20:32:57.920 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-15T20:32:57.920 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-15T20:32:57.920 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2024-09-15T20:32:57.920 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-15T20:32:58.263 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=null 2024-09-15T20:32:58.263 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test null '!=' null 2024-09-15T20:32:58.263 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1655: wait_for_clean: (( 1 >= 13 )) 2024-09-15T20:32:58.263 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-15T20:32:58.263 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 0.2 2024-09-15T20:32:58.464 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-15T20:32:58.464 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-15T20:32:58.465 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-15T20:32:58.465 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-15T20:32:58.465 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T20:32:58.465 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T20:32:58.465 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T20:32:58.465 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T20:32:58.736 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=0 2024-09-15T20:32:58.737 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-15T20:32:58.737 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T20:32:58.737 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T20:32:59.080 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 0 = 1 2024-09-15T20:32:59.080 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 0 '!=' 0 2024-09-15T20:32:59.080 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: get_is_making_recovery_progress 2024-09-15T20:32:59.080 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2024-09-15T20:32:59.080 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-15T20:32:59.080 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-15T20:32:59.080 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-15T20:32:59.080 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2024-09-15T20:32:59.080 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-15T20:32:59.424 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=null 2024-09-15T20:32:59.424 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test null '!=' null 2024-09-15T20:32:59.425 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1655: wait_for_clean: (( 2 >= 13 )) 2024-09-15T20:32:59.425 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-15T20:32:59.425 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 0.4 2024-09-15T20:32:59.826 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-15T20:32:59.826 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-15T20:32:59.826 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-15T20:32:59.826 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-15T20:32:59.826 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T20:32:59.826 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T20:32:59.826 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T20:32:59.826 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T20:33:00.097 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=1 2024-09-15T20:33:00.097 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-15T20:33:00.097 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T20:33:00.097 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T20:33:00.444 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 = 1 2024-09-15T20:33:00.445 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-15T20:33:00.445 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-15T20:33:00.445 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:557: TEST_auto_repair_bluestore_failed: local payload=ABCDEF 2024-09-15T20:33:00.445 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:558: TEST_auto_repair_bluestore_failed: echo ABCDEF 2024-09-15T20:33:00.445 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:559: TEST_auto_repair_bluestore_failed: seq 1 10 2024-09-15T20:33:00.445 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:559: TEST_auto_repair_bluestore_failed: for i in $(seq 1 10) 2024-09-15T20:33:00.445 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:561: TEST_auto_repair_bluestore_failed: rados --pool testpool put obj1 td/osd-scrub-repair/ORIGINAL 2024-09-15T20:33:00.578 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:559: TEST_auto_repair_bluestore_failed: for i in $(seq 1 10) 2024-09-15T20:33:00.578 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:561: TEST_auto_repair_bluestore_failed: rados --pool testpool put obj2 td/osd-scrub-repair/ORIGINAL 2024-09-15T20:33:00.704 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:559: TEST_auto_repair_bluestore_failed: for i in $(seq 1 10) 2024-09-15T20:33:00.704 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:561: TEST_auto_repair_bluestore_failed: rados --pool testpool put obj3 td/osd-scrub-repair/ORIGINAL 2024-09-15T20:33:00.829 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:559: TEST_auto_repair_bluestore_failed: for i in $(seq 1 10) 2024-09-15T20:33:00.829 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:561: TEST_auto_repair_bluestore_failed: rados --pool testpool put obj4 td/osd-scrub-repair/ORIGINAL 2024-09-15T20:33:00.947 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:559: TEST_auto_repair_bluestore_failed: for i in $(seq 1 10) 2024-09-15T20:33:00.947 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:561: TEST_auto_repair_bluestore_failed: rados --pool testpool put obj5 td/osd-scrub-repair/ORIGINAL 2024-09-15T20:33:01.080 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:559: TEST_auto_repair_bluestore_failed: for i in $(seq 1 10) 2024-09-15T20:33:01.080 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:561: TEST_auto_repair_bluestore_failed: rados --pool testpool put obj6 td/osd-scrub-repair/ORIGINAL 2024-09-15T20:33:01.214 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:559: TEST_auto_repair_bluestore_failed: for i in $(seq 1 10) 2024-09-15T20:33:01.215 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:561: TEST_auto_repair_bluestore_failed: rados --pool testpool put obj7 td/osd-scrub-repair/ORIGINAL 2024-09-15T20:33:01.332 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:559: TEST_auto_repair_bluestore_failed: for i in $(seq 1 10) 2024-09-15T20:33:01.332 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:561: TEST_auto_repair_bluestore_failed: rados --pool testpool put obj8 td/osd-scrub-repair/ORIGINAL 2024-09-15T20:33:01.500 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:559: TEST_auto_repair_bluestore_failed: for i in $(seq 1 10) 2024-09-15T20:33:01.500 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:561: TEST_auto_repair_bluestore_failed: rados --pool testpool put obj9 td/osd-scrub-repair/ORIGINAL 2024-09-15T20:33:01.626 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:559: TEST_auto_repair_bluestore_failed: for i in $(seq 1 10) 2024-09-15T20:33:01.626 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:561: TEST_auto_repair_bluestore_failed: rados --pool testpool put obj10 td/osd-scrub-repair/ORIGINAL 2024-09-15T20:33:01.752 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:566: TEST_auto_repair_bluestore_failed: get_not_primary testpool SOMETHING 2024-09-15T20:33:01.752 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1220: get_not_primary: local poolname=testpool 2024-09-15T20:33:01.752 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1221: get_not_primary: local objectname=SOMETHING 2024-09-15T20:33:01.752 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1223: get_not_primary: get_primary testpool SOMETHING 2024-09-15T20:33:01.752 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1187: get_primary: local poolname=testpool 2024-09-15T20:33:01.752 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1188: get_primary: local objectname=SOMETHING 2024-09-15T20:33:01.752 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1190: get_primary: ceph --format json osd map testpool SOMETHING 2024-09-15T20:33:01.753 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1191: get_primary: jq .acting_primary 2024-09-15T20:33:02.028 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1223: get_not_primary: local primary=1 2024-09-15T20:33:02.029 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1224: get_not_primary: ceph --format json osd map testpool SOMETHING 2024-09-15T20:33:02.029 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1225: get_not_primary: jq '.acting | map(select (. != 1)) | .[0]' 2024-09-15T20:33:02.308 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:566: TEST_auto_repair_bluestore_failed: objectstore_tool td/osd-scrub-repair 0 obj1 remove 2024-09-15T20:33:02.308 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2024-09-15T20:33:02.309 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2024-09-15T20:33:02.309 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=0 2024-09-15T20:33:02.309 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2024-09-15T20:33:02.309 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 obj1 remove 2024-09-15T20:33:02.309 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-09-15T20:33:02.309 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2024-09-15T20:33:02.309 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=0 2024-09-15T20:33:02.309 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2024-09-15T20:33:02.309 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2024-09-15T20:33:02.309 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-15T20:33:02.309 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-15T20:33:02.309 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-15T20:33:02.309 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-15T20:33:02.310 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-15T20:33:02.615 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-15T20:33:02.615 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 obj1 remove 2024-09-15T20:33:02.615 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-15T20:33:02.615 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-15T20:33:02.615 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=0 2024-09-15T20:33:02.615 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-15T20:33:02.615 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-09-15T20:33:02.615 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 obj1 remove 2024-09-15T20:33:04.221 INFO:tasks.workunit.client.0.smithi136.stdout:remove #1:ff7b1f36:::obj1:head# 2024-09-15T20:33:04.743 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-09-15T20:33:04.743 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2024-09-15T20:33:04.743 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-15T20:33:04.743 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=0 2024-09-15T20:33:04.743 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-15T20:33:04.743 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-09-15T20:33:04.743 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=ce4892c7-6c6d-4b4f-8e6f-07a3e11133e5 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-15T20:33:04.743 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T20:33:04.743 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T20:33:04.743 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T20:33:04.743 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-09-15T20:33:04.744 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-09-15T20:33:04.744 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-15T20:33:04.744 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-15T20:33:04.744 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-15T20:33:04.744 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-15T20:33:04.744 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-15T20:33:04.744 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-15T20:33:04.744 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-15T20:33:04.744 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T20:33:04.744 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52602 2024-09-15T20:33:04.744 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.52602/$cluster-$name.asok' 2024-09-15T20:33:04.744 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.52602/$cluster-$name.asok' 2024-09-15T20:33:04.744 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-15T20:33:04.745 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-15T20:33:04.745 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-15T20:33:04.745 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T20:33:04.745 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T20:33:04.745 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T20:33:04.745 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T20:33:04.745 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-15T20:33:04.745 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+='--osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0' 2024-09-15T20:33:04.745 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-09-15T20:33:04.746 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.0 2024-09-15T20:33:04.746 INFO:tasks.workunit.client.0.smithi136.stderr:start osd.0 2024-09-15T20:33:04.746 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 0 --fsid=ce4892c7-6c6d-4b4f-8e6f-07a3e11133e5 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.52602/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-09-15T20:33:04.746 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-09-15T20:33:04.746 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 0 = 0 ']' 2024-09-15T20:33:04.746 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-15T20:33:04.746 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-15T20:33:04.747 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-15T20:33:04.776 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:33:04.774+0000 7f023b545640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T20:33:04.779 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:33:04.777+0000 7f023b545640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T20:33:04.780 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:33:04.778+0000 7f023b545640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T20:33:05.034 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 0 2024-09-15T20:33:05.034 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-15T20:33:05.034 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2024-09-15T20:33:05.034 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-15T20:33:05.035 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-15T20:33:05.035 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:33:05.035 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-15T20:33:05.035 INFO:tasks.workunit.client.0.smithi136.stderr:0 2024-09-15T20:33:05.035 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:33:05.035 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T20:33:05.321 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:33:06.323 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:33:06.323 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:33:06.323 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-15T20:33:06.323 INFO:tasks.workunit.client.0.smithi136.stderr:1 2024-09-15T20:33:06.323 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:33:06.323 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T20:33:06.604 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:33:06.624 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:33:06.622+0000 7f023b545640 -1 Falling back to public interface 2024-09-15T20:33:07.606 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:33:07.606 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:33:07.606 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-15T20:33:07.606 INFO:tasks.workunit.client.0.smithi136.stderr:2 2024-09-15T20:33:07.606 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:33:07.606 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T20:33:07.890 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:33:08.892 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:33:08.892 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:33:08.892 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-15T20:33:08.892 INFO:tasks.workunit.client.0.smithi136.stderr:3 2024-09-15T20:33:08.892 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:33:08.892 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T20:33:09.177 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:33:09.375 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:33:09.373+0000 7f023b545640 -1 osd.0 21 log_to_monitors true 2024-09-15T20:33:10.178 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:33:10.178 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:33:10.179 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-15T20:33:10.179 INFO:tasks.workunit.client.0.smithi136.stderr:4 2024-09-15T20:33:10.179 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:33:10.179 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T20:33:10.513 INFO:tasks.workunit.client.0.smithi136.stderr:osd.0 up in weight 1 up_from 25 up_thru 0 down_at 22 last_clean_interval [5,21) [v2:127.0.0.1:6802/3399429214,v1:127.0.0.1:6803/3399429214] [v2:127.0.0.1:6804/3399429214,v1:127.0.0.1:6805/3399429214] exists,up 555689c8-c81e-4042-90bc-288636f201a7 2024-09-15T20:33:10.513 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-15T20:33:10.514 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-15T20:33:10.514 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-15T20:33:10.514 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2024-09-15T20:33:10.514 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-15T20:33:10.514 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-15T20:33:10.514 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-15T20:33:10.514 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-15T20:33:10.514 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-15T20:33:10.514 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-15T20:33:10.514 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-15T20:33:10.514 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-15T20:33:10.514 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-15T20:33:10.602 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-15T20:33:10.603 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-15T20:33:10.603 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-15T20:33:10.603 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-15T20:33:10.603 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-15T20:33:10.603 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-15T20:33:10.927 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-15T20:33:10.928 INFO:tasks.workunit.client.0.smithi136.stderr:1 2024-09-15T20:33:10.928 INFO:tasks.workunit.client.0.smithi136.stderr:2' 2024-09-15T20:33:10.928 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-15T20:33:10.928 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T20:33:10.928 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-15T20:33:11.030 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=107374182402 2024-09-15T20:33:11.031 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 107374182402 2024-09-15T20:33:11.031 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-107374182402' 2024-09-15T20:33:11.031 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T20:33:11.031 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-15T20:33:11.134 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=42949672974 2024-09-15T20:33:11.134 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 42949672974 2024-09-15T20:33:11.134 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-107374182402 1-42949672974' 2024-09-15T20:33:11.134 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T20:33:11.134 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-15T20:33:11.252 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=64424509449 2024-09-15T20:33:11.252 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 64424509449 2024-09-15T20:33:11.252 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-107374182402 1-42949672974 2-64424509449' 2024-09-15T20:33:11.252 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T20:33:11.252 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-107374182402 2024-09-15T20:33:11.252 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T20:33:11.253 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-15T20:33:11.253 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-107374182402 2024-09-15T20:33:11.253 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T20:33:11.254 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=107374182402 2024-09-15T20:33:11.255 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 107374182402' 2024-09-15T20:33:11.255 INFO:tasks.workunit.client.0.smithi136.stderr:waiting osd.0 seq 107374182402 2024-09-15T20:33:11.255 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T20:33:11.567 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 107374182402 -lt 107374182402 2024-09-15T20:33:11.567 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T20:33:11.567 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-42949672974 2024-09-15T20:33:11.567 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T20:33:11.568 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-15T20:33:11.569 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-42949672974 2024-09-15T20:33:11.569 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T20:33:11.570 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=42949672974 2024-09-15T20:33:11.570 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 42949672974' 2024-09-15T20:33:11.570 INFO:tasks.workunit.client.0.smithi136.stderr:waiting osd.1 seq 42949672974 2024-09-15T20:33:11.570 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-15T20:33:11.846 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 42949672974 -lt 42949672974 2024-09-15T20:33:11.846 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T20:33:11.847 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-64424509449 2024-09-15T20:33:11.847 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T20:33:11.848 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-15T20:33:11.848 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-64424509449 2024-09-15T20:33:11.848 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T20:33:11.849 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=64424509449 2024-09-15T20:33:11.849 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 64424509449' 2024-09-15T20:33:11.850 INFO:tasks.workunit.client.0.smithi136.stderr:waiting osd.2 seq 64424509449 2024-09-15T20:33:11.850 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-15T20:33:12.125 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 64424509448 -lt 64424509449 2024-09-15T20:33:12.125 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-15T20:33:13.128 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-15T20:33:13.128 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-15T20:33:13.420 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 64424509449 -lt 64424509449 2024-09-15T20:33:13.421 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-15T20:33:13.421 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T20:33:13.421 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T20:33:13.803 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 1 == 0 2024-09-15T20:33:13.804 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-15T20:33:13.804 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-15T20:33:13.804 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-15T20:33:13.804 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T20:33:13.804 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T20:33:13.804 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T20:33:13.804 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T20:33:14.078 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=1 2024-09-15T20:33:14.078 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-15T20:33:14.079 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T20:33:14.079 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T20:33:14.424 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 = 1 2024-09-15T20:33:14.424 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-15T20:33:14.424 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-15T20:33:14.424 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:568: TEST_auto_repair_bluestore_failed: get_not_primary testpool SOMETHING 2024-09-15T20:33:14.424 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1220: get_not_primary: local poolname=testpool 2024-09-15T20:33:14.424 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1221: get_not_primary: local objectname=SOMETHING 2024-09-15T20:33:14.425 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1223: get_not_primary: get_primary testpool SOMETHING 2024-09-15T20:33:14.425 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1187: get_primary: local poolname=testpool 2024-09-15T20:33:14.425 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1188: get_primary: local objectname=SOMETHING 2024-09-15T20:33:14.425 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1190: get_primary: ceph --format json osd map testpool SOMETHING 2024-09-15T20:33:14.425 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1191: get_primary: jq .acting_primary 2024-09-15T20:33:14.746 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1223: get_not_primary: local primary=1 2024-09-15T20:33:14.746 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1224: get_not_primary: ceph --format json osd map testpool SOMETHING 2024-09-15T20:33:14.746 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1225: get_not_primary: jq '.acting | map(select (. != 1)) | .[0]' 2024-09-15T20:33:15.025 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:568: TEST_auto_repair_bluestore_failed: objectstore_tool td/osd-scrub-repair 0 obj2 remove 2024-09-15T20:33:15.025 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2024-09-15T20:33:15.025 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2024-09-15T20:33:15.025 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=0 2024-09-15T20:33:15.025 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2024-09-15T20:33:15.025 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 obj2 remove 2024-09-15T20:33:15.025 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-09-15T20:33:15.025 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2024-09-15T20:33:15.026 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=0 2024-09-15T20:33:15.026 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2024-09-15T20:33:15.026 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2024-09-15T20:33:15.026 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-15T20:33:15.026 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-15T20:33:15.026 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-15T20:33:15.026 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-15T20:33:15.026 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-15T20:33:15.331 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-15T20:33:15.331 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 obj2 remove 2024-09-15T20:33:15.332 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-15T20:33:15.332 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-15T20:33:15.332 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=0 2024-09-15T20:33:15.332 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-15T20:33:15.332 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-09-15T20:33:15.332 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 obj2 remove 2024-09-15T20:33:16.682 INFO:tasks.workunit.client.0.smithi136.stdout:remove #1:104778fc:::obj2:head# 2024-09-15T20:33:17.203 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-09-15T20:33:17.203 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2024-09-15T20:33:17.203 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-15T20:33:17.203 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=0 2024-09-15T20:33:17.203 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-15T20:33:17.203 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-09-15T20:33:17.203 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=ce4892c7-6c6d-4b4f-8e6f-07a3e11133e5 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-15T20:33:17.203 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T20:33:17.203 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T20:33:17.203 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T20:33:17.204 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-09-15T20:33:17.204 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-09-15T20:33:17.204 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-15T20:33:17.204 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-15T20:33:17.204 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-15T20:33:17.204 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-15T20:33:17.204 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-15T20:33:17.204 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-15T20:33:17.204 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-15T20:33:17.204 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T20:33:17.204 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52602 2024-09-15T20:33:17.204 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.52602/$cluster-$name.asok' 2024-09-15T20:33:17.204 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.52602/$cluster-$name.asok' 2024-09-15T20:33:17.204 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-15T20:33:17.204 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-15T20:33:17.205 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-15T20:33:17.205 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T20:33:17.205 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T20:33:17.205 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T20:33:17.207 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T20:33:17.207 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-15T20:33:17.207 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+='--osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0' 2024-09-15T20:33:17.207 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-09-15T20:33:17.208 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.0 2024-09-15T20:33:17.208 INFO:tasks.workunit.client.0.smithi136.stderr:start osd.0 2024-09-15T20:33:17.208 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 0 --fsid=ce4892c7-6c6d-4b4f-8e6f-07a3e11133e5 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.52602/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-09-15T20:33:17.209 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-09-15T20:33:17.209 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 0 = 0 ']' 2024-09-15T20:33:17.209 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-15T20:33:17.209 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-15T20:33:17.209 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-15T20:33:17.236 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:33:17.233+0000 7fd0799bd640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T20:33:17.238 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:33:17.235+0000 7fd0799bd640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T20:33:17.239 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:33:17.237+0000 7fd0799bd640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T20:33:17.515 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 0 2024-09-15T20:33:17.516 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-15T20:33:17.516 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2024-09-15T20:33:17.516 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-15T20:33:17.516 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-15T20:33:17.516 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:33:17.516 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-15T20:33:17.517 INFO:tasks.workunit.client.0.smithi136.stderr:0 2024-09-15T20:33:17.517 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:33:17.517 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T20:33:17.789 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:33:18.791 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:33:18.791 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:33:18.791 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-15T20:33:18.791 INFO:tasks.workunit.client.0.smithi136.stderr:1 2024-09-15T20:33:18.791 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:33:18.791 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T20:33:18.834 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:33:18.832+0000 7fd0799bd640 -1 Falling back to public interface 2024-09-15T20:33:19.074 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:33:20.076 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:33:20.076 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:33:20.076 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-15T20:33:20.076 INFO:tasks.workunit.client.0.smithi136.stderr:2 2024-09-15T20:33:20.077 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:33:20.077 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T20:33:20.361 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:33:21.363 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:33:21.363 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:33:21.363 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-15T20:33:21.363 INFO:tasks.workunit.client.0.smithi136.stderr:3 2024-09-15T20:33:21.363 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:33:21.363 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T20:33:21.538 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:33:21.536+0000 7fd0799bd640 -1 osd.0 26 log_to_monitors true 2024-09-15T20:33:21.734 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:33:22.736 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:33:22.736 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:33:22.736 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-15T20:33:22.736 INFO:tasks.workunit.client.0.smithi136.stderr:4 2024-09-15T20:33:22.736 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:33:22.736 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T20:33:23.020 INFO:tasks.workunit.client.0.smithi136.stderr:osd.0 up in weight 1 up_from 30 up_thru 0 down_at 27 last_clean_interval [25,26) [v2:127.0.0.1:6802/3394315324,v1:127.0.0.1:6803/3394315324] [v2:127.0.0.1:6804/3394315324,v1:127.0.0.1:6805/3394315324] exists,up 555689c8-c81e-4042-90bc-288636f201a7 2024-09-15T20:33:23.020 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-15T20:33:23.020 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-15T20:33:23.020 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-15T20:33:23.020 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2024-09-15T20:33:23.020 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-15T20:33:23.020 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-15T20:33:23.021 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-15T20:33:23.021 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-15T20:33:23.021 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-15T20:33:23.021 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-15T20:33:23.021 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-15T20:33:23.021 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-15T20:33:23.021 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-15T20:33:23.110 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-15T20:33:23.110 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-15T20:33:23.110 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-15T20:33:23.110 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-15T20:33:23.110 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-15T20:33:23.111 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-15T20:33:23.410 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-15T20:33:23.410 INFO:tasks.workunit.client.0.smithi136.stderr:1 2024-09-15T20:33:23.410 INFO:tasks.workunit.client.0.smithi136.stderr:2' 2024-09-15T20:33:23.410 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-15T20:33:23.410 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T20:33:23.410 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-15T20:33:23.515 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=128849018882 2024-09-15T20:33:23.515 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 128849018882 2024-09-15T20:33:23.515 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-128849018882' 2024-09-15T20:33:23.515 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T20:33:23.515 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-15T20:33:23.618 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=42949672978 2024-09-15T20:33:23.618 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 42949672978 2024-09-15T20:33:23.618 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-128849018882 1-42949672978' 2024-09-15T20:33:23.618 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T20:33:23.618 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-15T20:33:23.720 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=64424509453 2024-09-15T20:33:23.720 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 64424509453 2024-09-15T20:33:23.721 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-128849018882 1-42949672978 2-64424509453' 2024-09-15T20:33:23.721 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T20:33:23.721 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-128849018882 2024-09-15T20:33:23.721 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T20:33:23.722 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-15T20:33:23.723 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-128849018882 2024-09-15T20:33:23.723 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T20:33:23.724 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=128849018882 2024-09-15T20:33:23.724 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 128849018882' 2024-09-15T20:33:23.724 INFO:tasks.workunit.client.0.smithi136.stderr:waiting osd.0 seq 128849018882 2024-09-15T20:33:23.724 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T20:33:23.997 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 1 -lt 128849018882 2024-09-15T20:33:23.997 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-15T20:33:24.998 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-15T20:33:24.998 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T20:33:25.281 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 128849018882 -lt 128849018882 2024-09-15T20:33:25.281 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T20:33:25.281 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-42949672978 2024-09-15T20:33:25.282 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T20:33:25.282 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-15T20:33:25.283 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-42949672978 2024-09-15T20:33:25.283 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T20:33:25.284 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=42949672978 2024-09-15T20:33:25.284 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 42949672978' 2024-09-15T20:33:25.284 INFO:tasks.workunit.client.0.smithi136.stderr:waiting osd.1 seq 42949672978 2024-09-15T20:33:25.284 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-15T20:33:25.589 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 42949672978 -lt 42949672978 2024-09-15T20:33:25.589 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T20:33:25.590 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-64424509453 2024-09-15T20:33:25.590 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T20:33:25.591 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-15T20:33:25.591 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-64424509453 2024-09-15T20:33:25.591 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T20:33:25.592 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=64424509453 2024-09-15T20:33:25.592 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 64424509453' 2024-09-15T20:33:25.592 INFO:tasks.workunit.client.0.smithi136.stderr:waiting osd.2 seq 64424509453 2024-09-15T20:33:25.593 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-15T20:33:25.868 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 64424509453 -lt 64424509453 2024-09-15T20:33:25.869 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-15T20:33:25.869 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T20:33:25.869 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T20:33:26.212 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 1 == 0 2024-09-15T20:33:26.213 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-15T20:33:26.213 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-15T20:33:26.213 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-15T20:33:26.213 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T20:33:26.213 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T20:33:26.213 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T20:33:26.213 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T20:33:26.485 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=1 2024-09-15T20:33:26.485 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-15T20:33:26.485 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T20:33:26.485 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T20:33:26.831 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 = 1 2024-09-15T20:33:26.831 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-15T20:33:26.831 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-15T20:33:26.831 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:569: TEST_auto_repair_bluestore_failed: get_primary testpool SOMETHING 2024-09-15T20:33:26.831 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1187: get_primary: local poolname=testpool 2024-09-15T20:33:26.831 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1188: get_primary: local objectname=SOMETHING 2024-09-15T20:33:26.831 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1190: get_primary: ceph --format json osd map testpool SOMETHING 2024-09-15T20:33:26.831 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1191: get_primary: jq .acting_primary 2024-09-15T20:33:27.108 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:569: TEST_auto_repair_bluestore_failed: objectstore_tool td/osd-scrub-repair 1 obj2 rm-attr _ 2024-09-15T20:33:27.108 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2024-09-15T20:33:27.108 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2024-09-15T20:33:27.108 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=1 2024-09-15T20:33:27.108 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2024-09-15T20:33:27.108 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 obj2 rm-attr _ 2024-09-15T20:33:27.108 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-09-15T20:33:27.109 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2024-09-15T20:33:27.109 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=1 2024-09-15T20:33:27.109 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2024-09-15T20:33:27.109 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-09-15T20:33:27.109 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-15T20:33:27.109 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-15T20:33:27.109 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-15T20:33:27.109 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-15T20:33:27.109 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-15T20:33:27.414 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-15T20:33:27.414 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 obj2 rm-attr _ 2024-09-15T20:33:27.414 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-15T20:33:27.415 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-15T20:33:27.415 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=1 2024-09-15T20:33:27.415 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-15T20:33:27.415 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-09-15T20:33:27.415 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 obj2 rm-attr _ 2024-09-15T20:33:29.297 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-09-15T20:33:29.297 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2024-09-15T20:33:29.297 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-15T20:33:29.297 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=1 2024-09-15T20:33:29.297 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-15T20:33:29.297 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-09-15T20:33:29.298 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=ce4892c7-6c6d-4b4f-8e6f-07a3e11133e5 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-15T20:33:29.298 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T20:33:29.298 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T20:33:29.298 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T20:33:29.298 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-09-15T20:33:29.298 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-09-15T20:33:29.298 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-15T20:33:29.298 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-15T20:33:29.298 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-15T20:33:29.298 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-15T20:33:29.298 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-15T20:33:29.298 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-15T20:33:29.298 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-15T20:33:29.298 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T20:33:29.299 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52602 2024-09-15T20:33:29.299 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.52602/$cluster-$name.asok' 2024-09-15T20:33:29.299 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.52602/$cluster-$name.asok' 2024-09-15T20:33:29.299 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-15T20:33:29.299 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-15T20:33:29.299 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-15T20:33:29.299 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T20:33:29.299 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T20:33:29.299 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T20:33:29.299 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T20:33:29.299 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-15T20:33:29.299 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+='--osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0' 2024-09-15T20:33:29.299 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-09-15T20:33:29.300 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.1 2024-09-15T20:33:29.301 INFO:tasks.workunit.client.0.smithi136.stderr:start osd.1 2024-09-15T20:33:29.301 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 1 --fsid=ce4892c7-6c6d-4b4f-8e6f-07a3e11133e5 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.52602/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-09-15T20:33:29.301 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-09-15T20:33:29.301 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 1 = 1 ']' 2024-09-15T20:33:29.301 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-15T20:33:29.301 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-15T20:33:29.301 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-15T20:33:29.329 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:33:29.327+0000 7fd33f40f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T20:33:29.332 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:33:29.330+0000 7fd33f40f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T20:33:29.333 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:33:29.331+0000 7fd33f40f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T20:33:29.583 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 1 2024-09-15T20:33:29.584 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-15T20:33:29.584 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2024-09-15T20:33:29.584 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-15T20:33:29.584 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-15T20:33:29.585 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:33:29.585 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-15T20:33:29.585 INFO:tasks.workunit.client.0.smithi136.stderr:0 2024-09-15T20:33:29.585 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:33:29.585 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T20:33:29.862 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:33:30.709 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:33:30.707+0000 7fd33f40f640 -1 Falling back to public interface 2024-09-15T20:33:30.863 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:33:30.864 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:33:30.864 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-15T20:33:30.864 INFO:tasks.workunit.client.0.smithi136.stderr:1 2024-09-15T20:33:30.864 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:33:30.864 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T20:33:31.145 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:33:32.146 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:33:32.146 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:33:32.147 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-15T20:33:32.147 INFO:tasks.workunit.client.0.smithi136.stderr:2 2024-09-15T20:33:32.147 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:33:32.147 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T20:33:32.435 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:33:33.437 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:33:33.437 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:33:33.437 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-15T20:33:33.437 INFO:tasks.workunit.client.0.smithi136.stderr:3 2024-09-15T20:33:33.437 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:33:33.437 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T20:33:33.619 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:33:33.617+0000 7fd33f40f640 -1 osd.1 31 log_to_monitors true 2024-09-15T20:33:33.725 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:33:34.727 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:33:34.727 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:33:34.727 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-15T20:33:34.727 INFO:tasks.workunit.client.0.smithi136.stderr:4 2024-09-15T20:33:34.727 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:33:34.727 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T20:33:35.015 INFO:tasks.workunit.client.0.smithi136.stderr:osd.1 up in weight 1 up_from 35 up_thru 35 down_at 32 last_clean_interval [10,31) [v2:127.0.0.1:6810/1862379850,v1:127.0.0.1:6811/1862379850] [v2:127.0.0.1:6812/1862379850,v1:127.0.0.1:6813/1862379850] exists,up a91f9a5c-e300-4dd1-9666-c0e98286384c 2024-09-15T20:33:35.015 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-15T20:33:35.015 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-15T20:33:35.015 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-15T20:33:35.015 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2024-09-15T20:33:35.015 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-15T20:33:35.015 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-15T20:33:35.015 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-15T20:33:35.015 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-15T20:33:35.016 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-15T20:33:35.016 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-15T20:33:35.016 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-15T20:33:35.016 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-15T20:33:35.017 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-15T20:33:35.105 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-15T20:33:35.105 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-15T20:33:35.105 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-15T20:33:35.105 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-15T20:33:35.105 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-15T20:33:35.105 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-15T20:33:35.394 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-15T20:33:35.394 INFO:tasks.workunit.client.0.smithi136.stderr:1 2024-09-15T20:33:35.394 INFO:tasks.workunit.client.0.smithi136.stderr:2' 2024-09-15T20:33:35.394 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-15T20:33:35.394 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T20:33:35.394 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-15T20:33:35.497 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=128849018885 2024-09-15T20:33:35.497 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 128849018885 2024-09-15T20:33:35.497 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-128849018885' 2024-09-15T20:33:35.498 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T20:33:35.498 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-15T20:33:35.601 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=150323855362 2024-09-15T20:33:35.601 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 150323855362 2024-09-15T20:33:35.601 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-128849018885 1-150323855362' 2024-09-15T20:33:35.601 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T20:33:35.601 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-15T20:33:35.703 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=64424509456 2024-09-15T20:33:35.703 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 64424509456 2024-09-15T20:33:35.703 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-128849018885 1-150323855362 2-64424509456' 2024-09-15T20:33:35.703 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T20:33:35.704 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-128849018885 2024-09-15T20:33:35.704 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T20:33:35.704 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-15T20:33:35.705 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-128849018885 2024-09-15T20:33:35.705 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T20:33:35.706 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=128849018885 2024-09-15T20:33:35.706 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 128849018885' 2024-09-15T20:33:35.706 INFO:tasks.workunit.client.0.smithi136.stderr:waiting osd.0 seq 128849018885 2024-09-15T20:33:35.706 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T20:33:35.980 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 128849018884 -lt 128849018885 2024-09-15T20:33:35.981 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-15T20:33:36.982 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-15T20:33:36.982 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T20:33:37.268 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 128849018886 -lt 128849018885 2024-09-15T20:33:37.268 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T20:33:37.269 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-150323855362 2024-09-15T20:33:37.269 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T20:33:37.269 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-15T20:33:37.270 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-150323855362 2024-09-15T20:33:37.270 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T20:33:37.271 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=150323855362 2024-09-15T20:33:37.271 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 150323855362' 2024-09-15T20:33:37.271 INFO:tasks.workunit.client.0.smithi136.stderr:waiting osd.1 seq 150323855362 2024-09-15T20:33:37.272 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-15T20:33:37.565 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 150323855362 -lt 150323855362 2024-09-15T20:33:37.565 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T20:33:37.565 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-64424509456 2024-09-15T20:33:37.565 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T20:33:37.566 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-15T20:33:37.567 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-64424509456 2024-09-15T20:33:37.567 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T20:33:37.568 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=64424509456 2024-09-15T20:33:37.568 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 64424509456' 2024-09-15T20:33:37.568 INFO:tasks.workunit.client.0.smithi136.stderr:waiting osd.2 seq 64424509456 2024-09-15T20:33:37.568 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-15T20:33:37.847 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 64424509456 -lt 64424509456 2024-09-15T20:33:37.847 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-15T20:33:37.847 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T20:33:37.847 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T20:33:38.198 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 1 == 0 2024-09-15T20:33:38.198 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-15T20:33:38.198 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-15T20:33:38.198 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-15T20:33:38.198 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T20:33:38.199 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T20:33:38.199 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T20:33:38.199 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T20:33:38.470 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=1 2024-09-15T20:33:38.471 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-15T20:33:38.471 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T20:33:38.471 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T20:33:38.816 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 = 1 2024-09-15T20:33:38.816 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-15T20:33:38.817 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-15T20:33:38.817 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:571: TEST_auto_repair_bluestore_failed: get_pg testpool obj1 2024-09-15T20:33:38.817 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1084: get_pg: local poolname=testpool 2024-09-15T20:33:38.817 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1085: get_pg: local objectname=obj1 2024-09-15T20:33:38.817 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1087: get_pg: ceph --format json osd map testpool obj1 2024-09-15T20:33:38.818 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1087: get_pg: jq -r .pgid 2024-09-15T20:33:39.096 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:571: TEST_auto_repair_bluestore_failed: local pgid=1.0 2024-09-15T20:33:39.097 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:572: TEST_auto_repair_bluestore_failed: get_primary testpool obj1 2024-09-15T20:33:39.097 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1187: get_primary: local poolname=testpool 2024-09-15T20:33:39.097 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1188: get_primary: local objectname=obj1 2024-09-15T20:33:39.098 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1190: get_primary: ceph --format json osd map testpool obj1 2024-09-15T20:33:39.098 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1191: get_primary: jq .acting_primary 2024-09-15T20:33:39.392 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:572: TEST_auto_repair_bluestore_failed: local primary=1 2024-09-15T20:33:39.393 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:573: TEST_auto_repair_bluestore_failed: get_last_scrub_stamp 1.0 2024-09-15T20:33:39.393 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-15T20:33:39.393 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-15T20:33:39.393 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-15T20:33:39.393 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-15T20:33:39.665 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:573: TEST_auto_repair_bluestore_failed: local last_scrub_stamp=2024-09-15T20:32:52.893757+0000 2024-09-15T20:33:39.665 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:574: TEST_auto_repair_bluestore_failed: ceph tell 1.0 schedule-deep-scrub 2024-09-15T20:33:39.762 INFO:tasks.workunit.client.0.smithi136.stdout:{ 2024-09-15T20:33:39.762 INFO:tasks.workunit.client.0.smithi136.stdout: "deep": true, 2024-09-15T20:33:39.762 INFO:tasks.workunit.client.0.smithi136.stdout: "must": false, 2024-09-15T20:33:39.762 INFO:tasks.workunit.client.0.smithi136.stdout: "stamp": "2024-09-01T20:31:59.760779+0000" 2024-09-15T20:33:39.762 INFO:tasks.workunit.client.0.smithi136.stdout:} 2024-09-15T20:33:39.771 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:577: TEST_auto_repair_bluestore_failed: wait_for_scrub 1.0 2024-09-15T20:32:52.893757+0000 2024-09-15T20:33:39.771 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: local pgid=1.0 2024-09-15T20:33:39.772 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2018: wait_for_scrub: local last_scrub=2024-09-15T20:32:52.893757+0000 2024-09-15T20:33:39.772 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2019: wait_for_scrub: local sname=last_scrub_stamp 2024-09-15T20:33:39.772 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i=0 )) 2024-09-15T20:33:39.772 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-15T20:33:39.772 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-15T20:33:39.772 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-15T20:33:39.772 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-15T20:33:39.772 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-15T20:33:39.772 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-15T20:33:40.041 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-15T20:32:52.893757+0000 '>' 2024-09-15T20:32:52.893757+0000 2024-09-15T20:33:40.041 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-15T20:33:41.042 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-15T20:33:41.042 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-15T20:33:41.043 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-15T20:33:41.043 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-15T20:33:41.043 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-15T20:33:41.043 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-15T20:33:41.043 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-15T20:33:41.327 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-15T20:32:52.893757+0000 '>' 2024-09-15T20:32:52.893757+0000 2024-09-15T20:33:41.327 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-15T20:33:42.328 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-15T20:33:42.328 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-15T20:33:42.329 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-15T20:33:42.329 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-15T20:33:42.329 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-15T20:33:42.329 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-15T20:33:42.329 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-15T20:33:42.610 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-15T20:32:52.893757+0000 '>' 2024-09-15T20:32:52.893757+0000 2024-09-15T20:33:42.610 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-15T20:33:43.611 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-15T20:33:43.612 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-15T20:33:43.612 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-15T20:33:43.612 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-15T20:33:43.612 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-15T20:33:43.613 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-15T20:33:43.613 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-15T20:33:43.922 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-15T20:32:52.893757+0000 '>' 2024-09-15T20:32:52.893757+0000 2024-09-15T20:33:43.922 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-15T20:33:44.925 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-15T20:33:44.925 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-15T20:33:44.925 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-15T20:33:44.925 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-15T20:33:44.925 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-15T20:33:44.925 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-15T20:33:44.925 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-15T20:33:45.202 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-15T20:33:43.667907+0000 '>' 2024-09-15T20:32:52.893757+0000 2024-09-15T20:33:45.203 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2023: wait_for_scrub: return 0 2024-09-15T20:33:45.203 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:578: TEST_auto_repair_bluestore_failed: wait_for_clean 2024-09-15T20:33:45.203 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-15T20:33:45.203 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-15T20:33:45.203 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-15T20:33:45.203 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-15T20:33:45.203 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-15T20:33:45.203 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-15T20:33:45.203 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-15T20:33:45.203 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-15T20:33:45.203 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-15T20:33:45.293 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-15T20:33:45.293 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-15T20:33:45.293 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-15T20:33:45.293 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-15T20:33:45.293 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-15T20:33:45.293 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-15T20:33:45.570 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-15T20:33:45.570 INFO:tasks.workunit.client.0.smithi136.stderr:1 2024-09-15T20:33:45.570 INFO:tasks.workunit.client.0.smithi136.stderr:2' 2024-09-15T20:33:45.571 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-15T20:33:45.571 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T20:33:45.571 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-15T20:33:45.673 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=128849018888 2024-09-15T20:33:45.674 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 128849018888 2024-09-15T20:33:45.674 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-128849018888' 2024-09-15T20:33:45.674 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T20:33:45.674 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-15T20:33:45.777 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=150323855365 2024-09-15T20:33:45.777 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 150323855365 2024-09-15T20:33:45.777 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-128849018888 1-150323855365' 2024-09-15T20:33:45.777 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T20:33:45.777 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-15T20:33:45.879 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=64424509459 2024-09-15T20:33:45.879 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 64424509459 2024-09-15T20:33:45.879 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-128849018888 1-150323855365 2-64424509459' 2024-09-15T20:33:45.880 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T20:33:45.880 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-128849018888 2024-09-15T20:33:45.880 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T20:33:45.880 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-15T20:33:45.881 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-128849018888 2024-09-15T20:33:45.881 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T20:33:45.882 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=128849018888 2024-09-15T20:33:45.882 INFO:tasks.workunit.client.0.smithi136.stdout:waiting osd.0 seq 128849018888 2024-09-15T20:33:45.883 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 128849018888' 2024-09-15T20:33:45.883 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T20:33:46.158 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 128849018887 -lt 128849018888 2024-09-15T20:33:46.158 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-15T20:33:47.159 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-15T20:33:47.160 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T20:33:47.447 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 128849018889 -lt 128849018888 2024-09-15T20:33:47.447 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T20:33:47.448 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-150323855365 2024-09-15T20:33:47.448 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T20:33:47.448 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-15T20:33:47.449 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-150323855365 2024-09-15T20:33:47.449 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T20:33:47.450 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=150323855365 2024-09-15T20:33:47.450 INFO:tasks.workunit.client.0.smithi136.stdout:waiting osd.1 seq 150323855365 2024-09-15T20:33:47.451 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 150323855365' 2024-09-15T20:33:47.451 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-15T20:33:47.756 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 150323855365 -lt 150323855365 2024-09-15T20:33:47.756 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T20:33:47.757 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-64424509459 2024-09-15T20:33:47.757 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T20:33:47.757 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-15T20:33:47.758 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-64424509459 2024-09-15T20:33:47.758 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T20:33:47.759 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=64424509459 2024-09-15T20:33:47.759 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 64424509459' 2024-09-15T20:33:47.759 INFO:tasks.workunit.client.0.smithi136.stdout:waiting osd.2 seq 64424509459 2024-09-15T20:33:47.760 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-15T20:33:48.034 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 64424509459 -lt 64424509459 2024-09-15T20:33:48.035 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-15T20:33:48.035 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T20:33:48.035 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T20:33:48.380 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 1 == 0 2024-09-15T20:33:48.380 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-15T20:33:48.381 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-15T20:33:48.381 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-15T20:33:48.381 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T20:33:48.381 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T20:33:48.381 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T20:33:48.381 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T20:33:48.650 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=1 2024-09-15T20:33:48.651 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-15T20:33:48.651 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T20:33:48.651 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T20:33:48.997 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 = 1 2024-09-15T20:33:48.997 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-15T20:33:48.997 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-15T20:33:48.998 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:579: TEST_auto_repair_bluestore_failed: flush_pg_stats 2024-09-15T20:33:48.998 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-15T20:33:48.998 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-15T20:33:49.275 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-15T20:33:49.275 INFO:tasks.workunit.client.0.smithi136.stderr:1 2024-09-15T20:33:49.275 INFO:tasks.workunit.client.0.smithi136.stderr:2' 2024-09-15T20:33:49.275 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-15T20:33:49.275 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T20:33:49.275 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-15T20:33:49.402 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=128849018890 2024-09-15T20:33:49.403 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 128849018890 2024-09-15T20:33:49.403 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-128849018890' 2024-09-15T20:33:49.403 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T20:33:49.403 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-15T20:33:49.505 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=150323855367 2024-09-15T20:33:49.506 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 150323855367 2024-09-15T20:33:49.506 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-128849018890 1-150323855367' 2024-09-15T20:33:49.506 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T20:33:49.506 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-15T20:33:49.608 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=64424509461 2024-09-15T20:33:49.608 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 64424509461 2024-09-15T20:33:49.608 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-128849018890 1-150323855367 2-64424509461' 2024-09-15T20:33:49.608 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T20:33:49.609 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-128849018890 2024-09-15T20:33:49.609 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T20:33:49.609 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-15T20:33:49.610 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-128849018890 2024-09-15T20:33:49.610 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T20:33:49.611 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=128849018890 2024-09-15T20:33:49.611 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 128849018890' 2024-09-15T20:33:49.611 INFO:tasks.workunit.client.0.smithi136.stdout:waiting osd.0 seq 128849018890 2024-09-15T20:33:49.611 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T20:33:49.887 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 128849018889 -lt 128849018890 2024-09-15T20:33:49.887 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-15T20:33:50.889 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-15T20:33:50.889 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T20:33:51.174 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 128849018889 -lt 128849018890 2024-09-15T20:33:51.174 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-15T20:33:52.175 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 299 -eq 0 ']' 2024-09-15T20:33:52.176 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T20:33:52.460 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 128849018890 -lt 128849018890 2024-09-15T20:33:52.460 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T20:33:52.461 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-150323855367 2024-09-15T20:33:52.461 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T20:33:52.462 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-15T20:33:52.462 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-150323855367 2024-09-15T20:33:52.462 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T20:33:52.463 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=150323855367 2024-09-15T20:33:52.463 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 150323855367' 2024-09-15T20:33:52.464 INFO:tasks.workunit.client.0.smithi136.stdout:waiting osd.1 seq 150323855367 2024-09-15T20:33:52.464 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-15T20:33:52.779 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 150323855367 -lt 150323855367 2024-09-15T20:33:52.779 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T20:33:52.779 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-64424509461 2024-09-15T20:33:52.780 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T20:33:52.780 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-15T20:33:52.781 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-64424509461 2024-09-15T20:33:52.781 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T20:33:52.782 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=64424509461 2024-09-15T20:33:52.782 INFO:tasks.workunit.client.0.smithi136.stdout:waiting osd.2 seq 64424509461 2024-09-15T20:33:52.782 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 64424509461' 2024-09-15T20:33:52.783 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-15T20:33:53.062 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 64424509461 -lt 64424509461 2024-09-15T20:33:53.063 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:580: TEST_auto_repair_bluestore_failed: grep scrub_finish td/osd-scrub-repair/osd.1.log 2024-09-15T20:33:53.065 INFO:tasks.workunit.client.0.smithi136.stdout:2024-09-15T20:33:42.655+0000 7fd322ddc640 10 osd.1 pg_epoch: 36 pg[1.0( v 21'10 (0'0,21'10] local-lis/les=35/36 n=10 ec=17/17 lis/c=35/35 les/c/f=36/36/0 sis=35) [1,0] r=0 lpr=35 crt=21'10 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep [ 1.0: AUTO_REPAIR ] ] scrubber: scrub_finish before flags: AUTO_REPAIR. repair state: no-repair. deep_scrub_on_error: 0 2024-09-15T20:33:53.065 INFO:tasks.workunit.client.0.smithi136.stdout:2024-09-15T20:33:42.655+0000 7fd322ddc640 10 osd.1 pg_epoch: 36 pg[1.0( v 21'10 (0'0,21'10] local-lis/les=35/36 n=10 ec=17/17 lis/c=35/35 les/c/f=36/36/0 sis=35) [1,0] r=0 lpr=35 crt=21'10 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep [ 1.0: AUTO_REPAIR ] ] scrubber: _scrub_finish info stats: valid m_is_repair: 1 2024-09-15T20:33:53.065 INFO:tasks.workunit.client.0.smithi136.stdout:2024-09-15T20:33:42.655+0000 7fd322ddc640 15 osd.1 pg_epoch: 36 pg[1.0( v 21'10 (0'0,21'10] local-lis/les=35/36 n=10 ec=17/17 lis/c=35/35 les/c/f=36/36/0 sis=35) [1,0] r=0 lpr=35 crt=21'10 lcod 0'0 mlcod 0'0 active+scrubbing+deep+repair [ 1.0: AUTO_REPAIR ] mbc={255={(1+0)=1}}] scrubber: scrub_finish: 5 errors. 2 errors fixed 2024-09-15T20:33:53.065 INFO:tasks.workunit.client.0.smithi136.stdout:2024-09-15T20:33:42.655+0000 7fd322ddc640 20 osd.1 pg_epoch: 36 pg[1.0( v 21'10 (0'0,21'10] local-lis/les=35/36 n=10 ec=17/17 lis/c=35/35 les/c/f=36/36/0 sis=35) [1,0] r=0 lpr=35 crt=21'10 lcod 0'0 mlcod 0'0 active+scrubbing+deep+repair [ 1.0: AUTO_REPAIR ] mbc={255={(1+0)=1}}] scrubber: scrub_finish Current 'required': 0 Planned 'req_scrub': 0 2024-09-15T20:33:53.065 INFO:tasks.workunit.client.0.smithi136.stdout:2024-09-15T20:33:42.655+0000 7fd322ddc640 19 osd.1 pg_epoch: 36 pg[1.0( v 21'10 (0'0,21'10] local-lis/les=35/36 n=10 ec=17/17 lis/c=35/35 les/c/f=36/36/0 sis=35) [1,0] r=0 lpr=35 crt=21'10 lcod 0'0 mlcod 0'0 active+scrubbing+deep+repair [ 1.0: AUTO_REPAIR ] mbc={255={(1+0)=1}}] scrubber: scrub_finish shard 1 num_omap_bytes = 0 num_omap_keys = 0 2024-09-15T20:33:53.065 INFO:tasks.workunit.client.0.smithi136.stdout:2024-09-15T20:33:43.666+0000 7fd322ddc640 10 osd.1 pg_epoch: 36 pg[1.0( v 21'10 (0'0,21'10] local-lis/les=35/36 n=10 ec=17/17 lis/c=35/35 les/c/f=36/36/0 sis=35) [1,0] r=0 lpr=35 crt=21'10 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep+inconsistent [ 1.0: AUTO_REPAIR CHECK_REPAIR ] ] scrubber: scrub_finish before flags: AUTO_REPAIR CHECK_REPAIR. repair state: no-repair. deep_scrub_on_error: 0 2024-09-15T20:33:53.065 INFO:tasks.workunit.client.0.smithi136.stdout:2024-09-15T20:33:43.666+0000 7fd322ddc640 10 osd.1 pg_epoch: 36 pg[1.0( v 21'10 (0'0,21'10] local-lis/les=35/36 n=10 ec=17/17 lis/c=35/35 les/c/f=36/36/0 sis=35) [1,0] r=0 lpr=35 crt=21'10 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep+inconsistent [ 1.0: AUTO_REPAIR CHECK_REPAIR ] ] scrubber: _scrub_finish info stats: valid m_is_repair: 1 2024-09-15T20:33:53.065 INFO:tasks.workunit.client.0.smithi136.stdout:2024-09-15T20:33:43.667+0000 7fd322ddc640 15 osd.1 pg_epoch: 36 pg[1.0( v 21'10 (0'0,21'10] local-lis/les=35/36 n=10 ec=17/17 lis/c=35/35 les/c/f=36/36/0 sis=35) [1,0] r=0 lpr=35 crt=21'10 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep+inconsistent [ 1.0: AUTO_REPAIR CHECK_REPAIR ] ] scrubber: scrub_finish: 3 errors. 0 errors fixed 2024-09-15T20:33:53.065 INFO:tasks.workunit.client.0.smithi136.stdout:2024-09-15T20:33:43.667+0000 7fd322ddc640 10 osd.1 pg_epoch: 36 pg[1.0( v 21'10 (0'0,21'10] local-lis/les=35/36 n=10 ec=17/17 lis/c=35/35 les/c/f=36/36/0 sis=35) [1,0] r=0 lpr=35 crt=21'10 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep+inconsistent+failed_repair [ 1.0: AUTO_REPAIR CHECK_REPAIR ] ] scrubber: scrub_finish 3 error(s) present with no repair possible 2024-09-15T20:33:53.066 INFO:tasks.workunit.client.0.smithi136.stdout:2024-09-15T20:33:43.667+0000 7fd322ddc640 19 osd.1 pg_epoch: 36 pg[1.0( v 21'10 (0'0,21'10] local-lis/les=35/36 n=10 ec=17/17 lis/c=35/35 les/c/f=36/36/0 sis=35) [1,0] r=0 lpr=35 crt=21'10 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep+inconsistent+failed_repair [ 1.0: AUTO_REPAIR CHECK_REPAIR ] ] scrubber: scrub_finish shard 1 num_omap_bytes = 0 num_omap_keys = 0 2024-09-15T20:33:53.066 INFO:tasks.workunit.client.0.smithi136.stdout:2024-09-15T20:33:43.667+0000 7fd322ddc640 10 osd.1 pg_epoch: 36 pg[1.0( v 21'10 (0'0,21'10] local-lis/les=35/36 n=10 ec=17/17 lis/c=35/35 les/c/f=36/36/0 sis=35) [1,0] r=0 lpr=35 crt=21'10 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep+inconsistent+failed_repair [ 1.0: AUTO_REPAIR ] ] scrubber: scrub_finish 3 error(s) still present after re-scrub 2024-09-15T20:33:53.066 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:581: TEST_auto_repair_bluestore_failed: grep -q 'scrub_finish.*still present after re-scrub' td/osd-scrub-repair/osd.1.log 2024-09-15T20:33:53.067 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:582: TEST_auto_repair_bluestore_failed: ceph pg dump pgs 2024-09-15T20:33:53.333 INFO:tasks.workunit.client.0.smithi136.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-15T20:33:53.333 INFO:tasks.workunit.client.0.smithi136.stdout:1.0 10 0 0 0 0 63 0 0 10 0 10 active+clean+inconsistent+failed_repair 2024-09-15T20:33:43.667971+0000 21'10 36:102 [1,0] 1 [1,0] 1 21'10 2024-09-15T20:33:43.667907+0000 21'10 2024-09-15T20:33:43.667907+0000 0 1 periodic scrub scheduled @ 2024-09-16T20:33:43.667907+0000 10 0 2024-09-15T20:33:53.333 INFO:tasks.workunit.client.0.smithi136.stdout: 2024-09-15T20:33:53.333 INFO:tasks.workunit.client.0.smithi136.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-15T20:33:53.333 INFO:tasks.workunit.client.0.smithi136.stderr:dumped pgs 2024-09-15T20:33:53.344 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:583: TEST_auto_repair_bluestore_failed: ceph pg dump pgs 2024-09-15T20:33:53.344 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:583: TEST_auto_repair_bluestore_failed: grep -q '^1.0.*+failed_repair' 2024-09-15T20:33:53.602 INFO:tasks.workunit.client.0.smithi136.stderr:dumped pgs 2024-09-15T20:33:53.615 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:587: TEST_auto_repair_bluestore_failed: get_not_primary testpool obj1 2024-09-15T20:33:53.615 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1220: get_not_primary: local poolname=testpool 2024-09-15T20:33:53.615 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1221: get_not_primary: local objectname=obj1 2024-09-15T20:33:53.615 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1223: get_not_primary: get_primary testpool obj1 2024-09-15T20:33:53.615 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1187: get_primary: local poolname=testpool 2024-09-15T20:33:53.615 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1188: get_primary: local objectname=obj1 2024-09-15T20:33:53.616 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1190: get_primary: ceph --format json osd map testpool obj1 2024-09-15T20:33:53.616 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1191: get_primary: jq .acting_primary 2024-09-15T20:33:53.893 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1223: get_not_primary: local primary=1 2024-09-15T20:33:53.893 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1224: get_not_primary: ceph --format json osd map testpool obj1 2024-09-15T20:33:53.893 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1225: get_not_primary: jq '.acting | map(select (. != 1)) | .[0]' 2024-09-15T20:33:54.174 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:587: TEST_auto_repair_bluestore_failed: objectstore_tool td/osd-scrub-repair 0 obj1 list-attrs 2024-09-15T20:33:54.174 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2024-09-15T20:33:54.174 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2024-09-15T20:33:54.174 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=0 2024-09-15T20:33:54.174 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2024-09-15T20:33:54.174 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 obj1 list-attrs 2024-09-15T20:33:54.174 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-09-15T20:33:54.174 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2024-09-15T20:33:54.174 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=0 2024-09-15T20:33:54.174 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2024-09-15T20:33:54.174 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2024-09-15T20:33:54.174 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-15T20:33:54.175 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-15T20:33:54.175 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-15T20:33:54.175 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-15T20:33:54.175 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-15T20:33:54.480 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-15T20:33:54.480 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 obj1 list-attrs 2024-09-15T20:33:54.480 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-15T20:33:54.480 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-15T20:33:54.480 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=0 2024-09-15T20:33:54.481 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-15T20:33:54.481 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-09-15T20:33:54.481 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 obj1 list-attrs 2024-09-15T20:33:54.951 INFO:tasks.workunit.client.0.smithi136.stdout:_ 2024-09-15T20:33:54.951 INFO:tasks.workunit.client.0.smithi136.stdout:snapset 2024-09-15T20:33:55.219 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-09-15T20:33:55.219 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2024-09-15T20:33:55.219 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-15T20:33:55.219 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=0 2024-09-15T20:33:55.219 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-15T20:33:55.219 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-09-15T20:33:55.219 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=ce4892c7-6c6d-4b4f-8e6f-07a3e11133e5 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-15T20:33:55.219 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T20:33:55.219 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T20:33:55.219 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T20:33:55.219 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-09-15T20:33:55.219 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-09-15T20:33:55.220 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-15T20:33:55.220 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-15T20:33:55.220 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-15T20:33:55.220 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-15T20:33:55.220 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-15T20:33:55.220 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-15T20:33:55.220 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-15T20:33:55.220 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T20:33:55.220 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52602 2024-09-15T20:33:55.220 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.52602/$cluster-$name.asok' 2024-09-15T20:33:55.220 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.52602/$cluster-$name.asok' 2024-09-15T20:33:55.220 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-15T20:33:55.220 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-15T20:33:55.221 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-15T20:33:55.221 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T20:33:55.221 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T20:33:55.221 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T20:33:55.221 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T20:33:55.221 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-15T20:33:55.221 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+='--osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0' 2024-09-15T20:33:55.221 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-09-15T20:33:55.221 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.0 2024-09-15T20:33:55.221 INFO:tasks.workunit.client.0.smithi136.stderr:start osd.0 2024-09-15T20:33:55.221 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 0 --fsid=ce4892c7-6c6d-4b4f-8e6f-07a3e11133e5 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.52602/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-09-15T20:33:55.221 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-09-15T20:33:55.222 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 0 = 0 ']' 2024-09-15T20:33:55.222 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-15T20:33:55.222 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-15T20:33:55.222 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-15T20:33:55.249 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:33:55.246+0000 7f49cb887640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T20:33:55.252 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:33:55.250+0000 7f49cb887640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T20:33:55.253 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:33:55.251+0000 7f49cb887640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T20:33:55.518 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 0 2024-09-15T20:33:55.518 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-15T20:33:55.518 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2024-09-15T20:33:55.518 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-15T20:33:55.518 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-15T20:33:55.518 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:33:55.518 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-15T20:33:55.518 INFO:tasks.workunit.client.0.smithi136.stderr:0 2024-09-15T20:33:55.518 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:33:55.519 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T20:33:55.792 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:33:56.613 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:33:56.611+0000 7f49cb887640 -1 Falling back to public interface 2024-09-15T20:33:56.794 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:33:56.794 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:33:56.794 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-15T20:33:56.794 INFO:tasks.workunit.client.0.smithi136.stderr:1 2024-09-15T20:33:56.794 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:33:56.794 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T20:33:57.079 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:33:58.080 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:33:58.081 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:33:58.081 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-15T20:33:58.081 INFO:tasks.workunit.client.0.smithi136.stderr:2 2024-09-15T20:33:58.081 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:33:58.081 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T20:33:58.387 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:33:59.389 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:33:59.389 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:33:59.389 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-15T20:33:59.389 INFO:tasks.workunit.client.0.smithi136.stderr:3 2024-09-15T20:33:59.389 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:33:59.390 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T20:33:59.604 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:33:59.601+0000 7f49cb887640 -1 osd.0 36 log_to_monitors true 2024-09-15T20:33:59.698 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:34:00.700 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:34:00.700 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:34:00.700 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-15T20:34:00.700 INFO:tasks.workunit.client.0.smithi136.stderr:4 2024-09-15T20:34:00.700 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:34:00.700 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T20:34:00.990 INFO:tasks.workunit.client.0.smithi136.stderr:osd.0 up in weight 1 up_from 40 up_thru 32 down_at 37 last_clean_interval [30,36) [v2:127.0.0.1:6802/2204807572,v1:127.0.0.1:6803/2204807572] [v2:127.0.0.1:6804/2204807572,v1:127.0.0.1:6805/2204807572] exists,up 555689c8-c81e-4042-90bc-288636f201a7 2024-09-15T20:34:00.990 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-15T20:34:00.990 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-15T20:34:00.990 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-15T20:34:00.991 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2024-09-15T20:34:00.991 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-15T20:34:00.991 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-15T20:34:00.991 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-15T20:34:00.991 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-15T20:34:00.991 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-15T20:34:00.991 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-15T20:34:00.991 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-15T20:34:00.991 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-15T20:34:00.991 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-15T20:34:01.080 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-15T20:34:01.080 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-15T20:34:01.080 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-15T20:34:01.080 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-15T20:34:01.080 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-15T20:34:01.080 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-15T20:34:01.429 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-15T20:34:01.429 INFO:tasks.workunit.client.0.smithi136.stderr:1 2024-09-15T20:34:01.429 INFO:tasks.workunit.client.0.smithi136.stderr:2' 2024-09-15T20:34:01.429 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-15T20:34:01.429 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T20:34:01.429 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-15T20:34:01.531 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=171798691842 2024-09-15T20:34:01.531 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 171798691842 2024-09-15T20:34:01.531 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-171798691842' 2024-09-15T20:34:01.531 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T20:34:01.531 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-15T20:34:01.634 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=150323855370 2024-09-15T20:34:01.635 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 150323855370 2024-09-15T20:34:01.635 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-171798691842 1-150323855370' 2024-09-15T20:34:01.635 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T20:34:01.635 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-15T20:34:01.737 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=64424509464 2024-09-15T20:34:01.737 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 64424509464 2024-09-15T20:34:01.737 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-171798691842 1-150323855370 2-64424509464' 2024-09-15T20:34:01.737 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T20:34:01.737 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-171798691842 2024-09-15T20:34:01.737 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T20:34:01.738 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-15T20:34:01.739 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-171798691842 2024-09-15T20:34:01.739 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T20:34:01.740 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=171798691842 2024-09-15T20:34:01.740 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 171798691842' 2024-09-15T20:34:01.740 INFO:tasks.workunit.client.0.smithi136.stderr:waiting osd.0 seq 171798691842 2024-09-15T20:34:01.740 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T20:34:02.017 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 1 -lt 171798691842 2024-09-15T20:34:02.017 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-15T20:34:03.018 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-15T20:34:03.019 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T20:34:03.313 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 171798691842 -lt 171798691842 2024-09-15T20:34:03.313 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T20:34:03.313 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-150323855370 2024-09-15T20:34:03.313 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T20:34:03.314 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-15T20:34:03.315 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-150323855370 2024-09-15T20:34:03.315 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T20:34:03.316 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=150323855370 2024-09-15T20:34:03.316 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 150323855370' 2024-09-15T20:34:03.316 INFO:tasks.workunit.client.0.smithi136.stderr:waiting osd.1 seq 150323855370 2024-09-15T20:34:03.316 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-15T20:34:03.596 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 150323855370 -lt 150323855370 2024-09-15T20:34:03.596 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T20:34:03.596 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-64424509464 2024-09-15T20:34:03.597 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T20:34:03.597 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-15T20:34:03.598 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-64424509464 2024-09-15T20:34:03.598 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T20:34:03.599 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=64424509464 2024-09-15T20:34:03.599 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 64424509464' 2024-09-15T20:34:03.599 INFO:tasks.workunit.client.0.smithi136.stderr:waiting osd.2 seq 64424509464 2024-09-15T20:34:03.599 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-15T20:34:03.904 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 64424509464 -lt 64424509464 2024-09-15T20:34:03.905 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-15T20:34:03.905 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T20:34:03.905 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T20:34:04.247 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 1 == 0 2024-09-15T20:34:04.247 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-15T20:34:04.247 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-15T20:34:04.247 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-15T20:34:04.247 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T20:34:04.248 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T20:34:04.248 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T20:34:04.248 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T20:34:04.520 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=1 2024-09-15T20:34:04.520 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-15T20:34:04.521 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T20:34:04.521 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T20:34:04.865 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 = 1 2024-09-15T20:34:04.865 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-15T20:34:04.865 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-15T20:34:04.865 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:588: TEST_auto_repair_bluestore_failed: rados --pool testpool get obj1 td/osd-scrub-repair/COPY 2024-09-15T20:34:04.894 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:589: TEST_auto_repair_bluestore_failed: diff td/osd-scrub-repair/ORIGINAL td/osd-scrub-repair/COPY 2024-09-15T20:34:04.895 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:590: TEST_auto_repair_bluestore_failed: grep scrub_finish td/osd-scrub-repair/osd.1.log 2024-09-15T20:34:04.897 INFO:tasks.workunit.client.0.smithi136.stdout:2024-09-15T20:33:42.655+0000 7fd322ddc640 10 osd.1 pg_epoch: 36 pg[1.0( v 21'10 (0'0,21'10] local-lis/les=35/36 n=10 ec=17/17 lis/c=35/35 les/c/f=36/36/0 sis=35) [1,0] r=0 lpr=35 crt=21'10 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep [ 1.0: AUTO_REPAIR ] ] scrubber: scrub_finish before flags: AUTO_REPAIR. repair state: no-repair. deep_scrub_on_error: 0 2024-09-15T20:34:04.898 INFO:tasks.workunit.client.0.smithi136.stdout:2024-09-15T20:33:42.655+0000 7fd322ddc640 10 osd.1 pg_epoch: 36 pg[1.0( v 21'10 (0'0,21'10] local-lis/les=35/36 n=10 ec=17/17 lis/c=35/35 les/c/f=36/36/0 sis=35) [1,0] r=0 lpr=35 crt=21'10 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep [ 1.0: AUTO_REPAIR ] ] scrubber: _scrub_finish info stats: valid m_is_repair: 1 2024-09-15T20:34:04.898 INFO:tasks.workunit.client.0.smithi136.stdout:2024-09-15T20:33:42.655+0000 7fd322ddc640 15 osd.1 pg_epoch: 36 pg[1.0( v 21'10 (0'0,21'10] local-lis/les=35/36 n=10 ec=17/17 lis/c=35/35 les/c/f=36/36/0 sis=35) [1,0] r=0 lpr=35 crt=21'10 lcod 0'0 mlcod 0'0 active+scrubbing+deep+repair [ 1.0: AUTO_REPAIR ] mbc={255={(1+0)=1}}] scrubber: scrub_finish: 5 errors. 2 errors fixed 2024-09-15T20:34:04.898 INFO:tasks.workunit.client.0.smithi136.stdout:2024-09-15T20:33:42.655+0000 7fd322ddc640 20 osd.1 pg_epoch: 36 pg[1.0( v 21'10 (0'0,21'10] local-lis/les=35/36 n=10 ec=17/17 lis/c=35/35 les/c/f=36/36/0 sis=35) [1,0] r=0 lpr=35 crt=21'10 lcod 0'0 mlcod 0'0 active+scrubbing+deep+repair [ 1.0: AUTO_REPAIR ] mbc={255={(1+0)=1}}] scrubber: scrub_finish Current 'required': 0 Planned 'req_scrub': 0 2024-09-15T20:34:04.898 INFO:tasks.workunit.client.0.smithi136.stdout:2024-09-15T20:33:42.655+0000 7fd322ddc640 19 osd.1 pg_epoch: 36 pg[1.0( v 21'10 (0'0,21'10] local-lis/les=35/36 n=10 ec=17/17 lis/c=35/35 les/c/f=36/36/0 sis=35) [1,0] r=0 lpr=35 crt=21'10 lcod 0'0 mlcod 0'0 active+scrubbing+deep+repair [ 1.0: AUTO_REPAIR ] mbc={255={(1+0)=1}}] scrubber: scrub_finish shard 1 num_omap_bytes = 0 num_omap_keys = 0 2024-09-15T20:34:04.898 INFO:tasks.workunit.client.0.smithi136.stdout:2024-09-15T20:33:43.666+0000 7fd322ddc640 10 osd.1 pg_epoch: 36 pg[1.0( v 21'10 (0'0,21'10] local-lis/les=35/36 n=10 ec=17/17 lis/c=35/35 les/c/f=36/36/0 sis=35) [1,0] r=0 lpr=35 crt=21'10 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep+inconsistent [ 1.0: AUTO_REPAIR CHECK_REPAIR ] ] scrubber: scrub_finish before flags: AUTO_REPAIR CHECK_REPAIR. repair state: no-repair. deep_scrub_on_error: 0 2024-09-15T20:34:04.898 INFO:tasks.workunit.client.0.smithi136.stdout:2024-09-15T20:33:43.666+0000 7fd322ddc640 10 osd.1 pg_epoch: 36 pg[1.0( v 21'10 (0'0,21'10] local-lis/les=35/36 n=10 ec=17/17 lis/c=35/35 les/c/f=36/36/0 sis=35) [1,0] r=0 lpr=35 crt=21'10 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep+inconsistent [ 1.0: AUTO_REPAIR CHECK_REPAIR ] ] scrubber: _scrub_finish info stats: valid m_is_repair: 1 2024-09-15T20:34:04.898 INFO:tasks.workunit.client.0.smithi136.stdout:2024-09-15T20:33:43.667+0000 7fd322ddc640 15 osd.1 pg_epoch: 36 pg[1.0( v 21'10 (0'0,21'10] local-lis/les=35/36 n=10 ec=17/17 lis/c=35/35 les/c/f=36/36/0 sis=35) [1,0] r=0 lpr=35 crt=21'10 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep+inconsistent [ 1.0: AUTO_REPAIR CHECK_REPAIR ] ] scrubber: scrub_finish: 3 errors. 0 errors fixed 2024-09-15T20:34:04.898 INFO:tasks.workunit.client.0.smithi136.stdout:2024-09-15T20:33:43.667+0000 7fd322ddc640 10 osd.1 pg_epoch: 36 pg[1.0( v 21'10 (0'0,21'10] local-lis/les=35/36 n=10 ec=17/17 lis/c=35/35 les/c/f=36/36/0 sis=35) [1,0] r=0 lpr=35 crt=21'10 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep+inconsistent+failed_repair [ 1.0: AUTO_REPAIR CHECK_REPAIR ] ] scrubber: scrub_finish 3 error(s) present with no repair possible 2024-09-15T20:34:04.898 INFO:tasks.workunit.client.0.smithi136.stdout:2024-09-15T20:33:43.667+0000 7fd322ddc640 19 osd.1 pg_epoch: 36 pg[1.0( v 21'10 (0'0,21'10] local-lis/les=35/36 n=10 ec=17/17 lis/c=35/35 les/c/f=36/36/0 sis=35) [1,0] r=0 lpr=35 crt=21'10 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep+inconsistent+failed_repair [ 1.0: AUTO_REPAIR CHECK_REPAIR ] ] scrubber: scrub_finish shard 1 num_omap_bytes = 0 num_omap_keys = 0 2024-09-15T20:34:04.898 INFO:tasks.workunit.client.0.smithi136.stdout:2024-09-15T20:33:43.667+0000 7fd322ddc640 10 osd.1 pg_epoch: 36 pg[1.0( v 21'10 (0'0,21'10] local-lis/les=35/36 n=10 ec=17/17 lis/c=35/35 les/c/f=36/36/0 sis=35) [1,0] r=0 lpr=35 crt=21'10 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep+inconsistent+failed_repair [ 1.0: AUTO_REPAIR ] ] scrubber: scrub_finish 3 error(s) still present after re-scrub 2024-09-15T20:34:04.899 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:593: TEST_auto_repair_bluestore_failed: get_primary testpool SOMETHING 2024-09-15T20:34:04.899 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1187: get_primary: local poolname=testpool 2024-09-15T20:34:04.899 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1188: get_primary: local objectname=SOMETHING 2024-09-15T20:34:04.899 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1190: get_primary: ceph --format json osd map testpool SOMETHING 2024-09-15T20:34:04.899 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1191: get_primary: jq .acting_primary 2024-09-15T20:34:05.190 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:593: TEST_auto_repair_bluestore_failed: objectstore_tool td/osd-scrub-repair 1 obj2 remove 2024-09-15T20:34:05.190 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2024-09-15T20:34:05.190 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2024-09-15T20:34:05.190 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=1 2024-09-15T20:34:05.190 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2024-09-15T20:34:05.190 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 obj2 remove 2024-09-15T20:34:05.190 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-09-15T20:34:05.191 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2024-09-15T20:34:05.191 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=1 2024-09-15T20:34:05.191 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2024-09-15T20:34:05.191 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-09-15T20:34:05.191 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-15T20:34:05.191 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-15T20:34:05.191 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-15T20:34:05.191 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-15T20:34:05.191 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-15T20:34:05.496 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-15T20:34:05.497 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 obj2 remove 2024-09-15T20:34:05.497 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-15T20:34:05.497 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-15T20:34:05.497 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=1 2024-09-15T20:34:05.497 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-15T20:34:05.497 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-09-15T20:34:05.497 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 obj2 remove 2024-09-15T20:34:06.804 INFO:tasks.workunit.client.0.smithi136.stderr:Error getting attr on : 1.0_head,#1:104778fc:::obj2:head#, (61) No data available 2024-09-15T20:34:06.804 INFO:tasks.workunit.client.0.smithi136.stdout:remove #1:104778fc:::obj2:head# 2024-09-15T20:34:07.323 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-09-15T20:34:07.323 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2024-09-15T20:34:07.323 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-15T20:34:07.323 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=1 2024-09-15T20:34:07.323 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-15T20:34:07.324 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-09-15T20:34:07.324 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=ce4892c7-6c6d-4b4f-8e6f-07a3e11133e5 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-15T20:34:07.324 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T20:34:07.324 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T20:34:07.324 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T20:34:07.324 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-09-15T20:34:07.324 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-09-15T20:34:07.324 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-15T20:34:07.324 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-15T20:34:07.324 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-15T20:34:07.324 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-15T20:34:07.325 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-15T20:34:07.325 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-15T20:34:07.325 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-15T20:34:07.325 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T20:34:07.325 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52602 2024-09-15T20:34:07.325 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.52602/$cluster-$name.asok' 2024-09-15T20:34:07.326 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.52602/$cluster-$name.asok' 2024-09-15T20:34:07.326 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-15T20:34:07.326 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-15T20:34:07.326 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-15T20:34:07.326 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T20:34:07.326 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T20:34:07.326 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T20:34:07.326 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T20:34:07.326 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-15T20:34:07.326 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+='--osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0' 2024-09-15T20:34:07.326 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-09-15T20:34:07.327 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.1 2024-09-15T20:34:07.327 INFO:tasks.workunit.client.0.smithi136.stderr:start osd.1 2024-09-15T20:34:07.327 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 1 --fsid=ce4892c7-6c6d-4b4f-8e6f-07a3e11133e5 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.52602/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-09-15T20:34:07.327 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-09-15T20:34:07.328 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 1 = 1 ']' 2024-09-15T20:34:07.328 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-15T20:34:07.328 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-15T20:34:07.328 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-15T20:34:07.357 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:34:07.354+0000 7ff3ffc14640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T20:34:07.390 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:34:07.388+0000 7ff3ffc14640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T20:34:07.391 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:34:07.389+0000 7ff3ffc14640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T20:34:07.609 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 1 2024-09-15T20:34:07.609 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-15T20:34:07.609 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2024-09-15T20:34:07.609 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-15T20:34:07.609 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-15T20:34:07.610 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:34:07.610 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-15T20:34:07.610 INFO:tasks.workunit.client.0.smithi136.stderr:0 2024-09-15T20:34:07.610 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:34:07.610 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T20:34:07.886 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:34:08.737 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:34:08.735+0000 7ff3ffc14640 -1 Falling back to public interface 2024-09-15T20:34:08.888 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:34:08.888 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:34:08.888 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-15T20:34:08.888 INFO:tasks.workunit.client.0.smithi136.stderr:1 2024-09-15T20:34:08.888 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:34:08.888 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T20:34:09.169 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:34:10.171 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:34:10.171 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:34:10.171 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-15T20:34:10.171 INFO:tasks.workunit.client.0.smithi136.stderr:2 2024-09-15T20:34:10.171 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:34:10.171 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T20:34:10.456 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:34:11.457 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:34:11.457 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:34:11.458 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-15T20:34:11.458 INFO:tasks.workunit.client.0.smithi136.stderr:3 2024-09-15T20:34:11.458 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:34:11.458 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T20:34:11.486 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:34:11.484+0000 7ff3ffc14640 -1 osd.1 41 log_to_monitors true 2024-09-15T20:34:11.755 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:34:12.757 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:34:12.757 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:34:12.757 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-15T20:34:12.757 INFO:tasks.workunit.client.0.smithi136.stderr:4 2024-09-15T20:34:12.757 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:34:12.757 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T20:34:13.045 INFO:tasks.workunit.client.0.smithi136.stderr:osd.1 up in weight 1 up_from 45 up_thru 45 down_at 42 last_clean_interval [35,41) [v2:127.0.0.1:6810/3544404086,v1:127.0.0.1:6811/3544404086] [v2:127.0.0.1:6812/3544404086,v1:127.0.0.1:6813/3544404086] exists,up a91f9a5c-e300-4dd1-9666-c0e98286384c 2024-09-15T20:34:13.045 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-15T20:34:13.045 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-15T20:34:13.045 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-15T20:34:13.045 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2024-09-15T20:34:13.045 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-15T20:34:13.045 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-15T20:34:13.045 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-15T20:34:13.045 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-15T20:34:13.045 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-15T20:34:13.046 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-15T20:34:13.046 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-15T20:34:13.046 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-15T20:34:13.046 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-15T20:34:13.135 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-15T20:34:13.135 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-15T20:34:13.135 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-15T20:34:13.135 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-15T20:34:13.135 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-15T20:34:13.136 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-15T20:34:13.414 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-15T20:34:13.414 INFO:tasks.workunit.client.0.smithi136.stderr:1 2024-09-15T20:34:13.414 INFO:tasks.workunit.client.0.smithi136.stderr:2' 2024-09-15T20:34:13.414 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-15T20:34:13.414 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T20:34:13.414 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-15T20:34:13.536 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=171798691845 2024-09-15T20:34:13.536 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 171798691845 2024-09-15T20:34:13.536 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-171798691845' 2024-09-15T20:34:13.536 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T20:34:13.537 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-15T20:34:13.646 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=193273528322 2024-09-15T20:34:13.646 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 193273528322 2024-09-15T20:34:13.646 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-171798691845 1-193273528322' 2024-09-15T20:34:13.647 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T20:34:13.647 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-15T20:34:13.749 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=64424509468 2024-09-15T20:34:13.749 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 64424509468 2024-09-15T20:34:13.749 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-171798691845 1-193273528322 2-64424509468' 2024-09-15T20:34:13.749 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T20:34:13.749 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-171798691845 2024-09-15T20:34:13.750 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T20:34:13.750 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-15T20:34:13.751 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-171798691845 2024-09-15T20:34:13.751 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T20:34:13.752 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=171798691845 2024-09-15T20:34:13.752 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 171798691845' 2024-09-15T20:34:13.752 INFO:tasks.workunit.client.0.smithi136.stderr:waiting osd.0 seq 171798691845 2024-09-15T20:34:13.752 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T20:34:14.028 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 171798691844 -lt 171798691845 2024-09-15T20:34:14.028 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-15T20:34:15.029 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-15T20:34:15.030 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T20:34:15.322 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 171798691846 -lt 171798691845 2024-09-15T20:34:15.322 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T20:34:15.323 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-193273528322 2024-09-15T20:34:15.323 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T20:34:15.323 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-15T20:34:15.324 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-193273528322 2024-09-15T20:34:15.324 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T20:34:15.325 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=193273528322 2024-09-15T20:34:15.325 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 193273528322' 2024-09-15T20:34:15.325 INFO:tasks.workunit.client.0.smithi136.stderr:waiting osd.1 seq 193273528322 2024-09-15T20:34:15.325 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-15T20:34:15.602 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 193273528322 -lt 193273528322 2024-09-15T20:34:15.602 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T20:34:15.603 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-64424509468 2024-09-15T20:34:15.603 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T20:34:15.603 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-15T20:34:15.604 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-64424509468 2024-09-15T20:34:15.604 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T20:34:15.605 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=64424509468 2024-09-15T20:34:15.605 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 64424509468' 2024-09-15T20:34:15.605 INFO:tasks.workunit.client.0.smithi136.stderr:waiting osd.2 seq 64424509468 2024-09-15T20:34:15.605 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-15T20:34:15.905 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 64424509468 -lt 64424509468 2024-09-15T20:34:15.905 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-15T20:34:15.905 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T20:34:15.905 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T20:34:16.252 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 1 == 0 2024-09-15T20:34:16.252 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-15T20:34:16.253 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-15T20:34:16.253 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-15T20:34:16.253 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T20:34:16.253 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T20:34:16.253 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T20:34:16.253 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T20:34:16.526 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=1 2024-09-15T20:34:16.527 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-15T20:34:16.527 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T20:34:16.527 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T20:34:16.869 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 = 1 2024-09-15T20:34:16.869 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-15T20:34:16.869 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-15T20:34:16.869 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:594: TEST_auto_repair_bluestore_failed: repair 1.0 2024-09-15T20:34:16.869 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1846: repair: local pgid=1.0 2024-09-15T20:34:16.869 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1847: repair: get_last_scrub_stamp 1.0 2024-09-15T20:34:16.869 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-15T20:34:16.869 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-15T20:34:16.869 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-15T20:34:16.870 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-15T20:34:17.144 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1847: repair: local last_scrub=2024-09-15T20:33:43.667907+0000 2024-09-15T20:34:17.144 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1848: repair: ceph pg repair 1.0 2024-09-15T20:34:17.417 INFO:tasks.workunit.client.0.smithi136.stderr:instructing pg 1.0 on osd.1 to repair 2024-09-15T20:34:17.429 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1849: repair: wait_for_scrub 1.0 2024-09-15T20:33:43.667907+0000 2024-09-15T20:34:17.429 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: local pgid=1.0 2024-09-15T20:34:17.429 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2018: wait_for_scrub: local last_scrub=2024-09-15T20:33:43.667907+0000 2024-09-15T20:34:17.429 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2019: wait_for_scrub: local sname=last_scrub_stamp 2024-09-15T20:34:17.429 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i=0 )) 2024-09-15T20:34:17.429 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-15T20:34:17.429 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-15T20:34:17.429 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-15T20:34:17.430 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-15T20:34:17.430 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-15T20:34:17.430 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-15T20:34:17.699 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-15T20:33:43.667907+0000 '>' 2024-09-15T20:33:43.667907+0000 2024-09-15T20:34:17.699 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-15T20:34:18.700 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-15T20:34:18.700 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-15T20:34:18.701 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-15T20:34:18.701 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-15T20:34:18.701 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-15T20:34:18.701 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-15T20:34:18.701 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-15T20:34:18.988 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-15T20:33:43.667907+0000 '>' 2024-09-15T20:33:43.667907+0000 2024-09-15T20:34:18.988 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-15T20:34:19.989 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-15T20:34:19.989 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-15T20:34:19.990 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-15T20:34:19.990 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-15T20:34:19.990 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-15T20:34:19.990 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-15T20:34:19.990 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-15T20:34:20.273 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-15T20:33:43.667907+0000 '>' 2024-09-15T20:33:43.667907+0000 2024-09-15T20:34:20.273 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-15T20:34:21.274 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-15T20:34:21.274 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-15T20:34:21.275 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-15T20:34:21.275 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-15T20:34:21.275 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-15T20:34:21.275 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-15T20:34:21.275 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-15T20:34:21.584 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-15T20:33:43.667907+0000 '>' 2024-09-15T20:33:43.667907+0000 2024-09-15T20:34:21.584 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-15T20:34:22.586 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-15T20:34:22.587 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-15T20:34:22.587 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-15T20:34:22.587 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-15T20:34:22.587 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-15T20:34:22.587 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-15T20:34:22.587 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-15T20:34:22.869 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-15T20:33:43.667907+0000 '>' 2024-09-15T20:33:43.667907+0000 2024-09-15T20:34:22.869 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-15T20:34:23.870 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-15T20:34:23.871 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-15T20:34:23.871 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-15T20:34:23.871 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-15T20:34:23.872 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-15T20:34:23.872 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-15T20:34:23.872 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-15T20:34:24.156 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-15T20:34:18.435874+0000 '>' 2024-09-15T20:33:43.667907+0000 2024-09-15T20:34:24.156 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2023: wait_for_scrub: return 0 2024-09-15T20:34:24.156 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1850: repair: sleep 2 2024-09-15T20:34:26.157 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:595: TEST_auto_repair_bluestore_failed: sleep 2 2024-09-15T20:34:28.158 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:597: TEST_auto_repair_bluestore_failed: flush_pg_stats 2024-09-15T20:34:28.158 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-15T20:34:28.159 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-15T20:34:28.449 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-15T20:34:28.449 INFO:tasks.workunit.client.0.smithi136.stderr:1 2024-09-15T20:34:28.449 INFO:tasks.workunit.client.0.smithi136.stderr:2' 2024-09-15T20:34:28.449 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-15T20:34:28.449 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T20:34:28.450 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-15T20:34:28.551 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=171798691849 2024-09-15T20:34:28.551 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 171798691849 2024-09-15T20:34:28.551 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-171798691849' 2024-09-15T20:34:28.551 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T20:34:28.552 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-15T20:34:28.655 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=193273528326 2024-09-15T20:34:28.656 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 193273528326 2024-09-15T20:34:28.656 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-171798691849 1-193273528326' 2024-09-15T20:34:28.656 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T20:34:28.656 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-15T20:34:28.758 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=64424509472 2024-09-15T20:34:28.758 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 64424509472 2024-09-15T20:34:28.758 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-171798691849 1-193273528326 2-64424509472' 2024-09-15T20:34:28.758 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T20:34:28.759 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-171798691849 2024-09-15T20:34:28.759 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T20:34:28.759 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-15T20:34:28.760 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-171798691849 2024-09-15T20:34:28.760 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T20:34:28.761 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=171798691849 2024-09-15T20:34:28.761 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 171798691849' 2024-09-15T20:34:28.761 INFO:tasks.workunit.client.0.smithi136.stdout:waiting osd.0 seq 171798691849 2024-09-15T20:34:28.761 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T20:34:29.035 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 171798691848 -lt 171798691849 2024-09-15T20:34:29.036 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-15T20:34:30.037 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-15T20:34:30.037 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T20:34:30.317 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 171798691849 -lt 171798691849 2024-09-15T20:34:30.317 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T20:34:30.318 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-193273528326 2024-09-15T20:34:30.318 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T20:34:30.319 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-15T20:34:30.319 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-193273528326 2024-09-15T20:34:30.319 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T20:34:30.320 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=193273528326 2024-09-15T20:34:30.321 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 193273528326' 2024-09-15T20:34:30.321 INFO:tasks.workunit.client.0.smithi136.stdout:waiting osd.1 seq 193273528326 2024-09-15T20:34:30.321 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-15T20:34:30.601 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 193273528326 -lt 193273528326 2024-09-15T20:34:30.601 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T20:34:30.601 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-64424509472 2024-09-15T20:34:30.602 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T20:34:30.602 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-15T20:34:30.603 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-64424509472 2024-09-15T20:34:30.603 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T20:34:30.603 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=64424509472 2024-09-15T20:34:30.604 INFO:tasks.workunit.client.0.smithi136.stdout:waiting osd.2 seq 64424509472 2024-09-15T20:34:30.604 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 64424509472' 2024-09-15T20:34:30.604 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-15T20:34:30.878 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 64424509472 -lt 64424509472 2024-09-15T20:34:30.878 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:598: TEST_auto_repair_bluestore_failed: ceph pg dump pgs 2024-09-15T20:34:31.135 INFO:tasks.workunit.client.0.smithi136.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-15T20:34:31.135 INFO:tasks.workunit.client.0.smithi136.stdout:1.0 9 0 0 0 0 63 0 0 10 0 10 active+clean 2024-09-15T20:34:18.435947+0000 21'10 46:142 [1,0] 1 [1,0] 1 21'10 2024-09-15T20:34:18.435874+0000 21'10 2024-09-15T20:34:18.435874+0000 0 1 periodic scrub scheduled @ 2024-09-16T20:34:18.435874+0000 9 0 2024-09-15T20:34:31.135 INFO:tasks.workunit.client.0.smithi136.stdout: 2024-09-15T20:34:31.136 INFO:tasks.workunit.client.0.smithi136.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-15T20:34:31.136 INFO:tasks.workunit.client.0.smithi136.stderr:dumped pgs 2024-09-15T20:34:31.147 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:599: TEST_auto_repair_bluestore_failed: ceph pg dump pgs 2024-09-15T20:34:31.147 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:599: TEST_auto_repair_bluestore_failed: grep -q -e '^1.0.* active+clean ' -e '^1.0.* active+clean+wait ' 2024-09-15T20:34:31.425 INFO:tasks.workunit.client.0.smithi136.stderr:dumped pgs 2024-09-15T20:34:31.437 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:600: TEST_auto_repair_bluestore_failed: grep scrub_finish td/osd-scrub-repair/osd.1.log 2024-09-15T20:34:31.439 INFO:tasks.workunit.client.0.smithi136.stdout:2024-09-15T20:33:42.655+0000 7fd322ddc640 10 osd.1 pg_epoch: 36 pg[1.0( v 21'10 (0'0,21'10] local-lis/les=35/36 n=10 ec=17/17 lis/c=35/35 les/c/f=36/36/0 sis=35) [1,0] r=0 lpr=35 crt=21'10 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep [ 1.0: AUTO_REPAIR ] ] scrubber: scrub_finish before flags: AUTO_REPAIR. repair state: no-repair. deep_scrub_on_error: 0 2024-09-15T20:34:31.439 INFO:tasks.workunit.client.0.smithi136.stdout:2024-09-15T20:33:42.655+0000 7fd322ddc640 10 osd.1 pg_epoch: 36 pg[1.0( v 21'10 (0'0,21'10] local-lis/les=35/36 n=10 ec=17/17 lis/c=35/35 les/c/f=36/36/0 sis=35) [1,0] r=0 lpr=35 crt=21'10 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep [ 1.0: AUTO_REPAIR ] ] scrubber: _scrub_finish info stats: valid m_is_repair: 1 2024-09-15T20:34:31.439 INFO:tasks.workunit.client.0.smithi136.stdout:2024-09-15T20:33:42.655+0000 7fd322ddc640 15 osd.1 pg_epoch: 36 pg[1.0( v 21'10 (0'0,21'10] local-lis/les=35/36 n=10 ec=17/17 lis/c=35/35 les/c/f=36/36/0 sis=35) [1,0] r=0 lpr=35 crt=21'10 lcod 0'0 mlcod 0'0 active+scrubbing+deep+repair [ 1.0: AUTO_REPAIR ] mbc={255={(1+0)=1}}] scrubber: scrub_finish: 5 errors. 2 errors fixed 2024-09-15T20:34:31.439 INFO:tasks.workunit.client.0.smithi136.stdout:2024-09-15T20:33:42.655+0000 7fd322ddc640 20 osd.1 pg_epoch: 36 pg[1.0( v 21'10 (0'0,21'10] local-lis/les=35/36 n=10 ec=17/17 lis/c=35/35 les/c/f=36/36/0 sis=35) [1,0] r=0 lpr=35 crt=21'10 lcod 0'0 mlcod 0'0 active+scrubbing+deep+repair [ 1.0: AUTO_REPAIR ] mbc={255={(1+0)=1}}] scrubber: scrub_finish Current 'required': 0 Planned 'req_scrub': 0 2024-09-15T20:34:31.439 INFO:tasks.workunit.client.0.smithi136.stdout:2024-09-15T20:33:42.655+0000 7fd322ddc640 19 osd.1 pg_epoch: 36 pg[1.0( v 21'10 (0'0,21'10] local-lis/les=35/36 n=10 ec=17/17 lis/c=35/35 les/c/f=36/36/0 sis=35) [1,0] r=0 lpr=35 crt=21'10 lcod 0'0 mlcod 0'0 active+scrubbing+deep+repair [ 1.0: AUTO_REPAIR ] mbc={255={(1+0)=1}}] scrubber: scrub_finish shard 1 num_omap_bytes = 0 num_omap_keys = 0 2024-09-15T20:34:31.440 INFO:tasks.workunit.client.0.smithi136.stdout:2024-09-15T20:33:43.666+0000 7fd322ddc640 10 osd.1 pg_epoch: 36 pg[1.0( v 21'10 (0'0,21'10] local-lis/les=35/36 n=10 ec=17/17 lis/c=35/35 les/c/f=36/36/0 sis=35) [1,0] r=0 lpr=35 crt=21'10 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep+inconsistent [ 1.0: AUTO_REPAIR CHECK_REPAIR ] ] scrubber: scrub_finish before flags: AUTO_REPAIR CHECK_REPAIR. repair state: no-repair. deep_scrub_on_error: 0 2024-09-15T20:34:31.440 INFO:tasks.workunit.client.0.smithi136.stdout:2024-09-15T20:33:43.666+0000 7fd322ddc640 10 osd.1 pg_epoch: 36 pg[1.0( v 21'10 (0'0,21'10] local-lis/les=35/36 n=10 ec=17/17 lis/c=35/35 les/c/f=36/36/0 sis=35) [1,0] r=0 lpr=35 crt=21'10 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep+inconsistent [ 1.0: AUTO_REPAIR CHECK_REPAIR ] ] scrubber: _scrub_finish info stats: valid m_is_repair: 1 2024-09-15T20:34:31.440 INFO:tasks.workunit.client.0.smithi136.stdout:2024-09-15T20:33:43.667+0000 7fd322ddc640 15 osd.1 pg_epoch: 36 pg[1.0( v 21'10 (0'0,21'10] local-lis/les=35/36 n=10 ec=17/17 lis/c=35/35 les/c/f=36/36/0 sis=35) [1,0] r=0 lpr=35 crt=21'10 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep+inconsistent [ 1.0: AUTO_REPAIR CHECK_REPAIR ] ] scrubber: scrub_finish: 3 errors. 0 errors fixed 2024-09-15T20:34:31.440 INFO:tasks.workunit.client.0.smithi136.stdout:2024-09-15T20:33:43.667+0000 7fd322ddc640 10 osd.1 pg_epoch: 36 pg[1.0( v 21'10 (0'0,21'10] local-lis/les=35/36 n=10 ec=17/17 lis/c=35/35 les/c/f=36/36/0 sis=35) [1,0] r=0 lpr=35 crt=21'10 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep+inconsistent+failed_repair [ 1.0: AUTO_REPAIR CHECK_REPAIR ] ] scrubber: scrub_finish 3 error(s) present with no repair possible 2024-09-15T20:34:31.440 INFO:tasks.workunit.client.0.smithi136.stdout:2024-09-15T20:33:43.667+0000 7fd322ddc640 19 osd.1 pg_epoch: 36 pg[1.0( v 21'10 (0'0,21'10] local-lis/les=35/36 n=10 ec=17/17 lis/c=35/35 les/c/f=36/36/0 sis=35) [1,0] r=0 lpr=35 crt=21'10 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep+inconsistent+failed_repair [ 1.0: AUTO_REPAIR CHECK_REPAIR ] ] scrubber: scrub_finish shard 1 num_omap_bytes = 0 num_omap_keys = 0 2024-09-15T20:34:31.440 INFO:tasks.workunit.client.0.smithi136.stdout:2024-09-15T20:33:43.667+0000 7fd322ddc640 10 osd.1 pg_epoch: 36 pg[1.0( v 21'10 (0'0,21'10] local-lis/les=35/36 n=10 ec=17/17 lis/c=35/35 les/c/f=36/36/0 sis=35) [1,0] r=0 lpr=35 crt=21'10 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep+inconsistent+failed_repair [ 1.0: AUTO_REPAIR ] ] scrubber: scrub_finish 3 error(s) still present after re-scrub 2024-09-15T20:34:31.441 INFO:tasks.workunit.client.0.smithi136.stdout:2024-09-15T20:34:18.434+0000 7ff3e3482640 10 osd.1 pg_epoch: 46 pg[1.0( v 21'10 (0'0,21'10] local-lis/les=45/46 n=10 ec=17/17 lis/c=45/45 les/c/f=46/46/0 sis=45) [1,0] r=0 lpr=45 crt=21'10 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep+repair [ 1.0: AUTO_REPAIR REQ_SCRUB ] MUST_REPAIR MUST_DEEP_SCRUB MUST_SCRUB planned REQ_SCRUB] scrubber: scrub_finish before flags: AUTO_REPAIR REQ_SCRUB. repair state: repair. deep_scrub_on_error: 0 2024-09-15T20:34:31.441 INFO:tasks.workunit.client.0.smithi136.stdout:2024-09-15T20:34:18.434+0000 7ff3e3482640 10 osd.1 pg_epoch: 46 pg[1.0( v 21'10 (0'0,21'10] local-lis/les=45/46 n=10 ec=17/17 lis/c=45/45 les/c/f=46/46/0 sis=45) [1,0] r=0 lpr=45 crt=21'10 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep+repair [ 1.0: AUTO_REPAIR REQ_SCRUB ] ] scrubber: _scrub_finish info stats: valid m_is_repair: 1 2024-09-15T20:34:31.441 INFO:tasks.workunit.client.0.smithi136.stdout:2024-09-15T20:34:18.434+0000 7ff3e3482640 15 osd.1 pg_epoch: 46 pg[1.0( v 21'10 (0'0,21'10] local-lis/les=45/46 n=9 ec=17/17 lis/c=45/45 les/c/f=46/46/0 sis=45) [1,0] r=0 lpr=45 crt=21'10 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep+repair [ 1.0: AUTO_REPAIR REQ_SCRUB ] ] scrubber: scrub_finish: 1 errors. 1 errors fixed 2024-09-15T20:34:31.441 INFO:tasks.workunit.client.0.smithi136.stdout:2024-09-15T20:34:18.434+0000 7ff3e3482640 20 osd.1 pg_epoch: 46 pg[1.0( v 21'10 (0'0,21'10] local-lis/les=45/46 n=9 ec=17/17 lis/c=45/45 les/c/f=46/46/0 sis=45) [1,0] r=0 lpr=45 crt=21'10 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep+repair [ 1.0: AUTO_REPAIR REQ_SCRUB ] ] scrubber: scrub_finish All may be fixed 2024-09-15T20:34:31.441 INFO:tasks.workunit.client.0.smithi136.stdout:2024-09-15T20:34:18.434+0000 7ff3e3482640 19 osd.1 pg_epoch: 46 pg[1.0( v 21'10 (0'0,21'10] local-lis/les=45/46 n=9 ec=17/17 lis/c=45/45 les/c/f=46/46/0 sis=45) [1,0] r=0 lpr=45 crt=21'10 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep+repair [ 1.0: AUTO_REPAIR REQ_SCRUB ] ] scrubber: scrub_finish shard 1 num_omap_bytes = 0 num_omap_keys = 0 2024-09-15T20:34:31.442 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:66: run: teardown td/osd-scrub-repair 2024-09-15T20:34:31.442 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/osd-scrub-repair 2024-09-15T20:34:31.442 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-15T20:34:31.442 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/osd-scrub-repair KILL 2024-09-15T20:34:31.442 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-15T20:34:31.442 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-15T20:34:31.442 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-15T20:34:31.442 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-15T20:34:31.442 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-15T20:34:31.548 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-15T20:34:31.548 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-15T20:34:31.549 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-15T20:34:31.549 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-15T20:34:31.551 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-15T20:34:31.551 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-15T20:34:31.551 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-15T20:34:31.552 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T20:34:31.552 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-15T20:34:31.552 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-15T20:34:31.552 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T20:34:31.553 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-15T20:34:31.554 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-15T20:34:31.554 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/osd-scrub-repair 2024-09-15T20:34:31.572 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-15T20:34:31.572 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T20:34:31.572 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52602 2024-09-15T20:34:31.572 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.52602 2024-09-15T20:34:31.574 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-15T20:34:31.574 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-15T20:34:31.574 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:63: run: for func in $funcs 2024-09-15T20:34:31.574 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:64: run: setup td/osd-scrub-repair 2024-09-15T20:34:31.574 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/osd-scrub-repair 2024-09-15T20:34:31.575 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/osd-scrub-repair 2024-09-15T20:34:31.575 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/osd-scrub-repair 2024-09-15T20:34:31.575 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-15T20:34:31.575 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/osd-scrub-repair KILL 2024-09-15T20:34:31.575 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-15T20:34:31.575 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-15T20:34:31.575 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-15T20:34:31.575 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-15T20:34:31.575 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-15T20:34:31.576 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-15T20:34:31.576 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-15T20:34:31.576 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-15T20:34:31.577 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-15T20:34:31.578 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-15T20:34:31.578 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-15T20:34:31.578 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-15T20:34:31.579 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T20:34:31.579 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-15T20:34:31.580 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-15T20:34:31.580 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T20:34:31.580 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-15T20:34:31.581 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-15T20:34:31.581 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/osd-scrub-repair 2024-09-15T20:34:31.582 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-15T20:34:31.583 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T20:34:31.583 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52602 2024-09-15T20:34:31.583 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.52602 2024-09-15T20:34:31.583 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-15T20:34:31.584 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-15T20:34:31.584 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/osd-scrub-repair 2024-09-15T20:34:31.585 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2024-09-15T20:34:31.585 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T20:34:31.585 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52602 2024-09-15T20:34:31.585 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.52602 2024-09-15T20:34:31.588 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2024-09-15T20:34:31.588 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 4096 -le 1024 ']' 2024-09-15T20:34:31.588 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2024-09-15T20:34:31.588 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/osd-scrub-repair 1' TERM HUP INT 2024-09-15T20:34:31.588 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:65: run: TEST_auto_repair_bluestore_failed_norecov td/osd-scrub-repair 2024-09-15T20:34:31.589 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:604: TEST_auto_repair_bluestore_failed_norecov: local dir=td/osd-scrub-repair 2024-09-15T20:34:31.589 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:605: TEST_auto_repair_bluestore_failed_norecov: local poolname=testpool 2024-09-15T20:34:31.589 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:608: TEST_auto_repair_bluestore_failed_norecov: run_mon td/osd-scrub-repair a 2024-09-15T20:34:31.589 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/osd-scrub-repair 2024-09-15T20:34:31.590 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2024-09-15T20:34:31.590 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2024-09-15T20:34:31.590 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2024-09-15T20:34:31.590 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/osd-scrub-repair/a 2024-09-15T20:34:31.590 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-scrub-repair/a --run-dir=td/osd-scrub-repair 2024-09-15T20:34:32.083 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2024-09-15T20:34:32.083 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-15T20:34:32.084 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-15T20:34:32.084 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-15T20:34:32.084 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T20:34:32.084 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52602 2024-09-15T20:34:32.084 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.52602/$cluster-$name.asok' 2024-09-15T20:34:32.084 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-scrub-repair/a '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.52602/$cluster-$name.asok' --mon-cluster-log-file=td/osd-scrub-repair/log --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-09-15T20:34:32.275 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2024-09-15T20:34:32.276 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2024-09-15T20:34:32.276 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-15T20:34:32.276 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-15T20:34:32.276 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2024-09-15T20:34:32.276 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2024-09-15T20:34:32.277 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-15T20:34:32.277 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-15T20:34:32.277 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-15T20:34:32.277 INFO:tasks.workunit.client.0.smithi136.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-15T20:34:32.277 INFO:tasks.workunit.client.0.smithi136.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T20:34:32.277 INFO:tasks.workunit.client.0.smithi136.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52602 2024-09-15T20:34:32.278 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.52602/ceph-mon.a.asok 2024-09-15T20:34:32.278 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-15T20:34:32.278 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.52602/ceph-mon.a.asok config get fsid 2024-09-15T20:34:32.355 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2024-09-15T20:34:32.355 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-15T20:34:32.355 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-15T20:34:32.355 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2024-09-15T20:34:32.355 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2024-09-15T20:34:32.356 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-15T20:34:32.356 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-15T20:34:32.356 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-15T20:34:32.356 INFO:tasks.workunit.client.0.smithi136.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-15T20:34:32.357 INFO:tasks.workunit.client.0.smithi136.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T20:34:32.357 INFO:tasks.workunit.client.0.smithi136.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52602 2024-09-15T20:34:32.357 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.52602/ceph-mon.a.asok 2024-09-15T20:34:32.357 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-15T20:34:32.357 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.52602/ceph-mon.a.asok config get mon_host 2024-09-15T20:34:32.435 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:609: TEST_auto_repair_bluestore_failed_norecov: run_mgr td/osd-scrub-repair x 2024-09-15T20:34:32.435 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: run_mgr: local dir=td/osd-scrub-repair 2024-09-15T20:34:32.435 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: run_mgr: shift 2024-09-15T20:34:32.435 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: run_mgr: local id=x 2024-09-15T20:34:32.435 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: run_mgr: shift 2024-09-15T20:34:32.436 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:548: run_mgr: local data=td/osd-scrub-repair/x 2024-09-15T20:34:32.436 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:550: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-15T20:34:33.251 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: get_asok_path 2024-09-15T20:34:33.251 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-15T20:34:33.252 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-15T20:34:33.252 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-15T20:34:33.252 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T20:34:33.252 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52602 2024-09-15T20:34:33.252 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.52602/$cluster-$name.asok' 2024-09-15T20:34:33.252 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-15T20:34:33.253 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-scrub-repair/x '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.52602/$cluster-$name.asok' --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-15T20:34:33.280 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:612: TEST_auto_repair_bluestore_failed_norecov: local 'ceph_osd_args=--osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0' 2024-09-15T20:34:33.280 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:613: TEST_auto_repair_bluestore_failed_norecov: seq 0 2 2024-09-15T20:34:33.285 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:613: TEST_auto_repair_bluestore_failed_norecov: for id in $(seq 0 2) 2024-09-15T20:34:33.285 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:614: TEST_auto_repair_bluestore_failed_norecov: run_osd td/osd-scrub-repair 0 --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-09-15T20:34:33.285 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-scrub-repair 2024-09-15T20:34:33.285 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-15T20:34:33.285 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=0 2024-09-15T20:34:33.285 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-15T20:34:33.285 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-scrub-repair/0 2024-09-15T20:34:33.285 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=ce4892c7-6c6d-4b4f-8e6f-07a3e11133e5 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-15T20:34:33.285 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T20:34:33.285 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T20:34:33.285 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T20:34:33.286 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-09-15T20:34:33.286 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-09-15T20:34:33.286 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-15T20:34:33.286 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-15T20:34:33.286 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-15T20:34:33.286 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-15T20:34:33.286 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-15T20:34:33.286 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-15T20:34:33.286 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-15T20:34:33.286 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T20:34:33.286 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52602 2024-09-15T20:34:33.286 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.52602/$cluster-$name.asok' 2024-09-15T20:34:33.286 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.52602/$cluster-$name.asok' 2024-09-15T20:34:33.286 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-15T20:34:33.287 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-15T20:34:33.287 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-15T20:34:33.287 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-15T20:34:33.287 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-15T20:34:33.287 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T20:34:33.287 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T20:34:33.287 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T20:34:33.287 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T20:34:33.287 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-15T20:34:33.287 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+='--osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0' 2024-09-15T20:34:33.287 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-scrub-repair/0 2024-09-15T20:34:33.287 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-15T20:34:33.288 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=8e99504b-f154-418f-a155-6debf8edaca9 2024-09-15T20:34:33.288 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd0 8e99504b-f154-418f-a155-6debf8edaca9' 2024-09-15T20:34:33.288 INFO:tasks.workunit.client.0.smithi136.stdout:add osd0 8e99504b-f154-418f-a155-6debf8edaca9 2024-09-15T20:34:33.288 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-15T20:34:33.304 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQDZROdm4A4AEhAArJp8zSpxO3XJKC2N12Rz4w== 2024-09-15T20:34:33.304 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQDZROdm4A4AEhAArJp8zSpxO3XJKC2N12Rz4w=="}' 2024-09-15T20:34:33.304 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 8e99504b-f154-418f-a155-6debf8edaca9 -i td/osd-scrub-repair/0/new.json 2024-09-15T20:34:33.508 INFO:tasks.workunit.client.0.smithi136.stdout:0 2024-09-15T20:34:33.518 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-scrub-repair/0/new.json 2024-09-15T20:34:33.518 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 0 --fsid=ce4892c7-6c6d-4b4f-8e6f-07a3e11133e5 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.52602/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --mkfs --key AQDZROdm4A4AEhAArJp8zSpxO3XJKC2N12Rz4w== --osd-uuid 8e99504b-f154-418f-a155-6debf8edaca9 2024-09-15T20:34:33.551 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:34:33.549+0000 7faaff520640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T20:34:33.558 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:34:33.556+0000 7faaff520640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T20:34:33.559 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:34:33.557+0000 7faaff520640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T20:34:33.560 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:34:33.558+0000 7faaff520640 -1 bdev(0x558ef09cdc00 td/osd-scrub-repair/0/block) open stat got: (1) Operation not permitted 2024-09-15T20:34:33.560 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:34:33.558+0000 7faaff520640 -1 bluestore(td/osd-scrub-repair/0) _read_fsid unparsable uuid 2024-09-15T20:34:39.003 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-scrub-repair/0/keyring 2024-09-15T20:34:39.004 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-15T20:34:39.004 INFO:tasks.workunit.client.0.smithi136.stdout:adding osd0 key to auth repository 2024-09-15T20:34:39.005 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd0 key to auth repository 2024-09-15T20:34:39.005 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-scrub-repair/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-15T20:34:39.369 INFO:tasks.workunit.client.0.smithi136.stdout:start osd.0 2024-09-15T20:34:39.370 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.0 2024-09-15T20:34:39.370 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 0 --fsid=ce4892c7-6c6d-4b4f-8e6f-07a3e11133e5 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.52602/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-09-15T20:34:39.370 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-15T20:34:39.370 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-15T20:34:39.371 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-15T20:34:39.399 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:34:39.397+0000 7f3111f02640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T20:34:39.401 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:34:39.399+0000 7f3111f02640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T20:34:39.402 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:34:39.400+0000 7f3111f02640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T20:34:39.648 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 0 2024-09-15T20:34:39.648 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-15T20:34:39.648 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2024-09-15T20:34:39.649 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-15T20:34:39.649 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-15T20:34:39.649 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:34:39.649 INFO:tasks.workunit.client.0.smithi136.stdout:0 2024-09-15T20:34:39.649 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-15T20:34:39.649 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:34:39.650 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T20:34:39.926 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:34:40.528 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:34:40.526+0000 7f3111f02640 -1 Falling back to public interface 2024-09-15T20:34:40.927 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:34:40.927 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:34:40.928 INFO:tasks.workunit.client.0.smithi136.stdout:1 2024-09-15T20:34:40.928 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-15T20:34:40.928 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:34:40.928 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T20:34:41.215 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:34:42.216 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:34:42.216 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:34:42.216 INFO:tasks.workunit.client.0.smithi136.stdout:2 2024-09-15T20:34:42.217 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-15T20:34:42.217 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:34:42.217 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T20:34:42.502 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:34:43.426 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:34:43.423+0000 7f3111f02640 -1 osd.0 0 log_to_monitors true 2024-09-15T20:34:43.503 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:34:43.503 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:34:43.504 INFO:tasks.workunit.client.0.smithi136.stdout:3 2024-09-15T20:34:43.504 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-15T20:34:43.504 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:34:43.504 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T20:34:43.851 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:34:44.852 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:34:44.852 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:34:44.853 INFO:tasks.workunit.client.0.smithi136.stdout:4 2024-09-15T20:34:44.853 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-15T20:34:44.853 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:34:44.853 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T20:34:45.173 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:34:46.175 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:34:46.175 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:34:46.175 INFO:tasks.workunit.client.0.smithi136.stdout:5 2024-09-15T20:34:46.175 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-15T20:34:46.176 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:34:46.176 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T20:34:46.460 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:34:47.461 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:34:47.461 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:34:47.461 INFO:tasks.workunit.client.0.smithi136.stdout:6 2024-09-15T20:34:47.462 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-15T20:34:47.462 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:34:47.462 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T20:34:47.745 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:34:48.747 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:34:48.747 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:34:48.747 INFO:tasks.workunit.client.0.smithi136.stdout:7 2024-09-15T20:34:48.747 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-15T20:34:48.748 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:34:48.748 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T20:34:49.032 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:34:50.033 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:34:50.034 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:34:50.034 INFO:tasks.workunit.client.0.smithi136.stdout:8 2024-09-15T20:34:50.034 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-15T20:34:50.034 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:34:50.034 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T20:34:50.335 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:34:51.337 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:34:51.337 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:34:51.337 INFO:tasks.workunit.client.0.smithi136.stdout:9 2024-09-15T20:34:51.338 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-15T20:34:51.338 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:34:51.338 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T20:34:51.624 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:34:52.625 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:34:52.625 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:34:52.626 INFO:tasks.workunit.client.0.smithi136.stdout:10 2024-09-15T20:34:52.626 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-15T20:34:52.626 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:34:52.626 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T20:34:52.980 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:34:53.981 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:34:53.982 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:34:53.982 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-15T20:34:53.982 INFO:tasks.workunit.client.0.smithi136.stdout:11 2024-09-15T20:34:53.982 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:34:53.983 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T20:34:54.351 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:34:55.352 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:34:55.352 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:34:55.353 INFO:tasks.workunit.client.0.smithi136.stdout:12 2024-09-15T20:34:55.353 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-15T20:34:55.353 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:34:55.353 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T20:34:55.969 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:34:56.971 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:34:56.971 INFO:tasks.workunit.client.0.smithi136.stdout:13 2024-09-15T20:34:56.972 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:34:56.972 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-15T20:34:56.972 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:34:56.972 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T20:34:57.900 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:34:58.901 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:34:58.901 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:34:58.902 INFO:tasks.workunit.client.0.smithi136.stdout:14 2024-09-15T20:34:58.902 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-15T20:34:58.902 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:34:58.902 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T20:34:59.707 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:35:00.708 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:35:00.708 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:35:00.709 INFO:tasks.workunit.client.0.smithi136.stdout:15 2024-09-15T20:35:00.709 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-15T20:35:00.709 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:35:00.709 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T20:35:01.474 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:35:02.475 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:35:02.476 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:35:02.476 INFO:tasks.workunit.client.0.smithi136.stdout:16 2024-09-15T20:35:02.476 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-15T20:35:02.476 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:35:02.477 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T20:35:03.204 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:35:04.205 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:35:04.206 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:35:04.206 INFO:tasks.workunit.client.0.smithi136.stdout:17 2024-09-15T20:35:04.206 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-15T20:35:04.207 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:35:04.207 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T20:35:04.811 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:35:05.813 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:35:05.813 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:35:05.814 INFO:tasks.workunit.client.0.smithi136.stdout:18 2024-09-15T20:35:05.815 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 18 2024-09-15T20:35:05.815 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:35:05.815 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T20:35:06.100 INFO:tasks.workunit.client.0.smithi136.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/3833634155,v1:127.0.0.1:6803/3833634155] [v2:127.0.0.1:6804/3833634155,v1:127.0.0.1:6805/3833634155] exists,up 8e99504b-f154-418f-a155-6debf8edaca9 2024-09-15T20:35:06.104 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-15T20:35:06.104 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-15T20:35:06.104 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-15T20:35:06.104 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:613: TEST_auto_repair_bluestore_failed_norecov: for id in $(seq 0 2) 2024-09-15T20:35:06.104 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:614: TEST_auto_repair_bluestore_failed_norecov: run_osd td/osd-scrub-repair 1 --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-09-15T20:35:06.104 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-scrub-repair 2024-09-15T20:35:06.104 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-15T20:35:06.104 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=1 2024-09-15T20:35:06.105 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-15T20:35:06.105 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-scrub-repair/1 2024-09-15T20:35:06.105 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=ce4892c7-6c6d-4b4f-8e6f-07a3e11133e5 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-15T20:35:06.105 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T20:35:06.105 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T20:35:06.105 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T20:35:06.105 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-09-15T20:35:06.105 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-09-15T20:35:06.105 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-15T20:35:06.105 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-15T20:35:06.105 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-15T20:35:06.105 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-15T20:35:06.105 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-15T20:35:06.106 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-15T20:35:06.106 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-15T20:35:06.106 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T20:35:06.106 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52602 2024-09-15T20:35:06.106 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.52602/$cluster-$name.asok' 2024-09-15T20:35:06.106 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.52602/$cluster-$name.asok' 2024-09-15T20:35:06.106 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-15T20:35:06.106 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-15T20:35:06.106 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-15T20:35:06.106 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-15T20:35:06.106 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-15T20:35:06.106 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T20:35:06.106 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T20:35:06.107 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T20:35:06.107 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T20:35:06.107 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-15T20:35:06.107 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+='--osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0' 2024-09-15T20:35:06.107 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-scrub-repair/1 2024-09-15T20:35:06.107 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-15T20:35:06.107 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=a3cff831-3849-4205-bcdf-59d3a90a6d99 2024-09-15T20:35:06.107 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd1 a3cff831-3849-4205-bcdf-59d3a90a6d99' 2024-09-15T20:35:06.107 INFO:tasks.workunit.client.0.smithi136.stdout:add osd1 a3cff831-3849-4205-bcdf-59d3a90a6d99 2024-09-15T20:35:06.108 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-15T20:35:06.122 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQD6ROdmF+UoBxAAtXs9UipzFP6STLvQxAt9Qw== 2024-09-15T20:35:06.122 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQD6ROdmF+UoBxAAtXs9UipzFP6STLvQxAt9Qw=="}' 2024-09-15T20:35:06.122 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new a3cff831-3849-4205-bcdf-59d3a90a6d99 -i td/osd-scrub-repair/1/new.json 2024-09-15T20:35:06.445 INFO:tasks.workunit.client.0.smithi136.stdout:1 2024-09-15T20:35:06.457 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-scrub-repair/1/new.json 2024-09-15T20:35:06.458 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 1 --fsid=ce4892c7-6c6d-4b4f-8e6f-07a3e11133e5 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.52602/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --mkfs --key AQD6ROdmF+UoBxAAtXs9UipzFP6STLvQxAt9Qw== --osd-uuid a3cff831-3849-4205-bcdf-59d3a90a6d99 2024-09-15T20:35:06.490 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:35:06.488+0000 7f162660b640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T20:35:06.530 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:35:06.528+0000 7f162660b640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T20:35:06.531 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:35:06.529+0000 7f162660b640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T20:35:06.531 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:35:06.529+0000 7f162660b640 -1 bdev(0x555c5b819c00 td/osd-scrub-repair/1/block) open stat got: (1) Operation not permitted 2024-09-15T20:35:06.532 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:35:06.530+0000 7f162660b640 -1 bluestore(td/osd-scrub-repair/1) _read_fsid unparsable uuid 2024-09-15T20:35:12.662 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-scrub-repair/1/keyring 2024-09-15T20:35:12.662 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-15T20:35:12.663 INFO:tasks.workunit.client.0.smithi136.stdout:adding osd1 key to auth repository 2024-09-15T20:35:12.663 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd1 key to auth repository 2024-09-15T20:35:12.663 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-scrub-repair/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-15T20:35:13.020 INFO:tasks.workunit.client.0.smithi136.stdout:start osd.1 2024-09-15T20:35:13.020 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.1 2024-09-15T20:35:13.021 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 1 --fsid=ce4892c7-6c6d-4b4f-8e6f-07a3e11133e5 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.52602/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-09-15T20:35:13.021 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-15T20:35:13.021 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-15T20:35:13.021 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-15T20:35:13.049 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:35:13.047+0000 7fd4b1e0b640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T20:35:13.052 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:35:13.050+0000 7fd4b1e0b640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T20:35:13.053 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:35:13.051+0000 7fd4b1e0b640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T20:35:13.323 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 1 2024-09-15T20:35:13.323 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-15T20:35:13.323 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2024-09-15T20:35:13.323 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-15T20:35:13.323 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-15T20:35:13.323 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:35:13.323 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-15T20:35:13.324 INFO:tasks.workunit.client.0.smithi136.stdout:0 2024-09-15T20:35:13.324 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:35:13.324 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T20:35:13.600 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:35:14.171 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:35:14.169+0000 7fd4b1e0b640 -1 Falling back to public interface 2024-09-15T20:35:14.601 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:35:14.602 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:35:14.602 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-15T20:35:14.602 INFO:tasks.workunit.client.0.smithi136.stdout:1 2024-09-15T20:35:14.602 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:35:14.603 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T20:35:14.885 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:35:15.887 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:35:15.887 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:35:15.887 INFO:tasks.workunit.client.0.smithi136.stdout:2 2024-09-15T20:35:15.888 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-15T20:35:15.888 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:35:15.888 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T20:35:16.173 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:35:17.054 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:35:17.052+0000 7fd4b1e0b640 -1 osd.1 0 log_to_monitors true 2024-09-15T20:35:17.175 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:35:17.175 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:35:17.175 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-15T20:35:17.175 INFO:tasks.workunit.client.0.smithi136.stdout:3 2024-09-15T20:35:17.175 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:35:17.176 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T20:35:17.568 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:35:18.569 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:35:18.569 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:35:18.570 INFO:tasks.workunit.client.0.smithi136.stdout:4 2024-09-15T20:35:18.570 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-15T20:35:18.570 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:35:18.570 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T20:35:18.899 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:35:19.901 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:35:19.901 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:35:19.901 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-15T20:35:19.901 INFO:tasks.workunit.client.0.smithi136.stdout:5 2024-09-15T20:35:19.902 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:35:19.902 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T20:35:20.224 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:35:21.225 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:35:21.226 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:35:21.226 INFO:tasks.workunit.client.0.smithi136.stdout:6 2024-09-15T20:35:21.226 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-15T20:35:21.227 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:35:21.227 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T20:35:21.673 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:35:22.675 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:35:22.675 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:35:22.675 INFO:tasks.workunit.client.0.smithi136.stdout:7 2024-09-15T20:35:22.676 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-15T20:35:22.676 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:35:22.676 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T20:35:23.072 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:35:24.071 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:35:24.071 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:35:24.071 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-15T20:35:24.071 INFO:tasks.workunit.client.0.smithi136.stdout:8 2024-09-15T20:35:24.072 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:35:24.072 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T20:35:24.380 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:35:25.381 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:35:25.382 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:35:25.382 INFO:tasks.workunit.client.0.smithi136.stdout:9 2024-09-15T20:35:25.382 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-15T20:35:25.382 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:35:25.383 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T20:35:25.681 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:35:26.682 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:35:26.683 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:35:26.683 INFO:tasks.workunit.client.0.smithi136.stdout:10 2024-09-15T20:35:26.683 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-15T20:35:26.684 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:35:26.684 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T20:35:27.053 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:35:28.054 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:35:28.054 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:35:28.055 INFO:tasks.workunit.client.0.smithi136.stdout:11 2024-09-15T20:35:28.055 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-15T20:35:28.055 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:35:28.055 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T20:35:28.345 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:35:29.346 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:35:29.346 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:35:29.347 INFO:tasks.workunit.client.0.smithi136.stdout:12 2024-09-15T20:35:29.347 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-15T20:35:29.347 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:35:29.347 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T20:35:29.920 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:35:30.921 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:35:30.921 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:35:30.922 INFO:tasks.workunit.client.0.smithi136.stdout:13 2024-09-15T20:35:30.922 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-15T20:35:30.922 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:35:30.922 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T20:35:31.361 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:35:32.362 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:35:32.362 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:35:32.363 INFO:tasks.workunit.client.0.smithi136.stdout:14 2024-09-15T20:35:32.363 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-15T20:35:32.363 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:35:32.363 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T20:35:32.785 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:35:33.786 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:35:33.786 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:35:33.786 INFO:tasks.workunit.client.0.smithi136.stdout:15 2024-09-15T20:35:33.787 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-15T20:35:33.787 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:35:33.787 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T20:35:34.126 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:35:35.127 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:35:35.128 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:35:35.128 INFO:tasks.workunit.client.0.smithi136.stdout:16 2024-09-15T20:35:35.128 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-15T20:35:35.129 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:35:35.129 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T20:35:35.537 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:35:36.538 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:35:36.538 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:35:36.539 INFO:tasks.workunit.client.0.smithi136.stdout:17 2024-09-15T20:35:36.539 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-15T20:35:36.540 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:35:36.540 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T20:35:37.092 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:35:38.093 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:35:38.093 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:35:38.094 INFO:tasks.workunit.client.0.smithi136.stdout:18 2024-09-15T20:35:38.094 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 18 2024-09-15T20:35:38.094 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:35:38.094 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T20:35:38.475 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:35:39.476 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:35:39.476 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:35:39.477 INFO:tasks.workunit.client.0.smithi136.stdout:19 2024-09-15T20:35:39.477 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 19 2024-09-15T20:35:39.477 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:35:39.477 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T20:35:39.922 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:35:40.923 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:35:40.924 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:35:40.924 INFO:tasks.workunit.client.0.smithi136.stdout:20 2024-09-15T20:35:40.924 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 20 2024-09-15T20:35:40.925 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:35:40.925 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T20:35:41.241 INFO:tasks.workunit.client.0.smithi136.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/4040339816,v1:127.0.0.1:6811/4040339816] [v2:127.0.0.1:6812/4040339816,v1:127.0.0.1:6813/4040339816] exists,up a3cff831-3849-4205-bcdf-59d3a90a6d99 2024-09-15T20:35:41.245 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-15T20:35:41.245 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-15T20:35:41.245 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-15T20:35:41.245 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:613: TEST_auto_repair_bluestore_failed_norecov: for id in $(seq 0 2) 2024-09-15T20:35:41.245 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:614: TEST_auto_repair_bluestore_failed_norecov: run_osd td/osd-scrub-repair 2 --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-09-15T20:35:41.245 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-scrub-repair 2024-09-15T20:35:41.245 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-15T20:35:41.246 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=2 2024-09-15T20:35:41.246 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-15T20:35:41.246 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-scrub-repair/2 2024-09-15T20:35:41.246 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=ce4892c7-6c6d-4b4f-8e6f-07a3e11133e5 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-15T20:35:41.246 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T20:35:41.246 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T20:35:41.246 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T20:35:41.246 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/2' 2024-09-15T20:35:41.246 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/2/journal' 2024-09-15T20:35:41.246 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-15T20:35:41.246 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-15T20:35:41.246 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-15T20:35:41.246 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-15T20:35:41.246 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-15T20:35:41.247 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-15T20:35:41.247 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-15T20:35:41.247 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T20:35:41.247 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52602 2024-09-15T20:35:41.247 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.52602/$cluster-$name.asok' 2024-09-15T20:35:41.247 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.52602/$cluster-$name.asok' 2024-09-15T20:35:41.247 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-15T20:35:41.247 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-15T20:35:41.247 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-15T20:35:41.247 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-15T20:35:41.247 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-15T20:35:41.247 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T20:35:41.247 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T20:35:41.248 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T20:35:41.248 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T20:35:41.248 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-15T20:35:41.248 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+='--osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0' 2024-09-15T20:35:41.248 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-scrub-repair/2 2024-09-15T20:35:41.248 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-15T20:35:41.248 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=e82fe6c6-1032-49d7-85b7-4b92c990a2f0 2024-09-15T20:35:41.248 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd2 e82fe6c6-1032-49d7-85b7-4b92c990a2f0' 2024-09-15T20:35:41.248 INFO:tasks.workunit.client.0.smithi136.stdout:add osd2 e82fe6c6-1032-49d7-85b7-4b92c990a2f0 2024-09-15T20:35:41.249 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-15T20:35:41.263 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQAdRedmkwqaDxAAKLrKjztIqDx4biFNbuDByA== 2024-09-15T20:35:41.264 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQAdRedmkwqaDxAAKLrKjztIqDx4biFNbuDByA=="}' 2024-09-15T20:35:41.264 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new e82fe6c6-1032-49d7-85b7-4b92c990a2f0 -i td/osd-scrub-repair/2/new.json 2024-09-15T20:35:41.639 INFO:tasks.workunit.client.0.smithi136.stdout:2 2024-09-15T20:35:41.650 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-scrub-repair/2/new.json 2024-09-15T20:35:41.651 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 2 --fsid=ce4892c7-6c6d-4b4f-8e6f-07a3e11133e5 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.52602/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --mkfs --key AQAdRedmkwqaDxAAKLrKjztIqDx4biFNbuDByA== --osd-uuid e82fe6c6-1032-49d7-85b7-4b92c990a2f0 2024-09-15T20:35:41.682 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:35:41.680+0000 7f1e4021e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T20:35:41.715 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:35:41.713+0000 7f1e4021e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T20:35:41.716 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:35:41.714+0000 7f1e4021e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T20:35:41.716 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:35:41.714+0000 7f1e4021e640 -1 bdev(0x5638b597fc00 td/osd-scrub-repair/2/block) open stat got: (1) Operation not permitted 2024-09-15T20:35:41.716 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:35:41.714+0000 7f1e4021e640 -1 bluestore(td/osd-scrub-repair/2) _read_fsid unparsable uuid 2024-09-15T20:35:47.642 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-scrub-repair/2/keyring 2024-09-15T20:35:47.643 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-15T20:35:47.644 INFO:tasks.workunit.client.0.smithi136.stdout:adding osd2 key to auth repository 2024-09-15T20:35:47.644 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd2 key to auth repository 2024-09-15T20:35:47.644 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-scrub-repair/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-15T20:35:48.000 INFO:tasks.workunit.client.0.smithi136.stdout:start osd.2 2024-09-15T20:35:48.001 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.2 2024-09-15T20:35:48.001 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 2 --fsid=ce4892c7-6c6d-4b4f-8e6f-07a3e11133e5 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.52602/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-09-15T20:35:48.001 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-15T20:35:48.002 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-15T20:35:48.002 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-15T20:35:48.030 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:35:48.028+0000 7f105481e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T20:35:48.032 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:35:48.030+0000 7f105481e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T20:35:48.033 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:35:48.031+0000 7f105481e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T20:35:48.279 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 2 2024-09-15T20:35:48.280 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-15T20:35:48.280 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=2 2024-09-15T20:35:48.280 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-15T20:35:48.280 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-15T20:35:48.280 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:35:48.280 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-15T20:35:48.280 INFO:tasks.workunit.client.0.smithi136.stdout:0 2024-09-15T20:35:48.281 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:35:48.281 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T20:35:48.557 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:35:49.395 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:35:49.393+0000 7f105481e640 -1 Falling back to public interface 2024-09-15T20:35:49.558 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:35:49.559 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:35:49.559 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-15T20:35:49.559 INFO:tasks.workunit.client.0.smithi136.stdout:1 2024-09-15T20:35:49.559 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:35:49.559 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T20:35:49.842 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:35:50.844 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:35:50.844 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:35:50.844 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-15T20:35:50.844 INFO:tasks.workunit.client.0.smithi136.stdout:2 2024-09-15T20:35:50.845 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:35:50.845 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T20:35:51.120 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:35:52.039 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:35:52.037+0000 7f105481e640 -1 osd.2 0 log_to_monitors true 2024-09-15T20:35:52.121 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:35:52.121 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:35:52.121 INFO:tasks.workunit.client.0.smithi136.stdout:3 2024-09-15T20:35:52.122 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-15T20:35:52.122 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:35:52.122 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T20:35:52.532 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:35:53.534 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:35:53.534 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:35:53.534 INFO:tasks.workunit.client.0.smithi136.stdout:4 2024-09-15T20:35:53.535 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-15T20:35:53.535 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:35:53.535 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T20:35:54.023 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:35:55.024 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:35:55.025 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:35:55.025 INFO:tasks.workunit.client.0.smithi136.stdout:5 2024-09-15T20:35:55.025 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-15T20:35:55.025 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:35:55.025 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T20:35:55.369 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:35:56.370 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:35:56.370 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:35:56.370 INFO:tasks.workunit.client.0.smithi136.stdout:6 2024-09-15T20:35:56.371 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-15T20:35:56.371 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:35:56.371 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T20:35:56.663 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:35:57.664 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:35:57.664 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:35:57.665 INFO:tasks.workunit.client.0.smithi136.stdout:7 2024-09-15T20:35:57.665 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-15T20:35:57.665 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:35:57.665 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T20:35:57.986 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:35:58.987 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:35:58.987 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:35:58.988 INFO:tasks.workunit.client.0.smithi136.stdout:8 2024-09-15T20:35:58.988 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-15T20:35:58.988 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:35:58.988 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T20:35:59.275 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:36:00.276 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:36:00.276 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:36:00.277 INFO:tasks.workunit.client.0.smithi136.stdout:9 2024-09-15T20:36:00.277 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-15T20:36:00.277 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:36:00.277 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T20:36:00.564 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:36:01.565 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:36:01.565 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:36:01.565 INFO:tasks.workunit.client.0.smithi136.stdout:10 2024-09-15T20:36:01.566 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-15T20:36:01.566 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:36:01.566 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T20:36:01.931 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:36:02.932 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:36:02.932 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:36:02.932 INFO:tasks.workunit.client.0.smithi136.stdout:11 2024-09-15T20:36:02.933 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-15T20:36:02.933 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:36:02.933 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T20:36:03.218 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:36:04.220 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:36:04.220 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:36:04.220 INFO:tasks.workunit.client.0.smithi136.stdout:12 2024-09-15T20:36:04.220 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-15T20:36:04.220 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:36:04.221 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T20:36:04.734 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:36:05.736 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:36:05.736 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:36:05.736 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-15T20:36:05.736 INFO:tasks.workunit.client.0.smithi136.stdout:13 2024-09-15T20:36:05.737 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:36:05.737 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T20:36:06.411 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:36:07.413 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:36:07.413 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:36:07.413 INFO:tasks.workunit.client.0.smithi136.stdout:14 2024-09-15T20:36:07.414 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-15T20:36:07.414 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:36:07.414 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T20:36:08.255 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:36:09.257 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:36:09.257 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:36:09.257 INFO:tasks.workunit.client.0.smithi136.stdout:15 2024-09-15T20:36:09.258 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-15T20:36:09.258 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:36:09.258 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T20:36:09.999 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:36:11.000 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:36:11.000 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:36:11.001 INFO:tasks.workunit.client.0.smithi136.stdout:16 2024-09-15T20:36:11.001 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-15T20:36:11.001 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:36:11.001 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T20:36:11.717 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:36:12.718 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:36:12.719 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:36:12.719 INFO:tasks.workunit.client.0.smithi136.stdout:17 2024-09-15T20:36:12.719 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-15T20:36:12.719 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:36:12.719 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T20:36:13.262 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:36:14.264 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:36:14.264 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:36:14.264 INFO:tasks.workunit.client.0.smithi136.stdout:18 2024-09-15T20:36:14.265 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 18 2024-09-15T20:36:14.265 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:36:14.265 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T20:36:14.583 INFO:tasks.workunit.client.0.smithi136.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/3127927163,v1:127.0.0.1:6819/3127927163] [v2:127.0.0.1:6820/3127927163,v1:127.0.0.1:6821/3127927163] exists,up e82fe6c6-1032-49d7-85b7-4b92c990a2f0 2024-09-15T20:36:14.583 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-15T20:36:14.583 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-15T20:36:14.583 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-15T20:36:14.583 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:617: TEST_auto_repair_bluestore_failed_norecov: create_pool testpool 1 1 2024-09-15T20:36:14.584 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create testpool 1 1 2024-09-15T20:36:14.954 INFO:tasks.workunit.client.0.smithi136.stderr:pool 'testpool' created 2024-09-15T20:36:14.965 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2024-09-15T20:36:15.966 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:618: TEST_auto_repair_bluestore_failed_norecov: ceph osd pool set testpool size 2 2024-09-15T20:36:16.445 INFO:tasks.workunit.client.0.smithi136.stderr:set pool 1 size to 2 2024-09-15T20:36:16.457 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:619: TEST_auto_repair_bluestore_failed_norecov: wait_for_clean 2024-09-15T20:36:16.457 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-15T20:36:16.457 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-15T20:36:16.457 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-15T20:36:16.457 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-15T20:36:16.458 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-15T20:36:16.458 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-15T20:36:16.458 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-15T20:36:16.458 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-15T20:36:16.458 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-15T20:36:16.549 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-15T20:36:16.549 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-15T20:36:16.549 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-15T20:36:16.549 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-15T20:36:16.549 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-15T20:36:16.549 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-15T20:36:16.834 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-15T20:36:16.834 INFO:tasks.workunit.client.0.smithi136.stderr:1 2024-09-15T20:36:16.834 INFO:tasks.workunit.client.0.smithi136.stderr:2' 2024-09-15T20:36:16.834 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-15T20:36:16.834 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T20:36:16.834 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-15T20:36:16.991 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836499 2024-09-15T20:36:16.992 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836499 2024-09-15T20:36:16.992 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836499' 2024-09-15T20:36:16.992 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T20:36:16.992 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-15T20:36:17.135 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=42949672971 2024-09-15T20:36:17.135 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 42949672971 2024-09-15T20:36:17.136 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836499 1-42949672971' 2024-09-15T20:36:17.136 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T20:36:17.136 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-15T20:36:17.288 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=64424509447 2024-09-15T20:36:17.289 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 64424509447 2024-09-15T20:36:17.289 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836499 1-42949672971 2-64424509447' 2024-09-15T20:36:17.289 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T20:36:17.289 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836499 2024-09-15T20:36:17.289 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T20:36:17.290 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-15T20:36:17.290 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836499 2024-09-15T20:36:17.290 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T20:36:17.291 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836499 2024-09-15T20:36:17.292 INFO:tasks.workunit.client.0.smithi136.stdout:waiting osd.0 seq 21474836499 2024-09-15T20:36:17.292 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836499' 2024-09-15T20:36:17.292 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T20:36:17.657 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836499 -lt 21474836499 2024-09-15T20:36:17.657 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T20:36:17.658 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-42949672971 2024-09-15T20:36:17.658 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T20:36:17.658 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-15T20:36:17.659 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-42949672971 2024-09-15T20:36:17.659 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T20:36:17.660 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=42949672971 2024-09-15T20:36:17.660 INFO:tasks.workunit.client.0.smithi136.stdout:waiting osd.1 seq 42949672971 2024-09-15T20:36:17.661 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 42949672971' 2024-09-15T20:36:17.661 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-15T20:36:17.960 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 42949672971 -lt 42949672971 2024-09-15T20:36:17.961 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T20:36:17.961 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-64424509447 2024-09-15T20:36:17.961 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T20:36:17.962 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-15T20:36:17.962 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-64424509447 2024-09-15T20:36:17.962 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T20:36:17.963 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=64424509447 2024-09-15T20:36:17.963 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 64424509447' 2024-09-15T20:36:17.964 INFO:tasks.workunit.client.0.smithi136.stdout:waiting osd.2 seq 64424509447 2024-09-15T20:36:17.964 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-15T20:36:18.240 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 6 -lt 64424509447 2024-09-15T20:36:18.241 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-15T20:36:19.242 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-15T20:36:19.242 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-15T20:36:19.531 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 64424509447 -lt 64424509447 2024-09-15T20:36:19.532 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-15T20:36:19.532 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T20:36:19.532 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T20:36:19.898 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 1 == 0 2024-09-15T20:36:19.899 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-15T20:36:19.899 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-15T20:36:19.900 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-15T20:36:19.900 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T20:36:19.900 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T20:36:19.900 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T20:36:19.900 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T20:36:20.174 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=0 2024-09-15T20:36:20.175 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-15T20:36:20.175 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T20:36:20.175 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T20:36:20.522 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 0 = 1 2024-09-15T20:36:20.523 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 0 '!=' -1 2024-09-15T20:36:20.523 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: loop=0 2024-09-15T20:36:20.523 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1652: wait_for_clean: num_active_clean=0 2024-09-15T20:36:20.523 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-15T20:36:20.523 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 0.1 2024-09-15T20:36:20.624 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-15T20:36:20.624 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-15T20:36:20.625 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-15T20:36:20.625 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-15T20:36:20.625 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T20:36:20.625 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T20:36:20.625 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T20:36:20.625 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T20:36:20.894 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=0 2024-09-15T20:36:20.894 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-15T20:36:20.894 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T20:36:20.894 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T20:36:21.238 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 0 = 1 2024-09-15T20:36:21.238 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 0 '!=' 0 2024-09-15T20:36:21.238 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: get_is_making_recovery_progress 2024-09-15T20:36:21.238 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2024-09-15T20:36:21.238 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-15T20:36:21.238 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-15T20:36:21.238 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-15T20:36:21.238 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2024-09-15T20:36:21.239 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-15T20:36:21.587 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=null 2024-09-15T20:36:21.606 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test null '!=' null 2024-09-15T20:36:21.607 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1655: wait_for_clean: (( 1 >= 13 )) 2024-09-15T20:36:21.607 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-15T20:36:21.607 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 0.2 2024-09-15T20:36:21.787 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-15T20:36:21.787 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-15T20:36:21.788 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-15T20:36:21.788 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-15T20:36:21.788 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T20:36:21.788 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T20:36:21.788 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T20:36:21.788 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T20:36:22.057 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=1 2024-09-15T20:36:22.057 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-15T20:36:22.058 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T20:36:22.058 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T20:36:22.407 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 = 1 2024-09-15T20:36:22.407 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-15T20:36:22.408 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-15T20:36:22.408 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:622: TEST_auto_repair_bluestore_failed_norecov: local payload=ABCDEF 2024-09-15T20:36:22.408 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:623: TEST_auto_repair_bluestore_failed_norecov: echo ABCDEF 2024-09-15T20:36:22.408 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:624: TEST_auto_repair_bluestore_failed_norecov: seq 1 10 2024-09-15T20:36:22.409 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:624: TEST_auto_repair_bluestore_failed_norecov: for i in $(seq 1 10) 2024-09-15T20:36:22.409 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:626: TEST_auto_repair_bluestore_failed_norecov: rados --pool testpool put obj1 td/osd-scrub-repair/ORIGINAL 2024-09-15T20:36:22.569 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:624: TEST_auto_repair_bluestore_failed_norecov: for i in $(seq 1 10) 2024-09-15T20:36:22.569 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:626: TEST_auto_repair_bluestore_failed_norecov: rados --pool testpool put obj2 td/osd-scrub-repair/ORIGINAL 2024-09-15T20:36:22.737 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:624: TEST_auto_repair_bluestore_failed_norecov: for i in $(seq 1 10) 2024-09-15T20:36:22.737 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:626: TEST_auto_repair_bluestore_failed_norecov: rados --pool testpool put obj3 td/osd-scrub-repair/ORIGINAL 2024-09-15T20:36:22.862 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:624: TEST_auto_repair_bluestore_failed_norecov: for i in $(seq 1 10) 2024-09-15T20:36:22.862 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:626: TEST_auto_repair_bluestore_failed_norecov: rados --pool testpool put obj4 td/osd-scrub-repair/ORIGINAL 2024-09-15T20:36:22.987 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:624: TEST_auto_repair_bluestore_failed_norecov: for i in $(seq 1 10) 2024-09-15T20:36:22.988 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:626: TEST_auto_repair_bluestore_failed_norecov: rados --pool testpool put obj5 td/osd-scrub-repair/ORIGINAL 2024-09-15T20:36:23.105 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:624: TEST_auto_repair_bluestore_failed_norecov: for i in $(seq 1 10) 2024-09-15T20:36:23.105 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:626: TEST_auto_repair_bluestore_failed_norecov: rados --pool testpool put obj6 td/osd-scrub-repair/ORIGINAL 2024-09-15T20:36:23.230 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:624: TEST_auto_repair_bluestore_failed_norecov: for i in $(seq 1 10) 2024-09-15T20:36:23.230 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:626: TEST_auto_repair_bluestore_failed_norecov: rados --pool testpool put obj7 td/osd-scrub-repair/ORIGINAL 2024-09-15T20:36:23.364 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:624: TEST_auto_repair_bluestore_failed_norecov: for i in $(seq 1 10) 2024-09-15T20:36:23.365 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:626: TEST_auto_repair_bluestore_failed_norecov: rados --pool testpool put obj8 td/osd-scrub-repair/ORIGINAL 2024-09-15T20:36:23.515 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:624: TEST_auto_repair_bluestore_failed_norecov: for i in $(seq 1 10) 2024-09-15T20:36:23.515 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:626: TEST_auto_repair_bluestore_failed_norecov: rados --pool testpool put obj9 td/osd-scrub-repair/ORIGINAL 2024-09-15T20:36:23.641 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:624: TEST_auto_repair_bluestore_failed_norecov: for i in $(seq 1 10) 2024-09-15T20:36:23.641 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:626: TEST_auto_repair_bluestore_failed_norecov: rados --pool testpool put obj10 td/osd-scrub-repair/ORIGINAL 2024-09-15T20:36:23.767 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:632: TEST_auto_repair_bluestore_failed_norecov: get_not_primary testpool SOMETHING 2024-09-15T20:36:23.767 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1220: get_not_primary: local poolname=testpool 2024-09-15T20:36:23.768 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1221: get_not_primary: local objectname=SOMETHING 2024-09-15T20:36:23.768 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1223: get_not_primary: get_primary testpool SOMETHING 2024-09-15T20:36:23.768 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1187: get_primary: local poolname=testpool 2024-09-15T20:36:23.768 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1188: get_primary: local objectname=SOMETHING 2024-09-15T20:36:23.768 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1190: get_primary: ceph --format json osd map testpool SOMETHING 2024-09-15T20:36:23.768 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1191: get_primary: jq .acting_primary 2024-09-15T20:36:24.047 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1223: get_not_primary: local primary=1 2024-09-15T20:36:24.048 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1224: get_not_primary: ceph --format json osd map testpool SOMETHING 2024-09-15T20:36:24.048 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1225: get_not_primary: jq '.acting | map(select (. != 1)) | .[0]' 2024-09-15T20:36:24.326 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:632: TEST_auto_repair_bluestore_failed_norecov: objectstore_tool td/osd-scrub-repair 0 obj1 remove 2024-09-15T20:36:24.326 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2024-09-15T20:36:24.326 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2024-09-15T20:36:24.326 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=0 2024-09-15T20:36:24.327 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2024-09-15T20:36:24.327 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 obj1 remove 2024-09-15T20:36:24.327 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-09-15T20:36:24.327 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2024-09-15T20:36:24.327 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=0 2024-09-15T20:36:24.327 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2024-09-15T20:36:24.327 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2024-09-15T20:36:24.327 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-15T20:36:24.327 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-15T20:36:24.327 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-15T20:36:24.327 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-15T20:36:24.327 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-15T20:36:24.629 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-15T20:36:24.629 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 obj1 remove 2024-09-15T20:36:24.630 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-15T20:36:24.630 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-15T20:36:24.630 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=0 2024-09-15T20:36:24.630 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-15T20:36:24.630 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-09-15T20:36:24.630 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 obj1 remove 2024-09-15T20:36:26.192 INFO:tasks.workunit.client.0.smithi136.stdout:remove #1:ff7b1f36:::obj1:head# 2024-09-15T20:36:26.711 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-09-15T20:36:26.711 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2024-09-15T20:36:26.711 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-15T20:36:26.711 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=0 2024-09-15T20:36:26.711 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-15T20:36:26.711 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-09-15T20:36:26.712 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=ce4892c7-6c6d-4b4f-8e6f-07a3e11133e5 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-15T20:36:26.712 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T20:36:26.712 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T20:36:26.712 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T20:36:26.712 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-09-15T20:36:26.712 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-09-15T20:36:26.712 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-15T20:36:26.712 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-15T20:36:26.712 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-15T20:36:26.712 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-15T20:36:26.712 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-15T20:36:26.712 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-15T20:36:26.712 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-15T20:36:26.713 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T20:36:26.713 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52602 2024-09-15T20:36:26.713 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.52602/$cluster-$name.asok' 2024-09-15T20:36:26.713 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.52602/$cluster-$name.asok' 2024-09-15T20:36:26.713 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-15T20:36:26.713 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-15T20:36:26.713 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-15T20:36:26.713 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T20:36:26.713 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T20:36:26.713 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T20:36:26.713 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T20:36:26.713 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-15T20:36:26.713 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+='--osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0' 2024-09-15T20:36:26.713 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-09-15T20:36:26.714 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.0 2024-09-15T20:36:26.714 INFO:tasks.workunit.client.0.smithi136.stderr:start osd.0 2024-09-15T20:36:26.715 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 0 --fsid=ce4892c7-6c6d-4b4f-8e6f-07a3e11133e5 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.52602/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-09-15T20:36:26.715 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-09-15T20:36:26.715 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 0 = 0 ']' 2024-09-15T20:36:26.715 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-15T20:36:26.715 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-15T20:36:26.715 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-15T20:36:26.744 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:36:26.742+0000 7fe8c3c0f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T20:36:26.746 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:36:26.744+0000 7fe8c3c0f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T20:36:26.747 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:36:26.745+0000 7fe8c3c0f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T20:36:26.999 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 0 2024-09-15T20:36:26.999 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-15T20:36:26.999 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2024-09-15T20:36:26.999 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-15T20:36:26.999 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-15T20:36:26.999 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:36:26.999 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-15T20:36:27.000 INFO:tasks.workunit.client.0.smithi136.stderr:0 2024-09-15T20:36:27.000 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:36:27.000 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T20:36:27.278 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:36:28.114 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:36:28.112+0000 7fe8c3c0f640 -1 Falling back to public interface 2024-09-15T20:36:28.280 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:36:28.280 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:36:28.280 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-15T20:36:28.281 INFO:tasks.workunit.client.0.smithi136.stderr:1 2024-09-15T20:36:28.281 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:36:28.281 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T20:36:28.566 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:36:29.567 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:36:29.568 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:36:29.568 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-15T20:36:29.568 INFO:tasks.workunit.client.0.smithi136.stderr:2 2024-09-15T20:36:29.568 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:36:29.568 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T20:36:29.852 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:36:30.751 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:36:30.749+0000 7fe8c3c0f640 -1 osd.0 21 log_to_monitors true 2024-09-15T20:36:30.853 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:36:30.853 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:36:30.854 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-15T20:36:30.854 INFO:tasks.workunit.client.0.smithi136.stderr:3 2024-09-15T20:36:30.854 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:36:30.854 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T20:36:31.198 INFO:tasks.workunit.client.0.smithi136.stderr:osd.0 up in weight 1 up_from 25 up_thru 0 down_at 22 last_clean_interval [5,21) [v2:127.0.0.1:6802/2894899867,v1:127.0.0.1:6803/2894899867] [v2:127.0.0.1:6804/2894899867,v1:127.0.0.1:6805/2894899867] exists,up 8e99504b-f154-418f-a155-6debf8edaca9 2024-09-15T20:36:31.198 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-15T20:36:31.198 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-15T20:36:31.198 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-15T20:36:31.198 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2024-09-15T20:36:31.198 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-15T20:36:31.198 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-15T20:36:31.198 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-15T20:36:31.198 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-15T20:36:31.199 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-15T20:36:31.199 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-15T20:36:31.199 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-15T20:36:31.199 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-15T20:36:31.199 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-15T20:36:31.288 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-15T20:36:31.288 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-15T20:36:31.288 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-15T20:36:31.288 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-15T20:36:31.288 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-15T20:36:31.288 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-15T20:36:31.586 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-15T20:36:31.586 INFO:tasks.workunit.client.0.smithi136.stderr:1 2024-09-15T20:36:31.586 INFO:tasks.workunit.client.0.smithi136.stderr:2' 2024-09-15T20:36:31.586 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-15T20:36:31.586 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T20:36:31.586 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-15T20:36:31.688 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=107374182402 2024-09-15T20:36:31.688 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 107374182402 2024-09-15T20:36:31.688 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-107374182402' 2024-09-15T20:36:31.688 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T20:36:31.688 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-15T20:36:31.792 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=42949672975 2024-09-15T20:36:31.792 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 42949672975 2024-09-15T20:36:31.792 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-107374182402 1-42949672975' 2024-09-15T20:36:31.792 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T20:36:31.792 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-15T20:36:31.894 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=64424509451 2024-09-15T20:36:31.894 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 64424509451 2024-09-15T20:36:31.894 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-107374182402 1-42949672975 2-64424509451' 2024-09-15T20:36:31.894 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T20:36:31.895 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-107374182402 2024-09-15T20:36:31.895 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T20:36:31.895 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-15T20:36:31.896 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-107374182402 2024-09-15T20:36:31.896 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T20:36:31.901 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=107374182402 2024-09-15T20:36:31.901 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 107374182402' 2024-09-15T20:36:31.901 INFO:tasks.workunit.client.0.smithi136.stderr:waiting osd.0 seq 107374182402 2024-09-15T20:36:31.901 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T20:36:32.177 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 1 -lt 107374182402 2024-09-15T20:36:32.178 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-15T20:36:33.179 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-15T20:36:33.179 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T20:36:33.466 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 107374182402 -lt 107374182402 2024-09-15T20:36:33.466 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T20:36:33.467 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-42949672975 2024-09-15T20:36:33.467 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T20:36:33.468 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-15T20:36:33.468 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-42949672975 2024-09-15T20:36:33.468 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T20:36:33.469 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=42949672975 2024-09-15T20:36:33.469 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 42949672975' 2024-09-15T20:36:33.469 INFO:tasks.workunit.client.0.smithi136.stderr:waiting osd.1 seq 42949672975 2024-09-15T20:36:33.470 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-15T20:36:33.756 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 42949672975 -lt 42949672975 2024-09-15T20:36:33.756 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T20:36:33.756 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-64424509451 2024-09-15T20:36:33.756 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T20:36:33.757 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-15T20:36:33.758 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-64424509451 2024-09-15T20:36:33.758 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T20:36:33.759 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=64424509451 2024-09-15T20:36:33.759 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 64424509451' 2024-09-15T20:36:33.759 INFO:tasks.workunit.client.0.smithi136.stderr:waiting osd.2 seq 64424509451 2024-09-15T20:36:33.759 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-15T20:36:34.078 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 64424509451 -lt 64424509451 2024-09-15T20:36:34.078 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-15T20:36:34.078 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T20:36:34.078 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T20:36:34.424 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 1 == 0 2024-09-15T20:36:34.424 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-15T20:36:34.425 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-15T20:36:34.425 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-15T20:36:34.425 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T20:36:34.425 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T20:36:34.425 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T20:36:34.425 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T20:36:34.700 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=0 2024-09-15T20:36:34.700 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-15T20:36:34.700 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T20:36:34.700 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T20:36:35.043 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 0 = 1 2024-09-15T20:36:35.043 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 0 '!=' -1 2024-09-15T20:36:35.043 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: loop=0 2024-09-15T20:36:35.043 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1652: wait_for_clean: num_active_clean=0 2024-09-15T20:36:35.043 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-15T20:36:35.043 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 0.1 2024-09-15T20:36:35.144 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-15T20:36:35.144 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-15T20:36:35.144 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-15T20:36:35.145 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-15T20:36:35.145 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T20:36:35.145 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T20:36:35.145 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T20:36:35.145 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T20:36:35.417 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=0 2024-09-15T20:36:35.417 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-15T20:36:35.417 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T20:36:35.417 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T20:36:35.764 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 0 = 1 2024-09-15T20:36:35.765 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 0 '!=' 0 2024-09-15T20:36:35.765 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: get_is_making_recovery_progress 2024-09-15T20:36:35.765 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2024-09-15T20:36:35.765 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-15T20:36:35.765 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-15T20:36:35.765 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-15T20:36:35.765 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2024-09-15T20:36:35.765 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-15T20:36:36.106 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=null 2024-09-15T20:36:36.107 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test null '!=' null 2024-09-15T20:36:36.107 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1655: wait_for_clean: (( 1 >= 13 )) 2024-09-15T20:36:36.107 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-15T20:36:36.107 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 0.2 2024-09-15T20:36:36.308 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-15T20:36:36.308 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-15T20:36:36.308 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-15T20:36:36.308 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-15T20:36:36.308 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T20:36:36.308 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T20:36:36.308 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T20:36:36.309 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T20:36:36.619 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=0 2024-09-15T20:36:36.620 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-15T20:36:36.620 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T20:36:36.620 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T20:36:36.966 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 0 = 1 2024-09-15T20:36:36.966 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 0 '!=' 0 2024-09-15T20:36:36.966 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: get_is_making_recovery_progress 2024-09-15T20:36:36.966 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2024-09-15T20:36:36.966 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-15T20:36:36.966 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-15T20:36:36.966 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-15T20:36:36.967 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2024-09-15T20:36:36.967 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-15T20:36:37.338 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=null 2024-09-15T20:36:37.338 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test null '!=' null 2024-09-15T20:36:37.338 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1655: wait_for_clean: (( 2 >= 13 )) 2024-09-15T20:36:37.338 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-15T20:36:37.338 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 0.4 2024-09-15T20:36:37.740 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-15T20:36:37.740 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-15T20:36:37.740 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-15T20:36:37.740 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-15T20:36:37.740 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T20:36:37.740 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T20:36:37.740 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T20:36:37.740 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T20:36:38.009 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=1 2024-09-15T20:36:38.009 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-15T20:36:38.009 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T20:36:38.009 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T20:36:38.354 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 = 1 2024-09-15T20:36:38.354 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-15T20:36:38.354 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-15T20:36:38.354 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:633: TEST_auto_repair_bluestore_failed_norecov: get_primary testpool SOMETHING 2024-09-15T20:36:38.354 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1187: get_primary: local poolname=testpool 2024-09-15T20:36:38.355 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1188: get_primary: local objectname=SOMETHING 2024-09-15T20:36:38.355 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1190: get_primary: ceph --format json osd map testpool SOMETHING 2024-09-15T20:36:38.355 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1191: get_primary: jq .acting_primary 2024-09-15T20:36:38.635 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:633: TEST_auto_repair_bluestore_failed_norecov: objectstore_tool td/osd-scrub-repair 1 obj1 rm-attr _ 2024-09-15T20:36:38.635 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2024-09-15T20:36:38.635 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2024-09-15T20:36:38.635 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=1 2024-09-15T20:36:38.635 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2024-09-15T20:36:38.635 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 obj1 rm-attr _ 2024-09-15T20:36:38.636 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-09-15T20:36:38.636 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2024-09-15T20:36:38.636 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=1 2024-09-15T20:36:38.636 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2024-09-15T20:36:38.636 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-09-15T20:36:38.636 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-15T20:36:38.636 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-15T20:36:38.636 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-15T20:36:38.636 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-15T20:36:38.636 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-15T20:36:38.941 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-15T20:36:38.941 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 obj1 rm-attr _ 2024-09-15T20:36:38.941 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-15T20:36:38.941 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-15T20:36:38.942 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=1 2024-09-15T20:36:38.942 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-15T20:36:38.942 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-09-15T20:36:38.942 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 obj1 rm-attr _ 2024-09-15T20:36:40.903 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-09-15T20:36:40.903 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2024-09-15T20:36:40.903 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-15T20:36:40.903 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=1 2024-09-15T20:36:40.903 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-15T20:36:40.903 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-09-15T20:36:40.903 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=ce4892c7-6c6d-4b4f-8e6f-07a3e11133e5 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-15T20:36:40.903 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T20:36:40.903 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T20:36:40.904 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T20:36:40.904 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-09-15T20:36:40.904 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-09-15T20:36:40.904 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-15T20:36:40.904 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-15T20:36:40.904 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-15T20:36:40.904 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-15T20:36:40.904 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-15T20:36:40.904 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-15T20:36:40.904 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-15T20:36:40.904 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T20:36:40.904 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52602 2024-09-15T20:36:40.904 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.52602/$cluster-$name.asok' 2024-09-15T20:36:40.904 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.52602/$cluster-$name.asok' 2024-09-15T20:36:40.905 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-15T20:36:40.905 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-15T20:36:40.905 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-15T20:36:40.905 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T20:36:40.905 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T20:36:40.905 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T20:36:40.905 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T20:36:40.905 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-15T20:36:40.905 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+='--osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0' 2024-09-15T20:36:40.905 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-09-15T20:36:40.906 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.1 2024-09-15T20:36:40.906 INFO:tasks.workunit.client.0.smithi136.stderr:start osd.1 2024-09-15T20:36:40.906 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 1 --fsid=ce4892c7-6c6d-4b4f-8e6f-07a3e11133e5 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.52602/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-09-15T20:36:40.906 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-09-15T20:36:40.907 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 1 = 1 ']' 2024-09-15T20:36:40.907 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-15T20:36:40.907 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-15T20:36:40.907 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-15T20:36:40.937 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:36:40.935+0000 7f051082c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T20:36:40.939 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:36:40.937+0000 7f051082c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T20:36:40.940 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:36:40.938+0000 7f051082c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T20:36:41.190 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 1 2024-09-15T20:36:41.190 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-15T20:36:41.190 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2024-09-15T20:36:41.190 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-15T20:36:41.190 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-15T20:36:41.190 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:36:41.190 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-15T20:36:41.190 INFO:tasks.workunit.client.0.smithi136.stderr:0 2024-09-15T20:36:41.190 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:36:41.190 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T20:36:41.490 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:36:42.304 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:36:42.302+0000 7f051082c640 -1 Falling back to public interface 2024-09-15T20:36:42.492 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:36:42.492 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:36:42.492 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-15T20:36:42.492 INFO:tasks.workunit.client.0.smithi136.stderr:1 2024-09-15T20:36:42.492 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:36:42.492 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T20:36:42.770 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:36:43.772 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:36:43.772 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:36:43.772 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-15T20:36:43.772 INFO:tasks.workunit.client.0.smithi136.stderr:2 2024-09-15T20:36:43.772 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:36:43.772 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T20:36:44.054 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:36:44.968 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:36:44.966+0000 7f051082c640 -1 osd.1 26 log_to_monitors true 2024-09-15T20:36:45.055 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:36:45.056 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:36:45.056 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-15T20:36:45.056 INFO:tasks.workunit.client.0.smithi136.stderr:3 2024-09-15T20:36:45.056 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:36:45.056 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T20:36:45.396 INFO:tasks.workunit.client.0.smithi136.stderr:osd.1 up in weight 1 up_from 30 up_thru 25 down_at 27 last_clean_interval [10,26) [v2:127.0.0.1:6810/2603851603,v1:127.0.0.1:6811/2603851603] [v2:127.0.0.1:6812/2603851603,v1:127.0.0.1:6813/2603851603] exists,up a3cff831-3849-4205-bcdf-59d3a90a6d99 2024-09-15T20:36:45.396 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-15T20:36:45.396 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-15T20:36:45.397 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-15T20:36:45.397 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2024-09-15T20:36:45.397 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-15T20:36:45.397 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-15T20:36:45.397 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-15T20:36:45.397 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-15T20:36:45.397 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-15T20:36:45.397 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-15T20:36:45.397 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-15T20:36:45.397 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-15T20:36:45.397 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-15T20:36:45.486 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-15T20:36:45.486 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-15T20:36:45.486 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-15T20:36:45.486 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-15T20:36:45.487 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-15T20:36:45.487 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-15T20:36:45.770 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-15T20:36:45.771 INFO:tasks.workunit.client.0.smithi136.stderr:1 2024-09-15T20:36:45.771 INFO:tasks.workunit.client.0.smithi136.stderr:2' 2024-09-15T20:36:45.771 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-15T20:36:45.771 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T20:36:45.771 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-15T20:36:45.874 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=107374182405 2024-09-15T20:36:45.874 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 107374182405 2024-09-15T20:36:45.874 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-107374182405' 2024-09-15T20:36:45.874 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T20:36:45.874 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-15T20:36:46.005 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=128849018882 2024-09-15T20:36:46.006 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 128849018882 2024-09-15T20:36:46.006 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-107374182405 1-128849018882' 2024-09-15T20:36:46.006 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T20:36:46.006 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-15T20:36:46.109 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=64424509455 2024-09-15T20:36:46.109 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 64424509455 2024-09-15T20:36:46.109 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-107374182405 1-128849018882 2-64424509455' 2024-09-15T20:36:46.109 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T20:36:46.109 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-107374182405 2024-09-15T20:36:46.110 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T20:36:46.110 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-15T20:36:46.111 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-107374182405 2024-09-15T20:36:46.111 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T20:36:46.112 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=107374182405 2024-09-15T20:36:46.112 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 107374182405' 2024-09-15T20:36:46.112 INFO:tasks.workunit.client.0.smithi136.stderr:waiting osd.0 seq 107374182405 2024-09-15T20:36:46.112 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T20:36:46.385 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 107374182404 -lt 107374182405 2024-09-15T20:36:46.386 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-15T20:36:47.387 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-15T20:36:47.387 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T20:36:47.670 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 107374182406 -lt 107374182405 2024-09-15T20:36:47.671 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T20:36:47.671 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-128849018882 2024-09-15T20:36:47.671 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T20:36:47.672 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-15T20:36:47.673 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-128849018882 2024-09-15T20:36:47.673 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T20:36:47.674 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=128849018882 2024-09-15T20:36:47.674 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 128849018882' 2024-09-15T20:36:47.674 INFO:tasks.workunit.client.0.smithi136.stderr:waiting osd.1 seq 128849018882 2024-09-15T20:36:47.674 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-15T20:36:47.950 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 128849018882 -lt 128849018882 2024-09-15T20:36:47.950 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T20:36:47.951 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-64424509455 2024-09-15T20:36:47.951 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T20:36:47.952 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-15T20:36:47.952 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-64424509455 2024-09-15T20:36:47.953 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T20:36:47.953 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=64424509455 2024-09-15T20:36:47.953 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 64424509455' 2024-09-15T20:36:47.954 INFO:tasks.workunit.client.0.smithi136.stderr:waiting osd.2 seq 64424509455 2024-09-15T20:36:47.954 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-15T20:36:48.300 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 64424509455 -lt 64424509455 2024-09-15T20:36:48.301 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-15T20:36:48.301 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T20:36:48.301 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T20:36:48.645 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 1 == 0 2024-09-15T20:36:48.646 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-15T20:36:48.646 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-15T20:36:48.646 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-15T20:36:48.646 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T20:36:48.646 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T20:36:48.646 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T20:36:48.646 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T20:36:48.919 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=0 2024-09-15T20:36:48.919 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-15T20:36:48.919 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T20:36:48.920 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T20:36:49.263 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 0 = 1 2024-09-15T20:36:49.263 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 0 '!=' -1 2024-09-15T20:36:49.263 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: loop=0 2024-09-15T20:36:49.264 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1652: wait_for_clean: num_active_clean=0 2024-09-15T20:36:49.264 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-15T20:36:49.264 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 0.1 2024-09-15T20:36:49.365 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-15T20:36:49.365 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-15T20:36:49.365 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-15T20:36:49.365 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-15T20:36:49.365 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T20:36:49.365 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T20:36:49.365 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T20:36:49.365 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T20:36:49.637 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=0 2024-09-15T20:36:49.637 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-15T20:36:49.638 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T20:36:49.638 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T20:36:49.980 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 0 = 1 2024-09-15T20:36:49.981 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 0 '!=' 0 2024-09-15T20:36:49.981 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: get_is_making_recovery_progress 2024-09-15T20:36:49.981 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2024-09-15T20:36:49.981 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-15T20:36:49.981 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-15T20:36:49.981 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-15T20:36:49.981 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2024-09-15T20:36:49.981 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-15T20:36:50.326 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=null 2024-09-15T20:36:50.326 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test null '!=' null 2024-09-15T20:36:50.326 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1655: wait_for_clean: (( 1 >= 13 )) 2024-09-15T20:36:50.326 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-15T20:36:50.326 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 0.2 2024-09-15T20:36:50.527 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-15T20:36:50.528 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-15T20:36:50.528 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-15T20:36:50.528 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-15T20:36:50.528 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T20:36:50.528 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T20:36:50.528 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T20:36:50.528 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T20:36:50.802 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=0 2024-09-15T20:36:50.802 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-15T20:36:50.802 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T20:36:50.802 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T20:36:51.145 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 0 = 1 2024-09-15T20:36:51.145 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 0 '!=' 0 2024-09-15T20:36:51.145 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: get_is_making_recovery_progress 2024-09-15T20:36:51.145 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2024-09-15T20:36:51.146 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-15T20:36:51.146 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-15T20:36:51.146 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-15T20:36:51.146 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2024-09-15T20:36:51.146 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-15T20:36:51.495 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=null 2024-09-15T20:36:51.495 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test null '!=' null 2024-09-15T20:36:51.495 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1655: wait_for_clean: (( 2 >= 13 )) 2024-09-15T20:36:51.495 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-15T20:36:51.495 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 0.4 2024-09-15T20:36:51.896 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-15T20:36:51.896 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-15T20:36:51.897 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-15T20:36:51.897 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-15T20:36:51.897 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T20:36:51.897 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T20:36:51.897 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T20:36:51.897 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T20:36:52.172 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=1 2024-09-15T20:36:52.259 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-15T20:36:52.259 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T20:36:52.260 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T20:36:52.520 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 = 1 2024-09-15T20:36:52.520 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-15T20:36:52.520 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-15T20:36:52.521 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:635: TEST_auto_repair_bluestore_failed_norecov: get_not_primary testpool SOMETHING 2024-09-15T20:36:52.521 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1220: get_not_primary: local poolname=testpool 2024-09-15T20:36:52.521 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1221: get_not_primary: local objectname=SOMETHING 2024-09-15T20:36:52.521 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1223: get_not_primary: get_primary testpool SOMETHING 2024-09-15T20:36:52.521 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1187: get_primary: local poolname=testpool 2024-09-15T20:36:52.521 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1188: get_primary: local objectname=SOMETHING 2024-09-15T20:36:52.521 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1190: get_primary: ceph --format json osd map testpool SOMETHING 2024-09-15T20:36:52.521 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1191: get_primary: jq .acting_primary 2024-09-15T20:36:52.804 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1223: get_not_primary: local primary=1 2024-09-15T20:36:52.805 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1224: get_not_primary: ceph --format json osd map testpool SOMETHING 2024-09-15T20:36:52.805 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1225: get_not_primary: jq '.acting | map(select (. != 1)) | .[0]' 2024-09-15T20:36:53.085 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:635: TEST_auto_repair_bluestore_failed_norecov: objectstore_tool td/osd-scrub-repair 0 obj2 remove 2024-09-15T20:36:53.085 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2024-09-15T20:36:53.085 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2024-09-15T20:36:53.085 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=0 2024-09-15T20:36:53.085 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2024-09-15T20:36:53.086 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 obj2 remove 2024-09-15T20:36:53.086 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-09-15T20:36:53.086 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2024-09-15T20:36:53.086 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=0 2024-09-15T20:36:53.086 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2024-09-15T20:36:53.086 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2024-09-15T20:36:53.086 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-15T20:36:53.087 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-15T20:36:53.087 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-15T20:36:53.087 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-15T20:36:53.087 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-15T20:36:53.391 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-15T20:36:53.391 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 obj2 remove 2024-09-15T20:36:53.391 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-15T20:36:53.391 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-15T20:36:53.391 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=0 2024-09-15T20:36:53.392 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-15T20:36:53.392 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-09-15T20:36:53.392 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 obj2 remove 2024-09-15T20:36:54.969 INFO:tasks.workunit.client.0.smithi136.stdout:remove #1:104778fc:::obj2:head# 2024-09-15T20:36:55.487 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-09-15T20:36:55.488 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2024-09-15T20:36:55.488 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-15T20:36:55.488 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=0 2024-09-15T20:36:55.488 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-15T20:36:55.488 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-09-15T20:36:55.488 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=ce4892c7-6c6d-4b4f-8e6f-07a3e11133e5 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-15T20:36:55.488 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T20:36:55.488 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T20:36:55.488 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T20:36:55.489 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-09-15T20:36:55.489 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-09-15T20:36:55.489 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-15T20:36:55.489 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-15T20:36:55.489 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-15T20:36:55.489 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-15T20:36:55.489 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-15T20:36:55.489 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-15T20:36:55.489 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-15T20:36:55.489 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T20:36:55.489 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52602 2024-09-15T20:36:55.489 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.52602/$cluster-$name.asok' 2024-09-15T20:36:55.489 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.52602/$cluster-$name.asok' 2024-09-15T20:36:55.490 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-15T20:36:55.490 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-15T20:36:55.490 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-15T20:36:55.490 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T20:36:55.490 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T20:36:55.490 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T20:36:55.490 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T20:36:55.490 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-15T20:36:55.490 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+='--osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0' 2024-09-15T20:36:55.490 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-09-15T20:36:55.490 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.0 2024-09-15T20:36:55.490 INFO:tasks.workunit.client.0.smithi136.stderr:start osd.0 2024-09-15T20:36:55.490 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 0 --fsid=ce4892c7-6c6d-4b4f-8e6f-07a3e11133e5 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.52602/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-09-15T20:36:55.491 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-09-15T20:36:55.491 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 0 = 0 ']' 2024-09-15T20:36:55.491 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-15T20:36:55.491 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-15T20:36:55.491 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-15T20:36:55.520 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:36:55.518+0000 7fab29f27640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T20:36:55.522 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:36:55.520+0000 7fab29f27640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T20:36:55.524 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:36:55.522+0000 7fab29f27640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T20:36:55.776 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 0 2024-09-15T20:36:55.776 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-15T20:36:55.776 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2024-09-15T20:36:55.776 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-15T20:36:55.776 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-15T20:36:55.776 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:36:55.776 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-15T20:36:55.776 INFO:tasks.workunit.client.0.smithi136.stderr:0 2024-09-15T20:36:55.777 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:36:55.777 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T20:36:56.054 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:36:56.896 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:36:56.894+0000 7fab29f27640 -1 Falling back to public interface 2024-09-15T20:36:57.055 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:36:57.056 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:36:57.056 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-15T20:36:57.056 INFO:tasks.workunit.client.0.smithi136.stderr:1 2024-09-15T20:36:57.056 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:36:57.056 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T20:36:57.341 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:36:58.343 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:36:58.344 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:36:58.344 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-15T20:36:58.344 INFO:tasks.workunit.client.0.smithi136.stderr:2 2024-09-15T20:36:58.344 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:36:58.344 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T20:36:58.632 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:36:59.633 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:36:59.633 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:36:59.633 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-15T20:36:59.634 INFO:tasks.workunit.client.0.smithi136.stderr:3 2024-09-15T20:36:59.634 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:36:59.634 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T20:36:59.636 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:36:59.634+0000 7fab29f27640 -1 osd.0 31 log_to_monitors true 2024-09-15T20:36:59.922 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:37:00.924 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:37:00.924 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:37:00.924 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-15T20:37:00.924 INFO:tasks.workunit.client.0.smithi136.stderr:4 2024-09-15T20:37:00.924 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:37:00.924 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T20:37:01.210 INFO:tasks.workunit.client.0.smithi136.stderr:osd.0 up in weight 1 up_from 35 up_thru 27 down_at 32 last_clean_interval [25,31) [v2:127.0.0.1:6802/925303394,v1:127.0.0.1:6803/925303394] [v2:127.0.0.1:6804/925303394,v1:127.0.0.1:6805/925303394] exists,up 8e99504b-f154-418f-a155-6debf8edaca9 2024-09-15T20:37:01.210 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-15T20:37:01.210 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-15T20:37:01.210 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-15T20:37:01.210 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2024-09-15T20:37:01.210 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-15T20:37:01.210 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-15T20:37:01.210 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-15T20:37:01.210 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-15T20:37:01.211 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-15T20:37:01.211 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-15T20:37:01.211 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-15T20:37:01.211 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-15T20:37:01.211 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-15T20:37:01.301 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-15T20:37:01.301 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-15T20:37:01.301 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-15T20:37:01.301 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-15T20:37:01.301 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-15T20:37:01.301 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-15T20:37:01.580 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-15T20:37:01.580 INFO:tasks.workunit.client.0.smithi136.stderr:1 2024-09-15T20:37:01.580 INFO:tasks.workunit.client.0.smithi136.stderr:2' 2024-09-15T20:37:01.580 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-15T20:37:01.580 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T20:37:01.581 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-15T20:37:01.684 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=150323855362 2024-09-15T20:37:01.684 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 150323855362 2024-09-15T20:37:01.684 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-150323855362' 2024-09-15T20:37:01.684 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T20:37:01.685 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-15T20:37:01.787 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=128849018886 2024-09-15T20:37:01.787 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 128849018886 2024-09-15T20:37:01.787 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-150323855362 1-128849018886' 2024-09-15T20:37:01.787 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T20:37:01.787 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-15T20:37:01.891 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=64424509459 2024-09-15T20:37:01.891 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 64424509459 2024-09-15T20:37:01.891 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-150323855362 1-128849018886 2-64424509459' 2024-09-15T20:37:01.891 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T20:37:01.892 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-150323855362 2024-09-15T20:37:01.892 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T20:37:01.892 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-15T20:37:01.893 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-150323855362 2024-09-15T20:37:01.893 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T20:37:01.894 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=150323855362 2024-09-15T20:37:01.894 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 150323855362' 2024-09-15T20:37:01.894 INFO:tasks.workunit.client.0.smithi136.stderr:waiting osd.0 seq 150323855362 2024-09-15T20:37:01.894 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T20:37:02.168 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 1 -lt 150323855362 2024-09-15T20:37:02.169 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-15T20:37:03.170 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-15T20:37:03.170 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T20:37:03.479 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 150323855362 -lt 150323855362 2024-09-15T20:37:03.479 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T20:37:03.479 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-128849018886 2024-09-15T20:37:03.479 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T20:37:03.480 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-15T20:37:03.481 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-128849018886 2024-09-15T20:37:03.481 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T20:37:03.482 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=128849018886 2024-09-15T20:37:03.482 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 128849018886' 2024-09-15T20:37:03.482 INFO:tasks.workunit.client.0.smithi136.stderr:waiting osd.1 seq 128849018886 2024-09-15T20:37:03.482 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-15T20:37:03.759 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 128849018886 -lt 128849018886 2024-09-15T20:37:03.759 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T20:37:03.760 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-64424509459 2024-09-15T20:37:03.760 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T20:37:03.760 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-15T20:37:03.761 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-64424509459 2024-09-15T20:37:03.761 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T20:37:03.762 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=64424509459 2024-09-15T20:37:03.762 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 64424509459' 2024-09-15T20:37:03.762 INFO:tasks.workunit.client.0.smithi136.stderr:waiting osd.2 seq 64424509459 2024-09-15T20:37:03.763 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-15T20:37:04.040 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 64424509459 -lt 64424509459 2024-09-15T20:37:04.040 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-15T20:37:04.041 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T20:37:04.041 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T20:37:04.386 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 1 == 0 2024-09-15T20:37:04.386 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-15T20:37:04.386 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-15T20:37:04.386 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-15T20:37:04.386 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T20:37:04.386 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T20:37:04.386 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T20:37:04.386 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T20:37:04.661 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=1 2024-09-15T20:37:04.661 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-15T20:37:04.662 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T20:37:04.662 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T20:37:05.005 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 = 1 2024-09-15T20:37:05.006 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-15T20:37:05.006 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-15T20:37:05.006 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:636: TEST_auto_repair_bluestore_failed_norecov: get_primary testpool SOMETHING 2024-09-15T20:37:05.006 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1187: get_primary: local poolname=testpool 2024-09-15T20:37:05.006 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1188: get_primary: local objectname=SOMETHING 2024-09-15T20:37:05.006 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1190: get_primary: ceph --format json osd map testpool SOMETHING 2024-09-15T20:37:05.006 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1191: get_primary: jq .acting_primary 2024-09-15T20:37:05.288 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:636: TEST_auto_repair_bluestore_failed_norecov: objectstore_tool td/osd-scrub-repair 1 obj2 rm-attr _ 2024-09-15T20:37:05.289 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2024-09-15T20:37:05.289 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2024-09-15T20:37:05.289 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=1 2024-09-15T20:37:05.289 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2024-09-15T20:37:05.289 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 obj2 rm-attr _ 2024-09-15T20:37:05.289 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-09-15T20:37:05.289 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2024-09-15T20:37:05.289 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=1 2024-09-15T20:37:05.289 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2024-09-15T20:37:05.289 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-09-15T20:37:05.289 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-15T20:37:05.289 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-15T20:37:05.289 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-15T20:37:05.289 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-15T20:37:05.290 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-15T20:37:05.595 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-15T20:37:05.595 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 obj2 rm-attr _ 2024-09-15T20:37:05.595 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-15T20:37:05.595 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-15T20:37:05.595 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=1 2024-09-15T20:37:05.595 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-15T20:37:05.596 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-09-15T20:37:05.596 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 obj2 rm-attr _ 2024-09-15T20:37:06.926 INFO:tasks.workunit.client.0.smithi136.stderr:Error getting attr on : 1.0_head,#1:ff7b1f36:::obj1:head#, (61) No data available 2024-09-15T20:37:07.454 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-09-15T20:37:07.455 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2024-09-15T20:37:07.455 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-15T20:37:07.455 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=1 2024-09-15T20:37:07.455 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-15T20:37:07.455 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-09-15T20:37:07.455 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=ce4892c7-6c6d-4b4f-8e6f-07a3e11133e5 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-15T20:37:07.455 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T20:37:07.455 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T20:37:07.455 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T20:37:07.455 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-09-15T20:37:07.455 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-09-15T20:37:07.456 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-15T20:37:07.456 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-15T20:37:07.456 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-15T20:37:07.456 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-15T20:37:07.456 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-15T20:37:07.456 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-15T20:37:07.456 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-15T20:37:07.456 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T20:37:07.456 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52602 2024-09-15T20:37:07.456 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.52602/$cluster-$name.asok' 2024-09-15T20:37:07.456 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.52602/$cluster-$name.asok' 2024-09-15T20:37:07.456 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-15T20:37:07.456 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-15T20:37:07.457 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-15T20:37:07.457 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T20:37:07.457 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T20:37:07.457 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T20:37:07.457 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T20:37:07.457 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-15T20:37:07.457 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+='--osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0' 2024-09-15T20:37:07.457 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-09-15T20:37:07.458 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.1 2024-09-15T20:37:07.458 INFO:tasks.workunit.client.0.smithi136.stderr:start osd.1 2024-09-15T20:37:07.458 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 1 --fsid=ce4892c7-6c6d-4b4f-8e6f-07a3e11133e5 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.52602/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-09-15T20:37:07.458 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-09-15T20:37:07.458 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 1 = 1 ']' 2024-09-15T20:37:07.458 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-15T20:37:07.458 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-15T20:37:07.458 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-15T20:37:07.488 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:37:07.486+0000 7f44b6e19640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T20:37:07.490 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:37:07.489+0000 7f44b6e19640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T20:37:07.492 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:37:07.490+0000 7f44b6e19640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T20:37:07.743 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 1 2024-09-15T20:37:07.743 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-15T20:37:07.743 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2024-09-15T20:37:07.743 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-15T20:37:07.743 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-15T20:37:07.743 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:37:07.743 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-15T20:37:07.744 INFO:tasks.workunit.client.0.smithi136.stderr:0 2024-09-15T20:37:07.744 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:37:07.744 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T20:37:08.016 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:37:08.600 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:37:08.598+0000 7f44b6e19640 -1 Falling back to public interface 2024-09-15T20:37:09.018 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:37:09.018 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:37:09.018 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-15T20:37:09.018 INFO:tasks.workunit.client.0.smithi136.stderr:1 2024-09-15T20:37:09.018 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:37:09.018 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T20:37:09.305 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:37:10.307 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:37:10.307 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:37:10.307 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-15T20:37:10.307 INFO:tasks.workunit.client.0.smithi136.stderr:2 2024-09-15T20:37:10.307 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:37:10.307 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T20:37:10.608 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:37:11.190 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:37:11.188+0000 7f44b6e19640 -1 osd.1 36 log_to_monitors true 2024-09-15T20:37:11.609 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:37:11.609 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:37:11.610 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-15T20:37:11.610 INFO:tasks.workunit.client.0.smithi136.stderr:3 2024-09-15T20:37:11.610 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:37:11.610 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T20:37:11.973 INFO:tasks.workunit.client.0.smithi136.stderr:osd.1 up in weight 1 up_from 40 up_thru 40 down_at 37 last_clean_interval [30,36) [v2:127.0.0.1:6810/206294033,v1:127.0.0.1:6811/206294033] [v2:127.0.0.1:6812/206294033,v1:127.0.0.1:6813/206294033] exists,up a3cff831-3849-4205-bcdf-59d3a90a6d99 2024-09-15T20:37:11.973 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-15T20:37:11.973 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-15T20:37:11.973 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-15T20:37:11.973 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2024-09-15T20:37:11.974 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-15T20:37:11.974 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-15T20:37:11.974 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-15T20:37:11.974 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-15T20:37:11.974 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-15T20:37:11.974 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-15T20:37:11.974 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-15T20:37:11.974 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-15T20:37:11.974 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-15T20:37:12.062 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-15T20:37:12.062 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-15T20:37:12.062 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-15T20:37:12.063 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-15T20:37:12.063 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-15T20:37:12.063 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-15T20:37:12.419 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-15T20:37:12.420 INFO:tasks.workunit.client.0.smithi136.stderr:1 2024-09-15T20:37:12.420 INFO:tasks.workunit.client.0.smithi136.stderr:2' 2024-09-15T20:37:12.420 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-15T20:37:12.420 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T20:37:12.420 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-15T20:37:12.523 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=150323855365 2024-09-15T20:37:12.523 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 150323855365 2024-09-15T20:37:12.523 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-150323855365' 2024-09-15T20:37:12.523 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T20:37:12.523 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-15T20:37:12.627 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=171798691842 2024-09-15T20:37:12.627 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 171798691842 2024-09-15T20:37:12.627 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-150323855365 1-171798691842' 2024-09-15T20:37:12.627 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T20:37:12.627 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-15T20:37:12.729 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=64424509462 2024-09-15T20:37:12.729 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 64424509462 2024-09-15T20:37:12.729 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-150323855365 1-171798691842 2-64424509462' 2024-09-15T20:37:12.729 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T20:37:12.729 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-150323855365 2024-09-15T20:37:12.729 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T20:37:12.730 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-15T20:37:12.731 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-150323855365 2024-09-15T20:37:12.731 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T20:37:12.732 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=150323855365 2024-09-15T20:37:12.732 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 150323855365' 2024-09-15T20:37:12.732 INFO:tasks.workunit.client.0.smithi136.stderr:waiting osd.0 seq 150323855365 2024-09-15T20:37:12.732 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T20:37:13.004 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 150323855364 -lt 150323855365 2024-09-15T20:37:13.004 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-15T20:37:14.006 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-15T20:37:14.006 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T20:37:14.293 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 150323855365 -lt 150323855365 2024-09-15T20:37:14.293 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T20:37:14.294 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-171798691842 2024-09-15T20:37:14.294 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T20:37:14.294 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-15T20:37:14.295 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-171798691842 2024-09-15T20:37:14.295 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T20:37:14.296 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=171798691842 2024-09-15T20:37:14.296 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 171798691842' 2024-09-15T20:37:14.296 INFO:tasks.workunit.client.0.smithi136.stderr:waiting osd.1 seq 171798691842 2024-09-15T20:37:14.296 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-15T20:37:14.571 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 171798691842 -lt 171798691842 2024-09-15T20:37:14.571 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T20:37:14.572 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-64424509462 2024-09-15T20:37:14.572 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T20:37:14.573 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-15T20:37:14.573 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-64424509462 2024-09-15T20:37:14.573 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T20:37:14.574 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=64424509462 2024-09-15T20:37:14.574 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 64424509462' 2024-09-15T20:37:14.574 INFO:tasks.workunit.client.0.smithi136.stderr:waiting osd.2 seq 64424509462 2024-09-15T20:37:14.575 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-15T20:37:14.854 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 64424509462 -lt 64424509462 2024-09-15T20:37:14.855 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-15T20:37:14.855 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T20:37:14.855 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T20:37:15.201 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 1 == 0 2024-09-15T20:37:15.201 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-15T20:37:15.201 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-15T20:37:15.201 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-15T20:37:15.201 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T20:37:15.202 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T20:37:15.202 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T20:37:15.202 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T20:37:15.477 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=1 2024-09-15T20:37:15.477 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-15T20:37:15.477 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T20:37:15.477 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T20:37:15.820 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 = 1 2024-09-15T20:37:15.821 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-15T20:37:15.821 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-15T20:37:15.821 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:637: TEST_auto_repair_bluestore_failed_norecov: ceph tell 'osd.*' config set osd_scrub_auto_repair true 2024-09-15T20:37:15.916 INFO:tasks.workunit.client.0.smithi136.stdout:osd.0: { 2024-09-15T20:37:15.916 INFO:tasks.workunit.client.0.smithi136.stdout: "success": "osd_delete_sleep = '' osd_delete_sleep_hdd = '' osd_delete_sleep_hybrid = '' osd_delete_sleep_ssd = '' osd_max_backfills = '' osd_recovery_max_active = '' osd_recovery_max_active_hdd = '' osd_recovery_max_active_ssd = '' osd_recovery_sleep = '' osd_recovery_sleep_hdd = '' osd_recovery_sleep_hybrid = '' osd_recovery_sleep_ssd = '' osd_scrub_auto_repair = '' (not observed, change may require restart) osd_scrub_sleep = '' osd_snap_trim_sleep = '' osd_snap_trim_sleep_hdd = '' osd_snap_trim_sleep_hybrid = '' osd_snap_trim_sleep_ssd = '' " 2024-09-15T20:37:15.916 INFO:tasks.workunit.client.0.smithi136.stdout:} 2024-09-15T20:37:15.929 INFO:tasks.workunit.client.0.smithi136.stdout:osd.1: { 2024-09-15T20:37:15.929 INFO:tasks.workunit.client.0.smithi136.stdout: "success": "osd_delete_sleep = '' osd_delete_sleep_hdd = '' osd_delete_sleep_hybrid = '' osd_delete_sleep_ssd = '' osd_max_backfills = '' osd_recovery_max_active = '' osd_recovery_max_active_hdd = '' osd_recovery_max_active_ssd = '' osd_recovery_sleep = '' osd_recovery_sleep_hdd = '' osd_recovery_sleep_hybrid = '' osd_recovery_sleep_ssd = '' osd_scrub_auto_repair = '' (not observed, change may require restart) osd_scrub_sleep = '' osd_snap_trim_sleep = '' osd_snap_trim_sleep_hdd = '' osd_snap_trim_sleep_hybrid = '' osd_snap_trim_sleep_ssd = '' " 2024-09-15T20:37:15.929 INFO:tasks.workunit.client.0.smithi136.stdout:} 2024-09-15T20:37:15.942 INFO:tasks.workunit.client.0.smithi136.stdout:osd.2: { 2024-09-15T20:37:15.942 INFO:tasks.workunit.client.0.smithi136.stdout: "success": "osd_scrub_auto_repair = '' (not observed, change may require restart) " 2024-09-15T20:37:15.942 INFO:tasks.workunit.client.0.smithi136.stdout:} 2024-09-15T20:37:15.952 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:639: TEST_auto_repair_bluestore_failed_norecov: get_pg testpool obj1 2024-09-15T20:37:15.952 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1084: get_pg: local poolname=testpool 2024-09-15T20:37:15.952 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1085: get_pg: local objectname=obj1 2024-09-15T20:37:15.952 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1087: get_pg: ceph --format json osd map testpool obj1 2024-09-15T20:37:15.952 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1087: get_pg: jq -r .pgid 2024-09-15T20:37:16.237 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:639: TEST_auto_repair_bluestore_failed_norecov: local pgid=1.0 2024-09-15T20:37:16.237 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:640: TEST_auto_repair_bluestore_failed_norecov: get_primary testpool obj1 2024-09-15T20:37:16.237 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1187: get_primary: local poolname=testpool 2024-09-15T20:37:16.237 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1188: get_primary: local objectname=obj1 2024-09-15T20:37:16.237 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1190: get_primary: ceph --format json osd map testpool obj1 2024-09-15T20:37:16.237 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1191: get_primary: jq .acting_primary 2024-09-15T20:37:16.548 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:640: TEST_auto_repair_bluestore_failed_norecov: local primary=1 2024-09-15T20:37:16.548 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:641: TEST_auto_repair_bluestore_failed_norecov: get_last_scrub_stamp 1.0 2024-09-15T20:37:16.549 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-15T20:37:16.549 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-15T20:37:16.549 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-15T20:37:16.549 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-15T20:37:16.822 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:641: TEST_auto_repair_bluestore_failed_norecov: local last_scrub_stamp=2024-09-15T20:36:14.901027+0000 2024-09-15T20:37:16.822 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:642: TEST_auto_repair_bluestore_failed_norecov: ceph tell 1.0 schedule-deep-scrub 2024-09-15T20:37:16.917 INFO:tasks.workunit.client.0.smithi136.stdout:{ 2024-09-15T20:37:16.918 INFO:tasks.workunit.client.0.smithi136.stdout: "deep": true, 2024-09-15T20:37:16.918 INFO:tasks.workunit.client.0.smithi136.stdout: "must": false, 2024-09-15T20:37:16.918 INFO:tasks.workunit.client.0.smithi136.stdout: "stamp": "2024-09-01T20:35:36.916475+0000" 2024-09-15T20:37:16.918 INFO:tasks.workunit.client.0.smithi136.stdout:} 2024-09-15T20:37:16.928 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:645: TEST_auto_repair_bluestore_failed_norecov: wait_for_scrub 1.0 2024-09-15T20:36:14.901027+0000 2024-09-15T20:37:16.928 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: local pgid=1.0 2024-09-15T20:37:16.928 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2018: wait_for_scrub: local last_scrub=2024-09-15T20:36:14.901027+0000 2024-09-15T20:37:16.928 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2019: wait_for_scrub: local sname=last_scrub_stamp 2024-09-15T20:37:16.928 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i=0 )) 2024-09-15T20:37:16.928 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-15T20:37:16.928 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-15T20:37:16.928 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-15T20:37:16.929 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-15T20:37:16.929 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-15T20:37:16.929 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-15T20:37:17.200 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-15T20:36:14.901027+0000 '>' 2024-09-15T20:36:14.901027+0000 2024-09-15T20:37:17.200 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-15T20:37:18.201 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-15T20:37:18.201 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-15T20:37:18.202 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-15T20:37:18.202 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-15T20:37:18.202 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-15T20:37:18.202 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-15T20:37:18.202 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-15T20:37:18.525 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-15T20:36:14.901027+0000 '>' 2024-09-15T20:36:14.901027+0000 2024-09-15T20:37:18.526 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-15T20:37:19.527 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-15T20:37:19.527 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-15T20:37:19.528 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-15T20:37:19.528 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-15T20:37:19.528 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-15T20:37:19.528 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-15T20:37:19.528 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-15T20:37:19.810 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-15T20:36:14.901027+0000 '>' 2024-09-15T20:36:14.901027+0000 2024-09-15T20:37:19.810 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-15T20:37:20.812 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-15T20:37:20.813 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-15T20:37:20.813 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-15T20:37:20.813 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-15T20:37:20.813 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-15T20:37:20.813 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-15T20:37:20.814 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-15T20:37:21.093 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-15T20:36:14.901027+0000 '>' 2024-09-15T20:36:14.901027+0000 2024-09-15T20:37:21.093 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-15T20:37:22.094 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-15T20:37:22.094 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-15T20:37:22.094 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-15T20:37:22.094 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-15T20:37:22.095 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-15T20:37:22.095 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-15T20:37:22.095 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-15T20:37:22.378 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-15T20:36:14.901027+0000 '>' 2024-09-15T20:36:14.901027+0000 2024-09-15T20:37:22.378 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-15T20:37:23.379 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-15T20:37:23.380 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-15T20:37:23.380 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-15T20:37:23.381 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-15T20:37:23.381 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-15T20:37:23.381 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-15T20:37:23.381 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-15T20:37:23.663 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-15T20:37:17.197067+0000 '>' 2024-09-15T20:36:14.901027+0000 2024-09-15T20:37:23.663 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2023: wait_for_scrub: return 0 2024-09-15T20:37:23.663 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:646: TEST_auto_repair_bluestore_failed_norecov: wait_for_clean 2024-09-15T20:37:23.663 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-15T20:37:23.663 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-15T20:37:23.664 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-15T20:37:23.664 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-15T20:37:23.664 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-15T20:37:23.664 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-15T20:37:23.664 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-15T20:37:23.665 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-15T20:37:23.665 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-15T20:37:23.752 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-15T20:37:23.753 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-15T20:37:23.753 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-15T20:37:23.753 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-15T20:37:23.755 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-15T20:37:23.755 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-15T20:37:24.029 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-15T20:37:24.029 INFO:tasks.workunit.client.0.smithi136.stderr:1 2024-09-15T20:37:24.030 INFO:tasks.workunit.client.0.smithi136.stderr:2' 2024-09-15T20:37:24.030 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-15T20:37:24.030 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T20:37:24.030 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-15T20:37:24.132 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=150323855368 2024-09-15T20:37:24.132 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 150323855368 2024-09-15T20:37:24.132 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-150323855368' 2024-09-15T20:37:24.133 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T20:37:24.133 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-15T20:37:24.235 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=171798691845 2024-09-15T20:37:24.235 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 171798691845 2024-09-15T20:37:24.236 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-150323855368 1-171798691845' 2024-09-15T20:37:24.236 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T20:37:24.236 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-15T20:37:24.339 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=64424509465 2024-09-15T20:37:24.339 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 64424509465 2024-09-15T20:37:24.339 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-150323855368 1-171798691845 2-64424509465' 2024-09-15T20:37:24.339 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T20:37:24.340 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-150323855368 2024-09-15T20:37:24.340 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T20:37:24.340 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-15T20:37:24.341 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-150323855368 2024-09-15T20:37:24.341 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T20:37:24.342 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=150323855368 2024-09-15T20:37:24.342 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 150323855368' 2024-09-15T20:37:24.342 INFO:tasks.workunit.client.0.smithi136.stdout:waiting osd.0 seq 150323855368 2024-09-15T20:37:24.343 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T20:37:24.647 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 150323855367 -lt 150323855368 2024-09-15T20:37:24.647 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-15T20:37:25.648 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-15T20:37:25.648 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T20:37:25.926 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 150323855369 -lt 150323855368 2024-09-15T20:37:25.926 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T20:37:25.927 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-171798691845 2024-09-15T20:37:25.927 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T20:37:25.928 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-15T20:37:25.928 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-171798691845 2024-09-15T20:37:25.928 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T20:37:25.929 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=171798691845 2024-09-15T20:37:25.929 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 171798691845' 2024-09-15T20:37:25.930 INFO:tasks.workunit.client.0.smithi136.stdout:waiting osd.1 seq 171798691845 2024-09-15T20:37:25.930 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-15T20:37:26.206 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 171798691845 -lt 171798691845 2024-09-15T20:37:26.206 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T20:37:26.206 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-64424509465 2024-09-15T20:37:26.206 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T20:37:26.207 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-15T20:37:26.208 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-64424509465 2024-09-15T20:37:26.208 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T20:37:26.209 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=64424509465 2024-09-15T20:37:26.209 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 64424509465' 2024-09-15T20:37:26.210 INFO:tasks.workunit.client.0.smithi136.stdout:waiting osd.2 seq 64424509465 2024-09-15T20:37:26.210 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-15T20:37:26.484 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 64424509466 -lt 64424509465 2024-09-15T20:37:26.485 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-15T20:37:26.485 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T20:37:26.485 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T20:37:26.827 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 1 == 0 2024-09-15T20:37:26.828 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-15T20:37:26.828 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-15T20:37:26.828 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-15T20:37:26.828 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T20:37:26.828 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T20:37:26.828 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T20:37:26.828 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T20:37:27.098 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=1 2024-09-15T20:37:27.099 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-15T20:37:27.099 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T20:37:27.099 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T20:37:27.482 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 = 1 2024-09-15T20:37:27.482 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-15T20:37:27.482 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-15T20:37:27.482 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:647: TEST_auto_repair_bluestore_failed_norecov: flush_pg_stats 2024-09-15T20:37:27.482 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-15T20:37:27.483 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-15T20:37:27.761 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-15T20:37:27.761 INFO:tasks.workunit.client.0.smithi136.stderr:1 2024-09-15T20:37:27.761 INFO:tasks.workunit.client.0.smithi136.stderr:2' 2024-09-15T20:37:27.762 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-15T20:37:27.762 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T20:37:27.762 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-15T20:37:27.864 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=150323855370 2024-09-15T20:37:27.864 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 150323855370 2024-09-15T20:37:27.864 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-150323855370' 2024-09-15T20:37:27.864 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T20:37:27.865 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-15T20:37:27.967 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=171798691847 2024-09-15T20:37:27.968 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 171798691847 2024-09-15T20:37:27.968 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-150323855370 1-171798691847' 2024-09-15T20:37:27.968 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T20:37:27.968 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-15T20:37:28.071 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=64424509467 2024-09-15T20:37:28.071 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 64424509467 2024-09-15T20:37:28.071 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-150323855370 1-171798691847 2-64424509467' 2024-09-15T20:37:28.071 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T20:37:28.071 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-150323855370 2024-09-15T20:37:28.072 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T20:37:28.072 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-15T20:37:28.073 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-150323855370 2024-09-15T20:37:28.073 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T20:37:28.074 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=150323855370 2024-09-15T20:37:28.074 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 150323855370' 2024-09-15T20:37:28.074 INFO:tasks.workunit.client.0.smithi136.stdout:waiting osd.0 seq 150323855370 2024-09-15T20:37:28.074 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T20:37:28.352 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 150323855369 -lt 150323855370 2024-09-15T20:37:28.352 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-15T20:37:29.353 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-15T20:37:29.353 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T20:37:29.672 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 150323855370 -lt 150323855370 2024-09-15T20:37:29.672 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T20:37:29.673 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-171798691847 2024-09-15T20:37:29.673 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T20:37:29.673 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-15T20:37:29.674 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-171798691847 2024-09-15T20:37:29.674 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T20:37:29.675 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=171798691847 2024-09-15T20:37:29.675 INFO:tasks.workunit.client.0.smithi136.stdout:waiting osd.1 seq 171798691847 2024-09-15T20:37:29.676 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 171798691847' 2024-09-15T20:37:29.676 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-15T20:37:29.956 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 171798691847 -lt 171798691847 2024-09-15T20:37:29.956 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T20:37:29.956 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-64424509467 2024-09-15T20:37:29.956 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T20:37:29.957 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-15T20:37:29.958 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-64424509467 2024-09-15T20:37:29.958 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T20:37:29.959 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=64424509467 2024-09-15T20:37:29.959 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 64424509467' 2024-09-15T20:37:29.959 INFO:tasks.workunit.client.0.smithi136.stdout:waiting osd.2 seq 64424509467 2024-09-15T20:37:29.959 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-15T20:37:30.234 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 64424509467 -lt 64424509467 2024-09-15T20:37:30.234 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:648: TEST_auto_repair_bluestore_failed_norecov: grep -q 'scrub_finish.*present with no repair possible' td/osd-scrub-repair/osd.1.log 2024-09-15T20:37:30.236 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:649: TEST_auto_repair_bluestore_failed_norecov: ceph pg dump pgs 2024-09-15T20:37:30.495 INFO:tasks.workunit.client.0.smithi136.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-15T20:37:30.495 INFO:tasks.workunit.client.0.smithi136.stdout:1.0 10 0 0 0 0 56 0 0 10 0 10 active+clean+inconsistent+failed_repair 2024-09-15T20:37:17.197120+0000 21'10 41:107 [1,0] 1 [1,0] 1 21'10 2024-09-15T20:37:17.197067+0000 21'10 2024-09-15T20:37:17.197067+0000 0 1 periodic scrub scheduled @ 2024-09-16T20:37:17.197067+0000 10 0 2024-09-15T20:37:30.495 INFO:tasks.workunit.client.0.smithi136.stdout: 2024-09-15T20:37:30.496 INFO:tasks.workunit.client.0.smithi136.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-15T20:37:30.496 INFO:tasks.workunit.client.0.smithi136.stderr:dumped pgs 2024-09-15T20:37:30.507 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:650: TEST_auto_repair_bluestore_failed_norecov: ceph pg dump pgs 2024-09-15T20:37:30.507 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:650: TEST_auto_repair_bluestore_failed_norecov: grep -q '^1.0.*+failed_repair' 2024-09-15T20:37:30.766 INFO:tasks.workunit.client.0.smithi136.stderr:dumped pgs 2024-09-15T20:37:30.779 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:66: run: teardown td/osd-scrub-repair 2024-09-15T20:37:30.779 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/osd-scrub-repair 2024-09-15T20:37:30.779 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-15T20:37:30.779 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/osd-scrub-repair KILL 2024-09-15T20:37:30.779 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-15T20:37:30.779 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-15T20:37:30.780 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-15T20:37:30.780 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-15T20:37:30.780 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-15T20:37:30.887 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-15T20:37:30.887 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-15T20:37:30.888 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-15T20:37:30.889 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-15T20:37:30.890 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-15T20:37:30.890 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-15T20:37:30.890 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-15T20:37:30.891 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T20:37:30.891 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-15T20:37:30.891 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-15T20:37:30.891 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T20:37:30.892 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-15T20:37:30.893 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-15T20:37:30.893 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/osd-scrub-repair 2024-09-15T20:37:30.911 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-15T20:37:30.911 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T20:37:30.911 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52602 2024-09-15T20:37:30.911 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.52602 2024-09-15T20:37:30.913 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-15T20:37:30.913 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-15T20:37:30.913 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:63: run: for func in $funcs 2024-09-15T20:37:30.913 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:64: run: setup td/osd-scrub-repair 2024-09-15T20:37:30.913 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/osd-scrub-repair 2024-09-15T20:37:30.913 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/osd-scrub-repair 2024-09-15T20:37:30.913 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/osd-scrub-repair 2024-09-15T20:37:30.913 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-15T20:37:30.914 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/osd-scrub-repair KILL 2024-09-15T20:37:30.914 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-15T20:37:30.914 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-15T20:37:30.914 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-15T20:37:30.914 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-15T20:37:30.914 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-15T20:37:30.914 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-15T20:37:30.915 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-15T20:37:30.915 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-15T20:37:30.916 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-15T20:37:30.917 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-15T20:37:30.917 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-15T20:37:30.917 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-15T20:37:30.918 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T20:37:30.918 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-15T20:37:30.918 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-15T20:37:30.918 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T20:37:30.919 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-15T20:37:30.920 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-15T20:37:30.921 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/osd-scrub-repair 2024-09-15T20:37:30.924 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-15T20:37:30.925 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T20:37:30.925 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52602 2024-09-15T20:37:30.925 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.52602 2024-09-15T20:37:30.925 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-15T20:37:30.925 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-15T20:37:30.925 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/osd-scrub-repair 2024-09-15T20:37:30.925 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2024-09-15T20:37:30.925 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T20:37:30.925 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52602 2024-09-15T20:37:30.925 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.52602 2024-09-15T20:37:30.927 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2024-09-15T20:37:30.927 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 4096 -le 1024 ']' 2024-09-15T20:37:30.927 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2024-09-15T20:37:30.927 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/osd-scrub-repair 1' TERM HUP INT 2024-09-15T20:37:30.927 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:65: run: TEST_auto_repair_bluestore_scrub td/osd-scrub-repair 2024-09-15T20:37:30.927 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:483: TEST_auto_repair_bluestore_scrub: local dir=td/osd-scrub-repair 2024-09-15T20:37:30.927 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:484: TEST_auto_repair_bluestore_scrub: local poolname=testpool 2024-09-15T20:37:30.928 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:487: TEST_auto_repair_bluestore_scrub: run_mon td/osd-scrub-repair a 2024-09-15T20:37:30.928 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/osd-scrub-repair 2024-09-15T20:37:30.928 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2024-09-15T20:37:30.928 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2024-09-15T20:37:30.928 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2024-09-15T20:37:30.928 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/osd-scrub-repair/a 2024-09-15T20:37:30.928 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-scrub-repair/a --run-dir=td/osd-scrub-repair 2024-09-15T20:37:31.388 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2024-09-15T20:37:31.388 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-15T20:37:31.388 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-15T20:37:31.388 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-15T20:37:31.388 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T20:37:31.388 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52602 2024-09-15T20:37:31.388 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.52602/$cluster-$name.asok' 2024-09-15T20:37:31.389 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-scrub-repair/a '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.52602/$cluster-$name.asok' --mon-cluster-log-file=td/osd-scrub-repair/log --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-09-15T20:37:31.580 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2024-09-15T20:37:31.581 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2024-09-15T20:37:31.581 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-15T20:37:31.581 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-15T20:37:31.581 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2024-09-15T20:37:31.582 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2024-09-15T20:37:31.582 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-15T20:37:31.582 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-15T20:37:31.582 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-15T20:37:31.583 INFO:tasks.workunit.client.0.smithi136.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-15T20:37:31.583 INFO:tasks.workunit.client.0.smithi136.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T20:37:31.583 INFO:tasks.workunit.client.0.smithi136.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52602 2024-09-15T20:37:31.583 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.52602/ceph-mon.a.asok 2024-09-15T20:37:31.583 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-15T20:37:31.583 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.52602/ceph-mon.a.asok config get fsid 2024-09-15T20:37:31.662 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2024-09-15T20:37:31.662 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-15T20:37:31.662 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-15T20:37:31.662 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2024-09-15T20:37:31.662 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2024-09-15T20:37:31.663 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-15T20:37:31.663 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-15T20:37:31.663 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-15T20:37:31.663 INFO:tasks.workunit.client.0.smithi136.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-15T20:37:31.663 INFO:tasks.workunit.client.0.smithi136.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T20:37:31.663 INFO:tasks.workunit.client.0.smithi136.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52602 2024-09-15T20:37:31.663 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.52602/ceph-mon.a.asok 2024-09-15T20:37:31.663 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-15T20:37:31.664 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.52602/ceph-mon.a.asok config get mon_host 2024-09-15T20:37:31.741 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:488: TEST_auto_repair_bluestore_scrub: run_mgr td/osd-scrub-repair x 2024-09-15T20:37:31.742 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: run_mgr: local dir=td/osd-scrub-repair 2024-09-15T20:37:31.742 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: run_mgr: shift 2024-09-15T20:37:31.742 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: run_mgr: local id=x 2024-09-15T20:37:31.742 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: run_mgr: shift 2024-09-15T20:37:31.742 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:548: run_mgr: local data=td/osd-scrub-repair/x 2024-09-15T20:37:31.742 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:550: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-15T20:37:32.556 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: get_asok_path 2024-09-15T20:37:32.557 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-15T20:37:32.557 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-15T20:37:32.557 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-15T20:37:32.557 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T20:37:32.557 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52602 2024-09-15T20:37:32.557 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.52602/$cluster-$name.asok' 2024-09-15T20:37:32.557 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-15T20:37:32.560 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-scrub-repair/x '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.52602/$cluster-$name.asok' --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-15T20:37:32.586 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:492: TEST_auto_repair_bluestore_scrub: local 'ceph_osd_args=--osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-scrub-backoff-ratio=0' 2024-09-15T20:37:32.586 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:493: TEST_auto_repair_bluestore_scrub: seq 0 2 2024-09-15T20:37:32.590 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:493: TEST_auto_repair_bluestore_scrub: for id in $(seq 0 2) 2024-09-15T20:37:32.590 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:494: TEST_auto_repair_bluestore_scrub: run_osd td/osd-scrub-repair 0 --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-scrub-backoff-ratio=0 2024-09-15T20:37:32.590 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-scrub-repair 2024-09-15T20:37:32.590 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-15T20:37:32.590 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=0 2024-09-15T20:37:32.590 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-15T20:37:32.590 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-scrub-repair/0 2024-09-15T20:37:32.591 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=ce4892c7-6c6d-4b4f-8e6f-07a3e11133e5 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-15T20:37:32.591 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T20:37:32.591 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T20:37:32.591 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T20:37:32.591 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-09-15T20:37:32.591 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-09-15T20:37:32.591 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-15T20:37:32.591 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-15T20:37:32.591 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-15T20:37:32.591 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-15T20:37:32.591 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-15T20:37:32.591 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-15T20:37:32.591 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-15T20:37:32.592 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T20:37:32.592 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52602 2024-09-15T20:37:32.592 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.52602/$cluster-$name.asok' 2024-09-15T20:37:32.592 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.52602/$cluster-$name.asok' 2024-09-15T20:37:32.592 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-15T20:37:32.592 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-15T20:37:32.592 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-15T20:37:32.592 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-15T20:37:32.592 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-15T20:37:32.592 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T20:37:32.592 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T20:37:32.592 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T20:37:32.592 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T20:37:32.592 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-15T20:37:32.593 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+='--osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-scrub-backoff-ratio=0' 2024-09-15T20:37:32.593 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-scrub-repair/0 2024-09-15T20:37:32.593 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-15T20:37:32.593 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=e9da054b-8a14-432f-bc7f-311133621289 2024-09-15T20:37:32.593 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd0 e9da054b-8a14-432f-bc7f-311133621289' 2024-09-15T20:37:32.594 INFO:tasks.workunit.client.0.smithi136.stdout:add osd0 e9da054b-8a14-432f-bc7f-311133621289 2024-09-15T20:37:32.594 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-15T20:37:32.610 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQCMRedmSQc2JBAAMTmYuNkvvXSVT1bfKpDGVw== 2024-09-15T20:37:32.610 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQCMRedmSQc2JBAAMTmYuNkvvXSVT1bfKpDGVw=="}' 2024-09-15T20:37:32.610 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new e9da054b-8a14-432f-bc7f-311133621289 -i td/osd-scrub-repair/0/new.json 2024-09-15T20:37:32.838 INFO:tasks.workunit.client.0.smithi136.stdout:0 2024-09-15T20:37:32.848 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-scrub-repair/0/new.json 2024-09-15T20:37:32.849 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 0 --fsid=ce4892c7-6c6d-4b4f-8e6f-07a3e11133e5 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.52602/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-scrub-backoff-ratio=0 --mkfs --key AQCMRedmSQc2JBAAMTmYuNkvvXSVT1bfKpDGVw== --osd-uuid e9da054b-8a14-432f-bc7f-311133621289 2024-09-15T20:37:32.881 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:37:32.879+0000 7f313e00e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T20:37:32.888 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:37:32.886+0000 7f313e00e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T20:37:32.889 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:37:32.887+0000 7f313e00e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T20:37:32.890 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:37:32.888+0000 7f313e00e640 -1 bdev(0x5562fe457c00 td/osd-scrub-repair/0/block) open stat got: (1) Operation not permitted 2024-09-15T20:37:32.890 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:37:32.888+0000 7f313e00e640 -1 bluestore(td/osd-scrub-repair/0) _read_fsid unparsable uuid 2024-09-15T20:37:38.612 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-scrub-repair/0/keyring 2024-09-15T20:37:38.612 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-15T20:37:38.613 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd0 key to auth repository 2024-09-15T20:37:38.613 INFO:tasks.workunit.client.0.smithi136.stdout:adding osd0 key to auth repository 2024-09-15T20:37:38.613 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-scrub-repair/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-15T20:37:38.968 INFO:tasks.workunit.client.0.smithi136.stdout:start osd.0 2024-09-15T20:37:38.969 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.0 2024-09-15T20:37:38.969 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 0 --fsid=ce4892c7-6c6d-4b4f-8e6f-07a3e11133e5 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.52602/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-scrub-backoff-ratio=0 2024-09-15T20:37:38.969 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-15T20:37:38.970 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-15T20:37:38.970 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-15T20:37:38.998 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:37:38.996+0000 7f152362e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T20:37:39.000 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:37:38.998+0000 7f152362e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T20:37:39.001 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:37:38.999+0000 7f152362e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T20:37:39.244 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 0 2024-09-15T20:37:39.244 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-15T20:37:39.244 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2024-09-15T20:37:39.244 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-15T20:37:39.244 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-15T20:37:39.245 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:37:39.245 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-15T20:37:39.245 INFO:tasks.workunit.client.0.smithi136.stdout:0 2024-09-15T20:37:39.245 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:37:39.245 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T20:37:39.520 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:37:40.520 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:37:40.520 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:37:40.521 INFO:tasks.workunit.client.0.smithi136.stdout:1 2024-09-15T20:37:40.521 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-15T20:37:40.521 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:37:40.521 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T20:37:40.622 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:37:40.620+0000 7f152362e640 -1 Falling back to public interface 2024-09-15T20:37:40.801 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:37:41.802 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:37:41.802 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:37:41.803 INFO:tasks.workunit.client.0.smithi136.stdout:2 2024-09-15T20:37:41.803 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-15T20:37:41.803 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:37:41.803 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T20:37:42.082 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:37:43.084 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:37:43.084 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:37:43.084 INFO:tasks.workunit.client.0.smithi136.stdout:3 2024-09-15T20:37:43.084 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-15T20:37:43.085 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:37:43.085 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T20:37:43.226 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:37:43.223+0000 7f152362e640 -1 osd.0 0 log_to_monitors true 2024-09-15T20:37:43.433 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:37:44.434 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:37:44.435 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:37:44.435 INFO:tasks.workunit.client.0.smithi136.stdout:4 2024-09-15T20:37:44.435 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-15T20:37:44.435 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:37:44.435 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T20:37:44.725 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:37:45.726 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:37:45.726 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:37:45.726 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-15T20:37:45.727 INFO:tasks.workunit.client.0.smithi136.stdout:5 2024-09-15T20:37:45.727 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:37:45.727 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T20:37:46.017 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:37:47.018 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:37:47.018 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:37:47.019 INFO:tasks.workunit.client.0.smithi136.stdout:6 2024-09-15T20:37:47.019 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-15T20:37:47.019 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:37:47.019 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T20:37:47.309 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:37:48.311 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:37:48.311 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:37:48.311 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-15T20:37:48.311 INFO:tasks.workunit.client.0.smithi136.stdout:7 2024-09-15T20:37:48.311 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:37:48.312 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T20:37:48.601 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:37:49.602 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:37:49.603 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:37:49.603 INFO:tasks.workunit.client.0.smithi136.stdout:8 2024-09-15T20:37:49.603 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-15T20:37:49.603 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:37:49.603 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T20:37:49.991 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:37:50.992 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:37:50.992 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:37:50.993 INFO:tasks.workunit.client.0.smithi136.stdout:9 2024-09-15T20:37:50.993 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-15T20:37:50.993 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:37:50.993 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T20:37:51.279 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:37:52.280 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:37:52.281 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:37:52.281 INFO:tasks.workunit.client.0.smithi136.stdout:10 2024-09-15T20:37:52.281 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-15T20:37:52.281 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:37:52.281 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T20:37:52.573 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:37:53.575 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:37:53.575 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:37:53.575 INFO:tasks.workunit.client.0.smithi136.stdout:11 2024-09-15T20:37:53.575 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-15T20:37:53.576 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:37:53.576 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T20:37:53.866 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:37:54.867 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:37:54.867 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:37:54.867 INFO:tasks.workunit.client.0.smithi136.stdout:12 2024-09-15T20:37:54.868 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-15T20:37:54.868 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:37:54.868 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T20:37:55.172 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:37:56.173 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:37:56.173 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:37:56.174 INFO:tasks.workunit.client.0.smithi136.stdout:13 2024-09-15T20:37:56.174 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-15T20:37:56.174 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:37:56.174 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T20:37:56.476 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:37:57.478 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:37:57.478 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:37:57.478 INFO:tasks.workunit.client.0.smithi136.stdout:14 2024-09-15T20:37:57.478 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-15T20:37:57.479 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:37:57.479 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T20:37:58.144 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:37:59.145 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:37:59.146 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:37:59.146 INFO:tasks.workunit.client.0.smithi136.stdout:15 2024-09-15T20:37:59.146 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-15T20:37:59.146 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:37:59.147 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T20:37:59.594 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:38:00.596 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:38:00.596 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:38:00.596 INFO:tasks.workunit.client.0.smithi136.stdout:16 2024-09-15T20:38:00.597 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-15T20:38:00.597 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:38:00.597 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T20:38:00.887 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:38:01.888 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:38:01.888 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:38:01.889 INFO:tasks.workunit.client.0.smithi136.stdout:17 2024-09-15T20:38:01.889 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-15T20:38:01.889 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:38:01.889 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T20:38:02.412 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:38:03.414 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:38:03.414 INFO:tasks.workunit.client.0.smithi136.stdout:18 2024-09-15T20:38:03.415 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:38:03.415 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 18 2024-09-15T20:38:03.415 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:38:03.415 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T20:38:04.022 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:38:05.024 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:38:05.024 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:38:05.024 INFO:tasks.workunit.client.0.smithi136.stdout:19 2024-09-15T20:38:05.025 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 19 2024-09-15T20:38:05.025 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:38:05.025 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T20:38:05.373 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:38:06.375 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:38:06.375 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:38:06.375 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 20 2024-09-15T20:38:06.375 INFO:tasks.workunit.client.0.smithi136.stdout:20 2024-09-15T20:38:06.376 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:38:06.376 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T20:38:06.665 INFO:tasks.workunit.client.0.smithi136.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/515632081,v1:127.0.0.1:6803/515632081] [v2:127.0.0.1:6804/515632081,v1:127.0.0.1:6805/515632081] exists,up e9da054b-8a14-432f-bc7f-311133621289 2024-09-15T20:38:06.669 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-15T20:38:06.669 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-15T20:38:06.669 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-15T20:38:06.669 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:493: TEST_auto_repair_bluestore_scrub: for id in $(seq 0 2) 2024-09-15T20:38:06.669 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:494: TEST_auto_repair_bluestore_scrub: run_osd td/osd-scrub-repair 1 --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-scrub-backoff-ratio=0 2024-09-15T20:38:06.669 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-scrub-repair 2024-09-15T20:38:06.669 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-15T20:38:06.670 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=1 2024-09-15T20:38:06.670 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-15T20:38:06.670 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-scrub-repair/1 2024-09-15T20:38:06.670 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=ce4892c7-6c6d-4b4f-8e6f-07a3e11133e5 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-15T20:38:06.670 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T20:38:06.670 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T20:38:06.670 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T20:38:06.670 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-09-15T20:38:06.670 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-09-15T20:38:06.670 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-15T20:38:06.670 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-15T20:38:06.670 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-15T20:38:06.670 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-15T20:38:06.670 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-15T20:38:06.671 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-15T20:38:06.671 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-15T20:38:06.671 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T20:38:06.671 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52602 2024-09-15T20:38:06.671 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.52602/$cluster-$name.asok' 2024-09-15T20:38:06.671 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.52602/$cluster-$name.asok' 2024-09-15T20:38:06.671 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-15T20:38:06.671 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-15T20:38:06.671 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-15T20:38:06.671 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-15T20:38:06.671 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-15T20:38:06.671 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T20:38:06.672 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T20:38:06.672 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T20:38:06.672 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T20:38:06.672 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-15T20:38:06.672 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+='--osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-scrub-backoff-ratio=0' 2024-09-15T20:38:06.672 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-scrub-repair/1 2024-09-15T20:38:06.672 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-15T20:38:06.672 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=722b27e5-5195-41eb-a2eb-f63048be6cc0 2024-09-15T20:38:06.673 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd1 722b27e5-5195-41eb-a2eb-f63048be6cc0' 2024-09-15T20:38:06.673 INFO:tasks.workunit.client.0.smithi136.stdout:add osd1 722b27e5-5195-41eb-a2eb-f63048be6cc0 2024-09-15T20:38:06.673 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-15T20:38:06.688 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQCuRedm6MrpKBAAynBbNa0939vETCCe/T2MZw== 2024-09-15T20:38:06.689 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQCuRedm6MrpKBAAynBbNa0939vETCCe/T2MZw=="}' 2024-09-15T20:38:06.689 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 722b27e5-5195-41eb-a2eb-f63048be6cc0 -i td/osd-scrub-repair/1/new.json 2024-09-15T20:38:07.017 INFO:tasks.workunit.client.0.smithi136.stdout:1 2024-09-15T20:38:07.028 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-scrub-repair/1/new.json 2024-09-15T20:38:07.029 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 1 --fsid=ce4892c7-6c6d-4b4f-8e6f-07a3e11133e5 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.52602/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-scrub-backoff-ratio=0 --mkfs --key AQCuRedm6MrpKBAAynBbNa0939vETCCe/T2MZw== --osd-uuid 722b27e5-5195-41eb-a2eb-f63048be6cc0 2024-09-15T20:38:07.061 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:38:07.059+0000 7fa27660c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T20:38:07.101 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:38:07.099+0000 7fa27660c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T20:38:07.102 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:38:07.100+0000 7fa27660c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T20:38:07.103 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:38:07.100+0000 7fa27660c640 -1 bdev(0x55bd67e1bc00 td/osd-scrub-repair/1/block) open stat got: (1) Operation not permitted 2024-09-15T20:38:07.103 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:38:07.100+0000 7fa27660c640 -1 bluestore(td/osd-scrub-repair/1) _read_fsid unparsable uuid 2024-09-15T20:38:12.553 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-scrub-repair/1/keyring 2024-09-15T20:38:12.553 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-15T20:38:12.554 INFO:tasks.workunit.client.0.smithi136.stdout:adding osd1 key to auth repository 2024-09-15T20:38:12.554 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd1 key to auth repository 2024-09-15T20:38:12.554 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-scrub-repair/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-15T20:38:12.917 INFO:tasks.workunit.client.0.smithi136.stdout:start osd.1 2024-09-15T20:38:12.918 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.1 2024-09-15T20:38:12.918 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 1 --fsid=ce4892c7-6c6d-4b4f-8e6f-07a3e11133e5 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.52602/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-scrub-backoff-ratio=0 2024-09-15T20:38:12.918 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-15T20:38:12.918 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-15T20:38:12.918 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-15T20:38:12.947 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:38:12.945+0000 7f5c4eaf1640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T20:38:12.950 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:38:12.948+0000 7f5c4eaf1640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T20:38:12.951 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:38:12.949+0000 7f5c4eaf1640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T20:38:13.199 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 1 2024-09-15T20:38:13.199 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-15T20:38:13.199 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2024-09-15T20:38:13.199 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-15T20:38:13.200 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-15T20:38:13.200 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:38:13.200 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-15T20:38:13.200 INFO:tasks.workunit.client.0.smithi136.stdout:0 2024-09-15T20:38:13.200 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:38:13.200 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T20:38:13.485 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:38:14.486 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:38:14.486 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:38:14.486 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-15T20:38:14.486 INFO:tasks.workunit.client.0.smithi136.stdout:1 2024-09-15T20:38:14.487 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:38:14.487 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T20:38:14.563 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:38:14.561+0000 7f5c4eaf1640 -1 Falling back to public interface 2024-09-15T20:38:14.776 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:38:15.777 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:38:15.777 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:38:15.777 INFO:tasks.workunit.client.0.smithi136.stdout:2 2024-09-15T20:38:15.778 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-15T20:38:15.778 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:38:15.778 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T20:38:16.061 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:38:17.062 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:38:17.062 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:38:17.062 INFO:tasks.workunit.client.0.smithi136.stdout:3 2024-09-15T20:38:17.063 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-15T20:38:17.063 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:38:17.063 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T20:38:17.207 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:38:17.205+0000 7f5c4eaf1640 -1 osd.1 0 log_to_monitors true 2024-09-15T20:38:17.343 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:38:18.344 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:38:18.344 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:38:18.345 INFO:tasks.workunit.client.0.smithi136.stdout:4 2024-09-15T20:38:18.345 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-15T20:38:18.345 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:38:18.345 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T20:38:18.679 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:38:19.680 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:38:19.680 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:38:19.681 INFO:tasks.workunit.client.0.smithi136.stdout:5 2024-09-15T20:38:19.681 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-15T20:38:19.681 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:38:19.681 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T20:38:20.030 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:38:21.031 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:38:21.031 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:38:21.032 INFO:tasks.workunit.client.0.smithi136.stdout:6 2024-09-15T20:38:21.032 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-15T20:38:21.032 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:38:21.032 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T20:38:21.324 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:38:22.325 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:38:22.325 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:38:22.326 INFO:tasks.workunit.client.0.smithi136.stdout:7 2024-09-15T20:38:22.326 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-15T20:38:22.326 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:38:22.326 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T20:38:22.677 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:38:23.678 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:38:23.679 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:38:23.679 INFO:tasks.workunit.client.0.smithi136.stdout:8 2024-09-15T20:38:23.679 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-15T20:38:23.679 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:38:23.679 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T20:38:24.077 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:38:25.078 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:38:25.079 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:38:25.079 INFO:tasks.workunit.client.0.smithi136.stdout:9 2024-09-15T20:38:25.079 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-15T20:38:25.079 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:38:25.080 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T20:38:25.480 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:38:26.481 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:38:26.481 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:38:26.481 INFO:tasks.workunit.client.0.smithi136.stdout:10 2024-09-15T20:38:26.482 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-15T20:38:26.482 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:38:26.482 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T20:38:26.811 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:38:27.812 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:38:27.812 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:38:27.812 INFO:tasks.workunit.client.0.smithi136.stdout:11 2024-09-15T20:38:27.813 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-15T20:38:27.813 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:38:27.813 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T20:38:28.241 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:38:29.243 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:38:29.243 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:38:29.243 INFO:tasks.workunit.client.0.smithi136.stdout:12 2024-09-15T20:38:29.243 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-15T20:38:29.244 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:38:29.244 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T20:38:29.863 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:38:30.864 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:38:30.865 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:38:30.865 INFO:tasks.workunit.client.0.smithi136.stdout:13 2024-09-15T20:38:30.865 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-15T20:38:30.866 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:38:30.866 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T20:38:31.657 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:38:32.658 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:38:32.659 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:38:32.659 INFO:tasks.workunit.client.0.smithi136.stdout:14 2024-09-15T20:38:32.659 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-15T20:38:32.660 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:38:32.660 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T20:38:33.310 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:38:34.312 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:38:34.312 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:38:34.312 INFO:tasks.workunit.client.0.smithi136.stdout:15 2024-09-15T20:38:34.312 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-15T20:38:34.313 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:38:34.313 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T20:38:34.916 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:38:35.917 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:38:35.917 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:38:35.917 INFO:tasks.workunit.client.0.smithi136.stdout:16 2024-09-15T20:38:35.918 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-15T20:38:35.918 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:38:35.918 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T20:38:36.449 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:38:37.451 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:38:37.451 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:38:37.451 INFO:tasks.workunit.client.0.smithi136.stdout:17 2024-09-15T20:38:37.451 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-15T20:38:37.452 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:38:37.452 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T20:38:38.018 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:38:39.020 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:38:39.020 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:38:39.020 INFO:tasks.workunit.client.0.smithi136.stdout:18 2024-09-15T20:38:39.021 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 18 2024-09-15T20:38:39.021 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:38:39.021 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T20:38:39.418 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:38:40.420 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:38:40.420 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:38:40.420 INFO:tasks.workunit.client.0.smithi136.stdout:19 2024-09-15T20:38:40.421 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 19 2024-09-15T20:38:40.421 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:38:40.421 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T20:38:40.708 INFO:tasks.workunit.client.0.smithi136.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/4078904186,v1:127.0.0.1:6811/4078904186] [v2:127.0.0.1:6812/4078904186,v1:127.0.0.1:6813/4078904186] exists,up 722b27e5-5195-41eb-a2eb-f63048be6cc0 2024-09-15T20:38:40.712 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-15T20:38:40.712 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-15T20:38:40.712 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-15T20:38:40.712 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:493: TEST_auto_repair_bluestore_scrub: for id in $(seq 0 2) 2024-09-15T20:38:40.712 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:494: TEST_auto_repair_bluestore_scrub: run_osd td/osd-scrub-repair 2 --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-scrub-backoff-ratio=0 2024-09-15T20:38:40.713 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-scrub-repair 2024-09-15T20:38:40.713 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-15T20:38:40.713 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=2 2024-09-15T20:38:40.713 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-15T20:38:40.713 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-scrub-repair/2 2024-09-15T20:38:40.713 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=ce4892c7-6c6d-4b4f-8e6f-07a3e11133e5 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-15T20:38:40.713 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T20:38:40.713 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T20:38:40.713 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T20:38:40.713 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/2' 2024-09-15T20:38:40.713 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/2/journal' 2024-09-15T20:38:40.713 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-15T20:38:40.713 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-15T20:38:40.713 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-15T20:38:40.714 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-15T20:38:40.714 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-15T20:38:40.714 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-15T20:38:40.714 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-15T20:38:40.714 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T20:38:40.714 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52602 2024-09-15T20:38:40.714 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.52602/$cluster-$name.asok' 2024-09-15T20:38:40.714 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.52602/$cluster-$name.asok' 2024-09-15T20:38:40.714 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-15T20:38:40.714 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-15T20:38:40.714 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-15T20:38:40.714 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-15T20:38:40.714 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-15T20:38:40.715 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T20:38:40.715 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T20:38:40.715 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T20:38:40.715 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T20:38:40.715 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-15T20:38:40.715 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+='--osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-scrub-backoff-ratio=0' 2024-09-15T20:38:40.715 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-scrub-repair/2 2024-09-15T20:38:40.715 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-15T20:38:40.715 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=c92ec994-d983-4ee7-bcb1-d4cfaf0574ec 2024-09-15T20:38:40.715 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd2 c92ec994-d983-4ee7-bcb1-d4cfaf0574ec' 2024-09-15T20:38:40.715 INFO:tasks.workunit.client.0.smithi136.stdout:add osd2 c92ec994-d983-4ee7-bcb1-d4cfaf0574ec 2024-09-15T20:38:40.716 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-15T20:38:40.730 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQDQRedm7J1uKxAAA79PfEf1ygJimmPpur818Q== 2024-09-15T20:38:40.731 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQDQRedm7J1uKxAAA79PfEf1ygJimmPpur818Q=="}' 2024-09-15T20:38:40.731 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new c92ec994-d983-4ee7-bcb1-d4cfaf0574ec -i td/osd-scrub-repair/2/new.json 2024-09-15T20:38:41.098 INFO:tasks.workunit.client.0.smithi136.stdout:2 2024-09-15T20:38:41.110 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-scrub-repair/2/new.json 2024-09-15T20:38:41.111 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 2 --fsid=ce4892c7-6c6d-4b4f-8e6f-07a3e11133e5 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.52602/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-scrub-backoff-ratio=0 --mkfs --key AQDQRedm7J1uKxAAA79PfEf1ygJimmPpur818Q== --osd-uuid c92ec994-d983-4ee7-bcb1-d4cfaf0574ec 2024-09-15T20:38:41.143 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:38:41.141+0000 7fcaa4eb4640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T20:38:41.200 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:38:41.198+0000 7fcaa4eb4640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T20:38:41.201 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:38:41.199+0000 7fcaa4eb4640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T20:38:41.201 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:38:41.199+0000 7fcaa4eb4640 -1 bdev(0x55bea7309c00 td/osd-scrub-repair/2/block) open stat got: (1) Operation not permitted 2024-09-15T20:38:41.201 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:38:41.199+0000 7fcaa4eb4640 -1 bluestore(td/osd-scrub-repair/2) _read_fsid unparsable uuid 2024-09-15T20:38:47.105 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-scrub-repair/2/keyring 2024-09-15T20:38:47.105 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-15T20:38:47.106 INFO:tasks.workunit.client.0.smithi136.stdout:adding osd2 key to auth repository 2024-09-15T20:38:47.106 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd2 key to auth repository 2024-09-15T20:38:47.107 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-scrub-repair/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-15T20:38:47.471 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.2 2024-09-15T20:38:47.471 INFO:tasks.workunit.client.0.smithi136.stdout:start osd.2 2024-09-15T20:38:47.471 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 2 --fsid=ce4892c7-6c6d-4b4f-8e6f-07a3e11133e5 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.52602/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-scrub-backoff-ratio=0 2024-09-15T20:38:47.471 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-15T20:38:47.472 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-15T20:38:47.472 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-15T20:38:47.504 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:38:47.499+0000 7f87dafd4640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T20:38:47.505 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:38:47.501+0000 7f87dafd4640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T20:38:47.505 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:38:47.502+0000 7f87dafd4640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T20:38:47.747 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 2 2024-09-15T20:38:47.747 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-15T20:38:47.747 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=2 2024-09-15T20:38:47.747 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-15T20:38:47.747 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-15T20:38:47.747 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:38:47.748 INFO:tasks.workunit.client.0.smithi136.stdout:0 2024-09-15T20:38:47.748 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-15T20:38:47.748 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:38:47.748 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T20:38:48.024 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:38:48.879 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:38:48.877+0000 7f87dafd4640 -1 Falling back to public interface 2024-09-15T20:38:49.025 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:38:49.025 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:38:49.025 INFO:tasks.workunit.client.0.smithi136.stdout:1 2024-09-15T20:38:49.026 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-15T20:38:49.026 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:38:49.026 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T20:38:49.314 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:38:50.315 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:38:50.315 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:38:50.315 INFO:tasks.workunit.client.0.smithi136.stdout:2 2024-09-15T20:38:50.316 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-15T20:38:50.316 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:38:50.316 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T20:38:50.626 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:38:51.628 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:38:51.628 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:38:51.629 INFO:tasks.workunit.client.0.smithi136.stdout:3 2024-09-15T20:38:51.630 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-15T20:38:51.630 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:38:51.630 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T20:38:51.790 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:38:51.787+0000 7f87dafd4640 -1 osd.2 0 log_to_monitors true 2024-09-15T20:38:51.989 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:38:52.991 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:38:52.991 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:38:52.991 INFO:tasks.workunit.client.0.smithi136.stdout:4 2024-09-15T20:38:52.992 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-15T20:38:52.992 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:38:52.992 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T20:38:53.281 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:38:54.282 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:38:54.283 INFO:tasks.workunit.client.0.smithi136.stdout:5 2024-09-15T20:38:54.283 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:38:54.284 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-15T20:38:54.284 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:38:54.284 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T20:38:54.570 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:38:55.572 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:38:55.572 INFO:tasks.workunit.client.0.smithi136.stdout:6 2024-09-15T20:38:55.573 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:38:55.573 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-15T20:38:55.573 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:38:55.573 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T20:38:55.860 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:38:56.861 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:38:56.861 INFO:tasks.workunit.client.0.smithi136.stdout:7 2024-09-15T20:38:56.862 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:38:56.862 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-15T20:38:56.862 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:38:56.862 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T20:38:57.168 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:38:58.170 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:38:58.170 INFO:tasks.workunit.client.0.smithi136.stdout:8 2024-09-15T20:38:58.170 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:38:58.170 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-15T20:38:58.171 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:38:58.171 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T20:38:58.585 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:38:59.586 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:38:59.586 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:38:59.587 INFO:tasks.workunit.client.0.smithi136.stdout:9 2024-09-15T20:38:59.587 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-15T20:38:59.587 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:38:59.587 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T20:38:59.876 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:39:00.877 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:39:00.878 INFO:tasks.workunit.client.0.smithi136.stdout:10 2024-09-15T20:39:00.878 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:39:00.878 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-15T20:39:00.878 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:39:00.878 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T20:39:01.243 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:39:02.244 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:39:02.244 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:39:02.245 INFO:tasks.workunit.client.0.smithi136.stdout:11 2024-09-15T20:39:02.245 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-15T20:39:02.245 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:39:02.245 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T20:39:02.583 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:39:03.585 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:39:03.585 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:39:03.585 INFO:tasks.workunit.client.0.smithi136.stdout:12 2024-09-15T20:39:03.586 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-15T20:39:03.586 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:39:03.586 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T20:39:03.870 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:39:04.872 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:39:04.872 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:39:04.872 INFO:tasks.workunit.client.0.smithi136.stdout:13 2024-09-15T20:39:04.873 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-15T20:39:04.873 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:39:04.873 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T20:39:05.624 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:39:06.625 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:39:06.625 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:39:06.625 INFO:tasks.workunit.client.0.smithi136.stdout:14 2024-09-15T20:39:06.626 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-15T20:39:06.626 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:39:06.626 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T20:39:06.981 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:39:07.983 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:39:07.983 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:39:07.983 INFO:tasks.workunit.client.0.smithi136.stdout:15 2024-09-15T20:39:07.984 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-15T20:39:07.984 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:39:07.984 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T20:39:08.518 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:39:09.519 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:39:09.519 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:39:09.519 INFO:tasks.workunit.client.0.smithi136.stdout:16 2024-09-15T20:39:09.520 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-15T20:39:09.520 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:39:09.520 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T20:39:10.329 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:39:11.331 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:39:11.331 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:39:11.331 INFO:tasks.workunit.client.0.smithi136.stdout:17 2024-09-15T20:39:11.331 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-15T20:39:11.332 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:39:11.332 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T20:39:12.065 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:39:13.067 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:39:13.067 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:39:13.067 INFO:tasks.workunit.client.0.smithi136.stdout:18 2024-09-15T20:39:13.068 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 18 2024-09-15T20:39:13.068 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:39:13.068 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T20:39:13.759 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:39:14.760 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:39:14.760 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:39:14.761 INFO:tasks.workunit.client.0.smithi136.stdout:19 2024-09-15T20:39:14.761 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 19 2024-09-15T20:39:14.761 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:39:14.761 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T20:39:15.115 INFO:tasks.workunit.client.0.smithi136.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/3011475035,v1:127.0.0.1:6819/3011475035] [v2:127.0.0.1:6820/3011475035,v1:127.0.0.1:6821/3011475035] exists,up c92ec994-d983-4ee7-bcb1-d4cfaf0574ec 2024-09-15T20:39:15.115 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-15T20:39:15.115 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-15T20:39:15.115 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-15T20:39:15.116 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:497: TEST_auto_repair_bluestore_scrub: create_pool testpool 1 1 2024-09-15T20:39:15.116 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create testpool 1 1 2024-09-15T20:39:15.538 INFO:tasks.workunit.client.0.smithi136.stderr:pool 'testpool' created 2024-09-15T20:39:15.550 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2024-09-15T20:39:16.551 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:498: TEST_auto_repair_bluestore_scrub: ceph osd pool set testpool size 2 2024-09-15T20:39:16.957 INFO:tasks.workunit.client.0.smithi136.stderr:set pool 1 size to 2 2024-09-15T20:39:16.968 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:499: TEST_auto_repair_bluestore_scrub: wait_for_clean 2024-09-15T20:39:16.968 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-15T20:39:16.968 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-15T20:39:16.969 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-15T20:39:16.969 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-15T20:39:16.969 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-15T20:39:16.969 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-15T20:39:16.969 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-15T20:39:16.970 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-15T20:39:16.970 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-15T20:39:17.059 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-15T20:39:17.059 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-15T20:39:17.060 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-15T20:39:17.060 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-15T20:39:17.060 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-15T20:39:17.060 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-15T20:39:17.428 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-15T20:39:17.428 INFO:tasks.workunit.client.0.smithi136.stderr:1 2024-09-15T20:39:17.429 INFO:tasks.workunit.client.0.smithi136.stderr:2' 2024-09-15T20:39:17.429 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-15T20:39:17.429 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T20:39:17.429 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-15T20:39:17.541 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836499 2024-09-15T20:39:17.541 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836499 2024-09-15T20:39:17.541 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836499' 2024-09-15T20:39:17.541 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T20:39:17.541 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-15T20:39:17.649 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=42949672975 2024-09-15T20:39:17.649 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 42949672975 2024-09-15T20:39:17.649 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836499 1-42949672975' 2024-09-15T20:39:17.649 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T20:39:17.650 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-15T20:39:17.785 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=64424509448 2024-09-15T20:39:17.786 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 64424509448 2024-09-15T20:39:17.786 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836499 1-42949672975 2-64424509448' 2024-09-15T20:39:17.786 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T20:39:17.786 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836499 2024-09-15T20:39:17.786 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T20:39:17.786 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-15T20:39:17.787 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836499 2024-09-15T20:39:17.787 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T20:39:17.788 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836499 2024-09-15T20:39:17.788 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836499' 2024-09-15T20:39:17.788 INFO:tasks.workunit.client.0.smithi136.stdout:waiting osd.0 seq 21474836499 2024-09-15T20:39:17.789 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T20:39:18.149 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836497 -lt 21474836499 2024-09-15T20:39:18.149 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-15T20:39:19.150 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-15T20:39:19.150 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T20:39:19.434 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836499 -lt 21474836499 2024-09-15T20:39:19.434 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T20:39:19.434 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-42949672975 2024-09-15T20:39:19.435 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T20:39:19.435 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-15T20:39:19.435 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-42949672975 2024-09-15T20:39:19.436 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T20:39:19.437 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=42949672975 2024-09-15T20:39:19.437 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 42949672975' 2024-09-15T20:39:19.437 INFO:tasks.workunit.client.0.smithi136.stdout:waiting osd.1 seq 42949672975 2024-09-15T20:39:19.437 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-15T20:39:19.715 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 42949672975 -lt 42949672975 2024-09-15T20:39:19.716 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T20:39:19.716 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-64424509448 2024-09-15T20:39:19.716 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T20:39:19.717 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-15T20:39:19.717 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-64424509448 2024-09-15T20:39:19.718 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T20:39:19.718 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=64424509448 2024-09-15T20:39:19.719 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 64424509448' 2024-09-15T20:39:19.719 INFO:tasks.workunit.client.0.smithi136.stdout:waiting osd.2 seq 64424509448 2024-09-15T20:39:19.719 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-15T20:39:19.993 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 64424509448 -lt 64424509448 2024-09-15T20:39:19.994 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-15T20:39:19.994 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T20:39:19.994 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T20:39:20.337 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 1 == 0 2024-09-15T20:39:20.337 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-15T20:39:20.337 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-15T20:39:20.338 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-15T20:39:20.338 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T20:39:20.338 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T20:39:20.338 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T20:39:20.338 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T20:39:20.609 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=0 2024-09-15T20:39:20.609 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-15T20:39:20.609 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T20:39:20.609 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T20:39:20.971 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 0 = 1 2024-09-15T20:39:20.971 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 0 '!=' -1 2024-09-15T20:39:20.971 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: loop=0 2024-09-15T20:39:20.971 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1652: wait_for_clean: num_active_clean=0 2024-09-15T20:39:20.971 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-15T20:39:20.971 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 0.1 2024-09-15T20:39:21.072 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-15T20:39:21.072 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-15T20:39:21.073 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-15T20:39:21.073 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-15T20:39:21.073 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T20:39:21.073 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T20:39:21.073 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T20:39:21.073 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T20:39:21.342 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=0 2024-09-15T20:39:21.343 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-15T20:39:21.343 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T20:39:21.343 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T20:39:21.696 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 0 = 1 2024-09-15T20:39:21.696 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 0 '!=' 0 2024-09-15T20:39:21.696 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: get_is_making_recovery_progress 2024-09-15T20:39:21.696 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2024-09-15T20:39:21.697 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-15T20:39:21.697 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-15T20:39:21.697 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-15T20:39:21.697 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2024-09-15T20:39:21.697 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-15T20:39:22.041 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=null 2024-09-15T20:39:22.041 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test null '!=' null 2024-09-15T20:39:22.041 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1655: wait_for_clean: (( 1 >= 13 )) 2024-09-15T20:39:22.041 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-15T20:39:22.041 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 0.2 2024-09-15T20:39:22.242 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-15T20:39:22.242 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-15T20:39:22.242 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-15T20:39:22.242 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-15T20:39:22.243 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T20:39:22.243 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T20:39:22.243 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T20:39:22.243 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T20:39:22.522 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=0 2024-09-15T20:39:22.523 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-15T20:39:22.523 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T20:39:22.523 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T20:39:22.902 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 0 = 1 2024-09-15T20:39:22.902 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 0 '!=' 0 2024-09-15T20:39:22.902 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: get_is_making_recovery_progress 2024-09-15T20:39:22.902 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2024-09-15T20:39:22.902 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-15T20:39:22.902 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-15T20:39:22.903 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-15T20:39:22.903 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2024-09-15T20:39:22.903 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-15T20:39:23.247 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=null 2024-09-15T20:39:23.248 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test null '!=' null 2024-09-15T20:39:23.248 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1655: wait_for_clean: (( 2 >= 13 )) 2024-09-15T20:39:23.248 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-15T20:39:23.248 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 0.4 2024-09-15T20:39:23.649 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-15T20:39:23.649 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-15T20:39:23.649 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-15T20:39:23.649 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-15T20:39:23.649 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T20:39:23.649 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T20:39:23.649 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T20:39:23.649 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T20:39:23.918 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=1 2024-09-15T20:39:23.918 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-15T20:39:23.918 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T20:39:23.918 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T20:39:24.266 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 = 1 2024-09-15T20:39:24.266 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-15T20:39:24.266 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-15T20:39:24.267 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:502: TEST_auto_repair_bluestore_scrub: local payload=ABCDEF 2024-09-15T20:39:24.267 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:503: TEST_auto_repair_bluestore_scrub: echo ABCDEF 2024-09-15T20:39:24.267 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:504: TEST_auto_repair_bluestore_scrub: rados --pool testpool put SOMETHING td/osd-scrub-repair/ORIGINAL 2024-09-15T20:39:24.398 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:508: TEST_auto_repair_bluestore_scrub: get_not_primary testpool SOMETHING 2024-09-15T20:39:24.398 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1220: get_not_primary: local poolname=testpool 2024-09-15T20:39:24.398 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1221: get_not_primary: local objectname=SOMETHING 2024-09-15T20:39:24.398 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1223: get_not_primary: get_primary testpool SOMETHING 2024-09-15T20:39:24.398 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1187: get_primary: local poolname=testpool 2024-09-15T20:39:24.399 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1188: get_primary: local objectname=SOMETHING 2024-09-15T20:39:24.399 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1190: get_primary: ceph --format json osd map testpool SOMETHING 2024-09-15T20:39:24.399 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1191: get_primary: jq .acting_primary 2024-09-15T20:39:24.677 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1223: get_not_primary: local primary=1 2024-09-15T20:39:24.677 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1224: get_not_primary: ceph --format json osd map testpool SOMETHING 2024-09-15T20:39:24.677 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1225: get_not_primary: jq '.acting | map(select (. != 1)) | .[0]' 2024-09-15T20:39:24.962 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:508: TEST_auto_repair_bluestore_scrub: objectstore_tool td/osd-scrub-repair 0 SOMETHING remove 2024-09-15T20:39:24.962 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2024-09-15T20:39:24.962 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2024-09-15T20:39:24.962 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=0 2024-09-15T20:39:24.963 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2024-09-15T20:39:24.963 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 SOMETHING remove 2024-09-15T20:39:24.963 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-09-15T20:39:24.963 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2024-09-15T20:39:24.963 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=0 2024-09-15T20:39:24.963 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2024-09-15T20:39:24.963 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2024-09-15T20:39:24.963 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-15T20:39:24.963 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-15T20:39:24.963 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-15T20:39:24.963 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-15T20:39:24.963 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-15T20:39:25.269 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-15T20:39:25.269 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 SOMETHING remove 2024-09-15T20:39:25.269 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-15T20:39:25.269 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-15T20:39:25.269 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=0 2024-09-15T20:39:25.269 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-15T20:39:25.269 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-09-15T20:39:25.270 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 SOMETHING remove 2024-09-15T20:39:26.925 INFO:tasks.workunit.client.0.smithi136.stdout:remove #1:eb822e21:::SOMETHING:head# 2024-09-15T20:39:27.447 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-scrub-backoff-ratio=0 2024-09-15T20:39:27.447 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2024-09-15T20:39:27.448 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-15T20:39:27.448 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=0 2024-09-15T20:39:27.448 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-15T20:39:27.448 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-09-15T20:39:27.448 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=ce4892c7-6c6d-4b4f-8e6f-07a3e11133e5 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-15T20:39:27.449 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T20:39:27.449 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T20:39:27.449 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T20:39:27.449 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-09-15T20:39:27.449 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-09-15T20:39:27.450 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-15T20:39:27.450 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-15T20:39:27.450 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-15T20:39:27.450 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-15T20:39:27.450 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-15T20:39:27.450 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-15T20:39:27.451 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-15T20:39:27.451 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T20:39:27.451 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52602 2024-09-15T20:39:27.451 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.52602/$cluster-$name.asok' 2024-09-15T20:39:27.451 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.52602/$cluster-$name.asok' 2024-09-15T20:39:27.451 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-15T20:39:27.452 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-15T20:39:27.452 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-15T20:39:27.452 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T20:39:27.452 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T20:39:27.452 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T20:39:27.453 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T20:39:27.453 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-15T20:39:27.453 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+='--osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-scrub-backoff-ratio=0' 2024-09-15T20:39:27.453 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-09-15T20:39:27.453 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.0 2024-09-15T20:39:27.453 INFO:tasks.workunit.client.0.smithi136.stderr:start osd.0 2024-09-15T20:39:27.454 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 0 --fsid=ce4892c7-6c6d-4b4f-8e6f-07a3e11133e5 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.52602/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-scrub-backoff-ratio=0 2024-09-15T20:39:27.454 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-09-15T20:39:27.454 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 0 = 0 ']' 2024-09-15T20:39:27.454 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-15T20:39:27.454 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-15T20:39:27.454 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-15T20:39:27.476 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:39:27.474+0000 7fe033ae6640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T20:39:27.478 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:39:27.476+0000 7fe033ae6640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T20:39:27.480 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:39:27.478+0000 7fe033ae6640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T20:39:27.732 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 0 2024-09-15T20:39:27.732 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-15T20:39:27.733 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2024-09-15T20:39:27.733 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-15T20:39:27.733 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-15T20:39:27.733 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:39:27.733 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-15T20:39:27.733 INFO:tasks.workunit.client.0.smithi136.stderr:0 2024-09-15T20:39:27.733 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:39:27.733 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T20:39:28.008 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:39:28.579 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:39:28.577+0000 7fe033ae6640 -1 Falling back to public interface 2024-09-15T20:39:29.010 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:39:29.010 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:39:29.010 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-15T20:39:29.010 INFO:tasks.workunit.client.0.smithi136.stderr:1 2024-09-15T20:39:29.010 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:39:29.010 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T20:39:29.294 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:39:30.295 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:39:30.295 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:39:30.296 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-15T20:39:30.296 INFO:tasks.workunit.client.0.smithi136.stderr:2 2024-09-15T20:39:30.296 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:39:30.296 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T20:39:30.584 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:39:31.060 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:39:31.057+0000 7fe033ae6640 -1 osd.0 21 log_to_monitors true 2024-09-15T20:39:31.586 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:39:31.586 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:39:31.586 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-15T20:39:31.586 INFO:tasks.workunit.client.0.smithi136.stderr:3 2024-09-15T20:39:31.586 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:39:31.586 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T20:39:31.894 INFO:tasks.workunit.client.0.smithi136.stderr:osd.0 up in weight 1 up_from 25 up_thru 0 down_at 22 last_clean_interval [5,21) [v2:127.0.0.1:6802/221700834,v1:127.0.0.1:6803/221700834] [v2:127.0.0.1:6804/221700834,v1:127.0.0.1:6805/221700834] exists,up e9da054b-8a14-432f-bc7f-311133621289 2024-09-15T20:39:31.895 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-15T20:39:31.895 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-15T20:39:31.895 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-15T20:39:31.895 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2024-09-15T20:39:31.895 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-15T20:39:31.895 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-15T20:39:31.895 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-15T20:39:31.895 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-15T20:39:31.896 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-15T20:39:31.896 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-15T20:39:31.896 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-15T20:39:31.896 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-15T20:39:31.896 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-15T20:39:31.985 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-15T20:39:31.985 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-15T20:39:31.985 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-15T20:39:31.985 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-15T20:39:31.985 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-15T20:39:31.988 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-15T20:39:32.268 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-15T20:39:32.269 INFO:tasks.workunit.client.0.smithi136.stderr:1 2024-09-15T20:39:32.269 INFO:tasks.workunit.client.0.smithi136.stderr:2' 2024-09-15T20:39:32.269 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-15T20:39:32.269 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T20:39:32.269 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-15T20:39:32.371 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=107374182402 2024-09-15T20:39:32.372 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 107374182402 2024-09-15T20:39:32.372 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-107374182402' 2024-09-15T20:39:32.372 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T20:39:32.372 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-15T20:39:32.475 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=42949672979 2024-09-15T20:39:32.475 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 42949672979 2024-09-15T20:39:32.475 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-107374182402 1-42949672979' 2024-09-15T20:39:32.475 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T20:39:32.475 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-15T20:39:32.578 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=64424509452 2024-09-15T20:39:32.578 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 64424509452 2024-09-15T20:39:32.579 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-107374182402 1-42949672979 2-64424509452' 2024-09-15T20:39:32.579 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T20:39:32.579 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-107374182402 2024-09-15T20:39:32.579 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T20:39:32.580 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-15T20:39:32.580 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-107374182402 2024-09-15T20:39:32.580 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T20:39:32.581 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=107374182402 2024-09-15T20:39:32.581 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 107374182402' 2024-09-15T20:39:32.582 INFO:tasks.workunit.client.0.smithi136.stderr:waiting osd.0 seq 107374182402 2024-09-15T20:39:32.582 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T20:39:32.859 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 107374182402 -lt 107374182402 2024-09-15T20:39:32.859 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T20:39:32.859 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-42949672979 2024-09-15T20:39:32.860 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T20:39:32.860 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-15T20:39:32.861 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-42949672979 2024-09-15T20:39:32.861 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T20:39:32.862 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=42949672979 2024-09-15T20:39:32.862 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 42949672979' 2024-09-15T20:39:32.862 INFO:tasks.workunit.client.0.smithi136.stderr:waiting osd.1 seq 42949672979 2024-09-15T20:39:32.862 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-15T20:39:33.182 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 42949672979 -lt 42949672979 2024-09-15T20:39:33.183 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T20:39:33.183 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-64424509452 2024-09-15T20:39:33.183 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T20:39:33.184 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-15T20:39:33.184 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-64424509452 2024-09-15T20:39:33.185 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T20:39:33.185 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=64424509452 2024-09-15T20:39:33.185 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 64424509452' 2024-09-15T20:39:33.186 INFO:tasks.workunit.client.0.smithi136.stderr:waiting osd.2 seq 64424509452 2024-09-15T20:39:33.186 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-15T20:39:33.463 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 64424509451 -lt 64424509452 2024-09-15T20:39:33.464 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-15T20:39:34.465 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-15T20:39:34.465 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-15T20:39:34.778 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 64424509452 -lt 64424509452 2024-09-15T20:39:34.779 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-15T20:39:34.779 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T20:39:34.779 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T20:39:35.123 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 1 == 0 2024-09-15T20:39:35.123 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-15T20:39:35.123 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-15T20:39:35.123 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-15T20:39:35.123 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T20:39:35.123 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T20:39:35.123 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T20:39:35.123 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T20:39:35.397 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=0 2024-09-15T20:39:35.398 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-15T20:39:35.398 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T20:39:35.398 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T20:39:35.764 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 0 = 1 2024-09-15T20:39:35.764 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 0 '!=' -1 2024-09-15T20:39:35.764 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: loop=0 2024-09-15T20:39:35.764 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1652: wait_for_clean: num_active_clean=0 2024-09-15T20:39:35.764 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-15T20:39:35.764 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 0.1 2024-09-15T20:39:35.866 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-15T20:39:35.866 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-15T20:39:35.866 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-15T20:39:35.866 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-15T20:39:35.866 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T20:39:35.866 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T20:39:35.866 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T20:39:35.866 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T20:39:36.137 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=0 2024-09-15T20:39:36.138 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-15T20:39:36.138 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T20:39:36.138 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T20:39:36.482 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 0 = 1 2024-09-15T20:39:36.482 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 0 '!=' 0 2024-09-15T20:39:36.482 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: get_is_making_recovery_progress 2024-09-15T20:39:36.482 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2024-09-15T20:39:36.483 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-15T20:39:36.483 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-15T20:39:36.483 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-15T20:39:36.483 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2024-09-15T20:39:36.483 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-15T20:39:36.835 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=null 2024-09-15T20:39:36.835 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test null '!=' null 2024-09-15T20:39:36.835 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1655: wait_for_clean: (( 1 >= 13 )) 2024-09-15T20:39:36.835 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-15T20:39:36.835 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 0.2 2024-09-15T20:39:37.036 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-15T20:39:37.036 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-15T20:39:37.037 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-15T20:39:37.037 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-15T20:39:37.037 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T20:39:37.037 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T20:39:37.037 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T20:39:37.037 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T20:39:37.310 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=1 2024-09-15T20:39:37.310 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-15T20:39:37.311 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T20:39:37.311 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T20:39:37.658 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 = 1 2024-09-15T20:39:37.658 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-15T20:39:37.658 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-15T20:39:37.658 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:510: TEST_auto_repair_bluestore_scrub: get_pg testpool SOMETHING 2024-09-15T20:39:37.658 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1084: get_pg: local poolname=testpool 2024-09-15T20:39:37.658 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1085: get_pg: local objectname=SOMETHING 2024-09-15T20:39:37.658 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1087: get_pg: ceph --format json osd map testpool SOMETHING 2024-09-15T20:39:37.658 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1087: get_pg: jq -r .pgid 2024-09-15T20:39:37.944 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:510: TEST_auto_repair_bluestore_scrub: local pgid=1.0 2024-09-15T20:39:37.945 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:511: TEST_auto_repair_bluestore_scrub: get_primary testpool SOMETHING 2024-09-15T20:39:37.946 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1187: get_primary: local poolname=testpool 2024-09-15T20:39:37.946 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1188: get_primary: local objectname=SOMETHING 2024-09-15T20:39:37.946 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1190: get_primary: ceph --format json osd map testpool SOMETHING 2024-09-15T20:39:37.946 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1191: get_primary: jq .acting_primary 2024-09-15T20:39:38.224 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:511: TEST_auto_repair_bluestore_scrub: local primary=1 2024-09-15T20:39:38.225 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:512: TEST_auto_repair_bluestore_scrub: get_last_scrub_stamp 1.0 2024-09-15T20:39:38.225 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-15T20:39:38.225 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-15T20:39:38.225 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-15T20:39:38.225 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-15T20:39:38.496 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:512: TEST_auto_repair_bluestore_scrub: local last_scrub_stamp=2024-09-15T20:39:15.419765+0000 2024-09-15T20:39:38.496 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:513: TEST_auto_repair_bluestore_scrub: ceph tell 1.0 schedule-scrub 2024-09-15T20:39:38.592 INFO:tasks.workunit.client.0.smithi136.stdout:{ 2024-09-15T20:39:38.592 INFO:tasks.workunit.client.0.smithi136.stdout: "deep": false, 2024-09-15T20:39:38.592 INFO:tasks.workunit.client.0.smithi136.stdout: "must": false, 2024-09-15T20:39:38.592 INFO:tasks.workunit.client.0.smithi136.stdout: "stamp": "2024-09-08T20:37:58.590656+0000" 2024-09-15T20:39:38.592 INFO:tasks.workunit.client.0.smithi136.stdout:} 2024-09-15T20:39:38.602 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:516: TEST_auto_repair_bluestore_scrub: wait_for_scrub 1.0 2024-09-15T20:39:15.419765+0000 2024-09-15T20:39:38.602 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: local pgid=1.0 2024-09-15T20:39:38.602 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2018: wait_for_scrub: local last_scrub=2024-09-15T20:39:15.419765+0000 2024-09-15T20:39:38.602 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2019: wait_for_scrub: local sname=last_scrub_stamp 2024-09-15T20:39:38.603 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i=0 )) 2024-09-15T20:39:38.603 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-15T20:39:38.603 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-15T20:39:38.603 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-15T20:39:38.603 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-15T20:39:38.603 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-15T20:39:38.603 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-15T20:39:38.893 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-15T20:39:15.419765+0000 '>' 2024-09-15T20:39:15.419765+0000 2024-09-15T20:39:38.893 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-15T20:39:39.894 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-15T20:39:39.894 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-15T20:39:39.895 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-15T20:39:39.895 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-15T20:39:39.895 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-15T20:39:39.896 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-15T20:39:39.896 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-15T20:39:40.180 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-15T20:39:15.419765+0000 '>' 2024-09-15T20:39:15.419765+0000 2024-09-15T20:39:40.180 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-15T20:39:41.181 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-15T20:39:41.181 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-15T20:39:41.182 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-15T20:39:41.182 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-15T20:39:41.183 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-15T20:39:41.183 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-15T20:39:41.183 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-15T20:39:41.505 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-15T20:39:15.419765+0000 '>' 2024-09-15T20:39:15.419765+0000 2024-09-15T20:39:41.505 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-15T20:39:42.506 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-15T20:39:42.506 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-15T20:39:42.507 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-15T20:39:42.507 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-15T20:39:42.507 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-15T20:39:42.507 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-15T20:39:42.508 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-15T20:39:42.794 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-15T20:39:40.375744+0000 '>' 2024-09-15T20:39:15.419765+0000 2024-09-15T20:39:42.794 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2023: wait_for_scrub: return 0 2024-09-15T20:39:42.794 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:517: TEST_auto_repair_bluestore_scrub: ceph pg dump pgs 2024-09-15T20:39:43.057 INFO:tasks.workunit.client.0.smithi136.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-15T20:39:43.057 INFO:tasks.workunit.client.0.smithi136.stdout:1.0 1 0 0 0 0 7 0 0 1 0 1 active+clean 2024-09-15T20:39:40.556042+0000 21'1 26:62 [1,0] 1 [1,0] 1 21'1 2024-09-15T20:39:40.375744+0000 21'1 2024-09-15T20:39:40.375744+0000 0 0 periodic scrub scheduled @ 2024-09-16T20:39:40.375744+0000 1 0 2024-09-15T20:39:43.057 INFO:tasks.workunit.client.0.smithi136.stdout: 2024-09-15T20:39:43.057 INFO:tasks.workunit.client.0.smithi136.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-15T20:39:43.057 INFO:tasks.workunit.client.0.smithi136.stderr:dumped pgs 2024-09-15T20:39:43.068 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:519: TEST_auto_repair_bluestore_scrub: sleep 2 2024-09-15T20:39:45.070 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:520: TEST_auto_repair_bluestore_scrub: ceph pg dump pgs 2024-09-15T20:39:45.344 INFO:tasks.workunit.client.0.smithi136.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-15T20:39:45.344 INFO:tasks.workunit.client.0.smithi136.stdout:1.0 1 0 0 0 0 7 0 0 1 0 1 active+clean 2024-09-15T20:39:40.556042+0000 21'1 26:62 [1,0] 1 [1,0] 1 21'1 2024-09-15T20:39:40.375744+0000 21'1 2024-09-15T20:39:40.375744+0000 0 0 periodic scrub scheduled @ 2024-09-16T20:39:40.375744+0000 1 0 2024-09-15T20:39:45.344 INFO:tasks.workunit.client.0.smithi136.stdout: 2024-09-15T20:39:45.344 INFO:tasks.workunit.client.0.smithi136.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-15T20:39:45.345 INFO:tasks.workunit.client.0.smithi136.stderr:dumped pgs 2024-09-15T20:39:45.355 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:521: TEST_auto_repair_bluestore_scrub: sleep 2 2024-09-15T20:39:47.356 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:522: TEST_auto_repair_bluestore_scrub: ceph pg dump pgs 2024-09-15T20:39:47.627 INFO:tasks.workunit.client.0.smithi136.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-15T20:39:47.627 INFO:tasks.workunit.client.0.smithi136.stdout:1.0 1 0 0 0 0 7 0 0 1 0 1 active+clean 2024-09-15T20:39:40.556042+0000 21'1 26:62 [1,0] 1 [1,0] 1 21'1 2024-09-15T20:39:40.375744+0000 21'1 2024-09-15T20:39:40.375744+0000 0 0 periodic scrub scheduled @ 2024-09-16T20:39:40.375744+0000 1 0 2024-09-15T20:39:47.628 INFO:tasks.workunit.client.0.smithi136.stdout: 2024-09-15T20:39:47.628 INFO:tasks.workunit.client.0.smithi136.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-15T20:39:47.628 INFO:tasks.workunit.client.0.smithi136.stderr:dumped pgs 2024-09-15T20:39:47.639 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:523: TEST_auto_repair_bluestore_scrub: sleep 5 2024-09-15T20:39:52.641 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:524: TEST_auto_repair_bluestore_scrub: wait_for_clean 2024-09-15T20:39:52.641 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-15T20:39:52.641 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-15T20:39:52.641 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-15T20:39:52.641 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-15T20:39:52.642 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-15T20:39:52.642 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-15T20:39:52.642 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-15T20:39:52.642 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-15T20:39:52.642 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-15T20:39:52.743 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-15T20:39:52.743 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-15T20:39:52.743 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-15T20:39:52.743 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-15T20:39:52.743 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-15T20:39:52.743 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-15T20:39:53.022 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-15T20:39:53.022 INFO:tasks.workunit.client.0.smithi136.stderr:1 2024-09-15T20:39:53.022 INFO:tasks.workunit.client.0.smithi136.stderr:2' 2024-09-15T20:39:53.023 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-15T20:39:53.023 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T20:39:53.023 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-15T20:39:53.126 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=107374182407 2024-09-15T20:39:53.126 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 107374182407 2024-09-15T20:39:53.126 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-107374182407' 2024-09-15T20:39:53.126 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T20:39:53.127 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-15T20:39:53.230 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=42949672984 2024-09-15T20:39:53.230 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 42949672984 2024-09-15T20:39:53.230 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-107374182407 1-42949672984' 2024-09-15T20:39:53.230 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T20:39:53.230 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-15T20:39:53.332 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=64424509457 2024-09-15T20:39:53.332 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 64424509457 2024-09-15T20:39:53.332 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-107374182407 1-42949672984 2-64424509457' 2024-09-15T20:39:53.332 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T20:39:53.332 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-107374182407 2024-09-15T20:39:53.333 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T20:39:53.333 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-15T20:39:53.334 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-107374182407 2024-09-15T20:39:53.334 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T20:39:53.335 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=107374182407 2024-09-15T20:39:53.335 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 107374182407' 2024-09-15T20:39:53.335 INFO:tasks.workunit.client.0.smithi136.stdout:waiting osd.0 seq 107374182407 2024-09-15T20:39:53.335 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T20:39:53.612 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 107374182406 -lt 107374182407 2024-09-15T20:39:53.612 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-15T20:39:54.613 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-15T20:39:54.613 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T20:39:54.903 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 107374182407 -lt 107374182407 2024-09-15T20:39:54.903 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T20:39:54.904 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-42949672984 2024-09-15T20:39:54.904 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T20:39:54.904 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-15T20:39:54.908 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-42949672984 2024-09-15T20:39:54.908 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T20:39:54.908 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=42949672984 2024-09-15T20:39:54.908 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 42949672984' 2024-09-15T20:39:54.909 INFO:tasks.workunit.client.0.smithi136.stdout:waiting osd.1 seq 42949672984 2024-09-15T20:39:54.909 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-15T20:39:55.181 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 42949672984 -lt 42949672984 2024-09-15T20:39:55.182 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T20:39:55.182 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-64424509457 2024-09-15T20:39:55.182 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T20:39:55.183 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-15T20:39:55.183 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-64424509457 2024-09-15T20:39:55.184 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T20:39:55.184 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=64424509457 2024-09-15T20:39:55.185 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 64424509457' 2024-09-15T20:39:55.185 INFO:tasks.workunit.client.0.smithi136.stdout:waiting osd.2 seq 64424509457 2024-09-15T20:39:55.185 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-15T20:39:55.462 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 64424509457 -lt 64424509457 2024-09-15T20:39:55.462 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-15T20:39:55.463 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T20:39:55.463 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T20:39:55.804 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 1 == 0 2024-09-15T20:39:55.804 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-15T20:39:55.804 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-15T20:39:55.804 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-15T20:39:55.804 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T20:39:55.804 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T20:39:55.804 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T20:39:55.804 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T20:39:56.080 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=1 2024-09-15T20:39:56.080 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-15T20:39:56.080 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T20:39:56.080 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T20:39:56.423 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 = 1 2024-09-15T20:39:56.423 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-15T20:39:56.423 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-15T20:39:56.423 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:525: TEST_auto_repair_bluestore_scrub: ceph pg dump pgs 2024-09-15T20:39:56.682 INFO:tasks.workunit.client.0.smithi136.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-15T20:39:56.683 INFO:tasks.workunit.client.0.smithi136.stdout:1.0 1 0 0 0 0 7 0 0 1 0 1 active+clean 2024-09-15T20:39:40.556042+0000 21'1 26:62 [1,0] 1 [1,0] 1 21'1 2024-09-15T20:39:40.375744+0000 21'1 2024-09-15T20:39:40.375744+0000 0 0 periodic scrub scheduled @ 2024-09-16T20:39:40.375744+0000 1 0 2024-09-15T20:39:56.683 INFO:tasks.workunit.client.0.smithi136.stdout: 2024-09-15T20:39:56.683 INFO:tasks.workunit.client.0.smithi136.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-15T20:39:56.683 INFO:tasks.workunit.client.0.smithi136.stderr:dumped pgs 2024-09-15T20:39:56.695 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:528: TEST_auto_repair_bluestore_scrub: get_not_primary testpool SOMETHING 2024-09-15T20:39:56.695 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1220: get_not_primary: local poolname=testpool 2024-09-15T20:39:56.695 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1221: get_not_primary: local objectname=SOMETHING 2024-09-15T20:39:56.695 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1223: get_not_primary: get_primary testpool SOMETHING 2024-09-15T20:39:56.695 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1187: get_primary: local poolname=testpool 2024-09-15T20:39:56.695 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1188: get_primary: local objectname=SOMETHING 2024-09-15T20:39:56.696 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1190: get_primary: ceph --format json osd map testpool SOMETHING 2024-09-15T20:39:56.696 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1191: get_primary: jq .acting_primary 2024-09-15T20:39:56.970 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1223: get_not_primary: local primary=1 2024-09-15T20:39:56.971 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1224: get_not_primary: ceph --format json osd map testpool SOMETHING 2024-09-15T20:39:56.971 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1225: get_not_primary: jq '.acting | map(select (. != 1)) | .[0]' 2024-09-15T20:39:57.250 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:528: TEST_auto_repair_bluestore_scrub: objectstore_tool td/osd-scrub-repair 0 SOMETHING list-attrs 2024-09-15T20:39:57.250 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2024-09-15T20:39:57.250 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2024-09-15T20:39:57.250 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=0 2024-09-15T20:39:57.251 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2024-09-15T20:39:57.251 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 SOMETHING list-attrs 2024-09-15T20:39:57.251 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-09-15T20:39:57.251 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2024-09-15T20:39:57.251 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=0 2024-09-15T20:39:57.251 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2024-09-15T20:39:57.251 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2024-09-15T20:39:57.251 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-15T20:39:57.251 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-15T20:39:57.251 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-15T20:39:57.251 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-15T20:39:57.251 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-15T20:39:57.557 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-15T20:39:57.557 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 SOMETHING list-attrs 2024-09-15T20:39:57.557 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-15T20:39:57.557 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-15T20:39:57.557 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=0 2024-09-15T20:39:57.557 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-15T20:39:57.557 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-09-15T20:39:57.557 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 SOMETHING list-attrs 2024-09-15T20:39:58.075 INFO:tasks.workunit.client.0.smithi136.stdout:_ 2024-09-15T20:39:58.075 INFO:tasks.workunit.client.0.smithi136.stdout:snapset 2024-09-15T20:39:58.343 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-scrub-backoff-ratio=0 2024-09-15T20:39:58.343 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2024-09-15T20:39:58.343 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-15T20:39:58.343 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=0 2024-09-15T20:39:58.343 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-15T20:39:58.343 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-09-15T20:39:58.344 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=ce4892c7-6c6d-4b4f-8e6f-07a3e11133e5 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-15T20:39:58.344 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T20:39:58.344 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T20:39:58.344 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T20:39:58.344 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-09-15T20:39:58.344 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-09-15T20:39:58.344 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-15T20:39:58.344 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-15T20:39:58.344 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-15T20:39:58.344 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-15T20:39:58.344 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-15T20:39:58.344 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-15T20:39:58.344 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-15T20:39:58.345 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T20:39:58.345 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52602 2024-09-15T20:39:58.345 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.52602/$cluster-$name.asok' 2024-09-15T20:39:58.345 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.52602/$cluster-$name.asok' 2024-09-15T20:39:58.345 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-15T20:39:58.345 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-15T20:39:58.345 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-15T20:39:58.345 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T20:39:58.345 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T20:39:58.345 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T20:39:58.345 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T20:39:58.345 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-15T20:39:58.345 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+='--osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-scrub-backoff-ratio=0' 2024-09-15T20:39:58.346 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-09-15T20:39:58.346 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.0 2024-09-15T20:39:58.346 INFO:tasks.workunit.client.0.smithi136.stderr:start osd.0 2024-09-15T20:39:58.346 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 0 --fsid=ce4892c7-6c6d-4b4f-8e6f-07a3e11133e5 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.52602/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-scrub-backoff-ratio=0 2024-09-15T20:39:58.346 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-09-15T20:39:58.346 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 0 = 0 ']' 2024-09-15T20:39:58.347 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-15T20:39:58.347 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-15T20:39:58.347 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-15T20:39:58.373 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:39:58.371+0000 7f58fd1cc640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T20:39:58.375 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:39:58.373+0000 7f58fd1cc640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T20:39:58.376 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:39:58.374+0000 7f58fd1cc640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T20:39:58.637 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 0 2024-09-15T20:39:58.637 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-15T20:39:58.638 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2024-09-15T20:39:58.638 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-15T20:39:58.638 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-15T20:39:58.638 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:39:58.638 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-15T20:39:58.638 INFO:tasks.workunit.client.0.smithi136.stderr:0 2024-09-15T20:39:58.638 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:39:58.638 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T20:39:58.914 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:39:59.479 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:39:59.477+0000 7f58fd1cc640 -1 Falling back to public interface 2024-09-15T20:39:59.915 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:39:59.916 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:39:59.916 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-15T20:39:59.916 INFO:tasks.workunit.client.0.smithi136.stderr:1 2024-09-15T20:39:59.916 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:39:59.916 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T20:40:00.199 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:40:01.200 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:40:01.201 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:40:01.201 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-15T20:40:01.201 INFO:tasks.workunit.client.0.smithi136.stderr:2 2024-09-15T20:40:01.201 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:40:01.201 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T20:40:01.480 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:40:02.303 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:40:02.300+0000 7f58fd1cc640 -1 osd.0 26 log_to_monitors true 2024-09-15T20:40:02.481 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:40:02.481 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:40:02.482 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-15T20:40:02.482 INFO:tasks.workunit.client.0.smithi136.stderr:3 2024-09-15T20:40:02.482 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:40:02.482 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T20:40:02.862 INFO:tasks.workunit.client.0.smithi136.stderr:osd.0 up in weight 1 up_from 30 up_thru 0 down_at 27 last_clean_interval [25,26) [v2:127.0.0.1:6802/2177424933,v1:127.0.0.1:6803/2177424933] [v2:127.0.0.1:6804/2177424933,v1:127.0.0.1:6805/2177424933] exists,up e9da054b-8a14-432f-bc7f-311133621289 2024-09-15T20:40:02.862 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-15T20:40:02.863 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-15T20:40:02.863 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-15T20:40:02.863 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2024-09-15T20:40:02.863 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-15T20:40:02.863 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-15T20:40:02.863 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-15T20:40:02.863 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-15T20:40:02.863 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-15T20:40:02.863 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-15T20:40:02.863 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-15T20:40:02.863 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-15T20:40:02.863 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-15T20:40:02.952 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-15T20:40:02.952 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-15T20:40:02.952 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-15T20:40:02.953 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-15T20:40:02.953 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-15T20:40:02.953 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-15T20:40:03.294 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-15T20:40:03.295 INFO:tasks.workunit.client.0.smithi136.stderr:1 2024-09-15T20:40:03.295 INFO:tasks.workunit.client.0.smithi136.stderr:2' 2024-09-15T20:40:03.295 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-15T20:40:03.296 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T20:40:03.296 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-15T20:40:03.398 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=128849018882 2024-09-15T20:40:03.399 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 128849018882 2024-09-15T20:40:03.399 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-128849018882' 2024-09-15T20:40:03.399 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T20:40:03.399 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-15T20:40:03.515 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=42949672987 2024-09-15T20:40:03.515 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 42949672987 2024-09-15T20:40:03.515 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-128849018882 1-42949672987' 2024-09-15T20:40:03.515 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T20:40:03.515 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-15T20:40:03.617 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=64424509460 2024-09-15T20:40:03.618 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 64424509460 2024-09-15T20:40:03.618 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-128849018882 1-42949672987 2-64424509460' 2024-09-15T20:40:03.618 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T20:40:03.618 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-128849018882 2024-09-15T20:40:03.618 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T20:40:03.619 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-15T20:40:03.619 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-128849018882 2024-09-15T20:40:03.619 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T20:40:03.620 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=128849018882 2024-09-15T20:40:03.620 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 128849018882' 2024-09-15T20:40:03.621 INFO:tasks.workunit.client.0.smithi136.stderr:waiting osd.0 seq 128849018882 2024-09-15T20:40:03.621 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T20:40:03.898 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 1 -lt 128849018882 2024-09-15T20:40:03.898 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-15T20:40:04.899 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-15T20:40:04.899 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T20:40:05.186 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 128849018882 -lt 128849018882 2024-09-15T20:40:05.186 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T20:40:05.187 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-42949672987 2024-09-15T20:40:05.187 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T20:40:05.187 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-15T20:40:05.188 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-42949672987 2024-09-15T20:40:05.188 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T20:40:05.189 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=42949672987 2024-09-15T20:40:05.189 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 42949672987' 2024-09-15T20:40:05.190 INFO:tasks.workunit.client.0.smithi136.stderr:waiting osd.1 seq 42949672987 2024-09-15T20:40:05.190 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-15T20:40:05.465 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 42949672987 -lt 42949672987 2024-09-15T20:40:05.465 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T20:40:05.466 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-64424509460 2024-09-15T20:40:05.466 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T20:40:05.467 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-15T20:40:05.467 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-64424509460 2024-09-15T20:40:05.467 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T20:40:05.468 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=64424509460 2024-09-15T20:40:05.468 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 64424509460' 2024-09-15T20:40:05.468 INFO:tasks.workunit.client.0.smithi136.stderr:waiting osd.2 seq 64424509460 2024-09-15T20:40:05.469 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-15T20:40:05.743 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 64424509460 -lt 64424509460 2024-09-15T20:40:05.744 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-15T20:40:05.744 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T20:40:05.744 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T20:40:06.085 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 1 == 0 2024-09-15T20:40:06.085 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-15T20:40:06.085 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-15T20:40:06.085 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-15T20:40:06.085 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T20:40:06.085 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T20:40:06.085 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T20:40:06.085 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T20:40:06.356 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=1 2024-09-15T20:40:06.356 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-15T20:40:06.356 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T20:40:06.356 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T20:40:06.711 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 = 1 2024-09-15T20:40:06.712 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-15T20:40:06.712 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-15T20:40:06.712 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:529: TEST_auto_repair_bluestore_scrub: rados --pool testpool get SOMETHING td/osd-scrub-repair/COPY 2024-09-15T20:40:06.774 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:530: TEST_auto_repair_bluestore_scrub: diff td/osd-scrub-repair/ORIGINAL td/osd-scrub-repair/COPY 2024-09-15T20:40:06.775 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:531: TEST_auto_repair_bluestore_scrub: grep scrub_finish td/osd-scrub-repair/osd.1.log 2024-09-15T20:40:06.777 INFO:tasks.workunit.client.0.smithi136.stdout:2024-09-15T20:39:39.372+0000 7f5c32a63640 10 osd.1 pg_epoch: 26 pg[1.0( v 21'1 (0'0,21'1] local-lis/les=25/26 n=1 ec=17/17 lis/c=25/25 les/c/f=26/26/0 sis=25) [1,0] r=0 lpr=25 crt=21'1 lcod 0'0 mlcod 0'0 active+clean+scrubbing [ 1.0: DEEP_SCRUB_ON_ERROR ] ] scrubber: scrub_finish before flags: DEEP_SCRUB_ON_ERROR. repair state: no-repair. deep_scrub_on_error: 1 2024-09-15T20:40:06.777 INFO:tasks.workunit.client.0.smithi136.stdout:2024-09-15T20:39:39.372+0000 7f5c32a63640 15 osd.1 pg_epoch: 26 pg[1.0( v 21'1 (0'0,21'1] local-lis/les=25/26 n=1 ec=17/17 lis/c=25/25 les/c/f=26/26/0 sis=25) [1,0] r=0 lpr=25 crt=21'1 lcod 0'0 mlcod 0'0 active+clean+scrubbing [ 1.0: DEEP_SCRUB_ON_ERROR ] ] scrubber: scrub_finish Try to auto repair after scrub errors 2024-09-15T20:40:06.777 INFO:tasks.workunit.client.0.smithi136.stdout:2024-09-15T20:39:39.372+0000 7f5c32a63640 10 osd.1 pg_epoch: 26 pg[1.0( v 21'1 (0'0,21'1] local-lis/les=25/26 n=1 ec=17/17 lis/c=25/25 les/c/f=26/26/0 sis=25) [1,0] r=0 lpr=25 crt=21'1 lcod 0'0 mlcod 0'0 active+clean+scrubbing [ 1.0: ] ] scrubber: _scrub_finish info stats: valid m_is_repair: 0 2024-09-15T20:40:06.777 INFO:tasks.workunit.client.0.smithi136.stdout:2024-09-15T20:39:40.374+0000 7f5c32a63640 10 osd.1 pg_epoch: 26 pg[1.0( v 21'1 (0'0,21'1] local-lis/les=25/26 n=1 ec=17/17 lis/c=25/25 les/c/f=26/26/0 sis=25) [1,0] r=0 lpr=25 crt=21'1 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep+inconsistent [ 1.0: AUTO_REPAIR ] NEED_AUTO] scrubber: scrub_finish before flags: AUTO_REPAIR. repair state: no-repair. deep_scrub_on_error: 0 2024-09-15T20:40:06.777 INFO:tasks.workunit.client.0.smithi136.stdout:2024-09-15T20:39:40.374+0000 7f5c32a63640 10 osd.1 pg_epoch: 26 pg[1.0( v 21'1 (0'0,21'1] local-lis/les=25/26 n=1 ec=17/17 lis/c=25/25 les/c/f=26/26/0 sis=25) [1,0] r=0 lpr=25 crt=21'1 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep+inconsistent [ 1.0: AUTO_REPAIR ] ] scrubber: _scrub_finish info stats: valid m_is_repair: 1 2024-09-15T20:40:06.777 INFO:tasks.workunit.client.0.smithi136.stdout:2024-09-15T20:39:40.374+0000 7f5c32a63640 15 osd.1 pg_epoch: 26 pg[1.0( v 21'1 (0'0,21'1] local-lis/les=25/26 n=1 ec=17/17 lis/c=25/25 les/c/f=26/26/0 sis=25) [1,0] r=0 lpr=25 crt=21'1 lcod 0'0 mlcod 0'0 active+scrubbing+deep+inconsistent+repair [ 1.0: AUTO_REPAIR ] mbc={255={(1+0)=1}}] scrubber: scrub_finish: 1 errors. 1 errors fixed 2024-09-15T20:40:06.777 INFO:tasks.workunit.client.0.smithi136.stdout:2024-09-15T20:39:40.374+0000 7f5c32a63640 20 osd.1 pg_epoch: 26 pg[1.0( v 21'1 (0'0,21'1] local-lis/les=25/26 n=1 ec=17/17 lis/c=25/25 les/c/f=26/26/0 sis=25) [1,0] r=0 lpr=25 crt=21'1 lcod 0'0 mlcod 0'0 active+scrubbing+deep+inconsistent+repair [ 1.0: AUTO_REPAIR ] mbc={255={(1+0)=1}}] scrubber: scrub_finish All may be fixed 2024-09-15T20:40:06.778 INFO:tasks.workunit.client.0.smithi136.stdout:2024-09-15T20:39:40.374+0000 7f5c32a63640 19 osd.1 pg_epoch: 26 pg[1.0( v 21'1 (0'0,21'1] local-lis/les=25/26 n=1 ec=17/17 lis/c=25/25 les/c/f=26/26/0 sis=25) [1,0] r=0 lpr=25 crt=21'1 lcod 0'0 mlcod 0'0 active+scrubbing+deep+inconsistent+repair [ 1.0: AUTO_REPAIR ] mbc={255={(1+0)=1}}] scrubber: scrub_finish shard 1 num_omap_bytes = 0 num_omap_keys = 0 2024-09-15T20:40:06.778 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:534: TEST_auto_repair_bluestore_scrub: ceph pg 1.0 query 2024-09-15T20:40:06.778 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:534: TEST_auto_repair_bluestore_scrub: jq .info.stats.stat_sum.num_objects_repaired 2024-09-15T20:40:06.897 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:534: TEST_auto_repair_bluestore_scrub: COUNT=1 2024-09-15T20:40:06.897 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:535: TEST_auto_repair_bluestore_scrub: test 1 = 1 2024-09-15T20:40:06.897 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:66: run: teardown td/osd-scrub-repair 2024-09-15T20:40:06.897 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/osd-scrub-repair 2024-09-15T20:40:06.897 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-15T20:40:06.897 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/osd-scrub-repair KILL 2024-09-15T20:40:06.897 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-15T20:40:06.897 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-15T20:40:06.897 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-15T20:40:06.897 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-15T20:40:06.898 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-15T20:40:07.004 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-15T20:40:07.004 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-15T20:40:07.005 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-15T20:40:07.005 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-15T20:40:07.006 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-15T20:40:07.007 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-15T20:40:07.007 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-15T20:40:07.008 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T20:40:07.008 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-15T20:40:07.008 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-15T20:40:07.008 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T20:40:07.009 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-15T20:40:07.010 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-15T20:40:07.010 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/osd-scrub-repair 2024-09-15T20:40:07.026 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-15T20:40:07.026 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T20:40:07.026 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52602 2024-09-15T20:40:07.026 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.52602 2024-09-15T20:40:07.027 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-15T20:40:07.028 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-15T20:40:07.028 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:63: run: for func in $funcs 2024-09-15T20:40:07.028 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:64: run: setup td/osd-scrub-repair 2024-09-15T20:40:07.028 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/osd-scrub-repair 2024-09-15T20:40:07.028 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/osd-scrub-repair 2024-09-15T20:40:07.028 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/osd-scrub-repair 2024-09-15T20:40:07.028 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-15T20:40:07.028 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/osd-scrub-repair KILL 2024-09-15T20:40:07.028 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-15T20:40:07.028 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-15T20:40:07.028 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-15T20:40:07.028 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-15T20:40:07.029 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-15T20:40:07.029 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-15T20:40:07.029 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-15T20:40:07.030 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-15T20:40:07.030 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-15T20:40:07.031 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-15T20:40:07.031 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-15T20:40:07.032 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-15T20:40:07.032 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T20:40:07.033 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-15T20:40:07.033 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-15T20:40:07.033 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T20:40:07.034 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-15T20:40:07.035 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-15T20:40:07.035 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/osd-scrub-repair 2024-09-15T20:40:07.036 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-15T20:40:07.036 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T20:40:07.036 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52602 2024-09-15T20:40:07.036 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.52602 2024-09-15T20:40:07.037 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-15T20:40:07.037 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-15T20:40:07.037 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/osd-scrub-repair 2024-09-15T20:40:07.038 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2024-09-15T20:40:07.038 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T20:40:07.038 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52602 2024-09-15T20:40:07.038 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.52602 2024-09-15T20:40:07.043 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2024-09-15T20:40:07.043 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 4096 -le 1024 ']' 2024-09-15T20:40:07.043 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2024-09-15T20:40:07.043 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/osd-scrub-repair 1' TERM HUP INT 2024-09-15T20:40:07.043 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:65: run: TEST_auto_repair_bluestore_tag td/osd-scrub-repair 2024-09-15T20:40:07.043 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:383: TEST_auto_repair_bluestore_tag: local dir=td/osd-scrub-repair 2024-09-15T20:40:07.043 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:384: TEST_auto_repair_bluestore_tag: local poolname=testpool 2024-09-15T20:40:07.043 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:387: TEST_auto_repair_bluestore_tag: run_mon td/osd-scrub-repair a 2024-09-15T20:40:07.043 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/osd-scrub-repair 2024-09-15T20:40:07.043 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2024-09-15T20:40:07.043 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2024-09-15T20:40:07.044 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2024-09-15T20:40:07.044 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/osd-scrub-repair/a 2024-09-15T20:40:07.044 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-scrub-repair/a --run-dir=td/osd-scrub-repair 2024-09-15T20:40:07.501 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2024-09-15T20:40:07.501 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-15T20:40:07.501 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-15T20:40:07.501 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-15T20:40:07.501 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T20:40:07.501 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52602 2024-09-15T20:40:07.501 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.52602/$cluster-$name.asok' 2024-09-15T20:40:07.501 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-scrub-repair/a '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.52602/$cluster-$name.asok' --mon-cluster-log-file=td/osd-scrub-repair/log --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-09-15T20:40:07.684 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2024-09-15T20:40:07.685 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2024-09-15T20:40:07.685 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-15T20:40:07.685 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-15T20:40:07.685 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2024-09-15T20:40:07.686 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2024-09-15T20:40:07.686 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-15T20:40:07.686 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-15T20:40:07.687 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-15T20:40:07.687 INFO:tasks.workunit.client.0.smithi136.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-15T20:40:07.687 INFO:tasks.workunit.client.0.smithi136.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T20:40:07.687 INFO:tasks.workunit.client.0.smithi136.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52602 2024-09-15T20:40:07.687 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.52602/ceph-mon.a.asok 2024-09-15T20:40:07.687 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-15T20:40:07.687 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.52602/ceph-mon.a.asok config get fsid 2024-09-15T20:40:07.767 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2024-09-15T20:40:07.767 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-15T20:40:07.767 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-15T20:40:07.768 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2024-09-15T20:40:07.768 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2024-09-15T20:40:07.768 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-15T20:40:07.768 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-15T20:40:07.768 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-15T20:40:07.769 INFO:tasks.workunit.client.0.smithi136.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-15T20:40:07.769 INFO:tasks.workunit.client.0.smithi136.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T20:40:07.769 INFO:tasks.workunit.client.0.smithi136.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52602 2024-09-15T20:40:07.769 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.52602/ceph-mon.a.asok 2024-09-15T20:40:07.769 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-15T20:40:07.769 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.52602/ceph-mon.a.asok config get mon_host 2024-09-15T20:40:07.847 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:388: TEST_auto_repair_bluestore_tag: run_mgr td/osd-scrub-repair x 2024-09-15T20:40:07.848 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: run_mgr: local dir=td/osd-scrub-repair 2024-09-15T20:40:07.848 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: run_mgr: shift 2024-09-15T20:40:07.848 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: run_mgr: local id=x 2024-09-15T20:40:07.848 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: run_mgr: shift 2024-09-15T20:40:07.848 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:548: run_mgr: local data=td/osd-scrub-repair/x 2024-09-15T20:40:07.848 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:550: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-15T20:40:08.653 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: get_asok_path 2024-09-15T20:40:08.653 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-15T20:40:08.653 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-15T20:40:08.653 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-15T20:40:08.653 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T20:40:08.653 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52602 2024-09-15T20:40:08.653 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.52602/$cluster-$name.asok' 2024-09-15T20:40:08.654 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-15T20:40:08.654 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-scrub-repair/x '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.52602/$cluster-$name.asok' --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-15T20:40:08.682 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:395: TEST_auto_repair_bluestore_tag: local 'ceph_osd_args=--osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-op-queue=wpq' 2024-09-15T20:40:08.682 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:396: TEST_auto_repair_bluestore_tag: seq 0 2 2024-09-15T20:40:08.686 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:396: TEST_auto_repair_bluestore_tag: for id in $(seq 0 2) 2024-09-15T20:40:08.686 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:397: TEST_auto_repair_bluestore_tag: run_osd td/osd-scrub-repair 0 --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-op-queue=wpq 2024-09-15T20:40:08.686 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-scrub-repair 2024-09-15T20:40:08.686 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-15T20:40:08.686 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=0 2024-09-15T20:40:08.686 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-15T20:40:08.686 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-scrub-repair/0 2024-09-15T20:40:08.686 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=ce4892c7-6c6d-4b4f-8e6f-07a3e11133e5 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-15T20:40:08.687 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T20:40:08.687 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T20:40:08.687 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T20:40:08.687 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-09-15T20:40:08.687 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-09-15T20:40:08.687 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-15T20:40:08.687 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-15T20:40:08.687 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-15T20:40:08.687 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-15T20:40:08.687 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-15T20:40:08.687 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-15T20:40:08.687 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-15T20:40:08.687 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T20:40:08.687 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52602 2024-09-15T20:40:08.688 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.52602/$cluster-$name.asok' 2024-09-15T20:40:08.688 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.52602/$cluster-$name.asok' 2024-09-15T20:40:08.688 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-15T20:40:08.688 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-15T20:40:08.688 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-15T20:40:08.688 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-15T20:40:08.688 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-15T20:40:08.688 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T20:40:08.688 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T20:40:08.688 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T20:40:08.688 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T20:40:08.688 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-15T20:40:08.688 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+='--osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-op-queue=wpq' 2024-09-15T20:40:08.688 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-scrub-repair/0 2024-09-15T20:40:08.688 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-15T20:40:08.689 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=0361f5e2-4cb7-4ba9-9b83-24c8876dabdb 2024-09-15T20:40:08.689 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd0 0361f5e2-4cb7-4ba9-9b83-24c8876dabdb' 2024-09-15T20:40:08.689 INFO:tasks.workunit.client.0.smithi136.stdout:add osd0 0361f5e2-4cb7-4ba9-9b83-24c8876dabdb 2024-09-15T20:40:08.689 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-15T20:40:08.706 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQAoRudmN6vvKRAADwxDVHxbG/vWaTqwyjuekA== 2024-09-15T20:40:08.706 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQAoRudmN6vvKRAADwxDVHxbG/vWaTqwyjuekA=="}' 2024-09-15T20:40:08.706 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 0361f5e2-4cb7-4ba9-9b83-24c8876dabdb -i td/osd-scrub-repair/0/new.json 2024-09-15T20:40:08.908 INFO:tasks.workunit.client.0.smithi136.stdout:0 2024-09-15T20:40:08.918 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-scrub-repair/0/new.json 2024-09-15T20:40:08.920 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 0 --fsid=ce4892c7-6c6d-4b4f-8e6f-07a3e11133e5 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.52602/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-op-queue=wpq --mkfs --key AQAoRudmN6vvKRAADwxDVHxbG/vWaTqwyjuekA== --osd-uuid 0361f5e2-4cb7-4ba9-9b83-24c8876dabdb 2024-09-15T20:40:08.952 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:40:08.949+0000 7f9b24c2e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T20:40:08.960 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:40:08.956+0000 7f9b24c2e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T20:40:08.961 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:40:08.958+0000 7f9b24c2e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T20:40:08.961 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:40:08.958+0000 7f9b24c2e640 -1 bdev(0x5605b2283c00 td/osd-scrub-repair/0/block) open stat got: (1) Operation not permitted 2024-09-15T20:40:08.961 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:40:08.958+0000 7f9b24c2e640 -1 bluestore(td/osd-scrub-repair/0) _read_fsid unparsable uuid 2024-09-15T20:40:14.739 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-scrub-repair/0/keyring 2024-09-15T20:40:14.739 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-15T20:40:14.740 INFO:tasks.workunit.client.0.smithi136.stdout:adding osd0 key to auth repository 2024-09-15T20:40:14.740 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd0 key to auth repository 2024-09-15T20:40:14.741 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-scrub-repair/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-15T20:40:15.117 INFO:tasks.workunit.client.0.smithi136.stdout:start osd.0 2024-09-15T20:40:15.118 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.0 2024-09-15T20:40:15.118 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 0 --fsid=ce4892c7-6c6d-4b4f-8e6f-07a3e11133e5 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.52602/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-op-queue=wpq 2024-09-15T20:40:15.118 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-15T20:40:15.118 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-15T20:40:15.119 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-15T20:40:15.148 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:40:15.146+0000 7f18dfb3e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T20:40:15.149 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:40:15.148+0000 7f18dfb3e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T20:40:15.151 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:40:15.149+0000 7f18dfb3e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T20:40:15.395 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 0 2024-09-15T20:40:15.395 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-15T20:40:15.395 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2024-09-15T20:40:15.395 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-15T20:40:15.395 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-15T20:40:15.395 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:40:15.396 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-15T20:40:15.396 INFO:tasks.workunit.client.0.smithi136.stdout:0 2024-09-15T20:40:15.396 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:40:15.396 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T20:40:15.710 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:40:16.712 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:40:16.712 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:40:16.712 INFO:tasks.workunit.client.0.smithi136.stdout:1 2024-09-15T20:40:16.713 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-15T20:40:16.713 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:40:16.713 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T20:40:16.762 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:40:16.760+0000 7f18dfb3e640 -1 Falling back to public interface 2024-09-15T20:40:16.991 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:40:17.993 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:40:17.993 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:40:17.993 INFO:tasks.workunit.client.0.smithi136.stdout:2 2024-09-15T20:40:17.993 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-15T20:40:17.994 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:40:17.994 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T20:40:18.277 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:40:19.279 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:40:19.279 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:40:19.279 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-15T20:40:19.279 INFO:tasks.workunit.client.0.smithi136.stdout:3 2024-09-15T20:40:19.280 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:40:19.280 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T20:40:19.365 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:40:19.363+0000 7f18dfb3e640 -1 osd.0 0 log_to_monitors true 2024-09-15T20:40:19.569 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:40:20.570 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:40:20.570 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:40:20.571 INFO:tasks.workunit.client.0.smithi136.stdout:4 2024-09-15T20:40:20.571 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-15T20:40:20.571 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:40:20.572 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T20:40:20.924 INFO:tasks.workunit.client.0.smithi136.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/41529000,v1:127.0.0.1:6803/41529000] [v2:127.0.0.1:6804/41529000,v1:127.0.0.1:6805/41529000] exists,up 0361f5e2-4cb7-4ba9-9b83-24c8876dabdb 2024-09-15T20:40:20.927 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-15T20:40:20.927 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-15T20:40:20.927 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-15T20:40:20.927 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:396: TEST_auto_repair_bluestore_tag: for id in $(seq 0 2) 2024-09-15T20:40:20.928 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:397: TEST_auto_repair_bluestore_tag: run_osd td/osd-scrub-repair 1 --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-op-queue=wpq 2024-09-15T20:40:20.928 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-scrub-repair 2024-09-15T20:40:20.928 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-15T20:40:20.928 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=1 2024-09-15T20:40:20.928 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-15T20:40:20.928 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-scrub-repair/1 2024-09-15T20:40:20.928 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=ce4892c7-6c6d-4b4f-8e6f-07a3e11133e5 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-15T20:40:20.928 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T20:40:20.928 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T20:40:20.928 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T20:40:20.928 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-09-15T20:40:20.928 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-09-15T20:40:20.928 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-15T20:40:20.929 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-15T20:40:20.929 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-15T20:40:20.929 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-15T20:40:20.929 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-15T20:40:20.929 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-15T20:40:20.929 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-15T20:40:20.929 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T20:40:20.929 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52602 2024-09-15T20:40:20.929 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.52602/$cluster-$name.asok' 2024-09-15T20:40:20.929 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.52602/$cluster-$name.asok' 2024-09-15T20:40:20.929 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-15T20:40:20.929 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-15T20:40:20.929 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-15T20:40:20.929 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-15T20:40:20.930 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-15T20:40:20.930 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T20:40:20.930 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T20:40:20.930 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T20:40:20.930 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T20:40:20.930 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-15T20:40:20.930 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+='--osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-op-queue=wpq' 2024-09-15T20:40:20.930 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-scrub-repair/1 2024-09-15T20:40:20.930 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-15T20:40:20.930 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=8915ab98-eb83-4a5e-a90b-96ef176392d1 2024-09-15T20:40:20.930 INFO:tasks.workunit.client.0.smithi136.stdout:add osd1 8915ab98-eb83-4a5e-a90b-96ef176392d1 2024-09-15T20:40:20.931 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd1 8915ab98-eb83-4a5e-a90b-96ef176392d1' 2024-09-15T20:40:20.931 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-15T20:40:20.946 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQA0RudmiiNEOBAA5f3bhx9OHpvIfJfMHAm0tQ== 2024-09-15T20:40:20.946 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQA0RudmiiNEOBAA5f3bhx9OHpvIfJfMHAm0tQ=="}' 2024-09-15T20:40:20.946 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 8915ab98-eb83-4a5e-a90b-96ef176392d1 -i td/osd-scrub-repair/1/new.json 2024-09-15T20:40:21.292 INFO:tasks.workunit.client.0.smithi136.stdout:1 2024-09-15T20:40:21.303 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-scrub-repair/1/new.json 2024-09-15T20:40:21.304 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 1 --fsid=ce4892c7-6c6d-4b4f-8e6f-07a3e11133e5 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.52602/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-op-queue=wpq --mkfs --key AQA0RudmiiNEOBAA5f3bhx9OHpvIfJfMHAm0tQ== --osd-uuid 8915ab98-eb83-4a5e-a90b-96ef176392d1 2024-09-15T20:40:21.336 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:40:21.334+0000 7f3a01217640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T20:40:21.385 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:40:21.383+0000 7f3a01217640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T20:40:21.386 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:40:21.384+0000 7f3a01217640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T20:40:21.386 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:40:21.384+0000 7f3a01217640 -1 bdev(0x5584fe89dc00 td/osd-scrub-repair/1/block) open stat got: (1) Operation not permitted 2024-09-15T20:40:21.386 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:40:21.384+0000 7f3a01217640 -1 bluestore(td/osd-scrub-repair/1) _read_fsid unparsable uuid 2024-09-15T20:40:27.758 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-scrub-repair/1/keyring 2024-09-15T20:40:27.759 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-15T20:40:27.759 INFO:tasks.workunit.client.0.smithi136.stdout:adding osd1 key to auth repository 2024-09-15T20:40:27.760 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd1 key to auth repository 2024-09-15T20:40:27.760 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-scrub-repair/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-15T20:40:28.132 INFO:tasks.workunit.client.0.smithi136.stdout:start osd.1 2024-09-15T20:40:28.132 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.1 2024-09-15T20:40:28.132 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 1 --fsid=ce4892c7-6c6d-4b4f-8e6f-07a3e11133e5 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.52602/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-op-queue=wpq 2024-09-15T20:40:28.132 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-15T20:40:28.133 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-15T20:40:28.133 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-15T20:40:28.161 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:40:28.159+0000 7fa20f60e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T20:40:28.163 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:40:28.161+0000 7fa20f60e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T20:40:28.164 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:40:28.162+0000 7fa20f60e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T20:40:28.409 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 1 2024-09-15T20:40:28.409 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-15T20:40:28.409 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2024-09-15T20:40:28.410 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-15T20:40:28.410 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-15T20:40:28.410 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:40:28.410 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-15T20:40:28.410 INFO:tasks.workunit.client.0.smithi136.stdout:0 2024-09-15T20:40:28.410 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:40:28.410 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T20:40:28.693 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:40:29.521 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:40:29.519+0000 7fa20f60e640 -1 Falling back to public interface 2024-09-15T20:40:29.695 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:40:29.695 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:40:29.695 INFO:tasks.workunit.client.0.smithi136.stdout:1 2024-09-15T20:40:29.696 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-15T20:40:29.696 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:40:29.696 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T20:40:29.981 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:40:30.982 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:40:30.982 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:40:30.982 INFO:tasks.workunit.client.0.smithi136.stdout:2 2024-09-15T20:40:30.983 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-15T20:40:30.983 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:40:30.983 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T20:40:31.268 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:40:32.270 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:40:32.270 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:40:32.270 INFO:tasks.workunit.client.0.smithi136.stdout:3 2024-09-15T20:40:32.271 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-15T20:40:32.271 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:40:32.271 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T20:40:32.416 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:40:32.414+0000 7fa20f60e640 -1 osd.1 0 log_to_monitors true 2024-09-15T20:40:32.616 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:40:33.618 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:40:33.618 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:40:33.618 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-15T20:40:33.619 INFO:tasks.workunit.client.0.smithi136.stdout:4 2024-09-15T20:40:33.619 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:40:33.620 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T20:40:33.902 INFO:tasks.workunit.client.0.smithi136.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/165194958,v1:127.0.0.1:6811/165194958] [v2:127.0.0.1:6812/165194958,v1:127.0.0.1:6813/165194958] exists,up 8915ab98-eb83-4a5e-a90b-96ef176392d1 2024-09-15T20:40:33.906 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-15T20:40:33.906 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-15T20:40:33.906 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-15T20:40:33.906 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:396: TEST_auto_repair_bluestore_tag: for id in $(seq 0 2) 2024-09-15T20:40:33.906 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:397: TEST_auto_repair_bluestore_tag: run_osd td/osd-scrub-repair 2 --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-op-queue=wpq 2024-09-15T20:40:33.906 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-scrub-repair 2024-09-15T20:40:33.906 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-15T20:40:33.906 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=2 2024-09-15T20:40:33.906 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-15T20:40:33.906 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-scrub-repair/2 2024-09-15T20:40:33.907 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=ce4892c7-6c6d-4b4f-8e6f-07a3e11133e5 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-15T20:40:33.907 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T20:40:33.907 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T20:40:33.907 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T20:40:33.907 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/2' 2024-09-15T20:40:33.907 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/2/journal' 2024-09-15T20:40:33.907 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-15T20:40:33.907 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-15T20:40:33.907 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-15T20:40:33.907 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-15T20:40:33.907 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-15T20:40:33.908 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-15T20:40:33.908 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-15T20:40:33.908 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T20:40:33.908 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52602 2024-09-15T20:40:33.908 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.52602/$cluster-$name.asok' 2024-09-15T20:40:33.908 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.52602/$cluster-$name.asok' 2024-09-15T20:40:33.908 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-15T20:40:33.908 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-15T20:40:33.908 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-15T20:40:33.908 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-15T20:40:33.908 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-15T20:40:33.908 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T20:40:33.909 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T20:40:33.909 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T20:40:33.909 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T20:40:33.909 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-15T20:40:33.909 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+='--osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-op-queue=wpq' 2024-09-15T20:40:33.909 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-scrub-repair/2 2024-09-15T20:40:33.910 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-15T20:40:33.910 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=05d70d31-ffd3-4abb-913c-a77fa2a8d969 2024-09-15T20:40:33.910 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd2 05d70d31-ffd3-4abb-913c-a77fa2a8d969' 2024-09-15T20:40:33.910 INFO:tasks.workunit.client.0.smithi136.stdout:add osd2 05d70d31-ffd3-4abb-913c-a77fa2a8d969 2024-09-15T20:40:33.910 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-15T20:40:33.924 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQBBRudm/gH2NhAAbx9R/IiFQYkabVanPS2nIw== 2024-09-15T20:40:33.924 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQBBRudm/gH2NhAAbx9R/IiFQYkabVanPS2nIw=="}' 2024-09-15T20:40:33.924 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 05d70d31-ffd3-4abb-913c-a77fa2a8d969 -i td/osd-scrub-repair/2/new.json 2024-09-15T20:40:34.277 INFO:tasks.workunit.client.0.smithi136.stdout:2 2024-09-15T20:40:34.289 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-scrub-repair/2/new.json 2024-09-15T20:40:34.290 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 2 --fsid=ce4892c7-6c6d-4b4f-8e6f-07a3e11133e5 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.52602/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-op-queue=wpq --mkfs --key AQBBRudm/gH2NhAAbx9R/IiFQYkabVanPS2nIw== --osd-uuid 05d70d31-ffd3-4abb-913c-a77fa2a8d969 2024-09-15T20:40:34.321 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:40:34.319+0000 7fe836520640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T20:40:34.362 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:40:34.360+0000 7fe836520640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T20:40:34.363 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:40:34.361+0000 7fe836520640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T20:40:34.363 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:40:34.361+0000 7fe836520640 -1 bdev(0x55f78fb55c00 td/osd-scrub-repair/2/block) open stat got: (1) Operation not permitted 2024-09-15T20:40:34.363 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:40:34.361+0000 7fe836520640 -1 bluestore(td/osd-scrub-repair/2) _read_fsid unparsable uuid 2024-09-15T20:40:40.469 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-scrub-repair/2/keyring 2024-09-15T20:40:40.470 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-15T20:40:40.471 INFO:tasks.workunit.client.0.smithi136.stdout:adding osd2 key to auth repository 2024-09-15T20:40:40.472 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd2 key to auth repository 2024-09-15T20:40:40.472 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-scrub-repair/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-15T20:40:40.839 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.2 2024-09-15T20:40:40.839 INFO:tasks.workunit.client.0.smithi136.stdout:start osd.2 2024-09-15T20:40:40.840 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 2 --fsid=ce4892c7-6c6d-4b4f-8e6f-07a3e11133e5 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.52602/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-op-queue=wpq 2024-09-15T20:40:40.840 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-15T20:40:40.840 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-15T20:40:40.840 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-15T20:40:40.869 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:40:40.867+0000 7fa6bbc16640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T20:40:40.871 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:40:40.869+0000 7fa6bbc16640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T20:40:40.872 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:40:40.870+0000 7fa6bbc16640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T20:40:41.118 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 2 2024-09-15T20:40:41.118 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-15T20:40:41.118 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=2 2024-09-15T20:40:41.118 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-15T20:40:41.118 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-15T20:40:41.118 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:40:41.119 INFO:tasks.workunit.client.0.smithi136.stdout:0 2024-09-15T20:40:41.119 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-15T20:40:41.119 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:40:41.119 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T20:40:41.396 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:40:41.979 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:40:41.977+0000 7fa6bbc16640 -1 Falling back to public interface 2024-09-15T20:40:42.398 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:40:42.398 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:40:42.398 INFO:tasks.workunit.client.0.smithi136.stdout:1 2024-09-15T20:40:42.398 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-15T20:40:42.399 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:40:42.399 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T20:40:42.698 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:40:43.699 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:40:43.699 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:40:43.699 INFO:tasks.workunit.client.0.smithi136.stdout:2 2024-09-15T20:40:43.700 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-15T20:40:43.700 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:40:43.700 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T20:40:43.993 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:40:44.619 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:40:44.617+0000 7fa6bbc16640 -1 osd.2 0 log_to_monitors true 2024-09-15T20:40:44.994 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:40:44.994 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:40:44.994 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-15T20:40:44.995 INFO:tasks.workunit.client.0.smithi136.stdout:3 2024-09-15T20:40:44.995 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:40:44.995 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T20:40:45.348 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:40:46.349 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:40:46.349 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:40:46.350 INFO:tasks.workunit.client.0.smithi136.stdout:4 2024-09-15T20:40:46.350 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-15T20:40:46.350 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:40:46.350 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T20:40:46.640 INFO:tasks.workunit.client.0.smithi136.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/1858071937,v1:127.0.0.1:6819/1858071937] [v2:127.0.0.1:6820/1858071937,v1:127.0.0.1:6821/1858071937] exists,up 05d70d31-ffd3-4abb-913c-a77fa2a8d969 2024-09-15T20:40:46.641 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-15T20:40:46.641 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-15T20:40:46.641 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-15T20:40:46.641 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:400: TEST_auto_repair_bluestore_tag: create_pool testpool 1 1 2024-09-15T20:40:46.642 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create testpool 1 1 2024-09-15T20:40:47.149 INFO:tasks.workunit.client.0.smithi136.stderr:pool 'testpool' created 2024-09-15T20:40:47.160 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2024-09-15T20:40:48.162 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:401: TEST_auto_repair_bluestore_tag: ceph osd pool set testpool size 2 2024-09-15T20:40:48.571 INFO:tasks.workunit.client.0.smithi136.stderr:set pool 1 size to 2 2024-09-15T20:40:48.584 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:402: TEST_auto_repair_bluestore_tag: wait_for_clean 2024-09-15T20:40:48.584 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-15T20:40:48.584 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-15T20:40:48.584 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-15T20:40:48.584 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-15T20:40:48.584 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-15T20:40:48.584 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-15T20:40:48.584 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-15T20:40:48.584 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-15T20:40:48.584 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-15T20:40:48.673 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-15T20:40:48.673 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-15T20:40:48.673 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-15T20:40:48.673 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-15T20:40:48.673 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-15T20:40:48.673 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-15T20:40:49.129 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-15T20:40:49.130 INFO:tasks.workunit.client.0.smithi136.stderr:1 2024-09-15T20:40:49.130 INFO:tasks.workunit.client.0.smithi136.stderr:2' 2024-09-15T20:40:49.130 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-15T20:40:49.130 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T20:40:49.130 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-15T20:40:49.269 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836487 2024-09-15T20:40:49.269 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836487 2024-09-15T20:40:49.269 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836487' 2024-09-15T20:40:49.269 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T20:40:49.269 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-15T20:40:49.372 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=42949672965 2024-09-15T20:40:49.372 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 42949672965 2024-09-15T20:40:49.372 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836487 1-42949672965' 2024-09-15T20:40:49.372 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T20:40:49.372 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-15T20:40:49.475 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=64424509442 2024-09-15T20:40:49.475 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 64424509442 2024-09-15T20:40:49.475 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836487 1-42949672965 2-64424509442' 2024-09-15T20:40:49.475 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T20:40:49.476 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836487 2024-09-15T20:40:49.476 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T20:40:49.476 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-15T20:40:49.477 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836487 2024-09-15T20:40:49.477 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T20:40:49.478 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836487 2024-09-15T20:40:49.478 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836487' 2024-09-15T20:40:49.478 INFO:tasks.workunit.client.0.smithi136.stdout:waiting osd.0 seq 21474836487 2024-09-15T20:40:49.478 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T20:40:49.756 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836486 -lt 21474836487 2024-09-15T20:40:49.756 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-15T20:40:50.757 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-15T20:40:50.757 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T20:40:51.043 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836487 -lt 21474836487 2024-09-15T20:40:51.043 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T20:40:51.043 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-42949672965 2024-09-15T20:40:51.043 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T20:40:51.044 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-15T20:40:51.045 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-42949672965 2024-09-15T20:40:51.045 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T20:40:51.046 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=42949672965 2024-09-15T20:40:51.046 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 42949672965' 2024-09-15T20:40:51.046 INFO:tasks.workunit.client.0.smithi136.stdout:waiting osd.1 seq 42949672965 2024-09-15T20:40:51.046 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-15T20:40:51.322 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 42949672965 -lt 42949672965 2024-09-15T20:40:51.323 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T20:40:51.323 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-64424509442 2024-09-15T20:40:51.323 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T20:40:51.324 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-15T20:40:51.324 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-64424509442 2024-09-15T20:40:51.324 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T20:40:51.325 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=64424509442 2024-09-15T20:40:51.325 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 64424509442' 2024-09-15T20:40:51.326 INFO:tasks.workunit.client.0.smithi136.stdout:waiting osd.2 seq 64424509442 2024-09-15T20:40:51.326 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-15T20:40:51.601 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 64424509442 -lt 64424509442 2024-09-15T20:40:51.602 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-15T20:40:51.602 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T20:40:51.602 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T20:40:51.947 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 1 == 0 2024-09-15T20:40:51.947 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-15T20:40:51.947 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-15T20:40:51.947 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-15T20:40:51.947 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T20:40:51.947 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T20:40:51.948 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T20:40:51.948 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T20:40:52.221 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=0 2024-09-15T20:40:52.221 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-15T20:40:52.222 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T20:40:52.222 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T20:40:52.562 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 0 = 1 2024-09-15T20:40:52.563 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 0 '!=' -1 2024-09-15T20:40:52.563 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: loop=0 2024-09-15T20:40:52.563 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1652: wait_for_clean: num_active_clean=0 2024-09-15T20:40:52.563 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-15T20:40:52.563 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 0.1 2024-09-15T20:40:52.663 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-15T20:40:52.664 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-15T20:40:52.664 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-15T20:40:52.664 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-15T20:40:52.665 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T20:40:52.665 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T20:40:52.665 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T20:40:52.665 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T20:40:52.939 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=0 2024-09-15T20:40:52.939 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-15T20:40:52.940 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T20:40:52.940 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T20:40:53.279 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 0 = 1 2024-09-15T20:40:53.279 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 0 '!=' 0 2024-09-15T20:40:53.279 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: get_is_making_recovery_progress 2024-09-15T20:40:53.279 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2024-09-15T20:40:53.279 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-15T20:40:53.279 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-15T20:40:53.279 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-15T20:40:53.279 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2024-09-15T20:40:53.279 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-15T20:40:53.624 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=null 2024-09-15T20:40:53.624 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test null '!=' null 2024-09-15T20:40:53.624 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1655: wait_for_clean: (( 1 >= 13 )) 2024-09-15T20:40:53.624 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-15T20:40:53.624 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 0.2 2024-09-15T20:40:53.825 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-15T20:40:53.825 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-15T20:40:53.826 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-15T20:40:53.826 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-15T20:40:53.826 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T20:40:53.826 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T20:40:53.826 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T20:40:53.826 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T20:40:54.097 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=0 2024-09-15T20:40:54.098 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-15T20:40:54.098 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T20:40:54.098 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T20:40:54.445 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 0 = 1 2024-09-15T20:40:54.445 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 0 '!=' 0 2024-09-15T20:40:54.446 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: get_is_making_recovery_progress 2024-09-15T20:40:54.446 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2024-09-15T20:40:54.446 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-15T20:40:54.446 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-15T20:40:54.446 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-15T20:40:54.446 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2024-09-15T20:40:54.446 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-15T20:40:54.795 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=null 2024-09-15T20:40:54.795 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test null '!=' null 2024-09-15T20:40:54.795 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1655: wait_for_clean: (( 2 >= 13 )) 2024-09-15T20:40:54.795 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-15T20:40:54.795 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 0.4 2024-09-15T20:40:55.196 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-15T20:40:55.196 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-15T20:40:55.196 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-15T20:40:55.197 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-15T20:40:55.197 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T20:40:55.197 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T20:40:55.197 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T20:40:55.197 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T20:40:55.471 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=1 2024-09-15T20:40:55.472 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-15T20:40:55.472 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T20:40:55.472 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T20:40:55.858 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 = 1 2024-09-15T20:40:55.858 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-15T20:40:55.858 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-15T20:40:55.858 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:405: TEST_auto_repair_bluestore_tag: local payload=ABCDEF 2024-09-15T20:40:55.858 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:406: TEST_auto_repair_bluestore_tag: echo ABCDEF 2024-09-15T20:40:55.858 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:407: TEST_auto_repair_bluestore_tag: rados --pool testpool put SOMETHING td/osd-scrub-repair/ORIGINAL 2024-09-15T20:40:55.998 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:411: TEST_auto_repair_bluestore_tag: get_not_primary testpool SOMETHING 2024-09-15T20:40:55.998 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1220: get_not_primary: local poolname=testpool 2024-09-15T20:40:55.998 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1221: get_not_primary: local objectname=SOMETHING 2024-09-15T20:40:55.998 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1223: get_not_primary: get_primary testpool SOMETHING 2024-09-15T20:40:55.998 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1187: get_primary: local poolname=testpool 2024-09-15T20:40:55.998 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1188: get_primary: local objectname=SOMETHING 2024-09-15T20:40:55.998 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1190: get_primary: ceph --format json osd map testpool SOMETHING 2024-09-15T20:40:55.998 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1191: get_primary: jq .acting_primary 2024-09-15T20:40:56.276 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1223: get_not_primary: local primary=1 2024-09-15T20:40:56.276 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1224: get_not_primary: ceph --format json osd map testpool SOMETHING 2024-09-15T20:40:56.276 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1225: get_not_primary: jq '.acting | map(select (. != 1)) | .[0]' 2024-09-15T20:40:56.560 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:411: TEST_auto_repair_bluestore_tag: objectstore_tool td/osd-scrub-repair 0 SOMETHING remove 2024-09-15T20:40:56.560 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2024-09-15T20:40:56.561 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2024-09-15T20:40:56.561 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=0 2024-09-15T20:40:56.561 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2024-09-15T20:40:56.561 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 SOMETHING remove 2024-09-15T20:40:56.561 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-09-15T20:40:56.561 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2024-09-15T20:40:56.561 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=0 2024-09-15T20:40:56.561 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2024-09-15T20:40:56.561 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2024-09-15T20:40:56.561 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-15T20:40:56.561 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-15T20:40:56.561 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-15T20:40:56.561 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-15T20:40:56.561 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-15T20:40:56.866 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-15T20:40:56.866 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 SOMETHING remove 2024-09-15T20:40:56.866 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-15T20:40:56.867 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-15T20:40:56.867 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=0 2024-09-15T20:40:56.867 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-15T20:40:56.867 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-09-15T20:40:56.867 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 SOMETHING remove 2024-09-15T20:40:58.537 INFO:tasks.workunit.client.0.smithi136.stdout:remove #1:eb822e21:::SOMETHING:head# 2024-09-15T20:40:59.055 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-op-queue=wpq 2024-09-15T20:40:59.056 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2024-09-15T20:40:59.056 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-15T20:40:59.056 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=0 2024-09-15T20:40:59.056 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-15T20:40:59.056 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-09-15T20:40:59.056 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=ce4892c7-6c6d-4b4f-8e6f-07a3e11133e5 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-15T20:40:59.056 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T20:40:59.056 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T20:40:59.056 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T20:40:59.056 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-09-15T20:40:59.056 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-09-15T20:40:59.056 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-15T20:40:59.056 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-15T20:40:59.057 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-15T20:40:59.057 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-15T20:40:59.057 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-15T20:40:59.057 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-15T20:40:59.057 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-15T20:40:59.057 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T20:40:59.057 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52602 2024-09-15T20:40:59.057 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.52602/$cluster-$name.asok' 2024-09-15T20:40:59.057 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.52602/$cluster-$name.asok' 2024-09-15T20:40:59.057 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-15T20:40:59.057 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-15T20:40:59.057 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-15T20:40:59.057 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T20:40:59.058 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T20:40:59.058 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T20:40:59.058 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T20:40:59.058 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-15T20:40:59.058 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+='--osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-op-queue=wpq' 2024-09-15T20:40:59.058 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-09-15T20:40:59.059 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.0 2024-09-15T20:40:59.059 INFO:tasks.workunit.client.0.smithi136.stderr:start osd.0 2024-09-15T20:40:59.059 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 0 --fsid=ce4892c7-6c6d-4b4f-8e6f-07a3e11133e5 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.52602/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-op-queue=wpq 2024-09-15T20:40:59.059 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-09-15T20:40:59.059 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 0 = 0 ']' 2024-09-15T20:40:59.059 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-15T20:40:59.060 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-15T20:40:59.060 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-15T20:40:59.089 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:40:59.087+0000 7f5c25558640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T20:40:59.091 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:40:59.089+0000 7f5c25558640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T20:40:59.093 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:40:59.090+0000 7f5c25558640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T20:40:59.341 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 0 2024-09-15T20:40:59.341 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-15T20:40:59.341 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2024-09-15T20:40:59.341 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-15T20:40:59.342 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-15T20:40:59.342 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:40:59.342 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-15T20:40:59.342 INFO:tasks.workunit.client.0.smithi136.stderr:0 2024-09-15T20:40:59.342 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:40:59.342 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T20:40:59.618 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:41:00.457 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:41:00.455+0000 7f5c25558640 -1 Falling back to public interface 2024-09-15T20:41:00.620 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:41:00.620 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:41:00.620 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-15T20:41:00.620 INFO:tasks.workunit.client.0.smithi136.stderr:1 2024-09-15T20:41:00.620 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:41:00.620 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T20:41:00.907 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:41:01.909 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:41:01.909 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:41:01.909 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-15T20:41:01.909 INFO:tasks.workunit.client.0.smithi136.stderr:2 2024-09-15T20:41:01.909 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:41:01.909 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T20:41:02.194 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:41:03.196 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:41:03.196 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:41:03.196 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-15T20:41:03.196 INFO:tasks.workunit.client.0.smithi136.stderr:3 2024-09-15T20:41:03.196 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:41:03.196 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T20:41:03.256 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:41:03.254+0000 7f5c25558640 -1 osd.0 20 log_to_monitors true 2024-09-15T20:41:03.490 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:41:04.491 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:41:04.492 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:41:04.492 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-15T20:41:04.492 INFO:tasks.workunit.client.0.smithi136.stderr:4 2024-09-15T20:41:04.492 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:41:04.492 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T20:41:04.782 INFO:tasks.workunit.client.0.smithi136.stderr:osd.0 up in weight 1 up_from 24 up_thru 0 down_at 21 last_clean_interval [5,20) [v2:127.0.0.1:6802/2034971029,v1:127.0.0.1:6803/2034971029] [v2:127.0.0.1:6804/2034971029,v1:127.0.0.1:6805/2034971029] exists,up 0361f5e2-4cb7-4ba9-9b83-24c8876dabdb 2024-09-15T20:41:04.782 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-15T20:41:04.782 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-15T20:41:04.782 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-15T20:41:04.782 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2024-09-15T20:41:04.782 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-15T20:41:04.782 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-15T20:41:04.782 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-15T20:41:04.782 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-15T20:41:04.783 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-15T20:41:04.783 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-15T20:41:04.783 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-15T20:41:04.783 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-15T20:41:04.783 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-15T20:41:04.871 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-15T20:41:04.871 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-15T20:41:04.871 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-15T20:41:04.871 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-15T20:41:04.872 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-15T20:41:04.872 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-15T20:41:05.172 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-15T20:41:05.172 INFO:tasks.workunit.client.0.smithi136.stderr:1 2024-09-15T20:41:05.172 INFO:tasks.workunit.client.0.smithi136.stderr:2' 2024-09-15T20:41:05.172 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-15T20:41:05.172 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T20:41:05.172 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-15T20:41:05.274 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=103079215106 2024-09-15T20:41:05.274 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 103079215106 2024-09-15T20:41:05.274 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-103079215106' 2024-09-15T20:41:05.274 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T20:41:05.274 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-15T20:41:05.378 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=42949672969 2024-09-15T20:41:05.379 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 42949672969 2024-09-15T20:41:05.379 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-103079215106 1-42949672969' 2024-09-15T20:41:05.379 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T20:41:05.379 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-15T20:41:05.481 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=64424509446 2024-09-15T20:41:05.482 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 64424509446 2024-09-15T20:41:05.482 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-103079215106 1-42949672969 2-64424509446' 2024-09-15T20:41:05.482 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T20:41:05.482 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-103079215106 2024-09-15T20:41:05.482 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T20:41:05.482 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-15T20:41:05.483 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-103079215106 2024-09-15T20:41:05.483 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T20:41:05.484 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=103079215106 2024-09-15T20:41:05.484 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 103079215106' 2024-09-15T20:41:05.484 INFO:tasks.workunit.client.0.smithi136.stderr:waiting osd.0 seq 103079215106 2024-09-15T20:41:05.485 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T20:41:05.759 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 1 -lt 103079215106 2024-09-15T20:41:05.759 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-15T20:41:06.760 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-15T20:41:06.760 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T20:41:07.039 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 103079215106 -lt 103079215106 2024-09-15T20:41:07.039 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T20:41:07.040 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-42949672969 2024-09-15T20:41:07.040 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T20:41:07.041 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-15T20:41:07.041 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-42949672969 2024-09-15T20:41:07.041 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T20:41:07.042 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=42949672969 2024-09-15T20:41:07.042 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 42949672969' 2024-09-15T20:41:07.042 INFO:tasks.workunit.client.0.smithi136.stderr:waiting osd.1 seq 42949672969 2024-09-15T20:41:07.043 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-15T20:41:07.319 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 42949672969 -lt 42949672969 2024-09-15T20:41:07.319 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T20:41:07.319 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-64424509446 2024-09-15T20:41:07.319 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T20:41:07.320 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-15T20:41:07.320 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-64424509446 2024-09-15T20:41:07.321 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T20:41:07.321 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=64424509446 2024-09-15T20:41:07.322 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 64424509446' 2024-09-15T20:41:07.322 INFO:tasks.workunit.client.0.smithi136.stderr:waiting osd.2 seq 64424509446 2024-09-15T20:41:07.322 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-15T20:41:07.597 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 64424509447 -lt 64424509446 2024-09-15T20:41:07.598 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-15T20:41:07.598 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T20:41:07.598 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T20:41:07.945 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 1 == 0 2024-09-15T20:41:07.945 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-15T20:41:07.945 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-15T20:41:07.946 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-15T20:41:07.946 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T20:41:07.946 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T20:41:07.946 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T20:41:07.946 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T20:41:08.216 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=1 2024-09-15T20:41:08.216 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-15T20:41:08.216 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T20:41:08.216 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T20:41:08.557 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 = 1 2024-09-15T20:41:08.557 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-15T20:41:08.557 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-15T20:41:08.557 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:413: TEST_auto_repair_bluestore_tag: get_pg testpool SOMETHING 2024-09-15T20:41:08.557 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1084: get_pg: local poolname=testpool 2024-09-15T20:41:08.557 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1085: get_pg: local objectname=SOMETHING 2024-09-15T20:41:08.557 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1087: get_pg: ceph --format json osd map testpool SOMETHING 2024-09-15T20:41:08.557 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1087: get_pg: jq -r .pgid 2024-09-15T20:41:08.855 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:413: TEST_auto_repair_bluestore_tag: local pgid=1.0 2024-09-15T20:41:08.856 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:414: TEST_auto_repair_bluestore_tag: get_primary testpool SOMETHING 2024-09-15T20:41:08.856 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1187: get_primary: local poolname=testpool 2024-09-15T20:41:08.856 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1188: get_primary: local objectname=SOMETHING 2024-09-15T20:41:08.856 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1190: get_primary: ceph --format json osd map testpool SOMETHING 2024-09-15T20:41:08.856 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1191: get_primary: jq .acting_primary 2024-09-15T20:41:09.134 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:414: TEST_auto_repair_bluestore_tag: local primary=1 2024-09-15T20:41:09.135 INFO:tasks.workunit.client.0.smithi136.stdout:Affected PG 1.0 w/ primary 1 2024-09-15T20:41:09.135 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:415: TEST_auto_repair_bluestore_tag: echo 'Affected PG ' 1.0 ' w/ primary ' 1 2024-09-15T20:41:09.135 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:416: TEST_auto_repair_bluestore_tag: get_last_scrub_stamp 1.0 2024-09-15T20:41:09.136 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-15T20:41:09.136 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-15T20:41:09.136 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-15T20:41:09.136 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-15T20:41:09.408 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:416: TEST_auto_repair_bluestore_tag: local last_scrub_stamp=2024-09-15T20:40:47.030721+0000 2024-09-15T20:41:09.409 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:417: TEST_auto_repair_bluestore_tag: initiate_and_fetch_state 1 1.0 3.0 2024-09-15T20:41:09.409 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:322: initiate_and_fetch_state: local the_osd=osd.1 2024-09-15T20:41:09.409 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:323: initiate_and_fetch_state: local pgid=1.0 2024-09-15T20:41:09.409 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:324: initiate_and_fetch_state: get_last_scrub_stamp 1.0 2024-09-15T20:41:09.409 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-15T20:41:09.409 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-15T20:41:09.409 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-15T20:41:09.409 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-15T20:41:09.679 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:324: initiate_and_fetch_state: local last_scrub=2024-09-15T20:40:47.030721+0000 2024-09-15T20:41:09.679 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:326: initiate_and_fetch_state: set_config osd 1 osd_scrub_sleep 3.0 2024-09-15T20:41:09.679 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1152: set_config: local daemon=osd 2024-09-15T20:41:09.679 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1153: set_config: local id=1 2024-09-15T20:41:09.679 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1154: set_config: local config=osd_scrub_sleep 2024-09-15T20:41:09.679 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1155: set_config: local value=3.0 2024-09-15T20:41:09.680 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: jq 'has("success")' 2024-09-15T20:41:09.680 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: get_asok_path osd.1 2024-09-15T20:41:09.680 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.1 2024-09-15T20:41:09.680 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.1 ']' 2024-09-15T20:41:09.680 INFO:tasks.workunit.client.0.smithi136.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-15T20:41:09.681 INFO:tasks.workunit.client.0.smithi136.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T20:41:09.681 INFO:tasks.workunit.client.0.smithi136.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52602 2024-09-15T20:41:09.681 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.52602/ceph-osd.1.asok 2024-09-15T20:41:09.681 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.52602/ceph-osd.1.asok config set osd_scrub_sleep 3.0 2024-09-15T20:41:09.765 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: test true == true 2024-09-15T20:41:09.765 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:327: initiate_and_fetch_state: set_config osd 1 osd_scrub_auto_repair true 2024-09-15T20:41:09.765 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1152: set_config: local daemon=osd 2024-09-15T20:41:09.766 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1153: set_config: local id=1 2024-09-15T20:41:09.766 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1154: set_config: local config=osd_scrub_auto_repair 2024-09-15T20:41:09.766 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1155: set_config: local value=true 2024-09-15T20:41:09.767 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: jq 'has("success")' 2024-09-15T20:41:09.767 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: get_asok_path osd.1 2024-09-15T20:41:09.767 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.1 2024-09-15T20:41:09.767 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.1 ']' 2024-09-15T20:41:09.767 INFO:tasks.workunit.client.0.smithi136.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-15T20:41:09.767 INFO:tasks.workunit.client.0.smithi136.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T20:41:09.767 INFO:tasks.workunit.client.0.smithi136.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52602 2024-09-15T20:41:09.767 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.52602/ceph-osd.1.asok 2024-09-15T20:41:09.767 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.52602/ceph-osd.1.asok config set osd_scrub_auto_repair true 2024-09-15T20:41:09.853 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: test true == true 2024-09-15T20:41:09.853 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:329: initiate_and_fetch_state: flush_pg_stats 2024-09-15T20:41:09.854 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-15T20:41:09.854 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-15T20:41:10.126 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-15T20:41:10.126 INFO:tasks.workunit.client.0.smithi136.stderr:1 2024-09-15T20:41:10.126 INFO:tasks.workunit.client.0.smithi136.stderr:2' 2024-09-15T20:41:10.126 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-15T20:41:10.127 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T20:41:10.127 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-15T20:41:10.228 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=103079215108 2024-09-15T20:41:10.229 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 103079215108 2024-09-15T20:41:10.229 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-103079215108' 2024-09-15T20:41:10.229 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T20:41:10.229 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-15T20:41:10.332 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=42949672971 2024-09-15T20:41:10.333 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 42949672971 2024-09-15T20:41:10.333 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-103079215108 1-42949672971' 2024-09-15T20:41:10.333 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T20:41:10.333 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-15T20:41:10.435 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=64424509448 2024-09-15T20:41:10.435 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 64424509448 2024-09-15T20:41:10.435 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-103079215108 1-42949672971 2-64424509448' 2024-09-15T20:41:10.435 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T20:41:10.435 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-103079215108 2024-09-15T20:41:10.435 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T20:41:10.436 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-15T20:41:10.436 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-103079215108 2024-09-15T20:41:10.437 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T20:41:10.437 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=103079215108 2024-09-15T20:41:10.437 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 103079215108' 2024-09-15T20:41:10.438 INFO:tasks.workunit.client.0.smithi136.stdout:waiting osd.0 seq 103079215108 2024-09-15T20:41:10.438 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T20:41:10.714 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 103079215107 -lt 103079215108 2024-09-15T20:41:10.714 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-15T20:41:11.716 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-15T20:41:11.716 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T20:41:11.998 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 103079215108 -lt 103079215108 2024-09-15T20:41:11.998 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T20:41:11.999 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-42949672971 2024-09-15T20:41:11.999 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T20:41:12.000 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-15T20:41:12.000 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-42949672971 2024-09-15T20:41:12.000 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T20:41:12.001 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=42949672971 2024-09-15T20:41:12.002 INFO:tasks.workunit.client.0.smithi136.stdout:waiting osd.1 seq 42949672971 2024-09-15T20:41:12.002 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 42949672971' 2024-09-15T20:41:12.002 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-15T20:41:12.281 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 42949672971 -lt 42949672971 2024-09-15T20:41:12.281 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T20:41:12.281 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-64424509448 2024-09-15T20:41:12.281 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T20:41:12.282 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-15T20:41:12.283 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-64424509448 2024-09-15T20:41:12.283 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T20:41:12.284 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=64424509448 2024-09-15T20:41:12.284 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 64424509448' 2024-09-15T20:41:12.284 INFO:tasks.workunit.client.0.smithi136.stdout:waiting osd.2 seq 64424509448 2024-09-15T20:41:12.284 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-15T20:41:12.561 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 64424509448 -lt 64424509448 2024-09-15T20:41:12.561 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:330: initiate_and_fetch_state: date --rfc-3339=ns 2024-09-15T20:41:12.561 INFO:tasks.workunit.client.0.smithi136.stdout:2024-09-15 20:41:12.560733264+00:00 2024-09-15T20:41:12.562 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:333: initiate_and_fetch_state: get_asok_path osd.1 2024-09-15T20:41:12.562 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.1 2024-09-15T20:41:12.562 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.1 ']' 2024-09-15T20:41:12.563 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-15T20:41:12.563 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T20:41:12.563 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52602 2024-09-15T20:41:12.563 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.52602/ceph-osd.1.asok 2024-09-15T20:41:12.563 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:333: initiate_and_fetch_state: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.52602/ceph-osd.1.asok schedule-deep-scrub 1.0 2024-09-15T20:41:12.652 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:336: initiate_and_fetch_state: (( i=0 )) 2024-09-15T20:41:12.652 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:336: initiate_and_fetch_state: (( i < 80 )) 2024-09-15T20:41:12.653 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:338: initiate_and_fetch_state: ceph pg 1.0 query --format json 2024-09-15T20:41:12.653 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:338: initiate_and_fetch_state: jq .state 2024-09-15T20:41:12.760 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:338: initiate_and_fetch_state: st='"active+clean"' 2024-09-15T20:41:12.760 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:339: initiate_and_fetch_state: echo 0 ') state now: ' '"active+clean"' 2024-09-15T20:41:12.760 INFO:tasks.workunit.client.0.smithi136.stdout:{"deep":true,"must":false,"stamp":"2024-09-01T20:39:32.643168+0000"}0 ) state now: "active+clean" 2024-09-15T20:41:12.761 INFO:tasks.workunit.client.0.smithi136.stdout:"active+clean" 2024-09-15T20:41:12.761 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:341: initiate_and_fetch_state: case "$st" in 2024-09-15T20:41:12.761 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:346: initiate_and_fetch_state: echo '"active+clean"' 2024-09-15T20:41:12.761 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:349: initiate_and_fetch_state: '[' 0 == 4 ']' 2024-09-15T20:41:12.761 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:352: initiate_and_fetch_state: sleep 0.3 2024-09-15T20:41:13.061 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:336: initiate_and_fetch_state: (( i++ )) 2024-09-15T20:41:13.083 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:336: initiate_and_fetch_state: (( i < 80 )) 2024-09-15T20:41:13.083 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:338: initiate_and_fetch_state: ceph pg 1.0 query --format json 2024-09-15T20:41:13.083 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:338: initiate_and_fetch_state: jq .state 2024-09-15T20:41:13.168 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:338: initiate_and_fetch_state: st='"active+clean"' 2024-09-15T20:41:13.168 INFO:tasks.workunit.client.0.smithi136.stdout:1 ) state now: "active+clean" 2024-09-15T20:41:13.168 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:339: initiate_and_fetch_state: echo 1 ') state now: ' '"active+clean"' 2024-09-15T20:41:13.169 INFO:tasks.workunit.client.0.smithi136.stdout:"active+clean" 2024-09-15T20:41:13.169 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:341: initiate_and_fetch_state: case "$st" in 2024-09-15T20:41:13.169 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:346: initiate_and_fetch_state: echo '"active+clean"' 2024-09-15T20:41:13.169 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:349: initiate_and_fetch_state: '[' 1 == 4 ']' 2024-09-15T20:41:13.169 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:352: initiate_and_fetch_state: sleep 0.3 2024-09-15T20:41:13.469 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:336: initiate_and_fetch_state: (( i++ )) 2024-09-15T20:41:13.470 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:336: initiate_and_fetch_state: (( i < 80 )) 2024-09-15T20:41:13.470 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:338: initiate_and_fetch_state: ceph pg 1.0 query --format json 2024-09-15T20:41:13.470 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:338: initiate_and_fetch_state: jq .state 2024-09-15T20:41:13.576 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:338: initiate_and_fetch_state: st='"active+clean"' 2024-09-15T20:41:13.577 INFO:tasks.workunit.client.0.smithi136.stdout:2 ) state now: "active+clean" 2024-09-15T20:41:13.577 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:339: initiate_and_fetch_state: echo 2 ') state now: ' '"active+clean"' 2024-09-15T20:41:13.577 INFO:tasks.workunit.client.0.smithi136.stdout:"active+clean" 2024-09-15T20:41:13.578 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:341: initiate_and_fetch_state: case "$st" in 2024-09-15T20:41:13.578 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:346: initiate_and_fetch_state: echo '"active+clean"' 2024-09-15T20:41:13.578 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:349: initiate_and_fetch_state: '[' 2 == 4 ']' 2024-09-15T20:41:13.578 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:352: initiate_and_fetch_state: sleep 0.3 2024-09-15T20:41:13.877 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:336: initiate_and_fetch_state: (( i++ )) 2024-09-15T20:41:13.878 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:336: initiate_and_fetch_state: (( i < 80 )) 2024-09-15T20:41:13.878 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:338: initiate_and_fetch_state: ceph pg 1.0 query --format json 2024-09-15T20:41:13.878 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:338: initiate_and_fetch_state: jq .state 2024-09-15T20:41:13.983 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:338: initiate_and_fetch_state: st='"active+clean"' 2024-09-15T20:41:13.984 INFO:tasks.workunit.client.0.smithi136.stdout:3 ) state now: "active+clean" 2024-09-15T20:41:13.984 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:339: initiate_and_fetch_state: echo 3 ') state now: ' '"active+clean"' 2024-09-15T20:41:13.984 INFO:tasks.workunit.client.0.smithi136.stdout:"active+clean" 2024-09-15T20:41:13.985 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:341: initiate_and_fetch_state: case "$st" in 2024-09-15T20:41:13.985 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:346: initiate_and_fetch_state: echo '"active+clean"' 2024-09-15T20:41:13.985 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:349: initiate_and_fetch_state: '[' 3 == 4 ']' 2024-09-15T20:41:13.985 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:352: initiate_and_fetch_state: sleep 0.3 2024-09-15T20:41:14.285 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:336: initiate_and_fetch_state: (( i++ )) 2024-09-15T20:41:14.285 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:336: initiate_and_fetch_state: (( i < 80 )) 2024-09-15T20:41:14.285 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:338: initiate_and_fetch_state: ceph pg 1.0 query --format json 2024-09-15T20:41:14.285 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:338: initiate_and_fetch_state: jq .state 2024-09-15T20:41:14.391 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:338: initiate_and_fetch_state: st='"active+clean"' 2024-09-15T20:41:14.391 INFO:tasks.workunit.client.0.smithi136.stdout:4 ) state now: "active+clean" 2024-09-15T20:41:14.392 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:339: initiate_and_fetch_state: echo 4 ') state now: ' '"active+clean"' 2024-09-15T20:41:14.392 INFO:tasks.workunit.client.0.smithi136.stdout:"active+clean" 2024-09-15T20:41:14.392 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:341: initiate_and_fetch_state: case "$st" in 2024-09-15T20:41:14.393 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:346: initiate_and_fetch_state: echo '"active+clean"' 2024-09-15T20:41:14.393 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:349: initiate_and_fetch_state: '[' 4 == 4 ']' 2024-09-15T20:41:14.393 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:350: initiate_and_fetch_state: echo 'loop --------> ' 4 2024-09-15T20:41:14.393 INFO:tasks.workunit.client.0.smithi136.stdout:loop --------> 4 2024-09-15T20:41:14.393 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:352: initiate_and_fetch_state: sleep 0.3 2024-09-15T20:41:14.692 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:336: initiate_and_fetch_state: (( i++ )) 2024-09-15T20:41:14.692 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:336: initiate_and_fetch_state: (( i < 80 )) 2024-09-15T20:41:14.693 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:338: initiate_and_fetch_state: ceph pg 1.0 query --format json 2024-09-15T20:41:14.693 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:338: initiate_and_fetch_state: jq .state 2024-09-15T20:41:14.838 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:338: initiate_and_fetch_state: st='"active+clean+scrubbing+deep"' 2024-09-15T20:41:14.838 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:339: initiate_and_fetch_state: echo 5 ') state now: ' '"active+clean+scrubbing+deep"' 2024-09-15T20:41:14.838 INFO:tasks.workunit.client.0.smithi136.stdout:5 ) state now: "active+clean+scrubbing+deep" 2024-09-15T20:41:14.839 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:341: initiate_and_fetch_state: case "$st" in 2024-09-15T20:41:14.839 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:343: initiate_and_fetch_state: echo 'found scrub' 2024-09-15T20:41:14.839 INFO:tasks.workunit.client.0.smithi136.stdout:found scrub 2024-09-15T20:41:14.839 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:343: initiate_and_fetch_state: return 0 2024-09-15T20:41:14.840 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:418: TEST_auto_repair_bluestore_tag: r=0 2024-09-15T20:41:14.840 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:419: TEST_auto_repair_bluestore_tag: echo 'initiate_and_fetch_state ret: ' 0 2024-09-15T20:41:14.840 INFO:tasks.workunit.client.0.smithi136.stdout:initiate_and_fetch_state ret: 0 2024-09-15T20:41:14.841 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:420: TEST_auto_repair_bluestore_tag: set_config osd td/osd-scrub-repair osd_scrub_sleep 0 2024-09-15T20:41:14.842 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1152: set_config: local daemon=osd 2024-09-15T20:41:14.842 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1153: set_config: local id=td/osd-scrub-repair 2024-09-15T20:41:14.842 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1154: set_config: local config=osd_scrub_sleep 2024-09-15T20:41:14.842 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1155: set_config: local value=0 2024-09-15T20:41:14.842 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: jq 'has("success")' 2024-09-15T20:41:14.842 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: get_asok_path osd.td/osd-scrub-repair 2024-09-15T20:41:14.842 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.td/osd-scrub-repair 2024-09-15T20:41:14.842 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.td/osd-scrub-repair ']' 2024-09-15T20:41:14.842 INFO:tasks.workunit.client.0.smithi136.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-15T20:41:14.842 INFO:tasks.workunit.client.0.smithi136.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T20:41:14.842 INFO:tasks.workunit.client.0.smithi136.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52602 2024-09-15T20:41:14.842 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.52602/ceph-osd.td/osd-scrub-repair.asok 2024-09-15T20:41:14.843 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.52602/ceph-osd.td/osd-scrub-repair.asok config set osd_scrub_sleep 0 2024-09-15T20:41:14.905 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: test == true 2024-09-15T20:41:14.905 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh: line 1159: test: ==: unary operator expected 2024-09-15T20:41:14.905 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:421: TEST_auto_repair_bluestore_tag: '[' 0 -ne 0 ']' 2024-09-15T20:41:14.905 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:425: TEST_auto_repair_bluestore_tag: wait_end_of_scrub 1 1.0 2024-09-15T20:41:14.905 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:360: wait_end_of_scrub: local the_osd=osd.1 2024-09-15T20:41:14.906 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:361: wait_end_of_scrub: local pgid=1.0 2024-09-15T20:41:14.906 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:363: wait_end_of_scrub: (( i=0 )) 2024-09-15T20:41:14.906 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:363: wait_end_of_scrub: (( i < 40 )) 2024-09-15T20:41:14.906 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:364: wait_end_of_scrub: ceph pg 1.0 query --format json 2024-09-15T20:41:14.906 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:364: wait_end_of_scrub: jq .state 2024-09-15T20:41:15.010 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:364: wait_end_of_scrub: st='"active+clean+scrubbing+deep"' 2024-09-15T20:41:15.011 INFO:tasks.workunit.client.0.smithi136.stdout:wait-scrub-end state now: "active+clean+scrubbing+deep" 2024-09-15T20:41:15.011 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:365: wait_end_of_scrub: echo 'wait-scrub-end state now: ' '"active+clean+scrubbing+deep"' 2024-09-15T20:41:15.011 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:366: wait_end_of_scrub: [[ "active+clean+scrubbing+deep" =~ (.*scrubbing.*) ]] 2024-09-15T20:41:15.011 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:367: wait_end_of_scrub: '[' 0 == 4 ']' 2024-09-15T20:41:15.011 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:370: wait_end_of_scrub: sleep 0.3 2024-09-15T20:41:15.312 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:363: wait_end_of_scrub: (( i++ )) 2024-09-15T20:41:15.312 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:363: wait_end_of_scrub: (( i < 40 )) 2024-09-15T20:41:15.312 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:364: wait_end_of_scrub: ceph pg 1.0 query --format json 2024-09-15T20:41:15.312 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:364: wait_end_of_scrub: jq .state 2024-09-15T20:41:15.419 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:364: wait_end_of_scrub: st='"active+clean+scrubbing+deep"' 2024-09-15T20:41:15.419 INFO:tasks.workunit.client.0.smithi136.stdout:wait-scrub-end state now: "active+clean+scrubbing+deep" 2024-09-15T20:41:15.420 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:365: wait_end_of_scrub: echo 'wait-scrub-end state now: ' '"active+clean+scrubbing+deep"' 2024-09-15T20:41:15.420 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:366: wait_end_of_scrub: [[ "active+clean+scrubbing+deep" =~ (.*scrubbing.*) ]] 2024-09-15T20:41:15.420 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:367: wait_end_of_scrub: '[' 1 == 4 ']' 2024-09-15T20:41:15.420 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:370: wait_end_of_scrub: sleep 0.3 2024-09-15T20:41:15.720 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:363: wait_end_of_scrub: (( i++ )) 2024-09-15T20:41:15.721 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:363: wait_end_of_scrub: (( i < 40 )) 2024-09-15T20:41:15.721 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:364: wait_end_of_scrub: ceph pg 1.0 query --format json 2024-09-15T20:41:15.721 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:364: wait_end_of_scrub: jq .state 2024-09-15T20:41:15.826 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:364: wait_end_of_scrub: st='"active+clean+scrubbing+deep"' 2024-09-15T20:41:15.827 INFO:tasks.workunit.client.0.smithi136.stdout:wait-scrub-end state now: "active+clean+scrubbing+deep" 2024-09-15T20:41:15.827 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:365: wait_end_of_scrub: echo 'wait-scrub-end state now: ' '"active+clean+scrubbing+deep"' 2024-09-15T20:41:15.827 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:366: wait_end_of_scrub: [[ "active+clean+scrubbing+deep" =~ (.*scrubbing.*) ]] 2024-09-15T20:41:15.828 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:367: wait_end_of_scrub: '[' 2 == 4 ']' 2024-09-15T20:41:15.828 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:370: wait_end_of_scrub: sleep 0.3 2024-09-15T20:41:16.129 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:363: wait_end_of_scrub: (( i++ )) 2024-09-15T20:41:16.129 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:363: wait_end_of_scrub: (( i < 40 )) 2024-09-15T20:41:16.129 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:364: wait_end_of_scrub: ceph pg 1.0 query --format json 2024-09-15T20:41:16.129 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:364: wait_end_of_scrub: jq .state 2024-09-15T20:41:16.237 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:364: wait_end_of_scrub: st='"active+clean+scrubbing+deep"' 2024-09-15T20:41:16.238 INFO:tasks.workunit.client.0.smithi136.stdout:wait-scrub-end state now: "active+clean+scrubbing+deep" 2024-09-15T20:41:16.238 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:365: wait_end_of_scrub: echo 'wait-scrub-end state now: ' '"active+clean+scrubbing+deep"' 2024-09-15T20:41:16.238 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:366: wait_end_of_scrub: [[ "active+clean+scrubbing+deep" =~ (.*scrubbing.*) ]] 2024-09-15T20:41:16.239 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:367: wait_end_of_scrub: '[' 3 == 4 ']' 2024-09-15T20:41:16.239 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:370: wait_end_of_scrub: sleep 0.3 2024-09-15T20:41:16.538 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:363: wait_end_of_scrub: (( i++ )) 2024-09-15T20:41:16.539 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:363: wait_end_of_scrub: (( i < 40 )) 2024-09-15T20:41:16.539 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:364: wait_end_of_scrub: ceph pg 1.0 query --format json 2024-09-15T20:41:16.539 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:364: wait_end_of_scrub: jq .state 2024-09-15T20:41:16.646 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:364: wait_end_of_scrub: st='"active+clean+scrubbing+deep"' 2024-09-15T20:41:16.646 INFO:tasks.workunit.client.0.smithi136.stdout:wait-scrub-end state now: "active+clean+scrubbing+deep" 2024-09-15T20:41:16.647 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:365: wait_end_of_scrub: echo 'wait-scrub-end state now: ' '"active+clean+scrubbing+deep"' 2024-09-15T20:41:16.647 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:366: wait_end_of_scrub: [[ "active+clean+scrubbing+deep" =~ (.*scrubbing.*) ]] 2024-09-15T20:41:16.647 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:367: wait_end_of_scrub: '[' 4 == 4 ']' 2024-09-15T20:41:16.647 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:368: wait_end_of_scrub: flush_pg_stats 2024-09-15T20:41:16.647 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-15T20:41:16.647 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-15T20:41:16.924 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-15T20:41:16.924 INFO:tasks.workunit.client.0.smithi136.stderr:1 2024-09-15T20:41:16.924 INFO:tasks.workunit.client.0.smithi136.stderr:2' 2024-09-15T20:41:16.924 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-15T20:41:16.924 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T20:41:16.924 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-15T20:41:17.028 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=103079215110 2024-09-15T20:41:17.028 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 103079215110 2024-09-15T20:41:17.028 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-103079215110' 2024-09-15T20:41:17.028 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T20:41:17.028 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-15T20:41:17.131 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=42949672973 2024-09-15T20:41:17.131 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 42949672973 2024-09-15T20:41:17.131 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-103079215110 1-42949672973' 2024-09-15T20:41:17.131 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T20:41:17.131 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-15T20:41:17.234 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=64424509451 2024-09-15T20:41:17.234 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 64424509451 2024-09-15T20:41:17.234 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-103079215110 1-42949672973 2-64424509451' 2024-09-15T20:41:17.234 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T20:41:17.235 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-103079215110 2024-09-15T20:41:17.235 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T20:41:17.235 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-15T20:41:17.236 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-103079215110 2024-09-15T20:41:17.236 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T20:41:17.237 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=103079215110 2024-09-15T20:41:17.237 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 103079215110' 2024-09-15T20:41:17.238 INFO:tasks.workunit.client.0.smithi136.stdout:waiting osd.0 seq 103079215110 2024-09-15T20:41:17.238 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T20:41:17.516 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 103079215109 -lt 103079215110 2024-09-15T20:41:17.516 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-15T20:41:18.517 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-15T20:41:18.517 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T20:41:18.807 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 103079215111 -lt 103079215110 2024-09-15T20:41:18.807 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T20:41:18.807 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-42949672973 2024-09-15T20:41:18.807 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T20:41:18.808 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-15T20:41:18.809 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-42949672973 2024-09-15T20:41:18.809 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T20:41:18.809 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=42949672973 2024-09-15T20:41:18.810 INFO:tasks.workunit.client.0.smithi136.stdout:waiting osd.1 seq 42949672973 2024-09-15T20:41:18.810 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 42949672973' 2024-09-15T20:41:18.810 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-15T20:41:19.087 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 42949672973 -lt 42949672973 2024-09-15T20:41:19.087 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T20:41:19.087 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-64424509451 2024-09-15T20:41:19.088 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T20:41:19.088 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-15T20:41:19.089 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-64424509451 2024-09-15T20:41:19.089 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T20:41:19.090 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=64424509451 2024-09-15T20:41:19.090 INFO:tasks.workunit.client.0.smithi136.stdout:waiting osd.2 seq 64424509451 2024-09-15T20:41:19.090 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 64424509451' 2024-09-15T20:41:19.092 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-15T20:41:19.367 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 64424509451 -lt 64424509451 2024-09-15T20:41:19.367 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:370: wait_end_of_scrub: sleep 0.3 2024-09-15T20:41:19.668 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:363: wait_end_of_scrub: (( i++ )) 2024-09-15T20:41:19.668 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:363: wait_end_of_scrub: (( i < 40 )) 2024-09-15T20:41:19.669 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:364: wait_end_of_scrub: ceph pg 1.0 query --format json 2024-09-15T20:41:19.669 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:364: wait_end_of_scrub: jq .state 2024-09-15T20:41:19.773 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:364: wait_end_of_scrub: st='"active+clean"' 2024-09-15T20:41:19.774 INFO:tasks.workunit.client.0.smithi136.stdout:wait-scrub-end state now: "active+clean" 2024-09-15T20:41:19.775 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:365: wait_end_of_scrub: echo 'wait-scrub-end state now: ' '"active+clean"' 2024-09-15T20:41:19.775 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:366: wait_end_of_scrub: [[ "active+clean" =~ (.*scrubbing.*) ]] 2024-09-15T20:41:19.775 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:366: wait_end_of_scrub: break 2024-09-15T20:41:19.775 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:373: wait_end_of_scrub: [[ "active+clean" =~ (.*scrubbing.*) ]] 2024-09-15T20:41:19.775 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:378: wait_end_of_scrub: return 0 2024-09-15T20:41:19.775 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:426: TEST_auto_repair_bluestore_tag: ceph pg dump pgs 2024-09-15T20:41:20.033 INFO:tasks.workunit.client.0.smithi136.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-15T20:41:20.033 INFO:tasks.workunit.client.0.smithi136.stdout:1.0 1 0 0 0 0 7 0 0 1 0 1 active+clean+scrubbing+deep 2024-09-15T20:41:14.395308+0000 20'1 25:47 [1,0] 1 [1,0] 1 0'0 2024-09-01T20:39:32.643168+0000 0'0 2024-09-01T20:39:32.643168+0000 0 0 deep scrubbing for 2s 0 0 2024-09-15T20:41:20.033 INFO:tasks.workunit.client.0.smithi136.stdout: 2024-09-15T20:41:20.033 INFO:tasks.workunit.client.0.smithi136.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-15T20:41:20.034 INFO:tasks.workunit.client.0.smithi136.stderr:dumped pgs 2024-09-15T20:41:20.045 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:430: TEST_auto_repair_bluestore_tag: get_not_primary testpool SOMETHING 2024-09-15T20:41:20.045 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1220: get_not_primary: local poolname=testpool 2024-09-15T20:41:20.045 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1221: get_not_primary: local objectname=SOMETHING 2024-09-15T20:41:20.045 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1223: get_not_primary: get_primary testpool SOMETHING 2024-09-15T20:41:20.046 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1187: get_primary: local poolname=testpool 2024-09-15T20:41:20.046 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1188: get_primary: local objectname=SOMETHING 2024-09-15T20:41:20.046 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1190: get_primary: ceph --format json osd map testpool SOMETHING 2024-09-15T20:41:20.046 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1191: get_primary: jq .acting_primary 2024-09-15T20:41:20.325 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1223: get_not_primary: local primary=1 2024-09-15T20:41:20.325 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1224: get_not_primary: ceph --format json osd map testpool SOMETHING 2024-09-15T20:41:20.325 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1225: get_not_primary: jq '.acting | map(select (. != 1)) | .[0]' 2024-09-15T20:41:20.622 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:430: TEST_auto_repair_bluestore_tag: objectstore_tool td/osd-scrub-repair 0 SOMETHING list-attrs 2024-09-15T20:41:20.622 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2024-09-15T20:41:20.622 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2024-09-15T20:41:20.622 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=0 2024-09-15T20:41:20.622 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2024-09-15T20:41:20.622 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 SOMETHING list-attrs 2024-09-15T20:41:20.622 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-09-15T20:41:20.622 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2024-09-15T20:41:20.622 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=0 2024-09-15T20:41:20.623 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2024-09-15T20:41:20.623 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2024-09-15T20:41:20.623 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-15T20:41:20.623 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-15T20:41:20.623 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-15T20:41:20.623 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-15T20:41:20.623 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-15T20:41:20.928 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-15T20:41:20.928 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 SOMETHING list-attrs 2024-09-15T20:41:20.928 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-15T20:41:20.928 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-15T20:41:20.928 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=0 2024-09-15T20:41:20.929 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-15T20:41:20.929 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-09-15T20:41:20.929 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 SOMETHING list-attrs 2024-09-15T20:41:21.365 INFO:tasks.workunit.client.0.smithi136.stdout:_ 2024-09-15T20:41:21.365 INFO:tasks.workunit.client.0.smithi136.stdout:snapset 2024-09-15T20:41:21.633 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-op-queue=wpq 2024-09-15T20:41:21.633 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2024-09-15T20:41:21.633 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-15T20:41:21.633 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=0 2024-09-15T20:41:21.633 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-15T20:41:21.633 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-09-15T20:41:21.633 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=ce4892c7-6c6d-4b4f-8e6f-07a3e11133e5 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-15T20:41:21.633 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T20:41:21.634 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T20:41:21.634 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T20:41:21.634 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-09-15T20:41:21.634 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-09-15T20:41:21.634 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-15T20:41:21.634 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-15T20:41:21.634 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-15T20:41:21.634 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-15T20:41:21.634 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-15T20:41:21.634 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-15T20:41:21.634 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-15T20:41:21.634 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T20:41:21.634 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52602 2024-09-15T20:41:21.635 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.52602/$cluster-$name.asok' 2024-09-15T20:41:21.635 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.52602/$cluster-$name.asok' 2024-09-15T20:41:21.635 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-15T20:41:21.635 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-15T20:41:21.635 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-15T20:41:21.635 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T20:41:21.635 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T20:41:21.635 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T20:41:21.635 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T20:41:21.635 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-15T20:41:21.635 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+='--osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-op-queue=wpq' 2024-09-15T20:41:21.635 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-09-15T20:41:21.635 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.0 2024-09-15T20:41:21.636 INFO:tasks.workunit.client.0.smithi136.stderr:start osd.0 2024-09-15T20:41:21.636 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 0 --fsid=ce4892c7-6c6d-4b4f-8e6f-07a3e11133e5 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.52602/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-op-queue=wpq 2024-09-15T20:41:21.636 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-09-15T20:41:21.636 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 0 = 0 ']' 2024-09-15T20:41:21.637 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-15T20:41:21.637 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-15T20:41:21.637 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-15T20:41:21.664 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:41:21.662+0000 7f5fbce0e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T20:41:21.667 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:41:21.665+0000 7f5fbce0e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T20:41:21.668 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:41:21.666+0000 7f5fbce0e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T20:41:21.916 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 0 2024-09-15T20:41:21.916 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-15T20:41:21.916 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2024-09-15T20:41:21.916 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-15T20:41:21.916 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-15T20:41:21.916 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:41:21.916 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-15T20:41:21.916 INFO:tasks.workunit.client.0.smithi136.stderr:0 2024-09-15T20:41:21.916 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:41:21.917 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T20:41:22.194 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:41:23.032 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:41:23.030+0000 7f5fbce0e640 -1 Falling back to public interface 2024-09-15T20:41:23.195 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:41:23.196 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:41:23.196 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-15T20:41:23.196 INFO:tasks.workunit.client.0.smithi136.stderr:1 2024-09-15T20:41:23.196 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:41:23.196 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T20:41:23.472 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:41:24.474 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:41:24.475 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:41:24.475 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-15T20:41:24.475 INFO:tasks.workunit.client.0.smithi136.stderr:2 2024-09-15T20:41:24.475 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:41:24.475 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T20:41:24.781 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:41:25.783 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:41:25.783 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:41:25.783 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-15T20:41:25.783 INFO:tasks.workunit.client.0.smithi136.stderr:3 2024-09-15T20:41:25.783 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:41:25.784 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T20:41:25.926 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:41:25.923+0000 7f5fbce0e640 -1 osd.0 25 log_to_monitors true 2024-09-15T20:41:26.120 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:41:27.122 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:41:27.122 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:41:27.122 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-15T20:41:27.122 INFO:tasks.workunit.client.0.smithi136.stderr:4 2024-09-15T20:41:27.122 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:41:27.122 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T20:41:27.402 INFO:tasks.workunit.client.0.smithi136.stderr:osd.0 up in weight 1 up_from 29 up_thru 0 down_at 26 last_clean_interval [24,25) [v2:127.0.0.1:6802/143362566,v1:127.0.0.1:6803/143362566] [v2:127.0.0.1:6804/143362566,v1:127.0.0.1:6805/143362566] exists,up 0361f5e2-4cb7-4ba9-9b83-24c8876dabdb 2024-09-15T20:41:27.402 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-15T20:41:27.402 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-15T20:41:27.402 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-15T20:41:27.402 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2024-09-15T20:41:27.402 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-15T20:41:27.402 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-15T20:41:27.402 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-15T20:41:27.402 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-15T20:41:27.403 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-15T20:41:27.403 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-15T20:41:27.403 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-15T20:41:27.403 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-15T20:41:27.403 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-15T20:41:27.492 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-15T20:41:27.492 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-15T20:41:27.492 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-15T20:41:27.492 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-15T20:41:27.493 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-15T20:41:27.493 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-15T20:41:27.767 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-15T20:41:27.767 INFO:tasks.workunit.client.0.smithi136.stderr:1 2024-09-15T20:41:27.767 INFO:tasks.workunit.client.0.smithi136.stderr:2' 2024-09-15T20:41:27.767 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-15T20:41:27.767 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T20:41:27.768 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-15T20:41:27.871 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=124554051586 2024-09-15T20:41:27.871 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 124554051586 2024-09-15T20:41:27.871 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-124554051586' 2024-09-15T20:41:27.871 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T20:41:27.871 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-15T20:41:27.975 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=42949672976 2024-09-15T20:41:27.975 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 42949672976 2024-09-15T20:41:27.975 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-124554051586 1-42949672976' 2024-09-15T20:41:27.975 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T20:41:27.975 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-15T20:41:28.077 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=64424509454 2024-09-15T20:41:28.077 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 64424509454 2024-09-15T20:41:28.078 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-124554051586 1-42949672976 2-64424509454' 2024-09-15T20:41:28.078 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T20:41:28.078 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-124554051586 2024-09-15T20:41:28.078 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T20:41:28.078 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-15T20:41:28.079 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-124554051586 2024-09-15T20:41:28.079 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T20:41:28.080 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=124554051586 2024-09-15T20:41:28.080 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 124554051586' 2024-09-15T20:41:28.081 INFO:tasks.workunit.client.0.smithi136.stderr:waiting osd.0 seq 124554051586 2024-09-15T20:41:28.081 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T20:41:28.358 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 1 -lt 124554051586 2024-09-15T20:41:28.359 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-15T20:41:29.360 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-15T20:41:29.360 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T20:41:29.645 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 124554051586 -lt 124554051586 2024-09-15T20:41:29.645 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T20:41:29.646 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-42949672976 2024-09-15T20:41:29.646 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T20:41:29.646 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-15T20:41:29.647 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-42949672976 2024-09-15T20:41:29.647 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T20:41:29.648 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=42949672976 2024-09-15T20:41:29.648 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 42949672976' 2024-09-15T20:41:29.649 INFO:tasks.workunit.client.0.smithi136.stderr:waiting osd.1 seq 42949672976 2024-09-15T20:41:29.649 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-15T20:41:29.933 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 42949672976 -lt 42949672976 2024-09-15T20:41:29.933 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T20:41:29.934 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-64424509454 2024-09-15T20:41:29.934 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T20:41:29.935 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-15T20:41:29.935 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-64424509454 2024-09-15T20:41:29.936 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T20:41:29.936 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=64424509454 2024-09-15T20:41:29.937 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 64424509454' 2024-09-15T20:41:29.937 INFO:tasks.workunit.client.0.smithi136.stderr:waiting osd.2 seq 64424509454 2024-09-15T20:41:29.937 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-15T20:41:30.213 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 64424509454 -lt 64424509454 2024-09-15T20:41:30.214 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-15T20:41:30.214 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T20:41:30.214 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T20:41:30.558 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 1 == 0 2024-09-15T20:41:30.558 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-15T20:41:30.558 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-15T20:41:30.558 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-15T20:41:30.558 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T20:41:30.558 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T20:41:30.558 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T20:41:30.558 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T20:41:30.832 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=1 2024-09-15T20:41:30.832 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-15T20:41:30.832 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T20:41:30.833 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T20:41:31.178 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 = 1 2024-09-15T20:41:31.178 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-15T20:41:31.178 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-15T20:41:31.178 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:431: TEST_auto_repair_bluestore_tag: get_not_primary testpool SOMETHING 2024-09-15T20:41:31.178 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1220: get_not_primary: local poolname=testpool 2024-09-15T20:41:31.179 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1221: get_not_primary: local objectname=SOMETHING 2024-09-15T20:41:31.179 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1223: get_not_primary: get_primary testpool SOMETHING 2024-09-15T20:41:31.179 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1187: get_primary: local poolname=testpool 2024-09-15T20:41:31.179 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1188: get_primary: local objectname=SOMETHING 2024-09-15T20:41:31.179 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1190: get_primary: ceph --format json osd map testpool SOMETHING 2024-09-15T20:41:31.179 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1191: get_primary: jq .acting_primary 2024-09-15T20:41:31.452 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1223: get_not_primary: local primary=1 2024-09-15T20:41:31.452 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1224: get_not_primary: ceph --format json osd map testpool SOMETHING 2024-09-15T20:41:31.452 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1225: get_not_primary: jq '.acting | map(select (. != 1)) | .[0]' 2024-09-15T20:41:31.730 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:431: TEST_auto_repair_bluestore_tag: objectstore_tool td/osd-scrub-repair 0 SOMETHING get-bytes td/osd-scrub-repair/COPY 2024-09-15T20:41:31.730 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2024-09-15T20:41:31.730 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2024-09-15T20:41:31.730 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=0 2024-09-15T20:41:31.730 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2024-09-15T20:41:31.730 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 SOMETHING get-bytes td/osd-scrub-repair/COPY 2024-09-15T20:41:31.730 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-09-15T20:41:31.730 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2024-09-15T20:41:31.730 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=0 2024-09-15T20:41:31.730 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2024-09-15T20:41:31.731 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2024-09-15T20:41:31.731 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-15T20:41:31.731 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-15T20:41:31.731 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-15T20:41:31.731 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-15T20:41:31.731 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-15T20:41:32.037 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-15T20:41:32.037 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 SOMETHING get-bytes td/osd-scrub-repair/COPY 2024-09-15T20:41:32.037 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-15T20:41:32.037 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-15T20:41:32.038 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=0 2024-09-15T20:41:32.038 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-15T20:41:32.038 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-09-15T20:41:32.038 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 SOMETHING get-bytes td/osd-scrub-repair/COPY 2024-09-15T20:41:32.713 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-op-queue=wpq 2024-09-15T20:41:32.714 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2024-09-15T20:41:32.714 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-15T20:41:32.714 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=0 2024-09-15T20:41:32.714 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-15T20:41:32.714 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-09-15T20:41:32.714 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=ce4892c7-6c6d-4b4f-8e6f-07a3e11133e5 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-15T20:41:32.715 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T20:41:32.715 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T20:41:32.715 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T20:41:32.715 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-09-15T20:41:32.715 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-09-15T20:41:32.715 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-15T20:41:32.715 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-15T20:41:32.716 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-15T20:41:32.716 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-15T20:41:32.716 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-15T20:41:32.716 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-15T20:41:32.716 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-15T20:41:32.716 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T20:41:32.717 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52602 2024-09-15T20:41:32.717 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.52602/$cluster-$name.asok' 2024-09-15T20:41:32.717 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.52602/$cluster-$name.asok' 2024-09-15T20:41:32.717 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-15T20:41:32.717 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-15T20:41:32.717 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-15T20:41:32.717 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T20:41:32.718 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T20:41:32.718 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T20:41:32.718 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T20:41:32.718 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-15T20:41:32.718 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+='--osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-op-queue=wpq' 2024-09-15T20:41:32.718 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-09-15T20:41:32.719 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.0 2024-09-15T20:41:32.719 INFO:tasks.workunit.client.0.smithi136.stderr:start osd.0 2024-09-15T20:41:32.719 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 0 --fsid=ce4892c7-6c6d-4b4f-8e6f-07a3e11133e5 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.52602/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-op-queue=wpq 2024-09-15T20:41:32.719 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-09-15T20:41:32.719 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 0 = 0 ']' 2024-09-15T20:41:32.719 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-15T20:41:32.719 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-15T20:41:32.720 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-15T20:41:32.741 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:41:32.739+0000 7f15aee16640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T20:41:32.743 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:41:32.741+0000 7f15aee16640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T20:41:32.744 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:41:32.742+0000 7f15aee16640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T20:41:33.004 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 0 2024-09-15T20:41:33.005 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-15T20:41:33.005 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2024-09-15T20:41:33.005 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-15T20:41:33.005 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-15T20:41:33.005 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:41:33.005 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-15T20:41:33.005 INFO:tasks.workunit.client.0.smithi136.stderr:0 2024-09-15T20:41:33.005 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:41:33.005 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T20:41:33.277 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:41:34.091 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:41:34.089+0000 7f15aee16640 -1 Falling back to public interface 2024-09-15T20:41:34.279 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:41:34.279 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:41:34.280 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-15T20:41:34.280 INFO:tasks.workunit.client.0.smithi136.stderr:1 2024-09-15T20:41:34.280 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:41:34.280 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T20:41:34.562 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:41:35.564 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:41:35.564 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:41:35.564 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-15T20:41:35.564 INFO:tasks.workunit.client.0.smithi136.stderr:2 2024-09-15T20:41:35.564 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:41:35.564 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T20:41:35.842 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:41:36.844 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:41:36.844 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:41:36.844 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-15T20:41:36.844 INFO:tasks.workunit.client.0.smithi136.stderr:3 2024-09-15T20:41:36.844 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:41:36.845 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T20:41:37.137 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:41:37.415 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:41:37.413+0000 7f15aee16640 -1 osd.0 30 log_to_monitors true 2024-09-15T20:41:38.138 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:41:38.138 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:41:38.138 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-15T20:41:38.139 INFO:tasks.workunit.client.0.smithi136.stderr:4 2024-09-15T20:41:38.139 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:41:38.139 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T20:41:38.550 INFO:tasks.workunit.client.0.smithi136.stderr:osd.0 up in weight 1 up_from 34 up_thru 0 down_at 31 last_clean_interval [29,30) [v2:127.0.0.1:6802/3064031352,v1:127.0.0.1:6803/3064031352] [v2:127.0.0.1:6804/3064031352,v1:127.0.0.1:6805/3064031352] exists,up 0361f5e2-4cb7-4ba9-9b83-24c8876dabdb 2024-09-15T20:41:38.550 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-15T20:41:38.550 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-15T20:41:38.550 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-15T20:41:38.550 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2024-09-15T20:41:38.550 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-15T20:41:38.551 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-15T20:41:38.551 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-15T20:41:38.551 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-15T20:41:38.551 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-15T20:41:38.551 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-15T20:41:38.551 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-15T20:41:38.551 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-15T20:41:38.551 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-15T20:41:38.637 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-15T20:41:38.637 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-15T20:41:38.637 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-15T20:41:38.637 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-15T20:41:38.637 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-15T20:41:38.637 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-15T20:41:38.949 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-15T20:41:38.949 INFO:tasks.workunit.client.0.smithi136.stderr:1 2024-09-15T20:41:38.950 INFO:tasks.workunit.client.0.smithi136.stderr:2' 2024-09-15T20:41:38.950 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-15T20:41:38.950 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T20:41:38.950 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-15T20:41:39.052 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=146028888066 2024-09-15T20:41:39.052 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 146028888066 2024-09-15T20:41:39.052 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-146028888066' 2024-09-15T20:41:39.052 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T20:41:39.052 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-15T20:41:39.155 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=42949672980 2024-09-15T20:41:39.156 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 42949672980 2024-09-15T20:41:39.156 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-146028888066 1-42949672980' 2024-09-15T20:41:39.156 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T20:41:39.156 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-15T20:41:39.260 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=64424509457 2024-09-15T20:41:39.261 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 64424509457 2024-09-15T20:41:39.261 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-146028888066 1-42949672980 2-64424509457' 2024-09-15T20:41:39.261 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T20:41:39.261 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-146028888066 2024-09-15T20:41:39.261 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T20:41:39.263 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-15T20:41:39.263 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-146028888066 2024-09-15T20:41:39.263 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T20:41:39.263 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=146028888066 2024-09-15T20:41:39.264 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 146028888066' 2024-09-15T20:41:39.264 INFO:tasks.workunit.client.0.smithi136.stderr:waiting osd.0 seq 146028888066 2024-09-15T20:41:39.264 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T20:41:39.541 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 1 -lt 146028888066 2024-09-15T20:41:39.541 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-15T20:41:40.542 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-15T20:41:40.542 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T20:41:40.840 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 146028888066 -lt 146028888066 2024-09-15T20:41:40.840 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T20:41:40.840 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-42949672980 2024-09-15T20:41:40.840 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T20:41:40.841 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-15T20:41:40.841 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-42949672980 2024-09-15T20:41:40.841 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T20:41:40.843 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=42949672980 2024-09-15T20:41:40.843 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 42949672980' 2024-09-15T20:41:40.843 INFO:tasks.workunit.client.0.smithi136.stderr:waiting osd.1 seq 42949672980 2024-09-15T20:41:40.843 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-15T20:41:41.119 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 42949672980 -lt 42949672980 2024-09-15T20:41:41.120 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T20:41:41.120 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-64424509457 2024-09-15T20:41:41.120 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T20:41:41.121 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-15T20:41:41.121 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-64424509457 2024-09-15T20:41:41.121 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T20:41:41.122 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=64424509457 2024-09-15T20:41:41.122 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 64424509457' 2024-09-15T20:41:41.123 INFO:tasks.workunit.client.0.smithi136.stderr:waiting osd.2 seq 64424509457 2024-09-15T20:41:41.123 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-15T20:41:41.398 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 64424509457 -lt 64424509457 2024-09-15T20:41:41.399 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-15T20:41:41.399 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T20:41:41.399 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T20:41:41.744 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 1 == 0 2024-09-15T20:41:41.744 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-15T20:41:41.744 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-15T20:41:41.744 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-15T20:41:41.744 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T20:41:41.744 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T20:41:41.744 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T20:41:41.744 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T20:41:42.016 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=1 2024-09-15T20:41:42.016 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-15T20:41:42.016 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T20:41:42.016 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T20:41:42.364 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 = 1 2024-09-15T20:41:42.364 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-15T20:41:42.364 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-15T20:41:42.364 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:432: TEST_auto_repair_bluestore_tag: diff td/osd-scrub-repair/ORIGINAL td/osd-scrub-repair/COPY 2024-09-15T20:41:42.364 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:433: TEST_auto_repair_bluestore_tag: grep scrub_finish td/osd-scrub-repair/osd.1.log 2024-09-15T20:41:42.366 INFO:tasks.workunit.client.0.smithi136.stdout:2024-09-15T20:41:17.399+0000 7fa1f57c1640 10 osd.1 pg_epoch: 25 pg[1.0( v 20'1 (0'0,20'1] local-lis/les=24/25 n=1 ec=17/17 lis/c=24/24 les/c/f=25/25/0 sis=24) [1,0] r=0 lpr=24 crt=20'1 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep [ 1.0: AUTO_REPAIR ] ] scrubber: scrub_finish before flags: AUTO_REPAIR. repair state: no-repair. deep_scrub_on_error: 0 2024-09-15T20:41:42.366 INFO:tasks.workunit.client.0.smithi136.stdout:2024-09-15T20:41:17.399+0000 7fa1f57c1640 10 osd.1 pg_epoch: 25 pg[1.0( v 20'1 (0'0,20'1] local-lis/les=24/25 n=1 ec=17/17 lis/c=24/24 les/c/f=25/25/0 sis=24) [1,0] r=0 lpr=24 crt=20'1 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep [ 1.0: AUTO_REPAIR ] ] scrubber: _scrub_finish info stats: valid m_is_repair: 1 2024-09-15T20:41:42.366 INFO:tasks.workunit.client.0.smithi136.stdout:2024-09-15T20:41:17.399+0000 7fa1f57c1640 15 osd.1 pg_epoch: 25 pg[1.0( v 20'1 (0'0,20'1] local-lis/les=24/25 n=1 ec=17/17 lis/c=24/24 les/c/f=25/25/0 sis=24) [1,0] r=0 lpr=24 crt=20'1 lcod 0'0 mlcod 0'0 active+scrubbing+deep+repair [ 1.0: AUTO_REPAIR ] mbc={255={(1+0)=1}}] scrubber: scrub_finish: 1 errors. 1 errors fixed 2024-09-15T20:41:42.366 INFO:tasks.workunit.client.0.smithi136.stdout:2024-09-15T20:41:17.399+0000 7fa1f57c1640 20 osd.1 pg_epoch: 25 pg[1.0( v 20'1 (0'0,20'1] local-lis/les=24/25 n=1 ec=17/17 lis/c=24/24 les/c/f=25/25/0 sis=24) [1,0] r=0 lpr=24 crt=20'1 lcod 0'0 mlcod 0'0 active+scrubbing+deep+repair [ 1.0: AUTO_REPAIR ] mbc={255={(1+0)=1}}] scrubber: scrub_finish All may be fixed 2024-09-15T20:41:42.366 INFO:tasks.workunit.client.0.smithi136.stdout:2024-09-15T20:41:17.399+0000 7fa1f57c1640 19 osd.1 pg_epoch: 25 pg[1.0( v 20'1 (0'0,20'1] local-lis/les=24/25 n=1 ec=17/17 lis/c=24/24 les/c/f=25/25/0 sis=24) [1,0] r=0 lpr=24 crt=20'1 lcod 0'0 mlcod 0'0 active+scrubbing+deep+repair [ 1.0: AUTO_REPAIR ] mbc={255={(1+0)=1}}] scrubber: scrub_finish shard 1 num_omap_bytes = 0 num_omap_keys = 0 2024-09-15T20:41:42.367 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:66: run: teardown td/osd-scrub-repair 2024-09-15T20:41:42.367 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/osd-scrub-repair 2024-09-15T20:41:42.367 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-15T20:41:42.368 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/osd-scrub-repair KILL 2024-09-15T20:41:42.368 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-15T20:41:42.368 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-15T20:41:42.368 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-15T20:41:42.368 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-15T20:41:42.368 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-15T20:41:42.475 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-15T20:41:42.475 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-15T20:41:42.476 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-15T20:41:42.476 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-15T20:41:42.477 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-15T20:41:42.477 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-15T20:41:42.477 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-15T20:41:42.478 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T20:41:42.478 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-15T20:41:42.479 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-15T20:41:42.479 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T20:41:42.479 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-15T20:41:42.480 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-15T20:41:42.481 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/osd-scrub-repair 2024-09-15T20:41:42.489 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-15T20:41:42.490 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T20:41:42.490 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52602 2024-09-15T20:41:42.490 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.52602 2024-09-15T20:41:42.491 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-15T20:41:42.491 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-15T20:41:42.491 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:63: run: for func in $funcs 2024-09-15T20:41:42.491 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:64: run: setup td/osd-scrub-repair 2024-09-15T20:41:42.491 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/osd-scrub-repair 2024-09-15T20:41:42.492 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/osd-scrub-repair 2024-09-15T20:41:42.492 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/osd-scrub-repair 2024-09-15T20:41:42.492 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-15T20:41:42.492 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/osd-scrub-repair KILL 2024-09-15T20:41:42.492 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-15T20:41:42.492 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-15T20:41:42.492 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-15T20:41:42.492 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-15T20:41:42.492 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-15T20:41:42.493 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-15T20:41:42.493 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-15T20:41:42.494 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-15T20:41:42.494 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-15T20:41:42.495 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-15T20:41:42.496 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-15T20:41:42.496 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-15T20:41:42.496 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T20:41:42.496 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-15T20:41:42.497 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-15T20:41:42.497 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T20:41:42.497 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-15T20:41:42.499 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-15T20:41:42.499 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/osd-scrub-repair 2024-09-15T20:41:42.500 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-15T20:41:42.500 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T20:41:42.500 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52602 2024-09-15T20:41:42.500 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.52602 2024-09-15T20:41:42.501 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-15T20:41:42.501 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-15T20:41:42.501 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/osd-scrub-repair 2024-09-15T20:41:42.502 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2024-09-15T20:41:42.502 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T20:41:42.502 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52602 2024-09-15T20:41:42.502 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.52602 2024-09-15T20:41:42.505 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2024-09-15T20:41:42.505 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 4096 -le 1024 ']' 2024-09-15T20:41:42.505 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2024-09-15T20:41:42.505 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/osd-scrub-repair 1' TERM HUP INT 2024-09-15T20:41:42.505 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:65: run: TEST_auto_repair_erasure_coded_appends td/osd-scrub-repair 2024-09-15T20:41:42.505 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:309: TEST_auto_repair_erasure_coded_appends: auto_repair_erasure_coded td/osd-scrub-repair false 2024-09-15T20:41:42.505 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:268: auto_repair_erasure_coded: local dir=td/osd-scrub-repair 2024-09-15T20:41:42.505 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:269: auto_repair_erasure_coded: local allow_overwrites=false 2024-09-15T20:41:42.506 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:270: auto_repair_erasure_coded: local poolname=ecpool 2024-09-15T20:41:42.506 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:273: auto_repair_erasure_coded: run_mon td/osd-scrub-repair a 2024-09-15T20:41:42.506 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/osd-scrub-repair 2024-09-15T20:41:42.506 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2024-09-15T20:41:42.506 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2024-09-15T20:41:42.506 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2024-09-15T20:41:42.506 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/osd-scrub-repair/a 2024-09-15T20:41:42.506 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-scrub-repair/a --run-dir=td/osd-scrub-repair 2024-09-15T20:41:42.962 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2024-09-15T20:41:42.962 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-15T20:41:42.962 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-15T20:41:42.962 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-15T20:41:42.962 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T20:41:42.962 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52602 2024-09-15T20:41:42.962 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.52602/$cluster-$name.asok' 2024-09-15T20:41:42.962 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-scrub-repair/a '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.52602/$cluster-$name.asok' --mon-cluster-log-file=td/osd-scrub-repair/log --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-09-15T20:41:43.128 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2024-09-15T20:41:43.129 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2024-09-15T20:41:43.129 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-15T20:41:43.130 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-15T20:41:43.130 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2024-09-15T20:41:43.130 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2024-09-15T20:41:43.130 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-15T20:41:43.130 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-15T20:41:43.130 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-15T20:41:43.131 INFO:tasks.workunit.client.0.smithi136.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-15T20:41:43.131 INFO:tasks.workunit.client.0.smithi136.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T20:41:43.131 INFO:tasks.workunit.client.0.smithi136.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52602 2024-09-15T20:41:43.131 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.52602/ceph-mon.a.asok 2024-09-15T20:41:43.131 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-15T20:41:43.131 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.52602/ceph-mon.a.asok config get fsid 2024-09-15T20:41:43.209 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2024-09-15T20:41:43.209 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-15T20:41:43.210 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-15T20:41:43.210 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2024-09-15T20:41:43.210 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2024-09-15T20:41:43.210 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-15T20:41:43.210 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-15T20:41:43.210 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-15T20:41:43.210 INFO:tasks.workunit.client.0.smithi136.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-15T20:41:43.210 INFO:tasks.workunit.client.0.smithi136.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T20:41:43.210 INFO:tasks.workunit.client.0.smithi136.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52602 2024-09-15T20:41:43.210 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.52602/ceph-mon.a.asok 2024-09-15T20:41:43.211 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-15T20:41:43.211 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.52602/ceph-mon.a.asok config get mon_host 2024-09-15T20:41:43.289 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:274: auto_repair_erasure_coded: run_mgr td/osd-scrub-repair x 2024-09-15T20:41:43.289 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: run_mgr: local dir=td/osd-scrub-repair 2024-09-15T20:41:43.289 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: run_mgr: shift 2024-09-15T20:41:43.289 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: run_mgr: local id=x 2024-09-15T20:41:43.289 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: run_mgr: shift 2024-09-15T20:41:43.289 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:548: run_mgr: local data=td/osd-scrub-repair/x 2024-09-15T20:41:43.289 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:550: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-15T20:41:44.099 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: get_asok_path 2024-09-15T20:41:44.099 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-15T20:41:44.099 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-15T20:41:44.099 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-15T20:41:44.099 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T20:41:44.099 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52602 2024-09-15T20:41:44.099 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.52602/$cluster-$name.asok' 2024-09-15T20:41:44.100 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-15T20:41:44.100 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-scrub-repair/x '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.52602/$cluster-$name.asok' --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-15T20:41:44.129 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:279: auto_repair_erasure_coded: local 'ceph_osd_args=--osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0' 2024-09-15T20:41:44.130 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:280: auto_repair_erasure_coded: seq 0 2 2024-09-15T20:41:44.133 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:280: auto_repair_erasure_coded: for id in $(seq 0 2) 2024-09-15T20:41:44.133 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:281: auto_repair_erasure_coded: run_osd td/osd-scrub-repair 0 --osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0 2024-09-15T20:41:44.133 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-scrub-repair 2024-09-15T20:41:44.134 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-15T20:41:44.134 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=0 2024-09-15T20:41:44.134 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-15T20:41:44.134 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-scrub-repair/0 2024-09-15T20:41:44.134 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=ce4892c7-6c6d-4b4f-8e6f-07a3e11133e5 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-15T20:41:44.134 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T20:41:44.134 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T20:41:44.134 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T20:41:44.134 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-09-15T20:41:44.134 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-09-15T20:41:44.134 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-15T20:41:44.134 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-15T20:41:44.134 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-15T20:41:44.134 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-15T20:41:44.135 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-15T20:41:44.135 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-15T20:41:44.135 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-15T20:41:44.135 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T20:41:44.135 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52602 2024-09-15T20:41:44.135 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.52602/$cluster-$name.asok' 2024-09-15T20:41:44.135 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.52602/$cluster-$name.asok' 2024-09-15T20:41:44.135 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-15T20:41:44.135 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-15T20:41:44.135 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-15T20:41:44.135 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-15T20:41:44.135 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-15T20:41:44.135 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T20:41:44.136 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T20:41:44.136 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T20:41:44.136 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T20:41:44.136 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-15T20:41:44.136 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+='--osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0' 2024-09-15T20:41:44.136 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-scrub-repair/0 2024-09-15T20:41:44.136 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-15T20:41:44.136 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=5a0bf27e-a01a-4349-8102-5637a3eeb7d8 2024-09-15T20:41:44.137 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd0 5a0bf27e-a01a-4349-8102-5637a3eeb7d8' 2024-09-15T20:41:44.137 INFO:tasks.workunit.client.0.smithi136.stdout:add osd0 5a0bf27e-a01a-4349-8102-5637a3eeb7d8 2024-09-15T20:41:44.137 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-15T20:41:44.154 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQCIRudmGfMICRAAEsDgI40aTACk/xsgV2TIZw== 2024-09-15T20:41:44.154 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQCIRudmGfMICRAAEsDgI40aTACk/xsgV2TIZw=="}' 2024-09-15T20:41:44.154 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 5a0bf27e-a01a-4349-8102-5637a3eeb7d8 -i td/osd-scrub-repair/0/new.json 2024-09-15T20:41:44.403 INFO:tasks.workunit.client.0.smithi136.stdout:0 2024-09-15T20:41:44.412 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-scrub-repair/0/new.json 2024-09-15T20:41:44.413 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 0 --fsid=ce4892c7-6c6d-4b4f-8e6f-07a3e11133e5 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.52602/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0 --mkfs --key AQCIRudmGfMICRAAEsDgI40aTACk/xsgV2TIZw== --osd-uuid 5a0bf27e-a01a-4349-8102-5637a3eeb7d8 2024-09-15T20:41:44.446 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:41:44.444+0000 7f70f800d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T20:41:44.454 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:41:44.452+0000 7f70f800d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T20:41:44.455 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:41:44.453+0000 7f70f800d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T20:41:44.455 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:41:44.453+0000 7f70f800d640 -1 bdev(0x56191e20bc00 td/osd-scrub-repair/0/block) open stat got: (1) Operation not permitted 2024-09-15T20:41:44.455 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:41:44.453+0000 7f70f800d640 -1 bluestore(td/osd-scrub-repair/0) _read_fsid unparsable uuid 2024-09-15T20:41:49.949 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-scrub-repair/0/keyring 2024-09-15T20:41:49.949 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-15T20:41:49.950 INFO:tasks.workunit.client.0.smithi136.stdout:adding osd0 key to auth repository 2024-09-15T20:41:49.950 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd0 key to auth repository 2024-09-15T20:41:49.950 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-scrub-repair/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-15T20:41:50.310 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.0 2024-09-15T20:41:50.311 INFO:tasks.workunit.client.0.smithi136.stdout:start osd.0 2024-09-15T20:41:50.311 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 0 --fsid=ce4892c7-6c6d-4b4f-8e6f-07a3e11133e5 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.52602/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0 2024-09-15T20:41:50.311 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-15T20:41:50.311 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-15T20:41:50.311 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-15T20:41:50.341 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:41:50.339+0000 7fa3ac929640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T20:41:50.343 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:41:50.341+0000 7fa3ac929640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T20:41:50.344 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:41:50.342+0000 7fa3ac929640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T20:41:50.589 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 0 2024-09-15T20:41:50.590 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-15T20:41:50.590 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2024-09-15T20:41:50.590 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-15T20:41:50.590 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-15T20:41:50.590 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:41:50.591 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-15T20:41:50.591 INFO:tasks.workunit.client.0.smithi136.stdout:0 2024-09-15T20:41:50.592 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:41:50.592 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T20:41:50.863 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:41:51.699 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:41:51.697+0000 7fa3ac929640 -1 Falling back to public interface 2024-09-15T20:41:51.864 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:41:51.864 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:41:51.865 INFO:tasks.workunit.client.0.smithi136.stdout:1 2024-09-15T20:41:51.865 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-15T20:41:51.865 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:41:51.865 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T20:41:52.153 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:41:53.155 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:41:53.155 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:41:53.155 INFO:tasks.workunit.client.0.smithi136.stdout:2 2024-09-15T20:41:53.155 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-15T20:41:53.155 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:41:53.156 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T20:41:53.445 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:41:54.446 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:41:54.446 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:41:54.447 INFO:tasks.workunit.client.0.smithi136.stdout:3 2024-09-15T20:41:54.447 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-15T20:41:54.447 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:41:54.447 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T20:41:54.582 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:41:54.580+0000 7fa3ac929640 -1 osd.0 0 log_to_monitors true 2024-09-15T20:41:54.756 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:41:55.757 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:41:55.758 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:41:55.758 INFO:tasks.workunit.client.0.smithi136.stdout:4 2024-09-15T20:41:55.758 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-15T20:41:55.759 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:41:55.759 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T20:41:56.046 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:41:57.047 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:41:57.047 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:41:57.048 INFO:tasks.workunit.client.0.smithi136.stdout:5 2024-09-15T20:41:57.048 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-15T20:41:57.048 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:41:57.048 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T20:41:57.334 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:41:58.335 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:41:58.335 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:41:58.336 INFO:tasks.workunit.client.0.smithi136.stdout:6 2024-09-15T20:41:58.336 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-15T20:41:58.336 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:41:58.336 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T20:41:58.622 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:41:59.623 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:41:59.624 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:41:59.624 INFO:tasks.workunit.client.0.smithi136.stdout:7 2024-09-15T20:41:59.624 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-15T20:41:59.624 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:41:59.625 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T20:41:59.912 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:42:00.914 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:42:00.914 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:42:00.914 INFO:tasks.workunit.client.0.smithi136.stdout:8 2024-09-15T20:42:00.915 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-15T20:42:00.915 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:42:00.915 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T20:42:01.323 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:42:02.324 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:42:02.325 INFO:tasks.workunit.client.0.smithi136.stdout:9 2024-09-15T20:42:02.325 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:42:02.325 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-15T20:42:02.325 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:42:02.326 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T20:42:02.611 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:42:03.612 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:42:03.612 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:42:03.612 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-15T20:42:03.612 INFO:tasks.workunit.client.0.smithi136.stdout:10 2024-09-15T20:42:03.613 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:42:03.613 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T20:42:03.899 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:42:04.900 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:42:04.901 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:42:04.901 INFO:tasks.workunit.client.0.smithi136.stdout:11 2024-09-15T20:42:04.902 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-15T20:42:04.902 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:42:04.902 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T20:42:05.191 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:42:06.192 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:42:06.192 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:42:06.192 INFO:tasks.workunit.client.0.smithi136.stdout:12 2024-09-15T20:42:06.193 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-15T20:42:06.193 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:42:06.193 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T20:42:06.964 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:42:07.965 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:42:07.965 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:42:07.965 INFO:tasks.workunit.client.0.smithi136.stdout:13 2024-09-15T20:42:07.966 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-15T20:42:07.966 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:42:07.966 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T20:42:08.933 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:42:09.934 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:42:09.934 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:42:09.935 INFO:tasks.workunit.client.0.smithi136.stdout:14 2024-09-15T20:42:09.936 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-15T20:42:09.936 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:42:09.937 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T20:42:10.366 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:42:11.367 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:42:11.367 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:42:11.367 INFO:tasks.workunit.client.0.smithi136.stdout:15 2024-09-15T20:42:11.368 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-15T20:42:11.368 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:42:11.368 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T20:42:12.058 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:42:13.059 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:42:13.059 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:42:13.060 INFO:tasks.workunit.client.0.smithi136.stdout:16 2024-09-15T20:42:13.060 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-15T20:42:13.060 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:42:13.060 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T20:42:13.773 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:42:14.774 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:42:14.774 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:42:14.774 INFO:tasks.workunit.client.0.smithi136.stdout:17 2024-09-15T20:42:14.775 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-15T20:42:14.775 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:42:14.775 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T20:42:15.625 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:42:16.626 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:42:16.626 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:42:16.626 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 18 2024-09-15T20:42:16.627 INFO:tasks.workunit.client.0.smithi136.stdout:18 2024-09-15T20:42:16.627 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:42:16.627 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T20:42:16.955 INFO:tasks.workunit.client.0.smithi136.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/3268487642,v1:127.0.0.1:6803/3268487642] [v2:127.0.0.1:6804/3268487642,v1:127.0.0.1:6805/3268487642] exists,up 5a0bf27e-a01a-4349-8102-5637a3eeb7d8 2024-09-15T20:42:16.962 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-15T20:42:16.962 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-15T20:42:16.962 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-15T20:42:16.963 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:280: auto_repair_erasure_coded: for id in $(seq 0 2) 2024-09-15T20:42:16.963 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:281: auto_repair_erasure_coded: run_osd td/osd-scrub-repair 1 --osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0 2024-09-15T20:42:16.963 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-scrub-repair 2024-09-15T20:42:16.963 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-15T20:42:16.963 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=1 2024-09-15T20:42:16.963 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-15T20:42:16.964 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-scrub-repair/1 2024-09-15T20:42:16.964 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=ce4892c7-6c6d-4b4f-8e6f-07a3e11133e5 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-15T20:42:16.964 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T20:42:16.964 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T20:42:16.964 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T20:42:16.964 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-09-15T20:42:16.965 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-09-15T20:42:16.965 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-15T20:42:16.965 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-15T20:42:16.965 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-15T20:42:16.965 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-15T20:42:16.965 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-15T20:42:16.965 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-15T20:42:16.965 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-15T20:42:16.965 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T20:42:16.965 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52602 2024-09-15T20:42:16.965 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.52602/$cluster-$name.asok' 2024-09-15T20:42:16.966 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.52602/$cluster-$name.asok' 2024-09-15T20:42:16.966 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-15T20:42:16.966 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-15T20:42:16.966 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-15T20:42:16.966 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-15T20:42:16.966 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-15T20:42:16.966 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T20:42:16.966 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T20:42:16.966 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T20:42:16.966 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T20:42:16.966 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-15T20:42:16.966 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+='--osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0' 2024-09-15T20:42:16.966 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-scrub-repair/1 2024-09-15T20:42:16.966 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-15T20:42:16.967 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=81c8d9ae-5d12-48b9-a757-184106ddccde 2024-09-15T20:42:16.967 INFO:tasks.workunit.client.0.smithi136.stdout:add osd1 81c8d9ae-5d12-48b9-a757-184106ddccde 2024-09-15T20:42:16.967 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd1 81c8d9ae-5d12-48b9-a757-184106ddccde' 2024-09-15T20:42:16.967 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-15T20:42:16.978 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQCoRudmM8IkOhAAjRMfCYb58G6oWuxdycWB2Q== 2024-09-15T20:42:16.978 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQCoRudmM8IkOhAAjRMfCYb58G6oWuxdycWB2Q=="}' 2024-09-15T20:42:16.978 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 81c8d9ae-5d12-48b9-a757-184106ddccde -i td/osd-scrub-repair/1/new.json 2024-09-15T20:42:17.301 INFO:tasks.workunit.client.0.smithi136.stdout:1 2024-09-15T20:42:17.313 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-scrub-repair/1/new.json 2024-09-15T20:42:17.314 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 1 --fsid=ce4892c7-6c6d-4b4f-8e6f-07a3e11133e5 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.52602/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0 --mkfs --key AQCoRudmM8IkOhAAjRMfCYb58G6oWuxdycWB2Q== --osd-uuid 81c8d9ae-5d12-48b9-a757-184106ddccde 2024-09-15T20:42:17.346 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:42:17.344+0000 7f704b21e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T20:42:17.386 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:42:17.383+0000 7f704b21e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T20:42:17.386 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:42:17.384+0000 7f704b21e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T20:42:17.387 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:42:17.385+0000 7f704b21e640 -1 bdev(0x55bff287fc00 td/osd-scrub-repair/1/block) open stat got: (1) Operation not permitted 2024-09-15T20:42:17.387 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:42:17.385+0000 7f704b21e640 -1 bluestore(td/osd-scrub-repair/1) _read_fsid unparsable uuid 2024-09-15T20:42:23.932 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-scrub-repair/1/keyring 2024-09-15T20:42:23.933 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-15T20:42:23.933 INFO:tasks.workunit.client.0.smithi136.stdout:adding osd1 key to auth repository 2024-09-15T20:42:23.934 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd1 key to auth repository 2024-09-15T20:42:23.934 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-scrub-repair/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-15T20:42:24.303 INFO:tasks.workunit.client.0.smithi136.stdout:start osd.1 2024-09-15T20:42:24.304 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.1 2024-09-15T20:42:24.304 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 1 --fsid=ce4892c7-6c6d-4b4f-8e6f-07a3e11133e5 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.52602/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0 2024-09-15T20:42:24.304 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-15T20:42:24.304 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-15T20:42:24.304 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-15T20:42:24.334 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:42:24.331+0000 7f4ef1dbb640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T20:42:24.335 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:42:24.333+0000 7f4ef1dbb640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T20:42:24.336 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:42:24.334+0000 7f4ef1dbb640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T20:42:24.581 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 1 2024-09-15T20:42:24.581 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-15T20:42:24.581 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2024-09-15T20:42:24.581 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-15T20:42:24.581 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-15T20:42:24.581 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:42:24.582 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-15T20:42:24.582 INFO:tasks.workunit.client.0.smithi136.stdout:0 2024-09-15T20:42:24.583 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:42:24.583 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T20:42:24.859 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:42:25.456 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:42:25.453+0000 7f4ef1dbb640 -1 Falling back to public interface 2024-09-15T20:42:25.861 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:42:25.861 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:42:25.861 INFO:tasks.workunit.client.0.smithi136.stdout:1 2024-09-15T20:42:25.862 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-15T20:42:25.862 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:42:25.862 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T20:42:26.161 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:42:27.163 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:42:27.163 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:42:27.163 INFO:tasks.workunit.client.0.smithi136.stdout:2 2024-09-15T20:42:27.164 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-15T20:42:27.164 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:42:27.164 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T20:42:27.453 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:42:28.455 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:42:28.455 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:42:28.455 INFO:tasks.workunit.client.0.smithi136.stdout:3 2024-09-15T20:42:28.456 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-15T20:42:28.456 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:42:28.456 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T20:42:28.517 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:42:28.514+0000 7f4ef1dbb640 -1 osd.1 0 log_to_monitors true 2024-09-15T20:42:28.800 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:42:29.802 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:42:29.802 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:42:29.802 INFO:tasks.workunit.client.0.smithi136.stdout:4 2024-09-15T20:42:29.803 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-15T20:42:29.803 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:42:29.803 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T20:42:30.093 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:42:31.094 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:42:31.094 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:42:31.094 INFO:tasks.workunit.client.0.smithi136.stdout:5 2024-09-15T20:42:31.095 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-15T20:42:31.095 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:42:31.095 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T20:42:31.383 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:42:32.385 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:42:32.385 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:42:32.385 INFO:tasks.workunit.client.0.smithi136.stdout:6 2024-09-15T20:42:32.386 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-15T20:42:32.386 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:42:32.386 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T20:42:32.707 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:42:33.709 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:42:33.709 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:42:33.709 INFO:tasks.workunit.client.0.smithi136.stdout:7 2024-09-15T20:42:33.710 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-15T20:42:33.710 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:42:33.710 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T20:42:33.996 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:42:34.997 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:42:34.997 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:42:34.997 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-15T20:42:34.998 INFO:tasks.workunit.client.0.smithi136.stdout:8 2024-09-15T20:42:34.998 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:42:34.998 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T20:42:35.355 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:42:36.357 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:42:36.357 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:42:36.357 INFO:tasks.workunit.client.0.smithi136.stdout:9 2024-09-15T20:42:36.358 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-15T20:42:36.358 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:42:36.358 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T20:42:36.729 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:42:37.730 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:42:37.731 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:42:37.731 INFO:tasks.workunit.client.0.smithi136.stdout:10 2024-09-15T20:42:37.731 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-15T20:42:37.731 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:42:37.732 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T20:42:38.019 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:42:39.020 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:42:39.021 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:42:39.021 INFO:tasks.workunit.client.0.smithi136.stdout:11 2024-09-15T20:42:39.021 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-15T20:42:39.021 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:42:39.021 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T20:42:39.309 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:42:40.310 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:42:40.311 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:42:40.311 INFO:tasks.workunit.client.0.smithi136.stdout:12 2024-09-15T20:42:40.311 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-15T20:42:40.312 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:42:40.312 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T20:42:40.673 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:42:41.675 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:42:41.675 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:42:41.675 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-15T20:42:41.675 INFO:tasks.workunit.client.0.smithi136.stdout:13 2024-09-15T20:42:41.676 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:42:41.676 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T20:42:42.305 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:42:43.306 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:42:43.306 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:42:43.307 INFO:tasks.workunit.client.0.smithi136.stdout:14 2024-09-15T20:42:43.307 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-15T20:42:43.307 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:42:43.307 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T20:42:43.879 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:42:44.880 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:42:44.880 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:42:44.881 INFO:tasks.workunit.client.0.smithi136.stdout:15 2024-09-15T20:42:44.881 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-15T20:42:44.881 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:42:44.881 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T20:42:45.634 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:42:46.635 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:42:46.636 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:42:46.636 INFO:tasks.workunit.client.0.smithi136.stdout:16 2024-09-15T20:42:46.636 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-15T20:42:46.637 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:42:46.637 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T20:42:47.715 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:42:48.717 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:42:48.717 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:42:48.717 INFO:tasks.workunit.client.0.smithi136.stdout:17 2024-09-15T20:42:48.718 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-15T20:42:48.718 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:42:48.718 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T20:42:49.449 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:42:50.451 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:42:50.451 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:42:50.451 INFO:tasks.workunit.client.0.smithi136.stdout:18 2024-09-15T20:42:50.452 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 18 2024-09-15T20:42:50.452 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:42:50.452 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T20:42:50.874 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:42:51.875 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:42:51.876 INFO:tasks.workunit.client.0.smithi136.stdout:19 2024-09-15T20:42:51.876 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:42:51.876 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 19 2024-09-15T20:42:51.876 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:42:51.876 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T20:42:52.154 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:42:53.156 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:42:53.156 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:42:53.156 INFO:tasks.workunit.client.0.smithi136.stdout:20 2024-09-15T20:42:53.156 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 20 2024-09-15T20:42:53.157 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:42:53.157 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T20:42:53.436 INFO:tasks.workunit.client.0.smithi136.stdout:osd.1 up in weight 1 up_from 16 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/1200421968,v1:127.0.0.1:6811/1200421968] [v2:127.0.0.1:6812/1200421968,v1:127.0.0.1:6813/1200421968] exists,up 81c8d9ae-5d12-48b9-a757-184106ddccde 2024-09-15T20:42:53.440 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-15T20:42:53.440 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-15T20:42:53.440 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-15T20:42:53.440 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:280: auto_repair_erasure_coded: for id in $(seq 0 2) 2024-09-15T20:42:53.440 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:281: auto_repair_erasure_coded: run_osd td/osd-scrub-repair 2 --osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0 2024-09-15T20:42:53.440 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-scrub-repair 2024-09-15T20:42:53.440 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-15T20:42:53.441 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=2 2024-09-15T20:42:53.441 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-15T20:42:53.441 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-scrub-repair/2 2024-09-15T20:42:53.441 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=ce4892c7-6c6d-4b4f-8e6f-07a3e11133e5 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-15T20:42:53.441 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T20:42:53.441 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T20:42:53.441 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T20:42:53.441 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/2' 2024-09-15T20:42:53.441 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/2/journal' 2024-09-15T20:42:53.441 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-15T20:42:53.441 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-15T20:42:53.441 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-15T20:42:53.441 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-15T20:42:53.441 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-15T20:42:53.442 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-15T20:42:53.442 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-15T20:42:53.442 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T20:42:53.442 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52602 2024-09-15T20:42:53.442 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.52602/$cluster-$name.asok' 2024-09-15T20:42:53.442 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.52602/$cluster-$name.asok' 2024-09-15T20:42:53.442 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-15T20:42:53.442 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-15T20:42:53.442 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-15T20:42:53.442 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-15T20:42:53.442 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-15T20:42:53.442 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T20:42:53.442 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T20:42:53.443 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T20:42:53.443 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T20:42:53.443 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-15T20:42:53.443 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+='--osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0' 2024-09-15T20:42:53.443 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-scrub-repair/2 2024-09-15T20:42:53.443 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-15T20:42:53.443 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=1f06c273-4b89-4f36-bb38-7315c679b7f8 2024-09-15T20:42:53.443 INFO:tasks.workunit.client.0.smithi136.stdout:add osd2 1f06c273-4b89-4f36-bb38-7315c679b7f8 2024-09-15T20:42:53.444 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd2 1f06c273-4b89-4f36-bb38-7315c679b7f8' 2024-09-15T20:42:53.444 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-15T20:42:53.458 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQDNRudmLzQwGxAAtyrBiA+iJvs/TUu+wPLGFg== 2024-09-15T20:42:53.458 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQDNRudmLzQwGxAAtyrBiA+iJvs/TUu+wPLGFg=="}' 2024-09-15T20:42:53.459 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 1f06c273-4b89-4f36-bb38-7315c679b7f8 -i td/osd-scrub-repair/2/new.json 2024-09-15T20:42:53.820 INFO:tasks.workunit.client.0.smithi136.stdout:2 2024-09-15T20:42:53.832 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-scrub-repair/2/new.json 2024-09-15T20:42:53.833 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 2 --fsid=ce4892c7-6c6d-4b4f-8e6f-07a3e11133e5 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.52602/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0 --mkfs --key AQDNRudmLzQwGxAAtyrBiA+iJvs/TUu+wPLGFg== --osd-uuid 1f06c273-4b89-4f36-bb38-7315c679b7f8 2024-09-15T20:42:53.865 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:42:53.863+0000 7f5630ad8640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T20:42:53.904 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:42:53.902+0000 7f5630ad8640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T20:42:53.905 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:42:53.903+0000 7f5630ad8640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T20:42:53.906 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:42:53.904+0000 7f5630ad8640 -1 bdev(0x55e713bd6800 td/osd-scrub-repair/2/block) open stat got: (1) Operation not permitted 2024-09-15T20:42:53.906 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:42:53.904+0000 7f5630ad8640 -1 bluestore(td/osd-scrub-repair/2) _read_fsid unparsable uuid 2024-09-15T20:43:00.328 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-scrub-repair/2/keyring 2024-09-15T20:43:00.328 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-15T20:43:00.329 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd2 key to auth repository 2024-09-15T20:43:00.329 INFO:tasks.workunit.client.0.smithi136.stdout:adding osd2 key to auth repository 2024-09-15T20:43:00.329 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-scrub-repair/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-15T20:43:00.693 INFO:tasks.workunit.client.0.smithi136.stdout:start osd.2 2024-09-15T20:43:00.694 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.2 2024-09-15T20:43:00.694 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 2 --fsid=ce4892c7-6c6d-4b4f-8e6f-07a3e11133e5 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.52602/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0 2024-09-15T20:43:00.694 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-15T20:43:00.694 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-15T20:43:00.694 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-15T20:43:00.724 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:43:00.721+0000 7fd7cd818640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T20:43:00.726 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:43:00.724+0000 7fd7cd818640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T20:43:00.727 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:43:00.725+0000 7fd7cd818640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T20:43:00.969 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 2 2024-09-15T20:43:00.969 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-15T20:43:00.969 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=2 2024-09-15T20:43:00.969 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-15T20:43:00.969 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-15T20:43:00.969 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:43:00.970 INFO:tasks.workunit.client.0.smithi136.stdout:0 2024-09-15T20:43:00.970 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-15T20:43:00.970 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:43:00.970 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T20:43:01.248 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:43:02.087 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:43:02.085+0000 7fd7cd818640 -1 Falling back to public interface 2024-09-15T20:43:02.249 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:43:02.249 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:43:02.249 INFO:tasks.workunit.client.0.smithi136.stdout:1 2024-09-15T20:43:02.250 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-15T20:43:02.250 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:43:02.250 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T20:43:02.552 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:43:03.553 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:43:03.554 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:43:03.554 INFO:tasks.workunit.client.0.smithi136.stdout:2 2024-09-15T20:43:03.555 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-15T20:43:03.555 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:43:03.555 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T20:43:03.871 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:43:04.786 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:43:04.784+0000 7fd7cd818640 -1 osd.2 0 log_to_monitors true 2024-09-15T20:43:04.873 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:43:04.873 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:43:04.873 INFO:tasks.workunit.client.0.smithi136.stdout:3 2024-09-15T20:43:04.874 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-15T20:43:04.874 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:43:04.874 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T20:43:05.300 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:43:06.301 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:43:06.302 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:43:06.302 INFO:tasks.workunit.client.0.smithi136.stdout:4 2024-09-15T20:43:06.302 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-15T20:43:06.303 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:43:06.303 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T20:43:06.592 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:43:07.593 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:43:07.593 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:43:07.593 INFO:tasks.workunit.client.0.smithi136.stdout:5 2024-09-15T20:43:07.594 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-15T20:43:07.594 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:43:07.594 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T20:43:07.881 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:43:08.882 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:43:08.882 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:43:08.882 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-15T20:43:08.883 INFO:tasks.workunit.client.0.smithi136.stdout:6 2024-09-15T20:43:08.883 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:43:08.883 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T20:43:09.246 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:43:10.247 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:43:10.248 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:43:10.248 INFO:tasks.workunit.client.0.smithi136.stdout:7 2024-09-15T20:43:10.249 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-15T20:43:10.249 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:43:10.249 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T20:43:10.529 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:43:11.531 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:43:11.531 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:43:11.531 INFO:tasks.workunit.client.0.smithi136.stdout:8 2024-09-15T20:43:11.532 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-15T20:43:11.532 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:43:11.532 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T20:43:11.818 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:43:12.820 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:43:12.820 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:43:12.820 INFO:tasks.workunit.client.0.smithi136.stdout:9 2024-09-15T20:43:12.821 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-15T20:43:12.821 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:43:12.821 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T20:43:13.262 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:43:14.264 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:43:14.264 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:43:14.264 INFO:tasks.workunit.client.0.smithi136.stdout:10 2024-09-15T20:43:14.265 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-15T20:43:14.265 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:43:14.265 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T20:43:14.631 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:43:15.632 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:43:15.632 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:43:15.633 INFO:tasks.workunit.client.0.smithi136.stdout:11 2024-09-15T20:43:15.633 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-15T20:43:15.633 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:43:15.633 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T20:43:15.920 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:43:16.921 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:43:16.921 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:43:16.921 INFO:tasks.workunit.client.0.smithi136.stdout:12 2024-09-15T20:43:16.922 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-15T20:43:16.922 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:43:16.922 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T20:43:17.207 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:43:18.209 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:43:18.209 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:43:18.209 INFO:tasks.workunit.client.0.smithi136.stdout:13 2024-09-15T20:43:18.209 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-15T20:43:18.210 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:43:18.210 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T20:43:19.134 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:43:20.135 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:43:20.135 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:43:20.136 INFO:tasks.workunit.client.0.smithi136.stdout:14 2024-09-15T20:43:20.136 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-15T20:43:20.136 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:43:20.136 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T20:43:21.055 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:43:22.057 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:43:22.057 INFO:tasks.workunit.client.0.smithi136.stdout:15 2024-09-15T20:43:22.058 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:43:22.058 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-15T20:43:22.058 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:43:22.058 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T20:43:22.345 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:43:23.346 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:43:23.346 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:43:23.346 INFO:tasks.workunit.client.0.smithi136.stdout:16 2024-09-15T20:43:23.347 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-15T20:43:23.347 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:43:23.347 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T20:43:23.943 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:43:24.945 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:43:24.945 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:43:24.945 INFO:tasks.workunit.client.0.smithi136.stdout:17 2024-09-15T20:43:24.946 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-15T20:43:24.946 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:43:24.946 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T20:43:25.682 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:43:26.683 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:43:26.684 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:43:26.684 INFO:tasks.workunit.client.0.smithi136.stdout:18 2024-09-15T20:43:26.684 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 18 2024-09-15T20:43:26.684 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:43:26.685 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T20:43:27.190 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:43:28.192 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:43:28.192 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:43:28.192 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 19 2024-09-15T20:43:28.192 INFO:tasks.workunit.client.0.smithi136.stdout:19 2024-09-15T20:43:28.193 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:43:28.193 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T20:43:28.485 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:43:29.487 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:43:29.487 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:43:29.487 INFO:tasks.workunit.client.0.smithi136.stdout:20 2024-09-15T20:43:29.488 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 20 2024-09-15T20:43:29.488 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:43:29.488 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T20:43:29.783 INFO:tasks.workunit.client.0.smithi136.stdout:osd.2 up in weight 1 up_from 33 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/1687028726,v1:127.0.0.1:6819/1687028726] [v2:127.0.0.1:6820/1687028726,v1:127.0.0.1:6821/1687028726] exists,up 1f06c273-4b89-4f36-bb38-7315c679b7f8 2024-09-15T20:43:29.783 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-15T20:43:29.784 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-15T20:43:29.784 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-15T20:43:29.784 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:283: auto_repair_erasure_coded: create_rbd_pool 2024-09-15T20:43:29.784 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:526: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2024-09-15T20:43:30.067 INFO:tasks.workunit.client.0.smithi136.stderr:pool 'rbd' does not exist 2024-09-15T20:43:30.078 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:527: create_rbd_pool: create_pool rbd 4 2024-09-15T20:43:30.078 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create rbd 4 2024-09-15T20:43:30.516 INFO:tasks.workunit.client.0.smithi136.stderr:pool 'rbd' created 2024-09-15T20:43:30.528 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2024-09-15T20:43:31.529 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:528: create_rbd_pool: rbd pool init rbd 2024-09-15T20:43:34.033 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:284: auto_repair_erasure_coded: wait_for_clean 2024-09-15T20:43:34.033 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-15T20:43:34.034 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-15T20:43:34.034 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-15T20:43:34.034 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-15T20:43:34.035 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-15T20:43:34.035 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-15T20:43:34.035 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-15T20:43:34.035 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-15T20:43:34.035 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-15T20:43:34.134 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-15T20:43:34.134 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-15T20:43:34.135 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-15T20:43:34.135 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-15T20:43:34.135 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-15T20:43:34.135 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-15T20:43:34.516 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-15T20:43:34.516 INFO:tasks.workunit.client.0.smithi136.stderr:1 2024-09-15T20:43:34.516 INFO:tasks.workunit.client.0.smithi136.stderr:2' 2024-09-15T20:43:34.516 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-15T20:43:34.516 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T20:43:34.516 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-15T20:43:34.626 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836503 2024-09-15T20:43:34.626 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836503 2024-09-15T20:43:34.626 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836503' 2024-09-15T20:43:34.626 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T20:43:34.626 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-15T20:43:34.729 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=68719476748 2024-09-15T20:43:34.729 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 68719476748 2024-09-15T20:43:34.729 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836503 1-68719476748' 2024-09-15T20:43:34.730 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T20:43:34.730 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-15T20:43:34.879 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=141733920777 2024-09-15T20:43:34.880 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 141733920777 2024-09-15T20:43:34.880 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836503 1-68719476748 2-141733920777' 2024-09-15T20:43:34.880 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T20:43:34.880 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836503 2024-09-15T20:43:34.880 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T20:43:34.880 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-15T20:43:34.881 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836503 2024-09-15T20:43:34.881 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T20:43:34.882 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836503 2024-09-15T20:43:34.882 INFO:tasks.workunit.client.0.smithi136.stdout:waiting osd.0 seq 21474836503 2024-09-15T20:43:34.883 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836503' 2024-09-15T20:43:34.883 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T20:43:35.161 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836502 -lt 21474836503 2024-09-15T20:43:35.161 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-15T20:43:36.161 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-15T20:43:36.161 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T20:43:36.450 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836503 -lt 21474836503 2024-09-15T20:43:36.450 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T20:43:36.451 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-68719476748 2024-09-15T20:43:36.451 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T20:43:36.452 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-15T20:43:36.452 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-68719476748 2024-09-15T20:43:36.452 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T20:43:36.453 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=68719476748 2024-09-15T20:43:36.454 INFO:tasks.workunit.client.0.smithi136.stdout:waiting osd.1 seq 68719476748 2024-09-15T20:43:36.454 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 68719476748' 2024-09-15T20:43:36.454 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-15T20:43:36.736 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 68719476748 -lt 68719476748 2024-09-15T20:43:36.736 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T20:43:36.737 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-141733920777 2024-09-15T20:43:36.737 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T20:43:36.738 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-15T20:43:36.738 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-141733920777 2024-09-15T20:43:36.738 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T20:43:36.739 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=141733920777 2024-09-15T20:43:36.740 INFO:tasks.workunit.client.0.smithi136.stdout:waiting osd.2 seq 141733920777 2024-09-15T20:43:36.740 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 141733920777' 2024-09-15T20:43:36.740 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-15T20:43:37.108 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 141733920777 -lt 141733920777 2024-09-15T20:43:37.108 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-15T20:43:37.108 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T20:43:37.108 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T20:43:37.454 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 4 == 0 2024-09-15T20:43:37.454 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-15T20:43:37.454 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-15T20:43:37.454 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-15T20:43:37.454 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T20:43:37.454 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T20:43:37.454 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T20:43:37.454 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T20:43:37.729 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=4 2024-09-15T20:43:37.730 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-15T20:43:37.730 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T20:43:37.730 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T20:43:38.071 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 = 4 2024-09-15T20:43:38.071 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-15T20:43:38.071 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-15T20:43:38.071 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:287: auto_repair_erasure_coded: create_ec_pool ecpool false k=2 m=1 2024-09-15T20:43:38.071 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2443: create_ec_pool: local pool_name=ecpool 2024-09-15T20:43:38.072 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2444: create_ec_pool: shift 2024-09-15T20:43:38.072 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2445: create_ec_pool: local allow_overwrites=false 2024-09-15T20:43:38.072 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2446: create_ec_pool: shift 2024-09-15T20:43:38.072 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2448: create_ec_pool: ceph osd erasure-code-profile set myprofile crush-failure-domain=osd k=2 m=1 2024-09-15T20:43:38.556 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2450: create_ec_pool: create_pool ecpool 1 1 erasure myprofile 2024-09-15T20:43:38.557 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create ecpool 1 1 erasure myprofile 2024-09-15T20:43:39.215 INFO:tasks.workunit.client.0.smithi136.stderr:pool 'ecpool' created 2024-09-15T20:43:39.226 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2024-09-15T20:43:40.230 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2452: create_ec_pool: '[' false = true ']' 2024-09-15T20:43:40.230 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2456: create_ec_pool: wait_for_clean 2024-09-15T20:43:40.230 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-15T20:43:40.231 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-15T20:43:40.231 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-15T20:43:40.231 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-15T20:43:40.231 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-15T20:43:40.231 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-15T20:43:40.231 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-15T20:43:40.232 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-15T20:43:40.232 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-15T20:43:40.330 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-15T20:43:40.331 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-15T20:43:40.331 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-15T20:43:40.331 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-15T20:43:40.331 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-15T20:43:40.331 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-15T20:43:40.643 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-15T20:43:40.644 INFO:tasks.workunit.client.0.smithi136.stderr:1 2024-09-15T20:43:40.644 INFO:tasks.workunit.client.0.smithi136.stderr:2' 2024-09-15T20:43:40.644 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-15T20:43:40.644 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T20:43:40.644 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-15T20:43:40.819 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836505 2024-09-15T20:43:40.819 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836505 2024-09-15T20:43:40.819 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836505' 2024-09-15T20:43:40.819 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T20:43:40.819 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-15T20:43:40.922 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=68719476750 2024-09-15T20:43:40.922 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 68719476750 2024-09-15T20:43:40.922 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836505 1-68719476750' 2024-09-15T20:43:40.922 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T20:43:40.922 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-15T20:43:41.029 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=141733920779 2024-09-15T20:43:41.029 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 141733920779 2024-09-15T20:43:41.029 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836505 1-68719476750 2-141733920779' 2024-09-15T20:43:41.029 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T20:43:41.029 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836505 2024-09-15T20:43:41.029 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T20:43:41.029 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-15T20:43:41.029 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836505 2024-09-15T20:43:41.030 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T20:43:41.030 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836505 2024-09-15T20:43:41.030 INFO:tasks.workunit.client.0.smithi136.stdout:waiting osd.0 seq 21474836505 2024-09-15T20:43:41.031 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836505' 2024-09-15T20:43:41.031 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T20:43:41.352 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836504 -lt 21474836505 2024-09-15T20:43:41.352 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-15T20:43:42.353 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-15T20:43:42.353 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T20:43:42.637 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836506 -lt 21474836505 2024-09-15T20:43:42.638 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T20:43:42.638 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-68719476750 2024-09-15T20:43:42.638 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T20:43:42.639 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-15T20:43:42.639 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-68719476750 2024-09-15T20:43:42.639 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T20:43:42.640 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=68719476750 2024-09-15T20:43:42.641 INFO:tasks.workunit.client.0.smithi136.stdout:waiting osd.1 seq 68719476750 2024-09-15T20:43:42.641 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 68719476750' 2024-09-15T20:43:42.641 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-15T20:43:43.054 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 68719476750 -lt 68719476750 2024-09-15T20:43:43.055 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T20:43:43.055 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-141733920779 2024-09-15T20:43:43.055 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T20:43:43.056 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-15T20:43:43.056 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-141733920779 2024-09-15T20:43:43.056 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T20:43:43.057 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=141733920779 2024-09-15T20:43:43.058 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 141733920779' 2024-09-15T20:43:43.058 INFO:tasks.workunit.client.0.smithi136.stdout:waiting osd.2 seq 141733920779 2024-09-15T20:43:43.058 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-15T20:43:43.338 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 141733920779 -lt 141733920779 2024-09-15T20:43:43.339 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-15T20:43:43.339 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T20:43:43.339 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T20:43:43.683 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 5 == 0 2024-09-15T20:43:43.684 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-15T20:43:43.684 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-15T20:43:43.684 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-15T20:43:43.684 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T20:43:43.684 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T20:43:43.684 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T20:43:43.684 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T20:43:43.957 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=5 2024-09-15T20:43:43.958 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-15T20:43:43.958 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T20:43:43.958 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T20:43:44.308 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 = 5 2024-09-15T20:43:44.309 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-15T20:43:44.309 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-15T20:43:44.309 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2457: create_ec_pool: return 0 2024-09-15T20:43:44.309 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:290: auto_repair_erasure_coded: local payload=ABCDEF 2024-09-15T20:43:44.309 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:291: auto_repair_erasure_coded: echo ABCDEF 2024-09-15T20:43:44.309 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:292: auto_repair_erasure_coded: rados --pool ecpool put SOMETHING td/osd-scrub-repair/ORIGINAL 2024-09-15T20:43:44.508 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:296: auto_repair_erasure_coded: get_not_primary ecpool SOMETHING 2024-09-15T20:43:44.508 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1220: get_not_primary: local poolname=ecpool 2024-09-15T20:43:44.508 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1221: get_not_primary: local objectname=SOMETHING 2024-09-15T20:43:44.508 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1223: get_not_primary: get_primary ecpool SOMETHING 2024-09-15T20:43:44.508 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1187: get_primary: local poolname=ecpool 2024-09-15T20:43:44.508 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1188: get_primary: local objectname=SOMETHING 2024-09-15T20:43:44.508 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1190: get_primary: ceph --format json osd map ecpool SOMETHING 2024-09-15T20:43:44.509 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1191: get_primary: jq .acting_primary 2024-09-15T20:43:44.938 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1223: get_not_primary: local primary=2 2024-09-15T20:43:44.938 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1224: get_not_primary: ceph --format json osd map ecpool SOMETHING 2024-09-15T20:43:44.939 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1225: get_not_primary: jq '.acting | map(select (. != 2)) | .[0]' 2024-09-15T20:43:45.219 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:296: auto_repair_erasure_coded: objectstore_tool td/osd-scrub-repair 1 SOMETHING remove 2024-09-15T20:43:45.219 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2024-09-15T20:43:45.219 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2024-09-15T20:43:45.220 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=1 2024-09-15T20:43:45.220 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2024-09-15T20:43:45.220 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 SOMETHING remove 2024-09-15T20:43:45.220 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-09-15T20:43:45.220 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2024-09-15T20:43:45.220 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=1 2024-09-15T20:43:45.220 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2024-09-15T20:43:45.220 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-09-15T20:43:45.220 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-15T20:43:45.220 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-15T20:43:45.220 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-15T20:43:45.220 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-15T20:43:45.220 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-15T20:43:45.525 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-15T20:43:45.526 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 SOMETHING remove 2024-09-15T20:43:45.526 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-15T20:43:45.526 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-15T20:43:45.526 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=1 2024-09-15T20:43:45.526 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-15T20:43:45.526 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-09-15T20:43:45.526 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 SOMETHING remove 2024-09-15T20:43:47.366 INFO:tasks.workunit.client.0.smithi136.stderr:Error getting attr on : 1.3_head,#-3:c0000000:::scrub_1.3:head#, (61) No data available 2024-09-15T20:43:47.366 INFO:tasks.workunit.client.0.smithi136.stderr:Error getting attr on : 1.0_head,#-3:00000000:::scrub_1.0:head#, (61) No data available 2024-09-15T20:43:47.366 INFO:tasks.workunit.client.0.smithi136.stdout:remove 1#2:eb822e21:::SOMETHING:head# 2024-09-15T20:43:47.887 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 --osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0 2024-09-15T20:43:47.888 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2024-09-15T20:43:47.888 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-15T20:43:47.888 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=1 2024-09-15T20:43:47.888 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-15T20:43:47.888 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-09-15T20:43:47.888 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=ce4892c7-6c6d-4b4f-8e6f-07a3e11133e5 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-15T20:43:47.888 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T20:43:47.888 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T20:43:47.888 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T20:43:47.888 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-09-15T20:43:47.889 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-09-15T20:43:47.889 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-15T20:43:47.889 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-15T20:43:47.889 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-15T20:43:47.889 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-15T20:43:47.889 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-15T20:43:47.889 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-15T20:43:47.889 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-15T20:43:47.889 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T20:43:47.889 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52602 2024-09-15T20:43:47.889 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.52602/$cluster-$name.asok' 2024-09-15T20:43:47.889 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.52602/$cluster-$name.asok' 2024-09-15T20:43:47.890 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-15T20:43:47.890 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-15T20:43:47.890 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-15T20:43:47.890 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T20:43:47.890 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T20:43:47.890 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T20:43:47.890 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T20:43:47.890 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-15T20:43:47.890 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+='--osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0' 2024-09-15T20:43:47.890 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-09-15T20:43:47.891 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.1 2024-09-15T20:43:47.891 INFO:tasks.workunit.client.0.smithi136.stderr:start osd.1 2024-09-15T20:43:47.891 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 1 --fsid=ce4892c7-6c6d-4b4f-8e6f-07a3e11133e5 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.52602/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0 2024-09-15T20:43:47.891 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-09-15T20:43:47.891 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 1 = 1 ']' 2024-09-15T20:43:47.892 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-15T20:43:47.892 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-15T20:43:47.892 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-15T20:43:47.921 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:43:47.919+0000 7f5828e18640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T20:43:47.923 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:43:47.921+0000 7f5828e18640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T20:43:47.925 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:43:47.922+0000 7f5828e18640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T20:43:48.177 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 1 2024-09-15T20:43:48.177 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-15T20:43:48.177 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2024-09-15T20:43:48.177 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-15T20:43:48.177 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-15T20:43:48.177 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:43:48.177 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-15T20:43:48.177 INFO:tasks.workunit.client.0.smithi136.stderr:0 2024-09-15T20:43:48.177 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:43:48.177 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T20:43:48.457 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:43:49.043 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:43:49.041+0000 7f5828e18640 -1 Falling back to public interface 2024-09-15T20:43:49.459 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:43:49.459 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:43:49.459 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-15T20:43:49.460 INFO:tasks.workunit.client.0.smithi136.stderr:1 2024-09-15T20:43:49.460 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:43:49.460 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T20:43:49.744 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:43:50.745 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:43:50.746 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:43:50.746 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-15T20:43:50.746 INFO:tasks.workunit.client.0.smithi136.stderr:2 2024-09-15T20:43:50.746 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:43:50.746 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T20:43:51.039 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:43:51.716 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:43:51.714+0000 7f5828e18640 -1 osd.1 56 log_to_monitors true 2024-09-15T20:43:52.040 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:43:52.041 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:43:52.041 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-15T20:43:52.041 INFO:tasks.workunit.client.0.smithi136.stderr:3 2024-09-15T20:43:52.041 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:43:52.041 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T20:43:52.423 INFO:tasks.workunit.client.0.smithi136.stderr:osd.1 up in weight 1 up_from 63 up_thru 36 down_at 57 last_clean_interval [16,56) [v2:127.0.0.1:6810/1318747612,v1:127.0.0.1:6811/1318747612] [v2:127.0.0.1:6812/1318747612,v1:127.0.0.1:6813/1318747612] exists,up 81c8d9ae-5d12-48b9-a757-184106ddccde 2024-09-15T20:43:52.423 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-15T20:43:52.423 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-15T20:43:52.423 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-15T20:43:52.423 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2024-09-15T20:43:52.423 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-15T20:43:52.423 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-15T20:43:52.423 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-15T20:43:52.423 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-15T20:43:52.423 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-15T20:43:52.424 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-15T20:43:52.424 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-15T20:43:52.424 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-15T20:43:52.424 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-15T20:43:52.513 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-15T20:43:52.513 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-15T20:43:52.514 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-15T20:43:52.514 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-15T20:43:52.514 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-15T20:43:52.514 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-15T20:43:53.039 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-15T20:43:53.040 INFO:tasks.workunit.client.0.smithi136.stderr:1 2024-09-15T20:43:53.040 INFO:tasks.workunit.client.0.smithi136.stderr:2' 2024-09-15T20:43:53.040 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-15T20:43:53.040 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T20:43:53.040 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-15T20:43:53.234 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836509 2024-09-15T20:43:53.235 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836509 2024-09-15T20:43:53.235 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836509' 2024-09-15T20:43:53.235 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T20:43:53.235 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-15T20:43:53.437 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=270582939650 2024-09-15T20:43:53.437 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 270582939650 2024-09-15T20:43:53.437 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836509 1-270582939650' 2024-09-15T20:43:53.438 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T20:43:53.438 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-15T20:43:53.553 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=141733920782 2024-09-15T20:43:53.554 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 141733920782 2024-09-15T20:43:53.554 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836509 1-270582939650 2-141733920782' 2024-09-15T20:43:53.554 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T20:43:53.554 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836509 2024-09-15T20:43:53.554 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T20:43:53.555 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-15T20:43:53.555 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836509 2024-09-15T20:43:53.555 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T20:43:53.556 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836509 2024-09-15T20:43:53.556 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836509' 2024-09-15T20:43:53.556 INFO:tasks.workunit.client.0.smithi136.stderr:waiting osd.0 seq 21474836509 2024-09-15T20:43:53.556 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T20:43:53.888 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836508 -lt 21474836509 2024-09-15T20:43:53.888 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-15T20:43:54.889 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-15T20:43:54.889 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T20:43:55.169 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836509 -lt 21474836509 2024-09-15T20:43:55.170 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T20:43:55.170 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-270582939650 2024-09-15T20:43:55.170 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T20:43:55.171 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-15T20:43:55.171 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-270582939650 2024-09-15T20:43:55.172 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T20:43:55.172 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=270582939650 2024-09-15T20:43:55.173 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 270582939650' 2024-09-15T20:43:55.173 INFO:tasks.workunit.client.0.smithi136.stderr:waiting osd.1 seq 270582939650 2024-09-15T20:43:55.173 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-15T20:43:55.458 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 270582939650 -lt 270582939650 2024-09-15T20:43:55.458 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T20:43:55.459 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-141733920782 2024-09-15T20:43:55.459 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T20:43:55.459 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-15T20:43:55.460 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-141733920782 2024-09-15T20:43:55.460 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T20:43:55.461 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=141733920782 2024-09-15T20:43:55.461 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 141733920782' 2024-09-15T20:43:55.461 INFO:tasks.workunit.client.0.smithi136.stderr:waiting osd.2 seq 141733920782 2024-09-15T20:43:55.462 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-15T20:43:55.741 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 141733920782 -lt 141733920782 2024-09-15T20:43:55.741 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-15T20:43:55.741 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T20:43:55.742 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T20:43:56.092 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 5 == 0 2024-09-15T20:43:56.092 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-15T20:43:56.092 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-15T20:43:56.092 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-15T20:43:56.092 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T20:43:56.092 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T20:43:56.092 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T20:43:56.092 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T20:43:56.365 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=4 2024-09-15T20:43:56.366 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-15T20:43:56.366 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T20:43:56.366 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T20:43:56.752 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 = 5 2024-09-15T20:43:56.752 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 4 '!=' -1 2024-09-15T20:43:56.752 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: loop=0 2024-09-15T20:43:56.752 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1652: wait_for_clean: num_active_clean=4 2024-09-15T20:43:56.752 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-15T20:43:56.753 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 0.1 2024-09-15T20:43:56.853 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-15T20:43:56.854 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-15T20:43:56.854 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-15T20:43:56.854 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-15T20:43:56.854 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T20:43:56.854 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T20:43:56.854 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T20:43:56.854 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T20:43:57.128 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=4 2024-09-15T20:43:57.128 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-15T20:43:57.129 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T20:43:57.129 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T20:43:57.491 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 = 5 2024-09-15T20:43:57.492 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 4 '!=' 4 2024-09-15T20:43:57.492 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: get_is_making_recovery_progress 2024-09-15T20:43:57.492 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2024-09-15T20:43:57.492 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-15T20:43:57.492 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-15T20:43:57.492 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-15T20:43:57.492 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2024-09-15T20:43:57.492 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-15T20:43:57.843 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=null 2024-09-15T20:43:57.844 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test null '!=' null 2024-09-15T20:43:57.844 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1655: wait_for_clean: (( 1 >= 13 )) 2024-09-15T20:43:57.844 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-15T20:43:57.844 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 0.2 2024-09-15T20:43:58.045 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-15T20:43:58.045 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-15T20:43:58.045 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-15T20:43:58.045 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-15T20:43:58.046 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T20:43:58.046 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T20:43:58.046 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T20:43:58.046 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T20:43:58.320 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=5 2024-09-15T20:43:58.320 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-15T20:43:58.320 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T20:43:58.320 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T20:43:58.668 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 = 5 2024-09-15T20:43:58.668 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-15T20:43:58.668 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-15T20:43:58.668 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:298: auto_repair_erasure_coded: get_pg ecpool SOMETHING 2024-09-15T20:43:58.668 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1084: get_pg: local poolname=ecpool 2024-09-15T20:43:58.668 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1085: get_pg: local objectname=SOMETHING 2024-09-15T20:43:58.668 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1087: get_pg: ceph --format json osd map ecpool SOMETHING 2024-09-15T20:43:58.668 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1087: get_pg: jq -r .pgid 2024-09-15T20:43:58.957 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:298: auto_repair_erasure_coded: local pgid=2.0 2024-09-15T20:43:58.958 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:299: auto_repair_erasure_coded: get_last_scrub_stamp 2.0 2024-09-15T20:43:58.958 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2024-09-15T20:43:58.958 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-15T20:43:58.958 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-15T20:43:58.958 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-09-15T20:43:59.263 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:299: auto_repair_erasure_coded: wait_for_scrub 2.0 2024-09-15T20:43:44.772743+0000 2024-09-15T20:43:59.263 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: local pgid=2.0 2024-09-15T20:43:59.263 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2018: wait_for_scrub: local last_scrub=2024-09-15T20:43:44.772743+0000 2024-09-15T20:43:59.263 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2019: wait_for_scrub: local sname=last_scrub_stamp 2024-09-15T20:43:59.263 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i=0 )) 2024-09-15T20:43:59.263 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-15T20:43:59.263 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-09-15T20:43:59.264 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2024-09-15T20:43:59.264 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-15T20:43:59.264 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-15T20:43:59.264 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-09-15T20:43:59.535 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-15T20:43:44.772743+0000 '>' 2024-09-15T20:43:44.772743+0000 2024-09-15T20:43:59.535 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-15T20:44:00.536 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-15T20:44:00.536 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-15T20:44:00.537 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-09-15T20:44:00.537 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2024-09-15T20:44:00.537 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-15T20:44:00.537 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-15T20:44:00.537 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-09-15T20:44:00.825 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-15T20:43:58.506624+0000 '>' 2024-09-15T20:43:44.772743+0000 2024-09-15T20:44:00.825 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2023: wait_for_scrub: return 0 2024-09-15T20:44:00.825 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:300: auto_repair_erasure_coded: wait_for_clean 2024-09-15T20:44:00.826 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-15T20:44:00.826 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-15T20:44:00.826 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-15T20:44:00.826 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-15T20:44:00.826 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-15T20:44:00.826 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-15T20:44:00.826 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-15T20:44:00.826 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-15T20:44:00.826 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-15T20:44:00.915 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-15T20:44:00.916 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-15T20:44:00.916 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-15T20:44:00.916 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-15T20:44:00.916 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-15T20:44:00.916 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-15T20:44:01.190 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-15T20:44:01.190 INFO:tasks.workunit.client.0.smithi136.stderr:1 2024-09-15T20:44:01.190 INFO:tasks.workunit.client.0.smithi136.stderr:2' 2024-09-15T20:44:01.191 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-15T20:44:01.191 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T20:44:01.191 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-15T20:44:01.294 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836511 2024-09-15T20:44:01.294 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836511 2024-09-15T20:44:01.294 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836511' 2024-09-15T20:44:01.295 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T20:44:01.295 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-15T20:44:01.406 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=270582939652 2024-09-15T20:44:01.406 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 270582939652 2024-09-15T20:44:01.406 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836511 1-270582939652' 2024-09-15T20:44:01.406 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T20:44:01.406 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-15T20:44:01.508 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=141733920785 2024-09-15T20:44:01.508 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 141733920785 2024-09-15T20:44:01.508 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836511 1-270582939652 2-141733920785' 2024-09-15T20:44:01.508 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T20:44:01.509 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836511 2024-09-15T20:44:01.509 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T20:44:01.509 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-15T20:44:01.510 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836511 2024-09-15T20:44:01.510 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T20:44:01.511 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836511 2024-09-15T20:44:01.511 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836511' 2024-09-15T20:44:01.512 INFO:tasks.workunit.client.0.smithi136.stdout:waiting osd.0 seq 21474836511 2024-09-15T20:44:01.512 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T20:44:01.787 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836510 -lt 21474836511 2024-09-15T20:44:01.787 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-15T20:44:02.788 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-15T20:44:02.788 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T20:44:03.075 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836512 -lt 21474836511 2024-09-15T20:44:03.075 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T20:44:03.075 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-270582939652 2024-09-15T20:44:03.075 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T20:44:03.076 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-15T20:44:03.077 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-270582939652 2024-09-15T20:44:03.077 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T20:44:03.078 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=270582939652 2024-09-15T20:44:03.078 INFO:tasks.workunit.client.0.smithi136.stdout:waiting osd.1 seq 270582939652 2024-09-15T20:44:03.079 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 270582939652' 2024-09-15T20:44:03.079 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-15T20:44:03.358 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 270582939653 -lt 270582939652 2024-09-15T20:44:03.358 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T20:44:03.359 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-141733920785 2024-09-15T20:44:03.359 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T20:44:03.360 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-15T20:44:03.360 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-141733920785 2024-09-15T20:44:03.361 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T20:44:03.361 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=141733920785 2024-09-15T20:44:03.361 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 141733920785' 2024-09-15T20:44:03.362 INFO:tasks.workunit.client.0.smithi136.stdout:waiting osd.2 seq 141733920785 2024-09-15T20:44:03.362 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-15T20:44:03.637 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 141733920785 -lt 141733920785 2024-09-15T20:44:03.638 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-15T20:44:03.638 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T20:44:03.638 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T20:44:03.985 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 5 == 0 2024-09-15T20:44:03.985 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-15T20:44:03.985 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-15T20:44:03.985 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-15T20:44:03.985 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T20:44:03.986 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T20:44:03.986 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T20:44:03.986 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T20:44:04.261 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=5 2024-09-15T20:44:04.265 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-15T20:44:04.265 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T20:44:04.265 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T20:44:04.796 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 = 5 2024-09-15T20:44:04.796 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-15T20:44:04.796 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-15T20:44:04.796 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:303: auto_repair_erasure_coded: get_not_primary ecpool SOMETHING 2024-09-15T20:44:04.796 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1220: get_not_primary: local poolname=ecpool 2024-09-15T20:44:04.797 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1221: get_not_primary: local objectname=SOMETHING 2024-09-15T20:44:04.797 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1223: get_not_primary: get_primary ecpool SOMETHING 2024-09-15T20:44:04.797 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1187: get_primary: local poolname=ecpool 2024-09-15T20:44:04.797 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1188: get_primary: local objectname=SOMETHING 2024-09-15T20:44:04.797 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1190: get_primary: ceph --format json osd map ecpool SOMETHING 2024-09-15T20:44:04.797 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1191: get_primary: jq .acting_primary 2024-09-15T20:44:05.152 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1223: get_not_primary: local primary=2 2024-09-15T20:44:05.152 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1224: get_not_primary: ceph --format json osd map ecpool SOMETHING 2024-09-15T20:44:05.152 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1225: get_not_primary: jq '.acting | map(select (. != 2)) | .[0]' 2024-09-15T20:44:05.436 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:303: auto_repair_erasure_coded: objectstore_tool td/osd-scrub-repair 1 SOMETHING list-attrs 2024-09-15T20:44:05.436 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2024-09-15T20:44:05.436 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2024-09-15T20:44:05.436 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=1 2024-09-15T20:44:05.436 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2024-09-15T20:44:05.436 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 SOMETHING list-attrs 2024-09-15T20:44:05.436 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-09-15T20:44:05.436 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2024-09-15T20:44:05.436 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=1 2024-09-15T20:44:05.436 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2024-09-15T20:44:05.436 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-09-15T20:44:05.436 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-15T20:44:05.437 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-15T20:44:05.437 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-15T20:44:05.437 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-15T20:44:05.437 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-15T20:44:05.742 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-15T20:44:05.742 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 SOMETHING list-attrs 2024-09-15T20:44:05.742 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-15T20:44:05.742 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-15T20:44:05.742 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=1 2024-09-15T20:44:05.742 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-15T20:44:05.742 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-09-15T20:44:05.742 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 SOMETHING list-attrs 2024-09-15T20:44:06.133 INFO:tasks.workunit.client.0.smithi136.stderr:Error getting attr on : 1.3_head,#-3:c0000000:::scrub_1.3:head#, (61) No data available 2024-09-15T20:44:06.133 INFO:tasks.workunit.client.0.smithi136.stderr:Error getting attr on : 1.0_head,#-3:00000000:::scrub_1.0:head#, (61) No data available 2024-09-15T20:44:06.133 INFO:tasks.workunit.client.0.smithi136.stdout:_ 2024-09-15T20:44:06.134 INFO:tasks.workunit.client.0.smithi136.stdout:hinfo_key 2024-09-15T20:44:06.134 INFO:tasks.workunit.client.0.smithi136.stdout:snapset 2024-09-15T20:44:06.401 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 --osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0 2024-09-15T20:44:06.401 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2024-09-15T20:44:06.401 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-15T20:44:06.401 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=1 2024-09-15T20:44:06.401 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-15T20:44:06.401 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-09-15T20:44:06.401 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=ce4892c7-6c6d-4b4f-8e6f-07a3e11133e5 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-15T20:44:06.402 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T20:44:06.402 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T20:44:06.402 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T20:44:06.402 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-09-15T20:44:06.402 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-09-15T20:44:06.402 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-15T20:44:06.402 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-15T20:44:06.402 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-15T20:44:06.402 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-15T20:44:06.402 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-15T20:44:06.402 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-15T20:44:06.402 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-15T20:44:06.402 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T20:44:06.403 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52602 2024-09-15T20:44:06.403 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.52602/$cluster-$name.asok' 2024-09-15T20:44:06.403 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.52602/$cluster-$name.asok' 2024-09-15T20:44:06.403 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-15T20:44:06.403 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-15T20:44:06.403 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-15T20:44:06.403 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T20:44:06.403 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T20:44:06.403 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T20:44:06.403 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T20:44:06.403 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-15T20:44:06.403 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+='--osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0' 2024-09-15T20:44:06.403 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-09-15T20:44:06.404 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.1 2024-09-15T20:44:06.404 INFO:tasks.workunit.client.0.smithi136.stderr:start osd.1 2024-09-15T20:44:06.404 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 1 --fsid=ce4892c7-6c6d-4b4f-8e6f-07a3e11133e5 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.52602/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0 2024-09-15T20:44:06.405 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-09-15T20:44:06.405 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 1 = 1 ']' 2024-09-15T20:44:06.405 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-15T20:44:06.405 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-15T20:44:06.405 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-15T20:44:06.437 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:44:06.434+0000 7f755fa15640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T20:44:06.504 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:44:06.502+0000 7f755fa15640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T20:44:06.505 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:44:06.503+0000 7f755fa15640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T20:44:06.705 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 1 2024-09-15T20:44:06.706 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-15T20:44:06.706 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2024-09-15T20:44:06.706 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-15T20:44:06.706 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-15T20:44:06.706 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:44:06.706 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-15T20:44:06.706 INFO:tasks.workunit.client.0.smithi136.stderr:0 2024-09-15T20:44:06.706 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:44:06.706 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T20:44:06.990 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:44:07.992 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:44:07.992 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:44:07.992 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-15T20:44:07.993 INFO:tasks.workunit.client.0.smithi136.stderr:1 2024-09-15T20:44:07.993 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:44:07.993 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T20:44:08.095 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:44:08.093+0000 7f755fa15640 -1 Falling back to public interface 2024-09-15T20:44:08.320 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:44:09.322 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:44:09.322 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:44:09.322 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-15T20:44:09.323 INFO:tasks.workunit.client.0.smithi136.stderr:2 2024-09-15T20:44:09.323 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:44:09.323 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T20:44:09.609 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:44:10.610 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:44:10.611 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:44:10.611 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-15T20:44:10.611 INFO:tasks.workunit.client.0.smithi136.stderr:3 2024-09-15T20:44:10.611 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:44:10.611 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T20:44:10.893 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:44:11.073 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:44:11.070+0000 7f755fa15640 -1 osd.1 71 log_to_monitors true 2024-09-15T20:44:11.894 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:44:11.894 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:44:11.894 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-15T20:44:11.894 INFO:tasks.workunit.client.0.smithi136.stderr:4 2024-09-15T20:44:11.894 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:44:11.894 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T20:44:12.226 INFO:tasks.workunit.client.0.smithi136.stderr:osd.1 up in weight 1 up_from 76 up_thru 63 down_at 72 last_clean_interval [63,71) [v2:127.0.0.1:6810/3052877396,v1:127.0.0.1:6811/3052877396] [v2:127.0.0.1:6812/3052877396,v1:127.0.0.1:6813/3052877396] exists,up 81c8d9ae-5d12-48b9-a757-184106ddccde 2024-09-15T20:44:12.227 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-15T20:44:12.227 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-15T20:44:12.227 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-15T20:44:12.227 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2024-09-15T20:44:12.227 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-15T20:44:12.227 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-15T20:44:12.227 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-15T20:44:12.227 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-15T20:44:12.228 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-15T20:44:12.228 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-15T20:44:12.228 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-15T20:44:12.228 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-15T20:44:12.228 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-15T20:44:12.316 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-15T20:44:12.316 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-15T20:44:12.316 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-15T20:44:12.316 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-15T20:44:12.316 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-15T20:44:12.316 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-15T20:44:12.620 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-15T20:44:12.620 INFO:tasks.workunit.client.0.smithi136.stderr:1 2024-09-15T20:44:12.620 INFO:tasks.workunit.client.0.smithi136.stderr:2' 2024-09-15T20:44:12.620 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-15T20:44:12.620 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T20:44:12.620 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-15T20:44:12.852 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836515 2024-09-15T20:44:12.853 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836515 2024-09-15T20:44:12.853 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836515' 2024-09-15T20:44:12.853 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T20:44:12.853 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-15T20:44:12.980 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=326417514498 2024-09-15T20:44:12.981 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 326417514498 2024-09-15T20:44:12.981 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836515 1-326417514498' 2024-09-15T20:44:12.981 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T20:44:12.981 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-15T20:44:13.106 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=141733920788 2024-09-15T20:44:13.106 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 141733920788 2024-09-15T20:44:13.106 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836515 1-326417514498 2-141733920788' 2024-09-15T20:44:13.106 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T20:44:13.106 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836515 2024-09-15T20:44:13.106 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T20:44:13.107 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-15T20:44:13.108 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836515 2024-09-15T20:44:13.108 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T20:44:13.109 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836515 2024-09-15T20:44:13.109 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836515' 2024-09-15T20:44:13.109 INFO:tasks.workunit.client.0.smithi136.stderr:waiting osd.0 seq 21474836515 2024-09-15T20:44:13.109 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T20:44:13.384 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836514 -lt 21474836515 2024-09-15T20:44:13.384 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-15T20:44:14.385 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-15T20:44:14.386 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T20:44:14.669 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836515 -lt 21474836515 2024-09-15T20:44:14.670 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T20:44:14.670 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-326417514498 2024-09-15T20:44:14.670 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T20:44:14.671 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-15T20:44:14.671 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-326417514498 2024-09-15T20:44:14.671 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T20:44:14.672 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=326417514498 2024-09-15T20:44:14.673 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 326417514498' 2024-09-15T20:44:14.673 INFO:tasks.workunit.client.0.smithi136.stderr:waiting osd.1 seq 326417514498 2024-09-15T20:44:14.673 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-15T20:44:14.949 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 326417514498 -lt 326417514498 2024-09-15T20:44:14.950 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T20:44:14.950 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-141733920788 2024-09-15T20:44:14.950 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T20:44:14.951 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-15T20:44:14.951 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-141733920788 2024-09-15T20:44:14.952 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T20:44:14.952 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=141733920788 2024-09-15T20:44:14.953 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 141733920788' 2024-09-15T20:44:14.953 INFO:tasks.workunit.client.0.smithi136.stderr:waiting osd.2 seq 141733920788 2024-09-15T20:44:14.953 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-15T20:44:15.255 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 141733920788 -lt 141733920788 2024-09-15T20:44:15.255 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-15T20:44:15.256 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T20:44:15.256 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T20:44:15.606 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 5 == 0 2024-09-15T20:44:15.606 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-15T20:44:15.606 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-15T20:44:15.606 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-15T20:44:15.607 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T20:44:15.607 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T20:44:15.607 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T20:44:15.607 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T20:44:15.881 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=5 2024-09-15T20:44:15.882 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-15T20:44:15.882 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T20:44:15.882 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T20:44:16.249 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 = 5 2024-09-15T20:44:16.249 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-15T20:44:16.249 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-15T20:44:16.249 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:304: auto_repair_erasure_coded: rados --pool ecpool get SOMETHING td/osd-scrub-repair/COPY 2024-09-15T20:44:16.312 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:305: auto_repair_erasure_coded: diff td/osd-scrub-repair/ORIGINAL td/osd-scrub-repair/COPY 2024-09-15T20:44:16.313 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:66: run: teardown td/osd-scrub-repair 2024-09-15T20:44:16.314 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/osd-scrub-repair 2024-09-15T20:44:16.314 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-15T20:44:16.314 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/osd-scrub-repair KILL 2024-09-15T20:44:16.314 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-15T20:44:16.314 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-15T20:44:16.314 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-15T20:44:16.314 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-15T20:44:16.314 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-15T20:44:16.422 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-15T20:44:16.422 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-15T20:44:16.422 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-15T20:44:16.423 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-15T20:44:16.424 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-15T20:44:16.424 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-15T20:44:16.424 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-15T20:44:16.426 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T20:44:16.426 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-15T20:44:16.426 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-15T20:44:16.427 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T20:44:16.427 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-15T20:44:16.427 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-15T20:44:16.427 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/osd-scrub-repair 2024-09-15T20:44:16.445 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-15T20:44:16.445 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T20:44:16.445 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52602 2024-09-15T20:44:16.445 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.52602 2024-09-15T20:44:16.447 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-15T20:44:16.447 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-15T20:44:16.447 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:63: run: for func in $funcs 2024-09-15T20:44:16.447 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:64: run: setup td/osd-scrub-repair 2024-09-15T20:44:16.447 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/osd-scrub-repair 2024-09-15T20:44:16.447 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/osd-scrub-repair 2024-09-15T20:44:16.447 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/osd-scrub-repair 2024-09-15T20:44:16.448 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-15T20:44:16.448 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/osd-scrub-repair KILL 2024-09-15T20:44:16.448 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-15T20:44:16.448 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-15T20:44:16.448 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-15T20:44:16.448 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-15T20:44:16.448 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-15T20:44:16.448 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-15T20:44:16.449 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-15T20:44:16.450 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-15T20:44:16.450 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-15T20:44:16.451 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-15T20:44:16.451 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-15T20:44:16.451 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-15T20:44:16.452 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T20:44:16.452 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-15T20:44:16.453 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-15T20:44:16.453 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T20:44:16.453 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-15T20:44:16.454 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-15T20:44:16.454 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/osd-scrub-repair 2024-09-15T20:44:16.456 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-15T20:44:16.456 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T20:44:16.456 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52602 2024-09-15T20:44:16.456 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.52602 2024-09-15T20:44:16.457 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-15T20:44:16.457 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-15T20:44:16.457 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/osd-scrub-repair 2024-09-15T20:44:16.458 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2024-09-15T20:44:16.458 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T20:44:16.458 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52602 2024-09-15T20:44:16.458 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.52602 2024-09-15T20:44:16.461 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2024-09-15T20:44:16.461 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 4096 -le 1024 ']' 2024-09-15T20:44:16.461 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2024-09-15T20:44:16.461 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/osd-scrub-repair 1' TERM HUP INT 2024-09-15T20:44:16.461 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:65: run: TEST_auto_repair_erasure_coded_overwrites td/osd-scrub-repair 2024-09-15T20:44:16.461 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:313: TEST_auto_repair_erasure_coded_overwrites: '[' true = true ']' 2024-09-15T20:44:16.461 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:314: TEST_auto_repair_erasure_coded_overwrites: auto_repair_erasure_coded td/osd-scrub-repair true 2024-09-15T20:44:16.461 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:268: auto_repair_erasure_coded: local dir=td/osd-scrub-repair 2024-09-15T20:44:16.461 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:269: auto_repair_erasure_coded: local allow_overwrites=true 2024-09-15T20:44:16.461 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:270: auto_repair_erasure_coded: local poolname=ecpool 2024-09-15T20:44:16.461 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:273: auto_repair_erasure_coded: run_mon td/osd-scrub-repair a 2024-09-15T20:44:16.461 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/osd-scrub-repair 2024-09-15T20:44:16.462 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2024-09-15T20:44:16.462 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2024-09-15T20:44:16.462 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2024-09-15T20:44:16.462 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/osd-scrub-repair/a 2024-09-15T20:44:16.462 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-scrub-repair/a --run-dir=td/osd-scrub-repair 2024-09-15T20:44:16.937 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2024-09-15T20:44:16.937 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-15T20:44:16.937 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-15T20:44:16.937 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-15T20:44:16.937 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T20:44:16.937 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52602 2024-09-15T20:44:16.937 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.52602/$cluster-$name.asok' 2024-09-15T20:44:16.937 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-scrub-repair/a '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.52602/$cluster-$name.asok' --mon-cluster-log-file=td/osd-scrub-repair/log --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-09-15T20:44:17.096 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2024-09-15T20:44:17.097 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2024-09-15T20:44:17.097 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-15T20:44:17.097 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-15T20:44:17.097 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2024-09-15T20:44:17.097 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2024-09-15T20:44:17.098 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-15T20:44:17.098 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-15T20:44:17.099 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-15T20:44:17.099 INFO:tasks.workunit.client.0.smithi136.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-15T20:44:17.099 INFO:tasks.workunit.client.0.smithi136.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T20:44:17.099 INFO:tasks.workunit.client.0.smithi136.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52602 2024-09-15T20:44:17.099 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.52602/ceph-mon.a.asok 2024-09-15T20:44:17.099 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-15T20:44:17.099 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.52602/ceph-mon.a.asok config get fsid 2024-09-15T20:44:17.180 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2024-09-15T20:44:17.181 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-15T20:44:17.181 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-15T20:44:17.181 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2024-09-15T20:44:17.181 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2024-09-15T20:44:17.181 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-15T20:44:17.181 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-15T20:44:17.181 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-15T20:44:17.182 INFO:tasks.workunit.client.0.smithi136.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-15T20:44:17.182 INFO:tasks.workunit.client.0.smithi136.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T20:44:17.182 INFO:tasks.workunit.client.0.smithi136.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52602 2024-09-15T20:44:17.182 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.52602/ceph-mon.a.asok 2024-09-15T20:44:17.182 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-15T20:44:17.182 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.52602/ceph-mon.a.asok config get mon_host 2024-09-15T20:44:17.259 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:274: auto_repair_erasure_coded: run_mgr td/osd-scrub-repair x 2024-09-15T20:44:17.259 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: run_mgr: local dir=td/osd-scrub-repair 2024-09-15T20:44:17.259 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: run_mgr: shift 2024-09-15T20:44:17.259 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: run_mgr: local id=x 2024-09-15T20:44:17.260 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: run_mgr: shift 2024-09-15T20:44:17.260 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:548: run_mgr: local data=td/osd-scrub-repair/x 2024-09-15T20:44:17.260 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:550: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-15T20:44:18.064 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: get_asok_path 2024-09-15T20:44:18.064 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-15T20:44:18.064 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-15T20:44:18.064 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-15T20:44:18.064 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T20:44:18.065 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52602 2024-09-15T20:44:18.065 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.52602/$cluster-$name.asok' 2024-09-15T20:44:18.065 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-15T20:44:18.065 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-scrub-repair/x '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.52602/$cluster-$name.asok' --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-15T20:44:18.093 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:279: auto_repair_erasure_coded: local 'ceph_osd_args=--osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0' 2024-09-15T20:44:18.093 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:280: auto_repair_erasure_coded: seq 0 2 2024-09-15T20:44:18.097 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:280: auto_repair_erasure_coded: for id in $(seq 0 2) 2024-09-15T20:44:18.097 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:281: auto_repair_erasure_coded: run_osd td/osd-scrub-repair 0 --osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0 2024-09-15T20:44:18.097 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-scrub-repair 2024-09-15T20:44:18.097 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-15T20:44:18.097 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=0 2024-09-15T20:44:18.097 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-15T20:44:18.097 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-scrub-repair/0 2024-09-15T20:44:18.097 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=ce4892c7-6c6d-4b4f-8e6f-07a3e11133e5 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-15T20:44:18.097 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T20:44:18.097 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T20:44:18.098 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T20:44:18.098 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-09-15T20:44:18.098 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-09-15T20:44:18.098 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-15T20:44:18.098 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-15T20:44:18.098 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-15T20:44:18.098 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-15T20:44:18.098 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-15T20:44:18.098 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-15T20:44:18.098 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-15T20:44:18.098 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T20:44:18.098 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52602 2024-09-15T20:44:18.098 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.52602/$cluster-$name.asok' 2024-09-15T20:44:18.098 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.52602/$cluster-$name.asok' 2024-09-15T20:44:18.099 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-15T20:44:18.099 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-15T20:44:18.099 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-15T20:44:18.099 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-15T20:44:18.099 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-15T20:44:18.099 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T20:44:18.099 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T20:44:18.099 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T20:44:18.099 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T20:44:18.099 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-15T20:44:18.099 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+='--osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0' 2024-09-15T20:44:18.099 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-scrub-repair/0 2024-09-15T20:44:18.101 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-15T20:44:18.101 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=aa1daf36-b500-48ce-a788-9916d2e28562 2024-09-15T20:44:18.101 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd0 aa1daf36-b500-48ce-a788-9916d2e28562' 2024-09-15T20:44:18.101 INFO:tasks.workunit.client.0.smithi136.stdout:add osd0 aa1daf36-b500-48ce-a788-9916d2e28562 2024-09-15T20:44:18.101 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-15T20:44:18.117 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQAiR+dmDn3VBhAA6QJ6mH0+xWf+s48xFG3JKQ== 2024-09-15T20:44:18.117 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQAiR+dmDn3VBhAA6QJ6mH0+xWf+s48xFG3JKQ=="}' 2024-09-15T20:44:18.117 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new aa1daf36-b500-48ce-a788-9916d2e28562 -i td/osd-scrub-repair/0/new.json 2024-09-15T20:44:18.320 INFO:tasks.workunit.client.0.smithi136.stdout:0 2024-09-15T20:44:18.330 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-scrub-repair/0/new.json 2024-09-15T20:44:18.331 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 0 --fsid=ce4892c7-6c6d-4b4f-8e6f-07a3e11133e5 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.52602/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0 --mkfs --key AQAiR+dmDn3VBhAA6QJ6mH0+xWf+s48xFG3JKQ== --osd-uuid aa1daf36-b500-48ce-a788-9916d2e28562 2024-09-15T20:44:18.364 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:44:18.362+0000 7fbafbc0c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T20:44:18.371 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:44:18.369+0000 7fbafbc0c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T20:44:18.373 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:44:18.371+0000 7fbafbc0c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T20:44:18.373 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:44:18.371+0000 7fbafbc0c640 -1 bdev(0x55b19206bc00 td/osd-scrub-repair/0/block) open stat got: (1) Operation not permitted 2024-09-15T20:44:18.373 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:44:18.371+0000 7fbafbc0c640 -1 bluestore(td/osd-scrub-repair/0) _read_fsid unparsable uuid 2024-09-15T20:44:24.088 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-scrub-repair/0/keyring 2024-09-15T20:44:24.088 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-15T20:44:24.089 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd0 key to auth repository 2024-09-15T20:44:24.089 INFO:tasks.workunit.client.0.smithi136.stdout:adding osd0 key to auth repository 2024-09-15T20:44:24.089 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-scrub-repair/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-15T20:44:24.452 INFO:tasks.workunit.client.0.smithi136.stdout:start osd.0 2024-09-15T20:44:24.452 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.0 2024-09-15T20:44:24.453 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 0 --fsid=ce4892c7-6c6d-4b4f-8e6f-07a3e11133e5 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.52602/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0 2024-09-15T20:44:24.453 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-15T20:44:24.453 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-15T20:44:24.453 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-15T20:44:24.483 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:44:24.481+0000 7f4ef9610640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T20:44:24.485 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:44:24.483+0000 7f4ef9610640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T20:44:24.486 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:44:24.484+0000 7f4ef9610640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T20:44:24.727 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 0 2024-09-15T20:44:24.727 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-15T20:44:24.727 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2024-09-15T20:44:24.727 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-15T20:44:24.727 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-15T20:44:24.727 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:44:24.727 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-15T20:44:24.727 INFO:tasks.workunit.client.0.smithi136.stdout:0 2024-09-15T20:44:24.728 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:44:24.728 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T20:44:25.023 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:44:25.827 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:44:25.825+0000 7f4ef9610640 -1 Falling back to public interface 2024-09-15T20:44:26.024 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:44:26.024 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:44:26.025 INFO:tasks.workunit.client.0.smithi136.stdout:1 2024-09-15T20:44:26.025 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-15T20:44:26.025 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:44:26.025 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T20:44:26.312 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:44:27.314 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:44:27.314 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:44:27.314 INFO:tasks.workunit.client.0.smithi136.stdout:2 2024-09-15T20:44:27.315 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-15T20:44:27.315 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:44:27.315 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T20:44:27.599 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:44:28.600 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:44:28.601 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:44:28.601 INFO:tasks.workunit.client.0.smithi136.stdout:3 2024-09-15T20:44:28.601 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-15T20:44:28.601 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:44:28.601 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T20:44:28.876 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:44:29.052 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:44:29.049+0000 7f4ef9610640 -1 osd.0 0 log_to_monitors true 2024-09-15T20:44:29.877 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:44:29.878 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:44:29.878 INFO:tasks.workunit.client.0.smithi136.stdout:4 2024-09-15T20:44:29.878 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-15T20:44:29.878 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:44:29.879 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T20:44:30.156 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:44:31.157 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:44:31.157 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:44:31.158 INFO:tasks.workunit.client.0.smithi136.stdout:5 2024-09-15T20:44:31.158 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-15T20:44:31.158 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:44:31.158 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T20:44:31.439 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:44:32.440 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:44:32.440 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:44:32.441 INFO:tasks.workunit.client.0.smithi136.stdout:6 2024-09-15T20:44:32.441 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-15T20:44:32.441 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:44:32.441 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T20:44:32.726 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:44:33.728 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:44:33.728 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:44:33.728 INFO:tasks.workunit.client.0.smithi136.stdout:7 2024-09-15T20:44:33.729 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-15T20:44:33.729 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:44:33.729 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T20:44:34.121 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:44:35.122 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:44:35.122 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:44:35.122 INFO:tasks.workunit.client.0.smithi136.stdout:8 2024-09-15T20:44:35.123 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-15T20:44:35.123 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:44:35.123 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T20:44:35.411 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:44:36.412 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:44:36.412 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:44:36.413 INFO:tasks.workunit.client.0.smithi136.stdout:9 2024-09-15T20:44:36.413 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-15T20:44:36.414 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:44:36.414 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T20:44:36.703 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:44:37.704 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:44:37.705 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:44:37.705 INFO:tasks.workunit.client.0.smithi136.stdout:10 2024-09-15T20:44:37.705 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-15T20:44:37.705 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:44:37.705 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T20:44:38.102 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:44:39.104 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:44:39.104 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:44:39.104 INFO:tasks.workunit.client.0.smithi136.stdout:11 2024-09-15T20:44:39.105 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-15T20:44:39.105 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:44:39.105 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T20:44:39.388 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:44:40.389 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:44:40.390 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:44:40.390 INFO:tasks.workunit.client.0.smithi136.stdout:12 2024-09-15T20:44:40.390 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-15T20:44:40.390 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:44:40.391 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T20:44:40.882 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:44:41.883 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:44:41.883 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:44:41.883 INFO:tasks.workunit.client.0.smithi136.stdout:13 2024-09-15T20:44:41.884 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-15T20:44:41.884 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:44:41.884 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T20:44:42.734 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:44:43.736 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:44:43.736 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:44:43.736 INFO:tasks.workunit.client.0.smithi136.stdout:14 2024-09-15T20:44:43.737 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-15T20:44:43.737 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:44:43.737 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T20:44:44.145 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:44:45.147 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:44:45.147 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:44:45.147 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-15T20:44:45.147 INFO:tasks.workunit.client.0.smithi136.stdout:15 2024-09-15T20:44:45.148 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:44:45.148 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T20:44:45.723 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:44:46.724 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:44:46.725 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:44:46.725 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-15T20:44:46.725 INFO:tasks.workunit.client.0.smithi136.stdout:16 2024-09-15T20:44:46.725 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:44:46.725 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T20:44:47.327 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:44:48.328 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:44:48.328 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:44:48.328 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-15T20:44:48.329 INFO:tasks.workunit.client.0.smithi136.stdout:17 2024-09-15T20:44:48.329 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:44:48.329 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T20:44:49.081 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:44:50.082 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:44:50.082 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:44:50.083 INFO:tasks.workunit.client.0.smithi136.stdout:18 2024-09-15T20:44:50.083 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 18 2024-09-15T20:44:50.083 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:44:50.083 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T20:44:50.815 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:44:51.816 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:44:51.816 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:44:51.817 INFO:tasks.workunit.client.0.smithi136.stdout:19 2024-09-15T20:44:51.817 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 19 2024-09-15T20:44:51.817 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:44:51.817 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T20:44:52.104 INFO:tasks.workunit.client.0.smithi136.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/3611340852,v1:127.0.0.1:6803/3611340852] [v2:127.0.0.1:6804/3611340852,v1:127.0.0.1:6805/3611340852] exists,up aa1daf36-b500-48ce-a788-9916d2e28562 2024-09-15T20:44:52.108 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-15T20:44:52.108 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-15T20:44:52.108 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-15T20:44:52.108 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:280: auto_repair_erasure_coded: for id in $(seq 0 2) 2024-09-15T20:44:52.108 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:281: auto_repair_erasure_coded: run_osd td/osd-scrub-repair 1 --osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0 2024-09-15T20:44:52.108 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-scrub-repair 2024-09-15T20:44:52.109 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-15T20:44:52.109 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=1 2024-09-15T20:44:52.109 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-15T20:44:52.109 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-scrub-repair/1 2024-09-15T20:44:52.109 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=ce4892c7-6c6d-4b4f-8e6f-07a3e11133e5 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-15T20:44:52.109 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T20:44:52.109 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T20:44:52.109 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T20:44:52.109 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-09-15T20:44:52.109 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-09-15T20:44:52.109 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-15T20:44:52.109 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-15T20:44:52.109 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-15T20:44:52.109 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-15T20:44:52.110 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-15T20:44:52.110 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-15T20:44:52.110 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-15T20:44:52.110 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T20:44:52.110 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52602 2024-09-15T20:44:52.110 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.52602/$cluster-$name.asok' 2024-09-15T20:44:52.110 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.52602/$cluster-$name.asok' 2024-09-15T20:44:52.110 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-15T20:44:52.110 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-15T20:44:52.110 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-15T20:44:52.110 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-15T20:44:52.110 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-15T20:44:52.110 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T20:44:52.110 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T20:44:52.111 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T20:44:52.111 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T20:44:52.111 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-15T20:44:52.111 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+='--osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0' 2024-09-15T20:44:52.111 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-scrub-repair/1 2024-09-15T20:44:52.111 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-15T20:44:52.111 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=0c7c4a12-04ee-4cd1-8f9d-8c94bcbf4510 2024-09-15T20:44:52.111 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd1 0c7c4a12-04ee-4cd1-8f9d-8c94bcbf4510' 2024-09-15T20:44:52.111 INFO:tasks.workunit.client.0.smithi136.stdout:add osd1 0c7c4a12-04ee-4cd1-8f9d-8c94bcbf4510 2024-09-15T20:44:52.112 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-15T20:44:52.127 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQBER+dmozRtBxAA6+STuGpn/WWWHetX+E5cOQ== 2024-09-15T20:44:52.127 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQBER+dmozRtBxAA6+STuGpn/WWWHetX+E5cOQ=="}' 2024-09-15T20:44:52.127 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 0c7c4a12-04ee-4cd1-8f9d-8c94bcbf4510 -i td/osd-scrub-repair/1/new.json 2024-09-15T20:44:52.483 INFO:tasks.workunit.client.0.smithi136.stdout:1 2024-09-15T20:44:52.494 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-scrub-repair/1/new.json 2024-09-15T20:44:52.495 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 1 --fsid=ce4892c7-6c6d-4b4f-8e6f-07a3e11133e5 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.52602/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0 --mkfs --key AQBER+dmozRtBxAA6+STuGpn/WWWHetX+E5cOQ== --osd-uuid 0c7c4a12-04ee-4cd1-8f9d-8c94bcbf4510 2024-09-15T20:44:52.527 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:44:52.525+0000 7f07025bd640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T20:44:52.559 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:44:52.557+0000 7f07025bd640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T20:44:52.560 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:44:52.558+0000 7f07025bd640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T20:44:52.561 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:44:52.558+0000 7f07025bd640 -1 bdev(0x562b88d37c00 td/osd-scrub-repair/1/block) open stat got: (1) Operation not permitted 2024-09-15T20:44:52.561 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:44:52.559+0000 7f07025bd640 -1 bluestore(td/osd-scrub-repair/1) _read_fsid unparsable uuid 2024-09-15T20:44:59.215 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-scrub-repair/1/keyring 2024-09-15T20:44:59.215 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-15T20:44:59.216 INFO:tasks.workunit.client.0.smithi136.stdout:adding osd1 key to auth repository 2024-09-15T20:44:59.216 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd1 key to auth repository 2024-09-15T20:44:59.216 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-scrub-repair/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-15T20:44:59.579 INFO:tasks.workunit.client.0.smithi136.stdout:start osd.1 2024-09-15T20:44:59.580 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.1 2024-09-15T20:44:59.580 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 1 --fsid=ce4892c7-6c6d-4b4f-8e6f-07a3e11133e5 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.52602/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0 2024-09-15T20:44:59.580 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-15T20:44:59.580 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-15T20:44:59.580 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-15T20:44:59.610 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:44:59.608+0000 7f769c1cc640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T20:44:59.612 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:44:59.610+0000 7f769c1cc640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T20:44:59.613 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:44:59.611+0000 7f769c1cc640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T20:44:59.856 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 1 2024-09-15T20:44:59.857 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-15T20:44:59.857 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2024-09-15T20:44:59.857 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-15T20:44:59.857 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-15T20:44:59.857 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:44:59.857 INFO:tasks.workunit.client.0.smithi136.stdout:0 2024-09-15T20:44:59.858 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-15T20:44:59.858 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:44:59.858 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T20:45:00.137 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:45:00.741 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:45:00.739+0000 7f769c1cc640 -1 Falling back to public interface 2024-09-15T20:45:01.138 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:45:01.139 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:45:01.139 INFO:tasks.workunit.client.0.smithi136.stdout:1 2024-09-15T20:45:01.139 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-15T20:45:01.139 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:45:01.139 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T20:45:01.423 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:45:02.424 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:45:02.424 INFO:tasks.workunit.client.0.smithi136.stdout:2 2024-09-15T20:45:02.425 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:45:02.425 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-15T20:45:02.425 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:45:02.425 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T20:45:02.711 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:45:03.620 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:45:03.618+0000 7f769c1cc640 -1 osd.1 0 log_to_monitors true 2024-09-15T20:45:03.712 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:45:03.712 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:45:03.713 INFO:tasks.workunit.client.0.smithi136.stdout:3 2024-09-15T20:45:03.713 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-15T20:45:03.713 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:45:03.713 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T20:45:04.097 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:45:05.098 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:45:05.098 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:45:05.098 INFO:tasks.workunit.client.0.smithi136.stdout:4 2024-09-15T20:45:05.099 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-15T20:45:05.099 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:45:05.099 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T20:45:05.412 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:45:06.414 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:45:06.414 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:45:06.414 INFO:tasks.workunit.client.0.smithi136.stdout:5 2024-09-15T20:45:06.415 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-15T20:45:06.415 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:45:06.415 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T20:45:06.966 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:45:07.968 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:45:07.968 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:45:07.968 INFO:tasks.workunit.client.0.smithi136.stdout:6 2024-09-15T20:45:07.969 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-15T20:45:07.969 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:45:07.969 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T20:45:08.339 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:45:09.340 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:45:09.340 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:45:09.340 INFO:tasks.workunit.client.0.smithi136.stdout:7 2024-09-15T20:45:09.341 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-15T20:45:09.341 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:45:09.341 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T20:45:09.636 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:45:10.637 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:45:10.638 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:45:10.638 INFO:tasks.workunit.client.0.smithi136.stdout:8 2024-09-15T20:45:10.638 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-15T20:45:10.638 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:45:10.639 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T20:45:10.924 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:45:11.925 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:45:11.925 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:45:11.926 INFO:tasks.workunit.client.0.smithi136.stdout:9 2024-09-15T20:45:11.926 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-15T20:45:11.926 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:45:11.926 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T20:45:12.217 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:45:13.218 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:45:13.218 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:45:13.218 INFO:tasks.workunit.client.0.smithi136.stdout:10 2024-09-15T20:45:13.219 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-15T20:45:13.219 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:45:13.219 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T20:45:13.604 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:45:14.606 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:45:14.606 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:45:14.606 INFO:tasks.workunit.client.0.smithi136.stdout:11 2024-09-15T20:45:14.606 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-15T20:45:14.607 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:45:14.607 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T20:45:14.896 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:45:15.897 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:45:15.898 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:45:15.898 INFO:tasks.workunit.client.0.smithi136.stdout:12 2024-09-15T20:45:15.898 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-15T20:45:15.898 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:45:15.899 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T20:45:16.346 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:45:17.347 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:45:17.347 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:45:17.347 INFO:tasks.workunit.client.0.smithi136.stdout:13 2024-09-15T20:45:17.348 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-15T20:45:17.348 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:45:17.348 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T20:45:17.926 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:45:18.929 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:45:19.019 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:45:19.020 INFO:tasks.workunit.client.0.smithi136.stdout:14 2024-09-15T20:45:19.020 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-15T20:45:19.020 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:45:19.020 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T20:45:19.880 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:45:20.884 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:45:20.884 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:45:20.885 INFO:tasks.workunit.client.0.smithi136.stdout:15 2024-09-15T20:45:20.885 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-15T20:45:20.885 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:45:20.885 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T20:45:21.186 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:45:22.188 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:45:22.188 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:45:22.188 INFO:tasks.workunit.client.0.smithi136.stdout:16 2024-09-15T20:45:22.189 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-15T20:45:22.189 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:45:22.189 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T20:45:22.973 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:45:23.974 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:45:23.974 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:45:23.974 INFO:tasks.workunit.client.0.smithi136.stdout:17 2024-09-15T20:45:23.975 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-15T20:45:23.975 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:45:23.975 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T20:45:24.788 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:45:25.786 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:45:25.786 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:45:25.786 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 18 2024-09-15T20:45:25.787 INFO:tasks.workunit.client.0.smithi136.stdout:18 2024-09-15T20:45:25.787 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:45:25.787 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T20:45:26.214 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:45:27.215 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:45:27.216 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:45:27.216 INFO:tasks.workunit.client.0.smithi136.stdout:19 2024-09-15T20:45:27.217 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 19 2024-09-15T20:45:27.217 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:45:27.217 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T20:45:27.501 INFO:tasks.workunit.client.0.smithi136.stdout:osd.1 up in weight 1 up_from 16 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/4004705055,v1:127.0.0.1:6811/4004705055] [v2:127.0.0.1:6812/4004705055,v1:127.0.0.1:6813/4004705055] exists,up 0c7c4a12-04ee-4cd1-8f9d-8c94bcbf4510 2024-09-15T20:45:27.506 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-15T20:45:27.507 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-15T20:45:27.507 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-15T20:45:27.507 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:280: auto_repair_erasure_coded: for id in $(seq 0 2) 2024-09-15T20:45:27.507 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:281: auto_repair_erasure_coded: run_osd td/osd-scrub-repair 2 --osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0 2024-09-15T20:45:27.507 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-scrub-repair 2024-09-15T20:45:27.507 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-15T20:45:27.507 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=2 2024-09-15T20:45:27.507 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-15T20:45:27.507 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-scrub-repair/2 2024-09-15T20:45:27.507 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=ce4892c7-6c6d-4b4f-8e6f-07a3e11133e5 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-15T20:45:27.508 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T20:45:27.508 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T20:45:27.508 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T20:45:27.508 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/2' 2024-09-15T20:45:27.508 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/2/journal' 2024-09-15T20:45:27.508 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-15T20:45:27.508 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-15T20:45:27.508 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-15T20:45:27.508 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-15T20:45:27.508 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-15T20:45:27.509 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-15T20:45:27.509 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-15T20:45:27.509 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T20:45:27.509 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52602 2024-09-15T20:45:27.509 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.52602/$cluster-$name.asok' 2024-09-15T20:45:27.509 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.52602/$cluster-$name.asok' 2024-09-15T20:45:27.509 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-15T20:45:27.509 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-15T20:45:27.509 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-15T20:45:27.509 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-15T20:45:27.509 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-15T20:45:27.509 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T20:45:27.509 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T20:45:27.509 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T20:45:27.510 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T20:45:27.510 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-15T20:45:27.510 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+='--osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0' 2024-09-15T20:45:27.510 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-scrub-repair/2 2024-09-15T20:45:27.510 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-15T20:45:27.510 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=b4e09345-0db6-43cb-8dfc-2c64a96cd5f6 2024-09-15T20:45:27.510 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd2 b4e09345-0db6-43cb-8dfc-2c64a96cd5f6' 2024-09-15T20:45:27.510 INFO:tasks.workunit.client.0.smithi136.stdout:add osd2 b4e09345-0db6-43cb-8dfc-2c64a96cd5f6 2024-09-15T20:45:27.511 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-15T20:45:27.523 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQBnR+dm5qMQHxAA3ft4Vh7SkPq26pLqW8ytvw== 2024-09-15T20:45:27.523 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQBnR+dm5qMQHxAA3ft4Vh7SkPq26pLqW8ytvw=="}' 2024-09-15T20:45:27.523 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new b4e09345-0db6-43cb-8dfc-2c64a96cd5f6 -i td/osd-scrub-repair/2/new.json 2024-09-15T20:45:27.880 INFO:tasks.workunit.client.0.smithi136.stdout:2 2024-09-15T20:45:27.891 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-scrub-repair/2/new.json 2024-09-15T20:45:27.892 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 2 --fsid=ce4892c7-6c6d-4b4f-8e6f-07a3e11133e5 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.52602/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0 --mkfs --key AQBnR+dm5qMQHxAA3ft4Vh7SkPq26pLqW8ytvw== --osd-uuid b4e09345-0db6-43cb-8dfc-2c64a96cd5f6 2024-09-15T20:45:27.925 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:45:27.922+0000 7fbfe1cd8640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T20:45:27.964 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:45:27.962+0000 7fbfe1cd8640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T20:45:27.965 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:45:27.963+0000 7fbfe1cd8640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T20:45:27.966 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:45:27.963+0000 7fbfe1cd8640 -1 bdev(0x55bf57bc1c00 td/osd-scrub-repair/2/block) open stat got: (1) Operation not permitted 2024-09-15T20:45:27.966 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:45:27.964+0000 7fbfe1cd8640 -1 bluestore(td/osd-scrub-repair/2) _read_fsid unparsable uuid 2024-09-15T20:45:34.422 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-scrub-repair/2/keyring 2024-09-15T20:45:34.422 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-15T20:45:34.424 INFO:tasks.workunit.client.0.smithi136.stdout:adding osd2 key to auth repository 2024-09-15T20:45:34.424 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd2 key to auth repository 2024-09-15T20:45:34.424 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-scrub-repair/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-15T20:45:34.798 INFO:tasks.workunit.client.0.smithi136.stdout:start osd.2 2024-09-15T20:45:34.799 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.2 2024-09-15T20:45:34.799 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 2 --fsid=ce4892c7-6c6d-4b4f-8e6f-07a3e11133e5 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.52602/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0 2024-09-15T20:45:34.799 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-15T20:45:34.799 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-15T20:45:34.799 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-15T20:45:34.829 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:45:34.825+0000 7fb132145640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T20:45:34.830 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:45:34.827+0000 7fb132145640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T20:45:34.831 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:45:34.828+0000 7fb132145640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T20:45:35.076 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 2 2024-09-15T20:45:35.076 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-15T20:45:35.076 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=2 2024-09-15T20:45:35.077 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-15T20:45:35.077 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-15T20:45:35.077 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:45:35.077 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-15T20:45:35.077 INFO:tasks.workunit.client.0.smithi136.stdout:0 2024-09-15T20:45:35.078 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:45:35.078 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T20:45:35.354 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:45:36.356 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:45:36.356 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:45:36.356 INFO:tasks.workunit.client.0.smithi136.stdout:1 2024-09-15T20:45:36.357 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-15T20:45:36.357 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:45:36.357 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T20:45:36.441 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:45:36.439+0000 7fb132145640 -1 Falling back to public interface 2024-09-15T20:45:36.659 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:45:37.661 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:45:37.661 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:45:37.662 INFO:tasks.workunit.client.0.smithi136.stdout:2 2024-09-15T20:45:37.662 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-15T20:45:37.663 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:45:37.663 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T20:45:37.949 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:45:38.950 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:45:38.950 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:45:38.950 INFO:tasks.workunit.client.0.smithi136.stdout:3 2024-09-15T20:45:38.951 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-15T20:45:38.951 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:45:38.951 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T20:45:39.234 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:45:39.882 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:45:39.880+0000 7fb132145640 -1 osd.2 0 log_to_monitors true 2024-09-15T20:45:40.236 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:45:40.236 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:45:40.236 INFO:tasks.workunit.client.0.smithi136.stdout:4 2024-09-15T20:45:40.237 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-15T20:45:40.237 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:45:40.237 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T20:45:40.528 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:45:41.529 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:45:41.530 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:45:41.530 INFO:tasks.workunit.client.0.smithi136.stdout:5 2024-09-15T20:45:41.530 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-15T20:45:41.530 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:45:41.530 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T20:45:42.026 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:45:43.028 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:45:43.028 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:45:43.028 INFO:tasks.workunit.client.0.smithi136.stdout:6 2024-09-15T20:45:43.029 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-15T20:45:43.029 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:45:43.029 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T20:45:43.316 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:45:44.318 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:45:44.318 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:45:44.318 INFO:tasks.workunit.client.0.smithi136.stdout:7 2024-09-15T20:45:44.318 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-15T20:45:44.319 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:45:44.319 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T20:45:44.684 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:45:45.685 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:45:45.686 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:45:45.686 INFO:tasks.workunit.client.0.smithi136.stdout:8 2024-09-15T20:45:45.686 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-15T20:45:45.686 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:45:45.687 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T20:45:45.977 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:45:46.978 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:45:46.978 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:45:46.979 INFO:tasks.workunit.client.0.smithi136.stdout:9 2024-09-15T20:45:46.979 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-15T20:45:46.979 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:45:46.979 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T20:45:47.264 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:45:48.265 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:45:48.265 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:45:48.266 INFO:tasks.workunit.client.0.smithi136.stdout:10 2024-09-15T20:45:48.266 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-15T20:45:48.266 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:45:48.266 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T20:45:48.556 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:45:49.557 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:45:49.557 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:45:49.558 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-15T20:45:49.558 INFO:tasks.workunit.client.0.smithi136.stdout:11 2024-09-15T20:45:49.558 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:45:49.558 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T20:45:49.940 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:45:50.942 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:45:50.942 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:45:50.942 INFO:tasks.workunit.client.0.smithi136.stdout:12 2024-09-15T20:45:50.942 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-15T20:45:50.943 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:45:50.943 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T20:45:51.263 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:45:52.264 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:45:52.264 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:45:52.264 INFO:tasks.workunit.client.0.smithi136.stdout:13 2024-09-15T20:45:52.265 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-15T20:45:52.265 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:45:52.265 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T20:45:52.934 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:45:53.936 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:45:53.936 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:45:53.936 INFO:tasks.workunit.client.0.smithi136.stdout:14 2024-09-15T20:45:53.937 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-15T20:45:53.937 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:45:53.937 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T20:45:54.445 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:45:55.446 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:45:55.447 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:45:55.447 INFO:tasks.workunit.client.0.smithi136.stdout:15 2024-09-15T20:45:55.447 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-15T20:45:55.448 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:45:55.448 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T20:45:56.042 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:45:57.044 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:45:57.044 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:45:57.044 INFO:tasks.workunit.client.0.smithi136.stdout:16 2024-09-15T20:45:57.044 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-15T20:45:57.045 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:45:57.045 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T20:45:57.355 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:45:58.357 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:45:58.357 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:45:58.357 INFO:tasks.workunit.client.0.smithi136.stdout:17 2024-09-15T20:45:58.358 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-15T20:45:58.358 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:45:58.358 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T20:45:58.644 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:45:59.646 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:45:59.646 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:45:59.646 INFO:tasks.workunit.client.0.smithi136.stdout:18 2024-09-15T20:45:59.646 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 18 2024-09-15T20:45:59.646 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:45:59.647 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T20:46:00.196 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:46:01.197 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:46:01.198 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:46:01.198 INFO:tasks.workunit.client.0.smithi136.stdout:19 2024-09-15T20:46:01.198 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 19 2024-09-15T20:46:01.198 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:46:01.199 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T20:46:01.775 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:46:02.776 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:46:02.777 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:46:02.777 INFO:tasks.workunit.client.0.smithi136.stdout:20 2024-09-15T20:46:02.777 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 20 2024-09-15T20:46:02.777 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:46:02.778 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T20:46:03.172 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:46:04.173 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:46:04.304 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:46:04.304 INFO:tasks.workunit.client.0.smithi136.stdout:21 2024-09-15T20:46:04.305 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 21 2024-09-15T20:46:04.305 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:46:04.305 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T20:46:04.458 INFO:tasks.workunit.client.0.smithi136.stdout:osd.2 up in weight 1 up_from 35 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/3179894099,v1:127.0.0.1:6819/3179894099] [v2:127.0.0.1:6820/3179894099,v1:127.0.0.1:6821/3179894099] exists,up b4e09345-0db6-43cb-8dfc-2c64a96cd5f6 2024-09-15T20:46:04.459 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-15T20:46:04.459 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-15T20:46:04.459 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-15T20:46:04.459 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:283: auto_repair_erasure_coded: create_rbd_pool 2024-09-15T20:46:04.459 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:526: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2024-09-15T20:46:04.725 INFO:tasks.workunit.client.0.smithi136.stderr:pool 'rbd' does not exist 2024-09-15T20:46:04.737 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:527: create_rbd_pool: create_pool rbd 4 2024-09-15T20:46:04.737 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create rbd 4 2024-09-15T20:46:05.321 INFO:tasks.workunit.client.0.smithi136.stderr:pool 'rbd' created 2024-09-15T20:46:05.333 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2024-09-15T20:46:06.334 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:528: create_rbd_pool: rbd pool init rbd 2024-09-15T20:46:08.753 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:284: auto_repair_erasure_coded: wait_for_clean 2024-09-15T20:46:08.753 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-15T20:46:08.753 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-15T20:46:08.753 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-15T20:46:08.754 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-15T20:46:08.754 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-15T20:46:08.754 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-15T20:46:08.754 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-15T20:46:08.755 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-15T20:46:08.755 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-15T20:46:08.854 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-15T20:46:08.854 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-15T20:46:08.854 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-15T20:46:08.854 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-15T20:46:08.854 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-15T20:46:08.854 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-15T20:46:09.198 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-15T20:46:09.198 INFO:tasks.workunit.client.0.smithi136.stderr:1 2024-09-15T20:46:09.198 INFO:tasks.workunit.client.0.smithi136.stderr:2' 2024-09-15T20:46:09.198 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-15T20:46:09.198 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T20:46:09.198 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-15T20:46:09.303 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836501 2024-09-15T20:46:09.303 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836501 2024-09-15T20:46:09.303 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836501' 2024-09-15T20:46:09.303 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T20:46:09.303 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-15T20:46:09.406 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=68719476752 2024-09-15T20:46:09.407 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 68719476752 2024-09-15T20:46:09.407 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836501 1-68719476752' 2024-09-15T20:46:09.407 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T20:46:09.407 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-15T20:46:09.510 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=150323855364 2024-09-15T20:46:09.510 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 150323855364 2024-09-15T20:46:09.510 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836501 1-68719476752 2-150323855364' 2024-09-15T20:46:09.510 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T20:46:09.510 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836501 2024-09-15T20:46:09.510 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T20:46:09.511 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-15T20:46:09.511 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836501 2024-09-15T20:46:09.511 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T20:46:09.512 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836501 2024-09-15T20:46:09.513 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836501' 2024-09-15T20:46:09.513 INFO:tasks.workunit.client.0.smithi136.stdout:waiting osd.0 seq 21474836501 2024-09-15T20:46:09.513 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T20:46:09.789 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836500 -lt 21474836501 2024-09-15T20:46:09.789 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-15T20:46:10.790 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-15T20:46:10.790 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T20:46:11.075 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836501 -lt 21474836501 2024-09-15T20:46:11.075 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T20:46:11.075 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-68719476752 2024-09-15T20:46:11.075 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T20:46:11.076 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-15T20:46:11.076 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-68719476752 2024-09-15T20:46:11.077 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T20:46:11.077 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=68719476752 2024-09-15T20:46:11.078 INFO:tasks.workunit.client.0.smithi136.stdout:waiting osd.1 seq 68719476752 2024-09-15T20:46:11.078 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 68719476752' 2024-09-15T20:46:11.078 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-15T20:46:11.356 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 68719476752 -lt 68719476752 2024-09-15T20:46:11.357 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T20:46:11.357 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-150323855364 2024-09-15T20:46:11.357 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T20:46:11.358 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-15T20:46:11.358 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-150323855364 2024-09-15T20:46:11.359 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T20:46:11.359 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=150323855364 2024-09-15T20:46:11.359 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 150323855364' 2024-09-15T20:46:11.360 INFO:tasks.workunit.client.0.smithi136.stdout:waiting osd.2 seq 150323855364 2024-09-15T20:46:11.360 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-15T20:46:11.639 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 150323855365 -lt 150323855364 2024-09-15T20:46:11.640 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-15T20:46:11.640 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T20:46:11.640 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T20:46:12.028 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 4 == 0 2024-09-15T20:46:12.028 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-15T20:46:12.030 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-15T20:46:12.030 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-15T20:46:12.030 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T20:46:12.030 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T20:46:12.030 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T20:46:12.030 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T20:46:12.304 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=4 2024-09-15T20:46:12.304 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-15T20:46:12.305 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T20:46:12.305 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T20:46:12.653 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 = 4 2024-09-15T20:46:12.653 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-15T20:46:12.654 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-15T20:46:12.654 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:287: auto_repair_erasure_coded: create_ec_pool ecpool true k=2 m=1 2024-09-15T20:46:12.654 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2443: create_ec_pool: local pool_name=ecpool 2024-09-15T20:46:12.654 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2444: create_ec_pool: shift 2024-09-15T20:46:12.654 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2445: create_ec_pool: local allow_overwrites=true 2024-09-15T20:46:12.654 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2446: create_ec_pool: shift 2024-09-15T20:46:12.655 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2448: create_ec_pool: ceph osd erasure-code-profile set myprofile crush-failure-domain=osd k=2 m=1 2024-09-15T20:46:13.038 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2450: create_ec_pool: create_pool ecpool 1 1 erasure myprofile 2024-09-15T20:46:13.038 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create ecpool 1 1 erasure myprofile 2024-09-15T20:46:13.622 INFO:tasks.workunit.client.0.smithi136.stderr:pool 'ecpool' created 2024-09-15T20:46:13.633 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2024-09-15T20:46:14.635 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2452: create_ec_pool: '[' true = true ']' 2024-09-15T20:46:14.635 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2453: create_ec_pool: ceph osd pool set ecpool allow_ec_overwrites true 2024-09-15T20:46:15.305 INFO:tasks.workunit.client.0.smithi136.stderr:set pool 2 allow_ec_overwrites to true 2024-09-15T20:46:15.318 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2456: create_ec_pool: wait_for_clean 2024-09-15T20:46:15.318 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-15T20:46:15.318 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-15T20:46:15.318 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-15T20:46:15.318 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-15T20:46:15.318 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-15T20:46:15.318 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-15T20:46:15.318 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-15T20:46:15.319 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-15T20:46:15.319 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-15T20:46:15.407 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-15T20:46:15.407 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-15T20:46:15.407 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-15T20:46:15.407 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-15T20:46:15.407 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-15T20:46:15.407 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-15T20:46:15.691 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-15T20:46:15.691 INFO:tasks.workunit.client.0.smithi136.stderr:1 2024-09-15T20:46:15.691 INFO:tasks.workunit.client.0.smithi136.stderr:2' 2024-09-15T20:46:15.692 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-15T20:46:15.692 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T20:46:15.692 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-15T20:46:15.819 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836503 2024-09-15T20:46:15.819 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836503 2024-09-15T20:46:15.819 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836503' 2024-09-15T20:46:15.819 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T20:46:15.819 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-15T20:46:15.922 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=68719476754 2024-09-15T20:46:15.922 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 68719476754 2024-09-15T20:46:15.922 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836503 1-68719476754' 2024-09-15T20:46:15.922 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T20:46:15.922 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-15T20:46:16.025 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=150323855367 2024-09-15T20:46:16.025 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 150323855367 2024-09-15T20:46:16.025 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836503 1-68719476754 2-150323855367' 2024-09-15T20:46:16.025 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T20:46:16.025 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836503 2024-09-15T20:46:16.025 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T20:46:16.026 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-15T20:46:16.026 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836503 2024-09-15T20:46:16.026 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T20:46:16.027 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836503 2024-09-15T20:46:16.027 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836503' 2024-09-15T20:46:16.028 INFO:tasks.workunit.client.0.smithi136.stdout:waiting osd.0 seq 21474836503 2024-09-15T20:46:16.028 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T20:46:16.331 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836503 -lt 21474836503 2024-09-15T20:46:16.332 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T20:46:16.332 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-68719476754 2024-09-15T20:46:16.332 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T20:46:16.332 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-15T20:46:16.333 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-68719476754 2024-09-15T20:46:16.333 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T20:46:16.334 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=68719476754 2024-09-15T20:46:16.334 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 68719476754' 2024-09-15T20:46:16.335 INFO:tasks.workunit.client.0.smithi136.stdout:waiting osd.1 seq 68719476754 2024-09-15T20:46:16.335 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-15T20:46:16.642 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 68719476754 -lt 68719476754 2024-09-15T20:46:16.642 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T20:46:16.642 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-150323855367 2024-09-15T20:46:16.642 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T20:46:16.643 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-15T20:46:16.644 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-150323855367 2024-09-15T20:46:16.644 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T20:46:16.645 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=150323855367 2024-09-15T20:46:16.645 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 150323855367' 2024-09-15T20:46:16.645 INFO:tasks.workunit.client.0.smithi136.stdout:waiting osd.2 seq 150323855367 2024-09-15T20:46:16.646 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-15T20:46:16.921 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 150323855367 -lt 150323855367 2024-09-15T20:46:16.921 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-15T20:46:16.922 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T20:46:16.922 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T20:46:17.266 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 5 == 0 2024-09-15T20:46:17.266 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-15T20:46:17.266 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-15T20:46:17.266 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-15T20:46:17.266 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T20:46:17.266 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T20:46:17.266 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T20:46:17.267 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T20:46:17.540 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=5 2024-09-15T20:46:17.541 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-15T20:46:17.541 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T20:46:17.541 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T20:46:17.925 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 = 5 2024-09-15T20:46:17.925 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-15T20:46:17.926 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-15T20:46:17.926 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2457: create_ec_pool: return 0 2024-09-15T20:46:17.926 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:290: auto_repair_erasure_coded: local payload=ABCDEF 2024-09-15T20:46:17.926 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:291: auto_repair_erasure_coded: echo ABCDEF 2024-09-15T20:46:17.926 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:292: auto_repair_erasure_coded: rados --pool ecpool put SOMETHING td/osd-scrub-repair/ORIGINAL 2024-09-15T20:46:18.252 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:296: auto_repair_erasure_coded: get_not_primary ecpool SOMETHING 2024-09-15T20:46:18.252 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1220: get_not_primary: local poolname=ecpool 2024-09-15T20:46:18.253 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1221: get_not_primary: local objectname=SOMETHING 2024-09-15T20:46:18.253 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1223: get_not_primary: get_primary ecpool SOMETHING 2024-09-15T20:46:18.253 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1187: get_primary: local poolname=ecpool 2024-09-15T20:46:18.253 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1188: get_primary: local objectname=SOMETHING 2024-09-15T20:46:18.253 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1190: get_primary: ceph --format json osd map ecpool SOMETHING 2024-09-15T20:46:18.254 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1191: get_primary: jq .acting_primary 2024-09-15T20:46:18.580 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1223: get_not_primary: local primary=2 2024-09-15T20:46:18.580 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1224: get_not_primary: ceph --format json osd map ecpool SOMETHING 2024-09-15T20:46:18.580 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1225: get_not_primary: jq '.acting | map(select (. != 2)) | .[0]' 2024-09-15T20:46:18.881 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:296: auto_repair_erasure_coded: objectstore_tool td/osd-scrub-repair 1 SOMETHING remove 2024-09-15T20:46:18.881 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2024-09-15T20:46:18.881 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2024-09-15T20:46:18.881 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=1 2024-09-15T20:46:18.881 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2024-09-15T20:46:18.882 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 SOMETHING remove 2024-09-15T20:46:18.882 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-09-15T20:46:18.882 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2024-09-15T20:46:18.882 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=1 2024-09-15T20:46:18.882 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2024-09-15T20:46:18.882 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-09-15T20:46:18.882 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-15T20:46:18.882 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-15T20:46:18.882 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-15T20:46:18.882 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-15T20:46:18.882 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-15T20:46:20.189 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-15T20:46:20.189 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 SOMETHING remove 2024-09-15T20:46:20.189 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-15T20:46:20.189 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-15T20:46:20.189 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=1 2024-09-15T20:46:20.189 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-15T20:46:20.189 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-09-15T20:46:20.189 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 SOMETHING remove 2024-09-15T20:46:21.608 INFO:tasks.workunit.client.0.smithi136.stderr:Error getting attr on : 1.3_head,#-3:c0000000:::scrub_1.3:head#, (61) No data available 2024-09-15T20:46:21.608 INFO:tasks.workunit.client.0.smithi136.stderr:Error getting attr on : 1.0_head,#-3:00000000:::scrub_1.0:head#, (61) No data available 2024-09-15T20:46:21.609 INFO:tasks.workunit.client.0.smithi136.stdout:remove 1#2:eb822e21:::SOMETHING:head# 2024-09-15T20:46:22.130 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 --osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0 2024-09-15T20:46:22.130 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2024-09-15T20:46:22.130 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-15T20:46:22.130 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=1 2024-09-15T20:46:22.130 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-15T20:46:22.131 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-09-15T20:46:22.131 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=ce4892c7-6c6d-4b4f-8e6f-07a3e11133e5 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-15T20:46:22.131 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T20:46:22.131 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T20:46:22.131 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T20:46:22.131 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-09-15T20:46:22.131 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-09-15T20:46:22.131 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-15T20:46:22.131 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-15T20:46:22.131 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-15T20:46:22.131 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-15T20:46:22.131 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-15T20:46:22.131 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-15T20:46:22.132 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-15T20:46:22.132 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T20:46:22.132 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52602 2024-09-15T20:46:22.132 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.52602/$cluster-$name.asok' 2024-09-15T20:46:22.132 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.52602/$cluster-$name.asok' 2024-09-15T20:46:22.132 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-15T20:46:22.132 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-15T20:46:22.132 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-15T20:46:22.132 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T20:46:22.132 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T20:46:22.132 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T20:46:22.132 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T20:46:22.132 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-15T20:46:22.132 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+='--osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0' 2024-09-15T20:46:22.133 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-09-15T20:46:22.133 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.1 2024-09-15T20:46:22.133 INFO:tasks.workunit.client.0.smithi136.stderr:start osd.1 2024-09-15T20:46:22.133 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 1 --fsid=ce4892c7-6c6d-4b4f-8e6f-07a3e11133e5 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.52602/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0 2024-09-15T20:46:22.133 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-09-15T20:46:22.133 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 1 = 1 ']' 2024-09-15T20:46:22.134 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-15T20:46:22.134 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-15T20:46:22.134 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-15T20:46:22.164 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:46:22.161+0000 7f450cbdd640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T20:46:22.167 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:46:22.163+0000 7f450cbdd640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T20:46:22.168 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:46:22.165+0000 7f450cbdd640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T20:46:22.418 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 1 2024-09-15T20:46:22.419 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-15T20:46:22.419 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2024-09-15T20:46:22.419 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-15T20:46:22.419 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-15T20:46:22.419 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:46:22.419 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-15T20:46:22.419 INFO:tasks.workunit.client.0.smithi136.stderr:0 2024-09-15T20:46:22.419 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:46:22.419 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T20:46:22.712 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:46:23.512 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:46:23.510+0000 7f450cbdd640 -1 Falling back to public interface 2024-09-15T20:46:23.713 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:46:23.713 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:46:23.713 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-15T20:46:23.714 INFO:tasks.workunit.client.0.smithi136.stderr:1 2024-09-15T20:46:23.714 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:46:23.714 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T20:46:24.001 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:46:25.002 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:46:25.003 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:46:25.003 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-15T20:46:25.003 INFO:tasks.workunit.client.0.smithi136.stderr:2 2024-09-15T20:46:25.003 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:46:25.003 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T20:46:25.287 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:46:26.288 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:46:26.288 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:46:26.288 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-15T20:46:26.288 INFO:tasks.workunit.client.0.smithi136.stderr:3 2024-09-15T20:46:26.288 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:46:26.289 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T20:46:26.582 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:46:26.580+0000 7f450cbdd640 -1 osd.1 56 log_to_monitors true 2024-09-15T20:46:26.594 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:46:27.596 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:46:27.596 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:46:27.596 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-15T20:46:27.596 INFO:tasks.workunit.client.0.smithi136.stderr:4 2024-09-15T20:46:27.596 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:46:27.597 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T20:46:27.896 INFO:tasks.workunit.client.0.smithi136.stderr:osd.1 up in weight 1 up_from 63 up_thru 63 down_at 57 last_clean_interval [16,56) [v2:127.0.0.1:6810/4156910076,v1:127.0.0.1:6811/4156910076] [v2:127.0.0.1:6812/4156910076,v1:127.0.0.1:6813/4156910076] exists,up 0c7c4a12-04ee-4cd1-8f9d-8c94bcbf4510 2024-09-15T20:46:27.896 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-15T20:46:27.896 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-15T20:46:27.896 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-15T20:46:27.896 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2024-09-15T20:46:27.896 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-15T20:46:27.896 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-15T20:46:27.896 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-15T20:46:27.896 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-15T20:46:27.896 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-15T20:46:27.896 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-15T20:46:27.897 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-15T20:46:27.897 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-15T20:46:27.897 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-15T20:46:27.985 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-15T20:46:27.985 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-15T20:46:27.985 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-15T20:46:27.985 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-15T20:46:27.985 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-15T20:46:27.985 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-15T20:46:28.397 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-15T20:46:28.397 INFO:tasks.workunit.client.0.smithi136.stderr:1 2024-09-15T20:46:28.397 INFO:tasks.workunit.client.0.smithi136.stderr:2' 2024-09-15T20:46:28.398 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-15T20:46:28.398 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T20:46:28.398 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-15T20:46:28.563 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836507 2024-09-15T20:46:28.563 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836507 2024-09-15T20:46:28.563 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836507' 2024-09-15T20:46:28.563 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T20:46:28.563 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-15T20:46:28.766 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=270582939650 2024-09-15T20:46:28.766 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 270582939650 2024-09-15T20:46:28.766 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836507 1-270582939650' 2024-09-15T20:46:28.766 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T20:46:28.767 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-15T20:46:28.870 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=150323855370 2024-09-15T20:46:28.870 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 150323855370 2024-09-15T20:46:28.870 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836507 1-270582939650 2-150323855370' 2024-09-15T20:46:28.870 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T20:46:28.871 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836507 2024-09-15T20:46:28.871 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T20:46:28.871 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-15T20:46:28.872 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836507 2024-09-15T20:46:28.872 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T20:46:28.873 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836507 2024-09-15T20:46:28.873 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836507' 2024-09-15T20:46:28.873 INFO:tasks.workunit.client.0.smithi136.stderr:waiting osd.0 seq 21474836507 2024-09-15T20:46:28.874 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T20:46:29.170 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836506 -lt 21474836507 2024-09-15T20:46:29.170 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-15T20:46:30.171 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-15T20:46:30.172 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T20:46:30.488 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836507 -lt 21474836507 2024-09-15T20:46:30.488 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T20:46:30.488 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-270582939650 2024-09-15T20:46:30.488 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T20:46:30.489 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-15T20:46:30.490 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-270582939650 2024-09-15T20:46:30.490 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T20:46:30.491 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=270582939650 2024-09-15T20:46:30.491 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 270582939650' 2024-09-15T20:46:30.491 INFO:tasks.workunit.client.0.smithi136.stderr:waiting osd.1 seq 270582939650 2024-09-15T20:46:30.491 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-15T20:46:30.796 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 270582939650 -lt 270582939650 2024-09-15T20:46:30.797 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T20:46:30.797 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-150323855370 2024-09-15T20:46:30.797 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T20:46:30.798 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-15T20:46:30.798 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-150323855370 2024-09-15T20:46:30.799 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T20:46:30.799 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=150323855370 2024-09-15T20:46:30.800 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 150323855370' 2024-09-15T20:46:30.800 INFO:tasks.workunit.client.0.smithi136.stderr:waiting osd.2 seq 150323855370 2024-09-15T20:46:30.800 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-15T20:46:31.093 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 150323855371 -lt 150323855370 2024-09-15T20:46:31.094 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-15T20:46:31.094 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T20:46:31.094 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T20:46:31.437 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 5 == 0 2024-09-15T20:46:31.437 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-15T20:46:31.437 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-15T20:46:31.437 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-15T20:46:31.437 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T20:46:31.437 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T20:46:31.437 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T20:46:31.437 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T20:46:31.711 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=5 2024-09-15T20:46:31.712 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-15T20:46:31.712 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T20:46:31.712 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T20:46:32.056 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 = 5 2024-09-15T20:46:32.056 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-15T20:46:32.057 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-15T20:46:32.057 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:298: auto_repair_erasure_coded: get_pg ecpool SOMETHING 2024-09-15T20:46:32.057 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1084: get_pg: local poolname=ecpool 2024-09-15T20:46:32.057 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1085: get_pg: local objectname=SOMETHING 2024-09-15T20:46:32.057 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1087: get_pg: ceph --format json osd map ecpool SOMETHING 2024-09-15T20:46:32.057 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1087: get_pg: jq -r .pgid 2024-09-15T20:46:32.337 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:298: auto_repair_erasure_coded: local pgid=2.0 2024-09-15T20:46:32.337 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:299: auto_repair_erasure_coded: get_last_scrub_stamp 2.0 2024-09-15T20:46:32.337 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2024-09-15T20:46:32.337 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-15T20:46:32.337 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-15T20:46:32.337 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-09-15T20:46:32.610 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:299: auto_repair_erasure_coded: wait_for_scrub 2.0 2024-09-15T20:46:18.554147+0000 2024-09-15T20:46:32.611 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: local pgid=2.0 2024-09-15T20:46:32.611 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2018: wait_for_scrub: local last_scrub=2024-09-15T20:46:18.554147+0000 2024-09-15T20:46:32.611 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2019: wait_for_scrub: local sname=last_scrub_stamp 2024-09-15T20:46:32.611 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i=0 )) 2024-09-15T20:46:32.611 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-15T20:46:32.611 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-09-15T20:46:32.611 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2024-09-15T20:46:32.611 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-15T20:46:32.611 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-15T20:46:32.611 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-09-15T20:46:32.887 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-15T20:46:18.554147+0000 '>' 2024-09-15T20:46:18.554147+0000 2024-09-15T20:46:32.888 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-15T20:46:33.888 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-15T20:46:33.889 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-15T20:46:33.889 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-09-15T20:46:33.889 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2024-09-15T20:46:33.890 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-15T20:46:33.890 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-15T20:46:33.890 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-09-15T20:46:34.172 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-15T20:46:18.554147+0000 '>' 2024-09-15T20:46:18.554147+0000 2024-09-15T20:46:34.173 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-15T20:46:35.174 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-15T20:46:35.174 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-15T20:46:35.174 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-09-15T20:46:35.175 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2024-09-15T20:46:35.175 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-15T20:46:35.175 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-15T20:46:35.175 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-09-15T20:46:35.456 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-15T20:46:18.554147+0000 '>' 2024-09-15T20:46:18.554147+0000 2024-09-15T20:46:35.457 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-15T20:46:36.458 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-15T20:46:36.458 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-15T20:46:36.458 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-09-15T20:46:36.459 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2024-09-15T20:46:36.459 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-15T20:46:36.459 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-15T20:46:36.459 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-09-15T20:46:36.741 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-15T20:46:18.554147+0000 '>' 2024-09-15T20:46:18.554147+0000 2024-09-15T20:46:36.741 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-15T20:46:37.742 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-15T20:46:37.742 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-15T20:46:37.743 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-09-15T20:46:37.743 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2024-09-15T20:46:37.743 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-15T20:46:37.743 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-15T20:46:37.743 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-09-15T20:46:38.028 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-15T20:46:18.554147+0000 '>' 2024-09-15T20:46:18.554147+0000 2024-09-15T20:46:38.028 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-15T20:46:39.029 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-15T20:46:39.029 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-15T20:46:39.030 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-09-15T20:46:39.030 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2024-09-15T20:46:39.030 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-15T20:46:39.030 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-15T20:46:39.030 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-09-15T20:46:39.313 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-15T20:46:18.554147+0000 '>' 2024-09-15T20:46:18.554147+0000 2024-09-15T20:46:39.313 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-15T20:46:40.314 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-15T20:46:40.314 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-15T20:46:40.315 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-09-15T20:46:40.315 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2024-09-15T20:46:40.315 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-15T20:46:40.315 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-15T20:46:40.315 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-09-15T20:46:40.595 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-15T20:46:34.726442+0000 '>' 2024-09-15T20:46:18.554147+0000 2024-09-15T20:46:40.596 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2023: wait_for_scrub: return 0 2024-09-15T20:46:40.596 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:300: auto_repair_erasure_coded: wait_for_clean 2024-09-15T20:46:40.596 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-15T20:46:40.596 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-15T20:46:40.596 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-15T20:46:40.596 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-15T20:46:40.596 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-15T20:46:40.596 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-15T20:46:40.596 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-15T20:46:40.597 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-15T20:46:40.597 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-15T20:46:40.685 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-15T20:46:40.685 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-15T20:46:40.685 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-15T20:46:40.685 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-15T20:46:40.686 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-15T20:46:40.686 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-15T20:46:41.006 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-15T20:46:41.006 INFO:tasks.workunit.client.0.smithi136.stderr:1 2024-09-15T20:46:41.006 INFO:tasks.workunit.client.0.smithi136.stderr:2' 2024-09-15T20:46:41.006 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-15T20:46:41.006 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T20:46:41.007 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-15T20:46:41.111 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836510 2024-09-15T20:46:41.111 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836510 2024-09-15T20:46:41.111 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836510' 2024-09-15T20:46:41.111 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T20:46:41.111 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-15T20:46:41.213 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=270582939653 2024-09-15T20:46:41.214 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 270582939653 2024-09-15T20:46:41.214 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836510 1-270582939653' 2024-09-15T20:46:41.214 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T20:46:41.214 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-15T20:46:41.316 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=150323855374 2024-09-15T20:46:41.316 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 150323855374 2024-09-15T20:46:41.316 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836510 1-270582939653 2-150323855374' 2024-09-15T20:46:41.316 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T20:46:41.317 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836510 2024-09-15T20:46:41.317 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T20:46:41.317 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-15T20:46:41.318 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836510 2024-09-15T20:46:41.318 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T20:46:41.319 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836510 2024-09-15T20:46:41.319 INFO:tasks.workunit.client.0.smithi136.stdout:waiting osd.0 seq 21474836510 2024-09-15T20:46:41.319 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836510' 2024-09-15T20:46:41.320 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T20:46:41.595 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836509 -lt 21474836510 2024-09-15T20:46:41.595 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-15T20:46:42.596 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-15T20:46:42.597 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T20:46:42.883 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836510 -lt 21474836510 2024-09-15T20:46:42.883 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T20:46:42.883 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-270582939653 2024-09-15T20:46:42.884 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T20:46:42.884 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-15T20:46:42.885 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-270582939653 2024-09-15T20:46:42.885 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T20:46:42.886 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=270582939653 2024-09-15T20:46:42.886 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 270582939653' 2024-09-15T20:46:42.886 INFO:tasks.workunit.client.0.smithi136.stdout:waiting osd.1 seq 270582939653 2024-09-15T20:46:42.886 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-15T20:46:43.163 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 270582939654 -lt 270582939653 2024-09-15T20:46:43.163 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T20:46:43.163 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-150323855374 2024-09-15T20:46:43.164 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T20:46:43.164 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-15T20:46:43.165 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-150323855374 2024-09-15T20:46:43.165 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T20:46:43.166 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=150323855374 2024-09-15T20:46:43.166 INFO:tasks.workunit.client.0.smithi136.stdout:waiting osd.2 seq 150323855374 2024-09-15T20:46:43.166 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 150323855374' 2024-09-15T20:46:43.166 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-15T20:46:43.445 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 150323855374 -lt 150323855374 2024-09-15T20:46:43.446 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-15T20:46:43.446 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T20:46:43.446 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T20:46:43.792 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 5 == 0 2024-09-15T20:46:43.792 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-15T20:46:43.792 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-15T20:46:43.792 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-15T20:46:43.792 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T20:46:43.793 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T20:46:43.793 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T20:46:43.793 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T20:46:44.073 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=5 2024-09-15T20:46:44.074 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-15T20:46:44.074 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T20:46:44.075 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T20:46:44.418 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 = 5 2024-09-15T20:46:44.419 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-15T20:46:44.419 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-15T20:46:44.419 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:303: auto_repair_erasure_coded: get_not_primary ecpool SOMETHING 2024-09-15T20:46:44.419 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1220: get_not_primary: local poolname=ecpool 2024-09-15T20:46:44.419 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1221: get_not_primary: local objectname=SOMETHING 2024-09-15T20:46:44.419 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1223: get_not_primary: get_primary ecpool SOMETHING 2024-09-15T20:46:44.420 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1187: get_primary: local poolname=ecpool 2024-09-15T20:46:44.420 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1188: get_primary: local objectname=SOMETHING 2024-09-15T20:46:44.420 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1190: get_primary: ceph --format json osd map ecpool SOMETHING 2024-09-15T20:46:44.420 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1191: get_primary: jq .acting_primary 2024-09-15T20:46:44.710 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1223: get_not_primary: local primary=2 2024-09-15T20:46:44.710 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1224: get_not_primary: ceph --format json osd map ecpool SOMETHING 2024-09-15T20:46:44.710 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1225: get_not_primary: jq '.acting | map(select (. != 2)) | .[0]' 2024-09-15T20:46:45.037 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:303: auto_repair_erasure_coded: objectstore_tool td/osd-scrub-repair 1 SOMETHING list-attrs 2024-09-15T20:46:45.037 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2024-09-15T20:46:45.037 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2024-09-15T20:46:45.037 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=1 2024-09-15T20:46:45.037 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2024-09-15T20:46:45.037 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 SOMETHING list-attrs 2024-09-15T20:46:45.037 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-09-15T20:46:45.037 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2024-09-15T20:46:45.037 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=1 2024-09-15T20:46:45.038 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2024-09-15T20:46:45.038 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-09-15T20:46:45.038 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-15T20:46:45.038 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-15T20:46:45.038 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-15T20:46:45.038 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-15T20:46:45.038 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-15T20:46:45.343 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-15T20:46:45.343 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 SOMETHING list-attrs 2024-09-15T20:46:45.343 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-15T20:46:45.343 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-15T20:46:45.344 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=1 2024-09-15T20:46:45.344 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-15T20:46:45.344 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-09-15T20:46:45.344 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 SOMETHING list-attrs 2024-09-15T20:46:45.764 INFO:tasks.workunit.client.0.smithi136.stderr:Error getting attr on : 1.3_head,#-3:c0000000:::scrub_1.3:head#, (61) No data available 2024-09-15T20:46:45.764 INFO:tasks.workunit.client.0.smithi136.stderr:Error getting attr on : 1.0_head,#-3:00000000:::scrub_1.0:head#, (61) No data available 2024-09-15T20:46:45.764 INFO:tasks.workunit.client.0.smithi136.stdout:_ 2024-09-15T20:46:45.764 INFO:tasks.workunit.client.0.smithi136.stdout:hinfo_key 2024-09-15T20:46:45.764 INFO:tasks.workunit.client.0.smithi136.stdout:snapset 2024-09-15T20:46:46.031 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 --osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0 2024-09-15T20:46:46.031 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2024-09-15T20:46:46.031 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-15T20:46:46.031 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=1 2024-09-15T20:46:46.031 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-15T20:46:46.031 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-09-15T20:46:46.031 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=ce4892c7-6c6d-4b4f-8e6f-07a3e11133e5 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-15T20:46:46.031 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T20:46:46.032 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T20:46:46.032 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T20:46:46.032 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-09-15T20:46:46.032 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-09-15T20:46:46.032 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-15T20:46:46.032 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-15T20:46:46.032 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-15T20:46:46.032 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-15T20:46:46.032 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-15T20:46:46.032 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-15T20:46:46.032 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-15T20:46:46.032 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T20:46:46.032 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52602 2024-09-15T20:46:46.033 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.52602/$cluster-$name.asok' 2024-09-15T20:46:46.033 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.52602/$cluster-$name.asok' 2024-09-15T20:46:46.033 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-15T20:46:46.033 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-15T20:46:46.033 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-15T20:46:46.033 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T20:46:46.033 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T20:46:46.033 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T20:46:46.033 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T20:46:46.033 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-15T20:46:46.033 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+='--osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0' 2024-09-15T20:46:46.033 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-09-15T20:46:46.033 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.1 2024-09-15T20:46:46.033 INFO:tasks.workunit.client.0.smithi136.stderr:start osd.1 2024-09-15T20:46:46.034 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 1 --fsid=ce4892c7-6c6d-4b4f-8e6f-07a3e11133e5 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.52602/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0 2024-09-15T20:46:46.034 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-09-15T20:46:46.034 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 1 = 1 ']' 2024-09-15T20:46:46.034 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-15T20:46:46.035 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-15T20:46:46.035 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-15T20:46:46.062 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:46:46.060+0000 7f231881a640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T20:46:46.064 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:46:46.062+0000 7f231881a640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T20:46:46.065 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:46:46.063+0000 7f231881a640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T20:46:46.475 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 1 2024-09-15T20:46:46.475 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-15T20:46:46.475 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2024-09-15T20:46:46.475 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-15T20:46:46.475 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-15T20:46:46.475 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:46:46.475 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-15T20:46:46.475 INFO:tasks.workunit.client.0.smithi136.stderr:0 2024-09-15T20:46:46.475 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:46:46.475 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T20:46:46.793 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:46:47.169 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:46:47.167+0000 7f231881a640 -1 Falling back to public interface 2024-09-15T20:46:47.795 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:46:47.795 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:46:47.795 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-15T20:46:47.795 INFO:tasks.workunit.client.0.smithi136.stderr:1 2024-09-15T20:46:47.795 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:46:47.795 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T20:46:48.083 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:46:49.085 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:46:49.085 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:46:49.085 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-15T20:46:49.085 INFO:tasks.workunit.client.0.smithi136.stderr:2 2024-09-15T20:46:49.085 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:46:49.085 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T20:46:49.365 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:46:50.367 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:46:50.367 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:46:50.367 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-15T20:46:50.367 INFO:tasks.workunit.client.0.smithi136.stderr:3 2024-09-15T20:46:50.367 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:46:50.367 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T20:46:50.652 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:46:50.818 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:46:50.816+0000 7f231881a640 -1 osd.1 75 log_to_monitors true 2024-09-15T20:46:51.653 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:46:51.654 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:46:51.654 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-15T20:46:51.654 INFO:tasks.workunit.client.0.smithi136.stderr:4 2024-09-15T20:46:51.654 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:46:51.654 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T20:46:52.105 INFO:tasks.workunit.client.0.smithi136.stderr:osd.1 up in weight 1 up_from 82 up_thru 82 down_at 76 last_clean_interval [63,75) [v2:127.0.0.1:6810/3541423060,v1:127.0.0.1:6811/3541423060] [v2:127.0.0.1:6812/3541423060,v1:127.0.0.1:6813/3541423060] exists,up 0c7c4a12-04ee-4cd1-8f9d-8c94bcbf4510 2024-09-15T20:46:52.105 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-15T20:46:52.105 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-15T20:46:52.105 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-15T20:46:52.106 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2024-09-15T20:46:52.106 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-15T20:46:52.106 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-15T20:46:52.106 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-15T20:46:52.106 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-15T20:46:52.106 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-15T20:46:52.107 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-15T20:46:52.107 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-15T20:46:52.107 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-15T20:46:52.107 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-15T20:46:52.196 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-15T20:46:52.196 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-15T20:46:52.196 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-15T20:46:52.196 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-15T20:46:52.196 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-15T20:46:52.196 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-15T20:46:52.546 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-15T20:46:52.546 INFO:tasks.workunit.client.0.smithi136.stderr:1 2024-09-15T20:46:52.546 INFO:tasks.workunit.client.0.smithi136.stderr:2' 2024-09-15T20:46:52.546 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-15T20:46:52.546 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T20:46:52.547 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-15T20:46:52.790 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836514 2024-09-15T20:46:52.790 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836514 2024-09-15T20:46:52.791 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836514' 2024-09-15T20:46:52.791 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T20:46:52.791 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-15T20:46:52.972 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=352187318274 2024-09-15T20:46:52.972 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 352187318274 2024-09-15T20:46:52.972 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836514 1-352187318274' 2024-09-15T20:46:52.972 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T20:46:52.972 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-15T20:46:53.109 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=150323855377 2024-09-15T20:46:53.109 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 150323855377 2024-09-15T20:46:53.109 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836514 1-352187318274 2-150323855377' 2024-09-15T20:46:53.109 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T20:46:53.110 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836514 2024-09-15T20:46:53.110 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T20:46:53.110 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-15T20:46:53.111 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836514 2024-09-15T20:46:53.111 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T20:46:53.112 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836514 2024-09-15T20:46:53.112 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836514' 2024-09-15T20:46:53.112 INFO:tasks.workunit.client.0.smithi136.stderr:waiting osd.0 seq 21474836514 2024-09-15T20:46:53.113 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T20:46:53.389 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836512 -lt 21474836514 2024-09-15T20:46:53.389 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-15T20:46:54.390 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-15T20:46:54.391 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T20:46:54.678 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836514 -lt 21474836514 2024-09-15T20:46:54.678 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T20:46:54.678 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-352187318274 2024-09-15T20:46:54.679 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T20:46:54.679 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-15T20:46:54.680 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-352187318274 2024-09-15T20:46:54.680 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T20:46:54.681 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=352187318274 2024-09-15T20:46:54.681 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 352187318274' 2024-09-15T20:46:54.681 INFO:tasks.workunit.client.0.smithi136.stderr:waiting osd.1 seq 352187318274 2024-09-15T20:46:54.681 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-15T20:46:54.974 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 352187318274 -lt 352187318274 2024-09-15T20:46:54.974 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T20:46:54.974 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-150323855377 2024-09-15T20:46:54.974 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T20:46:54.975 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-15T20:46:54.976 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-150323855377 2024-09-15T20:46:54.976 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T20:46:54.977 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=150323855377 2024-09-15T20:46:54.977 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 150323855377' 2024-09-15T20:46:54.977 INFO:tasks.workunit.client.0.smithi136.stderr:waiting osd.2 seq 150323855377 2024-09-15T20:46:54.977 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-15T20:46:55.282 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 150323855377 -lt 150323855377 2024-09-15T20:46:55.283 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-15T20:46:55.283 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T20:46:55.283 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T20:46:55.630 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 5 == 0 2024-09-15T20:46:55.630 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-15T20:46:55.631 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-15T20:46:55.631 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-15T20:46:55.631 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T20:46:55.631 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T20:46:55.631 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T20:46:55.631 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T20:46:55.904 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=4 2024-09-15T20:46:55.905 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-15T20:46:55.905 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T20:46:55.905 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T20:46:56.270 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 = 5 2024-09-15T20:46:56.270 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 4 '!=' -1 2024-09-15T20:46:56.271 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: loop=0 2024-09-15T20:46:56.271 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1652: wait_for_clean: num_active_clean=4 2024-09-15T20:46:56.271 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-15T20:46:56.271 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 0.1 2024-09-15T20:46:56.372 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-15T20:46:56.372 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-15T20:46:56.372 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-15T20:46:56.372 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-15T20:46:56.372 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T20:46:56.372 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T20:46:56.372 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T20:46:56.372 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T20:46:56.644 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=4 2024-09-15T20:46:56.644 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-15T20:46:56.644 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T20:46:56.645 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T20:46:56.991 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 = 5 2024-09-15T20:46:56.991 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 4 '!=' 4 2024-09-15T20:46:56.991 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: get_is_making_recovery_progress 2024-09-15T20:46:56.991 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2024-09-15T20:46:56.992 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-15T20:46:56.992 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-15T20:46:56.992 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-15T20:46:56.992 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2024-09-15T20:46:56.992 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-15T20:46:57.338 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=null 2024-09-15T20:46:57.338 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test null '!=' null 2024-09-15T20:46:57.338 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1655: wait_for_clean: (( 1 >= 13 )) 2024-09-15T20:46:57.338 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-15T20:46:57.338 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 0.2 2024-09-15T20:46:57.540 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-15T20:46:57.540 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-15T20:46:57.540 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-15T20:46:57.540 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-15T20:46:57.540 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T20:46:57.540 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T20:46:57.540 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T20:46:57.540 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T20:46:57.811 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=4 2024-09-15T20:46:57.811 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-15T20:46:57.811 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T20:46:57.812 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T20:46:58.320 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 = 5 2024-09-15T20:46:58.320 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 4 '!=' 4 2024-09-15T20:46:58.320 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: get_is_making_recovery_progress 2024-09-15T20:46:58.321 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2024-09-15T20:46:58.321 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-15T20:46:58.321 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-15T20:46:58.321 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-15T20:46:58.321 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2024-09-15T20:46:58.321 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-15T20:46:58.685 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=null 2024-09-15T20:46:58.685 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test null '!=' null 2024-09-15T20:46:58.685 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1655: wait_for_clean: (( 2 >= 13 )) 2024-09-15T20:46:58.685 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-15T20:46:58.685 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 0.4 2024-09-15T20:46:59.086 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-15T20:46:59.086 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-15T20:46:59.087 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-15T20:46:59.087 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-15T20:46:59.087 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T20:46:59.087 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T20:46:59.087 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T20:46:59.087 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T20:46:59.359 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=5 2024-09-15T20:46:59.359 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-15T20:46:59.359 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T20:46:59.359 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T20:46:59.705 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 = 5 2024-09-15T20:46:59.706 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-15T20:46:59.706 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-15T20:46:59.706 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:304: auto_repair_erasure_coded: rados --pool ecpool get SOMETHING td/osd-scrub-repair/COPY 2024-09-15T20:46:59.741 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:305: auto_repair_erasure_coded: diff td/osd-scrub-repair/ORIGINAL td/osd-scrub-repair/COPY 2024-09-15T20:46:59.743 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:66: run: teardown td/osd-scrub-repair 2024-09-15T20:46:59.743 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/osd-scrub-repair 2024-09-15T20:46:59.743 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-15T20:46:59.743 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/osd-scrub-repair KILL 2024-09-15T20:46:59.743 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-15T20:46:59.743 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-15T20:46:59.743 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-15T20:46:59.743 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-15T20:46:59.743 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-15T20:46:59.851 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-15T20:46:59.851 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-15T20:46:59.852 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-15T20:46:59.852 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-15T20:46:59.853 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-15T20:46:59.853 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-15T20:46:59.854 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-15T20:46:59.854 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T20:46:59.854 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-15T20:46:59.855 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-15T20:46:59.855 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T20:46:59.855 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-15T20:46:59.857 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-15T20:46:59.857 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/osd-scrub-repair 2024-09-15T20:46:59.875 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-15T20:46:59.875 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T20:46:59.875 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52602 2024-09-15T20:46:59.875 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.52602 2024-09-15T20:46:59.877 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-15T20:46:59.877 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-15T20:46:59.877 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:63: run: for func in $funcs 2024-09-15T20:46:59.877 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:64: run: setup td/osd-scrub-repair 2024-09-15T20:46:59.877 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/osd-scrub-repair 2024-09-15T20:46:59.877 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/osd-scrub-repair 2024-09-15T20:46:59.877 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/osd-scrub-repair 2024-09-15T20:46:59.877 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-15T20:46:59.877 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/osd-scrub-repair KILL 2024-09-15T20:46:59.878 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-15T20:46:59.878 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-15T20:46:59.878 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-15T20:46:59.878 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-15T20:46:59.878 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-15T20:46:59.878 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-15T20:46:59.879 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-15T20:46:59.879 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-15T20:46:59.880 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-15T20:46:59.882 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-15T20:46:59.882 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-15T20:46:59.882 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-15T20:46:59.882 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T20:46:59.882 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-15T20:46:59.882 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-15T20:46:59.883 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T20:46:59.883 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-15T20:46:59.884 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-15T20:46:59.884 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/osd-scrub-repair 2024-09-15T20:46:59.885 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-15T20:46:59.885 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T20:46:59.886 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52602 2024-09-15T20:46:59.886 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.52602 2024-09-15T20:46:59.886 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-15T20:46:59.886 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-15T20:46:59.886 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/osd-scrub-repair 2024-09-15T20:46:59.888 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2024-09-15T20:46:59.888 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T20:46:59.888 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52602 2024-09-15T20:46:59.888 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.52602 2024-09-15T20:46:59.890 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2024-09-15T20:46:59.890 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 4096 -le 1024 ']' 2024-09-15T20:46:59.890 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2024-09-15T20:46:59.890 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/osd-scrub-repair 1' TERM HUP INT 2024-09-15T20:46:59.891 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:65: run: TEST_corrupt_and_repair_jerasure_appends td/osd-scrub-repair 2024-09-15T20:46:59.891 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:815: TEST_corrupt_and_repair_jerasure_appends: corrupt_and_repair_jerasure td/osd-scrub-repair false 2024-09-15T20:46:59.891 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:798: corrupt_and_repair_jerasure: local dir=td/osd-scrub-repair 2024-09-15T20:46:59.891 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:799: corrupt_and_repair_jerasure: local allow_overwrites=false 2024-09-15T20:46:59.891 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:800: corrupt_and_repair_jerasure: local poolname=ecpool 2024-09-15T20:46:59.891 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:802: corrupt_and_repair_jerasure: run_mon td/osd-scrub-repair a 2024-09-15T20:46:59.891 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/osd-scrub-repair 2024-09-15T20:46:59.891 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2024-09-15T20:46:59.891 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2024-09-15T20:46:59.891 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2024-09-15T20:46:59.891 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/osd-scrub-repair/a 2024-09-15T20:46:59.891 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-scrub-repair/a --run-dir=td/osd-scrub-repair 2024-09-15T20:47:00.367 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2024-09-15T20:47:00.367 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-15T20:47:00.367 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-15T20:47:00.367 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-15T20:47:00.367 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T20:47:00.367 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52602 2024-09-15T20:47:00.367 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.52602/$cluster-$name.asok' 2024-09-15T20:47:00.367 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-scrub-repair/a '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.52602/$cluster-$name.asok' --mon-cluster-log-file=td/osd-scrub-repair/log --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-09-15T20:47:00.575 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2024-09-15T20:47:00.577 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2024-09-15T20:47:00.577 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-15T20:47:00.577 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-15T20:47:00.577 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2024-09-15T20:47:00.577 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2024-09-15T20:47:00.577 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-15T20:47:00.577 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-15T20:47:00.577 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-15T20:47:00.578 INFO:tasks.workunit.client.0.smithi136.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-15T20:47:00.578 INFO:tasks.workunit.client.0.smithi136.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T20:47:00.578 INFO:tasks.workunit.client.0.smithi136.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52602 2024-09-15T20:47:00.578 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.52602/ceph-mon.a.asok 2024-09-15T20:47:00.578 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-15T20:47:00.578 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.52602/ceph-mon.a.asok config get fsid 2024-09-15T20:47:00.657 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2024-09-15T20:47:00.657 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-15T20:47:00.657 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-15T20:47:00.657 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2024-09-15T20:47:00.657 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2024-09-15T20:47:00.657 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-15T20:47:00.657 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-15T20:47:00.658 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-15T20:47:00.658 INFO:tasks.workunit.client.0.smithi136.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-15T20:47:00.658 INFO:tasks.workunit.client.0.smithi136.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T20:47:00.658 INFO:tasks.workunit.client.0.smithi136.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52602 2024-09-15T20:47:00.658 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.52602/ceph-mon.a.asok 2024-09-15T20:47:00.658 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-15T20:47:00.658 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.52602/ceph-mon.a.asok config get mon_host 2024-09-15T20:47:00.736 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:803: corrupt_and_repair_jerasure: run_mgr td/osd-scrub-repair x 2024-09-15T20:47:00.736 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: run_mgr: local dir=td/osd-scrub-repair 2024-09-15T20:47:00.736 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: run_mgr: shift 2024-09-15T20:47:00.736 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: run_mgr: local id=x 2024-09-15T20:47:00.736 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: run_mgr: shift 2024-09-15T20:47:00.736 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:548: run_mgr: local data=td/osd-scrub-repair/x 2024-09-15T20:47:00.736 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:550: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-15T20:47:01.552 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: get_asok_path 2024-09-15T20:47:01.552 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-15T20:47:01.552 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-15T20:47:01.553 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-15T20:47:01.553 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T20:47:01.553 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52602 2024-09-15T20:47:01.553 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.52602/$cluster-$name.asok' 2024-09-15T20:47:01.553 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-15T20:47:01.554 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-scrub-repair/x '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.52602/$cluster-$name.asok' --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-15T20:47:01.581 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:804: corrupt_and_repair_jerasure: seq 0 3 2024-09-15T20:47:01.585 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:804: corrupt_and_repair_jerasure: for id in $(seq 0 3) 2024-09-15T20:47:01.585 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:805: corrupt_and_repair_jerasure: run_osd td/osd-scrub-repair 0 2024-09-15T20:47:01.585 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-scrub-repair 2024-09-15T20:47:01.586 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-15T20:47:01.586 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=0 2024-09-15T20:47:01.586 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-15T20:47:01.586 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-scrub-repair/0 2024-09-15T20:47:01.586 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=ce4892c7-6c6d-4b4f-8e6f-07a3e11133e5 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-15T20:47:01.586 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T20:47:01.586 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T20:47:01.586 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T20:47:01.586 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-09-15T20:47:01.586 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-09-15T20:47:01.586 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-15T20:47:01.586 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-15T20:47:01.586 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-15T20:47:01.587 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-15T20:47:01.587 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-15T20:47:01.587 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-15T20:47:01.587 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-15T20:47:01.587 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T20:47:01.587 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52602 2024-09-15T20:47:01.587 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.52602/$cluster-$name.asok' 2024-09-15T20:47:01.587 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.52602/$cluster-$name.asok' 2024-09-15T20:47:01.587 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-15T20:47:01.588 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-15T20:47:01.588 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-15T20:47:01.588 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-15T20:47:01.588 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-15T20:47:01.588 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T20:47:01.588 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T20:47:01.588 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T20:47:01.588 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T20:47:01.589 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-15T20:47:01.589 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-15T20:47:01.589 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-scrub-repair/0 2024-09-15T20:47:01.589 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-15T20:47:01.590 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=dc6acaea-8800-4195-8107-9392f24cb7c4 2024-09-15T20:47:01.590 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd0 dc6acaea-8800-4195-8107-9392f24cb7c4' 2024-09-15T20:47:01.590 INFO:tasks.workunit.client.0.smithi136.stdout:add osd0 dc6acaea-8800-4195-8107-9392f24cb7c4 2024-09-15T20:47:01.591 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-15T20:47:01.605 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQDFR+dmAnXoIxAAq/Pg3U40Nn0AUKR+Qx7TjQ== 2024-09-15T20:47:01.605 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQDFR+dmAnXoIxAAq/Pg3U40Nn0AUKR+Qx7TjQ=="}' 2024-09-15T20:47:01.605 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new dc6acaea-8800-4195-8107-9392f24cb7c4 -i td/osd-scrub-repair/0/new.json 2024-09-15T20:47:01.808 INFO:tasks.workunit.client.0.smithi136.stdout:0 2024-09-15T20:47:01.818 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-scrub-repair/0/new.json 2024-09-15T20:47:01.819 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 0 --fsid=ce4892c7-6c6d-4b4f-8e6f-07a3e11133e5 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.52602/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDFR+dmAnXoIxAAq/Pg3U40Nn0AUKR+Qx7TjQ== --osd-uuid dc6acaea-8800-4195-8107-9392f24cb7c4 2024-09-15T20:47:01.848 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:47:01.846+0000 7faf483b2640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T20:47:01.859 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:47:01.857+0000 7faf483b2640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T20:47:01.860 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:47:01.858+0000 7faf483b2640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T20:47:01.860 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:47:01.858+0000 7faf483b2640 -1 bdev(0x562fe28a3c00 td/osd-scrub-repair/0/block) open stat got: (1) Operation not permitted 2024-09-15T20:47:01.860 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:47:01.858+0000 7faf483b2640 -1 bluestore(td/osd-scrub-repair/0) _read_fsid unparsable uuid 2024-09-15T20:47:07.409 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-scrub-repair/0/keyring 2024-09-15T20:47:07.409 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-15T20:47:07.410 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd0 key to auth repository 2024-09-15T20:47:07.410 INFO:tasks.workunit.client.0.smithi136.stdout:adding osd0 key to auth repository 2024-09-15T20:47:07.411 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-scrub-repair/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-15T20:47:07.800 INFO:tasks.workunit.client.0.smithi136.stdout:start osd.0 2024-09-15T20:47:07.801 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.0 2024-09-15T20:47:07.801 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 0 --fsid=ce4892c7-6c6d-4b4f-8e6f-07a3e11133e5 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.52602/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T20:47:07.801 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-15T20:47:07.801 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-15T20:47:07.801 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-15T20:47:07.827 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:47:07.824+0000 7f287ab41640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T20:47:07.830 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:47:07.827+0000 7f287ab41640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T20:47:07.831 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:47:07.828+0000 7f287ab41640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T20:47:08.097 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 0 2024-09-15T20:47:08.098 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-15T20:47:08.098 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2024-09-15T20:47:08.098 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-15T20:47:08.098 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-15T20:47:08.098 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:47:08.098 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-15T20:47:08.098 INFO:tasks.workunit.client.0.smithi136.stdout:0 2024-09-15T20:47:08.099 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:47:08.099 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T20:47:08.373 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:47:09.374 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:47:09.375 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:47:09.375 INFO:tasks.workunit.client.0.smithi136.stdout:1 2024-09-15T20:47:09.376 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-15T20:47:09.376 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:47:09.376 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T20:47:09.443 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:47:09.441+0000 7f287ab41640 -1 Falling back to public interface 2024-09-15T20:47:09.660 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:47:10.661 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:47:10.661 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:47:10.661 INFO:tasks.workunit.client.0.smithi136.stdout:2 2024-09-15T20:47:10.662 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-15T20:47:10.662 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:47:10.662 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T20:47:10.946 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:47:11.948 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:47:11.948 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:47:11.948 INFO:tasks.workunit.client.0.smithi136.stdout:3 2024-09-15T20:47:11.949 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-15T20:47:11.949 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:47:11.949 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T20:47:12.233 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:47:12.657 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:47:12.654+0000 7f287ab41640 -1 osd.0 0 log_to_monitors true 2024-09-15T20:47:13.235 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:47:13.235 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:47:13.235 INFO:tasks.workunit.client.0.smithi136.stdout:4 2024-09-15T20:47:13.236 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-15T20:47:13.236 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:47:13.236 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T20:47:13.588 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:47:14.587 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:47:14.587 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:47:14.588 INFO:tasks.workunit.client.0.smithi136.stdout:5 2024-09-15T20:47:14.588 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-15T20:47:14.588 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:47:14.588 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T20:47:14.872 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:47:15.873 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:47:15.873 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:47:15.873 INFO:tasks.workunit.client.0.smithi136.stdout:6 2024-09-15T20:47:15.874 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-15T20:47:15.874 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:47:15.874 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T20:47:16.178 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:47:17.179 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:47:17.180 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:47:17.180 INFO:tasks.workunit.client.0.smithi136.stdout:7 2024-09-15T20:47:17.181 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-15T20:47:17.181 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:47:17.181 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T20:47:17.590 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:47:18.591 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:47:18.592 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:47:18.592 INFO:tasks.workunit.client.0.smithi136.stdout:8 2024-09-15T20:47:18.592 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-15T20:47:18.592 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:47:18.592 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T20:47:18.879 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:47:19.880 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:47:19.880 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:47:19.881 INFO:tasks.workunit.client.0.smithi136.stdout:9 2024-09-15T20:47:19.881 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-15T20:47:19.881 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:47:19.881 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T20:47:20.185 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:47:21.187 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:47:21.187 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:47:21.187 INFO:tasks.workunit.client.0.smithi136.stdout:10 2024-09-15T20:47:21.188 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-15T20:47:21.188 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:47:21.188 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T20:47:21.586 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:47:22.587 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:47:22.587 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:47:22.587 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-15T20:47:22.588 INFO:tasks.workunit.client.0.smithi136.stdout:11 2024-09-15T20:47:22.588 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:47:22.588 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T20:47:22.874 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:47:23.877 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:47:23.877 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:47:23.877 INFO:tasks.workunit.client.0.smithi136.stdout:12 2024-09-15T20:47:23.878 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-15T20:47:23.878 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:47:23.878 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T20:47:24.267 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:47:25.269 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:47:25.269 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:47:25.269 INFO:tasks.workunit.client.0.smithi136.stdout:13 2024-09-15T20:47:25.270 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-15T20:47:25.270 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:47:25.270 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T20:47:25.686 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:47:26.687 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:47:26.687 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:47:26.688 INFO:tasks.workunit.client.0.smithi136.stdout:14 2024-09-15T20:47:26.688 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-15T20:47:26.688 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:47:26.688 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T20:47:27.257 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:47:28.258 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:47:28.258 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:47:28.259 INFO:tasks.workunit.client.0.smithi136.stdout:15 2024-09-15T20:47:28.259 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-15T20:47:28.259 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:47:28.259 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T20:47:29.026 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:47:30.027 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:47:30.028 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:47:30.028 INFO:tasks.workunit.client.0.smithi136.stdout:16 2024-09-15T20:47:30.028 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-15T20:47:30.028 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:47:30.029 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T20:47:30.900 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:47:31.901 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:47:31.901 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:47:31.902 INFO:tasks.workunit.client.0.smithi136.stdout:17 2024-09-15T20:47:31.902 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-15T20:47:31.902 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:47:31.902 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T20:47:32.413 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:47:33.414 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:47:33.414 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:47:33.414 INFO:tasks.workunit.client.0.smithi136.stdout:18 2024-09-15T20:47:33.415 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 18 2024-09-15T20:47:33.415 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:47:33.415 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T20:47:34.199 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:47:35.200 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:47:35.200 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:47:35.200 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 19 2024-09-15T20:47:35.201 INFO:tasks.workunit.client.0.smithi136.stdout:19 2024-09-15T20:47:35.201 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:47:35.201 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T20:47:35.493 INFO:tasks.workunit.client.0.smithi136.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/4211371642,v1:127.0.0.1:6803/4211371642] [v2:127.0.0.1:6804/4211371642,v1:127.0.0.1:6805/4211371642] exists,up dc6acaea-8800-4195-8107-9392f24cb7c4 2024-09-15T20:47:35.499 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-15T20:47:35.499 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-15T20:47:35.500 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-15T20:47:35.500 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:804: corrupt_and_repair_jerasure: for id in $(seq 0 3) 2024-09-15T20:47:35.500 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:805: corrupt_and_repair_jerasure: run_osd td/osd-scrub-repair 1 2024-09-15T20:47:35.500 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-scrub-repair 2024-09-15T20:47:35.501 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-15T20:47:35.501 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=1 2024-09-15T20:47:35.502 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-15T20:47:35.502 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-scrub-repair/1 2024-09-15T20:47:35.502 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=ce4892c7-6c6d-4b4f-8e6f-07a3e11133e5 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-15T20:47:35.502 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T20:47:35.502 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T20:47:35.502 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T20:47:35.502 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-09-15T20:47:35.503 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-09-15T20:47:35.503 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-15T20:47:35.503 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-15T20:47:35.503 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-15T20:47:35.503 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-15T20:47:35.503 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-15T20:47:35.503 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-15T20:47:35.504 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-15T20:47:35.504 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T20:47:35.504 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52602 2024-09-15T20:47:35.504 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.52602/$cluster-$name.asok' 2024-09-15T20:47:35.504 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.52602/$cluster-$name.asok' 2024-09-15T20:47:35.505 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-15T20:47:35.505 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-15T20:47:35.505 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-15T20:47:35.505 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-15T20:47:35.505 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-15T20:47:35.505 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T20:47:35.506 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T20:47:35.506 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T20:47:35.506 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T20:47:35.506 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-15T20:47:35.507 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-15T20:47:35.507 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-scrub-repair/1 2024-09-15T20:47:35.507 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-15T20:47:35.507 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=ba9ca9c7-78b0-44cd-b086-00a47c922481 2024-09-15T20:47:35.507 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd1 ba9ca9c7-78b0-44cd-b086-00a47c922481' 2024-09-15T20:47:35.508 INFO:tasks.workunit.client.0.smithi136.stdout:add osd1 ba9ca9c7-78b0-44cd-b086-00a47c922481 2024-09-15T20:47:35.508 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-15T20:47:35.515 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQDnR+dmuJ2UHhAAyidzlLh9nM/nQttlz7TcgQ== 2024-09-15T20:47:35.515 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQDnR+dmuJ2UHhAAyidzlLh9nM/nQttlz7TcgQ=="}' 2024-09-15T20:47:35.515 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new ba9ca9c7-78b0-44cd-b086-00a47c922481 -i td/osd-scrub-repair/1/new.json 2024-09-15T20:47:35.993 INFO:tasks.workunit.client.0.smithi136.stdout:1 2024-09-15T20:47:36.005 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-scrub-repair/1/new.json 2024-09-15T20:47:36.006 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 1 --fsid=ce4892c7-6c6d-4b4f-8e6f-07a3e11133e5 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.52602/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDnR+dmuJ2UHhAAyidzlLh9nM/nQttlz7TcgQ== --osd-uuid ba9ca9c7-78b0-44cd-b086-00a47c922481 2024-09-15T20:47:36.035 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:47:36.033+0000 7fb5f821e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T20:47:36.078 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:47:36.075+0000 7fb5f821e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T20:47:36.079 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:47:36.076+0000 7fb5f821e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T20:47:36.079 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:47:36.076+0000 7fb5f821e640 -1 bdev(0x5624fa273c00 td/osd-scrub-repair/1/block) open stat got: (1) Operation not permitted 2024-09-15T20:47:36.080 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:47:36.077+0000 7fb5f821e640 -1 bluestore(td/osd-scrub-repair/1) _read_fsid unparsable uuid 2024-09-15T20:47:41.532 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-scrub-repair/1/keyring 2024-09-15T20:47:41.532 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-15T20:47:41.533 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd1 key to auth repository 2024-09-15T20:47:41.533 INFO:tasks.workunit.client.0.smithi136.stdout:adding osd1 key to auth repository 2024-09-15T20:47:41.533 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-scrub-repair/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-15T20:47:41.896 INFO:tasks.workunit.client.0.smithi136.stdout:start osd.1 2024-09-15T20:47:41.897 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.1 2024-09-15T20:47:41.897 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 1 --fsid=ce4892c7-6c6d-4b4f-8e6f-07a3e11133e5 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.52602/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T20:47:41.897 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-15T20:47:41.897 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-15T20:47:41.897 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-15T20:47:41.923 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:47:41.921+0000 7f489e00d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T20:47:41.925 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:47:41.923+0000 7f489e00d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T20:47:41.926 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:47:41.924+0000 7f489e00d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T20:47:42.172 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 1 2024-09-15T20:47:42.172 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-15T20:47:42.172 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2024-09-15T20:47:42.172 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-15T20:47:42.172 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-15T20:47:42.172 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:47:42.172 INFO:tasks.workunit.client.0.smithi136.stdout:0 2024-09-15T20:47:42.173 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-15T20:47:42.173 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:47:42.173 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T20:47:42.448 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:47:43.017 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:47:43.015+0000 7f489e00d640 -1 Falling back to public interface 2024-09-15T20:47:43.449 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:47:43.450 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:47:43.450 INFO:tasks.workunit.client.0.smithi136.stdout:1 2024-09-15T20:47:43.450 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-15T20:47:43.450 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:47:43.450 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T20:47:43.747 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:47:44.748 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:47:44.748 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:47:44.748 INFO:tasks.workunit.client.0.smithi136.stdout:2 2024-09-15T20:47:44.749 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-15T20:47:44.749 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:47:44.749 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T20:47:45.032 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:47:46.034 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:47:46.034 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:47:46.034 INFO:tasks.workunit.client.0.smithi136.stdout:3 2024-09-15T20:47:46.035 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-15T20:47:46.035 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:47:46.035 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T20:47:46.059 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:47:46.057+0000 7f489e00d640 -1 osd.1 0 log_to_monitors true 2024-09-15T20:47:46.384 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:47:47.386 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:47:47.386 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:47:47.386 INFO:tasks.workunit.client.0.smithi136.stdout:4 2024-09-15T20:47:47.386 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-15T20:47:47.387 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:47:47.387 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T20:47:47.674 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:47:48.675 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:47:48.675 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:47:48.675 INFO:tasks.workunit.client.0.smithi136.stdout:5 2024-09-15T20:47:48.676 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-15T20:47:48.676 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:47:48.676 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T20:47:49.086 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:47:50.088 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:47:50.088 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:47:50.088 INFO:tasks.workunit.client.0.smithi136.stdout:6 2024-09-15T20:47:50.089 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-15T20:47:50.089 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:47:50.089 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T20:47:50.373 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:47:51.374 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:47:51.374 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:47:51.375 INFO:tasks.workunit.client.0.smithi136.stdout:7 2024-09-15T20:47:51.375 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-15T20:47:51.375 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:47:51.375 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T20:47:51.664 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:47:52.665 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:47:52.666 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:47:52.666 INFO:tasks.workunit.client.0.smithi136.stdout:8 2024-09-15T20:47:52.666 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-15T20:47:52.667 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:47:52.667 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T20:47:52.954 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:47:53.955 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:47:53.956 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:47:53.956 INFO:tasks.workunit.client.0.smithi136.stdout:9 2024-09-15T20:47:53.956 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-15T20:47:53.956 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:47:53.957 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T20:47:54.245 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:47:55.247 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:47:55.247 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:47:55.248 INFO:tasks.workunit.client.0.smithi136.stdout:10 2024-09-15T20:47:55.249 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-15T20:47:55.249 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:47:55.249 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T20:47:55.536 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:47:56.538 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:47:56.538 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:47:56.538 INFO:tasks.workunit.client.0.smithi136.stdout:11 2024-09-15T20:47:56.538 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-15T20:47:56.539 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:47:56.539 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T20:47:56.892 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:47:57.893 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:47:57.894 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:47:57.894 INFO:tasks.workunit.client.0.smithi136.stdout:12 2024-09-15T20:47:57.894 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-15T20:47:57.894 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:47:57.894 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T20:47:58.468 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:47:59.472 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:47:59.472 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:47:59.473 INFO:tasks.workunit.client.0.smithi136.stdout:13 2024-09-15T20:47:59.473 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-15T20:47:59.473 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:47:59.473 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T20:48:00.265 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:48:01.266 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:48:01.266 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:48:01.266 INFO:tasks.workunit.client.0.smithi136.stdout:14 2024-09-15T20:48:01.267 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-15T20:48:01.267 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:48:01.267 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T20:48:01.750 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:48:02.751 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:48:02.751 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:48:02.752 INFO:tasks.workunit.client.0.smithi136.stdout:15 2024-09-15T20:48:02.752 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-15T20:48:02.752 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:48:02.752 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T20:48:03.452 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:48:04.454 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:48:04.454 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:48:04.454 INFO:tasks.workunit.client.0.smithi136.stdout:16 2024-09-15T20:48:04.455 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-15T20:48:04.455 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:48:04.455 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T20:48:05.145 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:48:06.147 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:48:06.147 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:48:06.147 INFO:tasks.workunit.client.0.smithi136.stdout:17 2024-09-15T20:48:06.148 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-15T20:48:06.148 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:48:06.148 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T20:48:06.571 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:48:07.572 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:48:07.572 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:48:07.572 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 18 2024-09-15T20:48:07.573 INFO:tasks.workunit.client.0.smithi136.stdout:18 2024-09-15T20:48:07.573 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:48:07.573 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T20:48:07.941 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:48:08.942 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:48:08.942 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:48:08.943 INFO:tasks.workunit.client.0.smithi136.stdout:19 2024-09-15T20:48:08.943 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 19 2024-09-15T20:48:08.943 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:48:08.943 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T20:48:09.230 INFO:tasks.workunit.client.0.smithi136.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/2498929416,v1:127.0.0.1:6811/2498929416] [v2:127.0.0.1:6812/2498929416,v1:127.0.0.1:6813/2498929416] exists,up ba9ca9c7-78b0-44cd-b086-00a47c922481 2024-09-15T20:48:09.234 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-15T20:48:09.234 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-15T20:48:09.234 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-15T20:48:09.234 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:804: corrupt_and_repair_jerasure: for id in $(seq 0 3) 2024-09-15T20:48:09.234 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:805: corrupt_and_repair_jerasure: run_osd td/osd-scrub-repair 2 2024-09-15T20:48:09.235 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-scrub-repair 2024-09-15T20:48:09.235 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-15T20:48:09.235 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=2 2024-09-15T20:48:09.235 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-15T20:48:09.235 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-scrub-repair/2 2024-09-15T20:48:09.235 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=ce4892c7-6c6d-4b4f-8e6f-07a3e11133e5 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-15T20:48:09.235 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T20:48:09.235 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T20:48:09.235 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T20:48:09.235 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/2' 2024-09-15T20:48:09.235 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/2/journal' 2024-09-15T20:48:09.235 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-15T20:48:09.235 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-15T20:48:09.236 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-15T20:48:09.236 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-15T20:48:09.236 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-15T20:48:09.236 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-15T20:48:09.236 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-15T20:48:09.236 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T20:48:09.236 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52602 2024-09-15T20:48:09.236 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.52602/$cluster-$name.asok' 2024-09-15T20:48:09.236 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.52602/$cluster-$name.asok' 2024-09-15T20:48:09.236 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-15T20:48:09.236 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-15T20:48:09.236 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-15T20:48:09.236 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-15T20:48:09.237 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-15T20:48:09.237 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T20:48:09.237 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T20:48:09.237 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T20:48:09.237 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T20:48:09.237 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-15T20:48:09.237 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-15T20:48:09.237 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-scrub-repair/2 2024-09-15T20:48:09.237 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-15T20:48:09.237 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=6eed0205-77e5-4f0f-9c89-cc7347b56baa 2024-09-15T20:48:09.237 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd2 6eed0205-77e5-4f0f-9c89-cc7347b56baa' 2024-09-15T20:48:09.238 INFO:tasks.workunit.client.0.smithi136.stdout:add osd2 6eed0205-77e5-4f0f-9c89-cc7347b56baa 2024-09-15T20:48:09.238 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-15T20:48:09.253 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQAJSOdmkyDwDhAARUP98H3/TH34H09HlqU9fw== 2024-09-15T20:48:09.253 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQAJSOdmkyDwDhAARUP98H3/TH34H09HlqU9fw=="}' 2024-09-15T20:48:09.253 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 6eed0205-77e5-4f0f-9c89-cc7347b56baa -i td/osd-scrub-repair/2/new.json 2024-09-15T20:48:09.607 INFO:tasks.workunit.client.0.smithi136.stdout:2 2024-09-15T20:48:09.618 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-scrub-repair/2/new.json 2024-09-15T20:48:09.619 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 2 --fsid=ce4892c7-6c6d-4b4f-8e6f-07a3e11133e5 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.52602/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAJSOdmkyDwDhAARUP98H3/TH34H09HlqU9fw== --osd-uuid 6eed0205-77e5-4f0f-9c89-cc7347b56baa 2024-09-15T20:48:09.648 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:48:09.645+0000 7fbaf400b640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T20:48:09.691 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:48:09.689+0000 7fbaf400b640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T20:48:09.692 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:48:09.690+0000 7fbaf400b640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T20:48:09.693 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:48:09.691+0000 7fbaf400b640 -1 bdev(0x558d10219c00 td/osd-scrub-repair/2/block) open stat got: (1) Operation not permitted 2024-09-15T20:48:09.693 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:48:09.691+0000 7fbaf400b640 -1 bluestore(td/osd-scrub-repair/2) _read_fsid unparsable uuid 2024-09-15T20:48:15.378 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-scrub-repair/2/keyring 2024-09-15T20:48:15.378 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-15T20:48:15.379 INFO:tasks.workunit.client.0.smithi136.stdout:adding osd2 key to auth repository 2024-09-15T20:48:15.380 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd2 key to auth repository 2024-09-15T20:48:15.380 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-scrub-repair/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-15T20:48:15.742 INFO:tasks.workunit.client.0.smithi136.stdout:start osd.2 2024-09-15T20:48:15.743 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.2 2024-09-15T20:48:15.743 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 2 --fsid=ce4892c7-6c6d-4b4f-8e6f-07a3e11133e5 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.52602/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T20:48:15.743 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-15T20:48:15.743 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-15T20:48:15.743 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-15T20:48:15.770 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:48:15.768+0000 7f1a9353e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T20:48:15.788 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:48:15.785+0000 7f1a9353e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T20:48:15.788 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:48:15.786+0000 7f1a9353e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T20:48:16.020 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 2 2024-09-15T20:48:16.020 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-15T20:48:16.020 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=2 2024-09-15T20:48:16.020 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-15T20:48:16.020 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-15T20:48:16.021 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:48:16.021 INFO:tasks.workunit.client.0.smithi136.stdout:0 2024-09-15T20:48:16.021 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-15T20:48:16.021 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:48:16.021 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T20:48:16.296 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:48:17.297 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:48:17.297 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:48:17.297 INFO:tasks.workunit.client.0.smithi136.stdout:1 2024-09-15T20:48:17.298 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-15T20:48:17.298 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:48:17.298 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T20:48:17.598 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:48:17.637 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:48:17.635+0000 7f1a9353e640 -1 Falling back to public interface 2024-09-15T20:48:18.599 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:48:18.600 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:48:18.600 INFO:tasks.workunit.client.0.smithi136.stdout:2 2024-09-15T20:48:18.600 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-15T20:48:18.600 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:48:18.601 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T20:48:18.884 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:48:19.885 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:48:19.886 INFO:tasks.workunit.client.0.smithi136.stdout:3 2024-09-15T20:48:19.886 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:48:19.886 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-15T20:48:19.886 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:48:19.886 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T20:48:20.164 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:48:20.329 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:48:20.327+0000 7f1a9353e640 -1 osd.2 0 log_to_monitors true 2024-09-15T20:48:21.165 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:48:21.166 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:48:21.166 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-15T20:48:21.166 INFO:tasks.workunit.client.0.smithi136.stdout:4 2024-09-15T20:48:21.166 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:48:21.166 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T20:48:21.452 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:48:22.454 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:48:22.454 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:48:22.454 INFO:tasks.workunit.client.0.smithi136.stdout:5 2024-09-15T20:48:22.455 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-15T20:48:22.455 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:48:22.455 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T20:48:22.741 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:48:23.743 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:48:23.743 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:48:23.743 INFO:tasks.workunit.client.0.smithi136.stdout:6 2024-09-15T20:48:23.744 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-15T20:48:23.744 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:48:23.744 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T20:48:24.032 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:48:25.033 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:48:25.034 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:48:25.034 INFO:tasks.workunit.client.0.smithi136.stdout:7 2024-09-15T20:48:25.034 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-15T20:48:25.034 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:48:25.035 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T20:48:25.421 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:48:26.422 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:48:26.422 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:48:26.422 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-15T20:48:26.423 INFO:tasks.workunit.client.0.smithi136.stdout:8 2024-09-15T20:48:26.423 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:48:26.423 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T20:48:26.710 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:48:27.711 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:48:27.712 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:48:27.712 INFO:tasks.workunit.client.0.smithi136.stdout:9 2024-09-15T20:48:27.712 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-15T20:48:27.713 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:48:27.713 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T20:48:28.009 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:48:29.010 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:48:29.010 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:48:29.010 INFO:tasks.workunit.client.0.smithi136.stdout:10 2024-09-15T20:48:29.011 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-15T20:48:29.011 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:48:29.011 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T20:48:29.405 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:48:30.406 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:48:30.406 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:48:30.406 INFO:tasks.workunit.client.0.smithi136.stdout:11 2024-09-15T20:48:30.407 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-15T20:48:30.407 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:48:30.407 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T20:48:30.696 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:48:31.697 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:48:31.697 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:48:31.697 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-15T20:48:31.698 INFO:tasks.workunit.client.0.smithi136.stdout:12 2024-09-15T20:48:31.698 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:48:31.698 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T20:48:31.986 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:48:32.987 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:48:32.988 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:48:32.988 INFO:tasks.workunit.client.0.smithi136.stdout:13 2024-09-15T20:48:32.988 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-15T20:48:32.988 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:48:32.988 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T20:48:33.720 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:48:34.722 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:48:34.722 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:48:34.723 INFO:tasks.workunit.client.0.smithi136.stdout:14 2024-09-15T20:48:34.723 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-15T20:48:34.723 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:48:34.723 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T20:48:35.309 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:48:36.310 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:48:36.310 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:48:36.310 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-15T20:48:36.311 INFO:tasks.workunit.client.0.smithi136.stdout:15 2024-09-15T20:48:36.311 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:48:36.311 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T20:48:36.806 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:48:37.808 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:48:37.808 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:48:37.809 INFO:tasks.workunit.client.0.smithi136.stdout:16 2024-09-15T20:48:37.809 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-15T20:48:37.809 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:48:37.809 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T20:48:38.474 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:48:39.475 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:48:39.476 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:48:39.476 INFO:tasks.workunit.client.0.smithi136.stdout:17 2024-09-15T20:48:39.477 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-15T20:48:39.477 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:48:39.477 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T20:48:39.861 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:48:40.861 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:48:40.861 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:48:40.861 INFO:tasks.workunit.client.0.smithi136.stdout:18 2024-09-15T20:48:40.862 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 18 2024-09-15T20:48:40.862 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:48:40.862 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T20:48:41.431 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:48:42.432 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:48:42.432 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:48:42.433 INFO:tasks.workunit.client.0.smithi136.stdout:19 2024-09-15T20:48:42.433 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 19 2024-09-15T20:48:42.433 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:48:42.433 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T20:48:43.057 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:48:44.059 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:48:44.059 INFO:tasks.workunit.client.0.smithi136.stdout:20 2024-09-15T20:48:44.060 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:48:44.060 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 20 2024-09-15T20:48:44.060 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:48:44.060 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T20:48:44.343 INFO:tasks.workunit.client.0.smithi136.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/1338555475,v1:127.0.0.1:6819/1338555475] [v2:127.0.0.1:6820/1338555475,v1:127.0.0.1:6821/1338555475] exists,up 6eed0205-77e5-4f0f-9c89-cc7347b56baa 2024-09-15T20:48:44.346 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-15T20:48:44.347 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-15T20:48:44.347 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-15T20:48:44.347 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:804: corrupt_and_repair_jerasure: for id in $(seq 0 3) 2024-09-15T20:48:44.347 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:805: corrupt_and_repair_jerasure: run_osd td/osd-scrub-repair 3 2024-09-15T20:48:44.347 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-scrub-repair 2024-09-15T20:48:44.347 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-15T20:48:44.347 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=3 2024-09-15T20:48:44.347 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-15T20:48:44.347 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-scrub-repair/3 2024-09-15T20:48:44.347 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=ce4892c7-6c6d-4b4f-8e6f-07a3e11133e5 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-15T20:48:44.347 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T20:48:44.347 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T20:48:44.347 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T20:48:44.348 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/3' 2024-09-15T20:48:44.348 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/3/journal' 2024-09-15T20:48:44.348 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-15T20:48:44.348 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-15T20:48:44.348 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-15T20:48:44.348 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-15T20:48:44.348 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-15T20:48:44.348 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-15T20:48:44.348 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-15T20:48:44.348 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T20:48:44.348 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52602 2024-09-15T20:48:44.348 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.52602/$cluster-$name.asok' 2024-09-15T20:48:44.348 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.52602/$cluster-$name.asok' 2024-09-15T20:48:44.349 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-15T20:48:44.349 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-15T20:48:44.349 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-15T20:48:44.349 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-15T20:48:44.349 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-15T20:48:44.349 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T20:48:44.349 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T20:48:44.349 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T20:48:44.349 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T20:48:44.349 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-15T20:48:44.349 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-15T20:48:44.349 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-scrub-repair/3 2024-09-15T20:48:44.349 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-15T20:48:44.350 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=48807f11-9c3a-49d3-8033-a247f29eadfa 2024-09-15T20:48:44.350 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd3 48807f11-9c3a-49d3-8033-a247f29eadfa' 2024-09-15T20:48:44.351 INFO:tasks.workunit.client.0.smithi136.stdout:add osd3 48807f11-9c3a-49d3-8033-a247f29eadfa 2024-09-15T20:48:44.351 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-15T20:48:44.369 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQAsSOdmwXe3FRAAsUfBwAcRYO+RIQf74NDDaQ== 2024-09-15T20:48:44.369 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQAsSOdmwXe3FRAAsUfBwAcRYO+RIQf74NDDaQ=="}' 2024-09-15T20:48:44.369 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 48807f11-9c3a-49d3-8033-a247f29eadfa -i td/osd-scrub-repair/3/new.json 2024-09-15T20:48:44.800 INFO:tasks.workunit.client.0.smithi136.stdout:3 2024-09-15T20:48:44.811 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-scrub-repair/3/new.json 2024-09-15T20:48:44.812 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 3 --fsid=ce4892c7-6c6d-4b4f-8e6f-07a3e11133e5 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/3 --osd-journal=td/osd-scrub-repair/3/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.52602/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAsSOdmwXe3FRAAsUfBwAcRYO+RIQf74NDDaQ== --osd-uuid 48807f11-9c3a-49d3-8033-a247f29eadfa 2024-09-15T20:48:44.842 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:48:44.839+0000 7faa6e633640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T20:48:44.876 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:48:44.874+0000 7faa6e633640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T20:48:44.877 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:48:44.875+0000 7faa6e633640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T20:48:44.877 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:48:44.875+0000 7faa6e633640 -1 bdev(0x55b2cb6abc00 td/osd-scrub-repair/3/block) open stat got: (1) Operation not permitted 2024-09-15T20:48:44.878 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:48:44.875+0000 7faa6e633640 -1 bluestore(td/osd-scrub-repair/3) _read_fsid unparsable uuid 2024-09-15T20:48:50.941 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-scrub-repair/3/keyring 2024-09-15T20:48:50.941 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-15T20:48:50.941 INFO:tasks.workunit.client.0.smithi136.stdout:adding osd3 key to auth repository 2024-09-15T20:48:50.941 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd3 key to auth repository 2024-09-15T20:48:50.942 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-scrub-repair/3/keyring auth add osd.3 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-15T20:48:51.299 INFO:tasks.workunit.client.0.smithi136.stdout:start osd.3 2024-09-15T20:48:51.299 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.3 2024-09-15T20:48:51.299 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 3 --fsid=ce4892c7-6c6d-4b4f-8e6f-07a3e11133e5 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/3 --osd-journal=td/osd-scrub-repair/3/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.52602/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T20:48:51.300 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-15T20:48:51.300 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-15T20:48:51.300 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-15T20:48:51.326 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:48:51.323+0000 7f5a1d40f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T20:48:51.327 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:48:51.325+0000 7f5a1d40f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T20:48:51.329 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:48:51.326+0000 7f5a1d40f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T20:48:51.575 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 3 2024-09-15T20:48:51.575 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-15T20:48:51.575 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=3 2024-09-15T20:48:51.575 INFO:tasks.workunit.client.0.smithi136.stdout:0 2024-09-15T20:48:51.576 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-15T20:48:51.576 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-15T20:48:51.576 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:48:51.576 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-15T20:48:51.576 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:48:51.576 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-15T20:48:51.851 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:48:52.695 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:48:52.693+0000 7f5a1d40f640 -1 Falling back to public interface 2024-09-15T20:48:52.852 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:48:52.853 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:48:52.853 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-15T20:48:52.853 INFO:tasks.workunit.client.0.smithi136.stdout:1 2024-09-15T20:48:52.853 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:48:52.854 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-15T20:48:53.141 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:48:54.142 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:48:54.142 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:48:54.142 INFO:tasks.workunit.client.0.smithi136.stdout:2 2024-09-15T20:48:54.143 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-15T20:48:54.143 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:48:54.143 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-15T20:48:54.429 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:48:55.365 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:48:55.362+0000 7f5a1d40f640 -1 osd.3 0 log_to_monitors true 2024-09-15T20:48:55.430 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:48:55.430 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:48:55.431 INFO:tasks.workunit.client.0.smithi136.stdout:3 2024-09-15T20:48:55.431 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-15T20:48:55.431 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:48:55.431 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-15T20:48:55.841 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:48:56.842 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:48:56.842 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:48:56.843 INFO:tasks.workunit.client.0.smithi136.stdout:4 2024-09-15T20:48:56.843 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-15T20:48:56.843 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:48:56.843 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-15T20:48:57.193 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:48:58.194 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:48:58.194 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:48:58.194 INFO:tasks.workunit.client.0.smithi136.stdout:5 2024-09-15T20:48:58.195 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-15T20:48:58.195 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:48:58.195 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-15T20:48:58.587 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:48:59.588 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:48:59.588 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:48:59.589 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-15T20:48:59.589 INFO:tasks.workunit.client.0.smithi136.stdout:6 2024-09-15T20:48:59.589 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:48:59.589 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-15T20:48:59.879 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:49:00.880 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:49:00.880 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:49:00.880 INFO:tasks.workunit.client.0.smithi136.stdout:7 2024-09-15T20:49:00.881 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-15T20:49:00.881 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:49:00.881 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-15T20:49:01.200 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:49:02.201 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:49:02.201 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:49:02.201 INFO:tasks.workunit.client.0.smithi136.stdout:8 2024-09-15T20:49:02.202 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-15T20:49:02.202 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:49:02.202 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-15T20:49:02.576 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:49:03.577 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:49:03.577 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:49:03.578 INFO:tasks.workunit.client.0.smithi136.stdout:9 2024-09-15T20:49:03.578 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-15T20:49:03.578 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:49:03.578 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-15T20:49:03.958 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:49:04.959 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:49:04.959 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:49:04.960 INFO:tasks.workunit.client.0.smithi136.stdout:10 2024-09-15T20:49:04.960 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-15T20:49:04.961 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:49:04.961 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-15T20:49:05.381 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:49:06.382 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:49:06.382 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:49:06.382 INFO:tasks.workunit.client.0.smithi136.stdout:11 2024-09-15T20:49:06.383 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-15T20:49:06.383 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:49:06.383 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-15T20:49:06.674 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:49:07.675 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:49:07.676 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:49:07.676 INFO:tasks.workunit.client.0.smithi136.stdout:12 2024-09-15T20:49:07.676 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-15T20:49:07.676 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:49:07.676 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-15T20:49:07.963 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:49:08.964 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:49:08.964 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:49:08.965 INFO:tasks.workunit.client.0.smithi136.stdout:13 2024-09-15T20:49:08.965 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-15T20:49:08.965 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:49:08.965 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-15T20:49:09.405 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:49:10.406 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:49:10.406 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:49:10.407 INFO:tasks.workunit.client.0.smithi136.stdout:14 2024-09-15T20:49:10.407 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-15T20:49:10.407 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:49:10.407 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-15T20:49:10.999 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:49:12.000 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:49:12.000 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:49:12.001 INFO:tasks.workunit.client.0.smithi136.stdout:15 2024-09-15T20:49:12.001 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-15T20:49:12.001 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:49:12.001 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-15T20:49:12.288 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:49:13.289 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:49:13.289 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:49:13.289 INFO:tasks.workunit.client.0.smithi136.stdout:16 2024-09-15T20:49:13.290 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-15T20:49:13.290 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:49:13.290 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-15T20:49:13.755 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:49:14.757 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:49:14.757 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:49:14.757 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-15T20:49:14.758 INFO:tasks.workunit.client.0.smithi136.stdout:17 2024-09-15T20:49:14.758 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:49:14.758 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-15T20:49:15.330 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:49:16.331 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:49:16.331 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:49:16.332 INFO:tasks.workunit.client.0.smithi136.stdout:18 2024-09-15T20:49:16.332 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 18 2024-09-15T20:49:16.332 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:49:16.333 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-15T20:49:16.677 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:49:17.678 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:49:17.678 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:49:17.679 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 19 2024-09-15T20:49:17.679 INFO:tasks.workunit.client.0.smithi136.stdout:19 2024-09-15T20:49:17.679 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:49:17.679 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-15T20:49:18.592 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:49:19.594 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:49:19.594 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:49:19.594 INFO:tasks.workunit.client.0.smithi136.stdout:20 2024-09-15T20:49:19.595 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 20 2024-09-15T20:49:19.595 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:49:19.595 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-15T20:49:19.875 INFO:tasks.workunit.client.0.smithi136.stdout:osd.3 up in weight 1 up_from 20 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6826/2070477689,v1:127.0.0.1:6827/2070477689] [v2:127.0.0.1:6828/2070477689,v1:127.0.0.1:6829/2070477689] exists,up 48807f11-9c3a-49d3-8033-a247f29eadfa 2024-09-15T20:49:19.876 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-15T20:49:19.876 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-15T20:49:19.876 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-15T20:49:19.876 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:807: corrupt_and_repair_jerasure: create_rbd_pool 2024-09-15T20:49:19.876 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:526: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2024-09-15T20:49:20.146 INFO:tasks.workunit.client.0.smithi136.stderr:pool 'rbd' does not exist 2024-09-15T20:49:20.157 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:527: create_rbd_pool: create_pool rbd 4 2024-09-15T20:49:20.157 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create rbd 4 2024-09-15T20:49:20.635 INFO:tasks.workunit.client.0.smithi136.stderr:pool 'rbd' created 2024-09-15T20:49:20.647 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2024-09-15T20:49:21.648 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:528: create_rbd_pool: rbd pool init rbd 2024-09-15T20:49:24.030 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:808: corrupt_and_repair_jerasure: wait_for_clean 2024-09-15T20:49:24.031 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-15T20:49:24.031 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-15T20:49:24.031 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-15T20:49:24.031 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-15T20:49:24.032 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-15T20:49:24.032 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-15T20:49:24.032 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-15T20:49:24.032 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-15T20:49:24.032 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-15T20:49:24.131 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-15T20:49:24.132 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-15T20:49:24.132 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-15T20:49:24.132 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-15T20:49:24.132 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-15T20:49:24.132 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-15T20:49:24.561 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-15T20:49:24.561 INFO:tasks.workunit.client.0.smithi136.stderr:1 2024-09-15T20:49:24.561 INFO:tasks.workunit.client.0.smithi136.stderr:2 2024-09-15T20:49:24.561 INFO:tasks.workunit.client.0.smithi136.stderr:3' 2024-09-15T20:49:24.561 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-15T20:49:24.561 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T20:49:24.561 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-15T20:49:24.665 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836509 2024-09-15T20:49:24.666 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836509 2024-09-15T20:49:24.666 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836509' 2024-09-15T20:49:24.666 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T20:49:24.666 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-15T20:49:24.770 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=42949672981 2024-09-15T20:49:24.770 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 42949672981 2024-09-15T20:49:24.770 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836509 1-42949672981' 2024-09-15T20:49:24.770 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T20:49:24.770 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-15T20:49:24.873 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=64424509453 2024-09-15T20:49:24.873 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 64424509453 2024-09-15T20:49:24.873 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836509 1-42949672981 2-64424509453' 2024-09-15T20:49:24.873 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T20:49:24.873 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-15T20:49:24.975 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=85899345926 2024-09-15T20:49:24.975 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 85899345926 2024-09-15T20:49:24.976 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836509 1-42949672981 2-64424509453 3-85899345926' 2024-09-15T20:49:24.976 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T20:49:24.976 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836509 2024-09-15T20:49:24.976 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T20:49:24.977 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-15T20:49:24.977 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836509 2024-09-15T20:49:24.977 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T20:49:24.978 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836509 2024-09-15T20:49:24.978 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836509' 2024-09-15T20:49:24.979 INFO:tasks.workunit.client.0.smithi136.stdout:waiting osd.0 seq 21474836509 2024-09-15T20:49:24.979 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T20:49:25.254 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836508 -lt 21474836509 2024-09-15T20:49:25.255 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-15T20:49:26.256 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-15T20:49:26.256 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T20:49:26.540 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836509 -lt 21474836509 2024-09-15T20:49:26.540 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T20:49:26.541 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-42949672981 2024-09-15T20:49:26.541 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T20:49:26.541 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-15T20:49:26.542 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-42949672981 2024-09-15T20:49:26.542 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T20:49:26.543 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=42949672981 2024-09-15T20:49:26.543 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 42949672981' 2024-09-15T20:49:26.543 INFO:tasks.workunit.client.0.smithi136.stdout:waiting osd.1 seq 42949672981 2024-09-15T20:49:26.543 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-15T20:49:26.845 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 42949672981 -lt 42949672981 2024-09-15T20:49:26.845 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T20:49:26.846 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-64424509453 2024-09-15T20:49:26.846 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T20:49:26.847 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-15T20:49:26.847 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-64424509453 2024-09-15T20:49:26.847 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T20:49:26.848 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=64424509453 2024-09-15T20:49:26.849 INFO:tasks.workunit.client.0.smithi136.stdout:waiting osd.2 seq 64424509453 2024-09-15T20:49:26.849 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 64424509453' 2024-09-15T20:49:26.849 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-15T20:49:27.160 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 64424509453 -lt 64424509453 2024-09-15T20:49:27.160 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T20:49:27.160 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 3-85899345926 2024-09-15T20:49:27.160 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T20:49:27.161 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=3 2024-09-15T20:49:27.162 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 3-85899345926 2024-09-15T20:49:27.162 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T20:49:27.163 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=85899345926 2024-09-15T20:49:27.163 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.3 seq 85899345926' 2024-09-15T20:49:27.163 INFO:tasks.workunit.client.0.smithi136.stdout:waiting osd.3 seq 85899345926 2024-09-15T20:49:27.164 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-15T20:49:27.479 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 85899345926 -lt 85899345926 2024-09-15T20:49:27.479 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-15T20:49:27.480 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T20:49:27.480 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T20:49:27.892 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 4 == 0 2024-09-15T20:49:27.893 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-15T20:49:27.893 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-15T20:49:27.893 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-15T20:49:27.893 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T20:49:27.893 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T20:49:27.893 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T20:49:27.893 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T20:49:28.169 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=4 2024-09-15T20:49:28.169 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-15T20:49:28.170 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T20:49:28.170 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T20:49:28.514 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 = 4 2024-09-15T20:49:28.514 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-15T20:49:28.514 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-15T20:49:28.514 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:810: corrupt_and_repair_jerasure: create_ec_pool ecpool false k=2 m=2 2024-09-15T20:49:28.514 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2443: create_ec_pool: local pool_name=ecpool 2024-09-15T20:49:28.515 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2444: create_ec_pool: shift 2024-09-15T20:49:28.515 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2445: create_ec_pool: local allow_overwrites=false 2024-09-15T20:49:28.515 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2446: create_ec_pool: shift 2024-09-15T20:49:28.515 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2448: create_ec_pool: ceph osd erasure-code-profile set myprofile crush-failure-domain=osd k=2 m=2 2024-09-15T20:49:29.011 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2450: create_ec_pool: create_pool ecpool 1 1 erasure myprofile 2024-09-15T20:49:29.011 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create ecpool 1 1 erasure myprofile 2024-09-15T20:49:29.661 INFO:tasks.workunit.client.0.smithi136.stderr:pool 'ecpool' created 2024-09-15T20:49:29.672 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2024-09-15T20:49:30.674 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2452: create_ec_pool: '[' false = true ']' 2024-09-15T20:49:30.675 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2456: create_ec_pool: wait_for_clean 2024-09-15T20:49:30.675 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-15T20:49:30.675 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-15T20:49:30.675 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-15T20:49:30.675 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-15T20:49:30.676 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-15T20:49:30.676 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-15T20:49:30.676 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-15T20:49:30.676 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-15T20:49:30.676 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-15T20:49:30.776 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-15T20:49:30.776 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-15T20:49:30.776 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-15T20:49:30.777 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-15T20:49:30.777 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-15T20:49:30.777 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-15T20:49:31.052 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-15T20:49:31.052 INFO:tasks.workunit.client.0.smithi136.stderr:1 2024-09-15T20:49:31.052 INFO:tasks.workunit.client.0.smithi136.stderr:2 2024-09-15T20:49:31.052 INFO:tasks.workunit.client.0.smithi136.stderr:3' 2024-09-15T20:49:31.052 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-15T20:49:31.052 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T20:49:31.052 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-15T20:49:31.155 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836512 2024-09-15T20:49:31.155 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836512 2024-09-15T20:49:31.155 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836512' 2024-09-15T20:49:31.156 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T20:49:31.156 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-15T20:49:31.280 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=42949672983 2024-09-15T20:49:31.280 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 42949672983 2024-09-15T20:49:31.280 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836512 1-42949672983' 2024-09-15T20:49:31.280 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T20:49:31.280 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-15T20:49:31.383 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=64424509455 2024-09-15T20:49:31.383 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 64424509455 2024-09-15T20:49:31.383 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836512 1-42949672983 2-64424509455' 2024-09-15T20:49:31.383 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T20:49:31.383 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-15T20:49:31.517 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=85899345928 2024-09-15T20:49:31.517 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 85899345928 2024-09-15T20:49:31.517 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836512 1-42949672983 2-64424509455 3-85899345928' 2024-09-15T20:49:31.517 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T20:49:31.518 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836512 2024-09-15T20:49:31.518 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T20:49:31.518 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-15T20:49:31.519 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836512 2024-09-15T20:49:31.519 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T20:49:31.520 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836512 2024-09-15T20:49:31.520 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836512' 2024-09-15T20:49:31.520 INFO:tasks.workunit.client.0.smithi136.stdout:waiting osd.0 seq 21474836512 2024-09-15T20:49:31.521 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T20:49:31.848 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836512 -lt 21474836512 2024-09-15T20:49:31.848 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T20:49:31.849 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-42949672983 2024-09-15T20:49:31.849 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T20:49:31.850 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-15T20:49:31.850 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-42949672983 2024-09-15T20:49:31.850 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T20:49:31.851 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=42949672983 2024-09-15T20:49:31.851 INFO:tasks.workunit.client.0.smithi136.stdout:waiting osd.1 seq 42949672983 2024-09-15T20:49:31.852 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 42949672983' 2024-09-15T20:49:31.852 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-15T20:49:32.127 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 42949672983 -lt 42949672983 2024-09-15T20:49:32.127 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T20:49:32.128 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-64424509455 2024-09-15T20:49:32.128 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T20:49:32.128 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-15T20:49:32.129 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-64424509455 2024-09-15T20:49:32.129 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T20:49:32.130 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=64424509455 2024-09-15T20:49:32.130 INFO:tasks.workunit.client.0.smithi136.stdout:waiting osd.2 seq 64424509455 2024-09-15T20:49:32.131 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 64424509455' 2024-09-15T20:49:32.131 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-15T20:49:32.408 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 64424509455 -lt 64424509455 2024-09-15T20:49:32.408 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T20:49:32.408 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 3-85899345928 2024-09-15T20:49:32.408 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T20:49:32.409 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=3 2024-09-15T20:49:32.410 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 3-85899345928 2024-09-15T20:49:32.410 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T20:49:32.411 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=85899345928 2024-09-15T20:49:32.411 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.3 seq 85899345928' 2024-09-15T20:49:32.411 INFO:tasks.workunit.client.0.smithi136.stdout:waiting osd.3 seq 85899345928 2024-09-15T20:49:32.411 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-15T20:49:32.687 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 85899345928 -lt 85899345928 2024-09-15T20:49:32.688 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-15T20:49:32.688 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T20:49:32.688 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T20:49:33.035 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 5 == 0 2024-09-15T20:49:33.035 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-15T20:49:33.035 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-15T20:49:33.035 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-15T20:49:33.036 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T20:49:33.036 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T20:49:33.036 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T20:49:33.036 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T20:49:33.307 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=5 2024-09-15T20:49:33.308 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-15T20:49:33.308 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T20:49:33.308 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T20:49:33.651 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 = 5 2024-09-15T20:49:33.651 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-15T20:49:33.652 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-15T20:49:33.652 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2457: create_ec_pool: return 0 2024-09-15T20:49:33.652 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:811: corrupt_and_repair_jerasure: corrupt_and_repair_erasure_coded td/osd-scrub-repair ecpool 2024-09-15T20:49:33.652 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:248: corrupt_and_repair_erasure_coded: local dir=td/osd-scrub-repair 2024-09-15T20:49:33.652 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:249: corrupt_and_repair_erasure_coded: local poolname=ecpool 2024-09-15T20:49:33.652 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:251: corrupt_and_repair_erasure_coded: add_something td/osd-scrub-repair ecpool 2024-09-15T20:49:33.652 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-09-15T20:49:33.652 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=ecpool 2024-09-15T20:49:33.652 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=SOMETHING 2024-09-15T20:49:33.652 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-09-15T20:49:33.652 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-09-15T20:49:33.652 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-09-15T20:49:34.035 INFO:tasks.workunit.client.0.smithi136.stderr:noscrub is set 2024-09-15T20:49:34.046 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-09-15T20:49:34.588 INFO:tasks.workunit.client.0.smithi136.stderr:nodeep-scrub is set 2024-09-15T20:49:34.601 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-09-15T20:49:34.601 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-09-15T20:49:34.601 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool ecpool put SOMETHING td/osd-scrub-repair/ORIGINAL 2024-09-15T20:49:34.852 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:253: corrupt_and_repair_erasure_coded: get_primary ecpool SOMETHING 2024-09-15T20:49:34.852 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1187: get_primary: local poolname=ecpool 2024-09-15T20:49:34.852 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1188: get_primary: local objectname=SOMETHING 2024-09-15T20:49:34.852 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1190: get_primary: ceph --format json osd map ecpool SOMETHING 2024-09-15T20:49:34.852 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1191: get_primary: jq .acting_primary 2024-09-15T20:49:35.133 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:253: corrupt_and_repair_erasure_coded: local primary=3 2024-09-15T20:49:35.133 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:254: corrupt_and_repair_erasure_coded: get_osds ecpool SOMETHING 2024-09-15T20:49:35.133 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1012: get_osds: local poolname=ecpool 2024-09-15T20:49:35.133 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1013: get_osds: local objectname=SOMETHING 2024-09-15T20:49:35.134 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:254: corrupt_and_repair_erasure_coded: sed -e s/3// 2024-09-15T20:49:35.134 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: ceph --format json osd map ecpool SOMETHING 2024-09-15T20:49:35.134 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: jq '.acting | .[]' 2024-09-15T20:49:35.412 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: local 'osds=3 2024-09-15T20:49:35.412 INFO:tasks.workunit.client.0.smithi136.stderr:1 2024-09-15T20:49:35.412 INFO:tasks.workunit.client.0.smithi136.stderr:2 2024-09-15T20:49:35.412 INFO:tasks.workunit.client.0.smithi136.stderr:0' 2024-09-15T20:49:35.412 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1018: get_osds: echo 3 1 2 0 2024-09-15T20:49:35.414 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:254: corrupt_and_repair_erasure_coded: osds=('1' '2' '0') 2024-09-15T20:49:35.414 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:254: corrupt_and_repair_erasure_coded: local -a osds 2024-09-15T20:49:35.414 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:255: corrupt_and_repair_erasure_coded: local not_primary_first=1 2024-09-15T20:49:35.414 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:256: corrupt_and_repair_erasure_coded: local not_primary_second=2 2024-09-15T20:49:35.415 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:259: corrupt_and_repair_erasure_coded: corrupt_and_repair_one td/osd-scrub-repair ecpool 3 2024-09-15T20:49:35.415 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:226: corrupt_and_repair_one: local dir=td/osd-scrub-repair 2024-09-15T20:49:35.415 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:227: corrupt_and_repair_one: local poolname=ecpool 2024-09-15T20:49:35.415 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:228: corrupt_and_repair_one: local osd=3 2024-09-15T20:49:35.415 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:233: corrupt_and_repair_one: objectstore_tool td/osd-scrub-repair 3 SOMETHING remove 2024-09-15T20:49:35.415 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2024-09-15T20:49:35.415 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2024-09-15T20:49:35.415 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=3 2024-09-15T20:49:35.415 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2024-09-15T20:49:35.415 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 3 SOMETHING remove 2024-09-15T20:49:35.415 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-09-15T20:49:35.415 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2024-09-15T20:49:35.415 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=3 2024-09-15T20:49:35.415 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2024-09-15T20:49:35.416 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.3 2024-09-15T20:49:35.416 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-15T20:49:35.416 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-15T20:49:35.416 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-15T20:49:35.416 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-15T20:49:35.416 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-15T20:49:35.720 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-15T20:49:35.720 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 3 SOMETHING remove 2024-09-15T20:49:35.720 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-15T20:49:35.720 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-15T20:49:35.721 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=3 2024-09-15T20:49:35.721 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-15T20:49:35.721 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/3 2024-09-15T20:49:35.721 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/3 SOMETHING remove 2024-09-15T20:49:37.819 INFO:tasks.workunit.client.0.smithi136.stdout:remove 0#2:eb822e21:::SOMETHING:head# 2024-09-15T20:49:38.337 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 3 2024-09-15T20:49:38.337 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2024-09-15T20:49:38.337 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-15T20:49:38.337 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=3 2024-09-15T20:49:38.338 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-15T20:49:38.338 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/3 2024-09-15T20:49:38.338 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=ce4892c7-6c6d-4b4f-8e6f-07a3e11133e5 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-15T20:49:38.338 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T20:49:38.338 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T20:49:38.338 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T20:49:38.338 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/3' 2024-09-15T20:49:38.338 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/3/journal' 2024-09-15T20:49:38.338 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-15T20:49:38.338 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-15T20:49:38.338 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-15T20:49:38.338 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-15T20:49:38.339 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-15T20:49:38.339 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-15T20:49:38.339 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-15T20:49:38.339 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T20:49:38.339 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52602 2024-09-15T20:49:38.339 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.52602/$cluster-$name.asok' 2024-09-15T20:49:38.339 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.52602/$cluster-$name.asok' 2024-09-15T20:49:38.339 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-15T20:49:38.339 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-15T20:49:38.339 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-15T20:49:38.339 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T20:49:38.339 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T20:49:38.339 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T20:49:38.340 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T20:49:38.340 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-15T20:49:38.340 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-15T20:49:38.340 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/3 2024-09-15T20:49:38.341 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.3 2024-09-15T20:49:38.341 INFO:tasks.workunit.client.0.smithi136.stderr:start osd.3 2024-09-15T20:49:38.341 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 3 --fsid=ce4892c7-6c6d-4b4f-8e6f-07a3e11133e5 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/3 --osd-journal=td/osd-scrub-repair/3/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.52602/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T20:49:38.341 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/3/whoami 2024-09-15T20:49:38.341 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 3 = 3 ']' 2024-09-15T20:49:38.341 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-15T20:49:38.341 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-15T20:49:38.341 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-15T20:49:38.369 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:49:38.366+0000 7fb172c19640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T20:49:38.371 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:49:38.369+0000 7fb172c19640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T20:49:38.372 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:49:38.370+0000 7fb172c19640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T20:49:38.628 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 3 2024-09-15T20:49:38.628 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-15T20:49:38.628 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=3 2024-09-15T20:49:38.628 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-15T20:49:38.628 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-15T20:49:38.628 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:49:38.628 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-15T20:49:38.629 INFO:tasks.workunit.client.0.smithi136.stderr:0 2024-09-15T20:49:38.629 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:49:38.629 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-15T20:49:38.905 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:49:39.730 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:49:39.728+0000 7fb172c19640 -1 Falling back to public interface 2024-09-15T20:49:39.906 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:49:39.906 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:49:39.906 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-15T20:49:39.907 INFO:tasks.workunit.client.0.smithi136.stderr:1 2024-09-15T20:49:39.907 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:49:39.907 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-15T20:49:40.190 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:49:41.192 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:49:41.192 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:49:41.192 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-15T20:49:41.192 INFO:tasks.workunit.client.0.smithi136.stderr:2 2024-09-15T20:49:41.192 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:49:41.192 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-15T20:49:41.479 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:49:42.454 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:49:42.452+0000 7fb172c19640 -1 osd.3 38 log_to_monitors true 2024-09-15T20:49:42.481 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:49:42.481 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:49:42.481 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-15T20:49:42.481 INFO:tasks.workunit.client.0.smithi136.stderr:3 2024-09-15T20:49:42.481 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:49:42.481 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-15T20:49:42.919 INFO:tasks.workunit.client.0.smithi136.stderr:osd.3 up in weight 1 up_from 43 up_thru 34 down_at 39 last_clean_interval [20,38) [v2:127.0.0.1:6826/2205277338,v1:127.0.0.1:6827/2205277338] [v2:127.0.0.1:6828/2205277338,v1:127.0.0.1:6829/2205277338] exists,up 48807f11-9c3a-49d3-8033-a247f29eadfa 2024-09-15T20:49:42.919 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-15T20:49:42.919 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-15T20:49:42.919 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-15T20:49:42.919 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2024-09-15T20:49:42.919 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-15T20:49:42.920 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-15T20:49:42.920 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-15T20:49:42.920 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-15T20:49:42.920 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-15T20:49:42.920 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-15T20:49:42.920 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-15T20:49:42.920 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-15T20:49:42.920 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-15T20:49:43.009 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-15T20:49:43.009 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-15T20:49:43.012 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-15T20:49:43.012 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-15T20:49:43.012 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-15T20:49:43.012 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-15T20:49:43.292 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-15T20:49:43.292 INFO:tasks.workunit.client.0.smithi136.stderr:1 2024-09-15T20:49:43.292 INFO:tasks.workunit.client.0.smithi136.stderr:2 2024-09-15T20:49:43.293 INFO:tasks.workunit.client.0.smithi136.stderr:3' 2024-09-15T20:49:43.293 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-15T20:49:43.293 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T20:49:43.293 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-15T20:49:43.485 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836515 2024-09-15T20:49:43.485 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836515 2024-09-15T20:49:43.485 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836515' 2024-09-15T20:49:43.485 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T20:49:43.485 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-15T20:49:43.596 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=42949672987 2024-09-15T20:49:43.597 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 42949672987 2024-09-15T20:49:43.597 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836515 1-42949672987' 2024-09-15T20:49:43.597 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T20:49:43.597 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-15T20:49:43.861 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=64424509458 2024-09-15T20:49:43.862 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 64424509458 2024-09-15T20:49:43.862 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836515 1-42949672987 2-64424509458' 2024-09-15T20:49:43.862 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T20:49:43.862 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-15T20:49:44.053 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=184683593730 2024-09-15T20:49:44.053 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 184683593730 2024-09-15T20:49:44.053 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836515 1-42949672987 2-64424509458 3-184683593730' 2024-09-15T20:49:44.053 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T20:49:44.054 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836515 2024-09-15T20:49:44.054 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T20:49:44.054 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-15T20:49:44.055 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836515 2024-09-15T20:49:44.055 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T20:49:44.056 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836515 2024-09-15T20:49:44.056 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836515' 2024-09-15T20:49:44.056 INFO:tasks.workunit.client.0.smithi136.stderr:waiting osd.0 seq 21474836515 2024-09-15T20:49:44.057 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T20:49:44.430 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836515 -lt 21474836515 2024-09-15T20:49:44.430 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T20:49:44.431 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-42949672987 2024-09-15T20:49:44.431 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T20:49:44.432 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-15T20:49:44.432 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-42949672987 2024-09-15T20:49:44.432 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T20:49:44.433 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=42949672987 2024-09-15T20:49:44.433 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 42949672987' 2024-09-15T20:49:44.433 INFO:tasks.workunit.client.0.smithi136.stderr:waiting osd.1 seq 42949672987 2024-09-15T20:49:44.434 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-15T20:49:44.750 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 42949672985 -lt 42949672987 2024-09-15T20:49:44.750 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-15T20:49:45.751 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-15T20:49:45.751 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-15T20:49:46.040 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 42949672987 -lt 42949672987 2024-09-15T20:49:46.040 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T20:49:46.040 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-64424509458 2024-09-15T20:49:46.040 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T20:49:46.040 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-15T20:49:46.041 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-64424509458 2024-09-15T20:49:46.041 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T20:49:46.042 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=64424509458 2024-09-15T20:49:46.042 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 64424509458' 2024-09-15T20:49:46.042 INFO:tasks.workunit.client.0.smithi136.stderr:waiting osd.2 seq 64424509458 2024-09-15T20:49:46.042 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-15T20:49:46.320 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 64424509459 -lt 64424509458 2024-09-15T20:49:46.320 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T20:49:46.321 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 3-184683593730 2024-09-15T20:49:46.321 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T20:49:46.321 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=3 2024-09-15T20:49:46.322 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 3-184683593730 2024-09-15T20:49:46.322 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T20:49:46.323 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=184683593730 2024-09-15T20:49:46.323 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.3 seq 184683593730' 2024-09-15T20:49:46.323 INFO:tasks.workunit.client.0.smithi136.stderr:waiting osd.3 seq 184683593730 2024-09-15T20:49:46.324 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-15T20:49:46.597 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 184683593730 -lt 184683593730 2024-09-15T20:49:46.597 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-15T20:49:46.597 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T20:49:46.597 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T20:49:46.946 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 5 == 0 2024-09-15T20:49:46.946 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-15T20:49:46.946 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-15T20:49:46.946 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-15T20:49:46.946 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T20:49:46.946 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T20:49:46.946 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T20:49:46.946 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T20:49:47.221 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=3 2024-09-15T20:49:47.222 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-15T20:49:47.222 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T20:49:47.222 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T20:49:47.564 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 3 = 5 2024-09-15T20:49:47.564 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 3 '!=' -1 2024-09-15T20:49:47.564 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: loop=0 2024-09-15T20:49:47.564 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1652: wait_for_clean: num_active_clean=3 2024-09-15T20:49:47.564 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-15T20:49:47.564 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 0.1 2024-09-15T20:49:47.665 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-15T20:49:47.665 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-15T20:49:47.665 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-15T20:49:47.665 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-15T20:49:47.665 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T20:49:47.665 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T20:49:47.666 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T20:49:47.666 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T20:49:47.938 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=4 2024-09-15T20:49:47.939 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-15T20:49:47.939 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T20:49:47.939 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T20:49:48.281 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 = 5 2024-09-15T20:49:48.281 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 4 '!=' 3 2024-09-15T20:49:48.281 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: loop=0 2024-09-15T20:49:48.282 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1652: wait_for_clean: num_active_clean=4 2024-09-15T20:49:48.282 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-15T20:49:48.282 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 0.1 2024-09-15T20:49:48.383 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-15T20:49:48.383 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-15T20:49:48.383 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-15T20:49:48.383 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-15T20:49:48.383 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T20:49:48.383 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T20:49:48.383 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T20:49:48.383 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T20:49:48.684 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=4 2024-09-15T20:49:48.684 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-15T20:49:48.685 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T20:49:48.685 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T20:49:49.026 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 = 5 2024-09-15T20:49:49.026 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 4 '!=' 4 2024-09-15T20:49:49.026 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: get_is_making_recovery_progress 2024-09-15T20:49:49.026 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2024-09-15T20:49:49.026 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-15T20:49:49.026 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-15T20:49:49.026 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-15T20:49:49.026 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2024-09-15T20:49:49.026 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-15T20:49:49.371 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=null 2024-09-15T20:49:49.371 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test null '!=' null 2024-09-15T20:49:49.371 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1655: wait_for_clean: (( 1 >= 13 )) 2024-09-15T20:49:49.371 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-15T20:49:49.371 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 0.2 2024-09-15T20:49:49.572 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-15T20:49:49.572 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-15T20:49:49.573 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-15T20:49:49.573 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-15T20:49:49.573 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T20:49:49.573 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T20:49:49.574 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T20:49:49.574 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T20:49:49.873 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=5 2024-09-15T20:49:49.873 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-15T20:49:49.873 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T20:49:49.873 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T20:49:50.217 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 = 5 2024-09-15T20:49:50.217 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-15T20:49:50.217 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-15T20:49:50.217 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:237: corrupt_and_repair_one: get_pg ecpool SOMETHING 2024-09-15T20:49:50.217 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1084: get_pg: local poolname=ecpool 2024-09-15T20:49:50.217 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1085: get_pg: local objectname=SOMETHING 2024-09-15T20:49:50.218 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1087: get_pg: ceph --format json osd map ecpool SOMETHING 2024-09-15T20:49:50.218 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1087: get_pg: jq -r .pgid 2024-09-15T20:49:50.494 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:237: corrupt_and_repair_one: local pg=2.0 2024-09-15T20:49:50.495 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:238: corrupt_and_repair_one: repair 2.0 2024-09-15T20:49:50.495 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1846: repair: local pgid=2.0 2024-09-15T20:49:50.495 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1847: repair: get_last_scrub_stamp 2.0 2024-09-15T20:49:50.495 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2024-09-15T20:49:50.495 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-15T20:49:50.495 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-15T20:49:50.495 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-09-15T20:49:50.769 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1847: repair: local last_scrub=2024-09-15T20:49:29.482036+0000 2024-09-15T20:49:50.769 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1848: repair: ceph pg repair 2.0 2024-09-15T20:49:51.031 INFO:tasks.workunit.client.0.smithi136.stderr:instructing pg 2.0s0 on osd.3 to repair 2024-09-15T20:49:51.042 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1849: repair: wait_for_scrub 2.0 2024-09-15T20:49:29.482036+0000 2024-09-15T20:49:51.043 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: local pgid=2.0 2024-09-15T20:49:51.043 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2018: wait_for_scrub: local last_scrub=2024-09-15T20:49:29.482036+0000 2024-09-15T20:49:51.043 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2019: wait_for_scrub: local sname=last_scrub_stamp 2024-09-15T20:49:51.043 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i=0 )) 2024-09-15T20:49:51.043 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-15T20:49:51.043 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-09-15T20:49:51.043 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2024-09-15T20:49:51.043 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-15T20:49:51.043 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-15T20:49:51.043 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-09-15T20:49:51.317 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-15T20:49:29.482036+0000 '>' 2024-09-15T20:49:29.482036+0000 2024-09-15T20:49:51.317 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-15T20:49:52.318 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-15T20:49:52.318 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-15T20:49:52.319 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-09-15T20:49:52.319 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2024-09-15T20:49:52.319 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-15T20:49:52.319 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-15T20:49:52.319 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-09-15T20:49:52.601 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-15T20:49:29.482036+0000 '>' 2024-09-15T20:49:29.482036+0000 2024-09-15T20:49:52.602 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-15T20:49:53.603 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-15T20:49:53.603 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-15T20:49:53.603 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-09-15T20:49:53.604 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2024-09-15T20:49:53.604 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-15T20:49:53.604 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-15T20:49:53.604 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-09-15T20:49:53.889 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-15T20:49:51.405772+0000 '>' 2024-09-15T20:49:29.482036+0000 2024-09-15T20:49:53.889 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2023: wait_for_scrub: return 0 2024-09-15T20:49:53.889 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1850: repair: sleep 2 2024-09-15T20:49:55.891 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:242: corrupt_and_repair_one: objectstore_tool td/osd-scrub-repair 3 SOMETHING list-attrs 2024-09-15T20:49:55.891 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2024-09-15T20:49:55.891 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2024-09-15T20:49:55.892 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=3 2024-09-15T20:49:55.892 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2024-09-15T20:49:55.892 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 3 SOMETHING list-attrs 2024-09-15T20:49:55.892 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-09-15T20:49:55.892 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2024-09-15T20:49:55.892 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=3 2024-09-15T20:49:55.892 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2024-09-15T20:49:55.892 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.3 2024-09-15T20:49:55.892 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-15T20:49:55.892 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-15T20:49:55.892 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-15T20:49:55.892 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-15T20:49:55.892 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-15T20:49:56.199 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-15T20:49:56.200 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 3 SOMETHING list-attrs 2024-09-15T20:49:56.200 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-15T20:49:56.200 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-15T20:49:56.200 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=3 2024-09-15T20:49:56.200 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-15T20:49:56.200 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/3 2024-09-15T20:49:56.200 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/3 SOMETHING list-attrs 2024-09-15T20:49:56.710 INFO:tasks.workunit.client.0.smithi136.stderr:Error getting attr on : 2.0s0_head,0#-4:00000000:::scrub_2.0s0:head#, (61) No data available 2024-09-15T20:49:56.710 INFO:tasks.workunit.client.0.smithi136.stdout:_ 2024-09-15T20:49:56.710 INFO:tasks.workunit.client.0.smithi136.stdout:hinfo_key 2024-09-15T20:49:56.710 INFO:tasks.workunit.client.0.smithi136.stdout:snapset 2024-09-15T20:49:56.978 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 3 2024-09-15T20:49:56.978 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2024-09-15T20:49:56.978 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-15T20:49:56.978 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=3 2024-09-15T20:49:56.978 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-15T20:49:56.978 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/3 2024-09-15T20:49:56.978 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=ce4892c7-6c6d-4b4f-8e6f-07a3e11133e5 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-15T20:49:56.978 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T20:49:56.978 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T20:49:56.979 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T20:49:56.979 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/3' 2024-09-15T20:49:56.979 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/3/journal' 2024-09-15T20:49:56.979 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-15T20:49:56.979 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-15T20:49:56.979 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-15T20:49:56.979 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-15T20:49:56.979 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-15T20:49:56.979 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-15T20:49:56.979 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-15T20:49:56.979 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T20:49:56.979 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52602 2024-09-15T20:49:56.980 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.52602/$cluster-$name.asok' 2024-09-15T20:49:56.980 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.52602/$cluster-$name.asok' 2024-09-15T20:49:56.980 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-15T20:49:56.980 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-15T20:49:56.980 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-15T20:49:56.980 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T20:49:56.980 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T20:49:56.980 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T20:49:56.980 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T20:49:56.980 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-15T20:49:56.980 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-15T20:49:56.980 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/3 2024-09-15T20:49:56.981 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.3 2024-09-15T20:49:56.981 INFO:tasks.workunit.client.0.smithi136.stderr:start osd.3 2024-09-15T20:49:56.982 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 3 --fsid=ce4892c7-6c6d-4b4f-8e6f-07a3e11133e5 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/3 --osd-journal=td/osd-scrub-repair/3/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.52602/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T20:49:56.982 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/3/whoami 2024-09-15T20:49:56.982 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 3 = 3 ']' 2024-09-15T20:49:56.982 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-15T20:49:56.982 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-15T20:49:56.982 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-15T20:49:57.009 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:49:57.007+0000 7f0750ef1640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T20:49:57.121 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:49:57.118+0000 7f0750ef1640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T20:49:57.122 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:49:57.120+0000 7f0750ef1640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T20:49:57.360 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 3 2024-09-15T20:49:57.360 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-15T20:49:57.360 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=3 2024-09-15T20:49:57.361 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-15T20:49:57.361 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-15T20:49:57.361 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:49:57.361 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-15T20:49:57.361 INFO:tasks.workunit.client.0.smithi136.stderr:0 2024-09-15T20:49:57.361 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:49:57.361 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-15T20:49:57.655 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:49:58.230 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:49:58.228+0000 7f0750ef1640 -1 Falling back to public interface 2024-09-15T20:49:58.657 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:49:58.657 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:49:58.657 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-15T20:49:58.657 INFO:tasks.workunit.client.0.smithi136.stderr:1 2024-09-15T20:49:58.657 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:49:58.657 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-15T20:49:58.942 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:49:59.944 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:49:59.944 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:49:59.944 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-15T20:49:59.944 INFO:tasks.workunit.client.0.smithi136.stderr:2 2024-09-15T20:49:59.945 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:49:59.945 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-15T20:50:00.231 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:50:01.175 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:50:01.173+0000 7f0750ef1640 -1 osd.3 45 log_to_monitors true 2024-09-15T20:50:01.232 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:50:01.232 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:50:01.233 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-15T20:50:01.233 INFO:tasks.workunit.client.0.smithi136.stderr:3 2024-09-15T20:50:01.233 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:50:01.233 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-15T20:50:01.636 INFO:tasks.workunit.client.0.smithi136.stderr:osd.3 up in weight 1 up_from 50 up_thru 43 down_at 46 last_clean_interval [43,45) [v2:127.0.0.1:6826/3316504916,v1:127.0.0.1:6827/3316504916] [v2:127.0.0.1:6828/3316504916,v1:127.0.0.1:6829/3316504916] exists,up 48807f11-9c3a-49d3-8033-a247f29eadfa 2024-09-15T20:50:01.636 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-15T20:50:01.636 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-15T20:50:01.637 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-15T20:50:01.637 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2024-09-15T20:50:01.637 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-15T20:50:01.637 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-15T20:50:01.637 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-15T20:50:01.637 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-15T20:50:01.638 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-15T20:50:01.638 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-15T20:50:01.638 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-15T20:50:01.638 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-15T20:50:01.638 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-15T20:50:01.727 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-15T20:50:01.727 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-15T20:50:01.728 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-15T20:50:01.728 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-15T20:50:01.728 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-15T20:50:01.728 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-15T20:50:02.218 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-15T20:50:02.218 INFO:tasks.workunit.client.0.smithi136.stderr:1 2024-09-15T20:50:02.218 INFO:tasks.workunit.client.0.smithi136.stderr:2 2024-09-15T20:50:02.218 INFO:tasks.workunit.client.0.smithi136.stderr:3' 2024-09-15T20:50:02.218 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-15T20:50:02.218 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T20:50:02.218 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-15T20:50:02.445 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836520 2024-09-15T20:50:02.446 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836520 2024-09-15T20:50:02.446 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836520' 2024-09-15T20:50:02.446 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T20:50:02.446 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-15T20:50:02.640 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=42949672991 2024-09-15T20:50:02.641 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 42949672991 2024-09-15T20:50:02.641 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836520 1-42949672991' 2024-09-15T20:50:02.641 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T20:50:02.641 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-15T20:50:02.755 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=64424509463 2024-09-15T20:50:02.755 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 64424509463 2024-09-15T20:50:02.756 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836520 1-42949672991 2-64424509463' 2024-09-15T20:50:02.756 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T20:50:02.756 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-15T20:50:02.903 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=214748364802 2024-09-15T20:50:02.904 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 214748364802 2024-09-15T20:50:02.904 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836520 1-42949672991 2-64424509463 3-214748364802' 2024-09-15T20:50:02.904 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T20:50:02.904 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836520 2024-09-15T20:50:02.904 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T20:50:02.904 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-15T20:50:02.905 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836520 2024-09-15T20:50:02.905 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T20:50:02.906 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836520 2024-09-15T20:50:02.908 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836520' 2024-09-15T20:50:02.908 INFO:tasks.workunit.client.0.smithi136.stderr:waiting osd.0 seq 21474836520 2024-09-15T20:50:02.909 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T20:50:03.182 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836519 -lt 21474836520 2024-09-15T20:50:03.183 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-15T20:50:04.184 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-15T20:50:04.184 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T20:50:04.471 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836520 -lt 21474836520 2024-09-15T20:50:04.471 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T20:50:04.471 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-42949672991 2024-09-15T20:50:04.471 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T20:50:04.472 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-15T20:50:04.473 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-42949672991 2024-09-15T20:50:04.473 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T20:50:04.474 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=42949672991 2024-09-15T20:50:04.474 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 42949672991' 2024-09-15T20:50:04.474 INFO:tasks.workunit.client.0.smithi136.stderr:waiting osd.1 seq 42949672991 2024-09-15T20:50:04.474 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-15T20:50:04.749 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 42949672991 -lt 42949672991 2024-09-15T20:50:04.750 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T20:50:04.750 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-64424509463 2024-09-15T20:50:04.750 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T20:50:04.751 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-15T20:50:04.751 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-64424509463 2024-09-15T20:50:04.752 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T20:50:04.753 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=64424509463 2024-09-15T20:50:04.753 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 64424509463' 2024-09-15T20:50:04.753 INFO:tasks.workunit.client.0.smithi136.stderr:waiting osd.2 seq 64424509463 2024-09-15T20:50:04.753 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-15T20:50:05.029 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 64424509463 -lt 64424509463 2024-09-15T20:50:05.030 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T20:50:05.030 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 3-214748364802 2024-09-15T20:50:05.030 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T20:50:05.031 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=3 2024-09-15T20:50:05.031 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 3-214748364802 2024-09-15T20:50:05.031 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T20:50:05.033 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=214748364802 2024-09-15T20:50:05.033 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.3 seq 214748364802' 2024-09-15T20:50:05.033 INFO:tasks.workunit.client.0.smithi136.stderr:waiting osd.3 seq 214748364802 2024-09-15T20:50:05.033 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-15T20:50:05.310 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 214748364802 -lt 214748364802 2024-09-15T20:50:05.310 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-15T20:50:05.310 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T20:50:05.311 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T20:50:05.663 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 5 == 0 2024-09-15T20:50:05.663 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-15T20:50:05.664 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-15T20:50:05.664 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-15T20:50:05.664 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T20:50:05.664 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T20:50:05.664 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T20:50:05.664 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T20:50:05.950 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=4 2024-09-15T20:50:05.950 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-15T20:50:05.950 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T20:50:05.951 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T20:50:06.297 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 = 5 2024-09-15T20:50:06.297 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 4 '!=' -1 2024-09-15T20:50:06.297 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: loop=0 2024-09-15T20:50:06.297 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1652: wait_for_clean: num_active_clean=4 2024-09-15T20:50:06.297 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-15T20:50:06.297 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 0.1 2024-09-15T20:50:06.397 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-15T20:50:06.398 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-15T20:50:06.398 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-15T20:50:06.399 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-15T20:50:06.399 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T20:50:06.399 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T20:50:06.399 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T20:50:06.399 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T20:50:06.670 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=4 2024-09-15T20:50:06.670 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-15T20:50:06.670 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T20:50:06.670 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T20:50:07.024 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 = 5 2024-09-15T20:50:07.024 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 4 '!=' 4 2024-09-15T20:50:07.024 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: get_is_making_recovery_progress 2024-09-15T20:50:07.024 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2024-09-15T20:50:07.024 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-15T20:50:07.024 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-15T20:50:07.025 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-15T20:50:07.025 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2024-09-15T20:50:07.025 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-15T20:50:07.368 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=null 2024-09-15T20:50:07.369 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test null '!=' null 2024-09-15T20:50:07.369 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1655: wait_for_clean: (( 1 >= 13 )) 2024-09-15T20:50:07.369 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-15T20:50:07.369 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 0.2 2024-09-15T20:50:07.569 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-15T20:50:07.569 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-15T20:50:07.570 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-15T20:50:07.570 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-15T20:50:07.570 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T20:50:07.570 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T20:50:07.570 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T20:50:07.571 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T20:50:07.844 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=5 2024-09-15T20:50:07.845 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-15T20:50:07.845 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T20:50:07.845 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T20:50:08.188 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 = 5 2024-09-15T20:50:08.188 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-15T20:50:08.189 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-15T20:50:08.189 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:243: corrupt_and_repair_one: rados --pool ecpool get SOMETHING td/osd-scrub-repair/COPY 2024-09-15T20:50:08.228 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:244: corrupt_and_repair_one: diff td/osd-scrub-repair/ORIGINAL td/osd-scrub-repair/COPY 2024-09-15T20:50:08.230 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:261: corrupt_and_repair_erasure_coded: corrupt_and_repair_one td/osd-scrub-repair ecpool 1 2024-09-15T20:50:08.231 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:226: corrupt_and_repair_one: local dir=td/osd-scrub-repair 2024-09-15T20:50:08.231 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:227: corrupt_and_repair_one: local poolname=ecpool 2024-09-15T20:50:08.231 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:228: corrupt_and_repair_one: local osd=1 2024-09-15T20:50:08.231 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:233: corrupt_and_repair_one: objectstore_tool td/osd-scrub-repair 1 SOMETHING remove 2024-09-15T20:50:08.231 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2024-09-15T20:50:08.231 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2024-09-15T20:50:08.231 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=1 2024-09-15T20:50:08.231 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2024-09-15T20:50:08.231 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 SOMETHING remove 2024-09-15T20:50:08.231 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-09-15T20:50:08.231 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2024-09-15T20:50:08.231 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=1 2024-09-15T20:50:08.231 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2024-09-15T20:50:08.232 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-09-15T20:50:08.232 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-15T20:50:08.232 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-15T20:50:08.232 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-15T20:50:08.232 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-15T20:50:08.232 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-15T20:50:08.536 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-15T20:50:08.537 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 SOMETHING remove 2024-09-15T20:50:08.537 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-15T20:50:08.537 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-15T20:50:08.537 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=1 2024-09-15T20:50:08.537 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-15T20:50:08.537 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-09-15T20:50:08.537 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 SOMETHING remove 2024-09-15T20:50:10.605 INFO:tasks.workunit.client.0.smithi136.stdout:remove 1#2:eb822e21:::SOMETHING:head# 2024-09-15T20:50:11.127 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2024-09-15T20:50:11.128 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2024-09-15T20:50:11.128 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-15T20:50:11.128 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=1 2024-09-15T20:50:11.128 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-15T20:50:11.128 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-09-15T20:50:11.128 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=ce4892c7-6c6d-4b4f-8e6f-07a3e11133e5 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-15T20:50:11.128 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T20:50:11.128 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T20:50:11.128 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T20:50:11.128 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-09-15T20:50:11.128 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-09-15T20:50:11.129 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-15T20:50:11.129 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-15T20:50:11.129 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-15T20:50:11.129 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-15T20:50:11.129 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-15T20:50:11.129 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-15T20:50:11.129 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-15T20:50:11.129 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T20:50:11.129 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52602 2024-09-15T20:50:11.129 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.52602/$cluster-$name.asok' 2024-09-15T20:50:11.129 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.52602/$cluster-$name.asok' 2024-09-15T20:50:11.129 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-15T20:50:11.129 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-15T20:50:11.130 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-15T20:50:11.130 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T20:50:11.130 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T20:50:11.130 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T20:50:11.130 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T20:50:11.130 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-15T20:50:11.130 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-15T20:50:11.130 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-09-15T20:50:11.130 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.1 2024-09-15T20:50:11.130 INFO:tasks.workunit.client.0.smithi136.stderr:start osd.1 2024-09-15T20:50:11.130 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 1 --fsid=ce4892c7-6c6d-4b4f-8e6f-07a3e11133e5 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.52602/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T20:50:11.130 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-09-15T20:50:11.131 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 1 = 1 ']' 2024-09-15T20:50:11.131 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-15T20:50:11.131 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-15T20:50:11.131 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-15T20:50:11.158 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:50:11.154+0000 7f81e6934640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T20:50:11.160 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:50:11.157+0000 7f81e6934640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T20:50:11.161 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:50:11.158+0000 7f81e6934640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T20:50:11.415 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 1 2024-09-15T20:50:11.415 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-15T20:50:11.415 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2024-09-15T20:50:11.415 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-15T20:50:11.415 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-15T20:50:11.415 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:50:11.415 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-15T20:50:11.415 INFO:tasks.workunit.client.0.smithi136.stderr:0 2024-09-15T20:50:11.415 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:50:11.415 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T20:50:11.700 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:50:12.702 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:50:12.702 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:50:12.702 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-15T20:50:12.702 INFO:tasks.workunit.client.0.smithi136.stderr:1 2024-09-15T20:50:12.702 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:50:12.702 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T20:50:12.990 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:50:13.015 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:50:13.012+0000 7f81e6934640 -1 Falling back to public interface 2024-09-15T20:50:13.992 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:50:13.992 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:50:13.992 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-15T20:50:13.992 INFO:tasks.workunit.client.0.smithi136.stderr:2 2024-09-15T20:50:13.992 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:50:13.992 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T20:50:14.275 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:50:15.276 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:50:15.277 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:50:15.277 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-15T20:50:15.277 INFO:tasks.workunit.client.0.smithi136.stderr:3 2024-09-15T20:50:15.277 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:50:15.277 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T20:50:15.561 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:50:15.770 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:50:15.752+0000 7f81e6934640 -1 osd.1 52 log_to_monitors true 2024-09-15T20:50:16.563 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:50:16.563 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:50:16.563 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-15T20:50:16.563 INFO:tasks.workunit.client.0.smithi136.stderr:4 2024-09-15T20:50:16.563 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:50:16.563 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T20:50:17.116 INFO:tasks.workunit.client.0.smithi136.stderr:osd.1 up in weight 1 up_from 57 up_thru 57 down_at 53 last_clean_interval [10,52) [v2:127.0.0.1:6810/385630105,v1:127.0.0.1:6811/385630105] [v2:127.0.0.1:6812/385630105,v1:127.0.0.1:6813/385630105] exists,up ba9ca9c7-78b0-44cd-b086-00a47c922481 2024-09-15T20:50:17.116 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-15T20:50:17.116 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-15T20:50:17.116 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-15T20:50:17.116 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2024-09-15T20:50:17.116 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-15T20:50:17.116 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-15T20:50:17.116 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-15T20:50:17.116 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-15T20:50:17.116 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-15T20:50:17.117 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-15T20:50:17.117 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-15T20:50:17.117 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-15T20:50:17.117 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-15T20:50:17.206 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-15T20:50:17.206 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-15T20:50:17.206 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-15T20:50:17.206 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-15T20:50:17.207 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-15T20:50:17.207 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-15T20:50:17.542 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-15T20:50:17.542 INFO:tasks.workunit.client.0.smithi136.stderr:1 2024-09-15T20:50:17.542 INFO:tasks.workunit.client.0.smithi136.stderr:2 2024-09-15T20:50:17.542 INFO:tasks.workunit.client.0.smithi136.stderr:3' 2024-09-15T20:50:17.542 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-15T20:50:17.542 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T20:50:17.542 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-15T20:50:17.792 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836524 2024-09-15T20:50:17.793 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836524 2024-09-15T20:50:17.793 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836524' 2024-09-15T20:50:17.793 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T20:50:17.793 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-15T20:50:17.896 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=244813135874 2024-09-15T20:50:17.896 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 244813135874 2024-09-15T20:50:17.897 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836524 1-244813135874' 2024-09-15T20:50:17.897 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T20:50:17.897 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-15T20:50:18.024 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=64424509467 2024-09-15T20:50:18.025 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 64424509467 2024-09-15T20:50:18.025 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836524 1-244813135874 2-64424509467' 2024-09-15T20:50:18.025 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T20:50:18.025 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-15T20:50:18.128 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=214748364806 2024-09-15T20:50:18.128 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 214748364806 2024-09-15T20:50:18.128 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836524 1-244813135874 2-64424509467 3-214748364806' 2024-09-15T20:50:18.128 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T20:50:18.129 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836524 2024-09-15T20:50:18.129 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T20:50:18.129 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-15T20:50:18.130 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836524 2024-09-15T20:50:18.130 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T20:50:18.131 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836524 2024-09-15T20:50:18.131 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836524' 2024-09-15T20:50:18.131 INFO:tasks.workunit.client.0.smithi136.stderr:waiting osd.0 seq 21474836524 2024-09-15T20:50:18.131 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T20:50:18.404 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836523 -lt 21474836524 2024-09-15T20:50:18.405 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-15T20:50:19.405 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-15T20:50:19.406 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T20:50:19.705 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836523 -lt 21474836524 2024-09-15T20:50:19.705 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-15T20:50:20.706 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 299 -eq 0 ']' 2024-09-15T20:50:20.706 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T20:50:20.990 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836524 -lt 21474836524 2024-09-15T20:50:20.991 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T20:50:20.991 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-244813135874 2024-09-15T20:50:20.991 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T20:50:20.992 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-15T20:50:20.993 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-244813135874 2024-09-15T20:50:20.993 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T20:50:20.994 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=244813135874 2024-09-15T20:50:20.994 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 244813135874' 2024-09-15T20:50:20.994 INFO:tasks.workunit.client.0.smithi136.stderr:waiting osd.1 seq 244813135874 2024-09-15T20:50:20.995 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-15T20:50:21.271 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 244813135874 -lt 244813135874 2024-09-15T20:50:21.271 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T20:50:21.271 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-64424509467 2024-09-15T20:50:21.271 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T20:50:21.272 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-15T20:50:21.273 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-64424509467 2024-09-15T20:50:21.273 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T20:50:21.273 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=64424509467 2024-09-15T20:50:21.274 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 64424509467' 2024-09-15T20:50:21.274 INFO:tasks.workunit.client.0.smithi136.stderr:waiting osd.2 seq 64424509467 2024-09-15T20:50:21.274 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-15T20:50:21.549 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 64424509468 -lt 64424509467 2024-09-15T20:50:21.549 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T20:50:21.550 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 3-214748364806 2024-09-15T20:50:21.550 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T20:50:21.550 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=3 2024-09-15T20:50:21.551 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 3-214748364806 2024-09-15T20:50:21.551 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T20:50:21.552 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=214748364806 2024-09-15T20:50:21.552 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.3 seq 214748364806' 2024-09-15T20:50:21.552 INFO:tasks.workunit.client.0.smithi136.stderr:waiting osd.3 seq 214748364806 2024-09-15T20:50:21.552 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-15T20:50:21.875 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 214748364807 -lt 214748364806 2024-09-15T20:50:21.876 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-15T20:50:21.876 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T20:50:21.876 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T20:50:22.222 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 5 == 0 2024-09-15T20:50:22.222 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-15T20:50:22.222 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-15T20:50:22.222 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-15T20:50:22.222 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T20:50:22.222 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T20:50:22.222 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T20:50:22.222 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T20:50:22.495 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=5 2024-09-15T20:50:22.496 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-15T20:50:22.496 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T20:50:22.496 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T20:50:22.840 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 = 5 2024-09-15T20:50:22.840 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-15T20:50:22.840 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-15T20:50:22.840 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:237: corrupt_and_repair_one: get_pg ecpool SOMETHING 2024-09-15T20:50:22.840 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1084: get_pg: local poolname=ecpool 2024-09-15T20:50:22.840 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1085: get_pg: local objectname=SOMETHING 2024-09-15T20:50:22.841 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1087: get_pg: ceph --format json osd map ecpool SOMETHING 2024-09-15T20:50:22.841 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1087: get_pg: jq -r .pgid 2024-09-15T20:50:23.120 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:237: corrupt_and_repair_one: local pg=2.0 2024-09-15T20:50:23.120 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:238: corrupt_and_repair_one: repair 2.0 2024-09-15T20:50:23.120 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1846: repair: local pgid=2.0 2024-09-15T20:50:23.120 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1847: repair: get_last_scrub_stamp 2.0 2024-09-15T20:50:23.120 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2024-09-15T20:50:23.120 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-15T20:50:23.120 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-15T20:50:23.120 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-09-15T20:50:23.391 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1847: repair: local last_scrub=2024-09-15T20:49:51.405772+0000 2024-09-15T20:50:23.391 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1848: repair: ceph pg repair 2.0 2024-09-15T20:50:23.652 INFO:tasks.workunit.client.0.smithi136.stderr:instructing pg 2.0s0 on osd.3 to repair 2024-09-15T20:50:23.664 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1849: repair: wait_for_scrub 2.0 2024-09-15T20:49:51.405772+0000 2024-09-15T20:50:23.664 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: local pgid=2.0 2024-09-15T20:50:23.664 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2018: wait_for_scrub: local last_scrub=2024-09-15T20:49:51.405772+0000 2024-09-15T20:50:23.664 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2019: wait_for_scrub: local sname=last_scrub_stamp 2024-09-15T20:50:23.664 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i=0 )) 2024-09-15T20:50:23.664 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-15T20:50:23.665 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-09-15T20:50:23.665 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2024-09-15T20:50:23.665 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-15T20:50:23.665 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-15T20:50:23.665 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-09-15T20:50:23.952 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-15T20:49:51.405772+0000 '>' 2024-09-15T20:49:51.405772+0000 2024-09-15T20:50:23.952 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-15T20:50:24.953 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-15T20:50:24.954 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-15T20:50:24.954 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-09-15T20:50:24.954 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2024-09-15T20:50:24.955 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-15T20:50:24.955 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-15T20:50:24.955 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-09-15T20:50:25.241 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-15T20:49:51.405772+0000 '>' 2024-09-15T20:49:51.405772+0000 2024-09-15T20:50:25.241 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-15T20:50:26.242 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-15T20:50:26.243 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-15T20:50:26.243 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-09-15T20:50:26.243 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2024-09-15T20:50:26.243 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-15T20:50:26.244 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-15T20:50:26.244 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-09-15T20:50:26.528 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-15T20:49:51.405772+0000 '>' 2024-09-15T20:49:51.405772+0000 2024-09-15T20:50:26.528 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-15T20:50:27.529 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-15T20:50:27.529 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-15T20:50:27.530 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-09-15T20:50:27.530 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2024-09-15T20:50:27.530 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-15T20:50:27.530 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-15T20:50:27.531 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-09-15T20:50:27.818 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-15T20:50:24.121965+0000 '>' 2024-09-15T20:49:51.405772+0000 2024-09-15T20:50:27.818 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2023: wait_for_scrub: return 0 2024-09-15T20:50:27.818 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1850: repair: sleep 2 2024-09-15T20:50:29.820 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:242: corrupt_and_repair_one: objectstore_tool td/osd-scrub-repair 3 SOMETHING list-attrs 2024-09-15T20:50:29.820 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2024-09-15T20:50:29.821 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2024-09-15T20:50:29.821 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=3 2024-09-15T20:50:29.821 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2024-09-15T20:50:29.821 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 3 SOMETHING list-attrs 2024-09-15T20:50:29.821 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-09-15T20:50:29.821 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2024-09-15T20:50:29.821 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=3 2024-09-15T20:50:29.821 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2024-09-15T20:50:29.821 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.3 2024-09-15T20:50:29.821 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-15T20:50:29.821 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-15T20:50:29.821 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-15T20:50:29.821 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-15T20:50:29.822 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-15T20:50:30.129 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-15T20:50:30.129 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 3 SOMETHING list-attrs 2024-09-15T20:50:30.129 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-15T20:50:30.129 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-15T20:50:30.129 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=3 2024-09-15T20:50:30.129 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-15T20:50:30.129 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/3 2024-09-15T20:50:30.129 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/3 SOMETHING list-attrs 2024-09-15T20:50:30.594 INFO:tasks.workunit.client.0.smithi136.stderr:Error getting attr on : 2.0s0_head,0#-4:00000000:::scrub_2.0s0:head#, (61) No data available 2024-09-15T20:50:30.595 INFO:tasks.workunit.client.0.smithi136.stdout:_ 2024-09-15T20:50:30.595 INFO:tasks.workunit.client.0.smithi136.stdout:hinfo_key 2024-09-15T20:50:30.595 INFO:tasks.workunit.client.0.smithi136.stdout:snapset 2024-09-15T20:50:30.862 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 3 2024-09-15T20:50:30.862 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2024-09-15T20:50:30.862 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-15T20:50:30.862 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=3 2024-09-15T20:50:30.862 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-15T20:50:30.862 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/3 2024-09-15T20:50:30.862 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=ce4892c7-6c6d-4b4f-8e6f-07a3e11133e5 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-15T20:50:30.863 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T20:50:30.863 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T20:50:30.863 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T20:50:30.863 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/3' 2024-09-15T20:50:30.863 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/3/journal' 2024-09-15T20:50:30.863 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-15T20:50:30.863 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-15T20:50:30.863 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-15T20:50:30.863 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-15T20:50:30.863 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-15T20:50:30.863 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-15T20:50:30.863 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-15T20:50:30.863 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T20:50:30.864 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52602 2024-09-15T20:50:30.864 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.52602/$cluster-$name.asok' 2024-09-15T20:50:30.864 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.52602/$cluster-$name.asok' 2024-09-15T20:50:30.864 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-15T20:50:30.864 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-15T20:50:30.864 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-15T20:50:30.864 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T20:50:30.864 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T20:50:30.864 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T20:50:30.864 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T20:50:30.864 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-15T20:50:30.864 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-15T20:50:30.864 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/3 2024-09-15T20:50:30.865 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.3 2024-09-15T20:50:30.865 INFO:tasks.workunit.client.0.smithi136.stderr:start osd.3 2024-09-15T20:50:30.865 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 3 --fsid=ce4892c7-6c6d-4b4f-8e6f-07a3e11133e5 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/3 --osd-journal=td/osd-scrub-repair/3/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.52602/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T20:50:30.865 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/3/whoami 2024-09-15T20:50:30.866 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 3 = 3 ']' 2024-09-15T20:50:30.866 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-15T20:50:30.866 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-15T20:50:30.866 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-15T20:50:30.893 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:50:30.890+0000 7f81c1c16640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T20:50:30.998 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:50:30.995+0000 7f81c1c16640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T20:50:30.999 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:50:30.997+0000 7f81c1c16640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T20:50:31.237 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 3 2024-09-15T20:50:31.237 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-15T20:50:31.238 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=3 2024-09-15T20:50:31.238 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-15T20:50:31.238 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-15T20:50:31.238 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:50:31.238 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-15T20:50:31.238 INFO:tasks.workunit.client.0.smithi136.stderr:0 2024-09-15T20:50:31.238 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:50:31.238 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-15T20:50:31.541 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:50:32.102 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:50:32.100+0000 7f81c1c16640 -1 Falling back to public interface 2024-09-15T20:50:32.543 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:50:32.543 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:50:32.543 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-15T20:50:32.543 INFO:tasks.workunit.client.0.smithi136.stderr:1 2024-09-15T20:50:32.543 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:50:32.543 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-15T20:50:32.830 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:50:33.831 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:50:33.832 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:50:33.832 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-15T20:50:33.832 INFO:tasks.workunit.client.0.smithi136.stderr:2 2024-09-15T20:50:33.832 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:50:33.832 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-15T20:50:34.136 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:50:34.962 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:50:34.960+0000 7f81c1c16640 -1 osd.3 59 log_to_monitors true 2024-09-15T20:50:35.138 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:50:35.138 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:50:35.138 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-15T20:50:35.138 INFO:tasks.workunit.client.0.smithi136.stderr:3 2024-09-15T20:50:35.138 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:50:35.138 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-15T20:50:35.469 INFO:tasks.workunit.client.0.smithi136.stderr:osd.3 up in weight 1 up_from 64 up_thru 57 down_at 60 last_clean_interval [50,59) [v2:127.0.0.1:6826/24175141,v1:127.0.0.1:6827/24175141] [v2:127.0.0.1:6828/24175141,v1:127.0.0.1:6829/24175141] exists,up 48807f11-9c3a-49d3-8033-a247f29eadfa 2024-09-15T20:50:35.469 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-15T20:50:35.470 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-15T20:50:35.470 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-15T20:50:35.470 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2024-09-15T20:50:35.470 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-15T20:50:35.470 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-15T20:50:35.470 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-15T20:50:35.470 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-15T20:50:35.470 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-15T20:50:35.470 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-15T20:50:35.470 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-15T20:50:35.470 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-15T20:50:35.470 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-15T20:50:35.559 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-15T20:50:35.559 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-15T20:50:35.559 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-15T20:50:35.559 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-15T20:50:35.559 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-15T20:50:35.559 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-15T20:50:35.922 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-15T20:50:35.922 INFO:tasks.workunit.client.0.smithi136.stderr:1 2024-09-15T20:50:35.922 INFO:tasks.workunit.client.0.smithi136.stderr:2 2024-09-15T20:50:35.922 INFO:tasks.workunit.client.0.smithi136.stderr:3' 2024-09-15T20:50:35.922 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-15T20:50:35.922 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T20:50:35.922 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-15T20:50:36.206 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836529 2024-09-15T20:50:36.206 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836529 2024-09-15T20:50:36.206 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836529' 2024-09-15T20:50:36.206 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T20:50:36.207 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-15T20:50:36.399 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=244813135879 2024-09-15T20:50:36.399 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 244813135879 2024-09-15T20:50:36.399 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836529 1-244813135879' 2024-09-15T20:50:36.399 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T20:50:36.399 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-15T20:50:36.564 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=64424509472 2024-09-15T20:50:36.564 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 64424509472 2024-09-15T20:50:36.564 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836529 1-244813135879 2-64424509472' 2024-09-15T20:50:36.565 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T20:50:36.565 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-15T20:50:36.667 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=274877906946 2024-09-15T20:50:36.668 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 274877906946 2024-09-15T20:50:36.668 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836529 1-244813135879 2-64424509472 3-274877906946' 2024-09-15T20:50:36.668 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T20:50:36.668 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836529 2024-09-15T20:50:36.668 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T20:50:36.669 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-15T20:50:36.669 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836529 2024-09-15T20:50:36.670 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T20:50:36.671 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836529 2024-09-15T20:50:36.671 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836529' 2024-09-15T20:50:36.671 INFO:tasks.workunit.client.0.smithi136.stderr:waiting osd.0 seq 21474836529 2024-09-15T20:50:36.671 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T20:50:37.014 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836527 -lt 21474836529 2024-09-15T20:50:37.014 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-15T20:50:38.015 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-15T20:50:38.016 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T20:50:38.299 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836529 -lt 21474836529 2024-09-15T20:50:38.300 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T20:50:38.300 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-244813135879 2024-09-15T20:50:38.300 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T20:50:38.301 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-15T20:50:38.301 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-244813135879 2024-09-15T20:50:38.302 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T20:50:38.302 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=244813135879 2024-09-15T20:50:38.302 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 244813135879' 2024-09-15T20:50:38.303 INFO:tasks.workunit.client.0.smithi136.stderr:waiting osd.1 seq 244813135879 2024-09-15T20:50:38.303 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-15T20:50:38.579 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 244813135879 -lt 244813135879 2024-09-15T20:50:38.579 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T20:50:38.579 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-64424509472 2024-09-15T20:50:38.579 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T20:50:38.580 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-15T20:50:38.581 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-64424509472 2024-09-15T20:50:38.581 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T20:50:38.582 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=64424509472 2024-09-15T20:50:38.582 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 64424509472' 2024-09-15T20:50:38.582 INFO:tasks.workunit.client.0.smithi136.stderr:waiting osd.2 seq 64424509472 2024-09-15T20:50:38.582 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-15T20:50:38.859 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 64424509472 -lt 64424509472 2024-09-15T20:50:38.859 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T20:50:38.859 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 3-274877906946 2024-09-15T20:50:38.859 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T20:50:38.860 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=3 2024-09-15T20:50:38.861 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 3-274877906946 2024-09-15T20:50:38.861 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T20:50:38.862 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=274877906946 2024-09-15T20:50:38.862 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.3 seq 274877906946' 2024-09-15T20:50:38.862 INFO:tasks.workunit.client.0.smithi136.stderr:waiting osd.3 seq 274877906946 2024-09-15T20:50:38.862 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-15T20:50:39.139 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 274877906946 -lt 274877906946 2024-09-15T20:50:39.140 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-15T20:50:39.140 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T20:50:39.140 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T20:50:39.487 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 5 == 0 2024-09-15T20:50:39.487 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-15T20:50:39.487 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-15T20:50:39.487 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-15T20:50:39.487 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T20:50:39.487 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T20:50:39.487 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T20:50:39.487 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T20:50:39.761 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=4 2024-09-15T20:50:39.762 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-15T20:50:39.762 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T20:50:39.762 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T20:50:40.106 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 = 5 2024-09-15T20:50:40.107 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 4 '!=' -1 2024-09-15T20:50:40.107 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: loop=0 2024-09-15T20:50:40.107 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1652: wait_for_clean: num_active_clean=4 2024-09-15T20:50:40.107 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-15T20:50:40.107 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 0.1 2024-09-15T20:50:40.207 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-15T20:50:40.207 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-15T20:50:40.208 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-15T20:50:40.208 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-15T20:50:40.208 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T20:50:40.208 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T20:50:40.208 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T20:50:40.209 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T20:50:40.478 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=4 2024-09-15T20:50:40.478 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-15T20:50:40.479 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T20:50:40.479 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T20:50:40.829 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 = 5 2024-09-15T20:50:40.829 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 4 '!=' 4 2024-09-15T20:50:40.829 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: get_is_making_recovery_progress 2024-09-15T20:50:40.829 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2024-09-15T20:50:40.829 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-15T20:50:40.829 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-15T20:50:40.829 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-15T20:50:40.829 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2024-09-15T20:50:40.829 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-15T20:50:41.173 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=null 2024-09-15T20:50:41.174 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test null '!=' null 2024-09-15T20:50:41.174 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1655: wait_for_clean: (( 1 >= 13 )) 2024-09-15T20:50:41.174 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-15T20:50:41.174 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 0.2 2024-09-15T20:50:41.374 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-15T20:50:41.375 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-15T20:50:41.376 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-15T20:50:41.376 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-15T20:50:41.376 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T20:50:41.376 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T20:50:41.376 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T20:50:41.376 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T20:50:41.646 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=5 2024-09-15T20:50:41.646 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-15T20:50:41.646 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T20:50:41.646 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T20:50:42.029 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 = 5 2024-09-15T20:50:42.029 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-15T20:50:42.029 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-15T20:50:42.029 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:243: corrupt_and_repair_one: rados --pool ecpool get SOMETHING td/osd-scrub-repair/COPY 2024-09-15T20:50:42.158 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:244: corrupt_and_repair_one: diff td/osd-scrub-repair/ORIGINAL td/osd-scrub-repair/COPY 2024-09-15T20:50:42.161 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:262: corrupt_and_repair_erasure_coded: corrupt_and_repair_two td/osd-scrub-repair ecpool 1 2 2024-09-15T20:50:42.161 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:185: corrupt_and_repair_two: local dir=td/osd-scrub-repair 2024-09-15T20:50:42.161 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:186: corrupt_and_repair_two: local poolname=ecpool 2024-09-15T20:50:42.161 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:187: corrupt_and_repair_two: local first=1 2024-09-15T20:50:42.161 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:188: corrupt_and_repair_two: local second=2 2024-09-15T20:50:42.161 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:193: corrupt_and_repair_two: pids= 2024-09-15T20:50:42.161 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:194: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 1 SOMETHING remove 2024-09-15T20:50:42.161 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2135: run_in_background: local pid_variable=pids 2024-09-15T20:50:42.162 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2136: run_in_background: shift 2024-09-15T20:50:42.162 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2140: run_in_background: eval 'pids+=" 149548"' 2024-09-15T20:50:42.162 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2140: run_in_background: pids+=' 149548' 2024-09-15T20:50:42.162 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:195: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 2 SOMETHING remove 2024-09-15T20:50:42.162 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2135: run_in_background: local pid_variable=pids 2024-09-15T20:50:42.162 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2136: run_in_background: shift 2024-09-15T20:50:42.162 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2140: run_in_background: eval 'pids+=" 149550"' 2024-09-15T20:50:42.162 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2140: run_in_background: pids+=' 149550' 2024-09-15T20:50:42.162 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:196: corrupt_and_repair_two: wait_background pids 2024-09-15T20:50:42.162 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: run_in_background: objectstore_tool td/osd-scrub-repair 1 SOMETHING remove 2024-09-15T20:50:42.162 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: wait_background: pids=' 149548 149550' 2024-09-15T20:50:42.162 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: wait_background: return_code=0 2024-09-15T20:50:42.163 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: wait_background: for pid in $pids 2024-09-15T20:50:42.163 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2175: wait_background: wait 149548 2024-09-15T20:50:42.163 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: run_in_background: sed 's/^/149551: /' 2024-09-15T20:50:42.163 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: run_in_background: objectstore_tool td/osd-scrub-repair 2 SOMETHING remove 2024-09-15T20:50:42.163 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: run_in_background: sed 's/^/149553: /' 2024-09-15T20:50:45.182 INFO:tasks.workunit.client.0.smithi136.stderr:149551: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2024-09-15T20:50:45.182 INFO:tasks.workunit.client.0.smithi136.stderr:149551: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2024-09-15T20:50:45.182 INFO:tasks.workunit.client.0.smithi136.stderr:149551: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=1 2024-09-15T20:50:45.182 INFO:tasks.workunit.client.0.smithi136.stderr:149551: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2024-09-15T20:50:45.182 INFO:tasks.workunit.client.0.smithi136.stderr:149551: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 SOMETHING remove 2024-09-15T20:50:45.182 INFO:tasks.workunit.client.0.smithi136.stderr:149551: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-09-15T20:50:45.182 INFO:tasks.workunit.client.0.smithi136.stderr:149551: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2024-09-15T20:50:45.182 INFO:tasks.workunit.client.0.smithi136.stderr:149551: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=1 2024-09-15T20:50:45.182 INFO:tasks.workunit.client.0.smithi136.stderr:149551: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2024-09-15T20:50:45.182 INFO:tasks.workunit.client.0.smithi136.stderr:149551: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-09-15T20:50:45.182 INFO:tasks.workunit.client.0.smithi136.stderr:149551: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-15T20:50:45.182 INFO:tasks.workunit.client.0.smithi136.stderr:149551: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-15T20:50:45.183 INFO:tasks.workunit.client.0.smithi136.stderr:149551: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-15T20:50:45.183 INFO:tasks.workunit.client.0.smithi136.stderr:149551: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-15T20:50:45.183 INFO:tasks.workunit.client.0.smithi136.stderr:149551: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-15T20:50:45.183 INFO:tasks.workunit.client.0.smithi136.stderr:149551: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-15T20:50:45.183 INFO:tasks.workunit.client.0.smithi136.stderr:149551: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 SOMETHING remove 2024-09-15T20:50:45.183 INFO:tasks.workunit.client.0.smithi136.stderr:149551: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-15T20:50:45.183 INFO:tasks.workunit.client.0.smithi136.stderr:149551: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-15T20:50:45.183 INFO:tasks.workunit.client.0.smithi136.stderr:149551: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=1 2024-09-15T20:50:45.183 INFO:tasks.workunit.client.0.smithi136.stderr:149551: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-15T20:50:45.183 INFO:tasks.workunit.client.0.smithi136.stderr:149551: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-09-15T20:50:45.183 INFO:tasks.workunit.client.0.smithi136.stderr:149551: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 SOMETHING remove 2024-09-15T20:50:45.183 INFO:tasks.workunit.client.0.smithi136.stderr:149551: remove 1#2:eb822e21:::SOMETHING:head# 2024-09-15T20:50:45.183 INFO:tasks.workunit.client.0.smithi136.stderr:149551: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2024-09-15T20:50:45.184 INFO:tasks.workunit.client.0.smithi136.stderr:149551: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2024-09-15T20:50:45.184 INFO:tasks.workunit.client.0.smithi136.stderr:149551: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-15T20:50:45.184 INFO:tasks.workunit.client.0.smithi136.stderr:149551: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=1 2024-09-15T20:50:45.184 INFO:tasks.workunit.client.0.smithi136.stderr:149551: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-15T20:50:45.184 INFO:tasks.workunit.client.0.smithi136.stderr:149551: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-09-15T20:50:45.184 INFO:tasks.workunit.client.0.smithi136.stderr:149551: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=ce4892c7-6c6d-4b4f-8e6f-07a3e11133e5 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-15T20:50:45.184 INFO:tasks.workunit.client.0.smithi136.stderr:149551: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T20:50:45.185 INFO:tasks.workunit.client.0.smithi136.stderr:149551: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T20:50:45.185 INFO:tasks.workunit.client.0.smithi136.stderr:149551: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T20:50:45.185 INFO:tasks.workunit.client.0.smithi136.stderr:149551: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-09-15T20:50:45.185 INFO:tasks.workunit.client.0.smithi136.stderr:149551: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-09-15T20:50:45.185 INFO:tasks.workunit.client.0.smithi136.stderr:149551: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-15T20:50:45.185 INFO:tasks.workunit.client.0.smithi136.stderr:149551: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-15T20:50:45.186 INFO:tasks.workunit.client.0.smithi136.stderr:149551: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-15T20:50:45.186 INFO:tasks.workunit.client.0.smithi136.stderr:149551: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-15T20:50:45.186 INFO:tasks.workunit.client.0.smithi136.stderr:149551: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-15T20:50:45.186 INFO:tasks.workunit.client.0.smithi136.stderr:149551: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-15T20:50:45.186 INFO:tasks.workunit.client.0.smithi136.stderr:149551: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-15T20:50:45.186 INFO:tasks.workunit.client.0.smithi136.stderr:149551: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T20:50:45.186 INFO:tasks.workunit.client.0.smithi136.stderr:149551: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52602 2024-09-15T20:50:45.186 INFO:tasks.workunit.client.0.smithi136.stderr:149551: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.52602/$cluster-$name.asok' 2024-09-15T20:50:45.186 INFO:tasks.workunit.client.0.smithi136.stderr:149551: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.52602/$cluster-$name.asok' 2024-09-15T20:50:45.186 INFO:tasks.workunit.client.0.smithi136.stderr:149551: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-15T20:50:45.186 INFO:tasks.workunit.client.0.smithi136.stderr:149551: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-15T20:50:45.186 INFO:tasks.workunit.client.0.smithi136.stderr:149551: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-15T20:50:45.186 INFO:tasks.workunit.client.0.smithi136.stderr:149551: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T20:50:45.187 INFO:tasks.workunit.client.0.smithi136.stderr:149551: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T20:50:45.187 INFO:tasks.workunit.client.0.smithi136.stderr:149551: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T20:50:45.187 INFO:tasks.workunit.client.0.smithi136.stderr:149551: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T20:50:45.187 INFO:tasks.workunit.client.0.smithi136.stderr:149551: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-15T20:50:45.187 INFO:tasks.workunit.client.0.smithi136.stderr:149551: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-15T20:50:45.187 INFO:tasks.workunit.client.0.smithi136.stderr:149551: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-09-15T20:50:45.187 INFO:tasks.workunit.client.0.smithi136.stderr:149551: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.1 2024-09-15T20:50:45.187 INFO:tasks.workunit.client.0.smithi136.stderr:149551: start osd.1 2024-09-15T20:50:45.187 INFO:tasks.workunit.client.0.smithi136.stderr:149551: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 1 --fsid=ce4892c7-6c6d-4b4f-8e6f-07a3e11133e5 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.52602/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T20:50:45.351 INFO:tasks.workunit.client.0.smithi136.stderr:1149553: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2024-09-15T20:50:45.351 INFO:tasks.workunit.client.0.smithi136.stderr:149553: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2024-09-15T20:50:45.351 INFO:tasks.workunit.client.0.smithi136.stderr:149553: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=2 2024-09-15T20:50:45.351 INFO:tasks.workunit.client.0.smithi136.stderr:149553: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2024-09-15T20:50:45.351 INFO:tasks.workunit.client.0.smithi136.stderr:149553: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 2 SOMETHING remove 2024-09-15T20:50:45.351 INFO:tasks.workunit.client.0.smithi136.stderr:149553: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-09-15T20:50:45.351 INFO:tasks.workunit.client.0.smithi136.stderr:149553: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2024-09-15T20:50:45.351 INFO:tasks.workunit.client.0.smithi136.stderr:149553: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=2 2024-09-15T20:50:45.352 INFO:tasks.workunit.client.0.smithi136.stderr:149553: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2024-09-15T20:50:45.352 INFO:tasks.workunit.client.0.smithi136.stderr:149553: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.2 2024-09-15T20:50:45.352 INFO:tasks.workunit.client.0.smithi136.stderr:149553: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-15T20:50:45.352 INFO:tasks.workunit.client.0.smithi136.stderr:149553: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-15T20:50:45.352 INFO:tasks.workunit.client.0.smithi136.stderr:149553: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-15T20:50:45.352 INFO:tasks.workunit.client.0.smithi136.stderr:149553: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-15T20:50:45.352 INFO:tasks.workunit.client.0.smithi136.stderr:149553: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-15T20:50:45.352 INFO:tasks.workunit.client.0.smithi136.stderr:149553: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-15T20:50:45.352 INFO:tasks.workunit.client.0.smithi136.stderr:149553: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 2 SOMETHING remove 2024-09-15T20:50:45.352 INFO:tasks.workunit.client.0.smithi136.stderr:149553: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-15T20:50:45.352 INFO:tasks.workunit.client.0.smithi136.stderr:149553: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-15T20:50:45.352 INFO:tasks.workunit.client.0.smithi136.stderr:149553: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=2 2024-09-15T20:50:45.352 INFO:tasks.workunit.client.0.smithi136.stderr:149553: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-15T20:50:45.353 INFO:tasks.workunit.client.0.smithi136.stderr:149553: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/2 2024-09-15T20:50:45.353 INFO:tasks.workunit.client.0.smithi136.stderr:149553: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/2 SOMETHING remove 2024-09-15T20:50:45.353 INFO:tasks.workunit.client.0.smithi136.stderr:149553: remove 2#2:eb822e21:::SOMETHING:head# 2024-09-15T20:50:45.353 INFO:tasks.workunit.client.0.smithi136.stderr:149553: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 2 2024-09-15T20:50:45.353 INFO:tasks.workunit.client.0.smithi136.stderr:149553: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2024-09-15T20:50:45.353 INFO:tasks.workunit.client.0.smithi136.stderr:149553: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-15T20:50:45.353 INFO:tasks.workunit.client.0.smithi136.stderr:149553: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=2 2024-09-15T20:50:45.353 INFO:tasks.workunit.client.0.smithi136.stderr:149553: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-15T20:50:45.353 INFO:tasks.workunit.client.0.smithi136.stderr:149553: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/2 2024-09-15T20:50:45.353 INFO:tasks.workunit.client.0.smithi136.stderr:149553: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=ce4892c7-6c6d-4b4f-8e6f-07a3e11133e5 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-15T20:50:45.353 INFO:tasks.workunit.client.0.smithi136.stderr:149553: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T20:50:45.354 INFO:tasks.workunit.client.0.smithi136.stderr:149553: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T20:50:45.354 INFO:tasks.workunit.client.0.smithi136.stderr:149553: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T20:50:45.354 INFO:tasks.workunit.client.0.smithi136.stderr:149553: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/2' 2024-09-15T20:50:45.354 INFO:tasks.workunit.client.0.smithi136.stderr:149553: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/2/journal' 2024-09-15T20:50:45.354 INFO:tasks.workunit.client.0.smithi136.stderr:149553: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-15T20:50:45.354 INFO:tasks.workunit.client.0.smithi136.stderr:149553: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-15T20:50:45.354 INFO:tasks.workunit.client.0.smithi136.stderr:149553: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-15T20:50:45.354 INFO:tasks.workunit.client.0.smithi136.stderr:149553: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-15T20:50:45.354 INFO:tasks.workunit.client.0.smithi136.stderr:149553: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-15T20:50:45.354 INFO:tasks.workunit.client.0.smithi136.stderr:149553: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-15T20:50:45.354 INFO:tasks.workunit.client.0.smithi136.stderr:149553: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-15T20:50:45.354 INFO:tasks.workunit.client.0.smithi136.stderr:149553: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T20:50:45.355 INFO:tasks.workunit.client.0.smithi136.stderr:149553: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52602 2024-09-15T20:50:45.355 INFO:tasks.workunit.client.0.smithi136.stderr:149553: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.52602/$cluster-$name.asok' 2024-09-15T20:50:45.355 INFO:tasks.workunit.client.0.smithi136.stderr:149553: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.52602/$cluster-$name.asok' 2024-09-15T20:50:45.355 INFO:tasks.workunit.client.0.smithi136.stderr:149553: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-15T20:50:45.355 INFO:tasks.workunit.client.0.smithi136.stderr:149553: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-15T20:50:45.355 INFO:tasks.workunit.client.0.smithi136.stderr:149553: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-15T20:50:45.355 INFO:tasks.workunit.client.0.smithi136.stderr:149553: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T20:50:45.355 INFO:tasks.workunit.client.0.smithi136.stderr:149553: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T20:50:45.355 INFO:tasks.workunit.client.0.smithi136.stderr:149553: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T20:50:45.355 INFO:tasks.workunit.client.0.smithi136.stderr:149553: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T20:50:45.355 INFO:tasks.workunit.client.0.smithi136.stderr:149553: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-15T20:50:45.355 INFO:tasks.workunit.client.0.smithi136.stderr:149553: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-15T20:50:45.355 INFO:tasks.workunit.client.0.smithi136.stderr:149553: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/2 2024-09-15T20:50:45.355 INFO:tasks.workunit.client.0.smithi136.stderr:149553: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.2 2024-09-15T20:50:45.355 INFO:tasks.workunit.client.0.smithi136.stderr:149553: start osd.2 2024-09-15T20:50:45.356 INFO:tasks.workunit.client.0.smithi136.stderr:149553: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 2 --fsid=ce4892c7-6c6d-4b4f-8e6f-07a3e11133e5 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.52602/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T20:50:49.771 INFO:tasks.workunit.client.0.smithi136.stderr:149551: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-09-15T20:50:49.772 INFO:tasks.workunit.client.0.smithi136.stderr:149551: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 1 = 1 ']' 2024-09-15T20:50:49.772 INFO:tasks.workunit.client.0.smithi136.stderr:149551: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-15T20:50:49.772 INFO:tasks.workunit.client.0.smithi136.stderr:149551: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-15T20:50:49.772 INFO:tasks.workunit.client.0.smithi136.stderr:149551: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-15T20:50:49.772 INFO:tasks.workunit.client.0.smithi136.stderr:149551: 2024-09-15T20:50:45.213+0000 7fc165664640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T20:50:49.772 INFO:tasks.workunit.client.0.smithi136.stderr:149551: 2024-09-15T20:50:45.216+0000 7fc165664640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T20:50:49.772 INFO:tasks.workunit.client.0.smithi136.stderr:149551: 2024-09-15T20:50:45.217+0000 7fc165664640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T20:50:49.772 INFO:tasks.workunit.client.0.smithi136.stderr:149551: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 1 2024-09-15T20:50:49.772 INFO:tasks.workunit.client.0.smithi136.stderr:149551: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-15T20:50:49.772 INFO:tasks.workunit.client.0.smithi136.stderr:149551: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2024-09-15T20:50:49.772 INFO:tasks.workunit.client.0.smithi136.stderr:149551: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-15T20:50:49.772 INFO:tasks.workunit.client.0.smithi136.stderr:149551: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-15T20:50:49.772 INFO:tasks.workunit.client.0.smithi136.stderr:149551: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:50:49.773 INFO:tasks.workunit.client.0.smithi136.stderr:149551: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-15T20:50:49.773 INFO:tasks.workunit.client.0.smithi136.stderr:149551: 0 2024-09-15T20:50:49.773 INFO:tasks.workunit.client.0.smithi136.stderr:149551: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:50:49.773 INFO:tasks.workunit.client.0.smithi136.stderr:149551: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T20:50:49.773 INFO:tasks.workunit.client.0.smithi136.stderr:149551: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:50:49.773 INFO:tasks.workunit.client.0.smithi136.stderr:149551: 2024-09-15T20:50:46.329+0000 7fc165664640 -1 Falling back to public interface 2024-09-15T20:50:49.773 INFO:tasks.workunit.client.0.smithi136.stderr:149551: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:50:49.773 INFO:tasks.workunit.client.0.smithi136.stderr:149551: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:50:49.773 INFO:tasks.workunit.client.0.smithi136.stderr:149551: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-15T20:50:49.773 INFO:tasks.workunit.client.0.smithi136.stderr:149551: 1 2024-09-15T20:50:49.773 INFO:tasks.workunit.client.0.smithi136.stderr:149551: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:50:49.773 INFO:tasks.workunit.client.0.smithi136.stderr:149551: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T20:50:49.773 INFO:tasks.workunit.client.0.smithi136.stderr:149551: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:50:49.773 INFO:tasks.workunit.client.0.smithi136.stderr:149551: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:50:49.774 INFO:tasks.workunit.client.0.smithi136.stderr:149551: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:50:49.774 INFO:tasks.workunit.client.0.smithi136.stderr:149551: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-15T20:50:49.774 INFO:tasks.workunit.client.0.smithi136.stderr:149551: 2 2024-09-15T20:50:49.774 INFO:tasks.workunit.client.0.smithi136.stderr:149551: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:50:49.774 INFO:tasks.workunit.client.0.smithi136.stderr:149551: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T20:50:49.774 INFO:tasks.workunit.client.0.smithi136.stderr:149551: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:50:49.774 INFO:tasks.workunit.client.0.smithi136.stderr:149551: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:50:49.774 INFO:tasks.workunit.client.0.smithi136.stderr:149551: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:50:49.774 INFO:tasks.workunit.client.0.smithi136.stderr:149551: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-15T20:50:49.774 INFO:tasks.workunit.client.0.smithi136.stderr:149551: 3 2024-09-15T20:50:49.774 INFO:tasks.workunit.client.0.smithi136.stderr:149551: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:50:49.774 INFO:tasks.workunit.client.0.smithi136.stderr:149551: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T20:50:49.774 INFO:tasks.workunit.client.0.smithi136.stderr:149551: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:50:49.940 INFO:tasks.workunit.client.0.smithi136.stderr:149551: 2024-09-15T20:50:49.769+0000 7fc165664640 -49553: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/2/whoami 2024-09-15T20:50:49.941 INFO:tasks.workunit.client.0.smithi136.stderr:149553: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 2 = 2 ']' 2024-09-15T20:50:49.941 INFO:tasks.workunit.client.0.smithi136.stderr:149553: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-15T20:50:49.941 INFO:tasks.workunit.client.0.smithi136.stderr:149553: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-15T20:50:49.941 INFO:tasks.workunit.client.0.smithi136.stderr:149553: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-15T20:50:49.941 INFO:tasks.workunit.client.0.smithi136.stderr:149553: 2024-09-15T20:50:45.378+0000 7f8cec5a2640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T20:50:49.941 INFO:tasks.workunit.client.0.smithi136.stderr:149553: 2024-09-15T20:50:45.381+0000 7f8cec5a2640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T20:50:49.941 INFO:tasks.workunit.client.0.smithi136.stderr:149553: 2024-09-15T20:50:45.382+0000 7f8cec5a2640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T20:50:49.941 INFO:tasks.workunit.client.0.smithi136.stderr:149553: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 2 2024-09-15T20:50:49.941 INFO:tasks.workunit.client.0.smithi136.stderr:149553: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-15T20:50:49.941 INFO:tasks.workunit.client.0.smithi136.stderr:149553: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=2 2024-09-15T20:50:49.941 INFO:tasks.workunit.client.0.smithi136.stderr:149553: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-15T20:50:49.941 INFO:tasks.workunit.client.0.smithi136.stderr:149553: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-15T20:50:49.941 INFO:tasks.workunit.client.0.smithi136.stderr:149553: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:50:49.941 INFO:tasks.workunit.client.0.smithi136.stderr:149553: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-15T20:50:49.942 INFO:tasks.workunit.client.0.smithi136.stderr:149553: 0 2024-09-15T20:50:49.942 INFO:tasks.workunit.client.0.smithi136.stderr:149553: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:50:49.942 INFO:tasks.workunit.client.0.smithi136.stderr:149553: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T20:50:49.942 INFO:tasks.workunit.client.0.smithi136.stderr:149553: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:50:49.942 INFO:tasks.workunit.client.0.smithi136.stderr:149553: 2024-09-15T20:50:46.502+0000 7f8cec5a2640 -1 Falling back to public interface 2024-09-15T20:50:49.942 INFO:tasks.workunit.client.0.smithi136.stderr:149553: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:50:49.942 INFO:tasks.workunit.client.0.smithi136.stderr:149553: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:50:49.942 INFO:tasks.workunit.client.0.smithi136.stderr:149553: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-15T20:50:49.942 INFO:tasks.workunit.client.0.smithi136.stderr:149553: 1 2024-09-15T20:50:49.942 INFO:tasks.workunit.client.0.smithi136.stderr:149553: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:50:49.942 INFO:tasks.workunit.client.0.smithi136.stderr:149553: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T20:50:49.942 INFO:tasks.workunit.client.0.smithi136.stderr:149553: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:50:49.942 INFO:tasks.workunit.client.0.smithi136.stderr:149553: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:50:49.943 INFO:tasks.workunit.client.0.smithi136.stderr:149553: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:50:49.943 INFO:tasks.workunit.client.0.smithi136.stderr:149553: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-15T20:50:49.943 INFO:tasks.workunit.client.0.smithi136.stderr:149553: 2 2024-09-15T20:50:49.943 INFO:tasks.workunit.client.0.smithi136.stderr:149553: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:50:49.943 INFO:tasks.workunit.client.0.smithi136.stderr:149553: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T20:50:49.943 INFO:tasks.workunit.client.0.smithi136.stderr:149553: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:50:49.943 INFO:tasks.workunit.client.0.smithi136.stderr:149553: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:50:49.943 INFO:tasks.workunit.client.0.smithi136.stderr:149553: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:50:49.943 INFO:tasks.workunit.client.0.smithi136.stderr:149553: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-15T20:50:49.943 INFO:tasks.workunit.client.0.smithi136.stderr:149553: 3 2024-09-15T20:50:49.943 INFO:tasks.workunit.client.0.smithi136.stderr:149553: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:50:49.943 INFO:tasks.workunit.client.0.smithi136.stderr:149553: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T20:50:49.943 INFO:tasks.workunit.client.0.smithi136.stderr:149553: 2024-09-15T20:50:49.677+0000 7f8cec5a2640 -1 osd.2 66 log_to_monitors true 2024-09-15T20:50:51.793 INFO:tasks.workunit.client.0.smithi136.stderr:149553: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers1 osd.1 66 log_to_monitors true 2024-09-15T20:50:51.793 INFO:tasks.workunit.client.0.smithi136.stderr:149551: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:50:51.794 INFO:tasks.workunit.client.0.smithi136.stderr:149551: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:50:51.794 INFO:tasks.workunit.client.0.smithi136.stderr:149551: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-15T20:50:51.794 INFO:tasks.workunit.client.0.smithi136.stderr:149551: 4 2024-09-15T20:50:51.794 INFO:tasks.workunit.client.0.smithi136.stderr:149551: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:50:51.794 INFO:tasks.workunit.client.0.smithi136.stderr:149551: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T20:50:51.794 INFO:tasks.workunit.client.0.smithi136.stderr:149551: osd.1 up in weight 1 up_from 73 up_thru 60 down_at 67 last_clean_interval [57,66) [v2:127.0.0.1:6810/2780829717,v1:127.0.0.1:6811/2780829717] [v2:127.0.0.1:6812/2780829717,v1:127.0.0.1:6813/2780829717] exists,up ba9ca9c7-78b0-44cd-b086-00a47c922481 2024-09-15T20:50:51.794 INFO:tasks.workunit.client.0.smithi136.stderr:149551: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-15T20:50:51.794 INFO:tasks.workunit.client.0.smithi136.stderr:149551: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-15T20:50:51.794 INFO:tasks.workunit.client.0.smithi136.stderr:149551: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-15T20:50:51.794 INFO:tasks.workunit.client.0.smithi136.stderr:149551: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2024-09-15T20:50:51.794 INFO:tasks.workunit.client.0.smithi136.stderr:149551: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-15T20:50:51.794 INFO:tasks.workunit.client.0.smithi136.stderr:149551: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-15T20:50:51.795 INFO:tasks.workunit.client.0.smithi136.stderr:149551: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-15T20:50:51.795 INFO:tasks.workunit.client.0.smithi136.stderr:149551: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-15T20:50:51.795 INFO:tasks.workunit.client.0.smithi136.stderr:149551: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-15T20:50:51.795 INFO:tasks.workunit.client.0.smithi136.stderr:149551: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-15T20:50:51.795 INFO:tasks.workunit.client.0.smithi136.stderr:149551: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-15T20:50:51.795 INFO:tasks.workunit.client.0.smithi136.stderr:149551: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-15T20:50:51.795 INFO:tasks.workunit.client.0.smithi136.stderr:149551: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-15T20:50:51.795 INFO:tasks.workunit.client.0.smithi136.stderr:149551: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-15T20:50:51.795 INFO:tasks.workunit.client.0.smithi136.stderr:149551: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-15T20:50:51.795 INFO:tasks.workunit.client.0.smithi136.stderr:149551: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-15T20:50:51.795 INFO:tasks.workunit.client.0.smithi136.stderr:149551: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-15T20:50:51.795 INFO:tasks.workunit.client.0.smithi136.stderr:149551: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-15T20:50:51.796 INFO:tasks.workunit.client.0.smithi136.stderr:149551: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-15T20:50:51.796 INFO:tasks.workunit.client.0.smithi136.stderr:149551: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-15T20:50:51.796 INFO:tasks.workunit.client.0.smithi136.stderr:149551: 1 2024-09-15T20:50:51.796 INFO:tasks.workunit.client.0.smithi136.stderr:149551: 2 2024-09-15T20:50:51.796 INFO:tasks.workunit.client.0.smithi136.stderr:149551: 3' 2024-09-15T20:50:51.796 INFO:tasks.workunit.client.0.smithi136.stderr:149551: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-15T20:50:51.796 INFO:tasks.workunit.client.0.smithi136.stderr:149551: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T20:50:51.796 INFO:tasks.workunit.client.0.smithi136.stderr:149551: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-15T20:50:51.796 INFO:tasks.workunit.client.0.smithi136.stderr:149551: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836533 2024-09-15T20:50:51.796 INFO:tasks.workunit.client.0.smithi136.stderr:149551: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836533 2024-09-15T20:50:51.796 INFO:tasks.workunit.client.0.smithi136.stderr:149551: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836533' 2024-09-15T20:50:51.796 INFO:tasks.workunit.client.0.smithi136.stderr:149551: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T20:50:51.982 INFO:tasks.workunit.client.0.smithi136.stderr:149551: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-help.sh:976: wait_for_osd: sleep 1 2024-09-15T20:50:51.982 INFO:tasks.workunit.client.0.smithi136.stderr:149553: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:50:51.982 INFO:tasks.workunit.client.0.smithi136.stderr:149553: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:50:51.982 INFO:tasks.workunit.client.0.smithi136.stderr:149553: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-15T20:50:51.982 INFO:tasks.workunit.client.0.smithi136.stderr:149553: 4 2024-09-15T20:50:51.982 INFO:tasks.workunit.client.0.smithi136.stderr:149553: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:50:51.982 INFO:tasks.workunit.client.0.smithi136.stderr:149553: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T20:50:51.983 INFO:tasks.workunit.client.0.smithi136.stderr:149553: osd.2 up in weight 1 up_from 72 up_thru 72 down_at 67 last_clean_interval [15,66) [v2:127.0.0.1:6818/1468525487,v1:127.0.0.1:6819/1468525487] [v2:127.0.0.1:6820/1468525487,v1:127.0.0.1:6821/1468525487] exists,up 6eed0205-77e5-4f0f-9c89-cc7347b56baa 2024-09-15T20:50:51.983 INFO:tasks.workunit.client.0.smithi136.stderr:149553: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-15T20:50:51.983 INFO:tasks.workunit.client.0.smithi136.stderr:149553: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-15T20:50:51.983 INFO:tasks.workunit.client.0.smithi136.stderr:149553: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-15T20:50:51.983 INFO:tasks.workunit.client.0.smithi136.stderr:149553: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2024-09-15T20:50:51.983 INFO:tasks.workunit.client.0.smithi136.stderr:149553: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-15T20:50:51.983 INFO:tasks.workunit.client.0.smithi136.stderr:149553: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-15T20:50:51.983 INFO:tasks.workunit.client.0.smithi136.stderr:149553: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-15T20:50:51.983 INFO:tasks.workunit.client.0.smithi136.stderr:149553: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-15T20:50:51.983 INFO:tasks.workunit.client.0.smithi136.stderr:149553: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-15T20:50:51.983 INFO:tasks.workunit.client.0.smithi136.stderr:149553: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-15T20:50:51.984 INFO:tasks.workunit.client.0.smithi136.stderr:149553: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-15T20:50:51.984 INFO:tasks.workunit.client.0.smithi136.stderr:149553: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-15T20:50:51.984 INFO:tasks.workunit.client.0.smithi136.stderr:149553: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-15T20:50:51.984 INFO:tasks.workunit.client.0.smithi136.stderr:149553: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-15T20:50:51.984 INFO:tasks.workunit.client.0.smithi136.stderr:149553: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-15T20:50:51.984 INFO:tasks.workunit.client.0.smithi136.stderr:149553: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-15T20:50:51.984 INFO:tasks.workunit.client.0.smithi136.stderr:149553: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-15T20:50:51.984 INFO:tasks.workunit.client.0.smithi136.stderr:149553: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-15T20:50:51.984 INFO:tasks.workunit.client.0.smithi136.stderr:149553: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-15T20:50:51.984 INFO:tasks.workunit.client.0.smithi136.stderr:149553: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-15T20:50:51.984 INFO:tasks.workunit.client.0.smithi136.stderr:149553: 1 2024-09-15T20:50:51.984 INFO:tasks.workunit.client.0.smithi136.stderr:149553: 2 2024-09-15T20:50:51.985 INFO:tasks.workunit.client.0.smithi136.stderr:149553: 3' 2024-09-15T20:50:51.985 INFO:tasks.workunit.client.0.smithi136.stderr:149553: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-15T20:50:51.985 INFO:tasks.workunit.client.0.smithi136.stderr:149553: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T20:50:51.985 INFO:tasks.workunit.client.0.smithi136.stderr:149553: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-15T20:50:51.985 INFO:tasks.workunit.client.0.smithi136.stderr:149553: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836534 2024-09-15T20:50:51.985 INFO:tasks.workunit.client.0.smithi136.stderr:149553: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836534 2024-09-15T20:50:51.985 INFO:tasks.workunit.client.0.smithi136.stderr:149553: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836534' 2024-09-15T20:50:51.985 INFO:tasks.workunit.client.0.smithi136.stderr:149553: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T20:50:53.983 INFO:tasks.workunit.client.0.smithi136.stderr:149553: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-15T20:50:53.984 INFO:tasks.workunit.client.0.smithi136.stderr:149551: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=313532612610 2024-09-15T20:50:53.984 INFO:tasks.workunit.client.0.smithi136.stderr:149551: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 313532612610 2024-09-15T20:50:53.984 INFO:tasks.workunit.client.0.smithi136.stderr:149551: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836533 1-313532612610' 2024-09-15T20:50:53.984 INFO:tasks.workunit.client.0.smithi136.stderr:149551: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T20:50:53.984 INFO:tasks.workunit.client.0.smithi136.stderr:149551: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-15T20:50:53.984 INFO:tasks.workunit.client.0.smithi136.stderr:149551: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=309237645314 2024-09-15T20:50:53.984 INFO:tasks.workunit.client.0.smithi136.stderr:149551: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 309237645314 2024-09-15T20:50:53.984 INFO:tasks.workunit.client.0.smithi136.stderr:149551: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836533 1-313532612610 2-309237645314' 2024-09-15T20:50:53.984 INFO:tasks.workunit.client.0.smithi136.stderr:149551: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T20:50:53.984 INFO:tasks.workunit.client.0.smithi136.stderr:149551: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-15T20:50:53.985 INFO:tasks.workunit.client.0.smithi136.stderr:149551: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=274877906950 2024-09-15T20:50:53.985 INFO:tasks.workunit.client.0.smithi136.stderr:149551: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 274877906950 2024-09-15T20:50:53.985 INFO:tasks.workunit.client.0.smithi136.stderr:149551: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836533 1-313532612610 2-309237645314 3-274877906950' 2024-09-15T20:50:53.985 INFO:tasks.workunit.client.0.smithi136.stderr:149551: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T20:50:53.985 INFO:tasks.workunit.client.0.smithi136.stderr:149551: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836533 2024-09-15T20:50:53.985 INFO:tasks.workunit.client.0.smithi136.stderr:149551: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T20:50:53.985 INFO:tasks.workunit.client.0.smithi136.stderr:149551: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-15T20:50:53.985 INFO:tasks.workunit.client.0.smithi136.stderr:149551: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836533 2024-09-15T20:50:53.985 INFO:tasks.workunit.client.0.smithi136.stderr:149551: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T20:50:53.985 INFO:tasks.workunit.client.0.smithi136.stderr:149551: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836533 2024-09-15T20:50:53.985 INFO:tasks.workunit.client.0.smithi136.stderr:149551: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836533' 2024-09-15T20:50:53.985 INFO:tasks.workunit.client.0.smithi136.stderr:149551: waiting osd.0 seq 21474836533 2024-09-15T20:50:53.986 INFO:tasks.workunit.client.0.smithi136.stderr:149551: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T20:50:53.986 INFO:tasks.workunit.client.0.smithi136.stderr:149551: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836532 -lt 21474836533 2024-09-15T20:50:53.986 INFO:tasks.workunit.client.0.smithi136.stderr:149551: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-15T20:50:53.986 INFO:tasks.workunit.client.0.smithi136.stderr:149551: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-15T20:50:53.986 INFO:tasks.workunit.client.0.smithi136.stderr:149551: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T20:50:53.986 INFO:tasks.workunit.client.0.smithi136.stderr:149551: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836534 -lt 21474836533 2024-09-15T20:50:53.986 INFO:tasks.workunit.client.0.smithi136.stderr:149551: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T20:50:53.986 INFO:tasks.workunit.client.0.smithi136.stderr:149551: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-313532612610 2024-09-15T20:50:53.986 INFO:tasks.workunit.client.0.smithi136.stderr:149551: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T20:50:53.986 INFO:tasks.workunit.client.0.smithi136.stderr:149551: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-15T20:50:53.986 INFO:tasks.workunit.client.0.smithi136.stderr:149551: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-313532612610 2024-09-15T20:50:53.986 INFO:tasks.workunit.client.0.smithi136.stderr:149551: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T20:50:54.122 INFO:tasks.workunit.client.0.smithi136.stderr:149551: /home/ubers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-15T20:50:54.122 INFO:tasks.workunit.client.0.smithi136.stderr:149553: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=313532612611 2024-09-15T20:50:54.122 INFO:tasks.workunit.client.0.smithi136.stderr:149553: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 313532612611 2024-09-15T20:50:54.122 INFO:tasks.workunit.client.0.smithi136.stderr:149553: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836534 1-313532612611' 2024-09-15T20:50:54.122 INFO:tasks.workunit.client.0.smithi136.stderr:149553: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T20:50:54.122 INFO:tasks.workunit.client.0.smithi136.stderr:149553: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-15T20:50:54.122 INFO:tasks.workunit.client.0.smithi136.stderr:149553: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=309237645315 2024-09-15T20:50:54.122 INFO:tasks.workunit.client.0.smithi136.stderr:149553: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 309237645315 2024-09-15T20:50:54.123 INFO:tasks.workunit.client.0.smithi136.stderr:149553: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836534 1-313532612611 2-309237645315' 2024-09-15T20:50:54.123 INFO:tasks.workunit.client.0.smithi136.stderr:149553: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T20:50:54.123 INFO:tasks.workunit.client.0.smithi136.stderr:149553: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-15T20:50:54.123 INFO:tasks.workunit.client.0.smithi136.stderr:149553: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=274877906951 2024-09-15T20:50:54.123 INFO:tasks.workunit.client.0.smithi136.stderr:149553: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 274877906951 2024-09-15T20:50:54.123 INFO:tasks.workunit.client.0.smithi136.stderr:149553: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836534 1-313532612611 2-309237645315 3-274877906951' 2024-09-15T20:50:54.123 INFO:tasks.workunit.client.0.smithi136.stderr:149553: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T20:50:54.123 INFO:tasks.workunit.client.0.smithi136.stderr:149553: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836534 2024-09-15T20:50:54.123 INFO:tasks.workunit.client.0.smithi136.stderr:149553: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T20:50:54.123 INFO:tasks.workunit.client.0.smithi136.stderr:149553: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-15T20:50:54.123 INFO:tasks.workunit.client.0.smithi136.stderr:149553: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836534 2024-09-15T20:50:54.123 INFO:tasks.workunit.client.0.smithi136.stderr:149553: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T20:50:54.124 INFO:tasks.workunit.client.0.smithi136.stderr:149553: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836534 2024-09-15T20:50:54.124 INFO:tasks.workunit.client.0.smithi136.stderr:149553: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836534' 2024-09-15T20:50:54.124 INFO:tasks.workunit.client.0.smithi136.stderr:149553: waiting osd.0 seq 21474836534 2024-09-15T20:50:54.124 INFO:tasks.workunit.client.0.smithi136.stderr:149553: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T20:50:54.124 INFO:tasks.workunit.client.0.smithi136.stderr:149553: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836532 -lt 21474836534 2024-09-15T20:50:54.124 INFO:tasks.workunit.client.0.smithi136.stderr:149553: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-15T20:50:54.124 INFO:tasks.workunit.client.0.smithi136.stderr:149553: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-15T20:50:54.124 INFO:tasks.workunit.client.0.smithi136.stderr:149553: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T20:50:54.124 INFO:tasks.workunit.client.0.smithi136.stderr:149553: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836534 -lt 21474836534 2024-09-15T20:50:54.124 INFO:tasks.workunit.client.0.smithi136.stderr:149553: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T20:50:54.124 INFO:tasks.workunit.client.0.smithi136.stderr:149553: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-313532612611 2024-09-15T20:50:54.124 INFO:tasks.workunit.client.0.smithi136.stderr:149553: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T20:50:54.124 INFO:tasks.workunit.client.0.smithi136.stderr:149553: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-15T20:50:54.125 INFO:tasks.workunit.client.0.smithi136.stderr:149553: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-313532612611 2024-09-15T20:50:54.125 INFO:tasks.workunit.client.0.smithi136.stderr:149553: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T20:50:55.276 INFO:tasks.workunit.client.0.smithi136.stderr:149553: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=313532612610 2024-09-15T20:50:55.276 INFO:tasks.workunit.client.0.smithi136.stderr:149551: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 313532612610' 2024-09-15T20:50:55.276 INFO:tasks.workunit.client.0.smithi136.stderr:149551: waiting osd.1 seq 313532612610 2024-09-15T20:50:55.276 INFO:tasks.workunit.client.0.smithi136.stderr:149551: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-15T20:50:55.276 INFO:tasks.workunit.client.0.smithi136.stderr:149551: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 313532612611 -lt 313532612610 2024-09-15T20:50:55.276 INFO:tasks.workunit.client.0.smithi136.stderr:149551: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T20:50:55.276 INFO:tasks.workunit.client.0.smithi136.stderr:149551: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-309237645314 2024-09-15T20:50:55.276 INFO:tasks.workunit.client.0.smithi136.stderr:149551: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T20:50:55.276 INFO:tasks.workunit.client.0.smithi136.stderr:149551: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-15T20:50:55.277 INFO:tasks.workunit.client.0.smithi136.stderr:149551: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-309237645314 2024-09-15T20:50:55.277 INFO:tasks.workunit.client.0.smithi136.stderr:149551: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T20:50:55.277 INFO:tasks.workunit.client.0.smithi136.stderr:149551: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=309237645314 2024-09-15T20:50:55.277 INFO:tasks.workunit.client.0.smithi136.stderr:149551: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 309237645314' 2024-09-15T20:50:55.277 INFO:tasks.workunit.client.0.smithi136.stderr:149551: waiting osd.2 seq 309237645314 2024-09-15T20:50:55.277 INFO:tasks.workunit.client.0.smithi136.stderr:149551: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-15T20:50:55.277 INFO:tasks.workunit.client.0.smithi136.stderr:149551: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 309237645315 -lt 309237645314 2024-09-15T20:50:55.277 INFO:tasks.workunit.client.0.smithi136.stderr:149551: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T20:50:55.277 INFO:tasks.workunit.client.0.smithi136.stderr:149551: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 3-274877906950 2024-09-15T20:50:55.277 INFO:tasks.workunit.client.0.smithi136.stderr:149551: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T20:50:55.277 INFO:tasks.workunit.client.0.smithi136.stderr:149551: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=3 2024-09-15T20:50:55.277 INFO:tasks.workunit.client.0.smithi136.stderr:149551: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 3-274877906950 2024-09-15T20:50:55.277 INFO:tasks.workunit.client.0.smithi136.stderr:149551: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T20:50:55.278 INFO:tasks.workunit.client.0.smithi136.stderr:149551: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=274877906950 2024-09-15T20:50:55.278 INFO:tasks.workunit.client.0.smithi136.stderr:149551: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.3 seq 274877906950' 2024-09-15T20:50:55.278 INFO:tasks.workunit.client.0.smithi136.stderr:149551: waiting osd.3 seq 274877906950 2024-09-15T20:50:55.278 INFO:tasks.workunit.client.0.smithi136.stderr:149551: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-15T20:50:55.278 INFO:tasks.workunit.client.0.smithi136.stderr:149551: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 274877906951 -lt 274877906950 2024-09-15T20:50:55.278 INFO:tasks.workunit.client.0.smithi136.stderr:149551: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-15T20:50:55.278 INFO:tasks.workunit.client.0.smithi136.stderr:149551: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T20:50:55.278 INFO:tasks.workunit.client.0.smithi136.stderr:149551: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T20:50:55.278 INFO:tasks.workunit.client.0.smithi136.stderr:149551: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 5 == 0 2024-09-15T20:50:55.278 INFO:tasks.workunit.client.0.smithi136.stderr:149551: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-15T20:50:55.278 INFO:tasks.workunit.client.0.smithi136.stderr:149551: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-15T20:50:55.278 INFO:tasks.workunit.client.0.smithi136.stderr:149551: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-15T20:50:55.278 INFO:tasks.workunit.client.0.smithi136.stderr:149551: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T20:50:55.407 INFO:tasks.workunit.client.0.smithi136.stderr:149551: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("suntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=313532612611 2024-09-15T20:50:55.408 INFO:tasks.workunit.client.0.smithi136.stderr:149553: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 313532612611' 2024-09-15T20:50:55.408 INFO:tasks.workunit.client.0.smithi136.stderr:149553: waiting osd.1 seq 313532612611 2024-09-15T20:50:55.408 INFO:tasks.workunit.client.0.smithi136.stderr:149553: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-15T20:50:55.408 INFO:tasks.workunit.client.0.smithi136.stderr:149553: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 313532612611 -lt 313532612611 2024-09-15T20:50:55.408 INFO:tasks.workunit.client.0.smithi136.stderr:149553: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T20:50:55.408 INFO:tasks.workunit.client.0.smithi136.stderr:149553: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-309237645315 2024-09-15T20:50:55.408 INFO:tasks.workunit.client.0.smithi136.stderr:149553: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T20:50:55.408 INFO:tasks.workunit.client.0.smithi136.stderr:149553: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-15T20:50:55.408 INFO:tasks.workunit.client.0.smithi136.stderr:149553: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-309237645315 2024-09-15T20:50:55.408 INFO:tasks.workunit.client.0.smithi136.stderr:149553: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T20:50:55.408 INFO:tasks.workunit.client.0.smithi136.stderr:149553: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=309237645315 2024-09-15T20:50:55.408 INFO:tasks.workunit.client.0.smithi136.stderr:149553: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 309237645315' 2024-09-15T20:50:55.409 INFO:tasks.workunit.client.0.smithi136.stderr:149553: waiting osd.2 seq 309237645315 2024-09-15T20:50:55.409 INFO:tasks.workunit.client.0.smithi136.stderr:149553: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-15T20:50:55.409 INFO:tasks.workunit.client.0.smithi136.stderr:149553: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 309237645315 -lt 309237645315 2024-09-15T20:50:55.409 INFO:tasks.workunit.client.0.smithi136.stderr:149553: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T20:50:55.409 INFO:tasks.workunit.client.0.smithi136.stderr:149553: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 3-274877906951 2024-09-15T20:50:55.409 INFO:tasks.workunit.client.0.smithi136.stderr:149553: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T20:50:55.409 INFO:tasks.workunit.client.0.smithi136.stderr:149553: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=3 2024-09-15T20:50:55.409 INFO:tasks.workunit.client.0.smithi136.stderr:149553: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 3-274877906951 2024-09-15T20:50:55.409 INFO:tasks.workunit.client.0.smithi136.stderr:149553: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T20:50:55.409 INFO:tasks.workunit.client.0.smithi136.stderr:149553: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=274877906951 2024-09-15T20:50:55.409 INFO:tasks.workunit.client.0.smithi136.stderr:149553: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.3 seq 274877906951' 2024-09-15T20:50:55.409 INFO:tasks.workunit.client.0.smithi136.stderr:149553: waiting osd.3 seq 274877906951 2024-09-15T20:50:55.409 INFO:tasks.workunit.client.0.smithi136.stderr:149553: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-15T20:50:55.410 INFO:tasks.workunit.client.0.smithi136.stderr:149553: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 274877906951 -lt 274877906951 2024-09-15T20:50:55.410 INFO:tasks.workunit.client.0.smithi136.stderr:149553: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-15T20:50:55.410 INFO:tasks.workunit.client.0.smithi136.stderr:149553: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T20:50:55.410 INFO:tasks.workunit.client.0.smithi136.stderr:149553: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T20:50:55.410 INFO:tasks.workunit.client.0.smithi136.stderr:149553: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 5 == 0 2024-09-15T20:50:55.410 INFO:tasks.workunit.client.0.smithi136.stderr:149553: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-15T20:50:55.410 INFO:tasks.workunit.client.0.smithi136.stderr:149553: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-15T20:50:55.410 INFO:tasks.workunit.client.0.smithi136.stderr:149553: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-15T20:50:55.410 INFO:tasks.workunit.client.0.smithi136.stderr:149553: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T20:50:56.709 INFO:tasks.workunit.client.0.smithi136.stderr:149553: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T20:50:56.709 INFO:tasks.workunit.client.0.smithi136.stderr:149551: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T20:50:56.709 INFO:tasks.workunit.client.0.smithi136.stderr:149551: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T20:50:56.709 INFO:tasks.workunit.client.0.smithi136.stderr:149551: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=2 2024-09-15T20:50:56.709 INFO:tasks.workunit.client.0.smithi136.stderr:149551: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-15T20:50:56.710 INFO:tasks.workunit.client.0.smithi136.stderr:149551: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T20:50:56.710 INFO:tasks.workunit.client.0.smithi136.stderr:149551: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T20:50:56.710 INFO:tasks.workunit.client.0.smithi136.stderr:149551: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 2 = 5 2024-09-15T20:50:56.710 INFO:tasks.workunit.client.0.smithi136.stderr:149551: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 2 '!=' -1 2024-09-15T20:50:56.710 INFO:tasks.workunit.client.0.smithi136.stderr:149551: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: loop=0 2024-09-15T20:50:56.710 INFO:tasks.workunit.client.0.smithi136.stderr:149551: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1652: wait_for_clean: num_active_clean=2 2024-09-15T20:50:56.710 INFO:tasks.workunit.client.0.smithi136.stderr:149551: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-15T20:50:56.710 INFO:tasks.workunit.client.0.smithi136.stderr:149551: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 0.1 2024-09-15T20:50:56.710 INFO:tasks.workunit.client.0.smithi136.stderr:149551: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-15T20:50:56.710 INFO:tasks.workunit.client.0.smithi136.stderr:149551: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-15T20:50:56.710 INFO:tasks.workunit.client.0.smithi136.stderr:149551: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-15T20:50:56.710 INFO:tasks.workunit.client.0.smithi136.stderr:149551: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-15T20:50:56.710 INFO:tasks.workunit.client.0.smithi136.stderr:149551: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T20:50:56.711 INFO:tasks.workunit.client.0.smithi136.stderr:149551: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T20:50:56.711 INFO:tasks.workunit.client.0.smithi136.stderr:149551: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T20:50:56.711 INFO:tasks.workunit.client.0.smithi136.stderr:149551: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T20:50:56.711 INFO:tasks.workunit.client.0.smithi136.stderr:149551: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=2 2024-09-15T20:50:56.711 INFO:tasks.workunit.client.0.smithi136.stderr:149551: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-15T20:50:56.711 INFO:tasks.workunit.client.0.smithi136.stderr:149551: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T20:50:56.711 INFO:tasks.workunit.client.0.smithi136.stderr:149551: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T20:50:56.711 INFO:tasks.workunit.client.0.smithi136.stderr:149551: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 2 = 5 2024-09-15T20:50:56.711 INFO:tasks.workunit.client.0.smithi136.stderr:149551: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 2 '!=' 2 2024-09-15T20:50:56.711 INFO:tasks.workunit.client.0.smithi136.stderr:149551: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: get_is_making_recovery_progress 2024-09-15T20:50:56.711 INFO:tasks.workunit.client.0.smithi136.stderr:149551: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2024-09-15T20:50:56.711 INFO:tasks.workunit.client.0.smithi136.stderr:149551: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-15T20:50:56.711 INFO:tasks.workunit.client.0.smithi136.stderr:149551: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-15T20:50:56.711 INFO:tasks.workunit.client.0.smithi136.stderr:149551: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-15T20:50:56.814 INFO:tasks.workunit.client.0.smithi136.stderr:149551: //tale") | not)' 2024-09-15T20:50:56.814 INFO:tasks.workunit.client.0.smithi136.stderr:149553: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T20:50:56.814 INFO:tasks.workunit.client.0.smithi136.stderr:149553: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T20:50:56.814 INFO:tasks.workunit.client.0.smithi136.stderr:149553: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=2 2024-09-15T20:50:56.815 INFO:tasks.workunit.client.0.smithi136.stderr:149553: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-15T20:50:56.815 INFO:tasks.workunit.client.0.smithi136.stderr:149553: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T20:50:56.815 INFO:tasks.workunit.client.0.smithi136.stderr:149553: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T20:50:56.815 INFO:tasks.workunit.client.0.smithi136.stderr:149553: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 2 = 5 2024-09-15T20:50:56.815 INFO:tasks.workunit.client.0.smithi136.stderr:149553: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 2 '!=' -1 2024-09-15T20:50:56.815 INFO:tasks.workunit.client.0.smithi136.stderr:149553: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: loop=0 2024-09-15T20:50:56.815 INFO:tasks.workunit.client.0.smithi136.stderr:149553: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1652: wait_for_clean: num_active_clean=2 2024-09-15T20:50:56.815 INFO:tasks.workunit.client.0.smithi136.stderr:149553: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-15T20:50:56.815 INFO:tasks.workunit.client.0.smithi136.stderr:149553: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 0.1 2024-09-15T20:50:56.815 INFO:tasks.workunit.client.0.smithi136.stderr:149553: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-15T20:50:56.815 INFO:tasks.workunit.client.0.smithi136.stderr:149553: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-15T20:50:56.815 INFO:tasks.workunit.client.0.smithi136.stderr:149553: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-15T20:50:56.816 INFO:tasks.workunit.client.0.smithi136.stderr:149553: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-15T20:50:56.816 INFO:tasks.workunit.client.0.smithi136.stderr:149553: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T20:50:56.816 INFO:tasks.workunit.client.0.smithi136.stderr:149553: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T20:50:56.816 INFO:tasks.workunit.client.0.smithi136.stderr:149553: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T20:50:56.816 INFO:tasks.workunit.client.0.smithi136.stderr:149553: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T20:50:56.816 INFO:tasks.workunit.client.0.smithi136.stderr:149553: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=2 2024-09-15T20:50:56.816 INFO:tasks.workunit.client.0.smithi136.stderr:149553: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-15T20:50:56.816 INFO:tasks.workunit.client.0.smithi136.stderr:149553: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T20:50:56.816 INFO:tasks.workunit.client.0.smithi136.stderr:149553: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T20:50:56.816 INFO:tasks.workunit.client.0.smithi136.stderr:149553: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 2 = 5 2024-09-15T20:50:56.816 INFO:tasks.workunit.client.0.smithi136.stderr:149553: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 2 '!=' 2 2024-09-15T20:50:56.816 INFO:tasks.workunit.client.0.smithi136.stderr:149553: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: get_is_making_recovery_progress 2024-09-15T20:50:56.817 INFO:tasks.workunit.client.0.smithi136.stderr:149553: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2024-09-15T20:50:56.817 INFO:tasks.workunit.client.0.smithi136.stderr:149553: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-15T20:50:56.817 INFO:tasks.workunit.client.0.smithi136.stderr:149553: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-15T20:50:56.817 INFO:tasks.workunit.client.0.smithi136.stderr:149553: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-15T20:50:57.921 INFO:tasks.workunit.client.0.smithi136.stderr:149553: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2024-09-15T20:50:57.921 INFO:tasks.workunit.client.0.smithi136.stderr:149551: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-15T20:50:57.921 INFO:tasks.workunit.client.0.smithi136.stderr:149551: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=null 2024-09-15T20:50:57.922 INFO:tasks.workunit.client.0.smithi136.stderr:149551: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test null '!=' null 2024-09-15T20:50:57.922 INFO:tasks.workunit.client.0.smithi136.stderr:149551: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1655: wait_for_clean: (( 1 >= 13 )) 2024-09-15T20:50:57.922 INFO:tasks.workunit.client.0.smithi136.stderr:149551: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-15T20:50:57.922 INFO:tasks.workunit.client.0.smithi136.stderr:149551: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 0.2 2024-09-15T20:50:57.922 INFO:tasks.workunit.client.0.smithi136.stderr:149551: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-15T20:50:57.922 INFO:tasks.workunit.client.0.smithi136.stderr:149551: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-15T20:50:57.922 INFO:tasks.workunit.client.0.smithi136.stderr:149551: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-15T20:50:57.922 INFO:tasks.workunit.client.0.smithi136.stderr:149551: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-15T20:50:57.922 INFO:tasks.workunit.client.0.smithi136.stderr:149551: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T20:50:57.922 INFO:tasks.workunit.client.0.smithi136.stderr:149551: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T20:50:57.922 INFO:tasks.workunit.client.0.smithi136.stderr:149551: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T20:50:57.922 INFO:tasks.workunit.client.0.smithi136.stderr:149551: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T20:50:57.922 INFO:tasks.workunit.client.0.smithi136.stderr:149551: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=5 2024-09-15T20:50:57.923 INFO:tasks.workunit.client.0.smithi136.stderr:149551: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-15T20:50:57.923 INFO:tasks.workunit.client.0.smithi136.stderr:149551: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T20:50:57.923 INFO:tasks.workunit.client.0.smithi136.stderr:149551: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T20:50:57.923 INFO:tasks.workunit.client.0.smithi136.stderr:149551: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 = 5 2024-09-15T20:50:57.923 INFO:tasks.workunit.client.0.smithi136.stderr:149551: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-15T20:50:57.923 INFO:tasks.workunit.client.0.smithi136.stderr:149551: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-15T20:50:57.923 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: run_in_background: return 0 2024-09-15T20:50:57.923 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: wait_background: for pid in $pids 2024-09-15T20:50:57.923 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2175: wait_background: wait 149550 2024-09-15T20:50:58.064 INFO:tasks.workunit.client.0.smithi136.stderr:home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2024-09-15T20:50:58.064 INFO:tasks.workunit.client.0.smithi136.stderr:149553: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-15T20:50:58.064 INFO:tasks.workunit.client.0.smithi136.stderr:149553: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=null 2024-09-15T20:50:58.065 INFO:tasks.workunit.client.0.smithi136.stderr:149553: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test null '!=' null 2024-09-15T20:50:58.065 INFO:tasks.workunit.client.0.smithi136.stderr:149553: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1655: wait_for_clean: (( 1 >= 13 )) 2024-09-15T20:50:58.065 INFO:tasks.workunit.client.0.smithi136.stderr:149553: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-15T20:50:58.065 INFO:tasks.workunit.client.0.smithi136.stderr:149553: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 0.2 2024-09-15T20:50:58.065 INFO:tasks.workunit.client.0.smithi136.stderr:149553: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-15T20:50:58.065 INFO:tasks.workunit.client.0.smithi136.stderr:149553: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-15T20:50:58.065 INFO:tasks.workunit.client.0.smithi136.stderr:149553: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-15T20:50:58.065 INFO:tasks.workunit.client.0.smithi136.stderr:149553: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-15T20:50:58.065 INFO:tasks.workunit.client.0.smithi136.stderr:149553: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T20:50:58.065 INFO:tasks.workunit.client.0.smithi136.stderr:149553: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T20:50:58.065 INFO:tasks.workunit.client.0.smithi136.stderr:149553: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T20:50:58.065 INFO:tasks.workunit.client.0.smithi136.stderr:149553: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T20:50:58.065 INFO:tasks.workunit.client.0.smithi136.stderr:149553: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=5 2024-09-15T20:50:58.066 INFO:tasks.workunit.client.0.smithi136.stderr:149553: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-15T20:50:58.066 INFO:tasks.workunit.client.0.smithi136.stderr:149553: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T20:50:58.066 INFO:tasks.workunit.client.0.smithi136.stderr:149553: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T20:50:58.066 INFO:tasks.workunit.client.0.smithi136.stderr:149553: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 = 5 2024-09-15T20:50:58.066 INFO:tasks.workunit.client.0.smithi136.stderr:149553: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-15T20:50:58.066 INFO:tasks.workunit.client.0.smithi136.stderr:149553: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-15T20:50:58.066 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: run_in_background: return 0 2024-09-15T20:50:58.066 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: wait_background: eval 'pids='\'''\''' 2024-09-15T20:50:58.066 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: wait_background: pids= 2024-09-15T20:50:58.066 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: wait_background: return 0 2024-09-15T20:50:58.066 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:197: corrupt_and_repair_two: return_code=0 2024-09-15T20:50:58.066 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:198: corrupt_and_repair_two: '[' 0 -ne 0 ']' 2024-09-15T20:50:58.066 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:203: corrupt_and_repair_two: get_pg ecpool SOMETHING 2024-09-15T20:50:58.066 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1084: get_pg: local poolname=ecpool 2024-09-15T20:50:58.066 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1085: get_pg: local objectname=SOMETHING 2024-09-15T20:50:58.067 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1087: get_pg: ceph --format json osd map ecpool SOMETHING 2024-09-15T20:50:58.067 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1087: get_pg: jq -r .pgid 2024-09-15T20:50:58.347 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:203: corrupt_and_repair_two: local pg=2.0 2024-09-15T20:50:58.347 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:204: corrupt_and_repair_two: repair 2.0 2024-09-15T20:50:58.347 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1846: repair: local pgid=2.0 2024-09-15T20:50:58.347 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1847: repair: get_last_scrub_stamp 2.0 2024-09-15T20:50:58.347 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2024-09-15T20:50:58.347 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-15T20:50:58.347 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-15T20:50:58.347 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-09-15T20:50:58.618 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1847: repair: local last_scrub=2024-09-15T20:50:24.121965+0000 2024-09-15T20:50:58.618 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1848: repair: ceph pg repair 2.0 2024-09-15T20:50:58.879 INFO:tasks.workunit.client.0.smithi136.stderr:instructing pg 2.0s0 on osd.3 to repair 2024-09-15T20:50:58.893 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1849: repair: wait_for_scrub 2.0 2024-09-15T20:50:24.121965+0000 2024-09-15T20:50:58.893 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: local pgid=2.0 2024-09-15T20:50:58.893 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2018: wait_for_scrub: local last_scrub=2024-09-15T20:50:24.121965+0000 2024-09-15T20:50:58.893 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2019: wait_for_scrub: local sname=last_scrub_stamp 2024-09-15T20:50:58.893 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i=0 )) 2024-09-15T20:50:58.893 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-15T20:50:58.893 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-09-15T20:50:58.893 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2024-09-15T20:50:58.894 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-15T20:50:58.894 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-15T20:50:58.894 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-09-15T20:50:59.166 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-15T20:50:24.121965+0000 '>' 2024-09-15T20:50:24.121965+0000 2024-09-15T20:50:59.166 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-15T20:51:00.167 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-15T20:51:00.168 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-15T20:51:00.168 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-09-15T20:51:00.168 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2024-09-15T20:51:00.169 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-15T20:51:00.169 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-15T20:51:00.169 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-09-15T20:51:00.447 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-15T20:50:24.121965+0000 '>' 2024-09-15T20:50:24.121965+0000 2024-09-15T20:51:00.447 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-15T20:51:01.448 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-15T20:51:01.449 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-15T20:51:01.449 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-09-15T20:51:01.449 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2024-09-15T20:51:01.449 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-15T20:51:01.450 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-15T20:51:01.450 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-09-15T20:51:01.736 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-15T20:50:58.922318+0000 '>' 2024-09-15T20:50:24.121965+0000 2024-09-15T20:51:01.736 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2023: wait_for_scrub: return 0 2024-09-15T20:51:01.736 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1850: repair: sleep 2 2024-09-15T20:51:03.739 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:208: corrupt_and_repair_two: pids= 2024-09-15T20:51:03.739 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:209: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 1 SOMETHING list-attrs 2024-09-15T20:51:03.739 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2135: run_in_background: local pid_variable=pids 2024-09-15T20:51:03.739 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2136: run_in_background: shift 2024-09-15T20:51:03.739 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2140: run_in_background: eval 'pids+=" 152663"' 2024-09-15T20:51:03.739 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2140: run_in_background: pids+=' 152663' 2024-09-15T20:51:03.740 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:210: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 2 SOMETHING list-attrs 2024-09-15T20:51:03.740 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2135: run_in_background: local pid_variable=pids 2024-09-15T20:51:03.740 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2136: run_in_background: shift 2024-09-15T20:51:03.740 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2140: run_in_background: eval 'pids+=" 152664"' 2024-09-15T20:51:03.740 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2140: run_in_background: pids+=' 152664' 2024-09-15T20:51:03.740 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:211: corrupt_and_repair_two: wait_background pids 2024-09-15T20:51:03.740 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: wait_background: pids=' 152663 152664' 2024-09-15T20:51:03.740 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: wait_background: return_code=0 2024-09-15T20:51:03.740 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: wait_background: for pid in $pids 2024-09-15T20:51:03.740 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: run_in_background: objectstore_tool td/osd-scrub-repair 1 SOMETHING list-attrs 2024-09-15T20:51:03.740 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2175: wait_background: wait 152663 2024-09-15T20:51:03.740 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: run_in_background: sed 's/^/152666: /' 2024-09-15T20:51:03.741 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: run_in_background: objectstore_tool td/osd-scrub-repair 2 SOMETHING list-attrs 2024-09-15T20:51:03.741 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: run_in_background: sed 's/^/152668: /' 2024-09-15T20:51:04.757 INFO:tasks.workunit.client.0.smithi136.stderr:152668: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2024-09-15T20:51:04.758 INFO:tasks.workunit.client.0.smithi136.stderr:152668: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2024-09-15T20:51:04.758 INFO:tasks.workunit.client.0.smithi136.stderr:152668: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=2 2024-09-15T20:51:04.758 INFO:tasks.workunit.client.0.smithi136.stderr:152668: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2024-09-15T20:51:04.758 INFO:tasks.workunit.client.0.smithi136.stderr:152668: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 2 SOMETHING list-attrs 2024-09-15T20:51:04.758 INFO:tasks.workunit.client.0.smithi136.stderr:152668: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-09-15T20:51:04.758 INFO:tasks.workunit.client.0.smithi136.stderr:152668: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2024-09-15T20:51:04.758 INFO:tasks.workunit.client.0.smithi136.stderr:152668: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=2 2024-09-15T20:51:04.758 INFO:tasks.workunit.client.0.smithi136.stderr:152668: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2024-09-15T20:51:04.758 INFO:tasks.workunit.client.0.smithi136.stderr:152668: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.2 2024-09-15T20:51:04.758 INFO:tasks.workunit.client.0.smithi136.stderr:152668: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-15T20:51:04.759 INFO:tasks.workunit.client.0.smithi136.stderr:152668: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-15T20:51:04.759 INFO:tasks.workunit.client.0.smithi136.stderr:152668: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-15T20:51:04.759 INFO:tasks.workunit.client.0.smithi136.stderr:152668: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-15T20:51:04.759 INFO:tasks.workunit.client.0.smithi136.stderr:152668: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-15T20:51:04.759 INFO:tasks.workunit.client.0.smithi136.stderr:152668: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-15T20:51:04.759 INFO:tasks.workunit.client.0.smithi136.stderr:152668: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 2 SOMETHING list-attrs 2024-09-15T20:51:04.759 INFO:tasks.workunit.client.0.smithi136.stderr:152668: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-15T20:51:04.759 INFO:tasks.workunit.client.0.smithi136.stderr:152668: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-15T20:51:04.759 INFO:tasks.workunit.client.0.smithi136.stderr:152668: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=2 2024-09-15T20:51:04.759 INFO:tasks.workunit.client.0.smithi136.stderr:152668: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-15T20:51:04.760 INFO:tasks.workunit.client.0.smithi136.stderr:152668: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/2 2024-09-15T20:51:04.760 INFO:tasks.workunit.client.0.smithi136.stderr:152668: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/2 SOMETHING list-attrs 2024-09-15T20:51:04.760 INFO:tasks.workunit.client.0.smithi136.stderr:152668: _ 2024-09-15T20:51:04.760 INFO:tasks.workunit.client.0.smithi136.stderr:152668: hinfo_key 2024-09-15T20:51:04.760 INFO:tasks.workunit.client.0.smithi136.stderr:152668: snapset 2024-09-15T20:51:04.760 INFO:tasks.workunit.client.0.smithi136.stderr:152668: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 2 2024-09-15T20:51:04.760 INFO:tasks.workunit.client.0.smithi136.stderr:152668: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2024-09-15T20:51:04.760 INFO:tasks.workunit.client.0.smithi136.stderr:152668: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-15T20:51:04.760 INFO:tasks.workunit.client.0.smithi136.stderr:152668: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=2 2024-09-15T20:51:04.760 INFO:tasks.workunit.client.0.smithi136.stderr:152668: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-15T20:51:04.760 INFO:tasks.workunit.client.0.smithi136.stderr:152668: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/2 2024-09-15T20:51:04.761 INFO:tasks.workunit.client.0.smithi136.stderr:152668: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=ce4892c7-6c6d-4b4f-8e6f-07a3e11133e5 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-15T20:51:04.761 INFO:tasks.workunit.client.0.smithi136.stderr:152668: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T20:51:04.761 INFO:tasks.workunit.client.0.smithi136.stderr:152668: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T20:51:04.761 INFO:tasks.workunit.client.0.smithi136.stderr:152668: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T20:51:04.761 INFO:tasks.workunit.client.0.smithi136.stderr:152668: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/2' 2024-09-15T20:51:04.761 INFO:tasks.workunit.client.0.smithi136.stderr:152668: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/2/journal' 2024-09-15T20:51:04.761 INFO:tasks.workunit.client.0.smithi136.stderr:152668: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-15T20:51:04.761 INFO:tasks.workunit.client.0.smithi136.stderr:152668: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-15T20:51:04.762 INFO:tasks.workunit.client.0.smithi136.stderr:152668: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-15T20:51:04.762 INFO:tasks.workunit.client.0.smithi136.stderr:152668: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-15T20:51:04.762 INFO:tasks.workunit.client.0.smithi136.stderr:152668: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-15T20:51:04.762 INFO:tasks.workunit.client.0.smithi136.stderr:152668: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-15T20:51:04.762 INFO:tasks.workunit.client.0.smithi136.stderr:152668: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-15T20:51:04.762 INFO:tasks.workunit.client.0.smithi136.stderr:152668: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T20:51:04.762 INFO:tasks.workunit.client.0.smithi136.stderr:152668: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52602 2024-09-15T20:51:04.762 INFO:tasks.workunit.client.0.smithi136.stderr:152668: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.52602/$cluster-$name.asok' 2024-09-15T20:51:04.762 INFO:tasks.workunit.client.0.smithi136.stderr:152668: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.52602/$cluster-$name.asok' 2024-09-15T20:51:04.762 INFO:tasks.workunit.client.0.smithi136.stderr:152668: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-15T20:51:04.762 INFO:tasks.workunit.client.0.smithi136.stderr:152668: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-15T20:51:04.762 INFO:tasks.workunit.client.0.smithi136.stderr:152668: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-15T20:51:04.763 INFO:tasks.workunit.client.0.smithi136.stderr:152668: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T20:51:04.763 INFO:tasks.workunit.client.0.smithi136.stderr:152668: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T20:51:04.763 INFO:tasks.workunit.client.0.smithi136.stderr:152668: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T20:51:04.763 INFO:tasks.workunit.client.0.smithi136.stderr:152668: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T20:51:04.763 INFO:tasks.workunit.client.0.smithi136.stderr:152668: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-15T20:51:04.763 INFO:tasks.workunit.client.0.smithi136.stderr:152668: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-15T20:51:04.763 INFO:tasks.workunit.client.0.smithi136.stderr:152668: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/2 2024-09-15T20:51:04.763 INFO:tasks.workunit.client.0.smithi136.stderr:152668: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.2 2024-09-15T20:51:04.763 INFO:tasks.workunit.client.0.smithi136.stderr:152668: start osd.2 2024-09-15T20:51:04.799 INFO:tasks.workunit.client.0.smithi136.stderr:152668: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 2 --fsid=ce4892c7-6c6d-4b4f-8e6f-07a3e11133e5 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.52602/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_reco152666: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2024-09-15T20:51:04.799 INFO:tasks.workunit.client.0.smithi136.stderr:152666: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2024-09-15T20:51:04.799 INFO:tasks.workunit.client.0.smithi136.stderr:152666: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=1 2024-09-15T20:51:04.799 INFO:tasks.workunit.client.0.smithi136.stderr:152666: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2024-09-15T20:51:04.799 INFO:tasks.workunit.client.0.smithi136.stderr:152666: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 SOMETHING list-attrs 2024-09-15T20:51:04.800 INFO:tasks.workunit.client.0.smithi136.stderr:152666: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-09-15T20:51:04.800 INFO:tasks.workunit.client.0.smithi136.stderr:152666: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2024-09-15T20:51:04.800 INFO:tasks.workunit.client.0.smithi136.stderr:152666: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=1 2024-09-15T20:51:04.800 INFO:tasks.workunit.client.0.smithi136.stderr:152666: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2024-09-15T20:51:04.800 INFO:tasks.workunit.client.0.smithi136.stderr:152666: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-09-15T20:51:04.800 INFO:tasks.workunit.client.0.smithi136.stderr:152666: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-15T20:51:04.800 INFO:tasks.workunit.client.0.smithi136.stderr:152666: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-15T20:51:04.800 INFO:tasks.workunit.client.0.smithi136.stderr:152666: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-15T20:51:04.800 INFO:tasks.workunit.client.0.smithi136.stderr:152666: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-15T20:51:04.800 INFO:tasks.workunit.client.0.smithi136.stderr:152666: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-15T20:51:04.800 INFO:tasks.workunit.client.0.smithi136.stderr:152666: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-15T20:51:04.800 INFO:tasks.workunit.client.0.smithi136.stderr:152666: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 SOMETHING list-attrs 2024-09-15T20:51:04.800 INFO:tasks.workunit.client.0.smithi136.stderr:152666: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-15T20:51:04.801 INFO:tasks.workunit.client.0.smithi136.stderr:152666: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-15T20:51:04.801 INFO:tasks.workunit.client.0.smithi136.stderr:152666: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=1 2024-09-15T20:51:04.801 INFO:tasks.workunit.client.0.smithi136.stderr:152666: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-15T20:51:04.801 INFO:tasks.workunit.client.0.smithi136.stderr:152666: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-09-15T20:51:04.801 INFO:tasks.workunit.client.0.smithi136.stderr:152666: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 SOMETHING list-attrs 2024-09-15T20:51:04.801 INFO:tasks.workunit.client.0.smithi136.stderr:152666: _ 2024-09-15T20:51:04.801 INFO:tasks.workunit.client.0.smithi136.stderr:152666: hinfo_key 2024-09-15T20:51:04.801 INFO:tasks.workunit.client.0.smithi136.stderr:152666: snapset 2024-09-15T20:51:04.801 INFO:tasks.workunit.client.0.smithi136.stderr:152666: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2024-09-15T20:51:04.801 INFO:tasks.workunit.client.0.smithi136.stderr:152666: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2024-09-15T20:51:04.801 INFO:tasks.workunit.client.0.smithi136.stderr:152666: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-15T20:51:04.801 INFO:tasks.workunit.client.0.smithi136.stderr:152666: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=1 2024-09-15T20:51:04.801 INFO:tasks.workunit.client.0.smithi136.stderr:152666: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-15T20:51:04.802 INFO:tasks.workunit.client.0.smithi136.stderr:152666: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-09-15T20:51:04.802 INFO:tasks.workunit.client.0.smithi136.stderr:152666: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=ce4892c7-6c6d-4b4f-8e6f-07a3e11133e5 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-15T20:51:04.802 INFO:tasks.workunit.client.0.smithi136.stderr:152666: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T20:51:04.802 INFO:tasks.workunit.client.0.smithi136.stderr:152666: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T20:51:04.802 INFO:tasks.workunit.client.0.smithi136.stderr:152666: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T20:51:04.802 INFO:tasks.workunit.client.0.smithi136.stderr:152666: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-09-15T20:51:04.802 INFO:tasks.workunit.client.0.smithi136.stderr:152666: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-09-15T20:51:04.802 INFO:tasks.workunit.client.0.smithi136.stderr:152666: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-15T20:51:04.802 INFO:tasks.workunit.client.0.smithi136.stderr:152666: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-15T20:51:04.803 INFO:tasks.workunit.client.0.smithi136.stderr:152666: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-15T20:51:04.803 INFO:tasks.workunit.client.0.smithi136.stderr:152666: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-15T20:51:04.803 INFO:tasks.workunit.client.0.smithi136.stderr:152666: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-15T20:51:04.803 INFO:tasks.workunit.client.0.smithi136.stderr:152666: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-15T20:51:04.803 INFO:tasks.workunit.client.0.smithi136.stderr:152666: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-15T20:51:04.803 INFO:tasks.workunit.client.0.smithi136.stderr:152666: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T20:51:04.803 INFO:tasks.workunit.client.0.smithi136.stderr:152666: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52602 2024-09-15T20:51:04.803 INFO:tasks.workunit.client.0.smithi136.stderr:152666: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.52602/$cluster-$name.asok' 2024-09-15T20:51:04.803 INFO:tasks.workunit.client.0.smithi136.stderr:152666: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.52602/$cluster-$name.asok' 2024-09-15T20:51:04.803 INFO:tasks.workunit.client.0.smithi136.stderr:152666: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-15T20:51:04.803 INFO:tasks.workunit.client.0.smithi136.stderr:152666: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-15T20:51:04.803 INFO:tasks.workunit.client.0.smithi136.stderr:152666: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-15T20:51:04.803 INFO:tasks.workunit.client.0.smithi136.stderr:152666: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T20:51:04.803 INFO:tasks.workunit.client.0.smithi136.stderr:152666: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T20:51:04.804 INFO:tasks.workunit.client.0.smithi136.stderr:152666: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T20:51:04.804 INFO:tasks.workunit.client.0.smithi136.stderr:152666: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T20:51:04.804 INFO:tasks.workunit.client.0.smithi136.stderr:152666: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-15T20:51:04.804 INFO:tasks.workunit.client.0.smithi136.stderr:152666: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-15T20:51:04.804 INFO:tasks.workunit.client.0.smithi136.stderr:152666: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-09-15T20:51:04.804 INFO:tasks.workunit.client.0.smithi136.stderr:152666: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.1 2024-09-15T20:51:04.804 INFO:tasks.workunit.client.0.smithi136.stderr:152666: start osd.1 2024-09-15T20:51:09.675 INFO:tasks.workunit.client.0.smithi136.stderr:152666: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 1 --fsid=ce4892c7-6c6d-4b4f-8e6f-07a3e11133e5 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.52602/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T20:51:09.676 INFO:tasks.workunit.client.0.smithi136.stderr:152666: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-09-15T20:51:09.676 INFO:tasks.workunit.client.0.smithi136.stderr:152666: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 1 = 1 ']' 2024-09-15T20:51:09.676 INFO:tasks.workunit.client.0.smithi136.stderr:152666: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-15T20:51:09.676 INFO:tasks.workunit.client.0.smithi136.stderr:152666: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-15T20:51:09.676 INFO:tasks.workunit.client.0.smithi136.stderr:152666: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-15T20:51:09.676 INFO:tasks.workunit.client.0.smithi136.stderr:152666: 2024-09-15T20:51:04.829+0000 7efd764e0640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T20:51:09.676 INFO:tasks.workunit.client.0.smithi136.stderr:152666: 2024-09-15T20:51:04.896+0000 7efd764e0640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T20:51:09.676 INFO:tasks.workunit.client.0.smithi136.stderr:152666: 2024-09-15T20:51:04.897+0000 7efd764e0640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T20:51:09.676 INFO:tasks.workunit.client.0.smithi136.stderr:152666: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 1 2024-09-15T20:51:09.676 INFO:tasks.workunit.client.0.smithi136.stderr:152666: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-15T20:51:09.676 INFO:tasks.workunit.client.0.smithi136.stderr:152666: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2024-09-15T20:51:09.676 INFO:tasks.workunit.client.0.smithi136.stderr:152666: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-15T20:51:09.676 INFO:tasks.workunit.client.0.smithi136.stderr:152666: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-15T20:51:09.677 INFO:tasks.workunit.client.0.smithi136.stderr:152666: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:51:09.677 INFO:tasks.workunit.client.0.smithi136.stderr:152666: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-15T20:51:09.677 INFO:tasks.workunit.client.0.smithi136.stderr:152666: 0 2024-09-15T20:51:09.677 INFO:tasks.workunit.client.0.smithi136.stderr:152666: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T20:51:09.677 INFO:tasks.workunit.client.0.smithi136.stderr:152666: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:51:09.677 INFO:tasks.workunit.client.0.smithi136.stderr:152666: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:51:09.677 INFO:tasks.workunit.client.0.smithi136.stderr:152666: 2024-09-15T20:51:06.503+0000 7efd764e0640 -1 Falling back to public interface 2024-09-15T20:51:09.677 INFO:tasks.workunit.client.0.smithi136.stderr:152666: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:51:09.677 INFO:tasks.workunit.client.0.smithi136.stderr:152666: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:51:09.677 INFO:tasks.workunit.client.0.smithi136.stderr:152666: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-15T20:51:09.677 INFO:tasks.workunit.client.0.smithi136.stderr:152666: 1 2024-09-15T20:51:09.677 INFO:tasks.workunit.client.0.smithi136.stderr:152666: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:51:09.678 INFO:tasks.workunit.client.0.smithi136.stderr:152666: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T20:51:09.678 INFO:tasks.workunit.client.0.smithi136.stderr:152666: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:51:09.678 INFO:tasks.workunit.client.0.smithi136.stderr:152666: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:51:09.678 INFO:tasks.workunit.client.0.smithi136.stderr:152666: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:51:09.678 INFO:tasks.workunit.client.0.smithi136.stderr:152666: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-15T20:51:09.678 INFO:tasks.workunit.client.0.smithi136.stderr:152666: 2 2024-09-15T20:51:09.678 INFO:tasks.workunit.client.0.smithi136.stderr:152666: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:51:09.678 INFO:tasks.workunit.client.0.smithi136.stderr:152666: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T20:51:09.678 INFO:tasks.workunit.client.0.smithi136.stderr:152666: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:51:09.678 INFO:tasks.workunit.client.0.smithi136.stderr:152666: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:51:09.678 INFO:tasks.workunit.client.0.smithi136.stderr:152666: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:51:09.678 INFO:tasks.workunit.client.0.smithi136.stderr:152666: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-15T20:51:09.678 INFO:tasks.workunit.client.0.smithi136.stderr:152666: 3 2024-09-15T20:51:09.678 INFO:tasks.workunit.client.0.smithi136.stderr:152666: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:51:09.679 INFO:tasks.workunit.client.0.smithi136.stderr:152666: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T20:51:09.679 INFO:tasks.workunit.client.0.smithi136.stderr:152666: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:51:10.481 INFO:tasks.workunit.client.0.smithi136.stderr:152666: 2024-09-15T20:51:09.673+0000 7efdvery_ops 2024-09-15T20:51:10.482 INFO:tasks.workunit.client.0.smithi136.stderr:152668: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/2/whoami 2024-09-15T20:51:10.482 INFO:tasks.workunit.client.0.smithi136.stderr:152668: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 2 = 2 ']' 2024-09-15T20:51:10.482 INFO:tasks.workunit.client.0.smithi136.stderr:152668: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-15T20:51:10.482 INFO:tasks.workunit.client.0.smithi136.stderr:152668: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-15T20:51:10.482 INFO:tasks.workunit.client.0.smithi136.stderr:152668: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-15T20:51:10.482 INFO:tasks.workunit.client.0.smithi136.stderr:152668: 2024-09-15T20:51:04.788+0000 7f972bfd9640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T20:51:10.482 INFO:tasks.workunit.client.0.smithi136.stderr:152668: 2024-09-15T20:51:04.896+0000 7f972bfd9640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T20:51:10.482 INFO:tasks.workunit.client.0.smithi136.stderr:152668: 2024-09-15T20:51:04.897+0000 7f972bfd9640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T20:51:10.482 INFO:tasks.workunit.client.0.smithi136.stderr:152668: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 2 2024-09-15T20:51:10.483 INFO:tasks.workunit.client.0.smithi136.stderr:152668: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-15T20:51:10.483 INFO:tasks.workunit.client.0.smithi136.stderr:152668: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=2 2024-09-15T20:51:10.483 INFO:tasks.workunit.client.0.smithi136.stderr:152668: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-15T20:51:10.483 INFO:tasks.workunit.client.0.smithi136.stderr:152668: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-15T20:51:10.483 INFO:tasks.workunit.client.0.smithi136.stderr:152668: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:51:10.483 INFO:tasks.workunit.client.0.smithi136.stderr:152668: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-15T20:51:10.483 INFO:tasks.workunit.client.0.smithi136.stderr:152668: 0 2024-09-15T20:51:10.483 INFO:tasks.workunit.client.0.smithi136.stderr:152668: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:51:10.483 INFO:tasks.workunit.client.0.smithi136.stderr:152668: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T20:51:10.483 INFO:tasks.workunit.client.0.smithi136.stderr:152668: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:51:10.483 INFO:tasks.workunit.client.0.smithi136.stderr:152668: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:51:10.483 INFO:tasks.workunit.client.0.smithi136.stderr:152668: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:51:10.484 INFO:tasks.workunit.client.0.smithi136.stderr:152668: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-15T20:51:10.484 INFO:tasks.workunit.client.0.smithi136.stderr:152668: 1 2024-09-15T20:51:10.484 INFO:tasks.workunit.client.0.smithi136.stderr:152668: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:51:10.484 INFO:tasks.workunit.client.0.smithi136.stderr:152668: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T20:51:10.484 INFO:tasks.workunit.client.0.smithi136.stderr:152668: 2024-09-15T20:51:06.740+0000 7f972bfd9640 -1 Falling back to public interface 2024-09-15T20:51:10.484 INFO:tasks.workunit.client.0.smithi136.stderr:152668: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:51:10.484 INFO:tasks.workunit.client.0.smithi136.stderr:152668: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:51:10.484 INFO:tasks.workunit.client.0.smithi136.stderr:152668: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:51:10.484 INFO:tasks.workunit.client.0.smithi136.stderr:152668: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-15T20:51:10.484 INFO:tasks.workunit.client.0.smithi136.stderr:152668: 2 2024-09-15T20:51:10.484 INFO:tasks.workunit.client.0.smithi136.stderr:152668: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:51:10.485 INFO:tasks.workunit.client.0.smithi136.stderr:152668: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T20:51:10.485 INFO:tasks.workunit.client.0.smithi136.stderr:152668: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:51:10.485 INFO:tasks.workunit.client.0.smithi136.stderr:152668: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:51:10.485 INFO:tasks.workunit.client.0.smithi136.stderr:152668: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:51:10.485 INFO:tasks.workunit.client.0.smithi136.stderr:152668: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-15T20:51:10.485 INFO:tasks.workunit.client.0.smithi136.stderr:152668: 3 2024-09-15T20:51:10.485 INFO:tasks.workunit.client.0.smithi136.stderr:152668: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:51:10.485 INFO:tasks.workunit.client.0.smithi136.stderr:152668: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T20:51:10.485 INFO:tasks.workunit.client.0.smithi136.stderr:152668: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:51:11.856 INFO:tasks.workunit.client.0.smithi136.stderr:152668: /home/ubuntu/cephtest/clone.clien764e0640 -1 osd.1 75 log_to_monitors true 2024-09-15T20:51:11.856 INFO:tasks.workunit.client.0.smithi136.stderr:152666: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:51:11.856 INFO:tasks.workunit.client.0.smithi136.stderr:152666: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:51:11.856 INFO:tasks.workunit.client.0.smithi136.stderr:152666: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-15T20:51:11.856 INFO:tasks.workunit.client.0.smithi136.stderr:152666: 4 2024-09-15T20:51:11.856 INFO:tasks.workunit.client.0.smithi136.stderr:152666: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:51:11.856 INFO:tasks.workunit.client.0.smithi136.stderr:152666: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T20:51:11.856 INFO:tasks.workunit.client.0.smithi136.stderr:152666: osd.1 up in weight 1 up_from 80 up_thru 80 down_at 76 last_clean_interval [73,75) [v2:127.0.0.1:6810/3175724141,v1:127.0.0.1:6811/3175724141] [v2:127.0.0.1:6812/3175724141,v1:127.0.0.1:6813/3175724141] exists,up ba9ca9c7-78b0-44cd-b086-00a47c922481 2024-09-15T20:51:11.856 INFO:tasks.workunit.client.0.smithi136.stderr:152666: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-15T20:51:11.856 INFO:tasks.workunit.client.0.smithi136.stderr:152666: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-15T20:51:11.856 INFO:tasks.workunit.client.0.smithi136.stderr:152666: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-15T20:51:11.857 INFO:tasks.workunit.client.0.smithi136.stderr:152666: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2024-09-15T20:51:11.857 INFO:tasks.workunit.client.0.smithi136.stderr:152666: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-15T20:51:11.857 INFO:tasks.workunit.client.0.smithi136.stderr:152666: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-15T20:51:11.857 INFO:tasks.workunit.client.0.smithi136.stderr:152666: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-15T20:51:11.857 INFO:tasks.workunit.client.0.smithi136.stderr:152666: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-15T20:51:11.857 INFO:tasks.workunit.client.0.smithi136.stderr:152666: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-15T20:51:11.857 INFO:tasks.workunit.client.0.smithi136.stderr:152666: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-15T20:51:11.857 INFO:tasks.workunit.client.0.smithi136.stderr:152666: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-15T20:51:11.857 INFO:tasks.workunit.client.0.smithi136.stderr:152666: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-15T20:51:11.857 INFO:tasks.workunit.client.0.smithi136.stderr:152666: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-15T20:51:11.857 INFO:tasks.workunit.client.0.smithi136.stderr:152666: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-15T20:51:11.857 INFO:tasks.workunit.client.0.smithi136.stderr:152666: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-15T20:51:11.857 INFO:tasks.workunit.client.0.smithi136.stderr:152666: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-15T20:51:11.858 INFO:tasks.workunit.client.0.smithi136.stderr:152666: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-15T20:51:11.858 INFO:tasks.workunit.client.0.smithi136.stderr:152666: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-15T20:51:11.858 INFO:tasks.workunit.client.0.smithi136.stderr:152666: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-15T20:51:11.858 INFO:tasks.workunit.client.0.smithi136.stderr:152666: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-15T20:51:11.858 INFO:tasks.workunit.client.0.smithi136.stderr:152666: 1 2024-09-15T20:51:11.858 INFO:tasks.workunit.client.0.smithi136.stderr:152666: 2 2024-09-15T20:51:11.858 INFO:tasks.workunit.client.0.smithi136.stderr:152666: 3' 2024-09-15T20:51:11.858 INFO:tasks.workunit.client.0.smithi136.stderr:152666: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-15T20:51:11.858 INFO:tasks.workunit.client.0.smithi136.stderr:152666: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T20:51:11.858 INFO:tasks.workunit.client.0.smithi136.stderr:152666: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-15T20:51:11.858 INFO:tasks.workunit.client.0.smithi136.stderr:152666: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836539 2024-09-15T20:51:11.858 INFO:tasks.workunit.client.0.smithi136.stderr:152666: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836539 2024-09-15T20:51:11.858 INFO:tasks.workunit.client.0.smithi136.stderr:152666: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836539' 2024-09-15T20:51:11.859 INFO:tasks.workunit.client.0.smithi136.stderr:152666: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T20:51:12.872 INFO:tasks.workunit.client.0.smithi136.stderr:152666: //home/ubuntu/cephtest/clone.client.0/qa/standalonet.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:51:12.872 INFO:tasks.workunit.client.0.smithi136.stderr:152668: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:51:12.873 INFO:tasks.workunit.client.0.smithi136.stderr:152668: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-15T20:51:12.873 INFO:tasks.workunit.client.0.smithi136.stderr:152668: 4 2024-09-15T20:51:12.873 INFO:tasks.workunit.client.0.smithi136.stderr:152668: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:51:12.873 INFO:tasks.workunit.client.0.smithi136.stderr:152668: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T20:51:12.873 INFO:tasks.workunit.client.0.smithi136.stderr:152668: 2024-09-15T20:51:10.862+0000 7f972bfd9640 -1 osd.2 75 log_to_monitors true 2024-09-15T20:51:12.873 INFO:tasks.workunit.client.0.smithi136.stderr:152668: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:51:12.873 INFO:tasks.workunit.client.0.smithi136.stderr:152668: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:51:12.873 INFO:tasks.workunit.client.0.smithi136.stderr:152668: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:51:12.873 INFO:tasks.workunit.client.0.smithi136.stderr:152668: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-15T20:51:12.873 INFO:tasks.workunit.client.0.smithi136.stderr:152668: 5 2024-09-15T20:51:12.873 INFO:tasks.workunit.client.0.smithi136.stderr:152668: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:51:12.873 INFO:tasks.workunit.client.0.smithi136.stderr:152668: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T20:51:12.873 INFO:tasks.workunit.client.0.smithi136.stderr:152668: osd.2 up in weight 1 up_from 84 up_thru 72 down_at 76 last_clean_interval [72,75) [v2:127.0.0.1:6818/3831243018,v1:127.0.0.1:6819/3831243018] [v2:127.0.0.1:6820/3831243018,v1:127.0.0.1:6821/3831243018] exists,up 6eed0205-77e5-4f0f-9c89-cc7347b56baa 2024-09-15T20:51:12.874 INFO:tasks.workunit.client.0.smithi136.stderr:152668: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-15T20:51:12.874 INFO:tasks.workunit.client.0.smithi136.stderr:152668: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-15T20:51:12.874 INFO:tasks.workunit.client.0.smithi136.stderr:152668: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-15T20:51:12.874 INFO:tasks.workunit.client.0.smithi136.stderr:152668: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2024-09-15T20:51:12.874 INFO:tasks.workunit.client.0.smithi136.stderr:152668: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-15T20:51:12.874 INFO:tasks.workunit.client.0.smithi136.stderr:152668: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-15T20:51:12.874 INFO:tasks.workunit.client.0.smithi136.stderr:152668: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-15T20:51:12.874 INFO:tasks.workunit.client.0.smithi136.stderr:152668: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-15T20:51:12.874 INFO:tasks.workunit.client.0.smithi136.stderr:152668: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-15T20:51:12.874 INFO:tasks.workunit.client.0.smithi136.stderr:152668: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-15T20:51:12.874 INFO:tasks.workunit.client.0.smithi136.stderr:152668: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-15T20:51:12.874 INFO:tasks.workunit.client.0.smithi136.stderr:152668: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-15T20:51:12.874 INFO:tasks.workunit.client.0.smithi136.stderr:152668: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-15T20:51:12.874 INFO:tasks.workunit.client.0.smithi136.stderr:152668: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-15T20:51:12.875 INFO:tasks.workunit.client.0.smithi136.stderr:152668: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-15T20:51:12.875 INFO:tasks.workunit.client.0.smithi136.stderr:152668: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-15T20:51:12.875 INFO:tasks.workunit.client.0.smithi136.stderr:152668: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-15T20:51:12.875 INFO:tasks.workunit.client.0.smithi136.stderr:152668: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-15T20:51:12.875 INFO:tasks.workunit.client.0.smithi136.stderr:152668: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-15T20:51:12.875 INFO:tasks.workunit.client.0.smithi136.stderr:152668: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-15T20:51:12.875 INFO:tasks.workunit.client.0.smithi136.stderr:152668: 1 2024-09-15T20:51:12.875 INFO:tasks.workunit.client.0.smithi136.stderr:152668: 2 2024-09-15T20:51:12.875 INFO:tasks.workunit.client.0.smithi136.stderr:152668: 3' 2024-09-15T20:51:12.875 INFO:tasks.workunit.client.0.smithi136.stderr:152668: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-15T20:51:12.875 INFO:tasks.workunit.client.0.smithi136.stderr:152668: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T20:51:14.055 INFO:tasks.workunit.client.0.smithi136.stderr:152668/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-15T20:51:14.056 INFO:tasks.workunit.client.0.smithi136.stderr:152666: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=343597383682 2024-09-15T20:51:14.056 INFO:tasks.workunit.client.0.smithi136.stderr:152666: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 343597383682 2024-09-15T20:51:14.056 INFO:tasks.workunit.client.0.smithi136.stderr:152666: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836539 1-343597383682' 2024-09-15T20:51:14.056 INFO:tasks.workunit.client.0.smithi136.stderr:152666: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T20:51:14.056 INFO:tasks.workunit.client.0.smithi136.stderr:152666: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-15T20:51:14.056 INFO:tasks.workunit.client.0.smithi136.stderr:152666: Error ENXIO: problem getting command descriptions from osd.2 2024-09-15T20:51:14.056 INFO:tasks.workunit.client.0.smithi136.stderr:152666: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq= 2024-09-15T20:51:14.056 INFO:tasks.workunit.client.0.smithi136.stderr:152666: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z '' 2024-09-15T20:51:14.056 INFO:tasks.workunit.client.0.smithi136.stderr:152666: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: continue 2024-09-15T20:51:14.057 INFO:tasks.workunit.client.0.smithi136.stderr:152666: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T20:51:14.057 INFO:tasks.workunit.client.0.smithi136.stderr:152666: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-15T20:51:14.057 INFO:tasks.workunit.client.0.smithi136.stderr:152666: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=274877906956 2024-09-15T20:51:14.057 INFO:tasks.workunit.client.0.smithi136.stderr:152666: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 274877906956 2024-09-15T20:51:14.057 INFO:tasks.workunit.client.0.smithi136.stderr:152666: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836539 1-343597383682 3-274877906956' 2024-09-15T20:51:14.057 INFO:tasks.workunit.client.0.smithi136.stderr:152666: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T20:51:14.057 INFO:tasks.workunit.client.0.smithi136.stderr:152666: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836539 2024-09-15T20:51:14.057 INFO:tasks.workunit.client.0.smithi136.stderr:152666: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T20:51:14.057 INFO:tasks.workunit.client.0.smithi136.stderr:152666: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-15T20:51:14.057 INFO:tasks.workunit.client.0.smithi136.stderr:152666: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836539 2024-09-15T20:51:14.057 INFO:tasks.workunit.client.0.smithi136.stderr:152666: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T20:51:14.057 INFO:tasks.workunit.client.0.smithi136.stderr:152666: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836539 2024-09-15T20:51:14.057 INFO:tasks.workunit.client.0.smithi136.stderr:152666: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836539' 2024-09-15T20:51:14.058 INFO:tasks.workunit.client.0.smithi136.stderr:152666: waiting osd.0 seq 21474836539 2024-09-15T20:51:14.058 INFO:tasks.workunit.client.0.smithi136.stderr:152666: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T20:51:14.058 INFO:tasks.workunit.client.0.smithi136.stderr:152666: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836538 -lt 21474836539 2024-09-15T20:51:14.058 INFO:tasks.workunit.client.0.smithi136.stderr:152666: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-15T20:51:14.058 INFO:tasks.workunit.client.0.smithi136.stderr:152666: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-15T20:51:14.058 INFO:tasks.workunit.client.0.smithi136.stderr:152666: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T20:51:14.058 INFO:tasks.workunit.client.0.smithi136.stderr:152666: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836540 -lt 21474836539 2024-09-15T20:51:14.058 INFO:tasks.workunit.client.0.smithi136.stderr:152666: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T20:51:14.058 INFO:tasks.workunit.client.0.smithi136.stderr:152666: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-343597383682 2024-09-15T20:51:14.058 INFO:tasks.workunit.client.0.smithi136.stderr:152666: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T20:51:14.058 INFO:tasks.workunit.client.0.smithi136.stderr:152666: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-15T20:51:14.058 INFO:tasks.workunit.client.0.smithi136.stderr:152666: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-343597383682 2024-09-15T20:51:14.058 INFO:tasks.workunit.client.0.smithi136.stderr:152666: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T20:51:14.196 INFO:tasks.workunit.client.0.smithi136.stderr:152666: /home/ubu: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-15T20:51:14.197 INFO:tasks.workunit.client.0.smithi136.stderr:152668: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836540 2024-09-15T20:51:14.197 INFO:tasks.workunit.client.0.smithi136.stderr:152668: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836540 2024-09-15T20:51:14.197 INFO:tasks.workunit.client.0.smithi136.stderr:152668: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836540' 2024-09-15T20:51:14.197 INFO:tasks.workunit.client.0.smithi136.stderr:152668: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T20:51:14.197 INFO:tasks.workunit.client.0.smithi136.stderr:152668: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-15T20:51:14.197 INFO:tasks.workunit.client.0.smithi136.stderr:152668: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=343597383683 2024-09-15T20:51:14.197 INFO:tasks.workunit.client.0.smithi136.stderr:152668: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 343597383683 2024-09-15T20:51:14.197 INFO:tasks.workunit.client.0.smithi136.stderr:152668: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836540 1-343597383683' 2024-09-15T20:51:14.197 INFO:tasks.workunit.client.0.smithi136.stderr:152668: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T20:51:14.197 INFO:tasks.workunit.client.0.smithi136.stderr:152668: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-15T20:51:14.197 INFO:tasks.workunit.client.0.smithi136.stderr:152668: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=360777252866 2024-09-15T20:51:14.198 INFO:tasks.workunit.client.0.smithi136.stderr:152668: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 360777252866 2024-09-15T20:51:14.198 INFO:tasks.workunit.client.0.smithi136.stderr:152668: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836540 1-343597383683 2-360777252866' 2024-09-15T20:51:14.198 INFO:tasks.workunit.client.0.smithi136.stderr:152668: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T20:51:14.198 INFO:tasks.workunit.client.0.smithi136.stderr:152668: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-15T20:51:14.198 INFO:tasks.workunit.client.0.smithi136.stderr:152668: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=274877906957 2024-09-15T20:51:14.198 INFO:tasks.workunit.client.0.smithi136.stderr:152668: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 274877906957 2024-09-15T20:51:14.198 INFO:tasks.workunit.client.0.smithi136.stderr:152668: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836540 1-343597383683 2-360777252866 3-274877906957' 2024-09-15T20:51:14.198 INFO:tasks.workunit.client.0.smithi136.stderr:152668: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T20:51:14.198 INFO:tasks.workunit.client.0.smithi136.stderr:152668: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836540 2024-09-15T20:51:14.198 INFO:tasks.workunit.client.0.smithi136.stderr:152668: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T20:51:14.198 INFO:tasks.workunit.client.0.smithi136.stderr:152668: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-15T20:51:14.198 INFO:tasks.workunit.client.0.smithi136.stderr:152668: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836540 2024-09-15T20:51:14.198 INFO:tasks.workunit.client.0.smithi136.stderr:152668: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T20:51:14.199 INFO:tasks.workunit.client.0.smithi136.stderr:152668: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836540 2024-09-15T20:51:14.199 INFO:tasks.workunit.client.0.smithi136.stderr:152668: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836540' 2024-09-15T20:51:14.199 INFO:tasks.workunit.client.0.smithi136.stderr:152668: waiting osd.0 seq 21474836540 2024-09-15T20:51:14.199 INFO:tasks.workunit.client.0.smithi136.stderr:152668: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T20:51:14.199 INFO:tasks.workunit.client.0.smithi136.stderr:152668: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836540 -lt 21474836540 2024-09-15T20:51:14.199 INFO:tasks.workunit.client.0.smithi136.stderr:152668: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T20:51:14.199 INFO:tasks.workunit.client.0.smithi136.stderr:152668: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-343597383683 2024-09-15T20:51:14.199 INFO:tasks.workunit.client.0.smithi136.stderr:152668: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T20:51:14.199 INFO:tasks.workunit.client.0.smithi136.stderr:152668: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-15T20:51:15.721 INFO:tasks.workunit.client.0.smithi136.stderr:152668: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-hntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=343597383682 2024-09-15T20:51:15.721 INFO:tasks.workunit.client.0.smithi136.stderr:152666: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 343597383682' 2024-09-15T20:51:15.721 INFO:tasks.workunit.client.0.smithi136.stderr:152666: waiting osd.1 seq 343597383682 2024-09-15T20:51:15.721 INFO:tasks.workunit.client.0.smithi136.stderr:152666: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-15T20:51:15.721 INFO:tasks.workunit.client.0.smithi136.stderr:152666: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 343597383683 -lt 343597383682 2024-09-15T20:51:15.721 INFO:tasks.workunit.client.0.smithi136.stderr:152666: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T20:51:15.721 INFO:tasks.workunit.client.0.smithi136.stderr:152666: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 3-274877906956 2024-09-15T20:51:15.721 INFO:tasks.workunit.client.0.smithi136.stderr:152666: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T20:51:15.721 INFO:tasks.workunit.client.0.smithi136.stderr:152666: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=3 2024-09-15T20:51:15.722 INFO:tasks.workunit.client.0.smithi136.stderr:152666: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 3-274877906956 2024-09-15T20:51:15.722 INFO:tasks.workunit.client.0.smithi136.stderr:152666: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T20:51:15.722 INFO:tasks.workunit.client.0.smithi136.stderr:152666: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=274877906956 2024-09-15T20:51:15.722 INFO:tasks.workunit.client.0.smithi136.stderr:152666: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.3 seq 274877906956' 2024-09-15T20:51:15.722 INFO:tasks.workunit.client.0.smithi136.stderr:152666: waiting osd.3 seq 274877906956 2024-09-15T20:51:15.722 INFO:tasks.workunit.client.0.smithi136.stderr:152666: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-15T20:51:15.722 INFO:tasks.workunit.client.0.smithi136.stderr:152666: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 274877906956 -lt 274877906956 2024-09-15T20:51:15.722 INFO:tasks.workunit.client.0.smithi136.stderr:152666: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-15T20:51:15.722 INFO:tasks.workunit.client.0.smithi136.stderr:152666: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T20:51:15.722 INFO:tasks.workunit.client.0.smithi136.stderr:152666: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T20:51:15.722 INFO:tasks.workunit.client.0.smithi136.stderr:152666: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 5 == 0 2024-09-15T20:51:15.722 INFO:tasks.workunit.client.0.smithi136.stderr:152666: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-15T20:51:15.722 INFO:tasks.workunit.client.0.smithi136.stderr:152666: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-15T20:51:15.722 INFO:tasks.workunit.client.0.smithi136.stderr:152666: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-15T20:51:15.723 INFO:tasks.workunit.client.0.smithi136.stderr:152666: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T20:51:15.723 INFO:tasks.workunit.client.0.smithi136.stderr:152666: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T20:51:15.723 INFO:tasks.workunit.client.0.smithi136.stderr:152666: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T20:51:15.723 INFO:tasks.workunit.client.0.smithi136.stderr:152666: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T20:51:15.723 INFO:tasks.workunit.client.0.smithi136.stderr:152666: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=1 2024-09-15T20:51:15.723 INFO:tasks.workunit.client.0.smithi136.stderr:152666: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-15T20:51:15.723 INFO:tasks.workunit.client.0.smithi136.stderr:152666: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T20:51:15.723 INFO:tasks.workunit.client.0.smithi136.stderr:152666: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T20:51:15.723 INFO:tasks.workunit.client.0.smithi136.stderr:152666: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 = 5 2024-09-15T20:51:15.723 INFO:tasks.workunit.client.0.smithi136.stderr:152666: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 1 '!=' -1 2024-09-15T20:51:15.723 INFO:tasks.workunit.client.0.smithi136.stderr:152666: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: loop=0 2024-09-15T20:51:16.483 INFO:tasks.workunit.client.0.smithi136.stderr:152666: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1652: wait_for_clean: num_active_clean=1 2024-09-15T20:51:16.484 INFO:tasks.workunit.client.0.smithi136.stderr:152666: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-15T20:51:16.484 INFO:tasks.workunit.client.0.smithi136.stderr:152666: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 0.1 2024-09-15T20:51:16.484 INFO:tasks.workunit.client.0.smithi136.stderr:152666: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-15T20:51:16.484 INFO:tasks.workunit.client.0.smithi136.stderr:152666: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-15T20:51:16.484 INFO:tasks.workunit.client.0.smithi136.stderr:152666: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-15T20:51:16.484 INFO:tasks.workunit.client.0.smithi136.stderr:152666: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-15T20:51:16.484 INFO:tasks.workunit.client.0.smithi136.stderr:152666: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T20:51:16.484 INFO:tasks.workunit.client.0.smithi136.stderr:152666: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T20:51:16.484 INFO:tasks.workunit.client.0.smithi136.stderr:152666: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T20:51:16.484 INFO:tasks.workunit.client.0.smithi136.stderr:152666: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T20:51:16.484 INFO:tasks.workunit.client.0.smithi136.stderr:152666: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=5 2024-09-15T20:51:16.485 INFO:tasks.workunit.client.0.smithi136.stderr:152666: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-15T20:51:16.485 INFO:tasks.workunit.client.0.smithi136.stderr:152666: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T20:51:16.485 INFO:tasks.workunit.client.0.smithi136.stderr:152666: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T20:51:16.485 INFO:tasks.workunit.client.0.smithi136.stderr:152666: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 = 5 2024-09-15T20:51:16.485 INFO:tasks.workunit.client.0.smithi136.stderr:152666: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-15T20:51:16.485 INFO:tasks.workunit.client.0.smithi136.stderr:152666: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-15T20:51:16.485 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: run_in_background: return 0 2024-09-15T20:51:16.485 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: wait_background: for pid in $pids 2024-09-15T20:51:16.485 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2175: wait_background: wait 152664 2024-09-15T20:51:16.823 INFO:tasks.workunit.client.0.smithi136.stderr:elpers.sh:2220: flush_pg_stats: echo 1-343597383683 2024-09-15T20:51:16.824 INFO:tasks.workunit.client.0.smithi136.stderr:152668: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T20:51:16.824 INFO:tasks.workunit.client.0.smithi136.stderr:152668: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=343597383683 2024-09-15T20:51:16.824 INFO:tasks.workunit.client.0.smithi136.stderr:152668: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 343597383683' 2024-09-15T20:51:16.824 INFO:tasks.workunit.client.0.smithi136.stderr:152668: waiting osd.1 seq 343597383683 2024-09-15T20:51:16.824 INFO:tasks.workunit.client.0.smithi136.stderr:152668: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-15T20:51:16.824 INFO:tasks.workunit.client.0.smithi136.stderr:152668: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 343597383683 -lt 343597383683 2024-09-15T20:51:16.824 INFO:tasks.workunit.client.0.smithi136.stderr:152668: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T20:51:16.824 INFO:tasks.workunit.client.0.smithi136.stderr:152668: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-360777252866 2024-09-15T20:51:16.824 INFO:tasks.workunit.client.0.smithi136.stderr:152668: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T20:51:16.824 INFO:tasks.workunit.client.0.smithi136.stderr:152668: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-15T20:51:16.825 INFO:tasks.workunit.client.0.smithi136.stderr:152668: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-360777252866 2024-09-15T20:51:16.825 INFO:tasks.workunit.client.0.smithi136.stderr:152668: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T20:51:16.825 INFO:tasks.workunit.client.0.smithi136.stderr:152668: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=360777252866 2024-09-15T20:51:16.825 INFO:tasks.workunit.client.0.smithi136.stderr:152668: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 360777252866' 2024-09-15T20:51:16.825 INFO:tasks.workunit.client.0.smithi136.stderr:152668: waiting osd.2 seq 360777252866 2024-09-15T20:51:16.825 INFO:tasks.workunit.client.0.smithi136.stderr:152668: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-15T20:51:16.825 INFO:tasks.workunit.client.0.smithi136.stderr:152668: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 360777252866 -lt 360777252866 2024-09-15T20:51:16.825 INFO:tasks.workunit.client.0.smithi136.stderr:152668: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T20:51:16.825 INFO:tasks.workunit.client.0.smithi136.stderr:152668: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 3-274877906957 2024-09-15T20:51:16.825 INFO:tasks.workunit.client.0.smithi136.stderr:152668: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T20:51:16.825 INFO:tasks.workunit.client.0.smithi136.stderr:152668: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=3 2024-09-15T20:51:16.825 INFO:tasks.workunit.client.0.smithi136.stderr:152668: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 3-274877906957 2024-09-15T20:51:16.825 INFO:tasks.workunit.client.0.smithi136.stderr:152668: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T20:51:16.826 INFO:tasks.workunit.client.0.smithi136.stderr:152668: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=274877906957 2024-09-15T20:51:16.826 INFO:tasks.workunit.client.0.smithi136.stderr:152668: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.3 seq 274877906957' 2024-09-15T20:51:16.826 INFO:tasks.workunit.client.0.smithi136.stderr:152668: waiting osd.3 seq 274877906957 2024-09-15T20:51:16.826 INFO:tasks.workunit.client.0.smithi136.stderr:152668: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-15T20:51:16.826 INFO:tasks.workunit.client.0.smithi136.stderr:152668: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 274877906956 -lt 274877906957 2024-09-15T20:51:16.826 INFO:tasks.workunit.client.0.smithi136.stderr:152668: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-15T20:51:16.826 INFO:tasks.workunit.client.0.smithi136.stderr:152668: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-15T20:51:16.827 INFO:tasks.workunit.client.0.smithi136.stderr:152668: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-15T20:51:16.827 INFO:tasks.workunit.client.0.smithi136.stderr:152668: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 274877906958 -lt 274877906957 2024-09-15T20:51:16.827 INFO:tasks.workunit.client.0.smithi136.stderr:152668: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-15T20:51:16.827 INFO:tasks.workunit.client.0.smithi136.stderr:152668: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T20:51:16.827 INFO:tasks.workunit.client.0.smithi136.stderr:152668: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T20:51:17.482 INFO:tasks.workunit.client.0.smithi136.stderr:152668: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 5 == 0 2024-09-15T20:51:17.483 INFO:tasks.workunit.client.0.smithi136.stderr:152668: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-15T20:51:17.483 INFO:tasks.workunit.client.0.smithi136.stderr:152668: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-15T20:51:17.483 INFO:tasks.workunit.client.0.smithi136.stderr:152668: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-15T20:51:17.483 INFO:tasks.workunit.client.0.smithi136.stderr:152668: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T20:51:17.483 INFO:tasks.workunit.client.0.smithi136.stderr:152668: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T20:51:17.483 INFO:tasks.workunit.client.0.smithi136.stderr:152668: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T20:51:17.483 INFO:tasks.workunit.client.0.smithi136.stderr:152668: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T20:51:17.483 INFO:tasks.workunit.client.0.smithi136.stderr:152668: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=5 2024-09-15T20:51:17.483 INFO:tasks.workunit.client.0.smithi136.stderr:152668: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-15T20:51:17.483 INFO:tasks.workunit.client.0.smithi136.stderr:152668: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T20:51:17.483 INFO:tasks.workunit.client.0.smithi136.stderr:152668: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T20:51:17.484 INFO:tasks.workunit.client.0.smithi136.stderr:152668: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 = 5 2024-09-15T20:51:17.484 INFO:tasks.workunit.client.0.smithi136.stderr:152668: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-15T20:51:17.484 INFO:tasks.workunit.client.0.smithi136.stderr:152668: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-15T20:51:17.484 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: run_in_background: return 0 2024-09-15T20:51:17.484 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: wait_background: eval 'pids='\'''\''' 2024-09-15T20:51:17.484 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: wait_background: pids= 2024-09-15T20:51:17.484 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: wait_background: return 0 2024-09-15T20:51:17.484 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:212: corrupt_and_repair_two: return_code=0 2024-09-15T20:51:17.484 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:213: corrupt_and_repair_two: '[' 0 -ne 0 ']' 2024-09-15T20:51:17.484 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:215: corrupt_and_repair_two: rados --pool ecpool get SOMETHING td/osd-scrub-repair/COPY 2024-09-15T20:51:17.514 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:216: corrupt_and_repair_two: diff td/osd-scrub-repair/ORIGINAL td/osd-scrub-repair/COPY 2024-09-15T20:51:17.517 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:263: corrupt_and_repair_erasure_coded: corrupt_and_repair_two td/osd-scrub-repair ecpool 3 1 2024-09-15T20:51:17.517 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:185: corrupt_and_repair_two: local dir=td/osd-scrub-repair 2024-09-15T20:51:17.517 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:186: corrupt_and_repair_two: local poolname=ecpool 2024-09-15T20:51:17.517 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:187: corrupt_and_repair_two: local first=3 2024-09-15T20:51:17.518 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:188: corrupt_and_repair_two: local second=1 2024-09-15T20:51:17.518 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:193: corrupt_and_repair_two: pids= 2024-09-15T20:51:17.518 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:194: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 3 SOMETHING remove 2024-09-15T20:51:17.518 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2135: run_in_background: local pid_variable=pids 2024-09-15T20:51:17.518 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2136: run_in_background: shift 2024-09-15T20:51:17.518 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2140: run_in_background: eval 'pids+=" 155087"' 2024-09-15T20:51:17.518 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2140: run_in_background: pids+=' 155087' 2024-09-15T20:51:17.523 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:195: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 1 SOMETHING remove 2024-09-15T20:51:17.523 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2135: run_in_background: local pid_variable=pids 2024-09-15T20:51:17.524 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2136: run_in_background: shift 2024-09-15T20:51:17.524 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2140: run_in_background: eval 'pids+=" 155089"' 2024-09-15T20:51:17.524 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: run_in_background: objectstore_tool td/osd-scrub-repair 3 SOMETHING remove 2024-09-15T20:51:17.524 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2140: run_in_background: pids+=' 155089' 2024-09-15T20:51:17.524 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:196: corrupt_and_repair_two: wait_background pids 2024-09-15T20:51:17.524 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: wait_background: pids=' 155087 155089' 2024-09-15T20:51:17.524 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: run_in_background: sed 's/^/155090: /' 2024-09-15T20:51:17.524 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: wait_background: return_code=0 2024-09-15T20:51:17.524 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: wait_background: for pid in $pids 2024-09-15T20:51:17.524 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2175: wait_background: wait 155087 2024-09-15T20:51:17.524 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: run_in_background: objectstore_tool td/osd-scrub-repair 1 SOMETHING remove 2024-09-15T20:51:17.524 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: run_in_background: sed 's/^/155092: /' 2024-09-15T20:51:20.626 INFO:tasks.workunit.client.0.smithi136.stderr:155090: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2024-09-15T20:51:20.626 INFO:tasks.workunit.client.0.smithi136.stderr:155090: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2024-09-15T20:51:20.626 INFO:tasks.workunit.client.0.smithi136.stderr:155090: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=3 2024-09-15T20:51:20.626 INFO:tasks.workunit.client.0.smithi136.stderr:155090: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2024-09-15T20:51:20.626 INFO:tasks.workunit.client.0.smithi136.stderr:155090: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 3 SOMETHING remove 2024-09-15T20:51:20.626 INFO:tasks.workunit.client.0.smithi136.stderr:155090: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-09-15T20:51:20.626 INFO:tasks.workunit.client.0.smithi136.stderr:155090: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2024-09-15T20:51:20.626 INFO:tasks.workunit.client.0.smithi136.stderr:155090: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=3 2024-09-15T20:51:20.627 INFO:tasks.workunit.client.0.smithi136.stderr:155090: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2024-09-15T20:51:20.627 INFO:tasks.workunit.client.0.smithi136.stderr:155090: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.3 2024-09-15T20:51:20.627 INFO:tasks.workunit.client.0.smithi136.stderr:155090: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-15T20:51:20.627 INFO:tasks.workunit.client.0.smithi136.stderr:155090: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-15T20:51:20.627 INFO:tasks.workunit.client.0.smithi136.stderr:155090: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-15T20:51:20.627 INFO:tasks.workunit.client.0.smithi136.stderr:155090: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-15T20:51:20.627 INFO:tasks.workunit.client.0.smithi136.stderr:155090: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-15T20:51:20.627 INFO:tasks.workunit.client.0.smithi136.stderr:155090: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-15T20:51:20.627 INFO:tasks.workunit.client.0.smithi136.stderr:155090: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 3 SOMETHING remove 2024-09-15T20:51:20.627 INFO:tasks.workunit.client.0.smithi136.stderr:155090: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-15T20:51:20.627 INFO:tasks.workunit.client.0.smithi136.stderr:155090: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-15T20:51:20.627 INFO:tasks.workunit.client.0.smithi136.stderr:155090: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=3 2024-09-15T20:51:20.627 INFO:tasks.workunit.client.0.smithi136.stderr:155090: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-15T20:51:20.628 INFO:tasks.workunit.client.0.smithi136.stderr:155090: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/3 2024-09-15T20:51:20.628 INFO:tasks.workunit.client.0.smithi136.stderr:155090: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/3 SOMETHING remove 2024-09-15T20:51:20.628 INFO:tasks.workunit.client.0.smithi136.stderr:155090: remove 0#2:eb822e21:::SOMETHING:head# 2024-09-15T20:51:20.628 INFO:tasks.workunit.client.0.smithi136.stderr:155090: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 3 2024-09-15T20:51:20.628 INFO:tasks.workunit.client.0.smithi136.stderr:155090: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2024-09-15T20:51:20.628 INFO:tasks.workunit.client.0.smithi136.stderr:155090: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-15T20:51:20.628 INFO:tasks.workunit.client.0.smithi136.stderr:155090: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=3 2024-09-15T20:51:20.628 INFO:tasks.workunit.client.0.smithi136.stderr:155090: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-15T20:51:20.628 INFO:tasks.workunit.client.0.smithi136.stderr:155090: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/3 2024-09-15T20:51:20.628 INFO:tasks.workunit.client.0.smithi136.stderr:155090: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=ce4892c7-6c6d-4b4f-8e6f-07a3e11133e5 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-15T20:51:20.628 INFO:tasks.workunit.client.0.smithi136.stderr:155090: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T20:51:20.629 INFO:tasks.workunit.client.0.smithi136.stderr:155090: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T20:51:20.629 INFO:tasks.workunit.client.0.smithi136.stderr:155090: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T20:51:20.629 INFO:tasks.workunit.client.0.smithi136.stderr:155090: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/3' 2024-09-15T20:51:20.629 INFO:tasks.workunit.client.0.smithi136.stderr:155090: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/3/journal' 2024-09-15T20:51:20.629 INFO:tasks.workunit.client.0.smithi136.stderr:155090: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-15T20:51:20.630 INFO:tasks.workunit.client.0.smithi136.stderr:155090: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-15T20:51:20.630 INFO:tasks.workunit.client.0.smithi136.stderr:155090: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-15T20:51:20.630 INFO:tasks.workunit.client.0.smithi136.stderr:155090: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-15T20:51:20.630 INFO:tasks.workunit.client.0.smithi136.stderr:155090: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-15T20:51:20.630 INFO:tasks.workunit.client.0.smithi136.stderr:155090: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-15T20:51:20.630 INFO:tasks.workunit.client.0.smithi136.stderr:155090: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-15T20:51:20.630 INFO:tasks.workunit.client.0.smithi136.stderr:155090: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T20:51:20.630 INFO:tasks.workunit.client.0.smithi136.stderr:155090: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52602 2024-09-15T20:51:20.630 INFO:tasks.workunit.client.0.smithi136.stderr:155090: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.52602/$cluster-$name.asok' 2024-09-15T20:51:20.630 INFO:tasks.workunit.client.0.smithi136.stderr:155090: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.52602/$cluster-$name.asok' 2024-09-15T20:51:20.630 INFO:tasks.workunit.client.0.smithi136.stderr:155090: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-15T20:51:20.630 INFO:tasks.workunit.client.0.smithi136.stderr:155090: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-15T20:51:20.630 INFO:tasks.workunit.client.0.smithi136.stderr:155090: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-15T20:51:20.630 INFO:tasks.workunit.client.0.smithi136.stderr:155090: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T20:51:20.631 INFO:tasks.workunit.client.0.smithi136.stderr:155090: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T20:51:20.631 INFO:tasks.workunit.client.0.smithi136.stderr:155090: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T20:51:20.631 INFO:tasks.workunit.client.0.smithi136.stderr:155090: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T20:51:20.631 INFO:tasks.workunit.client.0.smithi136.stderr:155090: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-15T20:51:20.631 INFO:tasks.workunit.client.0.smithi136.stderr:155090: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-15T20:51:20.631 INFO:tasks.workunit.client.0.smithi136.stderr:155090: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/3 2024-09-15T20:51:20.631 INFO:tasks.workunit.client.0.smithi136.stderr:155090: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.3 2024-09-15T20:51:20.631 INFO:tasks.workunit.client.0.smithi136.stderr:155090: start osd.3 2024-09-15T20:51:20.631 INFO:tasks.workunit.client.0.smithi136.stderr:155090: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 3 --fsid=ce4892c7-6c6d-4b4f-8e6f-07a3e11133e5 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/3 --osd-journal=td/osd-scrub-repair/3/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.52602/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T20:51:20.982 INFO:tasks.workunit.client.0.smithi136.stderr:1155092: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2024-09-15T20:51:20.982 INFO:tasks.workunit.client.0.smithi136.stderr:155092: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2024-09-15T20:51:20.982 INFO:tasks.workunit.client.0.smithi136.stderr:155092: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=1 2024-09-15T20:51:20.983 INFO:tasks.workunit.client.0.smithi136.stderr:155092: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2024-09-15T20:51:20.983 INFO:tasks.workunit.client.0.smithi136.stderr:155092: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 SOMETHING remove 2024-09-15T20:51:20.983 INFO:tasks.workunit.client.0.smithi136.stderr:155092: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-09-15T20:51:20.983 INFO:tasks.workunit.client.0.smithi136.stderr:155092: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2024-09-15T20:51:20.983 INFO:tasks.workunit.client.0.smithi136.stderr:155092: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=1 2024-09-15T20:51:20.983 INFO:tasks.workunit.client.0.smithi136.stderr:155092: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2024-09-15T20:51:20.983 INFO:tasks.workunit.client.0.smithi136.stderr:155092: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-09-15T20:51:20.983 INFO:tasks.workunit.client.0.smithi136.stderr:155092: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-15T20:51:20.983 INFO:tasks.workunit.client.0.smithi136.stderr:155092: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-15T20:51:20.983 INFO:tasks.workunit.client.0.smithi136.stderr:155092: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-15T20:51:20.983 INFO:tasks.workunit.client.0.smithi136.stderr:155092: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-15T20:51:20.983 INFO:tasks.workunit.client.0.smithi136.stderr:155092: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-15T20:51:20.983 INFO:tasks.workunit.client.0.smithi136.stderr:155092: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-15T20:51:20.983 INFO:tasks.workunit.client.0.smithi136.stderr:155092: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 SOMETHING remove 2024-09-15T20:51:20.984 INFO:tasks.workunit.client.0.smithi136.stderr:155092: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-15T20:51:20.984 INFO:tasks.workunit.client.0.smithi136.stderr:155092: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-15T20:51:20.984 INFO:tasks.workunit.client.0.smithi136.stderr:155092: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=1 2024-09-15T20:51:20.984 INFO:tasks.workunit.client.0.smithi136.stderr:155092: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-15T20:51:20.984 INFO:tasks.workunit.client.0.smithi136.stderr:155092: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-09-15T20:51:20.984 INFO:tasks.workunit.client.0.smithi136.stderr:155092: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 SOMETHING remove 2024-09-15T20:51:20.984 INFO:tasks.workunit.client.0.smithi136.stderr:155092: remove 1#2:eb822e21:::SOMETHING:head# 2024-09-15T20:51:20.984 INFO:tasks.workunit.client.0.smithi136.stderr:155092: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2024-09-15T20:51:20.984 INFO:tasks.workunit.client.0.smithi136.stderr:155092: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2024-09-15T20:51:20.984 INFO:tasks.workunit.client.0.smithi136.stderr:155092: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-15T20:51:20.984 INFO:tasks.workunit.client.0.smithi136.stderr:155092: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=1 2024-09-15T20:51:20.984 INFO:tasks.workunit.client.0.smithi136.stderr:155092: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-15T20:51:20.984 INFO:tasks.workunit.client.0.smithi136.stderr:155092: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-09-15T20:51:20.984 INFO:tasks.workunit.client.0.smithi136.stderr:155092: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=ce4892c7-6c6d-4b4f-8e6f-07a3e11133e5 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-15T20:51:20.985 INFO:tasks.workunit.client.0.smithi136.stderr:155092: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T20:51:20.985 INFO:tasks.workunit.client.0.smithi136.stderr:155092: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T20:51:20.985 INFO:tasks.workunit.client.0.smithi136.stderr:155092: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T20:51:20.985 INFO:tasks.workunit.client.0.smithi136.stderr:155092: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-09-15T20:51:20.985 INFO:tasks.workunit.client.0.smithi136.stderr:155092: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-09-15T20:51:20.985 INFO:tasks.workunit.client.0.smithi136.stderr:155092: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-15T20:51:20.985 INFO:tasks.workunit.client.0.smithi136.stderr:155092: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-15T20:51:20.988 INFO:tasks.workunit.client.0.smithi136.stderr:155092: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-15T20:51:20.988 INFO:tasks.workunit.client.0.smithi136.stderr:155092: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-15T20:51:20.988 INFO:tasks.workunit.client.0.smithi136.stderr:155092: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-15T20:51:20.988 INFO:tasks.workunit.client.0.smithi136.stderr:155092: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-15T20:51:20.988 INFO:tasks.workunit.client.0.smithi136.stderr:155092: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-15T20:51:20.988 INFO:tasks.workunit.client.0.smithi136.stderr:155092: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T20:51:20.989 INFO:tasks.workunit.client.0.smithi136.stderr:155092: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52602 2024-09-15T20:51:20.989 INFO:tasks.workunit.client.0.smithi136.stderr:155092: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.52602/$cluster-$name.asok' 2024-09-15T20:51:20.989 INFO:tasks.workunit.client.0.smithi136.stderr:155092: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.52602/$cluster-$name.asok' 2024-09-15T20:51:20.989 INFO:tasks.workunit.client.0.smithi136.stderr:155092: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-15T20:51:20.989 INFO:tasks.workunit.client.0.smithi136.stderr:155092: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-15T20:51:20.989 INFO:tasks.workunit.client.0.smithi136.stderr:155092: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-15T20:51:20.989 INFO:tasks.workunit.client.0.smithi136.stderr:155092: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T20:51:20.989 INFO:tasks.workunit.client.0.smithi136.stderr:155092: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T20:51:20.989 INFO:tasks.workunit.client.0.smithi136.stderr:155092: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T20:51:20.989 INFO:tasks.workunit.client.0.smithi136.stderr:155092: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T20:51:20.989 INFO:tasks.workunit.client.0.smithi136.stderr:155092: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-15T20:51:20.989 INFO:tasks.workunit.client.0.smithi136.stderr:155092: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-15T20:51:20.989 INFO:tasks.workunit.client.0.smithi136.stderr:155092: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-09-15T20:51:20.989 INFO:tasks.workunit.client.0.smithi136.stderr:155092: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.1 2024-09-15T20:51:20.990 INFO:tasks.workunit.client.0.smithi136.stderr:155092: start osd.1 2024-09-15T20:51:20.990 INFO:tasks.workunit.client.0.smithi136.stderr:155092: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 1 --fsid=ce4892c7-6c6d-4b4f-8e6f-07a3e11133e5 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.52602/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T20:51:25.373 INFO:tasks.workunit.client.0.smithi136.stderr:155090: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/3/whoami 2024-09-15T20:51:25.373 INFO:tasks.workunit.client.0.smithi136.stderr:155090: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 3 = 3 ']' 2024-09-15T20:51:25.373 INFO:tasks.workunit.client.0.smithi136.stderr:155090: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-15T20:51:25.374 INFO:tasks.workunit.client.0.smithi136.stderr:155090: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-15T20:51:25.374 INFO:tasks.workunit.client.0.smithi136.stderr:155090: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-15T20:51:25.374 INFO:tasks.workunit.client.0.smithi136.stderr:155090: 2024-09-15T20:51:20.656+0000 7f034ff4d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T20:51:25.374 INFO:tasks.workunit.client.0.smithi136.stderr:155090: 2024-09-15T20:51:20.658+0000 7f034ff4d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T20:51:25.374 INFO:tasks.workunit.client.0.smithi136.stderr:155090: 2024-09-15T20:51:20.660+0000 7f034ff4d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T20:51:25.374 INFO:tasks.workunit.client.0.smithi136.stderr:155090: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 3 2024-09-15T20:51:25.374 INFO:tasks.workunit.client.0.smithi136.stderr:155090: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-15T20:51:25.374 INFO:tasks.workunit.client.0.smithi136.stderr:155090: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=3 2024-09-15T20:51:25.374 INFO:tasks.workunit.client.0.smithi136.stderr:155090: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-15T20:51:25.374 INFO:tasks.workunit.client.0.smithi136.stderr:155090: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-15T20:51:25.374 INFO:tasks.workunit.client.0.smithi136.stderr:155090: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:51:25.374 INFO:tasks.workunit.client.0.smithi136.stderr:155090: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-15T20:51:25.374 INFO:tasks.workunit.client.0.smithi136.stderr:155090: 0 2024-09-15T20:51:25.375 INFO:tasks.workunit.client.0.smithi136.stderr:155090: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:51:25.375 INFO:tasks.workunit.client.0.smithi136.stderr:155090: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-15T20:51:25.375 INFO:tasks.workunit.client.0.smithi136.stderr:155090: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:51:25.375 INFO:tasks.workunit.client.0.smithi136.stderr:155090: 2024-09-15T20:51:21.803+0000 7f034ff4d640 -1 Falling back to public interface 2024-09-15T20:51:25.375 INFO:tasks.workunit.client.0.smithi136.stderr:155090: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:51:25.375 INFO:tasks.workunit.client.0.smithi136.stderr:155090: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:51:25.375 INFO:tasks.workunit.client.0.smithi136.stderr:155090: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-15T20:51:25.375 INFO:tasks.workunit.client.0.smithi136.stderr:155090: 1 2024-09-15T20:51:25.375 INFO:tasks.workunit.client.0.smithi136.stderr:155090: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:51:25.375 INFO:tasks.workunit.client.0.smithi136.stderr:155090: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-15T20:51:25.375 INFO:tasks.workunit.client.0.smithi136.stderr:155090: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:51:25.375 INFO:tasks.workunit.client.0.smithi136.stderr:155090: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:51:25.375 INFO:tasks.workunit.client.0.smithi136.stderr:155090: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:51:25.375 INFO:tasks.workunit.client.0.smithi136.stderr:155090: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-15T20:51:25.375 INFO:tasks.workunit.client.0.smithi136.stderr:155090: 2 2024-09-15T20:51:25.376 INFO:tasks.workunit.client.0.smithi136.stderr:155090: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:51:25.376 INFO:tasks.workunit.client.0.smithi136.stderr:155090: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-15T20:51:25.376 INFO:tasks.workunit.client.0.smithi136.stderr:155090: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:51:25.376 INFO:tasks.workunit.client.0.smithi136.stderr:155090: 2024-09-15T20:51:24.346+0000 7f034ff4d640 -1 osd.3 86 log_to_monitors true 2024-09-15T20:51:25.376 INFO:tasks.workunit.client.0.smithi136.stderr:155090: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:51:25.376 INFO:tasks.workunit.client.0.smithi136.stderr:155090: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:51:25.376 INFO:tasks.workunit.client.0.smithi136.stderr:155090: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-15T20:51:25.376 INFO:tasks.workunit.client.0.smithi136.stderr:155090: 3 2024-09-15T20:51:25.376 INFO:tasks.workunit.client.0.smithi136.stderr:155090: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:51:25.376 INFO:tasks.workunit.client.0.smithi136.stderr:155090: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-15T20:51:25.804 INFO:tasks.workunit.client.0.smithi136.stderr:155090: osd.3 up in weight 1 up_from 91 up_thru 84 down_at 87 last_c55092: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-09-15T20:51:25.804 INFO:tasks.workunit.client.0.smithi136.stderr:155092: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 1 = 1 ']' 2024-09-15T20:51:25.804 INFO:tasks.workunit.client.0.smithi136.stderr:155092: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-15T20:51:25.804 INFO:tasks.workunit.client.0.smithi136.stderr:155092: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-15T20:51:25.804 INFO:tasks.workunit.client.0.smithi136.stderr:155092: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-15T20:51:25.804 INFO:tasks.workunit.client.0.smithi136.stderr:155092: 2024-09-15T20:51:21.010+0000 7f0bb1210640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T20:51:25.804 INFO:tasks.workunit.client.0.smithi136.stderr:155092: 2024-09-15T20:51:21.013+0000 7f0bb1210640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T20:51:25.804 INFO:tasks.workunit.client.0.smithi136.stderr:155092: 2024-09-15T20:51:21.014+0000 7f0bb1210640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T20:51:25.805 INFO:tasks.workunit.client.0.smithi136.stderr:155092: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 1 2024-09-15T20:51:25.805 INFO:tasks.workunit.client.0.smithi136.stderr:155092: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-15T20:51:25.805 INFO:tasks.workunit.client.0.smithi136.stderr:155092: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2024-09-15T20:51:25.805 INFO:tasks.workunit.client.0.smithi136.stderr:155092: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-15T20:51:25.805 INFO:tasks.workunit.client.0.smithi136.stderr:155092: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-15T20:51:25.806 INFO:tasks.workunit.client.0.smithi136.stderr:155092: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:51:25.806 INFO:tasks.workunit.client.0.smithi136.stderr:155092: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-15T20:51:25.806 INFO:tasks.workunit.client.0.smithi136.stderr:155092: 0 2024-09-15T20:51:25.806 INFO:tasks.workunit.client.0.smithi136.stderr:155092: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:51:25.806 INFO:tasks.workunit.client.0.smithi136.stderr:155092: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T20:51:25.806 INFO:tasks.workunit.client.0.smithi136.stderr:155092: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:51:25.807 INFO:tasks.workunit.client.0.smithi136.stderr:155092: 2024-09-15T20:51:22.395+0000 7f0bb1210640 -1 Falling back to public interface 2024-09-15T20:51:25.807 INFO:tasks.workunit.client.0.smithi136.stderr:155092: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:51:25.807 INFO:tasks.workunit.client.0.smithi136.stderr:155092: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:51:25.807 INFO:tasks.workunit.client.0.smithi136.stderr:155092: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-15T20:51:25.807 INFO:tasks.workunit.client.0.smithi136.stderr:155092: 1 2024-09-15T20:51:25.807 INFO:tasks.workunit.client.0.smithi136.stderr:155092: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:51:25.807 INFO:tasks.workunit.client.0.smithi136.stderr:155092: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T20:51:25.808 INFO:tasks.workunit.client.0.smithi136.stderr:155092: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:51:25.808 INFO:tasks.workunit.client.0.smithi136.stderr:155092: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:51:25.808 INFO:tasks.workunit.client.0.smithi136.stderr:155092: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:51:25.808 INFO:tasks.workunit.client.0.smithi136.stderr:155092: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-15T20:51:25.808 INFO:tasks.workunit.client.0.smithi136.stderr:155092: 2 2024-09-15T20:51:25.808 INFO:tasks.workunit.client.0.smithi136.stderr:155092: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:51:25.809 INFO:tasks.workunit.client.0.smithi136.stderr:155092: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T20:51:25.809 INFO:tasks.workunit.client.0.smithi136.stderr:155092: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:51:25.809 INFO:tasks.workunit.client.0.smithi136.stderr:155092: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:51:25.809 INFO:tasks.workunit.client.0.smithi136.stderr:155092: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:51:25.809 INFO:tasks.workunit.client.0.smithi136.stderr:155092: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-15T20:51:25.809 INFO:tasks.workunit.client.0.smithi136.stderr:155092: 3 2024-09-15T20:51:25.809 INFO:tasks.workunit.client.0.smithi136.stderr:155092: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:51:25.810 INFO:tasks.workunit.client.0.smithi136.stderr:155092: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T20:51:25.810 INFO:tasks.workunit.client.0.smithi136.stderr:155092: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:51:26.183 INFO:tasks.workunit.client.0.smithi136.stderr:155092: 2024-09-15T20:51:25.801+0000 7f0bb1210640 -lean_interval [64,86) [v2:127.0.0.1:6810/161040107,v1:127.0.0.1:6811/161040107] [v2:127.0.0.1:6812/161040107,v1:127.0.0.1:6813/161040107] exists,up 48807f11-9c3a-49d3-8033-a247f29eadfa 2024-09-15T20:51:26.183 INFO:tasks.workunit.client.0.smithi136.stderr:155090: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-15T20:51:26.183 INFO:tasks.workunit.client.0.smithi136.stderr:155090: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-15T20:51:26.184 INFO:tasks.workunit.client.0.smithi136.stderr:155090: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-15T20:51:26.184 INFO:tasks.workunit.client.0.smithi136.stderr:155090: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2024-09-15T20:51:26.184 INFO:tasks.workunit.client.0.smithi136.stderr:155090: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-15T20:51:26.184 INFO:tasks.workunit.client.0.smithi136.stderr:155090: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-15T20:51:26.184 INFO:tasks.workunit.client.0.smithi136.stderr:155090: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-15T20:51:26.184 INFO:tasks.workunit.client.0.smithi136.stderr:155090: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-15T20:51:26.184 INFO:tasks.workunit.client.0.smithi136.stderr:155090: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-15T20:51:26.184 INFO:tasks.workunit.client.0.smithi136.stderr:155090: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-15T20:51:26.184 INFO:tasks.workunit.client.0.smithi136.stderr:155090: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-15T20:51:26.184 INFO:tasks.workunit.client.0.smithi136.stderr:155090: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-15T20:51:26.184 INFO:tasks.workunit.client.0.smithi136.stderr:155090: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-15T20:51:26.184 INFO:tasks.workunit.client.0.smithi136.stderr:155090: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-15T20:51:26.184 INFO:tasks.workunit.client.0.smithi136.stderr:155090: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-15T20:51:26.185 INFO:tasks.workunit.client.0.smithi136.stderr:155090: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-15T20:51:26.185 INFO:tasks.workunit.client.0.smithi136.stderr:155090: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-15T20:51:26.185 INFO:tasks.workunit.client.0.smithi136.stderr:155090: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-15T20:51:26.185 INFO:tasks.workunit.client.0.smithi136.stderr:155090: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-15T20:51:26.185 INFO:tasks.workunit.client.0.smithi136.stderr:155090: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-15T20:51:26.185 INFO:tasks.workunit.client.0.smithi136.stderr:155090: 1 2024-09-15T20:51:26.185 INFO:tasks.workunit.client.0.smithi136.stderr:155090: 2 2024-09-15T20:51:26.185 INFO:tasks.workunit.client.0.smithi136.stderr:155090: 3' 2024-09-15T20:51:26.185 INFO:tasks.workunit.client.0.smithi136.stderr:155090: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-15T20:51:26.185 INFO:tasks.workunit.client.0.smithi136.stderr:155090: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T20:51:26.185 INFO:tasks.workunit.client.0.smithi136.stderr:155090: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-15T20:51:26.185 INFO:tasks.workunit.client.0.smithi136.stderr:155090: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836543 2024-09-15T20:51:26.185 INFO:tasks.workunit.client.0.smithi136.stderr:155090: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836543 2024-09-15T20:51:26.186 INFO:tasks.workunit.client.0.smithi136.stderr:155090: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836543' 2024-09-15T20:51:26.186 INFO:tasks.workunit.client.0.smithi136.stderr:155090: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T20:51:26.186 INFO:tasks.workunit.client.0.smithi136.stderr:155090: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-15T20:51:26.186 INFO:tasks.workunit.client.0.smithi136.stderr:155090: Error ENXIO: problem getting command descriptions from osd.1 2024-09-15T20:51:26.186 INFO:tasks.workunit.client.0.smithi136.stderr:155090: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq= 2024-09-15T20:51:26.186 INFO:tasks.workunit.client.0.smithi136.stderr:155090: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z '' 2024-09-15T20:51:26.186 INFO:tasks.workunit.client.0.smithi136.stderr:155090: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: continue 2024-09-15T20:51:26.186 INFO:tasks.workunit.client.0.smithi136.stderr:155090: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T20:51:27.642 INFO:tasks.workunit.client.0.smithi136.stderr:155090: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg1 osd.1 86 log_to_monitors true 2024-09-15T20:51:27.643 INFO:tasks.workunit.client.0.smithi136.stderr:155092: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:51:27.643 INFO:tasks.workunit.client.0.smithi136.stderr:155092: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:51:27.643 INFO:tasks.workunit.client.0.smithi136.stderr:155092: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-15T20:51:27.643 INFO:tasks.workunit.client.0.smithi136.stderr:155092: 4 2024-09-15T20:51:27.643 INFO:tasks.workunit.client.0.smithi136.stderr:155092: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:51:27.643 INFO:tasks.workunit.client.0.smithi136.stderr:155092: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T20:51:27.643 INFO:tasks.workunit.client.0.smithi136.stderr:155092: osd.1 up in weight 1 up_from 94 up_thru 84 down_at 87 last_clean_interval [80,86) [v2:127.0.0.1:6826/3585448977,v1:127.0.0.1:6827/3585448977] [v2:127.0.0.1:6828/3585448977,v1:127.0.0.1:6829/3585448977] exists,up ba9ca9c7-78b0-44cd-b086-00a47c922481 2024-09-15T20:51:27.643 INFO:tasks.workunit.client.0.smithi136.stderr:155092: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-15T20:51:27.643 INFO:tasks.workunit.client.0.smithi136.stderr:155092: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-15T20:51:27.643 INFO:tasks.workunit.client.0.smithi136.stderr:155092: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-15T20:51:27.643 INFO:tasks.workunit.client.0.smithi136.stderr:155092: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2024-09-15T20:51:27.643 INFO:tasks.workunit.client.0.smithi136.stderr:155092: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-15T20:51:27.643 INFO:tasks.workunit.client.0.smithi136.stderr:155092: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-15T20:51:27.644 INFO:tasks.workunit.client.0.smithi136.stderr:155092: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-15T20:51:27.644 INFO:tasks.workunit.client.0.smithi136.stderr:155092: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-15T20:51:27.644 INFO:tasks.workunit.client.0.smithi136.stderr:155092: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-15T20:51:27.644 INFO:tasks.workunit.client.0.smithi136.stderr:155092: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-15T20:51:27.644 INFO:tasks.workunit.client.0.smithi136.stderr:155092: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-15T20:51:27.644 INFO:tasks.workunit.client.0.smithi136.stderr:155092: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-15T20:51:27.644 INFO:tasks.workunit.client.0.smithi136.stderr:155092: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-15T20:51:27.644 INFO:tasks.workunit.client.0.smithi136.stderr:155092: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-15T20:51:27.644 INFO:tasks.workunit.client.0.smithi136.stderr:155092: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-15T20:51:27.644 INFO:tasks.workunit.client.0.smithi136.stderr:155092: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-15T20:51:27.644 INFO:tasks.workunit.client.0.smithi136.stderr:155092: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-15T20:51:27.644 INFO:tasks.workunit.client.0.smithi136.stderr:155092: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-15T20:51:27.644 INFO:tasks.workunit.client.0.smithi136.stderr:155092: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-15T20:51:27.645 INFO:tasks.workunit.client.0.smithi136.stderr:155092: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-15T20:51:27.645 INFO:tasks.workunit.client.0.smithi136.stderr:155092: 1 2024-09-15T20:51:27.645 INFO:tasks.workunit.client.0.smithi136.stderr:155092: 2 2024-09-15T20:51:27.645 INFO:tasks.workunit.client.0.smithi136.stderr:155092: 3' 2024-09-15T20:51:27.645 INFO:tasks.workunit.client.0.smithi136.stderr:155092: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-15T20:51:27.645 INFO:tasks.workunit.client.0.smithi136.stderr:155092: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T20:51:27.645 INFO:tasks.workunit.client.0.smithi136.stderr:155092: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-15T20:51:27.645 INFO:tasks.workunit.client.0.smithi136.stderr:155092: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836545 2024-09-15T20:51:27.645 INFO:tasks.workunit.client.0.smithi136.stderr:155092: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836545 2024-09-15T20:51:27.645 INFO:tasks.workunit.client.0.smithi136.stderr:155092: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836545' 2024-09-15T20:51:27.645 INFO:tasks.workunit.client.0.smithi136.stderr:155092: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T20:51:28.717 INFO:tasks.workunit.client.0.smithi136.stderr:155092: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-help_stats: ceph tell osd.2 flush_pg_stats 2024-09-15T20:51:28.717 INFO:tasks.workunit.client.0.smithi136.stderr:155090: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=360777252869 2024-09-15T20:51:28.717 INFO:tasks.workunit.client.0.smithi136.stderr:155090: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 360777252869 2024-09-15T20:51:28.717 INFO:tasks.workunit.client.0.smithi136.stderr:155090: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836543 2-360777252869' 2024-09-15T20:51:28.717 INFO:tasks.workunit.client.0.smithi136.stderr:155090: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T20:51:28.717 INFO:tasks.workunit.client.0.smithi136.stderr:155090: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-15T20:51:28.717 INFO:tasks.workunit.client.0.smithi136.stderr:155090: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=390842023938 2024-09-15T20:51:28.717 INFO:tasks.workunit.client.0.smithi136.stderr:155090: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 390842023938 2024-09-15T20:51:28.717 INFO:tasks.workunit.client.0.smithi136.stderr:155090: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836543 2-360777252869 3-390842023938' 2024-09-15T20:51:28.717 INFO:tasks.workunit.client.0.smithi136.stderr:155090: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T20:51:28.718 INFO:tasks.workunit.client.0.smithi136.stderr:155090: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836543 2024-09-15T20:51:28.718 INFO:tasks.workunit.client.0.smithi136.stderr:155090: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T20:51:28.718 INFO:tasks.workunit.client.0.smithi136.stderr:155090: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-15T20:51:28.718 INFO:tasks.workunit.client.0.smithi136.stderr:155090: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836543 2024-09-15T20:51:28.718 INFO:tasks.workunit.client.0.smithi136.stderr:155090: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T20:51:28.718 INFO:tasks.workunit.client.0.smithi136.stderr:155090: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836543 2024-09-15T20:51:28.718 INFO:tasks.workunit.client.0.smithi136.stderr:155090: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836543' 2024-09-15T20:51:28.718 INFO:tasks.workunit.client.0.smithi136.stderr:155090: waiting osd.0 seq 21474836543 2024-09-15T20:51:28.718 INFO:tasks.workunit.client.0.smithi136.stderr:155090: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T20:51:28.718 INFO:tasks.workunit.client.0.smithi136.stderr:155090: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836542 -lt 21474836543 2024-09-15T20:51:28.718 INFO:tasks.workunit.client.0.smithi136.stderr:155090: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-15T20:51:28.718 INFO:tasks.workunit.client.0.smithi136.stderr:155090: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-15T20:51:28.718 INFO:tasks.workunit.client.0.smithi136.stderr:155090: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T20:51:28.719 INFO:tasks.workunit.client.0.smithi136.stderr:155090: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836544 -lt 21474836543 2024-09-15T20:51:28.719 INFO:tasks.workunit.client.0.smithi136.stderr:155090: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T20:51:28.719 INFO:tasks.workunit.client.0.smithi136.stderr:155090: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-360777252869 2024-09-15T20:51:28.719 INFO:tasks.workunit.client.0.smithi136.stderr:155090: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T20:51:28.719 INFO:tasks.workunit.client.0.smithi136.stderr:155090: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-15T20:51:28.719 INFO:tasks.workunit.client.0.smithi136.stderr:155090: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-360777252869 2024-09-15T20:51:28.719 INFO:tasks.workunit.client.0.smithi136.stderr:155090: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T20:51:28.719 INFO:tasks.workunit.client.0.smithi136.stderr:155090: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=360777252869 2024-09-15T20:51:28.719 INFO:tasks.workunit.client.0.smithi136.stderr:155090: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 360777252869' 2024-09-15T20:51:28.719 INFO:tasks.workunit.client.0.smithi136.stderr:155090: waiting osd.2 seq 360777252869 2024-09-15T20:51:28.719 INFO:tasks.workunit.client.0.smithi136.stderr:155090: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-15T20:51:28.719 INFO:tasks.workunit.client.0.smithi136.stderr:155090: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 360777252870 -lt 360777252869 2024-09-15T20:51:28.719 INFO:tasks.workunit.client.0.smithi136.stderr:155090: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T20:51:29.840 INFO:tasks.workunit.client.0.smithi136.stderr:155090: //home/ubunters.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-15T20:51:29.841 INFO:tasks.workunit.client.0.smithi136.stderr:155092: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=403726925826 2024-09-15T20:51:29.841 INFO:tasks.workunit.client.0.smithi136.stderr:155092: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 403726925826 2024-09-15T20:51:29.841 INFO:tasks.workunit.client.0.smithi136.stderr:155092: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836545 1-403726925826' 2024-09-15T20:51:29.841 INFO:tasks.workunit.client.0.smithi136.stderr:155092: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T20:51:29.841 INFO:tasks.workunit.client.0.smithi136.stderr:155092: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-15T20:51:29.841 INFO:tasks.workunit.client.0.smithi136.stderr:155092: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=360777252871 2024-09-15T20:51:29.841 INFO:tasks.workunit.client.0.smithi136.stderr:155092: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 360777252871 2024-09-15T20:51:29.841 INFO:tasks.workunit.client.0.smithi136.stderr:155092: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836545 1-403726925826 2-360777252871' 2024-09-15T20:51:29.841 INFO:tasks.workunit.client.0.smithi136.stderr:155092: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T20:51:29.841 INFO:tasks.workunit.client.0.smithi136.stderr:155092: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-15T20:51:29.841 INFO:tasks.workunit.client.0.smithi136.stderr:155092: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=390842023939 2024-09-15T20:51:29.841 INFO:tasks.workunit.client.0.smithi136.stderr:155092: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 390842023939 2024-09-15T20:51:29.841 INFO:tasks.workunit.client.0.smithi136.stderr:155092: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836545 1-403726925826 2-360777252871 3-390842023939' 2024-09-15T20:51:29.842 INFO:tasks.workunit.client.0.smithi136.stderr:155092: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T20:51:29.842 INFO:tasks.workunit.client.0.smithi136.stderr:155092: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836545 2024-09-15T20:51:29.842 INFO:tasks.workunit.client.0.smithi136.stderr:155092: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T20:51:29.842 INFO:tasks.workunit.client.0.smithi136.stderr:155092: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-15T20:51:29.842 INFO:tasks.workunit.client.0.smithi136.stderr:155092: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836545 2024-09-15T20:51:29.842 INFO:tasks.workunit.client.0.smithi136.stderr:155092: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T20:51:29.842 INFO:tasks.workunit.client.0.smithi136.stderr:155092: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836545 2024-09-15T20:51:29.842 INFO:tasks.workunit.client.0.smithi136.stderr:155092: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836545' 2024-09-15T20:51:29.842 INFO:tasks.workunit.client.0.smithi136.stderr:155092: waiting osd.0 seq 21474836545 2024-09-15T20:51:29.842 INFO:tasks.workunit.client.0.smithi136.stderr:155092: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T20:51:29.842 INFO:tasks.workunit.client.0.smithi136.stderr:155092: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836544 -lt 21474836545 2024-09-15T20:51:29.842 INFO:tasks.workunit.client.0.smithi136.stderr:155092: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-15T20:51:29.842 INFO:tasks.workunit.client.0.smithi136.stderr:155092: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-15T20:51:29.842 INFO:tasks.workunit.client.0.smithi136.stderr:155092: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T20:51:29.843 INFO:tasks.workunit.client.0.smithi136.stderr:155092: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836545 -lt 21474836545 2024-09-15T20:51:29.843 INFO:tasks.workunit.client.0.smithi136.stderr:155092: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T20:51:29.843 INFO:tasks.workunit.client.0.smithi136.stderr:155092: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-403726925826 2024-09-15T20:51:29.843 INFO:tasks.workunit.client.0.smithi136.stderr:155092: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T20:51:29.843 INFO:tasks.workunit.client.0.smithi136.stderr:155092: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-15T20:51:29.843 INFO:tasks.workunit.client.0.smithi136.stderr:155092: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-403726925826 2024-09-15T20:51:29.843 INFO:tasks.workunit.client.0.smithi136.stderr:155092: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T20:51:30.088 INFO:tasks.workunit.client.0.smithi136.stderr:155092: /home/ubu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 3-390842023938 2024-09-15T20:51:30.088 INFO:tasks.workunit.client.0.smithi136.stderr:155090: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T20:51:30.088 INFO:tasks.workunit.client.0.smithi136.stderr:155090: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=3 2024-09-15T20:51:30.088 INFO:tasks.workunit.client.0.smithi136.stderr:155090: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 3-390842023938 2024-09-15T20:51:30.088 INFO:tasks.workunit.client.0.smithi136.stderr:155090: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T20:51:30.088 INFO:tasks.workunit.client.0.smithi136.stderr:155090: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=390842023938 2024-09-15T20:51:30.088 INFO:tasks.workunit.client.0.smithi136.stderr:155090: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.3 seq 390842023938' 2024-09-15T20:51:30.089 INFO:tasks.workunit.client.0.smithi136.stderr:155090: waiting osd.3 seq 390842023938 2024-09-15T20:51:30.089 INFO:tasks.workunit.client.0.smithi136.stderr:155090: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-15T20:51:30.089 INFO:tasks.workunit.client.0.smithi136.stderr:155090: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 390842023938 -lt 390842023938 2024-09-15T20:51:30.089 INFO:tasks.workunit.client.0.smithi136.stderr:155090: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-15T20:51:30.089 INFO:tasks.workunit.client.0.smithi136.stderr:155090: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T20:51:30.089 INFO:tasks.workunit.client.0.smithi136.stderr:155090: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T20:51:30.089 INFO:tasks.workunit.client.0.smithi136.stderr:155090: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 5 == 0 2024-09-15T20:51:30.089 INFO:tasks.workunit.client.0.smithi136.stderr:155090: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-15T20:51:30.089 INFO:tasks.workunit.client.0.smithi136.stderr:155090: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-15T20:51:30.089 INFO:tasks.workunit.client.0.smithi136.stderr:155090: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-15T20:51:30.089 INFO:tasks.workunit.client.0.smithi136.stderr:155090: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T20:51:30.089 INFO:tasks.workunit.client.0.smithi136.stderr:155090: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T20:51:30.089 INFO:tasks.workunit.client.0.smithi136.stderr:155090: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T20:51:30.089 INFO:tasks.workunit.client.0.smithi136.stderr:155090: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T20:51:30.090 INFO:tasks.workunit.client.0.smithi136.stderr:155090: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=4 2024-09-15T20:51:30.090 INFO:tasks.workunit.client.0.smithi136.stderr:155090: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-15T20:51:30.090 INFO:tasks.workunit.client.0.smithi136.stderr:155090: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T20:51:30.090 INFO:tasks.workunit.client.0.smithi136.stderr:155090: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T20:51:30.090 INFO:tasks.workunit.client.0.smithi136.stderr:155090: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 = 5 2024-09-15T20:51:30.090 INFO:tasks.workunit.client.0.smithi136.stderr:155090: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 4 '!=' -1 2024-09-15T20:51:30.090 INFO:tasks.workunit.client.0.smithi136.stderr:155090: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: loop=0 2024-09-15T20:51:30.090 INFO:tasks.workunit.client.0.smithi136.stderr:155090: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1652: wait_for_clean: num_active_clean=4 2024-09-15T20:51:30.090 INFO:tasks.workunit.client.0.smithi136.stderr:155090: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-15T20:51:30.090 INFO:tasks.workunit.client.0.smithi136.stderr:155090: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 0.1 2024-09-15T20:51:30.090 INFO:tasks.workunit.client.0.smithi136.stderr:155090: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-15T20:51:30.090 INFO:tasks.workunit.client.0.smithi136.stderr:155090: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-15T20:51:30.090 INFO:tasks.workunit.client.0.smithi136.stderr:155090: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-15T20:51:31.129 INFO:tasks.workunit.client.0.smithi136.stderr:155090: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=403726925826 2024-09-15T20:51:31.129 INFO:tasks.workunit.client.0.smithi136.stderr:155092: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 403726925826' 2024-09-15T20:51:31.129 INFO:tasks.workunit.client.0.smithi136.stderr:155092: waiting osd.1 seq 403726925826 2024-09-15T20:51:31.129 INFO:tasks.workunit.client.0.smithi136.stderr:155092: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-15T20:51:31.129 INFO:tasks.workunit.client.0.smithi136.stderr:155092: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 403726925826 -lt 403726925826 2024-09-15T20:51:31.129 INFO:tasks.workunit.client.0.smithi136.stderr:155092: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T20:51:31.130 INFO:tasks.workunit.client.0.smithi136.stderr:155092: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-360777252871 2024-09-15T20:51:31.130 INFO:tasks.workunit.client.0.smithi136.stderr:155092: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T20:51:31.130 INFO:tasks.workunit.client.0.smithi136.stderr:155092: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-15T20:51:31.130 INFO:tasks.workunit.client.0.smithi136.stderr:155092: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-360777252871 2024-09-15T20:51:31.130 INFO:tasks.workunit.client.0.smithi136.stderr:155092: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T20:51:31.130 INFO:tasks.workunit.client.0.smithi136.stderr:155092: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=360777252871 2024-09-15T20:51:31.130 INFO:tasks.workunit.client.0.smithi136.stderr:155092: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 360777252871' 2024-09-15T20:51:31.130 INFO:tasks.workunit.client.0.smithi136.stderr:155092: waiting osd.2 seq 360777252871 2024-09-15T20:51:31.130 INFO:tasks.workunit.client.0.smithi136.stderr:155092: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-15T20:51:31.130 INFO:tasks.workunit.client.0.smithi136.stderr:155092: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 360777252871 -lt 360777252871 2024-09-15T20:51:31.130 INFO:tasks.workunit.client.0.smithi136.stderr:155092: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T20:51:31.130 INFO:tasks.workunit.client.0.smithi136.stderr:155092: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 3-390842023939 2024-09-15T20:51:31.130 INFO:tasks.workunit.client.0.smithi136.stderr:155092: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T20:51:31.130 INFO:tasks.workunit.client.0.smithi136.stderr:155092: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=3 2024-09-15T20:51:31.131 INFO:tasks.workunit.client.0.smithi136.stderr:155092: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 3-390842023939 2024-09-15T20:51:31.131 INFO:tasks.workunit.client.0.smithi136.stderr:155092: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T20:51:31.131 INFO:tasks.workunit.client.0.smithi136.stderr:155092: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=390842023939 2024-09-15T20:51:31.131 INFO:tasks.workunit.client.0.smithi136.stderr:155092: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.3 seq 390842023939' 2024-09-15T20:51:31.131 INFO:tasks.workunit.client.0.smithi136.stderr:155092: waiting osd.3 seq 390842023939 2024-09-15T20:51:31.131 INFO:tasks.workunit.client.0.smithi136.stderr:155092: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-15T20:51:31.131 INFO:tasks.workunit.client.0.smithi136.stderr:155092: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 390842023939 -lt 390842023939 2024-09-15T20:51:31.131 INFO:tasks.workunit.client.0.smithi136.stderr:155092: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-15T20:51:31.131 INFO:tasks.workunit.client.0.smithi136.stderr:155092: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T20:51:31.131 INFO:tasks.workunit.client.0.smithi136.stderr:155092: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T20:51:31.131 INFO:tasks.workunit.client.0.smithi136.stderr:155092: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 5 == 0 2024-09-15T20:51:31.131 INFO:tasks.workunit.client.0.smithi136.stderr:155092: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-15T20:51:31.131 INFO:tasks.workunit.client.0.smithi136.stderr:155092: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-15T20:51:31.131 INFO:tasks.workunit.client.0.smithi136.stderr:155092: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-15T20:51:31.131 INFO:tasks.workunit.client.0.smithi136.stderr:155092: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T20:51:31.227 INFO:tasks.workunit.client.0.smithi136.stderr:155092: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("sph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-15T20:51:31.228 INFO:tasks.workunit.client.0.smithi136.stderr:155090: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T20:51:31.228 INFO:tasks.workunit.client.0.smithi136.stderr:155090: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T20:51:31.228 INFO:tasks.workunit.client.0.smithi136.stderr:155090: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T20:51:31.228 INFO:tasks.workunit.client.0.smithi136.stderr:155090: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T20:51:31.228 INFO:tasks.workunit.client.0.smithi136.stderr:155090: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=4 2024-09-15T20:51:31.228 INFO:tasks.workunit.client.0.smithi136.stderr:155090: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-15T20:51:31.228 INFO:tasks.workunit.client.0.smithi136.stderr:155090: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T20:51:31.228 INFO:tasks.workunit.client.0.smithi136.stderr:155090: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T20:51:31.228 INFO:tasks.workunit.client.0.smithi136.stderr:155090: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 = 5 2024-09-15T20:51:31.228 INFO:tasks.workunit.client.0.smithi136.stderr:155090: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 4 '!=' 4 2024-09-15T20:51:31.228 INFO:tasks.workunit.client.0.smithi136.stderr:155090: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: get_is_making_recovery_progress 2024-09-15T20:51:31.228 INFO:tasks.workunit.client.0.smithi136.stderr:155090: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2024-09-15T20:51:31.228 INFO:tasks.workunit.client.0.smithi136.stderr:155090: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-15T20:51:31.229 INFO:tasks.workunit.client.0.smithi136.stderr:155090: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-15T20:51:31.229 INFO:tasks.workunit.client.0.smithi136.stderr:155090: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-15T20:51:31.229 INFO:tasks.workunit.client.0.smithi136.stderr:155090: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2024-09-15T20:51:31.229 INFO:tasks.workunit.client.0.smithi136.stderr:155090: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-15T20:51:31.229 INFO:tasks.workunit.client.0.smithi136.stderr:155090: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=1 2024-09-15T20:51:31.229 INFO:tasks.workunit.client.0.smithi136.stderr:155090: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test 1 '!=' null 2024-09-15T20:51:31.229 INFO:tasks.workunit.client.0.smithi136.stderr:155090: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: loop=0 2024-09-15T20:51:31.229 INFO:tasks.workunit.client.0.smithi136.stderr:155090: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-15T20:51:31.229 INFO:tasks.workunit.client.0.smithi136.stderr:155090: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 0.1 2024-09-15T20:51:31.229 INFO:tasks.workunit.client.0.smithi136.stderr:155090: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-15T20:51:31.229 INFO:tasks.workunit.client.0.smithi136.stderr:155090: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-15T20:51:31.229 INFO:tasks.workunit.client.0.smithi136.stderr:155090: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-15T20:51:31.229 INFO:tasks.workunit.client.0.smithi136.stderr:155090: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-15T20:51:31.229 INFO:tasks.workunit.client.0.smithi136.stderr:155090: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T20:51:31.230 INFO:tasks.workunit.client.0.smithi136.stderr:155090: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T20:51:31.230 INFO:tasks.workunit.client.0.smithi136.stderr:155090: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T20:51:32.600 INFO:tasks.workunit.client.0.smithi136.stderr:155090: //home/ubuntu/cephtest/clone.client.0/qa/standatale") | not)' 2024-09-15T20:51:32.600 INFO:tasks.workunit.client.0.smithi136.stderr:155092: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T20:51:32.600 INFO:tasks.workunit.client.0.smithi136.stderr:155092: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T20:51:32.600 INFO:tasks.workunit.client.0.smithi136.stderr:155092: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=4 2024-09-15T20:51:32.600 INFO:tasks.workunit.client.0.smithi136.stderr:155092: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-15T20:51:32.601 INFO:tasks.workunit.client.0.smithi136.stderr:155092: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T20:51:32.601 INFO:tasks.workunit.client.0.smithi136.stderr:155092: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T20:51:32.601 INFO:tasks.workunit.client.0.smithi136.stderr:155092: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 = 5 2024-09-15T20:51:32.601 INFO:tasks.workunit.client.0.smithi136.stderr:155092: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 4 '!=' -1 2024-09-15T20:51:32.601 INFO:tasks.workunit.client.0.smithi136.stderr:155092: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: loop=0 2024-09-15T20:51:32.601 INFO:tasks.workunit.client.0.smithi136.stderr:155092: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1652: wait_for_clean: num_active_clean=4 2024-09-15T20:51:32.601 INFO:tasks.workunit.client.0.smithi136.stderr:155092: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-15T20:51:32.601 INFO:tasks.workunit.client.0.smithi136.stderr:155092: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 0.1 2024-09-15T20:51:32.601 INFO:tasks.workunit.client.0.smithi136.stderr:155092: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-15T20:51:32.601 INFO:tasks.workunit.client.0.smithi136.stderr:155092: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-15T20:51:32.601 INFO:tasks.workunit.client.0.smithi136.stderr:155092: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-15T20:51:32.601 INFO:tasks.workunit.client.0.smithi136.stderr:155092: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-15T20:51:32.601 INFO:tasks.workunit.client.0.smithi136.stderr:155092: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T20:51:32.601 INFO:tasks.workunit.client.0.smithi136.stderr:155092: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T20:51:32.602 INFO:tasks.workunit.client.0.smithi136.stderr:155092: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T20:51:32.602 INFO:tasks.workunit.client.0.smithi136.stderr:155092: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T20:51:32.602 INFO:tasks.workunit.client.0.smithi136.stderr:155092: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=5 2024-09-15T20:51:32.602 INFO:tasks.workunit.client.0.smithi136.stderr:155092: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-15T20:51:32.602 INFO:tasks.workunit.client.0.smithi136.stderr:155092: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T20:51:32.602 INFO:tasks.workunit.client.0.smithi136.stderr:155092: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T20:51:32.602 INFO:tasks.workunit.client.0.smithi136.stderr:155092: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 = 5 2024-09-15T20:51:32.602 INFO:tasks.workunit.client.0.smithi136.stderr:155092: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-15T20:51:32.602 INFO:tasks.workunit.client.0.smithi136.stderr:155092: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-15T20:51:32.602 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: run_in_background: return 0 2024-09-15T20:51:32.641 INFO:tasks.workunit.client.0.smithi136.stderr:lone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T20:51:32.641 INFO:tasks.workunit.client.0.smithi136.stderr:155090: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=4 2024-09-15T20:51:32.641 INFO:tasks.workunit.client.0.smithi136.stderr:155090: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-15T20:51:32.642 INFO:tasks.workunit.client.0.smithi136.stderr:155090: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T20:51:32.642 INFO:tasks.workunit.client.0.smithi136.stderr:155090: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T20:51:32.642 INFO:tasks.workunit.client.0.smithi136.stderr:155090: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 = 5 2024-09-15T20:51:32.642 INFO:tasks.workunit.client.0.smithi136.stderr:155090: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 4 '!=' 4 2024-09-15T20:51:32.642 INFO:tasks.workunit.client.0.smithi136.stderr:155090: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: get_is_making_recovery_progress 2024-09-15T20:51:32.642 INFO:tasks.workunit.client.0.smithi136.stderr:155090: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2024-09-15T20:51:32.642 INFO:tasks.workunit.client.0.smithi136.stderr:155090: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-15T20:51:32.642 INFO:tasks.workunit.client.0.smithi136.stderr:155090: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-15T20:51:32.642 INFO:tasks.workunit.client.0.smithi136.stderr:155090: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-15T20:51:32.642 INFO:tasks.workunit.client.0.smithi136.stderr:155090: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2024-09-15T20:51:32.642 INFO:tasks.workunit.client.0.smithi136.stderr:155090: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-15T20:51:32.642 INFO:tasks.workunit.client.0.smithi136.stderr:155090: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=1 2024-09-15T20:51:32.642 INFO:tasks.workunit.client.0.smithi136.stderr:155090: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test 1 '!=' null 2024-09-15T20:51:32.642 INFO:tasks.workunit.client.0.smithi136.stderr:155090: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: loop=0 2024-09-15T20:51:32.643 INFO:tasks.workunit.client.0.smithi136.stderr:155090: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-15T20:51:32.643 INFO:tasks.workunit.client.0.smithi136.stderr:155090: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 0.1 2024-09-15T20:51:32.643 INFO:tasks.workunit.client.0.smithi136.stderr:155090: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-15T20:51:32.643 INFO:tasks.workunit.client.0.smithi136.stderr:155090: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-15T20:51:32.643 INFO:tasks.workunit.client.0.smithi136.stderr:155090: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-15T20:51:32.643 INFO:tasks.workunit.client.0.smithi136.stderr:155090: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-15T20:51:32.643 INFO:tasks.workunit.client.0.smithi136.stderr:155090: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T20:51:32.643 INFO:tasks.workunit.client.0.smithi136.stderr:155090: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T20:51:32.643 INFO:tasks.workunit.client.0.smithi136.stderr:155090: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T20:51:32.643 INFO:tasks.workunit.client.0.smithi136.stderr:155090: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T20:51:32.643 INFO:tasks.workunit.client.0.smithi136.stderr:155090: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=5 2024-09-15T20:51:32.643 INFO:tasks.workunit.client.0.smithi136.stderr:155090: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-15T20:51:32.643 INFO:tasks.workunit.client.0.smithi136.stderr:155090: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T20:51:33.005 INFO:tasks.workunit.client.0.smithi136.stderr:155090: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T20:51:33.005 INFO:tasks.workunit.client.0.smithi136.stderr:155090: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 = 5 2024-09-15T20:51:33.005 INFO:tasks.workunit.client.0.smithi136.stderr:155090: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-15T20:51:33.005 INFO:tasks.workunit.client.0.smithi136.stderr:155090: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-15T20:51:33.005 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: run_in_background: return 0 2024-09-15T20:51:33.005 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: wait_background: for pid in $pids 2024-09-15T20:51:33.005 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2175: wait_background: wait 155089 2024-09-15T20:51:33.005 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: wait_background: eval 'pids='\'''\''' 2024-09-15T20:51:33.006 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: wait_background: pids= 2024-09-15T20:51:33.006 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: wait_background: return 0 2024-09-15T20:51:33.006 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:197: corrupt_and_repair_two: return_code=0 2024-09-15T20:51:33.006 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:198: corrupt_and_repair_two: '[' 0 -ne 0 ']' 2024-09-15T20:51:33.006 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:203: corrupt_and_repair_two: get_pg ecpool SOMETHING 2024-09-15T20:51:33.006 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1084: get_pg: local poolname=ecpool 2024-09-15T20:51:33.006 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1085: get_pg: local objectname=SOMETHING 2024-09-15T20:51:33.006 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1087: get_pg: ceph --format json osd map ecpool SOMETHING 2024-09-15T20:51:33.006 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1087: get_pg: jq -r .pgid 2024-09-15T20:51:33.283 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:203: corrupt_and_repair_two: local pg=2.0 2024-09-15T20:51:33.283 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:204: corrupt_and_repair_two: repair 2.0 2024-09-15T20:51:33.283 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1846: repair: local pgid=2.0 2024-09-15T20:51:33.283 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1847: repair: get_last_scrub_stamp 2.0 2024-09-15T20:51:33.283 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2024-09-15T20:51:33.283 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-15T20:51:33.283 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-15T20:51:33.283 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-09-15T20:51:33.555 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1847: repair: local last_scrub=2024-09-15T20:50:58.922318+0000 2024-09-15T20:51:33.555 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1848: repair: ceph pg repair 2.0 2024-09-15T20:51:33.819 INFO:tasks.workunit.client.0.smithi136.stderr:instructing pg 2.0s0 on osd.3 to repair 2024-09-15T20:51:33.831 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1849: repair: wait_for_scrub 2.0 2024-09-15T20:50:58.922318+0000 2024-09-15T20:51:33.831 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: local pgid=2.0 2024-09-15T20:51:33.831 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2018: wait_for_scrub: local last_scrub=2024-09-15T20:50:58.922318+0000 2024-09-15T20:51:33.831 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2019: wait_for_scrub: local sname=last_scrub_stamp 2024-09-15T20:51:33.831 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i=0 )) 2024-09-15T20:51:33.832 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-15T20:51:33.832 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-09-15T20:51:33.832 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2024-09-15T20:51:33.832 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-15T20:51:33.832 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-15T20:51:33.832 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-09-15T20:51:34.102 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-15T20:50:58.922318+0000 '>' 2024-09-15T20:50:58.922318+0000 2024-09-15T20:51:34.102 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-15T20:51:35.103 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-15T20:51:35.103 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-15T20:51:35.104 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-09-15T20:51:35.104 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2024-09-15T20:51:35.104 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-15T20:51:35.104 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-15T20:51:35.104 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-09-15T20:51:35.388 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-15T20:50:58.922318+0000 '>' 2024-09-15T20:50:58.922318+0000 2024-09-15T20:51:35.388 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-15T20:51:36.389 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-15T20:51:36.389 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-15T20:51:36.390 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-09-15T20:51:36.390 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2024-09-15T20:51:36.390 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-15T20:51:36.390 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-15T20:51:36.390 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-09-15T20:51:36.672 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-15T20:51:34.358203+0000 '>' 2024-09-15T20:50:58.922318+0000 2024-09-15T20:51:36.672 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2023: wait_for_scrub: return 0 2024-09-15T20:51:36.672 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1850: repair: sleep 2 2024-09-15T20:51:38.674 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:208: corrupt_and_repair_two: pids= 2024-09-15T20:51:38.675 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:209: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 3 SOMETHING list-attrs 2024-09-15T20:51:38.675 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2135: run_in_background: local pid_variable=pids 2024-09-15T20:51:38.675 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2136: run_in_background: shift 2024-09-15T20:51:38.675 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2140: run_in_background: eval 'pids+=" 158149"' 2024-09-15T20:51:38.675 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2140: run_in_background: pids+=' 158149' 2024-09-15T20:51:38.675 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:210: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 1 SOMETHING list-attrs 2024-09-15T20:51:38.675 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2135: run_in_background: local pid_variable=pids 2024-09-15T20:51:38.675 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2136: run_in_background: shift 2024-09-15T20:51:38.675 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: run_in_background: objectstore_tool td/osd-scrub-repair 3 SOMETHING list-attrs 2024-09-15T20:51:38.675 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2140: run_in_background: eval 'pids+=" 158151"' 2024-09-15T20:51:38.675 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2140: run_in_background: pids+=' 158151' 2024-09-15T20:51:38.675 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:211: corrupt_and_repair_two: wait_background pids 2024-09-15T20:51:38.676 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: run_in_background: sed 's/^/158152: /' 2024-09-15T20:51:38.676 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: wait_background: pids=' 158149 158151' 2024-09-15T20:51:38.676 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: wait_background: return_code=0 2024-09-15T20:51:38.676 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: wait_background: for pid in $pids 2024-09-15T20:51:38.676 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2175: wait_background: wait 158149 2024-09-15T20:51:38.676 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: run_in_background: objectstore_tool td/osd-scrub-repair 1 SOMETHING list-attrs 2024-09-15T20:51:38.676 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: run_in_background: sed 's/^/158154: /' 2024-09-15T20:51:39.786 INFO:tasks.workunit.client.0.smithi136.stderr:158154: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2024-09-15T20:51:39.786 INFO:tasks.workunit.client.0.smithi136.stderr:158154: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2024-09-15T20:51:39.787 INFO:tasks.workunit.client.0.smithi136.stderr:158154: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=1 2024-09-15T20:51:39.787 INFO:tasks.workunit.client.0.smithi136.stderr:158154: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2024-09-15T20:51:39.787 INFO:tasks.workunit.client.0.smithi136.stderr:158154: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 SOMETHING list-attrs 2024-09-15T20:51:39.787 INFO:tasks.workunit.client.0.smithi136.stderr:158154: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-09-15T20:51:39.787 INFO:tasks.workunit.client.0.smithi136.stderr:158154: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2024-09-15T20:51:39.788 INFO:tasks.workunit.client.0.smithi136.stderr:158154: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=1 2024-09-15T20:51:39.788 INFO:tasks.workunit.client.0.smithi136.stderr:158154: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2024-09-15T20:51:39.788 INFO:tasks.workunit.client.0.smithi136.stderr:158154: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-09-15T20:51:39.788 INFO:tasks.workunit.client.0.smithi136.stderr:158154: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-15T20:51:39.788 INFO:tasks.workunit.client.0.smithi136.stderr:158154: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-15T20:51:39.788 INFO:tasks.workunit.client.0.smithi136.stderr:158154: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-15T20:51:39.789 INFO:tasks.workunit.client.0.smithi136.stderr:158154: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-15T20:51:39.789 INFO:tasks.workunit.client.0.smithi136.stderr:158154: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-15T20:51:39.789 INFO:tasks.workunit.client.0.smithi136.stderr:158154: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-15T20:51:39.789 INFO:tasks.workunit.client.0.smithi136.stderr:158154: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 SOMETHING list-attrs 2024-09-15T20:51:39.789 INFO:tasks.workunit.client.0.smithi136.stderr:158154: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-15T20:51:39.789 INFO:tasks.workunit.client.0.smithi136.stderr:158154: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-15T20:51:39.790 INFO:tasks.workunit.client.0.smithi136.stderr:158154: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=1 2024-09-15T20:51:39.790 INFO:tasks.workunit.client.0.smithi136.stderr:158154: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-15T20:51:39.790 INFO:tasks.workunit.client.0.smithi136.stderr:158154: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-09-15T20:51:39.790 INFO:tasks.workunit.client.0.smithi136.stderr:158154: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 SOMETHING list-attrs 2024-09-15T20:51:39.790 INFO:tasks.workunit.client.0.smithi136.stderr:158154: _ 2024-09-15T20:51:39.790 INFO:tasks.workunit.client.0.smithi136.stderr:158154: hinfo_key 2024-09-15T20:51:39.791 INFO:tasks.workunit.client.0.smithi136.stderr:158154: snapset 2024-09-15T20:51:39.791 INFO:tasks.workunit.client.0.smithi136.stderr:158154: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2024-09-15T20:51:39.791 INFO:tasks.workunit.client.0.smithi136.stderr:158154: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2024-09-15T20:51:39.791 INFO:tasks.workunit.client.0.smithi136.stderr:158154: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-15T20:51:39.791 INFO:tasks.workunit.client.0.smithi136.stderr:158154: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=1 2024-09-15T20:51:39.791 INFO:tasks.workunit.client.0.smithi136.stderr:158154: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-15T20:51:39.792 INFO:tasks.workunit.client.0.smithi136.stderr:158154: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-09-15T20:51:39.792 INFO:tasks.workunit.client.0.smithi136.stderr:158154: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=ce4892c7-6c6d-4b4f-8e6f-07a3e11133e5 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-15T20:51:39.792 INFO:tasks.workunit.client.0.smithi136.stderr:158154: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T20:51:39.793 INFO:tasks.workunit.client.0.smithi136.stderr:158154: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T20:51:39.793 INFO:tasks.workunit.client.0.smithi136.stderr:158154: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T20:51:39.793 INFO:tasks.workunit.client.0.smithi136.stderr:158154: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-09-15T20:51:39.793 INFO:tasks.workunit.client.0.smithi136.stderr:158154: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-09-15T20:51:39.793 INFO:tasks.workunit.client.0.smithi136.stderr:158154: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-15T20:51:39.794 INFO:tasks.workunit.client.0.smithi136.stderr:158154: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-15T20:51:39.794 INFO:tasks.workunit.client.0.smithi136.stderr:158154: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-15T20:51:39.794 INFO:tasks.workunit.client.0.smithi136.stderr:158154: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-15T20:51:39.794 INFO:tasks.workunit.client.0.smithi136.stderr:158154: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-15T20:51:39.794 INFO:tasks.workunit.client.0.smithi136.stderr:158154: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-15T20:51:39.794 INFO:tasks.workunit.client.0.smithi136.stderr:158154: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-15T20:51:39.795 INFO:tasks.workunit.client.0.smithi136.stderr:158154: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T20:51:39.795 INFO:tasks.workunit.client.0.smithi136.stderr:158154: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52602 2024-09-15T20:51:39.795 INFO:tasks.workunit.client.0.smithi136.stderr:158154: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.52602/$cluster-$name.asok' 2024-09-15T20:51:39.795 INFO:tasks.workunit.client.0.smithi136.stderr:158154: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.52602/$cluster-$name.asok' 2024-09-15T20:51:39.795 INFO:tasks.workunit.client.0.smithi136.stderr:158154: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-15T20:51:39.795 INFO:tasks.workunit.client.0.smithi136.stderr:158154: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-15T20:51:39.795 INFO:tasks.workunit.client.0.smithi136.stderr:158154: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-15T20:51:39.796 INFO:tasks.workunit.client.0.smithi136.stderr:158154: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T20:51:39.796 INFO:tasks.workunit.client.0.smithi136.stderr:158154: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T20:51:39.796 INFO:tasks.workunit.client.0.smithi136.stderr:158154: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T20:51:39.796 INFO:tasks.workunit.client.0.smithi136.stderr:158154: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T20:51:39.796 INFO:tasks.workunit.client.0.smithi136.stderr:158154: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-15T20:51:39.796 INFO:tasks.workunit.client.0.smithi136.stderr:158154: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-15T20:51:39.796 INFO:tasks.workunit.client.0.smithi136.stderr:158154: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-09-15T20:51:39.796 INFO:tasks.workunit.client.0.smithi136.stderr:158154: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.1 2024-09-15T20:51:39.796 INFO:tasks.workunit.client.0.smithi136.stderr:158154: start osd.1 2024-09-15T20:51:39.817 INFO:tasks.workunit.client.0.smithi136.stderr:158154: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 1 --fsid=ce4892c7-6c6d-4b4f-8e6f-07a3e11133e5 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.52602/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_reco158152: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2024-09-15T20:51:39.817 INFO:tasks.workunit.client.0.smithi136.stderr:158152: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2024-09-15T20:51:39.817 INFO:tasks.workunit.client.0.smithi136.stderr:158152: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=3 2024-09-15T20:51:39.817 INFO:tasks.workunit.client.0.smithi136.stderr:158152: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2024-09-15T20:51:39.817 INFO:tasks.workunit.client.0.smithi136.stderr:158152: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 3 SOMETHING list-attrs 2024-09-15T20:51:39.817 INFO:tasks.workunit.client.0.smithi136.stderr:158152: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-09-15T20:51:39.818 INFO:tasks.workunit.client.0.smithi136.stderr:158152: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2024-09-15T20:51:39.818 INFO:tasks.workunit.client.0.smithi136.stderr:158152: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=3 2024-09-15T20:51:39.818 INFO:tasks.workunit.client.0.smithi136.stderr:158152: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2024-09-15T20:51:39.818 INFO:tasks.workunit.client.0.smithi136.stderr:158152: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.3 2024-09-15T20:51:39.818 INFO:tasks.workunit.client.0.smithi136.stderr:158152: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-15T20:51:39.818 INFO:tasks.workunit.client.0.smithi136.stderr:158152: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-15T20:51:39.818 INFO:tasks.workunit.client.0.smithi136.stderr:158152: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-15T20:51:39.818 INFO:tasks.workunit.client.0.smithi136.stderr:158152: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-15T20:51:39.818 INFO:tasks.workunit.client.0.smithi136.stderr:158152: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-15T20:51:39.818 INFO:tasks.workunit.client.0.smithi136.stderr:158152: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-15T20:51:39.819 INFO:tasks.workunit.client.0.smithi136.stderr:158152: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 3 SOMETHING list-attrs 2024-09-15T20:51:39.819 INFO:tasks.workunit.client.0.smithi136.stderr:158152: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-15T20:51:39.819 INFO:tasks.workunit.client.0.smithi136.stderr:158152: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-15T20:51:39.819 INFO:tasks.workunit.client.0.smithi136.stderr:158152: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=3 2024-09-15T20:51:39.819 INFO:tasks.workunit.client.0.smithi136.stderr:158152: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-15T20:51:39.819 INFO:tasks.workunit.client.0.smithi136.stderr:158152: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/3 2024-09-15T20:51:39.819 INFO:tasks.workunit.client.0.smithi136.stderr:158152: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/3 SOMETHING list-attrs 2024-09-15T20:51:39.819 INFO:tasks.workunit.client.0.smithi136.stderr:158152: Error getting attr on : 2.0s0_head,0#-4:00000000:::scrub_2.0s0:head#, (61) No data available 2024-09-15T20:51:39.819 INFO:tasks.workunit.client.0.smithi136.stderr:158152: _ 2024-09-15T20:51:39.820 INFO:tasks.workunit.client.0.smithi136.stderr:158152: hinfo_key 2024-09-15T20:51:39.820 INFO:tasks.workunit.client.0.smithi136.stderr:158152: snapset 2024-09-15T20:51:39.820 INFO:tasks.workunit.client.0.smithi136.stderr:158152: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 3 2024-09-15T20:51:39.820 INFO:tasks.workunit.client.0.smithi136.stderr:158152: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2024-09-15T20:51:39.820 INFO:tasks.workunit.client.0.smithi136.stderr:158152: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-15T20:51:39.820 INFO:tasks.workunit.client.0.smithi136.stderr:158152: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=3 2024-09-15T20:51:39.820 INFO:tasks.workunit.client.0.smithi136.stderr:158152: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-15T20:51:39.820 INFO:tasks.workunit.client.0.smithi136.stderr:158152: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/3 2024-09-15T20:51:39.821 INFO:tasks.workunit.client.0.smithi136.stderr:158152: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=ce4892c7-6c6d-4b4f-8e6f-07a3e11133e5 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-15T20:51:39.821 INFO:tasks.workunit.client.0.smithi136.stderr:158152: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T20:51:39.821 INFO:tasks.workunit.client.0.smithi136.stderr:158152: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T20:51:39.821 INFO:tasks.workunit.client.0.smithi136.stderr:158152: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T20:51:39.821 INFO:tasks.workunit.client.0.smithi136.stderr:158152: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/3' 2024-09-15T20:51:39.822 INFO:tasks.workunit.client.0.smithi136.stderr:158152: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/3/journal' 2024-09-15T20:51:39.822 INFO:tasks.workunit.client.0.smithi136.stderr:158152: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-15T20:51:39.822 INFO:tasks.workunit.client.0.smithi136.stderr:158152: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-15T20:51:39.822 INFO:tasks.workunit.client.0.smithi136.stderr:158152: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-15T20:51:39.822 INFO:tasks.workunit.client.0.smithi136.stderr:158152: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-15T20:51:39.822 INFO:tasks.workunit.client.0.smithi136.stderr:158152: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-15T20:51:39.822 INFO:tasks.workunit.client.0.smithi136.stderr:158152: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-15T20:51:39.822 INFO:tasks.workunit.client.0.smithi136.stderr:158152: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-15T20:51:39.823 INFO:tasks.workunit.client.0.smithi136.stderr:158152: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T20:51:39.823 INFO:tasks.workunit.client.0.smithi136.stderr:158152: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52602 2024-09-15T20:51:39.823 INFO:tasks.workunit.client.0.smithi136.stderr:158152: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.52602/$cluster-$name.asok' 2024-09-15T20:51:39.823 INFO:tasks.workunit.client.0.smithi136.stderr:158152: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.52602/$cluster-$name.asok' 2024-09-15T20:51:39.823 INFO:tasks.workunit.client.0.smithi136.stderr:158152: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-15T20:51:39.823 INFO:tasks.workunit.client.0.smithi136.stderr:158152: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-15T20:51:39.823 INFO:tasks.workunit.client.0.smithi136.stderr:158152: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-15T20:51:39.823 INFO:tasks.workunit.client.0.smithi136.stderr:158152: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T20:51:39.823 INFO:tasks.workunit.client.0.smithi136.stderr:158152: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T20:51:39.824 INFO:tasks.workunit.client.0.smithi136.stderr:158152: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T20:51:39.824 INFO:tasks.workunit.client.0.smithi136.stderr:158152: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T20:51:39.824 INFO:tasks.workunit.client.0.smithi136.stderr:158152: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-15T20:51:39.824 INFO:tasks.workunit.client.0.smithi136.stderr:158152: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-15T20:51:39.824 INFO:tasks.workunit.client.0.smithi136.stderr:158152: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/3 2024-09-15T20:51:39.824 INFO:tasks.workunit.client.0.smithi136.stderr:158152: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.3 2024-09-15T20:51:39.824 INFO:tasks.workunit.client.0.smithi136.stderr:158152: start osd.3 2024-09-15T20:51:44.414 INFO:tasks.workunit.client.0.smithi136.stderr:158152: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 3 --fsid=ce4892c7-6c6d-4b4f-8e6f-07a3e11133e5 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/3 --osd-journal=td/osd-scrub-repair/3/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.52602/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T20:51:44.415 INFO:tasks.workunit.client.0.smithi136.stderr:158152: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/3/whoami 2024-09-15T20:51:44.415 INFO:tasks.workunit.client.0.smithi136.stderr:158152: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 3 = 3 ']' 2024-09-15T20:51:44.415 INFO:tasks.workunit.client.0.smithi136.stderr:158152: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-15T20:51:44.415 INFO:tasks.workunit.client.0.smithi136.stderr:158152: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-15T20:51:44.415 INFO:tasks.workunit.client.0.smithi136.stderr:158152: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-15T20:51:44.415 INFO:tasks.workunit.client.0.smithi136.stderr:158152: 2024-09-15T20:51:39.863+0000 7f19171b2640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T20:51:44.415 INFO:tasks.workunit.client.0.smithi136.stderr:158152: 2024-09-15T20:51:39.916+0000 7f19171b2640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T20:51:44.415 INFO:tasks.workunit.client.0.smithi136.stderr:158152: 2024-09-15T20:51:39.917+0000 7f19171b2640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T20:51:44.416 INFO:tasks.workunit.client.0.smithi136.stderr:158152: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 3 2024-09-15T20:51:44.416 INFO:tasks.workunit.client.0.smithi136.stderr:158152: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-15T20:51:44.416 INFO:tasks.workunit.client.0.smithi136.stderr:158152: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=3 2024-09-15T20:51:44.416 INFO:tasks.workunit.client.0.smithi136.stderr:158152: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-15T20:51:44.416 INFO:tasks.workunit.client.0.smithi136.stderr:158152: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-15T20:51:44.416 INFO:tasks.workunit.client.0.smithi136.stderr:158152: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:51:44.416 INFO:tasks.workunit.client.0.smithi136.stderr:158152: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-15T20:51:44.416 INFO:tasks.workunit.client.0.smithi136.stderr:158152: 0 2024-09-15T20:51:44.416 INFO:tasks.workunit.client.0.smithi136.stderr:158152: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:51:44.416 INFO:tasks.workunit.client.0.smithi136.stderr:158152: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-15T20:51:44.416 INFO:tasks.workunit.client.0.smithi136.stderr:158152: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:51:44.416 INFO:tasks.workunit.client.0.smithi136.stderr:158152: 2024-09-15T20:51:41.287+0000 7f19171b2640 -1 Falling back to public interface 2024-09-15T20:51:44.416 INFO:tasks.workunit.client.0.smithi136.stderr:158152: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:51:44.416 INFO:tasks.workunit.client.0.smithi136.stderr:158152: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:51:44.417 INFO:tasks.workunit.client.0.smithi136.stderr:158152: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-15T20:51:44.417 INFO:tasks.workunit.client.0.smithi136.stderr:158152: 1 2024-09-15T20:51:44.417 INFO:tasks.workunit.client.0.smithi136.stderr:158152: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:51:44.417 INFO:tasks.workunit.client.0.smithi136.stderr:158152: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-15T20:51:44.417 INFO:tasks.workunit.client.0.smithi136.stderr:158152: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:51:44.417 INFO:tasks.workunit.client.0.smithi136.stderr:158152: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:51:44.417 INFO:tasks.workunit.client.0.smithi136.stderr:158152: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:51:44.417 INFO:tasks.workunit.client.0.smithi136.stderr:158152: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-15T20:51:44.417 INFO:tasks.workunit.client.0.smithi136.stderr:158152: 2 2024-09-15T20:51:44.417 INFO:tasks.workunit.client.0.smithi136.stderr:158152: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:51:44.417 INFO:tasks.workunit.client.0.smithi136.stderr:158152: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-15T20:51:44.417 INFO:tasks.workunit.client.0.smithi136.stderr:158152: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:51:44.417 INFO:tasks.workunit.client.0.smithi136.stderr:158152: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:51:44.418 INFO:tasks.workunit.client.0.smithi136.stderr:158152: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:51:44.418 INFO:tasks.workunit.client.0.smithi136.stderr:158152: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-15T20:51:44.418 INFO:tasks.workunit.client.0.smithi136.stderr:158152: 3 2024-09-15T20:51:44.418 INFO:tasks.workunit.client.0.smithi136.stderr:158152: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:51:44.418 INFO:tasks.workunit.client.0.smithi136.stderr:158152: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-15T20:51:44.644 INFO:tasks.workunit.client.0.smithi136.stderr:158152: /home/ubuntu/cephtest/clone.client.very_ops 2024-09-15T20:51:44.645 INFO:tasks.workunit.client.0.smithi136.stderr:158154: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-09-15T20:51:44.645 INFO:tasks.workunit.client.0.smithi136.stderr:158154: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 1 = 1 ']' 2024-09-15T20:51:44.645 INFO:tasks.workunit.client.0.smithi136.stderr:158154: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-15T20:51:44.645 INFO:tasks.workunit.client.0.smithi136.stderr:158154: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-15T20:51:44.645 INFO:tasks.workunit.client.0.smithi136.stderr:158154: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-15T20:51:44.645 INFO:tasks.workunit.client.0.smithi136.stderr:158154: 2024-09-15T20:51:39.817+0000 7f6e15219640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T20:51:44.645 INFO:tasks.workunit.client.0.smithi136.stderr:158154: 2024-09-15T20:51:39.820+0000 7f6e15219640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T20:51:44.645 INFO:tasks.workunit.client.0.smithi136.stderr:158154: 2024-09-15T20:51:39.822+0000 7f6e15219640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T20:51:44.645 INFO:tasks.workunit.client.0.smithi136.stderr:158154: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 1 2024-09-15T20:51:44.645 INFO:tasks.workunit.client.0.smithi136.stderr:158154: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-15T20:51:44.645 INFO:tasks.workunit.client.0.smithi136.stderr:158154: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2024-09-15T20:51:44.645 INFO:tasks.workunit.client.0.smithi136.stderr:158154: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-15T20:51:44.645 INFO:tasks.workunit.client.0.smithi136.stderr:158154: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-15T20:51:44.646 INFO:tasks.workunit.client.0.smithi136.stderr:158154: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:51:44.646 INFO:tasks.workunit.client.0.smithi136.stderr:158154: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-15T20:51:44.646 INFO:tasks.workunit.client.0.smithi136.stderr:158154: 0 2024-09-15T20:51:44.646 INFO:tasks.workunit.client.0.smithi136.stderr:158154: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:51:44.646 INFO:tasks.workunit.client.0.smithi136.stderr:158154: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T20:51:44.646 INFO:tasks.workunit.client.0.smithi136.stderr:158154: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:51:44.646 INFO:tasks.workunit.client.0.smithi136.stderr:158154: 2024-09-15T20:51:41.162+0000 7f6e15219640 -1 Falling back to public interface 2024-09-15T20:51:44.646 INFO:tasks.workunit.client.0.smithi136.stderr:158154: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:51:44.646 INFO:tasks.workunit.client.0.smithi136.stderr:158154: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:51:44.646 INFO:tasks.workunit.client.0.smithi136.stderr:158154: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-15T20:51:44.646 INFO:tasks.workunit.client.0.smithi136.stderr:158154: 1 2024-09-15T20:51:44.646 INFO:tasks.workunit.client.0.smithi136.stderr:158154: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:51:44.646 INFO:tasks.workunit.client.0.smithi136.stderr:158154: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T20:51:44.646 INFO:tasks.workunit.client.0.smithi136.stderr:158154: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:51:44.647 INFO:tasks.workunit.client.0.smithi136.stderr:158154: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:51:44.647 INFO:tasks.workunit.client.0.smithi136.stderr:158154: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:51:44.647 INFO:tasks.workunit.client.0.smithi136.stderr:158154: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-15T20:51:44.647 INFO:tasks.workunit.client.0.smithi136.stderr:158154: 2 2024-09-15T20:51:44.647 INFO:tasks.workunit.client.0.smithi136.stderr:158154: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:51:44.647 INFO:tasks.workunit.client.0.smithi136.stderr:158154: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T20:51:44.647 INFO:tasks.workunit.client.0.smithi136.stderr:158154: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:51:44.647 INFO:tasks.workunit.client.0.smithi136.stderr:158154: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:51:44.647 INFO:tasks.workunit.client.0.smithi136.stderr:158154: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:51:44.647 INFO:tasks.workunit.client.0.smithi136.stderr:158154: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-15T20:51:44.647 INFO:tasks.workunit.client.0.smithi136.stderr:158154: 3 2024-09-15T20:51:44.647 INFO:tasks.workunit.client.0.smithi136.stderr:158154: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:51:44.647 INFO:tasks.workunit.client.0.smithi136.stderr:158154: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T20:51:44.647 INFO:tasks.workunit.client.0.smithi136.stderr:158154: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:51:46.417 INFO:tasks.workunit.client.0.smithi136.stderr:158154: 2024-09-15T20:51:44.641+0000 7f6e15219640 -1 osd.1 96 log_to_monitors true 2024-09-15T20:51:46.417 INFO:tasks.workunit.client.0.smithi136.stderr:158154: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:51:46.417 INFO:tasks.workunit.client.0.smithi136.stderr:158154: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:51:46.417 INFO:tasks.workunit.client.0.smithi136.stderr:158154: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-15T20:51:46.417 INFO:tasks.workunit.client.0.smithi136.stderr:158154: 4 2024-09-15T20:51:46.417 INFO:tasks.workunit.client.0.smithi136.stderr:158154: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:51:46.417 INFO:tasks.workunit.client.0.smithi136.stderr:158154: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T20:51:46.418 INFO:tasks.workunit.client.0.smithi136.stderr:158154: osd.1 up in weight 1 up_from 102 up_thru 94 down_at 97 last_clean_interval [94,96) [v2:127.0.0.1:6810/554664950,v1:127.0.0.1:6811/554664950] [v2:127.0.0.1:6812/554664950,v1:127.0.0.1:6813/554664950] exists,up ba9ca9c7-78b0-44cd-b086-00a47c922481 2024-09-15T20:51:46.418 INFO:tasks.workunit.client.0.smithi136.stderr:158154: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-15T20:51:46.418 INFO:tasks.workunit.client.0.smithi136.stderr:158154: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-15T20:51:46.418 INFO:tasks.workunit.client.0.smithi136.stderr:158154: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-15T20:51:46.418 INFO:tasks.workunit.client.0.smithi136.stderr:158154: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2024-09-15T20:51:46.418 INFO:tasks.workunit.client.0.smithi136.stderr:158154: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-15T20:51:46.418 INFO:tasks.workunit.client.0.smithi136.stderr:158154: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-15T20:51:46.418 INFO:tasks.workunit.client.0.smithi136.stderr:158154: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-15T20:51:46.418 INFO:tasks.workunit.client.0.smithi136.stderr:158154: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-15T20:51:46.418 INFO:tasks.workunit.client.0.smithi136.stderr:158154: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-15T20:51:46.418 INFO:tasks.workunit.client.0.smithi136.stderr:158154: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-15T20:51:46.418 INFO:tasks.workunit.client.0.smithi136.stderr:158154: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-15T20:51:46.418 INFO:tasks.workunit.client.0.smithi136.stderr:158154: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-15T20:51:46.418 INFO:tasks.workunit.client.0.smithi136.stderr:158154: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-15T20:51:46.419 INFO:tasks.workunit.client.0.smithi136.stderr:158154: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-15T20:51:46.419 INFO:tasks.workunit.client.0.smithi136.stderr:158154: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-15T20:51:46.419 INFO:tasks.workunit.client.0.smithi136.stderr:158154: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-15T20:51:46.419 INFO:tasks.workunit.client.0.smithi136.stderr:158154: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-15T20:51:46.419 INFO:tasks.workunit.client.0.smithi136.stderr:158154: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-15T20:51:46.419 INFO:tasks.workunit.client.0.smithi136.stderr:158154: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-15T20:51:46.419 INFO:tasks.workunit.client.0.smithi136.stderr:158154: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-15T20:51:46.419 INFO:tasks.workunit.client.0.smithi136.stderr:158154: 1 2024-09-15T20:51:46.419 INFO:tasks.workunit.client.0.smithi136.stderr:158154: 2 2024-09-15T20:51:46.419 INFO:tasks.workunit.client.0.smithi136.stderr:158154: 3' 2024-09-15T20:51:46.419 INFO:tasks.workunit.client.0.smithi136.stderr:158154: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-15T20:51:46.419 INFO:tasks.workunit.client.0.smithi136.stderr:158154: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T20:51:46.419 INFO:tasks.workunit.client.0.smithi136.stderr:158154: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-15T20:51:46.419 INFO:tasks.workunit.client.0.smithi136.stderr:158154: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836550 2024-09-15T20:51:46.420 INFO:tasks.workunit.client.0.smithi136.stderr:158154: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836550 2024-09-15T20:51:46.420 INFO:tasks.workunit.client.0.smithi136.stderr:158154: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836550' 2024-09-15T20:51:46.420 INFO:tasks.workunit.client.0.smithi136.stderr:158154: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T20:51:46.676 INFO:tasks.workunit.client.0.smithi136.stderr:158154: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ce0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:51:46.676 INFO:tasks.workunit.client.0.smithi136.stderr:158152: 2024-09-15T20:51:44.822+0000 7f19171b2640 -1 osd.3 96 log_to_monitors true 2024-09-15T20:51:46.676 INFO:tasks.workunit.client.0.smithi136.stderr:158152: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:51:46.676 INFO:tasks.workunit.client.0.smithi136.stderr:158152: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:51:46.676 INFO:tasks.workunit.client.0.smithi136.stderr:158152: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-15T20:51:46.676 INFO:tasks.workunit.client.0.smithi136.stderr:158152: 4 2024-09-15T20:51:46.676 INFO:tasks.workunit.client.0.smithi136.stderr:158152: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:51:46.677 INFO:tasks.workunit.client.0.smithi136.stderr:158152: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-15T20:51:46.677 INFO:tasks.workunit.client.0.smithi136.stderr:158152: osd.3 up in weight 1 up_from 102 up_thru 94 down_at 97 last_clean_interval [91,96) [v2:127.0.0.1:6826/314023090,v1:127.0.0.1:6827/314023090] [v2:127.0.0.1:6828/314023090,v1:127.0.0.1:6829/314023090] exists,up 48807f11-9c3a-49d3-8033-a247f29eadfa 2024-09-15T20:51:46.677 INFO:tasks.workunit.client.0.smithi136.stderr:158152: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-15T20:51:46.677 INFO:tasks.workunit.client.0.smithi136.stderr:158152: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-15T20:51:46.677 INFO:tasks.workunit.client.0.smithi136.stderr:158152: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-15T20:51:46.677 INFO:tasks.workunit.client.0.smithi136.stderr:158152: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2024-09-15T20:51:46.677 INFO:tasks.workunit.client.0.smithi136.stderr:158152: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-15T20:51:46.677 INFO:tasks.workunit.client.0.smithi136.stderr:158152: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-15T20:51:46.677 INFO:tasks.workunit.client.0.smithi136.stderr:158152: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-15T20:51:46.677 INFO:tasks.workunit.client.0.smithi136.stderr:158152: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-15T20:51:46.677 INFO:tasks.workunit.client.0.smithi136.stderr:158152: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-15T20:51:46.677 INFO:tasks.workunit.client.0.smithi136.stderr:158152: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-15T20:51:46.677 INFO:tasks.workunit.client.0.smithi136.stderr:158152: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-15T20:51:46.677 INFO:tasks.workunit.client.0.smithi136.stderr:158152: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-15T20:51:46.678 INFO:tasks.workunit.client.0.smithi136.stderr:158152: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-15T20:51:46.678 INFO:tasks.workunit.client.0.smithi136.stderr:158152: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-15T20:51:46.678 INFO:tasks.workunit.client.0.smithi136.stderr:158152: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-15T20:51:46.678 INFO:tasks.workunit.client.0.smithi136.stderr:158152: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-15T20:51:46.678 INFO:tasks.workunit.client.0.smithi136.stderr:158152: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-15T20:51:46.678 INFO:tasks.workunit.client.0.smithi136.stderr:158152: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-15T20:51:46.678 INFO:tasks.workunit.client.0.smithi136.stderr:158152: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-15T20:51:46.678 INFO:tasks.workunit.client.0.smithi136.stderr:158152: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-15T20:51:46.678 INFO:tasks.workunit.client.0.smithi136.stderr:158152: 1 2024-09-15T20:51:46.678 INFO:tasks.workunit.client.0.smithi136.stderr:158152: 2 2024-09-15T20:51:46.678 INFO:tasks.workunit.client.0.smithi136.stderr:158152: 3' 2024-09-15T20:51:46.678 INFO:tasks.workunit.client.0.smithi136.stderr:158152: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-15T20:51:46.678 INFO:tasks.workunit.client.0.smithi136.stderr:158152: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T20:51:46.678 INFO:tasks.workunit.client.0.smithi136.stderr:158152: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-15T20:51:46.679 INFO:tasks.workunit.client.0.smithi136.stderr:158152: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836551 2024-09-15T20:51:46.679 INFO:tasks.workunit.client.0.smithi136.stderr:158152: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836551 2024-09-15T20:51:46.679 INFO:tasks.workunit.client.0.smithi136.stderr:158152: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836551' 2024-09-15T20:51:48.723 INFO:tasks.workunit.client.0.smithi136.stderr:158152: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:ph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-15T20:51:48.723 INFO:tasks.workunit.client.0.smithi136.stderr:158154: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=438086664194 2024-09-15T20:51:48.723 INFO:tasks.workunit.client.0.smithi136.stderr:158154: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 438086664194 2024-09-15T20:51:48.723 INFO:tasks.workunit.client.0.smithi136.stderr:158154: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836550 1-438086664194' 2024-09-15T20:51:48.723 INFO:tasks.workunit.client.0.smithi136.stderr:158154: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T20:51:48.724 INFO:tasks.workunit.client.0.smithi136.stderr:158154: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-15T20:51:48.724 INFO:tasks.workunit.client.0.smithi136.stderr:158154: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=360777252875 2024-09-15T20:51:48.724 INFO:tasks.workunit.client.0.smithi136.stderr:158154: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 360777252875 2024-09-15T20:51:48.724 INFO:tasks.workunit.client.0.smithi136.stderr:158154: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836550 1-438086664194 2-360777252875' 2024-09-15T20:51:48.724 INFO:tasks.workunit.client.0.smithi136.stderr:158154: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T20:51:48.724 INFO:tasks.workunit.client.0.smithi136.stderr:158154: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-15T20:51:48.724 INFO:tasks.workunit.client.0.smithi136.stderr:158154: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=438086664194 2024-09-15T20:51:48.724 INFO:tasks.workunit.client.0.smithi136.stderr:158154: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 438086664194 2024-09-15T20:51:48.724 INFO:tasks.workunit.client.0.smithi136.stderr:158154: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836550 1-438086664194 2-360777252875 3-438086664194' 2024-09-15T20:51:48.724 INFO:tasks.workunit.client.0.smithi136.stderr:158154: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T20:51:48.724 INFO:tasks.workunit.client.0.smithi136.stderr:158154: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836550 2024-09-15T20:51:48.724 INFO:tasks.workunit.client.0.smithi136.stderr:158154: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T20:51:48.725 INFO:tasks.workunit.client.0.smithi136.stderr:158154: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-15T20:51:48.725 INFO:tasks.workunit.client.0.smithi136.stderr:158154: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836550 2024-09-15T20:51:48.725 INFO:tasks.workunit.client.0.smithi136.stderr:158154: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T20:51:48.725 INFO:tasks.workunit.client.0.smithi136.stderr:158154: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836550 2024-09-15T20:51:48.725 INFO:tasks.workunit.client.0.smithi136.stderr:158154: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836550' 2024-09-15T20:51:48.725 INFO:tasks.workunit.client.0.smithi136.stderr:158154: waiting osd.0 seq 21474836550 2024-09-15T20:51:48.725 INFO:tasks.workunit.client.0.smithi136.stderr:158154: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T20:51:48.725 INFO:tasks.workunit.client.0.smithi136.stderr:158154: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836548 -lt 21474836550 2024-09-15T20:51:48.725 INFO:tasks.workunit.client.0.smithi136.stderr:158154: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-15T20:51:48.725 INFO:tasks.workunit.client.0.smithi136.stderr:158154: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-15T20:51:48.725 INFO:tasks.workunit.client.0.smithi136.stderr:158154: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T20:51:48.725 INFO:tasks.workunit.client.0.smithi136.stderr:158154: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836551 -lt 21474836550 2024-09-15T20:51:48.725 INFO:tasks.workunit.client.0.smithi136.stderr:158154: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T20:51:48.726 INFO:tasks.workunit.client.0.smithi136.stderr:158154: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-438086664194 2024-09-15T20:51:48.726 INFO:tasks.workunit.client.0.smithi136.stderr:158154: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T20:51:48.726 INFO:tasks.workunit.client.0.smithi136.stderr:158154: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-15T20:51:48.726 INFO:tasks.workunit.client.0.smithi136.stderr:158154: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-438086664194 2024-09-15T20:51:48.726 INFO:tasks.workunit.client.0.smithi136.stderr:158154: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T20:51:48.895 INFO:tasks.workunit.client.0.smithi136.stderr:158154: /2209: flush_pg_stats: for osd in $ids 2024-09-15T20:51:48.895 INFO:tasks.workunit.client.0.smithi136.stderr:158152: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-15T20:51:48.895 INFO:tasks.workunit.client.0.smithi136.stderr:158152: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=438086664195 2024-09-15T20:51:48.895 INFO:tasks.workunit.client.0.smithi136.stderr:158152: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 438086664195 2024-09-15T20:51:48.896 INFO:tasks.workunit.client.0.smithi136.stderr:158152: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836551 1-438086664195' 2024-09-15T20:51:48.896 INFO:tasks.workunit.client.0.smithi136.stderr:158152: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T20:51:48.896 INFO:tasks.workunit.client.0.smithi136.stderr:158152: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-15T20:51:48.896 INFO:tasks.workunit.client.0.smithi136.stderr:158152: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=360777252877 2024-09-15T20:51:48.896 INFO:tasks.workunit.client.0.smithi136.stderr:158152: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 360777252877 2024-09-15T20:51:48.897 INFO:tasks.workunit.client.0.smithi136.stderr:158152: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836551 1-438086664195 2-360777252877' 2024-09-15T20:51:48.897 INFO:tasks.workunit.client.0.smithi136.stderr:158152: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T20:51:48.897 INFO:tasks.workunit.client.0.smithi136.stderr:158152: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-15T20:51:48.897 INFO:tasks.workunit.client.0.smithi136.stderr:158152: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=438086664195 2024-09-15T20:51:48.897 INFO:tasks.workunit.client.0.smithi136.stderr:158152: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 438086664195 2024-09-15T20:51:48.898 INFO:tasks.workunit.client.0.smithi136.stderr:158152: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836551 1-438086664195 2-360777252877 3-438086664195' 2024-09-15T20:51:48.898 INFO:tasks.workunit.client.0.smithi136.stderr:158152: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T20:51:48.898 INFO:tasks.workunit.client.0.smithi136.stderr:158152: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836551 2024-09-15T20:51:48.898 INFO:tasks.workunit.client.0.smithi136.stderr:158152: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T20:51:48.898 INFO:tasks.workunit.client.0.smithi136.stderr:158152: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-15T20:51:48.898 INFO:tasks.workunit.client.0.smithi136.stderr:158152: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836551 2024-09-15T20:51:48.899 INFO:tasks.workunit.client.0.smithi136.stderr:158152: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T20:51:48.899 INFO:tasks.workunit.client.0.smithi136.stderr:158152: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836551 2024-09-15T20:51:48.899 INFO:tasks.workunit.client.0.smithi136.stderr:158152: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836551' 2024-09-15T20:51:48.899 INFO:tasks.workunit.client.0.smithi136.stderr:158152: waiting osd.0 seq 21474836551 2024-09-15T20:51:48.899 INFO:tasks.workunit.client.0.smithi136.stderr:158152: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T20:51:48.900 INFO:tasks.workunit.client.0.smithi136.stderr:158152: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836548 -lt 21474836551 2024-09-15T20:51:48.900 INFO:tasks.workunit.client.0.smithi136.stderr:158152: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-15T20:51:48.900 INFO:tasks.workunit.client.0.smithi136.stderr:158152: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-15T20:51:48.900 INFO:tasks.workunit.client.0.smithi136.stderr:158152: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T20:51:48.901 INFO:tasks.workunit.client.0.smithi136.stderr:158152: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836551 -lt 21474836551 2024-09-15T20:51:48.901 INFO:tasks.workunit.client.0.smithi136.stderr:158152: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T20:51:48.901 INFO:tasks.workunit.client.0.smithi136.stderr:158152: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-438086664195 2024-09-15T20:51:48.901 INFO:tasks.workunit.client.0.smithi136.stderr:158152: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T20:51:48.901 INFO:tasks.workunit.client.0.smithi136.stderr:158152: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-15T20:51:48.902 INFO:tasks.workunit.client.0.smithi136.stderr:158152: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-438086664195 2024-09-15T20:51:50.037 INFO:tasks.workunit.client.0.smithi136.stderr:158152: //home/ubuntuhome/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=438086664194 2024-09-15T20:51:50.037 INFO:tasks.workunit.client.0.smithi136.stderr:158154: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 438086664194' 2024-09-15T20:51:50.037 INFO:tasks.workunit.client.0.smithi136.stderr:158154: waiting osd.1 seq 438086664194 2024-09-15T20:51:50.037 INFO:tasks.workunit.client.0.smithi136.stderr:158154: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-15T20:51:50.037 INFO:tasks.workunit.client.0.smithi136.stderr:158154: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 438086664195 -lt 438086664194 2024-09-15T20:51:50.037 INFO:tasks.workunit.client.0.smithi136.stderr:158154: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T20:51:50.037 INFO:tasks.workunit.client.0.smithi136.stderr:158154: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-360777252875 2024-09-15T20:51:50.038 INFO:tasks.workunit.client.0.smithi136.stderr:158154: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T20:51:50.038 INFO:tasks.workunit.client.0.smithi136.stderr:158154: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-15T20:51:50.038 INFO:tasks.workunit.client.0.smithi136.stderr:158154: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-360777252875 2024-09-15T20:51:50.038 INFO:tasks.workunit.client.0.smithi136.stderr:158154: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T20:51:50.038 INFO:tasks.workunit.client.0.smithi136.stderr:158154: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=360777252875 2024-09-15T20:51:50.038 INFO:tasks.workunit.client.0.smithi136.stderr:158154: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 360777252875' 2024-09-15T20:51:50.038 INFO:tasks.workunit.client.0.smithi136.stderr:158154: waiting osd.2 seq 360777252875 2024-09-15T20:51:50.038 INFO:tasks.workunit.client.0.smithi136.stderr:158154: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-15T20:51:50.038 INFO:tasks.workunit.client.0.smithi136.stderr:158154: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 360777252877 -lt 360777252875 2024-09-15T20:51:50.038 INFO:tasks.workunit.client.0.smithi136.stderr:158154: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T20:51:50.038 INFO:tasks.workunit.client.0.smithi136.stderr:158154: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 3-438086664194 2024-09-15T20:51:50.038 INFO:tasks.workunit.client.0.smithi136.stderr:158154: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T20:51:50.038 INFO:tasks.workunit.client.0.smithi136.stderr:158154: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=3 2024-09-15T20:51:50.039 INFO:tasks.workunit.client.0.smithi136.stderr:158154: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 3-438086664194 2024-09-15T20:51:50.039 INFO:tasks.workunit.client.0.smithi136.stderr:158154: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T20:51:50.039 INFO:tasks.workunit.client.0.smithi136.stderr:158154: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=438086664194 2024-09-15T20:51:50.039 INFO:tasks.workunit.client.0.smithi136.stderr:158154: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.3 seq 438086664194' 2024-09-15T20:51:50.039 INFO:tasks.workunit.client.0.smithi136.stderr:158154: waiting osd.3 seq 438086664194 2024-09-15T20:51:50.039 INFO:tasks.workunit.client.0.smithi136.stderr:158154: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-15T20:51:50.039 INFO:tasks.workunit.client.0.smithi136.stderr:158154: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 438086664195 -lt 438086664194 2024-09-15T20:51:50.039 INFO:tasks.workunit.client.0.smithi136.stderr:158154: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-15T20:51:50.039 INFO:tasks.workunit.client.0.smithi136.stderr:158154: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T20:51:50.039 INFO:tasks.workunit.client.0.smithi136.stderr:158154: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T20:51:50.039 INFO:tasks.workunit.client.0.smithi136.stderr:158154: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 5 == 0 2024-09-15T20:51:50.040 INFO:tasks.workunit.client.0.smithi136.stderr:158154: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-15T20:51:50.040 INFO:tasks.workunit.client.0.smithi136.stderr:158154: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-15T20:51:50.040 INFO:tasks.workunit.client.0.smithi136.stderr:158154: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-15T20:51:50.040 INFO:tasks.workunit.client.0.smithi136.stderr:158154: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T20:51:50.255 INFO:tasks.workunit.client.0.smithi136.stderr:158154: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(cont/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T20:51:50.255 INFO:tasks.workunit.client.0.smithi136.stderr:158152: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=438086664195 2024-09-15T20:51:50.255 INFO:tasks.workunit.client.0.smithi136.stderr:158152: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 438086664195' 2024-09-15T20:51:50.256 INFO:tasks.workunit.client.0.smithi136.stderr:158152: waiting osd.1 seq 438086664195 2024-09-15T20:51:50.256 INFO:tasks.workunit.client.0.smithi136.stderr:158152: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-15T20:51:50.256 INFO:tasks.workunit.client.0.smithi136.stderr:158152: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 438086664195 -lt 438086664195 2024-09-15T20:51:50.256 INFO:tasks.workunit.client.0.smithi136.stderr:158152: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T20:51:50.256 INFO:tasks.workunit.client.0.smithi136.stderr:158152: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-360777252877 2024-09-15T20:51:50.256 INFO:tasks.workunit.client.0.smithi136.stderr:158152: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T20:51:50.256 INFO:tasks.workunit.client.0.smithi136.stderr:158152: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-15T20:51:50.256 INFO:tasks.workunit.client.0.smithi136.stderr:158152: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-360777252877 2024-09-15T20:51:50.256 INFO:tasks.workunit.client.0.smithi136.stderr:158152: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T20:51:50.256 INFO:tasks.workunit.client.0.smithi136.stderr:158152: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=360777252877 2024-09-15T20:51:50.256 INFO:tasks.workunit.client.0.smithi136.stderr:158152: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 360777252877' 2024-09-15T20:51:50.257 INFO:tasks.workunit.client.0.smithi136.stderr:158152: waiting osd.2 seq 360777252877 2024-09-15T20:51:50.257 INFO:tasks.workunit.client.0.smithi136.stderr:158152: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-15T20:51:50.257 INFO:tasks.workunit.client.0.smithi136.stderr:158152: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 360777252877 -lt 360777252877 2024-09-15T20:51:50.257 INFO:tasks.workunit.client.0.smithi136.stderr:158152: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T20:51:50.257 INFO:tasks.workunit.client.0.smithi136.stderr:158152: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 3-438086664195 2024-09-15T20:51:50.257 INFO:tasks.workunit.client.0.smithi136.stderr:158152: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T20:51:50.257 INFO:tasks.workunit.client.0.smithi136.stderr:158152: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=3 2024-09-15T20:51:50.257 INFO:tasks.workunit.client.0.smithi136.stderr:158152: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 3-438086664195 2024-09-15T20:51:50.257 INFO:tasks.workunit.client.0.smithi136.stderr:158152: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T20:51:50.257 INFO:tasks.workunit.client.0.smithi136.stderr:158152: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=438086664195 2024-09-15T20:51:50.257 INFO:tasks.workunit.client.0.smithi136.stderr:158152: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.3 seq 438086664195' 2024-09-15T20:51:50.258 INFO:tasks.workunit.client.0.smithi136.stderr:158152: waiting osd.3 seq 438086664195 2024-09-15T20:51:50.258 INFO:tasks.workunit.client.0.smithi136.stderr:158152: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-15T20:51:50.258 INFO:tasks.workunit.client.0.smithi136.stderr:158152: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 438086664195 -lt 438086664195 2024-09-15T20:51:50.258 INFO:tasks.workunit.client.0.smithi136.stderr:158152: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-15T20:51:50.258 INFO:tasks.workunit.client.0.smithi136.stderr:158152: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T20:51:50.258 INFO:tasks.workunit.client.0.smithi136.stderr:158152: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T20:51:50.258 INFO:tasks.workunit.client.0.smithi136.stderr:158152: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 5 == 0 2024-09-15T20:51:50.259 INFO:tasks.workunit.client.0.smithi136.stderr:158152: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-15T20:51:50.259 INFO:tasks.workunit.client.0.smithi136.stderr:158152: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-15T20:51:50.259 INFO:tasks.workunit.client.0.smithi136.stderr:158152: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-15T20:51:50.259 INFO:tasks.workunit.client.0.smithi136.stderr:158152: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T20:51:51.477 INFO:tasks.workunit.client.0.smithi136.stderr:158152: //home/ubuntu/cephteains("stale") | not)' 2024-09-15T20:51:51.477 INFO:tasks.workunit.client.0.smithi136.stderr:158154: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T20:51:51.477 INFO:tasks.workunit.client.0.smithi136.stderr:158154: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T20:51:51.477 INFO:tasks.workunit.client.0.smithi136.stderr:158154: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=4 2024-09-15T20:51:51.477 INFO:tasks.workunit.client.0.smithi136.stderr:158154: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-15T20:51:51.478 INFO:tasks.workunit.client.0.smithi136.stderr:158154: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T20:51:51.478 INFO:tasks.workunit.client.0.smithi136.stderr:158154: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T20:51:51.478 INFO:tasks.workunit.client.0.smithi136.stderr:158154: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 = 5 2024-09-15T20:51:51.478 INFO:tasks.workunit.client.0.smithi136.stderr:158154: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 4 '!=' -1 2024-09-15T20:51:51.478 INFO:tasks.workunit.client.0.smithi136.stderr:158154: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: loop=0 2024-09-15T20:51:51.478 INFO:tasks.workunit.client.0.smithi136.stderr:158154: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1652: wait_for_clean: num_active_clean=4 2024-09-15T20:51:51.478 INFO:tasks.workunit.client.0.smithi136.stderr:158154: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-15T20:51:51.478 INFO:tasks.workunit.client.0.smithi136.stderr:158154: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 0.1 2024-09-15T20:51:51.478 INFO:tasks.workunit.client.0.smithi136.stderr:158154: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-15T20:51:51.478 INFO:tasks.workunit.client.0.smithi136.stderr:158154: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-15T20:51:51.478 INFO:tasks.workunit.client.0.smithi136.stderr:158154: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-15T20:51:51.478 INFO:tasks.workunit.client.0.smithi136.stderr:158154: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-15T20:51:51.478 INFO:tasks.workunit.client.0.smithi136.stderr:158154: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T20:51:51.479 INFO:tasks.workunit.client.0.smithi136.stderr:158154: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T20:51:51.479 INFO:tasks.workunit.client.0.smithi136.stderr:158154: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T20:51:51.479 INFO:tasks.workunit.client.0.smithi136.stderr:158154: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T20:51:51.479 INFO:tasks.workunit.client.0.smithi136.stderr:158154: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=4 2024-09-15T20:51:51.479 INFO:tasks.workunit.client.0.smithi136.stderr:158154: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-15T20:51:51.480 INFO:tasks.workunit.client.0.smithi136.stderr:158154: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T20:51:51.480 INFO:tasks.workunit.client.0.smithi136.stderr:158154: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T20:51:51.480 INFO:tasks.workunit.client.0.smithi136.stderr:158154: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 = 5 2024-09-15T20:51:51.480 INFO:tasks.workunit.client.0.smithi136.stderr:158154: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 4 '!=' 4 2024-09-15T20:51:51.480 INFO:tasks.workunit.client.0.smithi136.stderr:158154: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: get_is_making_recovery_progress 2024-09-15T20:51:51.480 INFO:tasks.workunit.client.0.smithi136.stderr:158154: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2024-09-15T20:51:51.480 INFO:tasks.workunit.client.0.smithi136.stderr:158154: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-15T20:51:51.480 INFO:tasks.workunit.client.0.smithi136.stderr:158154: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-15T20:51:51.480 INFO:tasks.workunit.client.0.smithi136.stderr:158154: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-15T20:51:51.707 INFO:tasks.workunit.client.0.smithi136.stderr:158st/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T20:51:51.708 INFO:tasks.workunit.client.0.smithi136.stderr:158152: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T20:51:51.708 INFO:tasks.workunit.client.0.smithi136.stderr:158152: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T20:51:51.708 INFO:tasks.workunit.client.0.smithi136.stderr:158152: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=4 2024-09-15T20:51:51.708 INFO:tasks.workunit.client.0.smithi136.stderr:158152: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-15T20:51:51.708 INFO:tasks.workunit.client.0.smithi136.stderr:158152: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T20:51:51.708 INFO:tasks.workunit.client.0.smithi136.stderr:158152: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T20:51:51.708 INFO:tasks.workunit.client.0.smithi136.stderr:158152: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 = 5 2024-09-15T20:51:51.708 INFO:tasks.workunit.client.0.smithi136.stderr:158152: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 4 '!=' -1 2024-09-15T20:51:51.708 INFO:tasks.workunit.client.0.smithi136.stderr:158152: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: loop=0 2024-09-15T20:51:51.708 INFO:tasks.workunit.client.0.smithi136.stderr:158152: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1652: wait_for_clean: num_active_clean=4 2024-09-15T20:51:51.708 INFO:tasks.workunit.client.0.smithi136.stderr:158152: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-15T20:51:51.708 INFO:tasks.workunit.client.0.smithi136.stderr:158152: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 0.1 2024-09-15T20:51:51.709 INFO:tasks.workunit.client.0.smithi136.stderr:158152: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-15T20:51:51.709 INFO:tasks.workunit.client.0.smithi136.stderr:158152: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-15T20:51:51.709 INFO:tasks.workunit.client.0.smithi136.stderr:158152: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-15T20:51:51.709 INFO:tasks.workunit.client.0.smithi136.stderr:158152: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-15T20:51:51.709 INFO:tasks.workunit.client.0.smithi136.stderr:158152: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T20:51:51.709 INFO:tasks.workunit.client.0.smithi136.stderr:158152: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T20:51:51.709 INFO:tasks.workunit.client.0.smithi136.stderr:158152: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T20:51:51.709 INFO:tasks.workunit.client.0.smithi136.stderr:158152: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T20:51:51.709 INFO:tasks.workunit.client.0.smithi136.stderr:158152: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=4 2024-09-15T20:51:51.709 INFO:tasks.workunit.client.0.smithi136.stderr:158152: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-15T20:51:51.709 INFO:tasks.workunit.client.0.smithi136.stderr:158152: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T20:51:51.709 INFO:tasks.workunit.client.0.smithi136.stderr:158152: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T20:51:51.709 INFO:tasks.workunit.client.0.smithi136.stderr:158152: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 = 5 2024-09-15T20:51:51.710 INFO:tasks.workunit.client.0.smithi136.stderr:158152: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 4 '!=' 4 2024-09-15T20:51:51.710 INFO:tasks.workunit.client.0.smithi136.stderr:158152: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: get_is_making_recovery_progress 2024-09-15T20:51:51.710 INFO:tasks.workunit.client.0.smithi136.stderr:158152: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2024-09-15T20:51:51.710 INFO:tasks.workunit.client.0.smithi136.stderr:158152: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-15T20:51:51.710 INFO:tasks.workunit.client.0.smithi136.stderr:158152: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-15T20:51:52.627 INFO:tasks.workunit.client.0.smithi136.stderr:158152: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-154: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2024-09-15T20:51:52.627 INFO:tasks.workunit.client.0.smithi136.stderr:158154: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-15T20:51:52.627 INFO:tasks.workunit.client.0.smithi136.stderr:158154: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=2 2024-09-15T20:51:52.628 INFO:tasks.workunit.client.0.smithi136.stderr:158154: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test 2 '!=' null 2024-09-15T20:51:52.628 INFO:tasks.workunit.client.0.smithi136.stderr:158154: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: loop=0 2024-09-15T20:51:52.628 INFO:tasks.workunit.client.0.smithi136.stderr:158154: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-15T20:51:52.628 INFO:tasks.workunit.client.0.smithi136.stderr:158154: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 0.1 2024-09-15T20:51:52.628 INFO:tasks.workunit.client.0.smithi136.stderr:158154: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-15T20:51:52.628 INFO:tasks.workunit.client.0.smithi136.stderr:158154: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-15T20:51:52.628 INFO:tasks.workunit.client.0.smithi136.stderr:158154: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-15T20:51:52.628 INFO:tasks.workunit.client.0.smithi136.stderr:158154: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-15T20:51:52.628 INFO:tasks.workunit.client.0.smithi136.stderr:158154: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T20:51:52.628 INFO:tasks.workunit.client.0.smithi136.stderr:158154: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T20:51:52.628 INFO:tasks.workunit.client.0.smithi136.stderr:158154: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T20:51:52.628 INFO:tasks.workunit.client.0.smithi136.stderr:158154: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T20:51:52.628 INFO:tasks.workunit.client.0.smithi136.stderr:158154: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=5 2024-09-15T20:51:52.629 INFO:tasks.workunit.client.0.smithi136.stderr:158154: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-15T20:51:52.629 INFO:tasks.workunit.client.0.smithi136.stderr:158154: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T20:51:52.629 INFO:tasks.workunit.client.0.smithi136.stderr:158154: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T20:51:52.629 INFO:tasks.workunit.client.0.smithi136.stderr:158154: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 = 5 2024-09-15T20:51:52.629 INFO:tasks.workunit.client.0.smithi136.stderr:158154: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-15T20:51:52.629 INFO:tasks.workunit.client.0.smithi136.stderr:158154: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-15T20:51:52.629 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: run_in_background: return 0 2024-09-15T20:51:52.834 INFO:tasks.workunit.client.0.smithi136.stderr:helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-15T20:51:52.834 INFO:tasks.workunit.client.0.smithi136.stderr:158152: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2024-09-15T20:51:52.834 INFO:tasks.workunit.client.0.smithi136.stderr:158152: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-15T20:51:52.834 INFO:tasks.workunit.client.0.smithi136.stderr:158152: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=2 2024-09-15T20:51:52.834 INFO:tasks.workunit.client.0.smithi136.stderr:158152: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test 2 '!=' null 2024-09-15T20:51:52.834 INFO:tasks.workunit.client.0.smithi136.stderr:158152: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: loop=0 2024-09-15T20:51:52.834 INFO:tasks.workunit.client.0.smithi136.stderr:158152: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-15T20:51:52.834 INFO:tasks.workunit.client.0.smithi136.stderr:158152: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 0.1 2024-09-15T20:51:52.834 INFO:tasks.workunit.client.0.smithi136.stderr:158152: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-15T20:51:52.834 INFO:tasks.workunit.client.0.smithi136.stderr:158152: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-15T20:51:52.834 INFO:tasks.workunit.client.0.smithi136.stderr:158152: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-15T20:51:52.835 INFO:tasks.workunit.client.0.smithi136.stderr:158152: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-15T20:51:52.835 INFO:tasks.workunit.client.0.smithi136.stderr:158152: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T20:51:52.835 INFO:tasks.workunit.client.0.smithi136.stderr:158152: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T20:51:52.835 INFO:tasks.workunit.client.0.smithi136.stderr:158152: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T20:51:52.835 INFO:tasks.workunit.client.0.smithi136.stderr:158152: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T20:51:52.835 INFO:tasks.workunit.client.0.smithi136.stderr:158152: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=5 2024-09-15T20:51:52.835 INFO:tasks.workunit.client.0.smithi136.stderr:158152: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-15T20:51:52.835 INFO:tasks.workunit.client.0.smithi136.stderr:158152: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T20:51:52.835 INFO:tasks.workunit.client.0.smithi136.stderr:158152: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T20:51:52.835 INFO:tasks.workunit.client.0.smithi136.stderr:158152: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 = 5 2024-09-15T20:51:52.835 INFO:tasks.workunit.client.0.smithi136.stderr:158152: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-15T20:51:52.835 INFO:tasks.workunit.client.0.smithi136.stderr:158152: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-15T20:51:52.835 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: run_in_background: return 0 2024-09-15T20:51:52.835 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: wait_background: for pid in $pids 2024-09-15T20:51:52.836 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2175: wait_background: wait 158151 2024-09-15T20:51:52.836 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: wait_background: eval 'pids='\'''\''' 2024-09-15T20:51:52.836 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: wait_background: pids= 2024-09-15T20:51:52.836 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: wait_background: return 0 2024-09-15T20:51:52.836 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:212: corrupt_and_repair_two: return_code=0 2024-09-15T20:51:52.836 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:213: corrupt_and_repair_two: '[' 0 -ne 0 ']' 2024-09-15T20:51:52.836 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:215: corrupt_and_repair_two: rados --pool ecpool get SOMETHING td/osd-scrub-repair/COPY 2024-09-15T20:51:52.878 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:216: corrupt_and_repair_two: diff td/osd-scrub-repair/ORIGINAL td/osd-scrub-repair/COPY 2024-09-15T20:51:52.879 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:66: run: teardown td/osd-scrub-repair 2024-09-15T20:51:52.879 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/osd-scrub-repair 2024-09-15T20:51:52.879 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-15T20:51:52.879 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/osd-scrub-repair KILL 2024-09-15T20:51:52.879 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-15T20:51:52.880 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-15T20:51:52.880 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-15T20:51:52.880 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-15T20:51:52.880 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-15T20:51:52.988 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-15T20:51:52.988 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-15T20:51:52.989 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-15T20:51:52.989 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-15T20:51:52.990 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-15T20:51:52.990 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-15T20:51:52.991 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-15T20:51:52.991 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T20:51:52.991 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-15T20:51:52.992 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-15T20:51:52.992 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T20:51:52.992 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-15T20:51:52.994 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-15T20:51:52.994 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/osd-scrub-repair 2024-09-15T20:51:53.025 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-15T20:51:53.025 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T20:51:53.025 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52602 2024-09-15T20:51:53.025 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.52602 2024-09-15T20:51:53.027 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-15T20:51:53.027 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-15T20:51:53.027 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:63: run: for func in $funcs 2024-09-15T20:51:53.027 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:64: run: setup td/osd-scrub-repair 2024-09-15T20:51:53.027 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/osd-scrub-repair 2024-09-15T20:51:53.027 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/osd-scrub-repair 2024-09-15T20:51:53.027 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/osd-scrub-repair 2024-09-15T20:51:53.027 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-15T20:51:53.027 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/osd-scrub-repair KILL 2024-09-15T20:51:53.027 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-15T20:51:53.027 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-15T20:51:53.028 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-15T20:51:53.028 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-15T20:51:53.028 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-15T20:51:53.028 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-15T20:51:53.028 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-15T20:51:53.030 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-15T20:51:53.030 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-15T20:51:53.030 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-15T20:51:53.031 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-15T20:51:53.031 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-15T20:51:53.032 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T20:51:53.032 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-15T20:51:53.032 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-15T20:51:53.032 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T20:51:53.033 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-15T20:51:53.034 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-15T20:51:53.034 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/osd-scrub-repair 2024-09-15T20:51:53.035 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-15T20:51:53.035 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T20:51:53.036 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52602 2024-09-15T20:51:53.036 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.52602 2024-09-15T20:51:53.037 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-15T20:51:53.037 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-15T20:51:53.037 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/osd-scrub-repair 2024-09-15T20:51:53.038 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2024-09-15T20:51:53.038 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T20:51:53.038 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52602 2024-09-15T20:51:53.038 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.52602 2024-09-15T20:51:53.040 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2024-09-15T20:51:53.040 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 4096 -le 1024 ']' 2024-09-15T20:51:53.040 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2024-09-15T20:51:53.041 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/osd-scrub-repair 1' TERM HUP INT 2024-09-15T20:51:53.041 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:65: run: TEST_corrupt_and_repair_jerasure_overwrites td/osd-scrub-repair 2024-09-15T20:51:53.041 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:819: TEST_corrupt_and_repair_jerasure_overwrites: '[' true = true ']' 2024-09-15T20:51:53.041 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:820: TEST_corrupt_and_repair_jerasure_overwrites: corrupt_and_repair_jerasure td/osd-scrub-repair true 2024-09-15T20:51:53.041 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:798: corrupt_and_repair_jerasure: local dir=td/osd-scrub-repair 2024-09-15T20:51:53.041 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:799: corrupt_and_repair_jerasure: local allow_overwrites=true 2024-09-15T20:51:53.041 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:800: corrupt_and_repair_jerasure: local poolname=ecpool 2024-09-15T20:51:53.041 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:802: corrupt_and_repair_jerasure: run_mon td/osd-scrub-repair a 2024-09-15T20:51:53.041 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/osd-scrub-repair 2024-09-15T20:51:53.041 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2024-09-15T20:51:53.041 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2024-09-15T20:51:53.041 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2024-09-15T20:51:53.041 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/osd-scrub-repair/a 2024-09-15T20:51:53.041 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-scrub-repair/a --run-dir=td/osd-scrub-repair 2024-09-15T20:51:53.499 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2024-09-15T20:51:53.499 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-15T20:51:53.499 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-15T20:51:53.499 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-15T20:51:53.499 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T20:51:53.499 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52602 2024-09-15T20:51:53.499 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.52602/$cluster-$name.asok' 2024-09-15T20:51:53.500 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-scrub-repair/a '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.52602/$cluster-$name.asok' --mon-cluster-log-file=td/osd-scrub-repair/log --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-09-15T20:51:53.699 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2024-09-15T20:51:53.700 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2024-09-15T20:51:53.700 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-15T20:51:53.700 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-15T20:51:53.700 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2024-09-15T20:51:53.701 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2024-09-15T20:51:53.701 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-15T20:51:53.701 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-15T20:51:53.701 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-15T20:51:53.702 INFO:tasks.workunit.client.0.smithi136.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-15T20:51:53.702 INFO:tasks.workunit.client.0.smithi136.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T20:51:53.702 INFO:tasks.workunit.client.0.smithi136.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52602 2024-09-15T20:51:53.702 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.52602/ceph-mon.a.asok 2024-09-15T20:51:53.702 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-15T20:51:53.702 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.52602/ceph-mon.a.asok config get fsid 2024-09-15T20:51:53.788 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2024-09-15T20:51:53.788 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-15T20:51:53.788 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-15T20:51:53.788 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2024-09-15T20:51:53.788 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2024-09-15T20:51:53.788 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-15T20:51:53.788 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-15T20:51:53.789 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-15T20:51:53.789 INFO:tasks.workunit.client.0.smithi136.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-15T20:51:53.789 INFO:tasks.workunit.client.0.smithi136.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T20:51:53.790 INFO:tasks.workunit.client.0.smithi136.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52602 2024-09-15T20:51:53.790 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.52602/ceph-mon.a.asok 2024-09-15T20:51:53.790 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-15T20:51:53.790 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.52602/ceph-mon.a.asok config get mon_host 2024-09-15T20:51:53.868 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:803: corrupt_and_repair_jerasure: run_mgr td/osd-scrub-repair x 2024-09-15T20:51:53.868 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: run_mgr: local dir=td/osd-scrub-repair 2024-09-15T20:51:53.868 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: run_mgr: shift 2024-09-15T20:51:53.868 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: run_mgr: local id=x 2024-09-15T20:51:53.868 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: run_mgr: shift 2024-09-15T20:51:53.868 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:548: run_mgr: local data=td/osd-scrub-repair/x 2024-09-15T20:51:53.868 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:550: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-15T20:51:54.683 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: get_asok_path 2024-09-15T20:51:54.683 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-15T20:51:54.683 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-15T20:51:54.683 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-15T20:51:54.683 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T20:51:54.684 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52602 2024-09-15T20:51:54.684 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.52602/$cluster-$name.asok' 2024-09-15T20:51:54.684 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-15T20:51:54.684 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-scrub-repair/x '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.52602/$cluster-$name.asok' --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-15T20:51:54.713 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:804: corrupt_and_repair_jerasure: seq 0 3 2024-09-15T20:51:54.717 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:804: corrupt_and_repair_jerasure: for id in $(seq 0 3) 2024-09-15T20:51:54.717 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:805: corrupt_and_repair_jerasure: run_osd td/osd-scrub-repair 0 2024-09-15T20:51:54.717 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-scrub-repair 2024-09-15T20:51:54.717 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-15T20:51:54.717 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=0 2024-09-15T20:51:54.717 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-15T20:51:54.717 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-scrub-repair/0 2024-09-15T20:51:54.717 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=ce4892c7-6c6d-4b4f-8e6f-07a3e11133e5 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-15T20:51:54.717 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T20:51:54.717 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T20:51:54.717 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T20:51:54.718 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-09-15T20:51:54.718 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-09-15T20:51:54.718 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-15T20:51:54.718 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-15T20:51:54.718 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-15T20:51:54.718 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-15T20:51:54.718 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-15T20:51:54.718 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-15T20:51:54.718 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-15T20:51:54.718 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T20:51:54.718 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52602 2024-09-15T20:51:54.718 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.52602/$cluster-$name.asok' 2024-09-15T20:51:54.718 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.52602/$cluster-$name.asok' 2024-09-15T20:51:54.718 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-15T20:51:54.719 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-15T20:51:54.719 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-15T20:51:54.719 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-15T20:51:54.719 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-15T20:51:54.719 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T20:51:54.719 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T20:51:54.719 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T20:51:54.719 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T20:51:54.719 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-15T20:51:54.719 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-15T20:51:54.719 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-scrub-repair/0 2024-09-15T20:51:54.719 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-15T20:51:54.720 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=81932c4b-1e94-460b-a286-00cb26282f5f 2024-09-15T20:51:54.720 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd0 81932c4b-1e94-460b-a286-00cb26282f5f' 2024-09-15T20:51:54.720 INFO:tasks.workunit.client.0.smithi136.stdout:add osd0 81932c4b-1e94-460b-a286-00cb26282f5f 2024-09-15T20:51:54.720 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-15T20:51:54.736 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQDqSOdmhk7BKxAAYw/59vyOZS8frtU0OgBn+w== 2024-09-15T20:51:54.737 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQDqSOdmhk7BKxAAYw/59vyOZS8frtU0OgBn+w=="}' 2024-09-15T20:51:54.737 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 81932c4b-1e94-460b-a286-00cb26282f5f -i td/osd-scrub-repair/0/new.json 2024-09-15T20:51:54.939 INFO:tasks.workunit.client.0.smithi136.stdout:0 2024-09-15T20:51:54.952 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-scrub-repair/0/new.json 2024-09-15T20:51:54.954 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 0 --fsid=ce4892c7-6c6d-4b4f-8e6f-07a3e11133e5 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.52602/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDqSOdmhk7BKxAAYw/59vyOZS8frtU0OgBn+w== --osd-uuid 81932c4b-1e94-460b-a286-00cb26282f5f 2024-09-15T20:51:54.983 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:51:54.980+0000 7f4b0f40d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T20:51:54.989 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:51:54.987+0000 7f4b0f40d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T20:51:54.991 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:51:54.989+0000 7f4b0f40d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T20:51:54.991 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:51:54.989+0000 7f4b0f40d640 -1 bdev(0x5559efda1c00 td/osd-scrub-repair/0/block) open stat got: (1) Operation not permitted 2024-09-15T20:51:54.992 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:51:54.989+0000 7f4b0f40d640 -1 bluestore(td/osd-scrub-repair/0) _read_fsid unparsable uuid 2024-09-15T20:52:00.484 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-scrub-repair/0/keyring 2024-09-15T20:52:00.484 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-15T20:52:00.485 INFO:tasks.workunit.client.0.smithi136.stdout:adding osd0 key to auth repository 2024-09-15T20:52:00.485 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd0 key to auth repository 2024-09-15T20:52:00.485 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-scrub-repair/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-15T20:52:00.844 INFO:tasks.workunit.client.0.smithi136.stdout:start osd.0 2024-09-15T20:52:00.845 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.0 2024-09-15T20:52:00.845 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 0 --fsid=ce4892c7-6c6d-4b4f-8e6f-07a3e11133e5 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.52602/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T20:52:00.845 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-15T20:52:00.845 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-15T20:52:00.845 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-15T20:52:00.872 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:52:00.869+0000 7fa970e0d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T20:52:00.874 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:52:00.871+0000 7fa970e0d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T20:52:00.875 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:52:00.873+0000 7fa970e0d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T20:52:01.125 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 0 2024-09-15T20:52:01.126 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-15T20:52:01.126 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2024-09-15T20:52:01.126 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-15T20:52:01.126 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-15T20:52:01.126 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:52:01.126 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-15T20:52:01.126 INFO:tasks.workunit.client.0.smithi136.stdout:0 2024-09-15T20:52:01.126 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:52:01.127 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T20:52:01.402 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:52:02.227 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:52:02.224+0000 7fa970e0d640 -1 Falling back to public interface 2024-09-15T20:52:02.403 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:52:02.403 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:52:02.404 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-15T20:52:02.404 INFO:tasks.workunit.client.0.smithi136.stdout:1 2024-09-15T20:52:02.404 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:52:02.404 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T20:52:02.692 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:52:03.693 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:52:03.694 INFO:tasks.workunit.client.0.smithi136.stdout:2 2024-09-15T20:52:03.694 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:52:03.694 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-15T20:52:03.694 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:52:03.694 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T20:52:03.982 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:52:04.983 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:52:04.983 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:52:04.984 INFO:tasks.workunit.client.0.smithi136.stdout:3 2024-09-15T20:52:04.984 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-15T20:52:04.984 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:52:04.984 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T20:52:05.117 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:52:05.115+0000 7fa970e0d640 -1 osd.0 0 log_to_monitors true 2024-09-15T20:52:05.291 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:52:06.293 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:52:06.293 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:52:06.293 INFO:tasks.workunit.client.0.smithi136.stdout:4 2024-09-15T20:52:06.294 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-15T20:52:06.294 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:52:06.294 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T20:52:06.581 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:52:07.583 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:52:07.583 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:52:07.583 INFO:tasks.workunit.client.0.smithi136.stdout:5 2024-09-15T20:52:07.584 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-15T20:52:07.584 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:52:07.584 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T20:52:07.882 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:52:08.883 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:52:08.884 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:52:08.884 INFO:tasks.workunit.client.0.smithi136.stdout:6 2024-09-15T20:52:08.884 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-15T20:52:08.884 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:52:08.885 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T20:52:09.170 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:52:10.172 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:52:10.172 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:52:10.172 INFO:tasks.workunit.client.0.smithi136.stdout:7 2024-09-15T20:52:10.173 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-15T20:52:10.173 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:52:10.173 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T20:52:10.458 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:52:11.459 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:52:11.460 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:52:11.460 INFO:tasks.workunit.client.0.smithi136.stdout:8 2024-09-15T20:52:11.460 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-15T20:52:11.460 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:52:11.461 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T20:52:11.747 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:52:12.749 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:52:12.749 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:52:12.749 INFO:tasks.workunit.client.0.smithi136.stdout:9 2024-09-15T20:52:12.750 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-15T20:52:12.750 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:52:12.750 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T20:52:13.078 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:52:14.079 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:52:14.079 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:52:14.080 INFO:tasks.workunit.client.0.smithi136.stdout:10 2024-09-15T20:52:14.080 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-15T20:52:14.080 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:52:14.080 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T20:52:14.367 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:52:15.368 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:52:15.369 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:52:15.369 INFO:tasks.workunit.client.0.smithi136.stdout:11 2024-09-15T20:52:15.369 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-15T20:52:15.369 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:52:15.370 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T20:52:15.663 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:52:16.665 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:52:16.665 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:52:16.665 INFO:tasks.workunit.client.0.smithi136.stdout:12 2024-09-15T20:52:16.666 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-15T20:52:16.666 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:52:16.666 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T20:52:17.402 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:52:18.403 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:52:18.404 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:52:18.404 INFO:tasks.workunit.client.0.smithi136.stdout:13 2024-09-15T20:52:18.405 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-15T20:52:18.405 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:52:18.405 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T20:52:19.335 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:52:20.336 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:52:20.337 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:52:20.337 INFO:tasks.workunit.client.0.smithi136.stdout:14 2024-09-15T20:52:20.337 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-15T20:52:20.337 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:52:20.337 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T20:52:20.745 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:52:21.747 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:52:21.747 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:52:21.747 INFO:tasks.workunit.client.0.smithi136.stdout:15 2024-09-15T20:52:21.747 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-15T20:52:21.748 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:52:21.748 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T20:52:22.339 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:52:23.340 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:52:23.341 INFO:tasks.workunit.client.0.smithi136.stdout:16 2024-09-15T20:52:23.341 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:52:23.341 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-15T20:52:23.341 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:52:23.341 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T20:52:24.002 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:52:25.003 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:52:25.003 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:52:25.003 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-15T20:52:25.004 INFO:tasks.workunit.client.0.smithi136.stdout:17 2024-09-15T20:52:25.004 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:52:25.004 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T20:52:25.744 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:52:26.745 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:52:26.745 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:52:26.746 INFO:tasks.workunit.client.0.smithi136.stdout:18 2024-09-15T20:52:26.746 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 18 2024-09-15T20:52:26.746 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:52:26.746 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T20:52:27.033 INFO:tasks.workunit.client.0.smithi136.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/2715825214,v1:127.0.0.1:6803/2715825214] [v2:127.0.0.1:6804/2715825214,v1:127.0.0.1:6805/2715825214] exists,up 81932c4b-1e94-460b-a286-00cb26282f5f 2024-09-15T20:52:27.037 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-15T20:52:27.037 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-15T20:52:27.037 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-15T20:52:27.037 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:804: corrupt_and_repair_jerasure: for id in $(seq 0 3) 2024-09-15T20:52:27.038 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:805: corrupt_and_repair_jerasure: run_osd td/osd-scrub-repair 1 2024-09-15T20:52:27.038 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-scrub-repair 2024-09-15T20:52:27.038 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-15T20:52:27.038 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=1 2024-09-15T20:52:27.038 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-15T20:52:27.038 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-scrub-repair/1 2024-09-15T20:52:27.038 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=ce4892c7-6c6d-4b4f-8e6f-07a3e11133e5 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-15T20:52:27.038 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T20:52:27.038 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T20:52:27.038 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T20:52:27.038 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-09-15T20:52:27.038 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-09-15T20:52:27.038 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-15T20:52:27.039 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-15T20:52:27.039 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-15T20:52:27.039 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-15T20:52:27.039 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-15T20:52:27.039 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-15T20:52:27.039 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-15T20:52:27.039 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T20:52:27.039 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52602 2024-09-15T20:52:27.039 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.52602/$cluster-$name.asok' 2024-09-15T20:52:27.039 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.52602/$cluster-$name.asok' 2024-09-15T20:52:27.039 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-15T20:52:27.039 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-15T20:52:27.039 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-15T20:52:27.039 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-15T20:52:27.040 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-15T20:52:27.040 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T20:52:27.040 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T20:52:27.040 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T20:52:27.040 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T20:52:27.040 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-15T20:52:27.040 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-15T20:52:27.040 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-scrub-repair/1 2024-09-15T20:52:27.040 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-15T20:52:27.040 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=4e733256-a0ac-49e1-b991-7e6e4a060afc 2024-09-15T20:52:27.040 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd1 4e733256-a0ac-49e1-b991-7e6e4a060afc' 2024-09-15T20:52:27.041 INFO:tasks.workunit.client.0.smithi136.stdout:add osd1 4e733256-a0ac-49e1-b991-7e6e4a060afc 2024-09-15T20:52:27.041 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-15T20:52:27.056 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQALSedmFdowAxAA+DNEtL026W4aWluVOkIEwA== 2024-09-15T20:52:27.056 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQALSedmFdowAxAA+DNEtL026W4aWluVOkIEwA=="}' 2024-09-15T20:52:27.056 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 4e733256-a0ac-49e1-b991-7e6e4a060afc -i td/osd-scrub-repair/1/new.json 2024-09-15T20:52:27.487 INFO:tasks.workunit.client.0.smithi136.stdout:1 2024-09-15T20:52:27.498 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-scrub-repair/1/new.json 2024-09-15T20:52:27.499 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 1 --fsid=ce4892c7-6c6d-4b4f-8e6f-07a3e11133e5 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.52602/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQALSedmFdowAxAA+DNEtL026W4aWluVOkIEwA== --osd-uuid 4e733256-a0ac-49e1-b991-7e6e4a060afc 2024-09-15T20:52:27.528 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:52:27.526+0000 7f77de41c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T20:52:27.572 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:52:27.569+0000 7f77de41c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T20:52:27.572 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:52:27.570+0000 7f77de41c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T20:52:27.573 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:52:27.571+0000 7f77de41c640 -1 bdev(0x558fc5531c00 td/osd-scrub-repair/1/block) open stat got: (1) Operation not permitted 2024-09-15T20:52:27.573 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:52:27.571+0000 7f77de41c640 -1 bluestore(td/osd-scrub-repair/1) _read_fsid unparsable uuid 2024-09-15T20:52:33.823 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-scrub-repair/1/keyring 2024-09-15T20:52:33.824 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-15T20:52:33.824 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd1 key to auth repository 2024-09-15T20:52:33.825 INFO:tasks.workunit.client.0.smithi136.stdout:adding osd1 key to auth repository 2024-09-15T20:52:33.825 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-scrub-repair/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-15T20:52:34.184 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.1 2024-09-15T20:52:34.185 INFO:tasks.workunit.client.0.smithi136.stdout:start osd.1 2024-09-15T20:52:34.185 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 1 --fsid=ce4892c7-6c6d-4b4f-8e6f-07a3e11133e5 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.52602/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T20:52:34.185 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-15T20:52:34.185 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-15T20:52:34.185 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-15T20:52:34.211 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:52:34.208+0000 7fa5b18b6640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T20:52:34.213 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:52:34.210+0000 7fa5b18b6640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T20:52:34.214 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:52:34.211+0000 7fa5b18b6640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T20:52:34.463 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 1 2024-09-15T20:52:34.464 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-15T20:52:34.464 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2024-09-15T20:52:34.464 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-15T20:52:34.464 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-15T20:52:34.464 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:52:34.464 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-15T20:52:34.464 INFO:tasks.workunit.client.0.smithi136.stdout:0 2024-09-15T20:52:34.465 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:52:34.465 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T20:52:34.743 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:52:35.324 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:52:35.322+0000 7fa5b18b6640 -1 Falling back to public interface 2024-09-15T20:52:35.745 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:52:35.745 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:52:35.745 INFO:tasks.workunit.client.0.smithi136.stdout:1 2024-09-15T20:52:35.746 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-15T20:52:35.746 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:52:35.746 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T20:52:36.030 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:52:37.031 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:52:37.031 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:52:37.032 INFO:tasks.workunit.client.0.smithi136.stdout:2 2024-09-15T20:52:37.032 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-15T20:52:37.032 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:52:37.032 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T20:52:37.319 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:52:37.990 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:52:37.988+0000 7fa5b18b6640 -1 osd.1 0 log_to_monitors true 2024-09-15T20:52:38.320 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:52:38.320 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:52:38.320 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-15T20:52:38.321 INFO:tasks.workunit.client.0.smithi136.stdout:3 2024-09-15T20:52:38.321 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:52:38.321 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T20:52:38.688 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:52:39.690 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:52:39.690 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:52:39.690 INFO:tasks.workunit.client.0.smithi136.stdout:4 2024-09-15T20:52:39.690 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-15T20:52:39.690 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:52:39.691 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T20:52:40.008 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:52:41.010 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:52:41.010 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:52:41.010 INFO:tasks.workunit.client.0.smithi136.stdout:5 2024-09-15T20:52:41.011 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-15T20:52:41.011 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:52:41.011 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T20:52:41.298 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:52:42.300 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:52:42.300 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:52:42.300 INFO:tasks.workunit.client.0.smithi136.stdout:6 2024-09-15T20:52:42.301 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-15T20:52:42.301 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:52:42.301 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T20:52:42.617 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:52:43.618 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:52:43.619 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:52:43.619 INFO:tasks.workunit.client.0.smithi136.stdout:7 2024-09-15T20:52:43.619 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-15T20:52:43.620 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:52:43.620 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T20:52:43.989 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:52:44.991 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:52:44.991 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:52:44.991 INFO:tasks.workunit.client.0.smithi136.stdout:8 2024-09-15T20:52:44.992 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-15T20:52:44.992 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:52:44.992 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T20:52:45.281 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:52:46.283 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:52:46.283 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:52:46.283 INFO:tasks.workunit.client.0.smithi136.stdout:9 2024-09-15T20:52:46.284 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-15T20:52:46.284 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:52:46.284 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T20:52:46.611 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:52:47.613 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:52:47.613 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:52:47.613 INFO:tasks.workunit.client.0.smithi136.stdout:10 2024-09-15T20:52:47.614 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-15T20:52:47.614 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:52:47.614 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T20:52:47.940 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:52:48.941 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:52:48.942 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:52:48.942 INFO:tasks.workunit.client.0.smithi136.stdout:11 2024-09-15T20:52:48.942 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-15T20:52:48.942 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:52:48.943 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T20:52:49.294 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:52:50.295 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:52:50.296 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:52:50.296 INFO:tasks.workunit.client.0.smithi136.stdout:12 2024-09-15T20:52:50.296 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-15T20:52:50.296 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:52:50.296 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T20:52:50.606 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:52:51.608 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:52:51.608 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:52:51.608 INFO:tasks.workunit.client.0.smithi136.stdout:13 2024-09-15T20:52:51.609 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-15T20:52:51.609 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:52:51.609 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T20:52:52.119 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:52:53.120 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:52:53.121 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:52:53.121 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-15T20:52:53.121 INFO:tasks.workunit.client.0.smithi136.stdout:14 2024-09-15T20:52:53.121 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:52:53.121 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T20:52:53.987 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:52:54.989 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:52:54.989 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:52:54.989 INFO:tasks.workunit.client.0.smithi136.stdout:15 2024-09-15T20:52:54.989 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-15T20:52:54.990 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:52:54.990 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T20:52:55.731 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:52:56.732 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:52:56.733 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:52:56.733 INFO:tasks.workunit.client.0.smithi136.stdout:16 2024-09-15T20:52:56.733 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-15T20:52:56.733 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:52:56.733 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T20:52:57.525 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:52:58.526 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:52:58.526 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:52:58.527 INFO:tasks.workunit.client.0.smithi136.stdout:17 2024-09-15T20:52:58.527 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-15T20:52:58.527 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:52:58.527 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T20:52:58.814 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:52:59.815 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:52:59.815 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:52:59.815 INFO:tasks.workunit.client.0.smithi136.stdout:18 2024-09-15T20:52:59.816 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 18 2024-09-15T20:52:59.816 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:52:59.816 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T20:53:00.454 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:53:01.455 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:53:01.455 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:53:01.456 INFO:tasks.workunit.client.0.smithi136.stdout:19 2024-09-15T20:53:01.456 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 19 2024-09-15T20:53:01.456 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:53:01.456 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T20:53:01.767 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:53:02.768 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:53:02.768 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:53:02.768 INFO:tasks.workunit.client.0.smithi136.stdout:20 2024-09-15T20:53:02.769 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 20 2024-09-15T20:53:02.769 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:53:02.769 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T20:53:03.093 INFO:tasks.workunit.client.0.smithi136.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/3402758861,v1:127.0.0.1:6811/3402758861] [v2:127.0.0.1:6812/3402758861,v1:127.0.0.1:6813/3402758861] exists,up 4e733256-a0ac-49e1-b991-7e6e4a060afc 2024-09-15T20:53:03.097 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-15T20:53:03.097 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-15T20:53:03.097 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-15T20:53:03.097 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:804: corrupt_and_repair_jerasure: for id in $(seq 0 3) 2024-09-15T20:53:03.097 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:805: corrupt_and_repair_jerasure: run_osd td/osd-scrub-repair 2 2024-09-15T20:53:03.097 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-scrub-repair 2024-09-15T20:53:03.097 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-15T20:53:03.098 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=2 2024-09-15T20:53:03.098 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-15T20:53:03.098 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-scrub-repair/2 2024-09-15T20:53:03.098 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=ce4892c7-6c6d-4b4f-8e6f-07a3e11133e5 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-15T20:53:03.098 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T20:53:03.098 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T20:53:03.098 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T20:53:03.098 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/2' 2024-09-15T20:53:03.098 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/2/journal' 2024-09-15T20:53:03.098 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-15T20:53:03.098 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-15T20:53:03.098 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-15T20:53:03.098 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-15T20:53:03.099 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-15T20:53:03.099 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-15T20:53:03.099 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-15T20:53:03.099 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T20:53:03.099 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52602 2024-09-15T20:53:03.099 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.52602/$cluster-$name.asok' 2024-09-15T20:53:03.099 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.52602/$cluster-$name.asok' 2024-09-15T20:53:03.099 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-15T20:53:03.099 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-15T20:53:03.099 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-15T20:53:03.099 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-15T20:53:03.099 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-15T20:53:03.099 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T20:53:03.099 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T20:53:03.100 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T20:53:03.100 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T20:53:03.100 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-15T20:53:03.100 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-15T20:53:03.100 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-scrub-repair/2 2024-09-15T20:53:03.100 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-15T20:53:03.100 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=f74e59aa-0b73-4bbf-a870-d7c571cf9f22 2024-09-15T20:53:03.100 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd2 f74e59aa-0b73-4bbf-a870-d7c571cf9f22' 2024-09-15T20:53:03.100 INFO:tasks.workunit.client.0.smithi136.stdout:add osd2 f74e59aa-0b73-4bbf-a870-d7c571cf9f22 2024-09-15T20:53:03.101 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-15T20:53:03.115 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQAvSedm5Ta9BhAAVzPwKs6F/QPljCce2eF28Q== 2024-09-15T20:53:03.115 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQAvSedm5Ta9BhAAVzPwKs6F/QPljCce2eF28Q=="}' 2024-09-15T20:53:03.115 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new f74e59aa-0b73-4bbf-a870-d7c571cf9f22 -i td/osd-scrub-repair/2/new.json 2024-09-15T20:53:03.484 INFO:tasks.workunit.client.0.smithi136.stdout:2 2024-09-15T20:53:03.496 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-scrub-repair/2/new.json 2024-09-15T20:53:03.497 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 2 --fsid=ce4892c7-6c6d-4b4f-8e6f-07a3e11133e5 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.52602/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAvSedm5Ta9BhAAVzPwKs6F/QPljCce2eF28Q== --osd-uuid f74e59aa-0b73-4bbf-a870-d7c571cf9f22 2024-09-15T20:53:03.526 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:53:03.523+0000 7f8d009a4640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T20:53:03.568 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:53:03.566+0000 7f8d009a4640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T20:53:03.569 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:53:03.567+0000 7f8d009a4640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T20:53:03.570 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:53:03.567+0000 7f8d009a4640 -1 bdev(0x5601eaf43c00 td/osd-scrub-repair/2/block) open stat got: (1) Operation not permitted 2024-09-15T20:53:03.570 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:53:03.567+0000 7f8d009a4640 -1 bluestore(td/osd-scrub-repair/2) _read_fsid unparsable uuid 2024-09-15T20:53:09.647 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-scrub-repair/2/keyring 2024-09-15T20:53:09.647 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-15T20:53:09.648 INFO:tasks.workunit.client.0.smithi136.stdout:adding osd2 key to auth repository 2024-09-15T20:53:09.648 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd2 key to auth repository 2024-09-15T20:53:09.649 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-scrub-repair/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-15T20:53:10.011 INFO:tasks.workunit.client.0.smithi136.stdout:start osd.2 2024-09-15T20:53:10.011 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.2 2024-09-15T20:53:10.012 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 2 --fsid=ce4892c7-6c6d-4b4f-8e6f-07a3e11133e5 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.52602/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T20:53:10.012 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-15T20:53:10.012 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-15T20:53:10.012 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-15T20:53:10.038 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:53:10.036+0000 7f7049d30640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T20:53:10.040 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:53:10.038+0000 7f7049d30640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T20:53:10.041 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:53:10.039+0000 7f7049d30640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T20:53:10.288 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 2 2024-09-15T20:53:10.288 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-15T20:53:10.288 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=2 2024-09-15T20:53:10.288 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-15T20:53:10.288 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-15T20:53:10.288 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:53:10.288 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-15T20:53:10.289 INFO:tasks.workunit.client.0.smithi136.stdout:0 2024-09-15T20:53:10.289 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:53:10.289 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T20:53:10.572 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:53:11.402 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:53:11.400+0000 7f7049d30640 -1 Falling back to public interface 2024-09-15T20:53:11.573 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:53:11.573 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:53:11.574 INFO:tasks.workunit.client.0.smithi136.stdout:1 2024-09-15T20:53:11.574 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-15T20:53:11.574 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:53:11.574 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T20:53:11.862 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:53:12.864 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:53:12.864 INFO:tasks.workunit.client.0.smithi136.stdout:2 2024-09-15T20:53:12.864 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:53:12.864 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-15T20:53:12.865 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:53:12.865 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T20:53:13.148 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:53:14.150 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:53:14.150 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:53:14.150 INFO:tasks.workunit.client.0.smithi136.stdout:3 2024-09-15T20:53:14.151 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-15T20:53:14.151 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:53:14.151 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T20:53:14.438 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:53:14.652 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:53:14.649+0000 7f7049d30640 -1 osd.2 0 log_to_monitors true 2024-09-15T20:53:15.440 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:53:15.440 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:53:15.440 INFO:tasks.workunit.client.0.smithi136.stdout:4 2024-09-15T20:53:15.441 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-15T20:53:15.441 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:53:15.441 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T20:53:15.772 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:53:16.773 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:53:16.774 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:53:16.774 INFO:tasks.workunit.client.0.smithi136.stdout:5 2024-09-15T20:53:16.774 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-15T20:53:16.774 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:53:16.775 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T20:53:17.163 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:53:18.164 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:53:18.165 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:53:18.165 INFO:tasks.workunit.client.0.smithi136.stdout:6 2024-09-15T20:53:18.165 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-15T20:53:18.165 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:53:18.166 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T20:53:18.449 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:53:19.450 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:53:19.451 INFO:tasks.workunit.client.0.smithi136.stdout:7 2024-09-15T20:53:19.451 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:53:19.451 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-15T20:53:19.451 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:53:19.452 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T20:53:19.730 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:53:20.731 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:53:20.732 INFO:tasks.workunit.client.0.smithi136.stdout:8 2024-09-15T20:53:20.732 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:53:20.732 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-15T20:53:20.732 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:53:20.733 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T20:53:21.034 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:53:22.035 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:53:22.036 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:53:22.036 INFO:tasks.workunit.client.0.smithi136.stdout:9 2024-09-15T20:53:22.036 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-15T20:53:22.036 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:53:22.037 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T20:53:22.370 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:53:23.372 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:53:23.372 INFO:tasks.workunit.client.0.smithi136.stdout:10 2024-09-15T20:53:23.373 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:53:23.373 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-15T20:53:23.373 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:53:23.373 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T20:53:23.710 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:53:24.711 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:53:24.711 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:53:24.711 INFO:tasks.workunit.client.0.smithi136.stdout:11 2024-09-15T20:53:24.712 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-15T20:53:24.712 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:53:24.712 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T20:53:25.047 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:53:26.049 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:53:26.049 INFO:tasks.workunit.client.0.smithi136.stdout:12 2024-09-15T20:53:26.050 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:53:26.050 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-15T20:53:26.050 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:53:26.050 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T20:53:26.393 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:53:27.395 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:53:27.395 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:53:27.395 INFO:tasks.workunit.client.0.smithi136.stdout:13 2024-09-15T20:53:27.395 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-15T20:53:27.395 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:53:27.395 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T20:53:27.920 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:53:28.922 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:53:28.922 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:53:28.922 INFO:tasks.workunit.client.0.smithi136.stdout:14 2024-09-15T20:53:28.923 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-15T20:53:28.923 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:53:28.923 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T20:53:29.696 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:53:30.697 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:53:30.697 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:53:30.698 INFO:tasks.workunit.client.0.smithi136.stdout:15 2024-09-15T20:53:30.698 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-15T20:53:30.698 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:53:30.698 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T20:53:31.530 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:53:32.531 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:53:32.532 INFO:tasks.workunit.client.0.smithi136.stdout:16 2024-09-15T20:53:32.532 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:53:32.532 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-15T20:53:32.533 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:53:32.533 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T20:53:33.315 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:53:34.316 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:53:34.317 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:53:34.317 INFO:tasks.workunit.client.0.smithi136.stdout:17 2024-09-15T20:53:34.317 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-15T20:53:34.317 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:53:34.318 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T20:53:34.919 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:53:35.920 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:53:35.921 INFO:tasks.workunit.client.0.smithi136.stdout:18 2024-09-15T20:53:35.921 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:53:35.921 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 18 2024-09-15T20:53:35.921 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:53:35.921 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T20:53:36.549 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:53:37.550 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:53:37.550 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:53:37.551 INFO:tasks.workunit.client.0.smithi136.stdout:19 2024-09-15T20:53:37.551 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 19 2024-09-15T20:53:37.551 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:53:37.551 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T20:53:37.976 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:53:38.977 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:53:38.978 INFO:tasks.workunit.client.0.smithi136.stdout:20 2024-09-15T20:53:38.978 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:53:38.978 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 20 2024-09-15T20:53:38.978 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:53:38.978 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T20:53:39.267 INFO:tasks.workunit.client.0.smithi136.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/450178718,v1:127.0.0.1:6819/450178718] [v2:127.0.0.1:6820/450178718,v1:127.0.0.1:6821/450178718] exists,up f74e59aa-0b73-4bbf-a870-d7c571cf9f22 2024-09-15T20:53:39.271 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-15T20:53:39.271 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-15T20:53:39.271 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-15T20:53:39.271 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:804: corrupt_and_repair_jerasure: for id in $(seq 0 3) 2024-09-15T20:53:39.271 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:805: corrupt_and_repair_jerasure: run_osd td/osd-scrub-repair 3 2024-09-15T20:53:39.271 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-scrub-repair 2024-09-15T20:53:39.271 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-15T20:53:39.271 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=3 2024-09-15T20:53:39.271 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-15T20:53:39.271 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-scrub-repair/3 2024-09-15T20:53:39.272 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=ce4892c7-6c6d-4b4f-8e6f-07a3e11133e5 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-15T20:53:39.272 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T20:53:39.272 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T20:53:39.272 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T20:53:39.272 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/3' 2024-09-15T20:53:39.272 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/3/journal' 2024-09-15T20:53:39.272 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-15T20:53:39.272 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-15T20:53:39.272 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-15T20:53:39.272 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-15T20:53:39.272 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-15T20:53:39.272 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-15T20:53:39.273 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-15T20:53:39.273 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T20:53:39.273 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52602 2024-09-15T20:53:39.273 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.52602/$cluster-$name.asok' 2024-09-15T20:53:39.273 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.52602/$cluster-$name.asok' 2024-09-15T20:53:39.273 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-15T20:53:39.273 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-15T20:53:39.273 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-15T20:53:39.273 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-15T20:53:39.273 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-15T20:53:39.273 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T20:53:39.273 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T20:53:39.273 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T20:53:39.274 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T20:53:39.274 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-15T20:53:39.274 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-15T20:53:39.274 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-scrub-repair/3 2024-09-15T20:53:39.274 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-15T20:53:39.274 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=b72fdf29-d86a-404d-a216-75e64b9c077b 2024-09-15T20:53:39.274 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd3 b72fdf29-d86a-404d-a216-75e64b9c077b' 2024-09-15T20:53:39.274 INFO:tasks.workunit.client.0.smithi136.stdout:add osd3 b72fdf29-d86a-404d-a216-75e64b9c077b 2024-09-15T20:53:39.274 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-15T20:53:39.289 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQBTSedm/igcERAARI+sk7nONQH18d3hBtOXvg== 2024-09-15T20:53:39.290 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQBTSedm/igcERAARI+sk7nONQH18d3hBtOXvg=="}' 2024-09-15T20:53:39.290 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new b72fdf29-d86a-404d-a216-75e64b9c077b -i td/osd-scrub-repair/3/new.json 2024-09-15T20:53:39.633 INFO:tasks.workunit.client.0.smithi136.stdout:3 2024-09-15T20:53:39.644 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-scrub-repair/3/new.json 2024-09-15T20:53:39.645 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 3 --fsid=ce4892c7-6c6d-4b4f-8e6f-07a3e11133e5 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/3 --osd-journal=td/osd-scrub-repair/3/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.52602/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBTSedm/igcERAARI+sk7nONQH18d3hBtOXvg== --osd-uuid b72fdf29-d86a-404d-a216-75e64b9c077b 2024-09-15T20:53:39.674 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:53:39.672+0000 7fa74745e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T20:53:39.717 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:53:39.715+0000 7fa74745e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T20:53:39.718 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:53:39.716+0000 7fa74745e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T20:53:39.719 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:53:39.716+0000 7fa74745e640 -1 bdev(0x5584315e5c00 td/osd-scrub-repair/3/block) open stat got: (1) Operation not permitted 2024-09-15T20:53:39.719 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:53:39.716+0000 7fa74745e640 -1 bluestore(td/osd-scrub-repair/3) _read_fsid unparsable uuid 2024-09-15T20:53:45.424 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-scrub-repair/3/keyring 2024-09-15T20:53:45.424 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-15T20:53:45.425 INFO:tasks.workunit.client.0.smithi136.stdout:adding osd3 key to auth repository 2024-09-15T20:53:45.425 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd3 key to auth repository 2024-09-15T20:53:45.425 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-scrub-repair/3/keyring auth add osd.3 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-15T20:53:45.782 INFO:tasks.workunit.client.0.smithi136.stdout:start osd.3 2024-09-15T20:53:45.782 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.3 2024-09-15T20:53:45.782 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 3 --fsid=ce4892c7-6c6d-4b4f-8e6f-07a3e11133e5 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/3 --osd-journal=td/osd-scrub-repair/3/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.52602/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T20:53:45.783 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-15T20:53:45.783 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-15T20:53:45.783 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-15T20:53:45.809 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:53:45.807+0000 7f9e19a0f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T20:53:45.811 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:53:45.809+0000 7f9e19a0f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T20:53:45.812 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:53:45.810+0000 7f9e19a0f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T20:53:46.060 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 3 2024-09-15T20:53:46.060 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-15T20:53:46.060 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=3 2024-09-15T20:53:46.060 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-15T20:53:46.060 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-15T20:53:46.060 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:53:46.060 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-15T20:53:46.061 INFO:tasks.workunit.client.0.smithi136.stdout:0 2024-09-15T20:53:46.061 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:53:46.061 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-15T20:53:46.340 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:53:47.341 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:53:47.341 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:53:47.341 INFO:tasks.workunit.client.0.smithi136.stdout:1 2024-09-15T20:53:47.342 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-15T20:53:47.342 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:53:47.342 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-15T20:53:47.431 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:53:47.429+0000 7f9e19a0f640 -1 Falling back to public interface 2024-09-15T20:53:47.629 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:53:48.630 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:53:48.630 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:53:48.630 INFO:tasks.workunit.client.0.smithi136.stdout:2 2024-09-15T20:53:48.631 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-15T20:53:48.631 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:53:48.631 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-15T20:53:48.922 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:53:49.923 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:53:49.924 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:53:49.924 INFO:tasks.workunit.client.0.smithi136.stdout:3 2024-09-15T20:53:49.924 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-15T20:53:49.924 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:53:49.924 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-15T20:53:50.044 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:53:50.041+0000 7f9e19a0f640 -1 osd.3 0 log_to_monitors true 2024-09-15T20:53:50.258 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:53:51.260 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:53:51.260 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:53:51.261 INFO:tasks.workunit.client.0.smithi136.stdout:4 2024-09-15T20:53:51.262 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-15T20:53:51.262 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:53:51.262 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-15T20:53:51.547 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:53:52.548 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:53:52.549 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:53:52.549 INFO:tasks.workunit.client.0.smithi136.stdout:5 2024-09-15T20:53:52.549 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-15T20:53:52.549 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:53:52.550 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-15T20:53:53.005 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:53:54.006 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:53:54.007 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:53:54.007 INFO:tasks.workunit.client.0.smithi136.stdout:6 2024-09-15T20:53:54.007 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-15T20:53:54.007 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:53:54.007 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-15T20:53:54.296 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:53:55.297 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:53:55.297 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:53:55.297 INFO:tasks.workunit.client.0.smithi136.stdout:7 2024-09-15T20:53:55.300 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-15T20:53:55.301 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:53:55.301 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-15T20:53:55.588 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:53:56.589 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:53:56.589 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:53:56.590 INFO:tasks.workunit.client.0.smithi136.stdout:8 2024-09-15T20:53:56.590 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-15T20:53:56.590 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:53:56.590 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-15T20:53:56.977 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:53:57.979 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:53:57.979 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:53:57.979 INFO:tasks.workunit.client.0.smithi136.stdout:9 2024-09-15T20:53:57.980 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-15T20:53:57.980 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:53:57.980 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-15T20:53:58.268 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:53:59.269 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:53:59.269 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:53:59.270 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-15T20:53:59.270 INFO:tasks.workunit.client.0.smithi136.stdout:10 2024-09-15T20:53:59.270 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:53:59.270 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-15T20:53:59.556 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:54:00.557 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:54:00.558 INFO:tasks.workunit.client.0.smithi136.stdout:11 2024-09-15T20:54:00.558 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:54:00.558 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-15T20:54:00.558 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:54:00.558 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-15T20:54:01.028 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:54:02.029 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:54:02.029 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:54:02.030 INFO:tasks.workunit.client.0.smithi136.stdout:12 2024-09-15T20:54:02.030 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-15T20:54:02.030 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:54:02.030 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-15T20:54:02.317 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:54:03.319 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:54:03.319 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:54:03.319 INFO:tasks.workunit.client.0.smithi136.stdout:13 2024-09-15T20:54:03.320 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-15T20:54:03.320 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:54:03.320 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-15T20:54:04.193 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:54:05.194 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:54:05.194 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:54:05.195 INFO:tasks.workunit.client.0.smithi136.stdout:14 2024-09-15T20:54:05.195 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-15T20:54:05.195 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:54:05.195 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-15T20:54:05.905 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:54:06.907 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:54:06.907 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:54:06.907 INFO:tasks.workunit.client.0.smithi136.stdout:15 2024-09-15T20:54:06.908 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-15T20:54:06.908 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:54:06.908 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-15T20:54:07.716 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:54:08.718 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:54:08.718 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:54:08.718 INFO:tasks.workunit.client.0.smithi136.stdout:16 2024-09-15T20:54:08.719 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-15T20:54:08.719 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:54:08.719 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-15T20:54:09.169 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:54:10.170 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:54:10.170 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:54:10.171 INFO:tasks.workunit.client.0.smithi136.stdout:17 2024-09-15T20:54:10.171 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-15T20:54:10.171 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:54:10.171 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-15T20:54:10.815 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:54:11.816 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:54:11.816 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:54:11.817 INFO:tasks.workunit.client.0.smithi136.stdout:18 2024-09-15T20:54:11.817 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 18 2024-09-15T20:54:11.817 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:54:11.817 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-15T20:54:12.695 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:54:13.697 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:54:13.697 INFO:tasks.workunit.client.0.smithi136.stdout:19 2024-09-15T20:54:13.697 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:54:13.698 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 19 2024-09-15T20:54:13.698 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:54:13.698 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-15T20:54:13.983 INFO:tasks.workunit.client.0.smithi136.stdout:osd.3 up in weight 1 up_from 20 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6826/1927468234,v1:127.0.0.1:6827/1927468234] [v2:127.0.0.1:6828/1927468234,v1:127.0.0.1:6829/1927468234] exists,up b72fdf29-d86a-404d-a216-75e64b9c077b 2024-09-15T20:54:13.983 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-15T20:54:13.984 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-15T20:54:13.984 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-15T20:54:13.984 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:807: corrupt_and_repair_jerasure: create_rbd_pool 2024-09-15T20:54:13.984 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:526: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2024-09-15T20:54:14.268 INFO:tasks.workunit.client.0.smithi136.stderr:pool 'rbd' does not exist 2024-09-15T20:54:14.279 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:527: create_rbd_pool: create_pool rbd 4 2024-09-15T20:54:14.279 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create rbd 4 2024-09-15T20:54:14.655 INFO:tasks.workunit.client.0.smithi136.stderr:pool 'rbd' created 2024-09-15T20:54:14.666 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2024-09-15T20:54:15.667 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:528: create_rbd_pool: rbd pool init rbd 2024-09-15T20:54:18.465 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:808: corrupt_and_repair_jerasure: wait_for_clean 2024-09-15T20:54:18.466 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-15T20:54:18.466 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-15T20:54:18.466 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-15T20:54:18.466 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-15T20:54:18.467 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-15T20:54:18.467 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-15T20:54:18.467 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-15T20:54:18.467 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-15T20:54:18.467 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-15T20:54:18.566 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-15T20:54:18.566 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-15T20:54:18.566 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-15T20:54:18.566 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-15T20:54:18.566 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-15T20:54:18.566 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-15T20:54:18.847 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-15T20:54:18.847 INFO:tasks.workunit.client.0.smithi136.stderr:1 2024-09-15T20:54:18.848 INFO:tasks.workunit.client.0.smithi136.stderr:2 2024-09-15T20:54:18.848 INFO:tasks.workunit.client.0.smithi136.stderr:3' 2024-09-15T20:54:18.848 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-15T20:54:18.848 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T20:54:18.848 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-15T20:54:18.952 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836506 2024-09-15T20:54:18.952 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836506 2024-09-15T20:54:18.952 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836506' 2024-09-15T20:54:18.952 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T20:54:18.952 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-15T20:54:19.054 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=42949672979 2024-09-15T20:54:19.055 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 42949672979 2024-09-15T20:54:19.055 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836506 1-42949672979' 2024-09-15T20:54:19.055 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T20:54:19.055 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-15T20:54:19.157 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=64424509454 2024-09-15T20:54:19.157 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 64424509454 2024-09-15T20:54:19.157 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836506 1-42949672979 2-64424509454' 2024-09-15T20:54:19.158 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T20:54:19.158 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-15T20:54:19.260 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=85899345929 2024-09-15T20:54:19.260 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 85899345929 2024-09-15T20:54:19.260 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836506 1-42949672979 2-64424509454 3-85899345929' 2024-09-15T20:54:19.260 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T20:54:19.261 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836506 2024-09-15T20:54:19.261 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T20:54:19.261 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-15T20:54:19.262 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836506 2024-09-15T20:54:19.262 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T20:54:19.263 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836506 2024-09-15T20:54:19.263 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836506' 2024-09-15T20:54:19.264 INFO:tasks.workunit.client.0.smithi136.stdout:waiting osd.0 seq 21474836506 2024-09-15T20:54:19.264 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T20:54:19.541 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836505 -lt 21474836506 2024-09-15T20:54:19.541 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-15T20:54:20.542 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-15T20:54:20.542 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T20:54:20.843 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836506 -lt 21474836506 2024-09-15T20:54:20.843 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T20:54:20.843 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-42949672979 2024-09-15T20:54:20.843 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T20:54:20.844 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-15T20:54:20.845 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-42949672979 2024-09-15T20:54:20.845 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T20:54:20.846 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=42949672979 2024-09-15T20:54:20.846 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 42949672979' 2024-09-15T20:54:20.847 INFO:tasks.workunit.client.0.smithi136.stdout:waiting osd.1 seq 42949672979 2024-09-15T20:54:20.847 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-15T20:54:21.123 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 42949672979 -lt 42949672979 2024-09-15T20:54:21.123 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T20:54:21.124 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-64424509454 2024-09-15T20:54:21.124 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T20:54:21.124 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-15T20:54:21.125 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-64424509454 2024-09-15T20:54:21.125 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T20:54:21.126 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=64424509454 2024-09-15T20:54:21.126 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 64424509454' 2024-09-15T20:54:21.126 INFO:tasks.workunit.client.0.smithi136.stdout:waiting osd.2 seq 64424509454 2024-09-15T20:54:21.127 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-15T20:54:21.402 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 64424509455 -lt 64424509454 2024-09-15T20:54:21.402 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T20:54:21.402 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 3-85899345929 2024-09-15T20:54:21.402 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T20:54:21.403 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=3 2024-09-15T20:54:21.404 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 3-85899345929 2024-09-15T20:54:21.404 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T20:54:21.405 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=85899345929 2024-09-15T20:54:21.405 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.3 seq 85899345929' 2024-09-15T20:54:21.405 INFO:tasks.workunit.client.0.smithi136.stdout:waiting osd.3 seq 85899345929 2024-09-15T20:54:21.405 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-15T20:54:21.681 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 85899345929 -lt 85899345929 2024-09-15T20:54:21.682 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-15T20:54:21.682 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T20:54:21.682 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T20:54:22.029 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 4 == 0 2024-09-15T20:54:22.029 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-15T20:54:22.029 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-15T20:54:22.029 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-15T20:54:22.029 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T20:54:22.029 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T20:54:22.029 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T20:54:22.030 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T20:54:22.302 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=4 2024-09-15T20:54:22.303 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-15T20:54:22.303 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T20:54:22.303 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T20:54:22.647 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 = 4 2024-09-15T20:54:22.647 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-15T20:54:22.647 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-15T20:54:22.647 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:810: corrupt_and_repair_jerasure: create_ec_pool ecpool true k=2 m=2 2024-09-15T20:54:22.647 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2443: create_ec_pool: local pool_name=ecpool 2024-09-15T20:54:22.647 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2444: create_ec_pool: shift 2024-09-15T20:54:22.648 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2445: create_ec_pool: local allow_overwrites=true 2024-09-15T20:54:22.648 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2446: create_ec_pool: shift 2024-09-15T20:54:22.648 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2448: create_ec_pool: ceph osd erasure-code-profile set myprofile crush-failure-domain=osd k=2 m=2 2024-09-15T20:54:23.140 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2450: create_ec_pool: create_pool ecpool 1 1 erasure myprofile 2024-09-15T20:54:23.141 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create ecpool 1 1 erasure myprofile 2024-09-15T20:54:23.731 INFO:tasks.workunit.client.0.smithi136.stderr:pool 'ecpool' created 2024-09-15T20:54:23.743 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2024-09-15T20:54:24.744 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2452: create_ec_pool: '[' true = true ']' 2024-09-15T20:54:24.745 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2453: create_ec_pool: ceph osd pool set ecpool allow_ec_overwrites true 2024-09-15T20:54:25.171 INFO:tasks.workunit.client.0.smithi136.stderr:set pool 2 allow_ec_overwrites to true 2024-09-15T20:54:25.183 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2456: create_ec_pool: wait_for_clean 2024-09-15T20:54:25.183 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-15T20:54:25.183 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-15T20:54:25.183 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-15T20:54:25.183 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-15T20:54:25.184 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-15T20:54:25.184 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-15T20:54:25.184 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-15T20:54:25.184 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-15T20:54:25.184 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-15T20:54:25.274 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-15T20:54:25.274 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-15T20:54:25.274 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-15T20:54:25.274 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-15T20:54:25.274 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-15T20:54:25.274 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-15T20:54:25.561 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-15T20:54:25.561 INFO:tasks.workunit.client.0.smithi136.stderr:1 2024-09-15T20:54:25.561 INFO:tasks.workunit.client.0.smithi136.stderr:2 2024-09-15T20:54:25.561 INFO:tasks.workunit.client.0.smithi136.stderr:3' 2024-09-15T20:54:25.561 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-15T20:54:25.561 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T20:54:25.561 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-15T20:54:25.685 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836508 2024-09-15T20:54:25.685 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836508 2024-09-15T20:54:25.685 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836508' 2024-09-15T20:54:25.685 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T20:54:25.685 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-15T20:54:25.788 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=42949672981 2024-09-15T20:54:25.789 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 42949672981 2024-09-15T20:54:25.789 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836508 1-42949672981' 2024-09-15T20:54:25.789 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T20:54:25.789 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-15T20:54:25.927 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=64424509457 2024-09-15T20:54:25.927 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 64424509457 2024-09-15T20:54:25.927 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836508 1-42949672981 2-64424509457' 2024-09-15T20:54:25.927 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T20:54:25.927 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-15T20:54:26.029 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=85899345931 2024-09-15T20:54:26.030 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 85899345931 2024-09-15T20:54:26.030 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836508 1-42949672981 2-64424509457 3-85899345931' 2024-09-15T20:54:26.030 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T20:54:26.030 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836508 2024-09-15T20:54:26.030 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T20:54:26.031 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-15T20:54:26.032 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836508 2024-09-15T20:54:26.032 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T20:54:26.033 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836508 2024-09-15T20:54:26.033 INFO:tasks.workunit.client.0.smithi136.stdout:waiting osd.0 seq 21474836508 2024-09-15T20:54:26.033 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836508' 2024-09-15T20:54:26.033 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T20:54:26.309 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836507 -lt 21474836508 2024-09-15T20:54:26.309 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-15T20:54:27.310 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-15T20:54:27.310 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T20:54:27.595 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836508 -lt 21474836508 2024-09-15T20:54:27.595 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T20:54:27.596 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-42949672981 2024-09-15T20:54:27.596 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T20:54:27.597 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-15T20:54:27.598 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-42949672981 2024-09-15T20:54:27.598 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T20:54:27.598 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=42949672981 2024-09-15T20:54:27.598 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 42949672981' 2024-09-15T20:54:27.599 INFO:tasks.workunit.client.0.smithi136.stdout:waiting osd.1 seq 42949672981 2024-09-15T20:54:27.599 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-15T20:54:27.877 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 42949672981 -lt 42949672981 2024-09-15T20:54:27.877 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T20:54:27.878 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-64424509457 2024-09-15T20:54:27.878 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T20:54:27.879 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-15T20:54:27.879 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-64424509457 2024-09-15T20:54:27.879 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T20:54:27.880 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=64424509457 2024-09-15T20:54:27.880 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 64424509457' 2024-09-15T20:54:27.881 INFO:tasks.workunit.client.0.smithi136.stdout:waiting osd.2 seq 64424509457 2024-09-15T20:54:27.881 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-15T20:54:28.157 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 64424509457 -lt 64424509457 2024-09-15T20:54:28.157 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T20:54:28.157 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 3-85899345931 2024-09-15T20:54:28.157 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T20:54:28.158 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=3 2024-09-15T20:54:28.159 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 3-85899345931 2024-09-15T20:54:28.159 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T20:54:28.160 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=85899345931 2024-09-15T20:54:28.160 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.3 seq 85899345931' 2024-09-15T20:54:28.160 INFO:tasks.workunit.client.0.smithi136.stdout:waiting osd.3 seq 85899345931 2024-09-15T20:54:28.160 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-15T20:54:28.438 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 85899345931 -lt 85899345931 2024-09-15T20:54:28.438 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-15T20:54:28.438 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T20:54:28.438 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T20:54:28.793 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 5 == 0 2024-09-15T20:54:28.793 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-15T20:54:28.793 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-15T20:54:28.793 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-15T20:54:28.793 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T20:54:28.793 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T20:54:28.794 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T20:54:28.794 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T20:54:29.095 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=5 2024-09-15T20:54:29.096 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-15T20:54:29.096 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T20:54:29.096 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T20:54:29.438 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 = 5 2024-09-15T20:54:29.438 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-15T20:54:29.438 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-15T20:54:29.438 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2457: create_ec_pool: return 0 2024-09-15T20:54:29.438 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:811: corrupt_and_repair_jerasure: corrupt_and_repair_erasure_coded td/osd-scrub-repair ecpool 2024-09-15T20:54:29.438 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:248: corrupt_and_repair_erasure_coded: local dir=td/osd-scrub-repair 2024-09-15T20:54:29.439 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:249: corrupt_and_repair_erasure_coded: local poolname=ecpool 2024-09-15T20:54:29.439 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:251: corrupt_and_repair_erasure_coded: add_something td/osd-scrub-repair ecpool 2024-09-15T20:54:29.439 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-09-15T20:54:29.439 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=ecpool 2024-09-15T20:54:29.439 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=SOMETHING 2024-09-15T20:54:29.439 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-09-15T20:54:29.439 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-09-15T20:54:29.439 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-09-15T20:54:29.807 INFO:tasks.workunit.client.0.smithi136.stderr:noscrub is set 2024-09-15T20:54:29.818 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-09-15T20:54:30.234 INFO:tasks.workunit.client.0.smithi136.stderr:nodeep-scrub is set 2024-09-15T20:54:30.246 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-09-15T20:54:30.246 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-09-15T20:54:30.246 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool ecpool put SOMETHING td/osd-scrub-repair/ORIGINAL 2024-09-15T20:54:30.727 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:253: corrupt_and_repair_erasure_coded: get_primary ecpool SOMETHING 2024-09-15T20:54:30.727 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1187: get_primary: local poolname=ecpool 2024-09-15T20:54:30.727 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1188: get_primary: local objectname=SOMETHING 2024-09-15T20:54:30.728 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1190: get_primary: ceph --format json osd map ecpool SOMETHING 2024-09-15T20:54:30.728 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1191: get_primary: jq .acting_primary 2024-09-15T20:54:31.062 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:253: corrupt_and_repair_erasure_coded: local primary=3 2024-09-15T20:54:31.062 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:254: corrupt_and_repair_erasure_coded: get_osds ecpool SOMETHING 2024-09-15T20:54:31.062 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:254: corrupt_and_repair_erasure_coded: sed -e s/3// 2024-09-15T20:54:31.063 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1012: get_osds: local poolname=ecpool 2024-09-15T20:54:31.063 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1013: get_osds: local objectname=SOMETHING 2024-09-15T20:54:31.063 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: ceph --format json osd map ecpool SOMETHING 2024-09-15T20:54:31.063 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: jq '.acting | .[]' 2024-09-15T20:54:31.339 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: local 'osds=3 2024-09-15T20:54:31.339 INFO:tasks.workunit.client.0.smithi136.stderr:1 2024-09-15T20:54:31.339 INFO:tasks.workunit.client.0.smithi136.stderr:2 2024-09-15T20:54:31.339 INFO:tasks.workunit.client.0.smithi136.stderr:0' 2024-09-15T20:54:31.339 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1018: get_osds: echo 3 1 2 0 2024-09-15T20:54:31.342 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:254: corrupt_and_repair_erasure_coded: osds=('1' '2' '0') 2024-09-15T20:54:31.342 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:254: corrupt_and_repair_erasure_coded: local -a osds 2024-09-15T20:54:31.342 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:255: corrupt_and_repair_erasure_coded: local not_primary_first=1 2024-09-15T20:54:31.342 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:256: corrupt_and_repair_erasure_coded: local not_primary_second=2 2024-09-15T20:54:31.342 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:259: corrupt_and_repair_erasure_coded: corrupt_and_repair_one td/osd-scrub-repair ecpool 3 2024-09-15T20:54:31.342 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:226: corrupt_and_repair_one: local dir=td/osd-scrub-repair 2024-09-15T20:54:31.342 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:227: corrupt_and_repair_one: local poolname=ecpool 2024-09-15T20:54:31.342 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:228: corrupt_and_repair_one: local osd=3 2024-09-15T20:54:31.342 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:233: corrupt_and_repair_one: objectstore_tool td/osd-scrub-repair 3 SOMETHING remove 2024-09-15T20:54:31.342 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2024-09-15T20:54:31.343 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2024-09-15T20:54:31.343 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=3 2024-09-15T20:54:31.343 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2024-09-15T20:54:31.343 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 3 SOMETHING remove 2024-09-15T20:54:31.343 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-09-15T20:54:31.343 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2024-09-15T20:54:31.343 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=3 2024-09-15T20:54:31.343 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2024-09-15T20:54:31.343 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.3 2024-09-15T20:54:31.343 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-15T20:54:31.343 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-15T20:54:31.343 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-15T20:54:31.343 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-15T20:54:31.344 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-15T20:54:32.649 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-15T20:54:32.649 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 3 SOMETHING remove 2024-09-15T20:54:32.649 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-15T20:54:32.649 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-15T20:54:32.649 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=3 2024-09-15T20:54:32.649 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-15T20:54:32.649 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/3 2024-09-15T20:54:32.649 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/3 SOMETHING remove 2024-09-15T20:54:33.997 INFO:tasks.workunit.client.0.smithi136.stdout:remove 0#2:eb822e21:::SOMETHING:head# 2024-09-15T20:54:34.516 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 3 2024-09-15T20:54:34.516 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2024-09-15T20:54:34.517 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-15T20:54:34.517 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=3 2024-09-15T20:54:34.517 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-15T20:54:34.517 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/3 2024-09-15T20:54:34.517 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=ce4892c7-6c6d-4b4f-8e6f-07a3e11133e5 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-15T20:54:34.517 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T20:54:34.517 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T20:54:34.517 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T20:54:34.517 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/3' 2024-09-15T20:54:34.517 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/3/journal' 2024-09-15T20:54:34.517 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-15T20:54:34.518 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-15T20:54:34.518 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-15T20:54:34.518 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-15T20:54:34.518 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-15T20:54:34.518 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-15T20:54:34.518 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-15T20:54:34.518 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T20:54:34.518 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52602 2024-09-15T20:54:34.518 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.52602/$cluster-$name.asok' 2024-09-15T20:54:34.518 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.52602/$cluster-$name.asok' 2024-09-15T20:54:34.518 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-15T20:54:34.518 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-15T20:54:34.519 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-15T20:54:34.519 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T20:54:34.519 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T20:54:34.519 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T20:54:34.519 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T20:54:34.519 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-15T20:54:34.519 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-15T20:54:34.519 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/3 2024-09-15T20:54:34.519 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.3 2024-09-15T20:54:34.519 INFO:tasks.workunit.client.0.smithi136.stderr:start osd.3 2024-09-15T20:54:34.519 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 3 --fsid=ce4892c7-6c6d-4b4f-8e6f-07a3e11133e5 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/3 --osd-journal=td/osd-scrub-repair/3/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.52602/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T20:54:34.520 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/3/whoami 2024-09-15T20:54:34.520 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 3 = 3 ']' 2024-09-15T20:54:34.520 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-15T20:54:34.521 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-15T20:54:34.521 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-15T20:54:34.546 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:54:34.544+0000 7f78b0ba4640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T20:54:34.549 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:54:34.546+0000 7f78b0ba4640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T20:54:34.550 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:54:34.547+0000 7f78b0ba4640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T20:54:34.807 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 3 2024-09-15T20:54:34.807 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-15T20:54:34.807 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=3 2024-09-15T20:54:34.807 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-15T20:54:34.807 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-15T20:54:34.807 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:54:34.807 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-15T20:54:34.807 INFO:tasks.workunit.client.0.smithi136.stderr:0 2024-09-15T20:54:34.807 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:54:34.808 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-15T20:54:35.126 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:54:35.901 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:54:35.899+0000 7f78b0ba4640 -1 Falling back to public interface 2024-09-15T20:54:36.128 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:54:36.128 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:54:36.128 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-15T20:54:36.128 INFO:tasks.workunit.client.0.smithi136.stderr:1 2024-09-15T20:54:36.128 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:54:36.128 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-15T20:54:36.414 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:54:37.416 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:54:37.416 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:54:37.416 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-15T20:54:37.416 INFO:tasks.workunit.client.0.smithi136.stderr:2 2024-09-15T20:54:37.416 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:54:37.416 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-15T20:54:37.703 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:54:38.566 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:54:38.564+0000 7f78b0ba4640 -1 osd.3 39 log_to_monitors true 2024-09-15T20:54:38.705 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:54:38.705 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:54:38.705 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-15T20:54:38.705 INFO:tasks.workunit.client.0.smithi136.stderr:3 2024-09-15T20:54:38.705 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:54:38.705 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-15T20:54:39.009 INFO:tasks.workunit.client.0.smithi136.stderr:osd.3 up in weight 1 up_from 43 up_thru 34 down_at 40 last_clean_interval [20,39) [v2:127.0.0.1:6826/3163839928,v1:127.0.0.1:6827/3163839928] [v2:127.0.0.1:6828/3163839928,v1:127.0.0.1:6829/3163839928] exists,up b72fdf29-d86a-404d-a216-75e64b9c077b 2024-09-15T20:54:39.009 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-15T20:54:39.009 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-15T20:54:39.009 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-15T20:54:39.009 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2024-09-15T20:54:39.009 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-15T20:54:39.009 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-15T20:54:39.009 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-15T20:54:39.010 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-15T20:54:39.010 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-15T20:54:39.010 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-15T20:54:39.010 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-15T20:54:39.010 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-15T20:54:39.010 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-15T20:54:39.099 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-15T20:54:39.100 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-15T20:54:39.100 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-15T20:54:39.100 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-15T20:54:39.100 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-15T20:54:39.100 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-15T20:54:39.494 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-15T20:54:39.494 INFO:tasks.workunit.client.0.smithi136.stderr:1 2024-09-15T20:54:39.494 INFO:tasks.workunit.client.0.smithi136.stderr:2 2024-09-15T20:54:39.494 INFO:tasks.workunit.client.0.smithi136.stderr:3' 2024-09-15T20:54:39.494 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-15T20:54:39.494 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T20:54:39.494 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-15T20:54:39.789 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836512 2024-09-15T20:54:39.790 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836512 2024-09-15T20:54:39.790 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836512' 2024-09-15T20:54:39.790 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T20:54:39.790 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-15T20:54:39.945 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=42949672985 2024-09-15T20:54:39.945 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 42949672985 2024-09-15T20:54:39.945 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836512 1-42949672985' 2024-09-15T20:54:39.945 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T20:54:39.945 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-15T20:54:40.181 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=64424509461 2024-09-15T20:54:40.181 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 64424509461 2024-09-15T20:54:40.181 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836512 1-42949672985 2-64424509461' 2024-09-15T20:54:40.182 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T20:54:40.182 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-15T20:54:40.285 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=184683593730 2024-09-15T20:54:40.286 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 184683593730 2024-09-15T20:54:40.286 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836512 1-42949672985 2-64424509461 3-184683593730' 2024-09-15T20:54:40.286 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T20:54:40.286 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836512 2024-09-15T20:54:40.286 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T20:54:40.287 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-15T20:54:40.287 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836512 2024-09-15T20:54:40.288 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T20:54:40.288 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836512 2024-09-15T20:54:40.288 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836512' 2024-09-15T20:54:40.289 INFO:tasks.workunit.client.0.smithi136.stderr:waiting osd.0 seq 21474836512 2024-09-15T20:54:40.289 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T20:54:40.679 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836511 -lt 21474836512 2024-09-15T20:54:40.679 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-15T20:54:41.680 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-15T20:54:41.681 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T20:54:41.966 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836512 -lt 21474836512 2024-09-15T20:54:41.966 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T20:54:41.966 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-42949672985 2024-09-15T20:54:41.967 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T20:54:41.967 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-15T20:54:41.968 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-42949672985 2024-09-15T20:54:41.968 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T20:54:41.969 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=42949672985 2024-09-15T20:54:41.969 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 42949672985' 2024-09-15T20:54:41.969 INFO:tasks.workunit.client.0.smithi136.stderr:waiting osd.1 seq 42949672985 2024-09-15T20:54:41.969 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-15T20:54:42.245 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 42949672985 -lt 42949672985 2024-09-15T20:54:42.245 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T20:54:42.245 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-64424509461 2024-09-15T20:54:42.246 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T20:54:42.246 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-15T20:54:42.247 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-64424509461 2024-09-15T20:54:42.247 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T20:54:42.248 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=64424509461 2024-09-15T20:54:42.248 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 64424509461' 2024-09-15T20:54:42.248 INFO:tasks.workunit.client.0.smithi136.stderr:waiting osd.2 seq 64424509461 2024-09-15T20:54:42.248 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-15T20:54:42.524 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 64424509461 -lt 64424509461 2024-09-15T20:54:42.524 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T20:54:42.525 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 3-184683593730 2024-09-15T20:54:42.525 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T20:54:42.525 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=3 2024-09-15T20:54:42.526 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 3-184683593730 2024-09-15T20:54:42.526 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T20:54:42.527 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=184683593730 2024-09-15T20:54:42.527 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.3 seq 184683593730' 2024-09-15T20:54:42.527 INFO:tasks.workunit.client.0.smithi136.stderr:waiting osd.3 seq 184683593730 2024-09-15T20:54:42.528 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-15T20:54:42.821 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 184683593730 -lt 184683593730 2024-09-15T20:54:42.821 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-15T20:54:42.821 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T20:54:42.821 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T20:54:43.168 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 5 == 0 2024-09-15T20:54:43.168 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-15T20:54:43.168 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-15T20:54:43.168 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-15T20:54:43.168 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T20:54:43.168 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T20:54:43.169 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T20:54:43.169 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T20:54:43.439 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=5 2024-09-15T20:54:43.440 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-15T20:54:43.440 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T20:54:43.440 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T20:54:43.784 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 = 5 2024-09-15T20:54:43.784 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-15T20:54:43.784 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-15T20:54:43.784 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:237: corrupt_and_repair_one: get_pg ecpool SOMETHING 2024-09-15T20:54:43.784 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1084: get_pg: local poolname=ecpool 2024-09-15T20:54:43.784 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1085: get_pg: local objectname=SOMETHING 2024-09-15T20:54:43.784 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1087: get_pg: ceph --format json osd map ecpool SOMETHING 2024-09-15T20:54:43.784 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1087: get_pg: jq -r .pgid 2024-09-15T20:54:44.090 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:237: corrupt_and_repair_one: local pg=2.0 2024-09-15T20:54:44.090 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:238: corrupt_and_repair_one: repair 2.0 2024-09-15T20:54:44.090 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1846: repair: local pgid=2.0 2024-09-15T20:54:44.090 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1847: repair: get_last_scrub_stamp 2.0 2024-09-15T20:54:44.090 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2024-09-15T20:54:44.090 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-15T20:54:44.090 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-15T20:54:44.090 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-09-15T20:54:44.369 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1847: repair: local last_scrub=2024-09-15T20:54:23.611852+0000 2024-09-15T20:54:44.369 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1848: repair: ceph pg repair 2.0 2024-09-15T20:54:44.628 INFO:tasks.workunit.client.0.smithi136.stderr:instructing pg 2.0s0 on osd.3 to repair 2024-09-15T20:54:44.640 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1849: repair: wait_for_scrub 2.0 2024-09-15T20:54:23.611852+0000 2024-09-15T20:54:44.640 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: local pgid=2.0 2024-09-15T20:54:44.640 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2018: wait_for_scrub: local last_scrub=2024-09-15T20:54:23.611852+0000 2024-09-15T20:54:44.640 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2019: wait_for_scrub: local sname=last_scrub_stamp 2024-09-15T20:54:44.640 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i=0 )) 2024-09-15T20:54:44.640 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-15T20:54:44.640 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-09-15T20:54:44.641 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2024-09-15T20:54:44.641 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-15T20:54:44.641 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-15T20:54:44.641 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-09-15T20:54:44.914 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-15T20:54:23.611852+0000 '>' 2024-09-15T20:54:23.611852+0000 2024-09-15T20:54:44.914 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-15T20:54:45.915 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-15T20:54:45.915 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-15T20:54:45.916 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-09-15T20:54:45.916 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2024-09-15T20:54:45.916 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-15T20:54:45.916 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-15T20:54:45.916 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-09-15T20:54:46.195 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-15T20:54:23.611852+0000 '>' 2024-09-15T20:54:23.611852+0000 2024-09-15T20:54:46.195 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-15T20:54:47.197 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-15T20:54:47.197 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-15T20:54:47.198 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-09-15T20:54:47.198 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2024-09-15T20:54:47.198 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-15T20:54:47.198 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-15T20:54:47.198 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-09-15T20:54:47.478 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-15T20:54:23.611852+0000 '>' 2024-09-15T20:54:23.611852+0000 2024-09-15T20:54:47.478 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-15T20:54:48.479 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-15T20:54:48.479 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-15T20:54:48.480 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-09-15T20:54:48.480 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2024-09-15T20:54:48.480 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-15T20:54:48.480 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-15T20:54:48.481 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-09-15T20:54:48.762 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-15T20:54:23.611852+0000 '>' 2024-09-15T20:54:23.611852+0000 2024-09-15T20:54:48.762 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-15T20:54:49.763 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-15T20:54:49.763 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-15T20:54:49.764 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-09-15T20:54:49.764 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2024-09-15T20:54:49.764 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-15T20:54:49.764 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-15T20:54:49.765 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-09-15T20:54:50.074 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-15T20:54:23.611852+0000 '>' 2024-09-15T20:54:23.611852+0000 2024-09-15T20:54:50.075 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-15T20:54:51.076 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-15T20:54:51.076 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-15T20:54:51.077 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-09-15T20:54:51.077 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2024-09-15T20:54:51.077 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-15T20:54:51.077 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-15T20:54:51.077 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-09-15T20:54:51.360 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-15T20:54:45.471700+0000 '>' 2024-09-15T20:54:23.611852+0000 2024-09-15T20:54:51.360 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2023: wait_for_scrub: return 0 2024-09-15T20:54:51.360 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1850: repair: sleep 2 2024-09-15T20:54:53.362 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:242: corrupt_and_repair_one: objectstore_tool td/osd-scrub-repair 3 SOMETHING list-attrs 2024-09-15T20:54:53.362 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2024-09-15T20:54:53.362 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2024-09-15T20:54:53.362 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=3 2024-09-15T20:54:53.363 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2024-09-15T20:54:53.363 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 3 SOMETHING list-attrs 2024-09-15T20:54:53.363 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-09-15T20:54:53.363 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2024-09-15T20:54:53.363 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=3 2024-09-15T20:54:53.363 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2024-09-15T20:54:53.363 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.3 2024-09-15T20:54:53.363 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-15T20:54:53.363 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-15T20:54:53.363 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-15T20:54:53.363 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-15T20:54:53.363 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-15T20:54:53.671 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-15T20:54:53.671 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 3 SOMETHING list-attrs 2024-09-15T20:54:53.671 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-15T20:54:53.671 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-15T20:54:53.671 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=3 2024-09-15T20:54:53.671 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-15T20:54:53.671 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/3 2024-09-15T20:54:53.671 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/3 SOMETHING list-attrs 2024-09-15T20:54:54.201 INFO:tasks.workunit.client.0.smithi136.stderr:Error getting attr on : 2.0s0_head,0#-4:00000000:::scrub_2.0s0:head#, (61) No data available 2024-09-15T20:54:54.201 INFO:tasks.workunit.client.0.smithi136.stdout:_ 2024-09-15T20:54:54.201 INFO:tasks.workunit.client.0.smithi136.stdout:hinfo_key 2024-09-15T20:54:54.201 INFO:tasks.workunit.client.0.smithi136.stdout:snapset 2024-09-15T20:54:54.483 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 3 2024-09-15T20:54:54.483 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2024-09-15T20:54:54.483 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-15T20:54:54.484 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=3 2024-09-15T20:54:54.484 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-15T20:54:54.484 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/3 2024-09-15T20:54:54.484 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=ce4892c7-6c6d-4b4f-8e6f-07a3e11133e5 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-15T20:54:54.485 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T20:54:54.485 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T20:54:54.485 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T20:54:54.486 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/3' 2024-09-15T20:54:54.486 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/3/journal' 2024-09-15T20:54:54.486 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-15T20:54:54.486 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-15T20:54:54.486 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-15T20:54:54.486 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-15T20:54:54.486 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-15T20:54:54.486 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-15T20:54:54.486 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-15T20:54:54.486 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T20:54:54.486 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52602 2024-09-15T20:54:54.486 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.52602/$cluster-$name.asok' 2024-09-15T20:54:54.486 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.52602/$cluster-$name.asok' 2024-09-15T20:54:54.487 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-15T20:54:54.487 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-15T20:54:54.487 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-15T20:54:54.487 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T20:54:54.487 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T20:54:54.487 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T20:54:54.487 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T20:54:54.487 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-15T20:54:54.487 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-15T20:54:54.487 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/3 2024-09-15T20:54:54.487 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.3 2024-09-15T20:54:54.487 INFO:tasks.workunit.client.0.smithi136.stderr:start osd.3 2024-09-15T20:54:54.488 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 3 --fsid=ce4892c7-6c6d-4b4f-8e6f-07a3e11133e5 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/3 --osd-journal=td/osd-scrub-repair/3/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.52602/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T20:54:54.488 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/3/whoami 2024-09-15T20:54:54.488 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 3 = 3 ']' 2024-09-15T20:54:54.488 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-15T20:54:54.488 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-15T20:54:54.488 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-15T20:54:54.506 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:54:54.502+0000 7f3470e15640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T20:54:54.620 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:54:54.618+0000 7f3470e15640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T20:54:54.622 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:54:54.619+0000 7f3470e15640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T20:54:54.950 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 3 2024-09-15T20:54:54.950 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-15T20:54:54.950 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=3 2024-09-15T20:54:54.950 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-15T20:54:54.951 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-15T20:54:54.951 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:54:54.951 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-15T20:54:54.951 INFO:tasks.workunit.client.0.smithi136.stderr:0 2024-09-15T20:54:54.951 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:54:54.951 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-15T20:54:55.258 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:54:55.724 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:54:55.722+0000 7f3470e15640 -1 Falling back to public interface 2024-09-15T20:54:56.259 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:54:56.259 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:54:56.260 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-15T20:54:56.260 INFO:tasks.workunit.client.0.smithi136.stderr:1 2024-09-15T20:54:56.260 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:54:56.260 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-15T20:54:56.536 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:54:57.537 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:54:57.537 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:54:57.537 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-15T20:54:57.538 INFO:tasks.workunit.client.0.smithi136.stderr:2 2024-09-15T20:54:57.538 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:54:57.538 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-15T20:54:57.816 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:54:58.818 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:54:58.818 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:54:58.818 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-15T20:54:58.818 INFO:tasks.workunit.client.0.smithi136.stderr:3 2024-09-15T20:54:58.818 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:54:58.818 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-15T20:54:58.983 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:54:58.980+0000 7f3470e15640 -1 osd.3 45 log_to_monitors true 2024-09-15T20:54:59.107 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:55:00.109 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:55:00.110 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:55:00.110 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-15T20:55:00.110 INFO:tasks.workunit.client.0.smithi136.stderr:4 2024-09-15T20:55:00.110 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:55:00.110 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-15T20:55:00.581 INFO:tasks.workunit.client.0.smithi136.stderr:osd.3 up in weight 1 up_from 50 up_thru 50 down_at 46 last_clean_interval [43,45) [v2:127.0.0.1:6826/1779420035,v1:127.0.0.1:6827/1779420035] [v2:127.0.0.1:6828/1779420035,v1:127.0.0.1:6829/1779420035] exists,up b72fdf29-d86a-404d-a216-75e64b9c077b 2024-09-15T20:55:00.581 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-15T20:55:00.581 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-15T20:55:00.581 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-15T20:55:00.581 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2024-09-15T20:55:00.581 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-15T20:55:00.581 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-15T20:55:00.581 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-15T20:55:00.581 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-15T20:55:00.582 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-15T20:55:00.582 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-15T20:55:00.582 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-15T20:55:00.582 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-15T20:55:00.583 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-15T20:55:00.671 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-15T20:55:00.671 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-15T20:55:00.671 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-15T20:55:00.671 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-15T20:55:00.671 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-15T20:55:00.672 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-15T20:55:00.988 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-15T20:55:00.988 INFO:tasks.workunit.client.0.smithi136.stderr:1 2024-09-15T20:55:00.988 INFO:tasks.workunit.client.0.smithi136.stderr:2 2024-09-15T20:55:00.988 INFO:tasks.workunit.client.0.smithi136.stderr:3' 2024-09-15T20:55:00.988 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-15T20:55:00.988 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T20:55:00.988 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-15T20:55:01.091 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836517 2024-09-15T20:55:01.092 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836517 2024-09-15T20:55:01.092 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836517' 2024-09-15T20:55:01.092 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T20:55:01.092 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-15T20:55:01.196 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=42949672990 2024-09-15T20:55:01.196 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 42949672990 2024-09-15T20:55:01.196 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836517 1-42949672990' 2024-09-15T20:55:01.196 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T20:55:01.196 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-15T20:55:01.298 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=64424509466 2024-09-15T20:55:01.298 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 64424509466 2024-09-15T20:55:01.298 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836517 1-42949672990 2-64424509466' 2024-09-15T20:55:01.298 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T20:55:01.298 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-15T20:55:01.400 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=214748364802 2024-09-15T20:55:01.400 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 214748364802 2024-09-15T20:55:01.400 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836517 1-42949672990 2-64424509466 3-214748364802' 2024-09-15T20:55:01.400 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T20:55:01.401 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836517 2024-09-15T20:55:01.401 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T20:55:01.401 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-15T20:55:01.402 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836517 2024-09-15T20:55:01.402 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T20:55:01.403 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836517 2024-09-15T20:55:01.403 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836517' 2024-09-15T20:55:01.403 INFO:tasks.workunit.client.0.smithi136.stderr:waiting osd.0 seq 21474836517 2024-09-15T20:55:01.403 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T20:55:01.677 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836516 -lt 21474836517 2024-09-15T20:55:01.678 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-15T20:55:02.680 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-15T20:55:02.680 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T20:55:02.976 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836518 -lt 21474836517 2024-09-15T20:55:02.976 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T20:55:02.976 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-42949672990 2024-09-15T20:55:02.976 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T20:55:02.977 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-15T20:55:02.978 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-42949672990 2024-09-15T20:55:02.978 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T20:55:02.979 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=42949672990 2024-09-15T20:55:02.980 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 42949672990' 2024-09-15T20:55:02.980 INFO:tasks.workunit.client.0.smithi136.stderr:waiting osd.1 seq 42949672990 2024-09-15T20:55:02.980 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-15T20:55:03.254 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 42949672991 -lt 42949672990 2024-09-15T20:55:03.254 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T20:55:03.254 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-64424509466 2024-09-15T20:55:03.254 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T20:55:03.255 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-15T20:55:03.256 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-64424509466 2024-09-15T20:55:03.256 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T20:55:03.257 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=64424509466 2024-09-15T20:55:03.257 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 64424509466' 2024-09-15T20:55:03.257 INFO:tasks.workunit.client.0.smithi136.stderr:waiting osd.2 seq 64424509466 2024-09-15T20:55:03.257 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-15T20:55:03.533 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 64424509466 -lt 64424509466 2024-09-15T20:55:03.533 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T20:55:03.533 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 3-214748364802 2024-09-15T20:55:03.534 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T20:55:03.534 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=3 2024-09-15T20:55:03.535 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 3-214748364802 2024-09-15T20:55:03.535 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T20:55:03.536 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=214748364802 2024-09-15T20:55:03.536 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.3 seq 214748364802' 2024-09-15T20:55:03.536 INFO:tasks.workunit.client.0.smithi136.stderr:waiting osd.3 seq 214748364802 2024-09-15T20:55:03.536 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-15T20:55:03.813 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 214748364802 -lt 214748364802 2024-09-15T20:55:03.813 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-15T20:55:03.814 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T20:55:03.814 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T20:55:04.157 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 5 == 0 2024-09-15T20:55:04.157 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-15T20:55:04.157 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-15T20:55:04.157 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-15T20:55:04.157 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T20:55:04.157 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T20:55:04.158 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T20:55:04.158 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T20:55:04.429 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=5 2024-09-15T20:55:04.429 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-15T20:55:04.430 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T20:55:04.430 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T20:55:04.800 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 = 5 2024-09-15T20:55:04.800 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-15T20:55:04.800 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-15T20:55:04.800 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:243: corrupt_and_repair_one: rados --pool ecpool get SOMETHING td/osd-scrub-repair/COPY 2024-09-15T20:55:04.872 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:244: corrupt_and_repair_one: diff td/osd-scrub-repair/ORIGINAL td/osd-scrub-repair/COPY 2024-09-15T20:55:04.874 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:261: corrupt_and_repair_erasure_coded: corrupt_and_repair_one td/osd-scrub-repair ecpool 1 2024-09-15T20:55:04.874 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:226: corrupt_and_repair_one: local dir=td/osd-scrub-repair 2024-09-15T20:55:04.874 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:227: corrupt_and_repair_one: local poolname=ecpool 2024-09-15T20:55:04.874 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:228: corrupt_and_repair_one: local osd=1 2024-09-15T20:55:04.874 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:233: corrupt_and_repair_one: objectstore_tool td/osd-scrub-repair 1 SOMETHING remove 2024-09-15T20:55:04.874 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2024-09-15T20:55:04.874 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2024-09-15T20:55:04.874 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=1 2024-09-15T20:55:04.875 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2024-09-15T20:55:04.875 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 SOMETHING remove 2024-09-15T20:55:04.875 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-09-15T20:55:04.875 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2024-09-15T20:55:04.875 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=1 2024-09-15T20:55:04.875 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2024-09-15T20:55:04.875 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-09-15T20:55:04.875 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-15T20:55:04.875 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-15T20:55:04.875 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-15T20:55:04.875 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-15T20:55:04.875 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-15T20:55:05.180 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-15T20:55:05.180 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 SOMETHING remove 2024-09-15T20:55:05.180 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-15T20:55:05.180 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-15T20:55:05.181 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=1 2024-09-15T20:55:05.181 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-15T20:55:05.181 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-09-15T20:55:05.181 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 SOMETHING remove 2024-09-15T20:55:07.078 INFO:tasks.workunit.client.0.smithi136.stdout:remove 1#2:eb822e21:::SOMETHING:head# 2024-09-15T20:55:07.595 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2024-09-15T20:55:07.596 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2024-09-15T20:55:07.596 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-15T20:55:07.596 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=1 2024-09-15T20:55:07.596 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-15T20:55:07.596 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-09-15T20:55:07.596 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=ce4892c7-6c6d-4b4f-8e6f-07a3e11133e5 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-15T20:55:07.596 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T20:55:07.596 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T20:55:07.596 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T20:55:07.596 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-09-15T20:55:07.596 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-09-15T20:55:07.596 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-15T20:55:07.597 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-15T20:55:07.597 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-15T20:55:07.597 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-15T20:55:07.597 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-15T20:55:07.597 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-15T20:55:07.597 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-15T20:55:07.597 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T20:55:07.597 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52602 2024-09-15T20:55:07.597 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.52602/$cluster-$name.asok' 2024-09-15T20:55:07.597 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.52602/$cluster-$name.asok' 2024-09-15T20:55:07.597 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-15T20:55:07.597 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-15T20:55:07.597 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-15T20:55:07.598 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T20:55:07.598 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T20:55:07.598 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T20:55:07.598 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T20:55:07.598 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-15T20:55:07.598 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-15T20:55:07.598 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-09-15T20:55:07.599 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.1 2024-09-15T20:55:07.599 INFO:tasks.workunit.client.0.smithi136.stderr:start osd.1 2024-09-15T20:55:07.599 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 1 --fsid=ce4892c7-6c6d-4b4f-8e6f-07a3e11133e5 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.52602/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T20:55:07.599 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-09-15T20:55:07.599 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 1 = 1 ']' 2024-09-15T20:55:07.599 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-15T20:55:07.599 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-15T20:55:07.599 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-15T20:55:07.627 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:55:07.624+0000 7f96b1bfd640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T20:55:07.629 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:55:07.626+0000 7f96b1bfd640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T20:55:07.630 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:55:07.627+0000 7f96b1bfd640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T20:55:07.882 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 1 2024-09-15T20:55:07.883 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-15T20:55:07.883 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2024-09-15T20:55:07.883 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-15T20:55:07.883 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-15T20:55:07.883 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:55:07.883 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-15T20:55:07.883 INFO:tasks.workunit.client.0.smithi136.stderr:0 2024-09-15T20:55:07.883 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:55:07.883 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T20:55:08.159 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:55:09.000 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:55:08.997+0000 7f96b1bfd640 -1 Falling back to public interface 2024-09-15T20:55:09.161 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:55:09.161 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:55:09.161 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-15T20:55:09.161 INFO:tasks.workunit.client.0.smithi136.stderr:1 2024-09-15T20:55:09.161 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:55:09.161 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T20:55:09.443 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:55:10.444 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:55:10.444 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:55:10.444 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-15T20:55:10.444 INFO:tasks.workunit.client.0.smithi136.stderr:2 2024-09-15T20:55:10.444 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:55:10.445 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T20:55:10.733 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:55:11.657 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:55:11.654+0000 7f96b1bfd640 -1 osd.1 52 log_to_monitors true 2024-09-15T20:55:11.734 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:55:11.735 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:55:11.735 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-15T20:55:11.735 INFO:tasks.workunit.client.0.smithi136.stderr:3 2024-09-15T20:55:11.735 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:55:11.735 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T20:55:12.128 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:55:13.133 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:55:13.133 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:55:13.133 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-15T20:55:13.134 INFO:tasks.workunit.client.0.smithi136.stderr:4 2024-09-15T20:55:13.134 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:55:13.134 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T20:55:13.416 INFO:tasks.workunit.client.0.smithi136.stderr:osd.1 up in weight 1 up_from 57 up_thru 57 down_at 53 last_clean_interval [10,52) [v2:127.0.0.1:6810/3354814746,v1:127.0.0.1:6811/3354814746] [v2:127.0.0.1:6812/3354814746,v1:127.0.0.1:6813/3354814746] exists,up 4e733256-a0ac-49e1-b991-7e6e4a060afc 2024-09-15T20:55:13.416 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-15T20:55:13.416 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-15T20:55:13.416 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-15T20:55:13.416 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2024-09-15T20:55:13.416 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-15T20:55:13.416 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-15T20:55:13.416 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-15T20:55:13.416 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-15T20:55:13.416 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-15T20:55:13.417 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-15T20:55:13.417 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-15T20:55:13.417 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-15T20:55:13.417 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-15T20:55:13.506 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-15T20:55:13.506 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-15T20:55:13.506 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-15T20:55:13.506 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-15T20:55:13.506 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-15T20:55:13.506 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-15T20:55:13.898 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-15T20:55:13.898 INFO:tasks.workunit.client.0.smithi136.stderr:1 2024-09-15T20:55:13.898 INFO:tasks.workunit.client.0.smithi136.stderr:2 2024-09-15T20:55:13.898 INFO:tasks.workunit.client.0.smithi136.stderr:3' 2024-09-15T20:55:13.898 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-15T20:55:13.898 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T20:55:13.898 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-15T20:55:14.001 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836521 2024-09-15T20:55:14.001 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836521 2024-09-15T20:55:14.001 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836521' 2024-09-15T20:55:14.001 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T20:55:14.001 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-15T20:55:14.104 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=244813135874 2024-09-15T20:55:14.105 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 244813135874 2024-09-15T20:55:14.105 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836521 1-244813135874' 2024-09-15T20:55:14.105 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T20:55:14.105 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-15T20:55:14.207 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=64424509469 2024-09-15T20:55:14.207 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 64424509469 2024-09-15T20:55:14.208 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836521 1-244813135874 2-64424509469' 2024-09-15T20:55:14.208 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T20:55:14.208 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-15T20:55:14.312 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=214748364806 2024-09-15T20:55:14.312 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 214748364806 2024-09-15T20:55:14.312 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836521 1-244813135874 2-64424509469 3-214748364806' 2024-09-15T20:55:14.312 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T20:55:14.312 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836521 2024-09-15T20:55:14.312 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T20:55:14.313 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-15T20:55:14.313 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836521 2024-09-15T20:55:14.313 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T20:55:14.314 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836521 2024-09-15T20:55:14.314 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836521' 2024-09-15T20:55:14.314 INFO:tasks.workunit.client.0.smithi136.stderr:waiting osd.0 seq 21474836521 2024-09-15T20:55:14.315 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T20:55:14.586 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836520 -lt 21474836521 2024-09-15T20:55:14.587 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-15T20:55:15.588 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-15T20:55:15.588 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T20:55:15.869 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836521 -lt 21474836521 2024-09-15T20:55:15.869 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T20:55:15.869 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-244813135874 2024-09-15T20:55:15.869 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T20:55:15.870 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-15T20:55:15.871 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-244813135874 2024-09-15T20:55:15.871 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T20:55:15.872 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=244813135874 2024-09-15T20:55:15.872 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 244813135874' 2024-09-15T20:55:15.872 INFO:tasks.workunit.client.0.smithi136.stderr:waiting osd.1 seq 244813135874 2024-09-15T20:55:15.872 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-15T20:55:16.148 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 244813135874 -lt 244813135874 2024-09-15T20:55:16.148 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T20:55:16.149 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-64424509469 2024-09-15T20:55:16.149 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T20:55:16.150 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-15T20:55:16.150 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-64424509469 2024-09-15T20:55:16.151 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T20:55:16.152 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=64424509469 2024-09-15T20:55:16.152 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 64424509469' 2024-09-15T20:55:16.152 INFO:tasks.workunit.client.0.smithi136.stderr:waiting osd.2 seq 64424509469 2024-09-15T20:55:16.152 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-15T20:55:16.429 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 64424509470 -lt 64424509469 2024-09-15T20:55:16.429 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T20:55:16.430 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 3-214748364806 2024-09-15T20:55:16.430 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T20:55:16.430 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=3 2024-09-15T20:55:16.431 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 3-214748364806 2024-09-15T20:55:16.431 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T20:55:16.432 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=214748364806 2024-09-15T20:55:16.432 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.3 seq 214748364806' 2024-09-15T20:55:16.432 INFO:tasks.workunit.client.0.smithi136.stderr:waiting osd.3 seq 214748364806 2024-09-15T20:55:16.433 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-15T20:55:16.711 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 214748364806 -lt 214748364806 2024-09-15T20:55:16.712 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-15T20:55:16.712 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T20:55:16.712 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T20:55:17.060 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 5 == 0 2024-09-15T20:55:17.060 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-15T20:55:17.060 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-15T20:55:17.060 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-15T20:55:17.060 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T20:55:17.060 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T20:55:17.061 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T20:55:17.061 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T20:55:17.333 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=5 2024-09-15T20:55:17.333 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-15T20:55:17.333 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T20:55:17.334 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T20:55:17.675 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 = 5 2024-09-15T20:55:17.676 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-15T20:55:17.676 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-15T20:55:17.676 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:237: corrupt_and_repair_one: get_pg ecpool SOMETHING 2024-09-15T20:55:17.676 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1084: get_pg: local poolname=ecpool 2024-09-15T20:55:17.676 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1085: get_pg: local objectname=SOMETHING 2024-09-15T20:55:17.676 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1087: get_pg: ceph --format json osd map ecpool SOMETHING 2024-09-15T20:55:17.676 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1087: get_pg: jq -r .pgid 2024-09-15T20:55:17.965 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:237: corrupt_and_repair_one: local pg=2.0 2024-09-15T20:55:17.965 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:238: corrupt_and_repair_one: repair 2.0 2024-09-15T20:55:17.965 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1846: repair: local pgid=2.0 2024-09-15T20:55:17.965 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1847: repair: get_last_scrub_stamp 2.0 2024-09-15T20:55:17.966 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2024-09-15T20:55:17.966 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-15T20:55:17.966 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-15T20:55:17.966 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-09-15T20:55:18.240 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1847: repair: local last_scrub=2024-09-15T20:54:45.471700+0000 2024-09-15T20:55:18.240 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1848: repair: ceph pg repair 2.0 2024-09-15T20:55:18.499 INFO:tasks.workunit.client.0.smithi136.stderr:instructing pg 2.0s0 on osd.3 to repair 2024-09-15T20:55:18.511 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1849: repair: wait_for_scrub 2.0 2024-09-15T20:54:45.471700+0000 2024-09-15T20:55:18.511 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: local pgid=2.0 2024-09-15T20:55:18.512 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2018: wait_for_scrub: local last_scrub=2024-09-15T20:54:45.471700+0000 2024-09-15T20:55:18.512 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2019: wait_for_scrub: local sname=last_scrub_stamp 2024-09-15T20:55:18.512 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i=0 )) 2024-09-15T20:55:18.512 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-15T20:55:18.512 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-09-15T20:55:18.512 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2024-09-15T20:55:18.512 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-15T20:55:18.512 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-15T20:55:18.512 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-09-15T20:55:18.785 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-15T20:54:45.471700+0000 '>' 2024-09-15T20:54:45.471700+0000 2024-09-15T20:55:18.785 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-15T20:55:19.787 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-15T20:55:19.787 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-15T20:55:19.787 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-09-15T20:55:19.787 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2024-09-15T20:55:19.788 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-15T20:55:19.788 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-15T20:55:19.788 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-09-15T20:55:20.112 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-15T20:54:45.471700+0000 '>' 2024-09-15T20:54:45.471700+0000 2024-09-15T20:55:20.112 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-15T20:55:21.113 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-15T20:55:21.114 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-15T20:55:21.114 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-09-15T20:55:21.115 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2024-09-15T20:55:21.115 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-15T20:55:21.115 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-15T20:55:21.115 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-09-15T20:55:21.397 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-15T20:55:19.155903+0000 '>' 2024-09-15T20:54:45.471700+0000 2024-09-15T20:55:21.397 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2023: wait_for_scrub: return 0 2024-09-15T20:55:21.397 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1850: repair: sleep 2 2024-09-15T20:55:23.399 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:242: corrupt_and_repair_one: objectstore_tool td/osd-scrub-repair 3 SOMETHING list-attrs 2024-09-15T20:55:23.399 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2024-09-15T20:55:23.400 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2024-09-15T20:55:23.400 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=3 2024-09-15T20:55:23.400 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2024-09-15T20:55:23.400 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 3 SOMETHING list-attrs 2024-09-15T20:55:23.400 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-09-15T20:55:23.400 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2024-09-15T20:55:23.400 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=3 2024-09-15T20:55:23.400 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2024-09-15T20:55:23.400 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.3 2024-09-15T20:55:23.400 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-15T20:55:23.400 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-15T20:55:23.400 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-15T20:55:23.400 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-15T20:55:23.401 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-15T20:55:23.708 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-15T20:55:23.708 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 3 SOMETHING list-attrs 2024-09-15T20:55:23.708 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-15T20:55:23.708 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-15T20:55:23.708 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=3 2024-09-15T20:55:23.708 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-15T20:55:23.708 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/3 2024-09-15T20:55:23.709 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/3 SOMETHING list-attrs 2024-09-15T20:55:24.228 INFO:tasks.workunit.client.0.smithi136.stderr:Error getting attr on : 2.0s0_head,0#-4:00000000:::scrub_2.0s0:head#, (61) No data available 2024-09-15T20:55:24.228 INFO:tasks.workunit.client.0.smithi136.stdout:_ 2024-09-15T20:55:24.229 INFO:tasks.workunit.client.0.smithi136.stdout:hinfo_key 2024-09-15T20:55:24.229 INFO:tasks.workunit.client.0.smithi136.stdout:snapset 2024-09-15T20:55:24.503 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 3 2024-09-15T20:55:24.503 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2024-09-15T20:55:24.503 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-15T20:55:24.503 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=3 2024-09-15T20:55:24.503 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-15T20:55:24.504 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/3 2024-09-15T20:55:24.504 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=ce4892c7-6c6d-4b4f-8e6f-07a3e11133e5 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-15T20:55:24.504 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T20:55:24.504 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T20:55:24.504 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T20:55:24.505 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/3' 2024-09-15T20:55:24.505 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/3/journal' 2024-09-15T20:55:24.505 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-15T20:55:24.505 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-15T20:55:24.505 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-15T20:55:24.505 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-15T20:55:24.506 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-15T20:55:24.506 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-15T20:55:24.506 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-15T20:55:24.506 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T20:55:24.506 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52602 2024-09-15T20:55:24.507 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.52602/$cluster-$name.asok' 2024-09-15T20:55:24.507 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.52602/$cluster-$name.asok' 2024-09-15T20:55:24.507 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-15T20:55:24.507 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-15T20:55:24.507 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-15T20:55:24.507 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T20:55:24.508 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T20:55:24.508 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T20:55:24.508 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T20:55:24.508 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-15T20:55:24.508 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-15T20:55:24.508 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/3 2024-09-15T20:55:24.509 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.3 2024-09-15T20:55:24.509 INFO:tasks.workunit.client.0.smithi136.stderr:start osd.3 2024-09-15T20:55:24.509 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 3 --fsid=ce4892c7-6c6d-4b4f-8e6f-07a3e11133e5 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/3 --osd-journal=td/osd-scrub-repair/3/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.52602/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T20:55:24.509 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/3/whoami 2024-09-15T20:55:24.509 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 3 = 3 ']' 2024-09-15T20:55:24.510 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-15T20:55:24.510 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-15T20:55:24.510 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-15T20:55:24.529 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:55:24.527+0000 7fed30644640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T20:55:24.532 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:55:24.529+0000 7fed30644640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T20:55:24.534 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:55:24.531+0000 7fed30644640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T20:55:24.989 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 3 2024-09-15T20:55:24.989 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-15T20:55:24.989 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=3 2024-09-15T20:55:24.989 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-15T20:55:24.989 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-15T20:55:24.989 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:55:24.989 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-15T20:55:24.989 INFO:tasks.workunit.client.0.smithi136.stderr:0 2024-09-15T20:55:24.989 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:55:24.990 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-15T20:55:25.290 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:55:26.122 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:55:26.120+0000 7fed30644640 -1 Falling back to public interface 2024-09-15T20:55:26.291 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:55:26.292 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:55:26.292 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-15T20:55:26.292 INFO:tasks.workunit.client.0.smithi136.stderr:1 2024-09-15T20:55:26.292 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:55:26.292 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-15T20:55:26.581 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:55:27.583 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:55:27.583 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:55:27.583 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-15T20:55:27.583 INFO:tasks.workunit.client.0.smithi136.stderr:2 2024-09-15T20:55:27.583 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:55:27.583 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-15T20:55:27.858 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:55:28.860 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:55:28.860 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:55:28.860 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-15T20:55:28.860 INFO:tasks.workunit.client.0.smithi136.stderr:3 2024-09-15T20:55:28.860 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:55:28.860 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-15T20:55:29.140 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:55:29.565 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:55:29.562+0000 7fed30644640 -1 osd.3 59 log_to_monitors true 2024-09-15T20:55:30.141 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:55:30.141 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:55:30.141 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-15T20:55:30.141 INFO:tasks.workunit.client.0.smithi136.stderr:4 2024-09-15T20:55:30.142 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:55:30.142 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-15T20:55:30.758 INFO:tasks.workunit.client.0.smithi136.stderr:osd.3 up in weight 1 up_from 64 up_thru 64 down_at 60 last_clean_interval [50,59) [v2:127.0.0.1:6826/3630482946,v1:127.0.0.1:6827/3630482946] [v2:127.0.0.1:6828/3630482946,v1:127.0.0.1:6829/3630482946] exists,up b72fdf29-d86a-404d-a216-75e64b9c077b 2024-09-15T20:55:30.758 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-15T20:55:30.758 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-15T20:55:30.759 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-15T20:55:30.759 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2024-09-15T20:55:30.759 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-15T20:55:30.759 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-15T20:55:30.759 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-15T20:55:30.759 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-15T20:55:30.759 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-15T20:55:30.759 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-15T20:55:30.759 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-15T20:55:30.759 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-15T20:55:30.759 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-15T20:55:30.845 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-15T20:55:30.845 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-15T20:55:30.846 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-15T20:55:30.846 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-15T20:55:30.846 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-15T20:55:30.846 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-15T20:55:31.369 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-15T20:55:31.369 INFO:tasks.workunit.client.0.smithi136.stderr:1 2024-09-15T20:55:31.369 INFO:tasks.workunit.client.0.smithi136.stderr:2 2024-09-15T20:55:31.370 INFO:tasks.workunit.client.0.smithi136.stderr:3' 2024-09-15T20:55:31.370 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-15T20:55:31.370 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T20:55:31.370 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-15T20:55:31.515 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836525 2024-09-15T20:55:31.515 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836525 2024-09-15T20:55:31.515 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836525' 2024-09-15T20:55:31.515 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T20:55:31.516 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-15T20:55:31.658 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=244813135878 2024-09-15T20:55:31.658 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 244813135878 2024-09-15T20:55:31.658 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836525 1-244813135878' 2024-09-15T20:55:31.658 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T20:55:31.658 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-15T20:55:31.759 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=64424509474 2024-09-15T20:55:31.759 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 64424509474 2024-09-15T20:55:31.759 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836525 1-244813135878 2-64424509474' 2024-09-15T20:55:31.759 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T20:55:31.759 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-15T20:55:31.862 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=274877906946 2024-09-15T20:55:31.862 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 274877906946 2024-09-15T20:55:31.862 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836525 1-244813135878 2-64424509474 3-274877906946' 2024-09-15T20:55:31.862 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T20:55:31.863 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836525 2024-09-15T20:55:31.863 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T20:55:31.863 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-15T20:55:31.864 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836525 2024-09-15T20:55:31.864 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T20:55:31.865 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836525 2024-09-15T20:55:31.865 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836525' 2024-09-15T20:55:31.865 INFO:tasks.workunit.client.0.smithi136.stderr:waiting osd.0 seq 21474836525 2024-09-15T20:55:31.865 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T20:55:32.140 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836524 -lt 21474836525 2024-09-15T20:55:32.140 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-15T20:55:33.141 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-15T20:55:33.141 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T20:55:33.423 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836526 -lt 21474836525 2024-09-15T20:55:33.423 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T20:55:33.423 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-244813135878 2024-09-15T20:55:33.424 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T20:55:33.424 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-15T20:55:33.425 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-244813135878 2024-09-15T20:55:33.425 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T20:55:33.426 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=244813135878 2024-09-15T20:55:33.426 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 244813135878' 2024-09-15T20:55:33.426 INFO:tasks.workunit.client.0.smithi136.stderr:waiting osd.1 seq 244813135878 2024-09-15T20:55:33.426 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-15T20:55:33.703 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 244813135879 -lt 244813135878 2024-09-15T20:55:33.704 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T20:55:33.704 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-64424509474 2024-09-15T20:55:33.704 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T20:55:33.705 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-15T20:55:33.706 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-64424509474 2024-09-15T20:55:33.706 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T20:55:33.707 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=64424509474 2024-09-15T20:55:33.707 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 64424509474' 2024-09-15T20:55:33.707 INFO:tasks.workunit.client.0.smithi136.stderr:waiting osd.2 seq 64424509474 2024-09-15T20:55:33.707 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-15T20:55:33.982 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 64424509474 -lt 64424509474 2024-09-15T20:55:33.983 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T20:55:33.983 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 3-274877906946 2024-09-15T20:55:33.983 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T20:55:33.984 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=3 2024-09-15T20:55:33.985 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 3-274877906946 2024-09-15T20:55:33.985 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T20:55:33.986 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=274877906946 2024-09-15T20:55:33.986 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.3 seq 274877906946' 2024-09-15T20:55:33.986 INFO:tasks.workunit.client.0.smithi136.stderr:waiting osd.3 seq 274877906946 2024-09-15T20:55:33.986 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-15T20:55:34.261 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 274877906946 -lt 274877906946 2024-09-15T20:55:34.262 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-15T20:55:34.262 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T20:55:34.262 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T20:55:34.609 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 5 == 0 2024-09-15T20:55:34.609 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-15T20:55:34.609 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-15T20:55:34.609 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-15T20:55:34.609 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T20:55:34.609 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T20:55:34.609 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T20:55:34.609 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T20:55:34.882 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=5 2024-09-15T20:55:34.882 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-15T20:55:34.883 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T20:55:34.883 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T20:55:35.228 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 = 5 2024-09-15T20:55:35.228 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-15T20:55:35.229 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-15T20:55:35.229 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:243: corrupt_and_repair_one: rados --pool ecpool get SOMETHING td/osd-scrub-repair/COPY 2024-09-15T20:55:35.273 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:244: corrupt_and_repair_one: diff td/osd-scrub-repair/ORIGINAL td/osd-scrub-repair/COPY 2024-09-15T20:55:35.276 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:262: corrupt_and_repair_erasure_coded: corrupt_and_repair_two td/osd-scrub-repair ecpool 1 2 2024-09-15T20:55:35.276 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:185: corrupt_and_repair_two: local dir=td/osd-scrub-repair 2024-09-15T20:55:35.277 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:186: corrupt_and_repair_two: local poolname=ecpool 2024-09-15T20:55:35.277 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:187: corrupt_and_repair_two: local first=1 2024-09-15T20:55:35.277 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:188: corrupt_and_repair_two: local second=2 2024-09-15T20:55:35.277 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:193: corrupt_and_repair_two: pids= 2024-09-15T20:55:35.277 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:194: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 1 SOMETHING remove 2024-09-15T20:55:35.277 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2135: run_in_background: local pid_variable=pids 2024-09-15T20:55:35.277 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2136: run_in_background: shift 2024-09-15T20:55:35.277 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2140: run_in_background: eval 'pids+=" 174631"' 2024-09-15T20:55:35.277 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2140: run_in_background: pids+=' 174631' 2024-09-15T20:55:35.277 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:195: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 2 SOMETHING remove 2024-09-15T20:55:35.277 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2135: run_in_background: local pid_variable=pids 2024-09-15T20:55:35.277 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2136: run_in_background: shift 2024-09-15T20:55:35.277 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2140: run_in_background: eval 'pids+=" 174633"' 2024-09-15T20:55:35.278 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: run_in_background: objectstore_tool td/osd-scrub-repair 1 SOMETHING remove 2024-09-15T20:55:35.278 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2140: run_in_background: pids+=' 174633' 2024-09-15T20:55:35.278 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:196: corrupt_and_repair_two: wait_background pids 2024-09-15T20:55:35.278 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: wait_background: pids=' 174631 174633' 2024-09-15T20:55:35.278 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: wait_background: return_code=0 2024-09-15T20:55:35.278 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: wait_background: for pid in $pids 2024-09-15T20:55:35.278 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: run_in_background: sed 's/^/174634: /' 2024-09-15T20:55:35.278 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2175: wait_background: wait 174631 2024-09-15T20:55:35.278 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: run_in_background: objectstore_tool td/osd-scrub-repair 2 SOMETHING remove 2024-09-15T20:55:35.278 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: run_in_background: sed 's/^/174636: /' 2024-09-15T20:55:38.449 INFO:tasks.workunit.client.0.smithi136.stderr:174634: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2024-09-15T20:55:38.450 INFO:tasks.workunit.client.0.smithi136.stderr:174634: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2024-09-15T20:55:38.450 INFO:tasks.workunit.client.0.smithi136.stderr:174634: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=1 2024-09-15T20:55:38.450 INFO:tasks.workunit.client.0.smithi136.stderr:174634: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2024-09-15T20:55:38.450 INFO:tasks.workunit.client.0.smithi136.stderr:174634: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 SOMETHING remove 2024-09-15T20:55:38.450 INFO:tasks.workunit.client.0.smithi136.stderr:174634: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-09-15T20:55:38.450 INFO:tasks.workunit.client.0.smithi136.stderr:174634: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2024-09-15T20:55:38.450 INFO:tasks.workunit.client.0.smithi136.stderr:174634: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=1 2024-09-15T20:55:38.450 INFO:tasks.workunit.client.0.smithi136.stderr:174634: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2024-09-15T20:55:38.450 INFO:tasks.workunit.client.0.smithi136.stderr:174634: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-09-15T20:55:38.450 INFO:tasks.workunit.client.0.smithi136.stderr:174634: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-15T20:55:38.450 INFO:tasks.workunit.client.0.smithi136.stderr:174634: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-15T20:55:38.450 INFO:tasks.workunit.client.0.smithi136.stderr:174634: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-15T20:55:38.451 INFO:tasks.workunit.client.0.smithi136.stderr:174634: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-15T20:55:38.451 INFO:tasks.workunit.client.0.smithi136.stderr:174634: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-15T20:55:38.451 INFO:tasks.workunit.client.0.smithi136.stderr:174634: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-15T20:55:38.451 INFO:tasks.workunit.client.0.smithi136.stderr:174634: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 SOMETHING remove 2024-09-15T20:55:38.451 INFO:tasks.workunit.client.0.smithi136.stderr:174634: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-15T20:55:38.451 INFO:tasks.workunit.client.0.smithi136.stderr:174634: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-15T20:55:38.451 INFO:tasks.workunit.client.0.smithi136.stderr:174634: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=1 2024-09-15T20:55:38.451 INFO:tasks.workunit.client.0.smithi136.stderr:174634: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-15T20:55:38.451 INFO:tasks.workunit.client.0.smithi136.stderr:174634: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-09-15T20:55:38.451 INFO:tasks.workunit.client.0.smithi136.stderr:174634: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 SOMETHING remove 2024-09-15T20:55:38.451 INFO:tasks.workunit.client.0.smithi136.stderr:174634: remove 1#2:eb822e21:::SOMETHING:head# 2024-09-15T20:55:38.451 INFO:tasks.workunit.client.0.smithi136.stderr:174634: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2024-09-15T20:55:38.451 INFO:tasks.workunit.client.0.smithi136.stderr:174634: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2024-09-15T20:55:38.451 INFO:tasks.workunit.client.0.smithi136.stderr:174634: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-15T20:55:38.452 INFO:tasks.workunit.client.0.smithi136.stderr:174634: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=1 2024-09-15T20:55:38.452 INFO:tasks.workunit.client.0.smithi136.stderr:174634: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-15T20:55:38.452 INFO:tasks.workunit.client.0.smithi136.stderr:174634: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-09-15T20:55:38.452 INFO:tasks.workunit.client.0.smithi136.stderr:174634: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=ce4892c7-6c6d-4b4f-8e6f-07a3e11133e5 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-15T20:55:38.452 INFO:tasks.workunit.client.0.smithi136.stderr:174634: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T20:55:38.453 INFO:tasks.workunit.client.0.smithi136.stderr:174634: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T20:55:38.453 INFO:tasks.workunit.client.0.smithi136.stderr:174634: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T20:55:38.453 INFO:tasks.workunit.client.0.smithi136.stderr:174634: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-09-15T20:55:38.453 INFO:tasks.workunit.client.0.smithi136.stderr:174634: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-09-15T20:55:38.453 INFO:tasks.workunit.client.0.smithi136.stderr:174634: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-15T20:55:38.453 INFO:tasks.workunit.client.0.smithi136.stderr:174634: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-15T20:55:38.453 INFO:tasks.workunit.client.0.smithi136.stderr:174634: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-15T20:55:38.453 INFO:tasks.workunit.client.0.smithi136.stderr:174634: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-15T20:55:38.453 INFO:tasks.workunit.client.0.smithi136.stderr:174634: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-15T20:55:38.454 INFO:tasks.workunit.client.0.smithi136.stderr:174634: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-15T20:55:38.454 INFO:tasks.workunit.client.0.smithi136.stderr:174634: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-15T20:55:38.454 INFO:tasks.workunit.client.0.smithi136.stderr:174634: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T20:55:38.454 INFO:tasks.workunit.client.0.smithi136.stderr:174634: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52602 2024-09-15T20:55:38.454 INFO:tasks.workunit.client.0.smithi136.stderr:174634: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.52602/$cluster-$name.asok' 2024-09-15T20:55:38.454 INFO:tasks.workunit.client.0.smithi136.stderr:174634: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.52602/$cluster-$name.asok' 2024-09-15T20:55:38.454 INFO:tasks.workunit.client.0.smithi136.stderr:174634: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-15T20:55:38.454 INFO:tasks.workunit.client.0.smithi136.stderr:174634: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-15T20:55:38.454 INFO:tasks.workunit.client.0.smithi136.stderr:174634: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-15T20:55:38.454 INFO:tasks.workunit.client.0.smithi136.stderr:174634: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T20:55:38.454 INFO:tasks.workunit.client.0.smithi136.stderr:174634: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T20:55:38.454 INFO:tasks.workunit.client.0.smithi136.stderr:174634: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T20:55:38.454 INFO:tasks.workunit.client.0.smithi136.stderr:174634: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T20:55:38.454 INFO:tasks.workunit.client.0.smithi136.stderr:174634: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-15T20:55:38.455 INFO:tasks.workunit.client.0.smithi136.stderr:174634: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-15T20:55:38.455 INFO:tasks.workunit.client.0.smithi136.stderr:174634: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-09-15T20:55:38.455 INFO:tasks.workunit.client.0.smithi136.stderr:174634: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.1 2024-09-15T20:55:38.455 INFO:tasks.workunit.client.0.smithi136.stderr:174634: start osd.1 2024-09-15T20:55:38.455 INFO:tasks.workunit.client.0.smithi136.stderr:174634: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 1 --fsid=ce4892c7-6c6d-4b4f-8e6f-07a3e11133e5 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.52602/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T20:55:38.628 INFO:tasks.workunit.client.0.smithi136.stderr:1174636: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2024-09-15T20:55:38.628 INFO:tasks.workunit.client.0.smithi136.stderr:174636: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2024-09-15T20:55:38.628 INFO:tasks.workunit.client.0.smithi136.stderr:174636: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=2 2024-09-15T20:55:38.628 INFO:tasks.workunit.client.0.smithi136.stderr:174636: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2024-09-15T20:55:38.629 INFO:tasks.workunit.client.0.smithi136.stderr:174636: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 2 SOMETHING remove 2024-09-15T20:55:38.629 INFO:tasks.workunit.client.0.smithi136.stderr:174636: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-09-15T20:55:38.629 INFO:tasks.workunit.client.0.smithi136.stderr:174636: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2024-09-15T20:55:38.629 INFO:tasks.workunit.client.0.smithi136.stderr:174636: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=2 2024-09-15T20:55:38.629 INFO:tasks.workunit.client.0.smithi136.stderr:174636: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2024-09-15T20:55:38.629 INFO:tasks.workunit.client.0.smithi136.stderr:174636: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.2 2024-09-15T20:55:38.629 INFO:tasks.workunit.client.0.smithi136.stderr:174636: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-15T20:55:38.629 INFO:tasks.workunit.client.0.smithi136.stderr:174636: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-15T20:55:38.629 INFO:tasks.workunit.client.0.smithi136.stderr:174636: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-15T20:55:38.629 INFO:tasks.workunit.client.0.smithi136.stderr:174636: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-15T20:55:38.629 INFO:tasks.workunit.client.0.smithi136.stderr:174636: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-15T20:55:38.629 INFO:tasks.workunit.client.0.smithi136.stderr:174636: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-15T20:55:38.629 INFO:tasks.workunit.client.0.smithi136.stderr:174636: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 2 SOMETHING remove 2024-09-15T20:55:38.630 INFO:tasks.workunit.client.0.smithi136.stderr:174636: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-15T20:55:38.630 INFO:tasks.workunit.client.0.smithi136.stderr:174636: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-15T20:55:38.630 INFO:tasks.workunit.client.0.smithi136.stderr:174636: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=2 2024-09-15T20:55:38.630 INFO:tasks.workunit.client.0.smithi136.stderr:174636: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-15T20:55:38.630 INFO:tasks.workunit.client.0.smithi136.stderr:174636: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/2 2024-09-15T20:55:38.630 INFO:tasks.workunit.client.0.smithi136.stderr:174636: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/2 SOMETHING remove 2024-09-15T20:55:38.630 INFO:tasks.workunit.client.0.smithi136.stderr:174636: remove 2#2:eb822e21:::SOMETHING:head# 2024-09-15T20:55:38.630 INFO:tasks.workunit.client.0.smithi136.stderr:174636: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 2 2024-09-15T20:55:38.630 INFO:tasks.workunit.client.0.smithi136.stderr:174636: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2024-09-15T20:55:38.630 INFO:tasks.workunit.client.0.smithi136.stderr:174636: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-15T20:55:38.630 INFO:tasks.workunit.client.0.smithi136.stderr:174636: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=2 2024-09-15T20:55:38.630 INFO:tasks.workunit.client.0.smithi136.stderr:174636: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-15T20:55:38.630 INFO:tasks.workunit.client.0.smithi136.stderr:174636: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/2 2024-09-15T20:55:38.630 INFO:tasks.workunit.client.0.smithi136.stderr:174636: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=ce4892c7-6c6d-4b4f-8e6f-07a3e11133e5 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-15T20:55:38.630 INFO:tasks.workunit.client.0.smithi136.stderr:174636: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T20:55:38.631 INFO:tasks.workunit.client.0.smithi136.stderr:174636: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T20:55:38.631 INFO:tasks.workunit.client.0.smithi136.stderr:174636: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T20:55:38.631 INFO:tasks.workunit.client.0.smithi136.stderr:174636: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/2' 2024-09-15T20:55:38.631 INFO:tasks.workunit.client.0.smithi136.stderr:174636: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/2/journal' 2024-09-15T20:55:38.631 INFO:tasks.workunit.client.0.smithi136.stderr:174636: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-15T20:55:38.632 INFO:tasks.workunit.client.0.smithi136.stderr:174636: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-15T20:55:38.632 INFO:tasks.workunit.client.0.smithi136.stderr:174636: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-15T20:55:38.632 INFO:tasks.workunit.client.0.smithi136.stderr:174636: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-15T20:55:38.632 INFO:tasks.workunit.client.0.smithi136.stderr:174636: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-15T20:55:38.632 INFO:tasks.workunit.client.0.smithi136.stderr:174636: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-15T20:55:38.632 INFO:tasks.workunit.client.0.smithi136.stderr:174636: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-15T20:55:38.632 INFO:tasks.workunit.client.0.smithi136.stderr:174636: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T20:55:38.632 INFO:tasks.workunit.client.0.smithi136.stderr:174636: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52602 2024-09-15T20:55:38.632 INFO:tasks.workunit.client.0.smithi136.stderr:174636: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.52602/$cluster-$name.asok' 2024-09-15T20:55:38.632 INFO:tasks.workunit.client.0.smithi136.stderr:174636: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.52602/$cluster-$name.asok' 2024-09-15T20:55:38.632 INFO:tasks.workunit.client.0.smithi136.stderr:174636: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-15T20:55:38.633 INFO:tasks.workunit.client.0.smithi136.stderr:174636: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-15T20:55:38.633 INFO:tasks.workunit.client.0.smithi136.stderr:174636: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-15T20:55:38.633 INFO:tasks.workunit.client.0.smithi136.stderr:174636: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T20:55:38.633 INFO:tasks.workunit.client.0.smithi136.stderr:174636: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T20:55:38.633 INFO:tasks.workunit.client.0.smithi136.stderr:174636: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T20:55:38.633 INFO:tasks.workunit.client.0.smithi136.stderr:174636: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T20:55:38.633 INFO:tasks.workunit.client.0.smithi136.stderr:174636: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-15T20:55:38.633 INFO:tasks.workunit.client.0.smithi136.stderr:174636: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-15T20:55:38.633 INFO:tasks.workunit.client.0.smithi136.stderr:174636: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/2 2024-09-15T20:55:38.633 INFO:tasks.workunit.client.0.smithi136.stderr:174636: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.2 2024-09-15T20:55:38.633 INFO:tasks.workunit.client.0.smithi136.stderr:174636: start osd.2 2024-09-15T20:55:38.633 INFO:tasks.workunit.client.0.smithi136.stderr:174636: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 2 --fsid=ce4892c7-6c6d-4b4f-8e6f-07a3e11133e5 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.52602/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T20:55:42.966 INFO:tasks.workunit.client.0.smithi136.stderr:174634: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-09-15T20:55:42.967 INFO:tasks.workunit.client.0.smithi136.stderr:174634: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 1 = 1 ']' 2024-09-15T20:55:42.967 INFO:tasks.workunit.client.0.smithi136.stderr:174634: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-15T20:55:42.967 INFO:tasks.workunit.client.0.smithi136.stderr:174634: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-15T20:55:42.967 INFO:tasks.workunit.client.0.smithi136.stderr:174634: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-15T20:55:42.967 INFO:tasks.workunit.client.0.smithi136.stderr:174634: 2024-09-15T20:55:38.480+0000 7fcc2fae0640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T20:55:42.967 INFO:tasks.workunit.client.0.smithi136.stderr:174634: 2024-09-15T20:55:38.482+0000 7fcc2fae0640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T20:55:42.967 INFO:tasks.workunit.client.0.smithi136.stderr:174634: 2024-09-15T20:55:38.484+0000 7fcc2fae0640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T20:55:42.967 INFO:tasks.workunit.client.0.smithi136.stderr:174634: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 1 2024-09-15T20:55:42.967 INFO:tasks.workunit.client.0.smithi136.stderr:174634: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-15T20:55:42.967 INFO:tasks.workunit.client.0.smithi136.stderr:174634: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2024-09-15T20:55:42.967 INFO:tasks.workunit.client.0.smithi136.stderr:174634: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-15T20:55:42.968 INFO:tasks.workunit.client.0.smithi136.stderr:174634: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-15T20:55:42.968 INFO:tasks.workunit.client.0.smithi136.stderr:174634: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:55:42.968 INFO:tasks.workunit.client.0.smithi136.stderr:174634: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-15T20:55:42.968 INFO:tasks.workunit.client.0.smithi136.stderr:174634: 0 2024-09-15T20:55:42.968 INFO:tasks.workunit.client.0.smithi136.stderr:174634: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:55:42.968 INFO:tasks.workunit.client.0.smithi136.stderr:174634: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T20:55:42.968 INFO:tasks.workunit.client.0.smithi136.stderr:174634: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:55:42.968 INFO:tasks.workunit.client.0.smithi136.stderr:174634: 2024-09-15T20:55:39.612+0000 7fcc2fae0640 -1 Falling back to public interface 2024-09-15T20:55:42.968 INFO:tasks.workunit.client.0.smithi136.stderr:174634: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:55:42.968 INFO:tasks.workunit.client.0.smithi136.stderr:174634: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:55:42.968 INFO:tasks.workunit.client.0.smithi136.stderr:174634: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-15T20:55:42.969 INFO:tasks.workunit.client.0.smithi136.stderr:174634: 1 2024-09-15T20:55:42.969 INFO:tasks.workunit.client.0.smithi136.stderr:174634: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:55:42.969 INFO:tasks.workunit.client.0.smithi136.stderr:174634: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T20:55:42.969 INFO:tasks.workunit.client.0.smithi136.stderr:174634: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:55:42.969 INFO:tasks.workunit.client.0.smithi136.stderr:174634: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:55:42.969 INFO:tasks.workunit.client.0.smithi136.stderr:174634: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:55:42.969 INFO:tasks.workunit.client.0.smithi136.stderr:174634: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-15T20:55:42.969 INFO:tasks.workunit.client.0.smithi136.stderr:174634: 2 2024-09-15T20:55:42.969 INFO:tasks.workunit.client.0.smithi136.stderr:174634: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:55:42.969 INFO:tasks.workunit.client.0.smithi136.stderr:174634: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T20:55:42.969 INFO:tasks.workunit.client.0.smithi136.stderr:174634: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:55:42.969 INFO:tasks.workunit.client.0.smithi136.stderr:174634: 2024-09-15T20:55:42.364+0000 7fcc2fae0640 -1 osd.1 66 log_to_monitors true 2024-09-15T20:55:42.969 INFO:tasks.workunit.client.0.smithi136.stderr:174634: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:55:42.970 INFO:tasks.workunit.client.0.smithi136.stderr:174634: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:55:42.970 INFO:tasks.workunit.client.0.smithi136.stderr:174634: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-15T20:55:42.970 INFO:tasks.workunit.client.0.smithi136.stderr:174634: 3 2024-09-15T20:55:42.970 INFO:tasks.workunit.client.0.smithi136.stderr:174634: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:55:42.970 INFO:tasks.workunit.client.0.smithi136.stderr:174634: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T20:55:43.978 INFO:tasks.workunit.client.0.smithi136.stderr:174634: osd.1 up in weight 1 up_from 71 up_thru 60 down_at 67 last_clean_interval [57,66) [v2:127.0.0.1:6810/1867674301,v1:127.0.0.1:6811/1867674301] [v2:127.0.0.1:6812/1867674301,v1:127.0.0.1:6813/1867674301] exists,up 4e733256-a0ac-49e1-b991-7e6e4a060afc 2024-09-15T20:55:43.978 INFO:tasks.workunit.client.0.smithi136.stderr:174634: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-15T20:55:43.978 INFO:tasks.workunit.client.0.smithi136.stderr:174634: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-15T20:55:43.979 INFO:tasks.workunit.client.0.smithi136.stderr:174634: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-15T20:55:43.979 INFO:tasks.workunit.client.0.smithi136.stderr:174634: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2024-09-15T20:55:43.979 INFO:tasks.workunit.client.0.smithi136.stderr:174634: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-15T20:55:43.979 INFO:tasks.workunit.client.0.smithi136.stderr:174634: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-15T20:55:43.979 INFO:tasks.workunit.client.0.smithi136.stderr:174634: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-15T20:55:43.979 INFO:tasks.workunit.client.0.smithi136.stderr:174634: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-15T20:55:43.979 INFO:tasks.workunit.client.0.smithi136.stderr:174634: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-15T20:55:43.979 INFO:tasks.workunit.client.0.smithi136.stderr:174634: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-15T20:55:43.979 INFO:tasks.workunit.client.0.smithi136.stderr:174634: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-15T20:55:43.979 INFO:tasks.workunit.client.0.smithi136.stderr:174634: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-15T20:55:43.979 INFO:tasks.workunit.client.0.smithi136.stderr:174634: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-15T20:55:43.979 INFO:tasks.workunit.client.0.smithi136.stderr:174634: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-15T20:55:43.979 INFO:tasks.workunit.client.0.smithi136.stderr:174634: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-15T20:55:43.979 INFO:tasks.workunit.client.0.smithi136.stderr:174634: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-15T20:55:43.980 INFO:tasks.workunit.client.0.smithi136.stderr:174634: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-15T20:55:43.980 INFO:tasks.workunit.client.0.smithi136.stderr:174634: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-15T20:55:43.980 INFO:tasks.workunit.client.0.smithi136.stderr:174634: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-15T20:55:43.980 INFO:tasks.workunit.client.0.smithi136.stderr:174634: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-15T20:55:43.980 INFO:tasks.workunit.client.0.smithi136.stderr:174634: 1 2024-09-15T20:55:43.980 INFO:tasks.workunit.client.0.smithi136.stderr:174634: 2 2024-09-15T20:55:43.980 INFO:tasks.workunit.client.0.smithi136.stderr:174634: 3' 2024-09-15T20:55:43.980 INFO:tasks.workunit.client.0.smithi136.stderr:174634: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-15T20:55:43.980 INFO:tasks.workunit.client.0.smithi136.stderr:174634: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T20:55:43.980 INFO:tasks.workunit.client.0.smithi136.stderr:174634: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-15T20:55:43.980 INFO:tasks.workunit.client.0.smithi136.stderr:174634: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836529 2024-09-15T20:55:43.980 INFO:tasks.workunit.client.0.smithi136.stderr:174634: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836529 2024-09-15T20:55:43.980 INFO:tasks.workunit.client.0.smithi136.stderr:174634: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836529' 2024-09-15T20:55:43.980 INFO:tasks.workunit.client.0.smithi136.stderr:174634: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T20:55:43.981 INFO:tasks.workunit.client.0.smithi136.stderr:174634: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-15T20:55:43.981 INFO:tasks.workunit.client.0.smithi136.stderr:174634: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=304942678018 2024-09-15T20:55:43.981 INFO:tasks.workunit.client.0.smithi136.stderr:174634: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 304942678018 2024-09-15T20:55:43.981 INFO:tasks.workunit.client.0.smithi136.stderr:174634: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836529 1-304942678018' 2024-09-15T20:55:43.981 INFO:tasks.workunit.client.0.smithi136.stderr:174634: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T20:55:44.109 INFO:tasks.workunit.client.0.smithi136.stderr:174634: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph t74636: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/2/whoami 2024-09-15T20:55:44.110 INFO:tasks.workunit.client.0.smithi136.stderr:174636: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 2 = 2 ']' 2024-09-15T20:55:44.110 INFO:tasks.workunit.client.0.smithi136.stderr:174636: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-15T20:55:44.110 INFO:tasks.workunit.client.0.smithi136.stderr:174636: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-15T20:55:44.110 INFO:tasks.workunit.client.0.smithi136.stderr:174636: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-15T20:55:44.110 INFO:tasks.workunit.client.0.smithi136.stderr:174636: 2024-09-15T20:55:38.655+0000 7fa0378e4640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T20:55:44.110 INFO:tasks.workunit.client.0.smithi136.stderr:174636: 2024-09-15T20:55:38.657+0000 7fa0378e4640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T20:55:44.110 INFO:tasks.workunit.client.0.smithi136.stderr:174636: 2024-09-15T20:55:38.658+0000 7fa0378e4640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T20:55:44.110 INFO:tasks.workunit.client.0.smithi136.stderr:174636: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 2 2024-09-15T20:55:44.110 INFO:tasks.workunit.client.0.smithi136.stderr:174636: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-15T20:55:44.110 INFO:tasks.workunit.client.0.smithi136.stderr:174636: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=2 2024-09-15T20:55:44.110 INFO:tasks.workunit.client.0.smithi136.stderr:174636: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-15T20:55:44.110 INFO:tasks.workunit.client.0.smithi136.stderr:174636: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-15T20:55:44.110 INFO:tasks.workunit.client.0.smithi136.stderr:174636: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:55:44.111 INFO:tasks.workunit.client.0.smithi136.stderr:174636: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-15T20:55:44.111 INFO:tasks.workunit.client.0.smithi136.stderr:174636: 0 2024-09-15T20:55:44.111 INFO:tasks.workunit.client.0.smithi136.stderr:174636: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:55:44.111 INFO:tasks.workunit.client.0.smithi136.stderr:174636: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T20:55:44.111 INFO:tasks.workunit.client.0.smithi136.stderr:174636: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:55:44.111 INFO:tasks.workunit.client.0.smithi136.stderr:174636: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:55:44.111 INFO:tasks.workunit.client.0.smithi136.stderr:174636: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:55:44.111 INFO:tasks.workunit.client.0.smithi136.stderr:174636: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-15T20:55:44.111 INFO:tasks.workunit.client.0.smithi136.stderr:174636: 1 2024-09-15T20:55:44.111 INFO:tasks.workunit.client.0.smithi136.stderr:174636: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:55:44.111 INFO:tasks.workunit.client.0.smithi136.stderr:174636: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T20:55:44.111 INFO:tasks.workunit.client.0.smithi136.stderr:174636: 2024-09-15T20:55:40.261+0000 7fa0378e4640 -1 Falling back to public interface 2024-09-15T20:55:44.111 INFO:tasks.workunit.client.0.smithi136.stderr:174636: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:55:44.111 INFO:tasks.workunit.client.0.smithi136.stderr:174636: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:55:44.112 INFO:tasks.workunit.client.0.smithi136.stderr:174636: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:55:44.112 INFO:tasks.workunit.client.0.smithi136.stderr:174636: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-15T20:55:44.112 INFO:tasks.workunit.client.0.smithi136.stderr:174636: 2 2024-09-15T20:55:44.112 INFO:tasks.workunit.client.0.smithi136.stderr:174636: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:55:44.112 INFO:tasks.workunit.client.0.smithi136.stderr:174636: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T20:55:44.112 INFO:tasks.workunit.client.0.smithi136.stderr:174636: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:55:44.112 INFO:tasks.workunit.client.0.smithi136.stderr:174636: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:55:44.112 INFO:tasks.workunit.client.0.smithi136.stderr:174636: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:55:44.112 INFO:tasks.workunit.client.0.smithi136.stderr:174636: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-15T20:55:44.112 INFO:tasks.workunit.client.0.smithi136.stderr:174636: 3 2024-09-15T20:55:44.112 INFO:tasks.workunit.client.0.smithi136.stderr:174636: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:55:44.112 INFO:tasks.workunit.client.0.smithi136.stderr:174636: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T20:55:44.112 INFO:tasks.workunit.client.0.smithi136.stderr:174636: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:55:46.218 INFO:tasks.workunit.client.0.smithi136.stderr:174636: 2024-09-15T20:55:44.083+0000 7fa0378e4640 -1 osd.2 66 log_to_monitors true 2024-09-15T20:55:46.218 INFO:tasks.workunit.client.0.smithi136.stderr:174636: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:55:46.219 INFO:tasks.workunit.client.0.smithi136.stderr:174636: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:55:46.219 INFO:tasks.workunit.client.0.smithi136.stderr:174636: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-15T20:55:46.219 INFO:tasks.workunit.client.0.smithi136.stderr:174636: 4 2024-09-15T20:55:46.219 INFO:tasks.workunit.client.0.smithi136.stderr:174636: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:55:46.219 INFO:tasks.workunit.client.0.smithi136.stderr:174636: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T20:55:46.219 INFO:tasks.workunit.client.0.smithi136.stderr:174636: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:55:46.219 INFO:tasks.workunit.client.0.smithi136.stderr:174636: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:55:46.219 INFO:tasks.workunit.client.0.smithi136.stderr:174636: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:55:46.219 INFO:tasks.workunit.client.0.smithi136.stderr:174636: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-15T20:55:46.219 INFO:tasks.workunit.client.0.smithi136.stderr:174636: 5 2024-09-15T20:55:46.219 INFO:tasks.workunit.client.0.smithi136.stderr:174636: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:55:46.219 INFO:tasks.workunit.client.0.smithi136.stderr:174636: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T20:55:46.219 INFO:tasks.workunit.client.0.smithi136.stderr:174636: osd.2 up in weight 1 up_from 75 up_thru 53 down_at 67 last_clean_interval [15,66) [v2:127.0.0.1:6818/4095853313,v1:127.0.0.1:6819/4095853313] [v2:127.0.0.1:6820/4095853313,v1:127.0.0.1:6821/4095853313] exists,up f74e59aa-0b73-4bbf-a870-d7c571cf9f22 2024-09-15T20:55:46.219 INFO:tasks.workunit.client.0.smithi136.stderr:174636: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-15T20:55:46.220 INFO:tasks.workunit.client.0.smithi136.stderr:174636: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-15T20:55:46.220 INFO:tasks.workunit.client.0.smithi136.stderr:174636: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-15T20:55:46.220 INFO:tasks.workunit.client.0.smithi136.stderr:174636: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2024-09-15T20:55:46.220 INFO:tasks.workunit.client.0.smithi136.stderr:174636: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-15T20:55:46.220 INFO:tasks.workunit.client.0.smithi136.stderr:174636: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-15T20:55:46.220 INFO:tasks.workunit.client.0.smithi136.stderr:174636: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-15T20:55:46.220 INFO:tasks.workunit.client.0.smithi136.stderr:174636: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-15T20:55:46.220 INFO:tasks.workunit.client.0.smithi136.stderr:174636: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-15T20:55:46.220 INFO:tasks.workunit.client.0.smithi136.stderr:174636: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-15T20:55:46.220 INFO:tasks.workunit.client.0.smithi136.stderr:174636: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-15T20:55:46.220 INFO:tasks.workunit.client.0.smithi136.stderr:174636: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-15T20:55:46.220 INFO:tasks.workunit.client.0.smithi136.stderr:174636: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-15T20:55:46.220 INFO:tasks.workunit.client.0.smithi136.stderr:174636: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-15T20:55:46.220 INFO:tasks.workunit.client.0.smithi136.stderr:174636: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-15T20:55:46.221 INFO:tasks.workunit.client.0.smithi136.stderr:174636: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-15T20:55:46.221 INFO:tasks.workunit.client.0.smithi136.stderr:174636: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-15T20:55:46.221 INFO:tasks.workunit.client.0.smithi136.stderr:174636: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-15T20:55:46.221 INFO:tasks.workunit.client.0.smithi136.stderr:174636: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-15T20:55:46.221 INFO:tasks.workunit.client.0.smithi136.stderr:174636: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-15T20:55:46.221 INFO:tasks.workunit.client.0.smithi136.stderr:174636: 1 2024-09-15T20:55:46.221 INFO:tasks.workunit.client.0.smithi136.stderr:174636: 2 2024-09-15T20:55:46.221 INFO:tasks.workunit.client.0.smithi136.stderr:174636: 3' 2024-09-15T20:55:46.221 INFO:tasks.workunit.client.0.smithi136.stderr:174636: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-15T20:55:46.221 INFO:tasks.workunit.client.0.smithi136.stderr:174636: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T20:55:46.484 INFO:tasks.workunit.client.0.smithi136.stderr:174636: //home/uell osd.2 flush_pg_stats 2024-09-15T20:55:46.484 INFO:tasks.workunit.client.0.smithi136.stderr:174634: Error ENXIO: problem getting command descriptions from osd.2 2024-09-15T20:55:46.484 INFO:tasks.workunit.client.0.smithi136.stderr:174634: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq= 2024-09-15T20:55:46.484 INFO:tasks.workunit.client.0.smithi136.stderr:174634: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z '' 2024-09-15T20:55:46.485 INFO:tasks.workunit.client.0.smithi136.stderr:174634: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: continue 2024-09-15T20:55:46.485 INFO:tasks.workunit.client.0.smithi136.stderr:174634: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T20:55:46.485 INFO:tasks.workunit.client.0.smithi136.stderr:174634: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-15T20:55:46.485 INFO:tasks.workunit.client.0.smithi136.stderr:174634: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=274877906949 2024-09-15T20:55:46.485 INFO:tasks.workunit.client.0.smithi136.stderr:174634: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 274877906949 2024-09-15T20:55:46.485 INFO:tasks.workunit.client.0.smithi136.stderr:174634: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836529 1-304942678018 3-274877906949' 2024-09-15T20:55:46.485 INFO:tasks.workunit.client.0.smithi136.stderr:174634: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T20:55:46.485 INFO:tasks.workunit.client.0.smithi136.stderr:174634: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836529 2024-09-15T20:55:46.485 INFO:tasks.workunit.client.0.smithi136.stderr:174634: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T20:55:46.485 INFO:tasks.workunit.client.0.smithi136.stderr:174634: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-15T20:55:46.485 INFO:tasks.workunit.client.0.smithi136.stderr:174634: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836529 2024-09-15T20:55:46.485 INFO:tasks.workunit.client.0.smithi136.stderr:174634: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T20:55:46.485 INFO:tasks.workunit.client.0.smithi136.stderr:174634: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836529 2024-09-15T20:55:46.486 INFO:tasks.workunit.client.0.smithi136.stderr:174634: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836529' 2024-09-15T20:55:46.486 INFO:tasks.workunit.client.0.smithi136.stderr:174634: waiting osd.0 seq 21474836529 2024-09-15T20:55:46.486 INFO:tasks.workunit.client.0.smithi136.stderr:174634: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T20:55:46.486 INFO:tasks.workunit.client.0.smithi136.stderr:174634: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836528 -lt 21474836529 2024-09-15T20:55:46.486 INFO:tasks.workunit.client.0.smithi136.stderr:174634: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-15T20:55:46.486 INFO:tasks.workunit.client.0.smithi136.stderr:174634: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-15T20:55:46.486 INFO:tasks.workunit.client.0.smithi136.stderr:174634: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T20:55:46.486 INFO:tasks.workunit.client.0.smithi136.stderr:174634: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836529 -lt 21474836529 2024-09-15T20:55:46.486 INFO:tasks.workunit.client.0.smithi136.stderr:174634: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T20:55:46.486 INFO:tasks.workunit.client.0.smithi136.stderr:174634: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-304942678018 2024-09-15T20:55:46.486 INFO:tasks.workunit.client.0.smithi136.stderr:174634: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T20:55:46.486 INFO:tasks.workunit.client.0.smithi136.stderr:174634: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-15T20:55:46.486 INFO:tasks.workunit.client.0.smithi136.stderr:174634: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-304942678018 2024-09-15T20:55:46.486 INFO:tasks.workunit.client.0.smithi136.stderr:174634: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T20:55:46.487 INFO:tasks.workunit.client.0.smithi136.stderr:174634: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=304942678018 2024-09-15T20:55:46.487 INFO:tasks.workunit.client.0.smithi136.stderr:174634: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 304942678018' 2024-09-15T20:55:46.487 INFO:tasks.workunit.client.0.smithi136.stderr:174634: waiting osd.1 seq 304942678018 2024-09-15T20:55:46.487 INFO:tasks.workunit.client.0.smithi136.stderr:174634: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-15T20:55:46.487 INFO:tasks.workunit.client.0.smithi136.stderr:174634: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 304942678018 -lt 304942678018 2024-09-15T20:55:46.487 INFO:tasks.workunit.client.0.smithi136.stderr:174634: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T20:55:47.000 INFO:tasks.workunit.client.0.smithi136.stderr:174634: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-15T20:55:47.000 INFO:tasks.workunit.client.0.smithi136.stderr:174636: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836530 2024-09-15T20:55:47.000 INFO:tasks.workunit.client.0.smithi136.stderr:174636: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836530 2024-09-15T20:55:47.000 INFO:tasks.workunit.client.0.smithi136.stderr:174636: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836530' 2024-09-15T20:55:47.000 INFO:tasks.workunit.client.0.smithi136.stderr:174636: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T20:55:47.000 INFO:tasks.workunit.client.0.smithi136.stderr:174636: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-15T20:55:47.000 INFO:tasks.workunit.client.0.smithi136.stderr:174636: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=304942678019 2024-09-15T20:55:47.000 INFO:tasks.workunit.client.0.smithi136.stderr:174636: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 304942678019 2024-09-15T20:55:47.000 INFO:tasks.workunit.client.0.smithi136.stderr:174636: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836530 1-304942678019' 2024-09-15T20:55:47.001 INFO:tasks.workunit.client.0.smithi136.stderr:174636: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T20:55:47.001 INFO:tasks.workunit.client.0.smithi136.stderr:174636: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-15T20:55:47.001 INFO:tasks.workunit.client.0.smithi136.stderr:174636: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=322122547202 2024-09-15T20:55:47.001 INFO:tasks.workunit.client.0.smithi136.stderr:174636: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 322122547202 2024-09-15T20:55:47.001 INFO:tasks.workunit.client.0.smithi136.stderr:174636: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836530 1-304942678019 2-322122547202' 2024-09-15T20:55:47.001 INFO:tasks.workunit.client.0.smithi136.stderr:174636: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T20:55:47.001 INFO:tasks.workunit.client.0.smithi136.stderr:174636: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-15T20:55:47.001 INFO:tasks.workunit.client.0.smithi136.stderr:174636: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=274877906951 2024-09-15T20:55:47.001 INFO:tasks.workunit.client.0.smithi136.stderr:174636: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 274877906951 2024-09-15T20:55:47.001 INFO:tasks.workunit.client.0.smithi136.stderr:174636: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836530 1-304942678019 2-322122547202 3-274877906951' 2024-09-15T20:55:47.001 INFO:tasks.workunit.client.0.smithi136.stderr:174636: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T20:55:47.001 INFO:tasks.workunit.client.0.smithi136.stderr:174636: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836530 2024-09-15T20:55:47.001 INFO:tasks.workunit.client.0.smithi136.stderr:174636: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T20:55:47.001 INFO:tasks.workunit.client.0.smithi136.stderr:174636: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-15T20:55:47.002 INFO:tasks.workunit.client.0.smithi136.stderr:174636: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836530 2024-09-15T20:55:47.002 INFO:tasks.workunit.client.0.smithi136.stderr:174636: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T20:55:47.002 INFO:tasks.workunit.client.0.smithi136.stderr:174636: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836530 2024-09-15T20:55:47.002 INFO:tasks.workunit.client.0.smithi136.stderr:174636: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836530' 2024-09-15T20:55:47.002 INFO:tasks.workunit.client.0.smithi136.stderr:174636: waiting osd.0 seq 21474836530 2024-09-15T20:55:47.002 INFO:tasks.workunit.client.0.smithi136.stderr:174636: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T20:55:47.002 INFO:tasks.workunit.client.0.smithi136.stderr:174636: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836530 -lt 21474836530 2024-09-15T20:55:47.002 INFO:tasks.workunit.client.0.smithi136.stderr:174636: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T20:55:47.002 INFO:tasks.workunit.client.0.smithi136.stderr:174636: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-304942678019 2024-09-15T20:55:47.002 INFO:tasks.workunit.client.0.smithi136.stderr:174636: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T20:55:47.002 INFO:tasks.workunit.client.0.smithi136.stderr:174636: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-15T20:55:47.861 INFO:tasks.workunit.client.0.smithi136.stderr:174636: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:buntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 3-274877906949 2024-09-15T20:55:47.861 INFO:tasks.workunit.client.0.smithi136.stderr:174634: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T20:55:47.861 INFO:tasks.workunit.client.0.smithi136.stderr:174634: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=3 2024-09-15T20:55:47.862 INFO:tasks.workunit.client.0.smithi136.stderr:174634: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 3-274877906949 2024-09-15T20:55:47.862 INFO:tasks.workunit.client.0.smithi136.stderr:174634: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T20:55:47.862 INFO:tasks.workunit.client.0.smithi136.stderr:174634: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=274877906949 2024-09-15T20:55:47.862 INFO:tasks.workunit.client.0.smithi136.stderr:174634: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.3 seq 274877906949' 2024-09-15T20:55:47.862 INFO:tasks.workunit.client.0.smithi136.stderr:174634: waiting osd.3 seq 274877906949 2024-09-15T20:55:47.862 INFO:tasks.workunit.client.0.smithi136.stderr:174634: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-15T20:55:47.862 INFO:tasks.workunit.client.0.smithi136.stderr:174634: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 274877906949 -lt 274877906949 2024-09-15T20:55:47.862 INFO:tasks.workunit.client.0.smithi136.stderr:174634: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-15T20:55:47.862 INFO:tasks.workunit.client.0.smithi136.stderr:174634: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T20:55:47.862 INFO:tasks.workunit.client.0.smithi136.stderr:174634: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T20:55:47.862 INFO:tasks.workunit.client.0.smithi136.stderr:174634: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 5 == 0 2024-09-15T20:55:47.862 INFO:tasks.workunit.client.0.smithi136.stderr:174634: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-15T20:55:47.862 INFO:tasks.workunit.client.0.smithi136.stderr:174634: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-15T20:55:47.863 INFO:tasks.workunit.client.0.smithi136.stderr:174634: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-15T20:55:47.863 INFO:tasks.workunit.client.0.smithi136.stderr:174634: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T20:55:47.863 INFO:tasks.workunit.client.0.smithi136.stderr:174634: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T20:55:47.863 INFO:tasks.workunit.client.0.smithi136.stderr:174634: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T20:55:47.863 INFO:tasks.workunit.client.0.smithi136.stderr:174634: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T20:55:47.863 INFO:tasks.workunit.client.0.smithi136.stderr:174634: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=5 2024-09-15T20:55:47.863 INFO:tasks.workunit.client.0.smithi136.stderr:174634: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-15T20:55:47.863 INFO:tasks.workunit.client.0.smithi136.stderr:174634: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T20:55:47.863 INFO:tasks.workunit.client.0.smithi136.stderr:174634: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T20:55:47.863 INFO:tasks.workunit.client.0.smithi136.stderr:174634: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 = 5 2024-09-15T20:55:47.863 INFO:tasks.workunit.client.0.smithi136.stderr:174634: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-15T20:55:47.863 INFO:tasks.workunit.client.0.smithi136.stderr:174634: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-15T20:55:47.863 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: run_in_background: return 0 2024-09-15T20:55:47.863 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: wait_background: for pid in $pids 2024-09-15T20:55:47.863 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2175: wait_background: wait 174633 2024-09-15T20:55:48.306 INFO:tasks.workunit.client.0.smithi136.stderr:2220: flush_pg_stats: echo 1-304942678019 2024-09-15T20:55:48.306 INFO:tasks.workunit.client.0.smithi136.stderr:174636: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T20:55:48.306 INFO:tasks.workunit.client.0.smithi136.stderr:174636: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=304942678019 2024-09-15T20:55:48.306 INFO:tasks.workunit.client.0.smithi136.stderr:174636: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 304942678019' 2024-09-15T20:55:48.307 INFO:tasks.workunit.client.0.smithi136.stderr:174636: waiting osd.1 seq 304942678019 2024-09-15T20:55:48.307 INFO:tasks.workunit.client.0.smithi136.stderr:174636: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-15T20:55:48.307 INFO:tasks.workunit.client.0.smithi136.stderr:174636: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 304942678019 -lt 304942678019 2024-09-15T20:55:48.307 INFO:tasks.workunit.client.0.smithi136.stderr:174636: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T20:55:48.307 INFO:tasks.workunit.client.0.smithi136.stderr:174636: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-322122547202 2024-09-15T20:55:48.307 INFO:tasks.workunit.client.0.smithi136.stderr:174636: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T20:55:48.307 INFO:tasks.workunit.client.0.smithi136.stderr:174636: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-15T20:55:48.307 INFO:tasks.workunit.client.0.smithi136.stderr:174636: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-322122547202 2024-09-15T20:55:48.307 INFO:tasks.workunit.client.0.smithi136.stderr:174636: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T20:55:48.307 INFO:tasks.workunit.client.0.smithi136.stderr:174636: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=322122547202 2024-09-15T20:55:48.307 INFO:tasks.workunit.client.0.smithi136.stderr:174636: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 322122547202' 2024-09-15T20:55:48.308 INFO:tasks.workunit.client.0.smithi136.stderr:174636: waiting osd.2 seq 322122547202 2024-09-15T20:55:48.308 INFO:tasks.workunit.client.0.smithi136.stderr:174636: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-15T20:55:48.308 INFO:tasks.workunit.client.0.smithi136.stderr:174636: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 322122547202 -lt 322122547202 2024-09-15T20:55:48.308 INFO:tasks.workunit.client.0.smithi136.stderr:174636: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T20:55:48.308 INFO:tasks.workunit.client.0.smithi136.stderr:174636: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 3-274877906951 2024-09-15T20:55:48.308 INFO:tasks.workunit.client.0.smithi136.stderr:174636: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T20:55:48.308 INFO:tasks.workunit.client.0.smithi136.stderr:174636: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=3 2024-09-15T20:55:48.308 INFO:tasks.workunit.client.0.smithi136.stderr:174636: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 3-274877906951 2024-09-15T20:55:48.308 INFO:tasks.workunit.client.0.smithi136.stderr:174636: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T20:55:48.308 INFO:tasks.workunit.client.0.smithi136.stderr:174636: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=274877906951 2024-09-15T20:55:48.308 INFO:tasks.workunit.client.0.smithi136.stderr:174636: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.3 seq 274877906951' 2024-09-15T20:55:48.308 INFO:tasks.workunit.client.0.smithi136.stderr:174636: waiting osd.3 seq 274877906951 2024-09-15T20:55:48.308 INFO:tasks.workunit.client.0.smithi136.stderr:174636: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-15T20:55:48.309 INFO:tasks.workunit.client.0.smithi136.stderr:174636: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 274877906951 -lt 274877906951 2024-09-15T20:55:48.309 INFO:tasks.workunit.client.0.smithi136.stderr:174636: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-15T20:55:48.309 INFO:tasks.workunit.client.0.smithi136.stderr:174636: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T20:55:48.309 INFO:tasks.workunit.client.0.smithi136.stderr:174636: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T20:55:48.309 INFO:tasks.workunit.client.0.smithi136.stderr:174636: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 5 == 0 2024-09-15T20:55:48.309 INFO:tasks.workunit.client.0.smithi136.stderr:174636: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-15T20:55:48.309 INFO:tasks.workunit.client.0.smithi136.stderr:174636: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-15T20:55:48.309 INFO:tasks.workunit.client.0.smithi136.stderr:174636: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-15T20:55:48.933 INFO:tasks.workunit.client.0.smithi136.stderr:174636: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T20:55:48.933 INFO:tasks.workunit.client.0.smithi136.stderr:174636: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T20:55:48.934 INFO:tasks.workunit.client.0.smithi136.stderr:174636: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T20:55:48.934 INFO:tasks.workunit.client.0.smithi136.stderr:174636: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T20:55:48.934 INFO:tasks.workunit.client.0.smithi136.stderr:174636: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=5 2024-09-15T20:55:48.934 INFO:tasks.workunit.client.0.smithi136.stderr:174636: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-15T20:55:48.934 INFO:tasks.workunit.client.0.smithi136.stderr:174636: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T20:55:48.934 INFO:tasks.workunit.client.0.smithi136.stderr:174636: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T20:55:48.934 INFO:tasks.workunit.client.0.smithi136.stderr:174636: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 = 5 2024-09-15T20:55:48.934 INFO:tasks.workunit.client.0.smithi136.stderr:174636: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-15T20:55:48.934 INFO:tasks.workunit.client.0.smithi136.stderr:174636: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-15T20:55:48.934 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: run_in_background: return 0 2024-09-15T20:55:48.934 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: wait_background: eval 'pids='\'''\''' 2024-09-15T20:55:48.934 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: wait_background: pids= 2024-09-15T20:55:48.934 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: wait_background: return 0 2024-09-15T20:55:48.934 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:197: corrupt_and_repair_two: return_code=0 2024-09-15T20:55:48.935 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:198: corrupt_and_repair_two: '[' 0 -ne 0 ']' 2024-09-15T20:55:48.935 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:203: corrupt_and_repair_two: get_pg ecpool SOMETHING 2024-09-15T20:55:48.935 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1084: get_pg: local poolname=ecpool 2024-09-15T20:55:48.935 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1085: get_pg: local objectname=SOMETHING 2024-09-15T20:55:48.935 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1087: get_pg: ceph --format json osd map ecpool SOMETHING 2024-09-15T20:55:48.935 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1087: get_pg: jq -r .pgid 2024-09-15T20:55:49.213 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:203: corrupt_and_repair_two: local pg=2.0 2024-09-15T20:55:49.213 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:204: corrupt_and_repair_two: repair 2.0 2024-09-15T20:55:49.213 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1846: repair: local pgid=2.0 2024-09-15T20:55:49.213 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1847: repair: get_last_scrub_stamp 2.0 2024-09-15T20:55:49.213 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2024-09-15T20:55:49.213 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-15T20:55:49.213 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-15T20:55:49.213 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-09-15T20:55:49.490 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1847: repair: local last_scrub=2024-09-15T20:55:19.155903+0000 2024-09-15T20:55:49.490 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1848: repair: ceph pg repair 2.0 2024-09-15T20:55:49.751 INFO:tasks.workunit.client.0.smithi136.stderr:instructing pg 2.0s0 on osd.3 to repair 2024-09-15T20:55:49.763 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1849: repair: wait_for_scrub 2.0 2024-09-15T20:55:19.155903+0000 2024-09-15T20:55:49.763 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: local pgid=2.0 2024-09-15T20:55:49.764 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2018: wait_for_scrub: local last_scrub=2024-09-15T20:55:19.155903+0000 2024-09-15T20:55:49.764 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2019: wait_for_scrub: local sname=last_scrub_stamp 2024-09-15T20:55:49.764 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i=0 )) 2024-09-15T20:55:49.764 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-15T20:55:49.764 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-09-15T20:55:49.764 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2024-09-15T20:55:49.764 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-15T20:55:49.764 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-15T20:55:49.764 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-09-15T20:55:50.034 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-15T20:55:19.155903+0000 '>' 2024-09-15T20:55:19.155903+0000 2024-09-15T20:55:50.034 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-15T20:55:51.036 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-15T20:55:51.036 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-15T20:55:51.037 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-09-15T20:55:51.037 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2024-09-15T20:55:51.037 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-15T20:55:51.037 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-15T20:55:51.037 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-09-15T20:55:51.320 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-15T20:55:19.155903+0000 '>' 2024-09-15T20:55:19.155903+0000 2024-09-15T20:55:51.320 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-15T20:55:52.321 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-15T20:55:52.321 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-15T20:55:52.322 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-09-15T20:55:52.322 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2024-09-15T20:55:52.322 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-15T20:55:52.322 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-15T20:55:52.322 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-09-15T20:55:52.608 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-15T20:55:19.155903+0000 '>' 2024-09-15T20:55:19.155903+0000 2024-09-15T20:55:52.608 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-15T20:55:53.609 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-15T20:55:53.609 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-15T20:55:53.610 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-09-15T20:55:53.610 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2024-09-15T20:55:53.610 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-15T20:55:53.610 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-15T20:55:53.610 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-09-15T20:55:53.895 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-15T20:55:19.155903+0000 '>' 2024-09-15T20:55:19.155903+0000 2024-09-15T20:55:53.895 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-15T20:55:54.896 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-15T20:55:54.896 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-15T20:55:54.897 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-09-15T20:55:54.897 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2024-09-15T20:55:54.897 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-15T20:55:54.898 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-15T20:55:54.898 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-09-15T20:55:55.179 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-15T20:55:19.155903+0000 '>' 2024-09-15T20:55:19.155903+0000 2024-09-15T20:55:55.179 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-15T20:55:56.180 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-15T20:55:56.181 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-15T20:55:56.181 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-09-15T20:55:56.181 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2024-09-15T20:55:56.182 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-15T20:55:56.182 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-15T20:55:56.182 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-09-15T20:55:56.461 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-15T20:55:19.155903+0000 '>' 2024-09-15T20:55:19.155903+0000 2024-09-15T20:55:56.462 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-15T20:55:57.463 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-15T20:55:57.463 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-15T20:55:57.463 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-09-15T20:55:57.464 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2024-09-15T20:55:57.464 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-15T20:55:57.464 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-15T20:55:57.464 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-09-15T20:55:57.744 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-15T20:55:50.632381+0000 '>' 2024-09-15T20:55:19.155903+0000 2024-09-15T20:55:57.744 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2023: wait_for_scrub: return 0 2024-09-15T20:55:57.744 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1850: repair: sleep 2 2024-09-15T20:55:59.747 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:208: corrupt_and_repair_two: pids= 2024-09-15T20:55:59.747 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:209: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 1 SOMETHING list-attrs 2024-09-15T20:55:59.747 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2135: run_in_background: local pid_variable=pids 2024-09-15T20:55:59.747 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2136: run_in_background: shift 2024-09-15T20:55:59.747 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2140: run_in_background: eval 'pids+=" 177549"' 2024-09-15T20:55:59.747 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2140: run_in_background: pids+=' 177549' 2024-09-15T20:55:59.748 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:210: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 2 SOMETHING list-attrs 2024-09-15T20:55:59.748 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2135: run_in_background: local pid_variable=pids 2024-09-15T20:55:59.748 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2136: run_in_background: shift 2024-09-15T20:55:59.748 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2140: run_in_background: eval 'pids+=" 177551"' 2024-09-15T20:55:59.748 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: run_in_background: objectstore_tool td/osd-scrub-repair 1 SOMETHING list-attrs 2024-09-15T20:55:59.748 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2140: run_in_background: pids+=' 177551' 2024-09-15T20:55:59.748 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:211: corrupt_and_repair_two: wait_background pids 2024-09-15T20:55:59.748 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: wait_background: pids=' 177549 177551' 2024-09-15T20:55:59.748 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: run_in_background: sed 's/^/177552: /' 2024-09-15T20:55:59.748 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: wait_background: return_code=0 2024-09-15T20:55:59.748 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: wait_background: for pid in $pids 2024-09-15T20:55:59.748 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2175: wait_background: wait 177549 2024-09-15T20:55:59.748 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: run_in_background: objectstore_tool td/osd-scrub-repair 2 SOMETHING list-attrs 2024-09-15T20:55:59.749 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: run_in_background: sed 's/^/177554: /' 2024-09-15T20:56:00.812 INFO:tasks.workunit.client.0.smithi136.stderr:177552: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2024-09-15T20:56:00.812 INFO:tasks.workunit.client.0.smithi136.stderr:177552: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2024-09-15T20:56:00.812 INFO:tasks.workunit.client.0.smithi136.stderr:177552: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=1 2024-09-15T20:56:00.812 INFO:tasks.workunit.client.0.smithi136.stderr:177552: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2024-09-15T20:56:00.812 INFO:tasks.workunit.client.0.smithi136.stderr:177552: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 SOMETHING list-attrs 2024-09-15T20:56:00.812 INFO:tasks.workunit.client.0.smithi136.stderr:177552: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-09-15T20:56:00.812 INFO:tasks.workunit.client.0.smithi136.stderr:177552: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2024-09-15T20:56:00.812 INFO:tasks.workunit.client.0.smithi136.stderr:177552: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=1 2024-09-15T20:56:00.813 INFO:tasks.workunit.client.0.smithi136.stderr:177552: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2024-09-15T20:56:00.813 INFO:tasks.workunit.client.0.smithi136.stderr:177552: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-09-15T20:56:00.813 INFO:tasks.workunit.client.0.smithi136.stderr:177552: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-15T20:56:00.813 INFO:tasks.workunit.client.0.smithi136.stderr:177552: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-15T20:56:00.813 INFO:tasks.workunit.client.0.smithi136.stderr:177552: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-15T20:56:00.813 INFO:tasks.workunit.client.0.smithi136.stderr:177552: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-15T20:56:00.813 INFO:tasks.workunit.client.0.smithi136.stderr:177552: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-15T20:56:00.813 INFO:tasks.workunit.client.0.smithi136.stderr:177552: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-15T20:56:00.813 INFO:tasks.workunit.client.0.smithi136.stderr:177552: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 SOMETHING list-attrs 2024-09-15T20:56:00.813 INFO:tasks.workunit.client.0.smithi136.stderr:177552: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-15T20:56:00.813 INFO:tasks.workunit.client.0.smithi136.stderr:177552: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-15T20:56:00.814 INFO:tasks.workunit.client.0.smithi136.stderr:177552: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=1 2024-09-15T20:56:00.814 INFO:tasks.workunit.client.0.smithi136.stderr:177552: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-15T20:56:00.814 INFO:tasks.workunit.client.0.smithi136.stderr:177552: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-09-15T20:56:00.814 INFO:tasks.workunit.client.0.smithi136.stderr:177552: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 SOMETHING list-attrs 2024-09-15T20:56:00.814 INFO:tasks.workunit.client.0.smithi136.stderr:177552: _ 2024-09-15T20:56:00.814 INFO:tasks.workunit.client.0.smithi136.stderr:177552: hinfo_key 2024-09-15T20:56:00.814 INFO:tasks.workunit.client.0.smithi136.stderr:177552: snapset 2024-09-15T20:56:00.814 INFO:tasks.workunit.client.0.smithi136.stderr:177552: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2024-09-15T20:56:00.814 INFO:tasks.workunit.client.0.smithi136.stderr:177552: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2024-09-15T20:56:00.814 INFO:tasks.workunit.client.0.smithi136.stderr:177552: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-15T20:56:00.814 INFO:tasks.workunit.client.0.smithi136.stderr:177552: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=1 2024-09-15T20:56:00.814 INFO:tasks.workunit.client.0.smithi136.stderr:177552: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-15T20:56:00.814 INFO:tasks.workunit.client.0.smithi136.stderr:177552: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-09-15T20:56:00.814 INFO:tasks.workunit.client.0.smithi136.stderr:177552: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=ce4892c7-6c6d-4b4f-8e6f-07a3e11133e5 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-15T20:56:00.815 INFO:tasks.workunit.client.0.smithi136.stderr:177552: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T20:56:00.815 INFO:tasks.workunit.client.0.smithi136.stderr:177552: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T20:56:00.815 INFO:tasks.workunit.client.0.smithi136.stderr:177552: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T20:56:00.815 INFO:tasks.workunit.client.0.smithi136.stderr:177552: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-09-15T20:56:00.815 INFO:tasks.workunit.client.0.smithi136.stderr:177552: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-09-15T20:56:00.816 INFO:tasks.workunit.client.0.smithi136.stderr:177552: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-15T20:56:00.816 INFO:tasks.workunit.client.0.smithi136.stderr:177552: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-15T20:56:00.816 INFO:tasks.workunit.client.0.smithi136.stderr:177552: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-15T20:56:00.816 INFO:tasks.workunit.client.0.smithi136.stderr:177552: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-15T20:56:00.816 INFO:tasks.workunit.client.0.smithi136.stderr:177552: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-15T20:56:00.816 INFO:tasks.workunit.client.0.smithi136.stderr:177552: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-15T20:56:00.816 INFO:tasks.workunit.client.0.smithi136.stderr:177552: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-15T20:56:00.816 INFO:tasks.workunit.client.0.smithi136.stderr:177552: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T20:56:00.816 INFO:tasks.workunit.client.0.smithi136.stderr:177552: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52602 2024-09-15T20:56:00.816 INFO:tasks.workunit.client.0.smithi136.stderr:177552: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.52602/$cluster-$name.asok' 2024-09-15T20:56:00.817 INFO:tasks.workunit.client.0.smithi136.stderr:177552: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.52602/$cluster-$name.asok' 2024-09-15T20:56:00.817 INFO:tasks.workunit.client.0.smithi136.stderr:177552: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-15T20:56:00.817 INFO:tasks.workunit.client.0.smithi136.stderr:177552: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-15T20:56:00.817 INFO:tasks.workunit.client.0.smithi136.stderr:177552: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-15T20:56:00.817 INFO:tasks.workunit.client.0.smithi136.stderr:177552: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T20:56:00.817 INFO:tasks.workunit.client.0.smithi136.stderr:177552: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T20:56:00.817 INFO:tasks.workunit.client.0.smithi136.stderr:177552: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T20:56:00.818 INFO:tasks.workunit.client.0.smithi136.stderr:177552: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T20:56:00.818 INFO:tasks.workunit.client.0.smithi136.stderr:177552: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-15T20:56:00.818 INFO:tasks.workunit.client.0.smithi136.stderr:177552: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-15T20:56:00.818 INFO:tasks.workunit.client.0.smithi136.stderr:177552: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-09-15T20:56:00.818 INFO:tasks.workunit.client.0.smithi136.stderr:177552: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.1 2024-09-15T20:56:00.818 INFO:tasks.workunit.client.0.smithi136.stderr:177552: start osd.1 2024-09-15T20:56:00.867 INFO:tasks.workunit.client.0.smithi136.stderr:177552: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 1 --fsid=ce4892c7-6c6d-4b4f-8e6f-07a3e11133e5 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.52602/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_reco177554: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2024-09-15T20:56:00.867 INFO:tasks.workunit.client.0.smithi136.stderr:177554: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2024-09-15T20:56:00.868 INFO:tasks.workunit.client.0.smithi136.stderr:177554: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=2 2024-09-15T20:56:00.868 INFO:tasks.workunit.client.0.smithi136.stderr:177554: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2024-09-15T20:56:00.868 INFO:tasks.workunit.client.0.smithi136.stderr:177554: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 2 SOMETHING list-attrs 2024-09-15T20:56:00.868 INFO:tasks.workunit.client.0.smithi136.stderr:177554: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-09-15T20:56:00.868 INFO:tasks.workunit.client.0.smithi136.stderr:177554: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2024-09-15T20:56:00.868 INFO:tasks.workunit.client.0.smithi136.stderr:177554: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=2 2024-09-15T20:56:00.868 INFO:tasks.workunit.client.0.smithi136.stderr:177554: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2024-09-15T20:56:00.868 INFO:tasks.workunit.client.0.smithi136.stderr:177554: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.2 2024-09-15T20:56:00.868 INFO:tasks.workunit.client.0.smithi136.stderr:177554: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-15T20:56:00.868 INFO:tasks.workunit.client.0.smithi136.stderr:177554: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-15T20:56:00.869 INFO:tasks.workunit.client.0.smithi136.stderr:177554: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-15T20:56:00.869 INFO:tasks.workunit.client.0.smithi136.stderr:177554: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-15T20:56:00.869 INFO:tasks.workunit.client.0.smithi136.stderr:177554: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-15T20:56:00.869 INFO:tasks.workunit.client.0.smithi136.stderr:177554: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-15T20:56:00.869 INFO:tasks.workunit.client.0.smithi136.stderr:177554: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 2 SOMETHING list-attrs 2024-09-15T20:56:00.869 INFO:tasks.workunit.client.0.smithi136.stderr:177554: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-15T20:56:00.870 INFO:tasks.workunit.client.0.smithi136.stderr:177554: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-15T20:56:00.870 INFO:tasks.workunit.client.0.smithi136.stderr:177554: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=2 2024-09-15T20:56:00.870 INFO:tasks.workunit.client.0.smithi136.stderr:177554: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-15T20:56:00.870 INFO:tasks.workunit.client.0.smithi136.stderr:177554: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/2 2024-09-15T20:56:00.870 INFO:tasks.workunit.client.0.smithi136.stderr:177554: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/2 SOMETHING list-attrs 2024-09-15T20:56:00.870 INFO:tasks.workunit.client.0.smithi136.stderr:177554: _ 2024-09-15T20:56:00.870 INFO:tasks.workunit.client.0.smithi136.stderr:177554: hinfo_key 2024-09-15T20:56:00.870 INFO:tasks.workunit.client.0.smithi136.stderr:177554: snapset 2024-09-15T20:56:00.870 INFO:tasks.workunit.client.0.smithi136.stderr:177554: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 2 2024-09-15T20:56:00.870 INFO:tasks.workunit.client.0.smithi136.stderr:177554: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2024-09-15T20:56:00.870 INFO:tasks.workunit.client.0.smithi136.stderr:177554: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-15T20:56:00.871 INFO:tasks.workunit.client.0.smithi136.stderr:177554: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=2 2024-09-15T20:56:00.871 INFO:tasks.workunit.client.0.smithi136.stderr:177554: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-15T20:56:00.871 INFO:tasks.workunit.client.0.smithi136.stderr:177554: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/2 2024-09-15T20:56:00.871 INFO:tasks.workunit.client.0.smithi136.stderr:177554: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=ce4892c7-6c6d-4b4f-8e6f-07a3e11133e5 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-15T20:56:00.871 INFO:tasks.workunit.client.0.smithi136.stderr:177554: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T20:56:00.871 INFO:tasks.workunit.client.0.smithi136.stderr:177554: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T20:56:00.872 INFO:tasks.workunit.client.0.smithi136.stderr:177554: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T20:56:00.872 INFO:tasks.workunit.client.0.smithi136.stderr:177554: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/2' 2024-09-15T20:56:00.872 INFO:tasks.workunit.client.0.smithi136.stderr:177554: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/2/journal' 2024-09-15T20:56:00.872 INFO:tasks.workunit.client.0.smithi136.stderr:177554: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-15T20:56:00.872 INFO:tasks.workunit.client.0.smithi136.stderr:177554: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-15T20:56:00.873 INFO:tasks.workunit.client.0.smithi136.stderr:177554: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-15T20:56:00.873 INFO:tasks.workunit.client.0.smithi136.stderr:177554: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-15T20:56:00.873 INFO:tasks.workunit.client.0.smithi136.stderr:177554: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-15T20:56:00.873 INFO:tasks.workunit.client.0.smithi136.stderr:177554: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-15T20:56:00.873 INFO:tasks.workunit.client.0.smithi136.stderr:177554: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-15T20:56:00.873 INFO:tasks.workunit.client.0.smithi136.stderr:177554: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T20:56:00.873 INFO:tasks.workunit.client.0.smithi136.stderr:177554: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52602 2024-09-15T20:56:00.874 INFO:tasks.workunit.client.0.smithi136.stderr:177554: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.52602/$cluster-$name.asok' 2024-09-15T20:56:00.874 INFO:tasks.workunit.client.0.smithi136.stderr:177554: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.52602/$cluster-$name.asok' 2024-09-15T20:56:00.874 INFO:tasks.workunit.client.0.smithi136.stderr:177554: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-15T20:56:00.874 INFO:tasks.workunit.client.0.smithi136.stderr:177554: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-15T20:56:00.874 INFO:tasks.workunit.client.0.smithi136.stderr:177554: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-15T20:56:00.874 INFO:tasks.workunit.client.0.smithi136.stderr:177554: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T20:56:00.875 INFO:tasks.workunit.client.0.smithi136.stderr:177554: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T20:56:00.875 INFO:tasks.workunit.client.0.smithi136.stderr:177554: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T20:56:00.875 INFO:tasks.workunit.client.0.smithi136.stderr:177554: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T20:56:00.875 INFO:tasks.workunit.client.0.smithi136.stderr:177554: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-15T20:56:00.875 INFO:tasks.workunit.client.0.smithi136.stderr:177554: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-15T20:56:00.875 INFO:tasks.workunit.client.0.smithi136.stderr:177554: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/2 2024-09-15T20:56:00.877 INFO:tasks.workunit.client.0.smithi136.stderr:177554: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.2 2024-09-15T20:56:00.877 INFO:tasks.workunit.client.0.smithi136.stderr:177554: start osd.2 2024-09-15T20:56:05.547 INFO:tasks.workunit.client.0.smithi136.stderr:177554: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 2 --fsid=ce4892c7-6c6d-4b4f-8e6f-07a3e11133e5 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.52602/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T20:56:05.547 INFO:tasks.workunit.client.0.smithi136.stderr:177554: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/2/whoami 2024-09-15T20:56:05.547 INFO:tasks.workunit.client.0.smithi136.stderr:177554: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 2 = 2 ']' 2024-09-15T20:56:05.547 INFO:tasks.workunit.client.0.smithi136.stderr:177554: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-15T20:56:05.547 INFO:tasks.workunit.client.0.smithi136.stderr:177554: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-15T20:56:05.547 INFO:tasks.workunit.client.0.smithi136.stderr:177554: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-15T20:56:05.547 INFO:tasks.workunit.client.0.smithi136.stderr:177554: 2024-09-15T20:56:00.897+0000 7f9d57145640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T20:56:05.547 INFO:tasks.workunit.client.0.smithi136.stderr:177554: 2024-09-15T20:56:01.004+0000 7f9d57145640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T20:56:05.547 INFO:tasks.workunit.client.0.smithi136.stderr:177554: 2024-09-15T20:56:01.006+0000 7f9d57145640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T20:56:05.547 INFO:tasks.workunit.client.0.smithi136.stderr:177554: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 2 2024-09-15T20:56:05.547 INFO:tasks.workunit.client.0.smithi136.stderr:177554: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-15T20:56:05.548 INFO:tasks.workunit.client.0.smithi136.stderr:177554: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=2 2024-09-15T20:56:05.548 INFO:tasks.workunit.client.0.smithi136.stderr:177554: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-15T20:56:05.548 INFO:tasks.workunit.client.0.smithi136.stderr:177554: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-15T20:56:05.548 INFO:tasks.workunit.client.0.smithi136.stderr:177554: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:56:05.548 INFO:tasks.workunit.client.0.smithi136.stderr:177554: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-15T20:56:05.548 INFO:tasks.workunit.client.0.smithi136.stderr:177554: 0 2024-09-15T20:56:05.548 INFO:tasks.workunit.client.0.smithi136.stderr:177554: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:56:05.548 INFO:tasks.workunit.client.0.smithi136.stderr:177554: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T20:56:05.548 INFO:tasks.workunit.client.0.smithi136.stderr:177554: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:56:05.548 INFO:tasks.workunit.client.0.smithi136.stderr:177554: 2024-09-15T20:56:02.130+0000 7f9d57145640 -1 Falling back to public interface 2024-09-15T20:56:05.548 INFO:tasks.workunit.client.0.smithi136.stderr:177554: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:56:05.548 INFO:tasks.workunit.client.0.smithi136.stderr:177554: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:56:05.548 INFO:tasks.workunit.client.0.smithi136.stderr:177554: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-15T20:56:05.548 INFO:tasks.workunit.client.0.smithi136.stderr:177554: 1 2024-09-15T20:56:05.549 INFO:tasks.workunit.client.0.smithi136.stderr:177554: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:56:05.549 INFO:tasks.workunit.client.0.smithi136.stderr:177554: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T20:56:05.549 INFO:tasks.workunit.client.0.smithi136.stderr:177554: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:56:05.549 INFO:tasks.workunit.client.0.smithi136.stderr:177554: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:56:05.549 INFO:tasks.workunit.client.0.smithi136.stderr:177554: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:56:05.549 INFO:tasks.workunit.client.0.smithi136.stderr:177554: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-15T20:56:05.549 INFO:tasks.workunit.client.0.smithi136.stderr:177554: 2 2024-09-15T20:56:05.549 INFO:tasks.workunit.client.0.smithi136.stderr:177554: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:56:05.549 INFO:tasks.workunit.client.0.smithi136.stderr:177554: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T20:56:05.549 INFO:tasks.workunit.client.0.smithi136.stderr:177554: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:56:05.549 INFO:tasks.workunit.client.0.smithi136.stderr:177554: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:56:05.549 INFO:tasks.workunit.client.0.smithi136.stderr:177554: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:56:05.549 INFO:tasks.workunit.client.0.smithi136.stderr:177554: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-15T20:56:05.550 INFO:tasks.workunit.client.0.smithi136.stderr:177554: 3 2024-09-15T20:56:05.550 INFO:tasks.workunit.client.0.smithi136.stderr:177554: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:56:05.550 INFO:tasks.workunit.client.0.smithi136.stderr:177554: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T20:56:05.550 INFO:tasks.workunit.client.0.smithi136.stderr:177554: 2024-09-15T20:56:05.255+0000 7f9d57145640 -1 osd.2 76 log_to_monitors true 2024-09-15T20:56:06.291 INFO:tasks.workunit.client.0.smithi136.stderr:177554: /home/ubuntu/cephtest/clone.client.0/qa/standalone/cevery_ops 2024-09-15T20:56:06.292 INFO:tasks.workunit.client.0.smithi136.stderr:177552: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-09-15T20:56:06.292 INFO:tasks.workunit.client.0.smithi136.stderr:177552: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 1 = 1 ']' 2024-09-15T20:56:06.292 INFO:tasks.workunit.client.0.smithi136.stderr:177552: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-15T20:56:06.292 INFO:tasks.workunit.client.0.smithi136.stderr:177552: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-15T20:56:06.292 INFO:tasks.workunit.client.0.smithi136.stderr:177552: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-15T20:56:06.292 INFO:tasks.workunit.client.0.smithi136.stderr:177552: 2024-09-15T20:56:00.843+0000 7f2d7b3bb640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T20:56:06.292 INFO:tasks.workunit.client.0.smithi136.stderr:177552: 2024-09-15T20:56:01.005+0000 7f2d7b3bb640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T20:56:06.293 INFO:tasks.workunit.client.0.smithi136.stderr:177552: 2024-09-15T20:56:01.006+0000 7f2d7b3bb640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T20:56:06.293 INFO:tasks.workunit.client.0.smithi136.stderr:177552: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 1 2024-09-15T20:56:06.293 INFO:tasks.workunit.client.0.smithi136.stderr:177552: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-15T20:56:06.293 INFO:tasks.workunit.client.0.smithi136.stderr:177552: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2024-09-15T20:56:06.293 INFO:tasks.workunit.client.0.smithi136.stderr:177552: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-15T20:56:06.293 INFO:tasks.workunit.client.0.smithi136.stderr:177552: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-15T20:56:06.293 INFO:tasks.workunit.client.0.smithi136.stderr:177552: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:56:06.294 INFO:tasks.workunit.client.0.smithi136.stderr:177552: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-15T20:56:06.294 INFO:tasks.workunit.client.0.smithi136.stderr:177552: 0 2024-09-15T20:56:06.294 INFO:tasks.workunit.client.0.smithi136.stderr:177552: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:56:06.294 INFO:tasks.workunit.client.0.smithi136.stderr:177552: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T20:56:06.294 INFO:tasks.workunit.client.0.smithi136.stderr:177552: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:56:06.294 INFO:tasks.workunit.client.0.smithi136.stderr:177552: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:56:06.294 INFO:tasks.workunit.client.0.smithi136.stderr:177552: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:56:06.295 INFO:tasks.workunit.client.0.smithi136.stderr:177552: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-15T20:56:06.295 INFO:tasks.workunit.client.0.smithi136.stderr:177552: 1 2024-09-15T20:56:06.295 INFO:tasks.workunit.client.0.smithi136.stderr:177552: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:56:06.295 INFO:tasks.workunit.client.0.smithi136.stderr:177552: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T20:56:06.295 INFO:tasks.workunit.client.0.smithi136.stderr:177552: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:56:06.295 INFO:tasks.workunit.client.0.smithi136.stderr:177552: 2024-09-15T20:56:02.874+0000 7f2d7b3bb640 -1 Falling back to public interface 2024-09-15T20:56:06.295 INFO:tasks.workunit.client.0.smithi136.stderr:177552: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:56:06.296 INFO:tasks.workunit.client.0.smithi136.stderr:177552: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:56:06.296 INFO:tasks.workunit.client.0.smithi136.stderr:177552: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-15T20:56:06.296 INFO:tasks.workunit.client.0.smithi136.stderr:177552: 2 2024-09-15T20:56:06.296 INFO:tasks.workunit.client.0.smithi136.stderr:177552: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:56:06.296 INFO:tasks.workunit.client.0.smithi136.stderr:177552: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T20:56:06.296 INFO:tasks.workunit.client.0.smithi136.stderr:177552: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:56:06.296 INFO:tasks.workunit.client.0.smithi136.stderr:177552: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:56:06.297 INFO:tasks.workunit.client.0.smithi136.stderr:177552: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:56:06.297 INFO:tasks.workunit.client.0.smithi136.stderr:177552: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-15T20:56:06.297 INFO:tasks.workunit.client.0.smithi136.stderr:177552: 3 2024-09-15T20:56:06.297 INFO:tasks.workunit.client.0.smithi136.stderr:177552: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:56:06.297 INFO:tasks.workunit.client.0.smithi136.stderr:177552: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T20:56:06.297 INFO:tasks.workunit.client.0.smithi136.stderr:177552: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:56:07.465 INFO:tasks.workunit.client.0.smithi136.stderr:177552: 2024-09-15T20:56:06.288+0000 7f2dph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:56:07.465 INFO:tasks.workunit.client.0.smithi136.stderr:177554: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:56:07.465 INFO:tasks.workunit.client.0.smithi136.stderr:177554: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:56:07.465 INFO:tasks.workunit.client.0.smithi136.stderr:177554: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-15T20:56:07.466 INFO:tasks.workunit.client.0.smithi136.stderr:177554: 4 2024-09-15T20:56:07.466 INFO:tasks.workunit.client.0.smithi136.stderr:177554: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:56:07.466 INFO:tasks.workunit.client.0.smithi136.stderr:177554: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T20:56:07.466 INFO:tasks.workunit.client.0.smithi136.stderr:177554: osd.2 up in weight 1 up_from 81 up_thru 81 down_at 77 last_clean_interval [75,76) [v2:127.0.0.1:6810/2860883189,v1:127.0.0.1:6811/2860883189] [v2:127.0.0.1:6812/2860883189,v1:127.0.0.1:6813/2860883189] exists,up f74e59aa-0b73-4bbf-a870-d7c571cf9f22 2024-09-15T20:56:07.466 INFO:tasks.workunit.client.0.smithi136.stderr:177554: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-15T20:56:07.466 INFO:tasks.workunit.client.0.smithi136.stderr:177554: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-15T20:56:07.466 INFO:tasks.workunit.client.0.smithi136.stderr:177554: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-15T20:56:07.466 INFO:tasks.workunit.client.0.smithi136.stderr:177554: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2024-09-15T20:56:07.466 INFO:tasks.workunit.client.0.smithi136.stderr:177554: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-15T20:56:07.466 INFO:tasks.workunit.client.0.smithi136.stderr:177554: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-15T20:56:07.466 INFO:tasks.workunit.client.0.smithi136.stderr:177554: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-15T20:56:07.466 INFO:tasks.workunit.client.0.smithi136.stderr:177554: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-15T20:56:07.467 INFO:tasks.workunit.client.0.smithi136.stderr:177554: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-15T20:56:07.467 INFO:tasks.workunit.client.0.smithi136.stderr:177554: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-15T20:56:07.467 INFO:tasks.workunit.client.0.smithi136.stderr:177554: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-15T20:56:07.467 INFO:tasks.workunit.client.0.smithi136.stderr:177554: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-15T20:56:07.467 INFO:tasks.workunit.client.0.smithi136.stderr:177554: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-15T20:56:07.467 INFO:tasks.workunit.client.0.smithi136.stderr:177554: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-15T20:56:07.467 INFO:tasks.workunit.client.0.smithi136.stderr:177554: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-15T20:56:07.467 INFO:tasks.workunit.client.0.smithi136.stderr:177554: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-15T20:56:07.467 INFO:tasks.workunit.client.0.smithi136.stderr:177554: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-15T20:56:07.467 INFO:tasks.workunit.client.0.smithi136.stderr:177554: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-15T20:56:07.467 INFO:tasks.workunit.client.0.smithi136.stderr:177554: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-15T20:56:07.467 INFO:tasks.workunit.client.0.smithi136.stderr:177554: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-15T20:56:07.467 INFO:tasks.workunit.client.0.smithi136.stderr:177554: 1 2024-09-15T20:56:07.467 INFO:tasks.workunit.client.0.smithi136.stderr:177554: 2 2024-09-15T20:56:07.468 INFO:tasks.workunit.client.0.smithi136.stderr:177554: 3' 2024-09-15T20:56:07.468 INFO:tasks.workunit.client.0.smithi136.stderr:177554: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-15T20:56:07.468 INFO:tasks.workunit.client.0.smithi136.stderr:177554: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T20:56:07.468 INFO:tasks.workunit.client.0.smithi136.stderr:177554: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-15T20:56:07.468 INFO:tasks.workunit.client.0.smithi136.stderr:177554: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836535 2024-09-15T20:56:07.468 INFO:tasks.workunit.client.0.smithi136.stderr:177554: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836535 2024-09-15T20:56:07.468 INFO:tasks.workunit.client.0.smithi136.stderr:177554: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836535' 2024-09-15T20:56:07.468 INFO:tasks.workunit.client.0.smithi136.stderr:177554: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T20:56:08.721 INFO:tasks.workunit.client.0.smithi136.stderr:177554: //home/ubuntu/cephtest/clone.client.0/qa/standalone7b3bb640 -1 osd.1 76 log_to_monitors true 2024-09-15T20:56:08.722 INFO:tasks.workunit.client.0.smithi136.stderr:177552: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:56:08.722 INFO:tasks.workunit.client.0.smithi136.stderr:177552: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:56:08.722 INFO:tasks.workunit.client.0.smithi136.stderr:177552: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-15T20:56:08.722 INFO:tasks.workunit.client.0.smithi136.stderr:177552: 4 2024-09-15T20:56:08.722 INFO:tasks.workunit.client.0.smithi136.stderr:177552: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:56:08.722 INFO:tasks.workunit.client.0.smithi136.stderr:177552: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T20:56:08.722 INFO:tasks.workunit.client.0.smithi136.stderr:177552: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:56:08.722 INFO:tasks.workunit.client.0.smithi136.stderr:177552: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:56:08.722 INFO:tasks.workunit.client.0.smithi136.stderr:177552: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:56:08.722 INFO:tasks.workunit.client.0.smithi136.stderr:177552: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-15T20:56:08.722 INFO:tasks.workunit.client.0.smithi136.stderr:177552: 5 2024-09-15T20:56:08.722 INFO:tasks.workunit.client.0.smithi136.stderr:177552: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:56:08.722 INFO:tasks.workunit.client.0.smithi136.stderr:177552: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T20:56:08.723 INFO:tasks.workunit.client.0.smithi136.stderr:177552: osd.1 up in weight 1 up_from 85 up_thru 75 down_at 77 last_clean_interval [71,76) [v2:127.0.0.1:6818/3128743965,v1:127.0.0.1:6819/3128743965] [v2:127.0.0.1:6820/3128743965,v1:127.0.0.1:6821/3128743965] exists,up 4e733256-a0ac-49e1-b991-7e6e4a060afc 2024-09-15T20:56:08.723 INFO:tasks.workunit.client.0.smithi136.stderr:177552: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-15T20:56:08.723 INFO:tasks.workunit.client.0.smithi136.stderr:177552: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-15T20:56:08.723 INFO:tasks.workunit.client.0.smithi136.stderr:177552: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-15T20:56:08.723 INFO:tasks.workunit.client.0.smithi136.stderr:177552: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2024-09-15T20:56:08.723 INFO:tasks.workunit.client.0.smithi136.stderr:177552: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-15T20:56:08.723 INFO:tasks.workunit.client.0.smithi136.stderr:177552: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-15T20:56:08.723 INFO:tasks.workunit.client.0.smithi136.stderr:177552: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-15T20:56:08.723 INFO:tasks.workunit.client.0.smithi136.stderr:177552: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-15T20:56:08.723 INFO:tasks.workunit.client.0.smithi136.stderr:177552: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-15T20:56:08.723 INFO:tasks.workunit.client.0.smithi136.stderr:177552: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-15T20:56:08.723 INFO:tasks.workunit.client.0.smithi136.stderr:177552: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-15T20:56:08.723 INFO:tasks.workunit.client.0.smithi136.stderr:177552: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-15T20:56:08.724 INFO:tasks.workunit.client.0.smithi136.stderr:177552: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-15T20:56:08.724 INFO:tasks.workunit.client.0.smithi136.stderr:177552: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-15T20:56:08.724 INFO:tasks.workunit.client.0.smithi136.stderr:177552: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-15T20:56:08.724 INFO:tasks.workunit.client.0.smithi136.stderr:177552: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-15T20:56:08.724 INFO:tasks.workunit.client.0.smithi136.stderr:177552: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-15T20:56:08.724 INFO:tasks.workunit.client.0.smithi136.stderr:177552: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-15T20:56:08.724 INFO:tasks.workunit.client.0.smithi136.stderr:177552: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-15T20:56:08.724 INFO:tasks.workunit.client.0.smithi136.stderr:177552: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-15T20:56:08.724 INFO:tasks.workunit.client.0.smithi136.stderr:177552: 1 2024-09-15T20:56:08.724 INFO:tasks.workunit.client.0.smithi136.stderr:177552: 2 2024-09-15T20:56:08.724 INFO:tasks.workunit.client.0.smithi136.stderr:177552: 3' 2024-09-15T20:56:08.724 INFO:tasks.workunit.client.0.smithi136.stderr:177552: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-15T20:56:08.724 INFO:tasks.workunit.client.0.smithi136.stderr:177552: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T20:56:10.016 INFO:tasks.workunit.client.0.smithi136.stderr:177552/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-15T20:56:10.016 INFO:tasks.workunit.client.0.smithi136.stderr:177554: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=2 2024-09-15T20:56:10.016 INFO:tasks.workunit.client.0.smithi136.stderr:177554: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 2 2024-09-15T20:56:10.016 INFO:tasks.workunit.client.0.smithi136.stderr:177554: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836535 1-2' 2024-09-15T20:56:10.016 INFO:tasks.workunit.client.0.smithi136.stderr:177554: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T20:56:10.016 INFO:tasks.workunit.client.0.smithi136.stderr:177554: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-15T20:56:10.016 INFO:tasks.workunit.client.0.smithi136.stderr:177554: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=347892350978 2024-09-15T20:56:10.016 INFO:tasks.workunit.client.0.smithi136.stderr:177554: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 347892350978 2024-09-15T20:56:10.016 INFO:tasks.workunit.client.0.smithi136.stderr:177554: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836535 1-2 2-347892350978' 2024-09-15T20:56:10.016 INFO:tasks.workunit.client.0.smithi136.stderr:177554: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T20:56:10.017 INFO:tasks.workunit.client.0.smithi136.stderr:177554: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-15T20:56:10.017 INFO:tasks.workunit.client.0.smithi136.stderr:177554: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=274877906956 2024-09-15T20:56:10.017 INFO:tasks.workunit.client.0.smithi136.stderr:177554: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 274877906956 2024-09-15T20:56:10.017 INFO:tasks.workunit.client.0.smithi136.stderr:177554: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836535 1-2 2-347892350978 3-274877906956' 2024-09-15T20:56:10.017 INFO:tasks.workunit.client.0.smithi136.stderr:177554: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T20:56:10.017 INFO:tasks.workunit.client.0.smithi136.stderr:177554: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836535 2024-09-15T20:56:10.017 INFO:tasks.workunit.client.0.smithi136.stderr:177554: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T20:56:10.017 INFO:tasks.workunit.client.0.smithi136.stderr:177554: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-15T20:56:10.017 INFO:tasks.workunit.client.0.smithi136.stderr:177554: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836535 2024-09-15T20:56:10.017 INFO:tasks.workunit.client.0.smithi136.stderr:177554: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T20:56:10.017 INFO:tasks.workunit.client.0.smithi136.stderr:177554: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836535 2024-09-15T20:56:10.017 INFO:tasks.workunit.client.0.smithi136.stderr:177554: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836535' 2024-09-15T20:56:10.017 INFO:tasks.workunit.client.0.smithi136.stderr:177554: waiting osd.0 seq 21474836535 2024-09-15T20:56:10.018 INFO:tasks.workunit.client.0.smithi136.stderr:177554: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T20:56:10.018 INFO:tasks.workunit.client.0.smithi136.stderr:177554: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836534 -lt 21474836535 2024-09-15T20:56:10.018 INFO:tasks.workunit.client.0.smithi136.stderr:177554: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-15T20:56:10.018 INFO:tasks.workunit.client.0.smithi136.stderr:177554: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-15T20:56:10.018 INFO:tasks.workunit.client.0.smithi136.stderr:177554: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T20:56:10.018 INFO:tasks.workunit.client.0.smithi136.stderr:177554: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836536 -lt 21474836535 2024-09-15T20:56:10.018 INFO:tasks.workunit.client.0.smithi136.stderr:177554: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T20:56:10.018 INFO:tasks.workunit.client.0.smithi136.stderr:177554: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-2 2024-09-15T20:56:10.018 INFO:tasks.workunit.client.0.smithi136.stderr:177554: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T20:56:10.018 INFO:tasks.workunit.client.0.smithi136.stderr:177554: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-15T20:56:10.018 INFO:tasks.workunit.client.0.smithi136.stderr:177554: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-2 2024-09-15T20:56:10.018 INFO:tasks.workunit.client.0.smithi136.stderr:177554: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T20:56:11.019 INFO:tasks.workunit.client.0.smithi136.stderr:177554: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: fl: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-15T20:56:11.019 INFO:tasks.workunit.client.0.smithi136.stderr:177552: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836537 2024-09-15T20:56:11.019 INFO:tasks.workunit.client.0.smithi136.stderr:177552: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836537 2024-09-15T20:56:11.019 INFO:tasks.workunit.client.0.smithi136.stderr:177552: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836537' 2024-09-15T20:56:11.019 INFO:tasks.workunit.client.0.smithi136.stderr:177552: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T20:56:11.019 INFO:tasks.workunit.client.0.smithi136.stderr:177552: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-15T20:56:11.019 INFO:tasks.workunit.client.0.smithi136.stderr:177552: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=365072220163 2024-09-15T20:56:11.019 INFO:tasks.workunit.client.0.smithi136.stderr:177552: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 365072220163 2024-09-15T20:56:11.019 INFO:tasks.workunit.client.0.smithi136.stderr:177552: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836537 1-365072220163' 2024-09-15T20:56:11.019 INFO:tasks.workunit.client.0.smithi136.stderr:177552: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T20:56:11.020 INFO:tasks.workunit.client.0.smithi136.stderr:177552: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-15T20:56:11.020 INFO:tasks.workunit.client.0.smithi136.stderr:177552: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=347892350979 2024-09-15T20:56:11.020 INFO:tasks.workunit.client.0.smithi136.stderr:177552: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 347892350979 2024-09-15T20:56:11.020 INFO:tasks.workunit.client.0.smithi136.stderr:177552: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836537 1-365072220163 2-347892350979' 2024-09-15T20:56:11.020 INFO:tasks.workunit.client.0.smithi136.stderr:177552: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T20:56:11.020 INFO:tasks.workunit.client.0.smithi136.stderr:177552: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-15T20:56:11.020 INFO:tasks.workunit.client.0.smithi136.stderr:177552: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=274877906957 2024-09-15T20:56:11.020 INFO:tasks.workunit.client.0.smithi136.stderr:177552: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 274877906957 2024-09-15T20:56:11.020 INFO:tasks.workunit.client.0.smithi136.stderr:177552: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836537 1-365072220163 2-347892350979 3-274877906957' 2024-09-15T20:56:11.020 INFO:tasks.workunit.client.0.smithi136.stderr:177552: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T20:56:11.021 INFO:tasks.workunit.client.0.smithi136.stderr:177552: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836537 2024-09-15T20:56:11.021 INFO:tasks.workunit.client.0.smithi136.stderr:177552: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T20:56:11.021 INFO:tasks.workunit.client.0.smithi136.stderr:177552: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-15T20:56:11.021 INFO:tasks.workunit.client.0.smithi136.stderr:177552: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836537 2024-09-15T20:56:11.021 INFO:tasks.workunit.client.0.smithi136.stderr:177552: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T20:56:11.021 INFO:tasks.workunit.client.0.smithi136.stderr:177552: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836537 2024-09-15T20:56:11.021 INFO:tasks.workunit.client.0.smithi136.stderr:177552: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836537' 2024-09-15T20:56:11.021 INFO:tasks.workunit.client.0.smithi136.stderr:177552: waiting osd.0 seq 21474836537 2024-09-15T20:56:11.021 INFO:tasks.workunit.client.0.smithi136.stderr:177552: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T20:56:11.021 INFO:tasks.workunit.client.0.smithi136.stderr:177552: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836536 -lt 21474836537 2024-09-15T20:56:11.021 INFO:tasks.workunit.client.0.smithi136.stderr:177552: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-15T20:56:11.022 INFO:tasks.workunit.client.0.smithi136.stderr:177552: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-15T20:56:11.022 INFO:tasks.workunit.client.0.smithi136.stderr:177552: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T20:56:11.022 INFO:tasks.workunit.client.0.smithi136.stderr:177552: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836537 -lt 21474836537 2024-09-15T20:56:11.245 INFO:tasks.workunit.client.0.smithi136.stderr:177552: /home/ubuntu/cephtest/clone.clush_pg_stats: seq=2 2024-09-15T20:56:11.245 INFO:tasks.workunit.client.0.smithi136.stderr:177554: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 2' 2024-09-15T20:56:11.245 INFO:tasks.workunit.client.0.smithi136.stderr:177554: waiting osd.1 seq 2 2024-09-15T20:56:11.245 INFO:tasks.workunit.client.0.smithi136.stderr:177554: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-15T20:56:11.245 INFO:tasks.workunit.client.0.smithi136.stderr:177554: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 2 -lt 2 2024-09-15T20:56:11.245 INFO:tasks.workunit.client.0.smithi136.stderr:177554: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T20:56:11.245 INFO:tasks.workunit.client.0.smithi136.stderr:177554: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-347892350978 2024-09-15T20:56:11.245 INFO:tasks.workunit.client.0.smithi136.stderr:177554: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T20:56:11.245 INFO:tasks.workunit.client.0.smithi136.stderr:177554: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-15T20:56:11.245 INFO:tasks.workunit.client.0.smithi136.stderr:177554: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-347892350978 2024-09-15T20:56:11.245 INFO:tasks.workunit.client.0.smithi136.stderr:177554: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T20:56:11.245 INFO:tasks.workunit.client.0.smithi136.stderr:177554: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=347892350978 2024-09-15T20:56:11.246 INFO:tasks.workunit.client.0.smithi136.stderr:177554: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 347892350978' 2024-09-15T20:56:11.246 INFO:tasks.workunit.client.0.smithi136.stderr:177554: waiting osd.2 seq 347892350978 2024-09-15T20:56:11.246 INFO:tasks.workunit.client.0.smithi136.stderr:177554: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-15T20:56:11.246 INFO:tasks.workunit.client.0.smithi136.stderr:177554: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 347892350978 -lt 347892350978 2024-09-15T20:56:11.246 INFO:tasks.workunit.client.0.smithi136.stderr:177554: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T20:56:11.246 INFO:tasks.workunit.client.0.smithi136.stderr:177554: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 3-274877906956 2024-09-15T20:56:11.246 INFO:tasks.workunit.client.0.smithi136.stderr:177554: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T20:56:11.246 INFO:tasks.workunit.client.0.smithi136.stderr:177554: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=3 2024-09-15T20:56:11.246 INFO:tasks.workunit.client.0.smithi136.stderr:177554: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 3-274877906956 2024-09-15T20:56:11.246 INFO:tasks.workunit.client.0.smithi136.stderr:177554: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T20:56:11.246 INFO:tasks.workunit.client.0.smithi136.stderr:177554: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=274877906956 2024-09-15T20:56:11.247 INFO:tasks.workunit.client.0.smithi136.stderr:177554: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.3 seq 274877906956' 2024-09-15T20:56:11.247 INFO:tasks.workunit.client.0.smithi136.stderr:177554: waiting osd.3 seq 274877906956 2024-09-15T20:56:11.247 INFO:tasks.workunit.client.0.smithi136.stderr:177554: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-15T20:56:11.247 INFO:tasks.workunit.client.0.smithi136.stderr:177554: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 274877906958 -lt 274877906956 2024-09-15T20:56:11.247 INFO:tasks.workunit.client.0.smithi136.stderr:177554: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-15T20:56:11.247 INFO:tasks.workunit.client.0.smithi136.stderr:177554: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T20:56:11.247 INFO:tasks.workunit.client.0.smithi136.stderr:177554: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T20:56:11.247 INFO:tasks.workunit.client.0.smithi136.stderr:177554: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 5 == 0 2024-09-15T20:56:11.247 INFO:tasks.workunit.client.0.smithi136.stderr:177554: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-15T20:56:11.247 INFO:tasks.workunit.client.0.smithi136.stderr:177554: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-15T20:56:11.247 INFO:tasks.workunit.client.0.smithi136.stderr:177554: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-15T20:56:11.247 INFO:tasks.workunit.client.0.smithi136.stderr:177554: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T20:56:11.247 INFO:tasks.workunit.client.0.smithi136.stderr:177554: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T20:56:12.291 INFO:tasks.workunit.client.0.smithi136.stderr:177554: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceient.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T20:56:12.291 INFO:tasks.workunit.client.0.smithi136.stderr:177552: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-365072220163 2024-09-15T20:56:12.291 INFO:tasks.workunit.client.0.smithi136.stderr:177552: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T20:56:12.291 INFO:tasks.workunit.client.0.smithi136.stderr:177552: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-15T20:56:12.291 INFO:tasks.workunit.client.0.smithi136.stderr:177552: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-365072220163 2024-09-15T20:56:12.291 INFO:tasks.workunit.client.0.smithi136.stderr:177552: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T20:56:12.292 INFO:tasks.workunit.client.0.smithi136.stderr:177552: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=365072220163 2024-09-15T20:56:12.292 INFO:tasks.workunit.client.0.smithi136.stderr:177552: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 365072220163' 2024-09-15T20:56:12.292 INFO:tasks.workunit.client.0.smithi136.stderr:177552: waiting osd.1 seq 365072220163 2024-09-15T20:56:12.292 INFO:tasks.workunit.client.0.smithi136.stderr:177552: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-15T20:56:12.292 INFO:tasks.workunit.client.0.smithi136.stderr:177552: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 365072220163 -lt 365072220163 2024-09-15T20:56:12.292 INFO:tasks.workunit.client.0.smithi136.stderr:177552: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T20:56:12.292 INFO:tasks.workunit.client.0.smithi136.stderr:177552: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-347892350979 2024-09-15T20:56:12.292 INFO:tasks.workunit.client.0.smithi136.stderr:177552: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T20:56:12.292 INFO:tasks.workunit.client.0.smithi136.stderr:177552: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-15T20:56:12.292 INFO:tasks.workunit.client.0.smithi136.stderr:177552: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-347892350979 2024-09-15T20:56:12.292 INFO:tasks.workunit.client.0.smithi136.stderr:177552: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T20:56:12.292 INFO:tasks.workunit.client.0.smithi136.stderr:177552: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=347892350979 2024-09-15T20:56:12.293 INFO:tasks.workunit.client.0.smithi136.stderr:177552: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 347892350979' 2024-09-15T20:56:12.293 INFO:tasks.workunit.client.0.smithi136.stderr:177552: waiting osd.2 seq 347892350979 2024-09-15T20:56:12.293 INFO:tasks.workunit.client.0.smithi136.stderr:177552: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-15T20:56:12.293 INFO:tasks.workunit.client.0.smithi136.stderr:177552: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 347892350980 -lt 347892350979 2024-09-15T20:56:12.293 INFO:tasks.workunit.client.0.smithi136.stderr:177552: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T20:56:12.293 INFO:tasks.workunit.client.0.smithi136.stderr:177552: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 3-274877906957 2024-09-15T20:56:12.293 INFO:tasks.workunit.client.0.smithi136.stderr:177552: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T20:56:12.293 INFO:tasks.workunit.client.0.smithi136.stderr:177552: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=3 2024-09-15T20:56:12.293 INFO:tasks.workunit.client.0.smithi136.stderr:177552: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 3-274877906957 2024-09-15T20:56:12.293 INFO:tasks.workunit.client.0.smithi136.stderr:177552: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T20:56:12.293 INFO:tasks.workunit.client.0.smithi136.stderr:177552: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=274877906957 2024-09-15T20:56:12.293 INFO:tasks.workunit.client.0.smithi136.stderr:177552: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.3 seq 274877906957' 2024-09-15T20:56:12.293 INFO:tasks.workunit.client.0.smithi136.stderr:177552: waiting osd.3 seq 274877906957 2024-09-15T20:56:12.294 INFO:tasks.workunit.client.0.smithi136.stderr:177552: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-15T20:56:12.294 INFO:tasks.workunit.client.0.smithi136.stderr:177552: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 274877906958 -lt 274877906957 2024-09-15T20:56:12.294 INFO:tasks.workunit.client.0.smithi136.stderr:177552: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-15T20:56:12.294 INFO:tasks.workunit.client.0.smithi136.stderr:177552: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T20:56:12.294 INFO:tasks.workunit.client.0.smithi136.stderr:177552: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T20:56:12.726 INFO:tasks.workunit.client.0.smithi136.stderr:177552: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test ph --format json pg dump pgs 2024-09-15T20:56:12.727 INFO:tasks.workunit.client.0.smithi136.stderr:177554: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T20:56:12.727 INFO:tasks.workunit.client.0.smithi136.stderr:177554: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=4 2024-09-15T20:56:12.727 INFO:tasks.workunit.client.0.smithi136.stderr:177554: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-15T20:56:12.727 INFO:tasks.workunit.client.0.smithi136.stderr:177554: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T20:56:12.727 INFO:tasks.workunit.client.0.smithi136.stderr:177554: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T20:56:12.727 INFO:tasks.workunit.client.0.smithi136.stderr:177554: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 = 5 2024-09-15T20:56:12.727 INFO:tasks.workunit.client.0.smithi136.stderr:177554: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 4 '!=' -1 2024-09-15T20:56:12.727 INFO:tasks.workunit.client.0.smithi136.stderr:177554: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: loop=0 2024-09-15T20:56:12.727 INFO:tasks.workunit.client.0.smithi136.stderr:177554: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1652: wait_for_clean: num_active_clean=4 2024-09-15T20:56:12.727 INFO:tasks.workunit.client.0.smithi136.stderr:177554: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-15T20:56:12.727 INFO:tasks.workunit.client.0.smithi136.stderr:177554: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 0.1 2024-09-15T20:56:12.728 INFO:tasks.workunit.client.0.smithi136.stderr:177554: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-15T20:56:12.728 INFO:tasks.workunit.client.0.smithi136.stderr:177554: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-15T20:56:12.728 INFO:tasks.workunit.client.0.smithi136.stderr:177554: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-15T20:56:12.728 INFO:tasks.workunit.client.0.smithi136.stderr:177554: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-15T20:56:12.728 INFO:tasks.workunit.client.0.smithi136.stderr:177554: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T20:56:12.728 INFO:tasks.workunit.client.0.smithi136.stderr:177554: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T20:56:12.728 INFO:tasks.workunit.client.0.smithi136.stderr:177554: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T20:56:12.728 INFO:tasks.workunit.client.0.smithi136.stderr:177554: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T20:56:12.728 INFO:tasks.workunit.client.0.smithi136.stderr:177554: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=4 2024-09-15T20:56:12.728 INFO:tasks.workunit.client.0.smithi136.stderr:177554: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-15T20:56:12.728 INFO:tasks.workunit.client.0.smithi136.stderr:177554: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T20:56:12.728 INFO:tasks.workunit.client.0.smithi136.stderr:177554: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T20:56:12.729 INFO:tasks.workunit.client.0.smithi136.stderr:177554: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 = 5 2024-09-15T20:56:12.729 INFO:tasks.workunit.client.0.smithi136.stderr:177554: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 4 '!=' 4 2024-09-15T20:56:12.729 INFO:tasks.workunit.client.0.smithi136.stderr:177554: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: get_is_making_recovery_progress 2024-09-15T20:56:12.729 INFO:tasks.workunit.client.0.smithi136.stderr:177554: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2024-09-15T20:56:12.729 INFO:tasks.workunit.client.0.smithi136.stderr:177554: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-15T20:56:12.729 INFO:tasks.workunit.client.0.smithi136.stderr:177554: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-15T20:56:12.729 INFO:tasks.workunit.client.0.smithi136.stderr:177554: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-15T20:56:13.692 INFO:tasks.workunit.client.0.smithi136.stderr:177554: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format js5 == 0 2024-09-15T20:56:13.692 INFO:tasks.workunit.client.0.smithi136.stderr:177552: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-15T20:56:13.692 INFO:tasks.workunit.client.0.smithi136.stderr:177552: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-15T20:56:13.692 INFO:tasks.workunit.client.0.smithi136.stderr:177552: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-15T20:56:13.692 INFO:tasks.workunit.client.0.smithi136.stderr:177552: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T20:56:13.692 INFO:tasks.workunit.client.0.smithi136.stderr:177552: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T20:56:13.693 INFO:tasks.workunit.client.0.smithi136.stderr:177552: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T20:56:13.693 INFO:tasks.workunit.client.0.smithi136.stderr:177552: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T20:56:13.693 INFO:tasks.workunit.client.0.smithi136.stderr:177552: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=4 2024-09-15T20:56:13.693 INFO:tasks.workunit.client.0.smithi136.stderr:177552: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-15T20:56:13.693 INFO:tasks.workunit.client.0.smithi136.stderr:177552: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T20:56:13.693 INFO:tasks.workunit.client.0.smithi136.stderr:177552: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T20:56:13.693 INFO:tasks.workunit.client.0.smithi136.stderr:177552: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 = 5 2024-09-15T20:56:13.693 INFO:tasks.workunit.client.0.smithi136.stderr:177552: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 4 '!=' -1 2024-09-15T20:56:13.693 INFO:tasks.workunit.client.0.smithi136.stderr:177552: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: loop=0 2024-09-15T20:56:13.693 INFO:tasks.workunit.client.0.smithi136.stderr:177552: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1652: wait_for_clean: num_active_clean=4 2024-09-15T20:56:13.693 INFO:tasks.workunit.client.0.smithi136.stderr:177552: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-15T20:56:13.693 INFO:tasks.workunit.client.0.smithi136.stderr:177552: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 0.1 2024-09-15T20:56:13.693 INFO:tasks.workunit.client.0.smithi136.stderr:177552: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-15T20:56:13.694 INFO:tasks.workunit.client.0.smithi136.stderr:177552: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-15T20:56:13.694 INFO:tasks.workunit.client.0.smithi136.stderr:177552: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-15T20:56:13.694 INFO:tasks.workunit.client.0.smithi136.stderr:177552: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-15T20:56:13.694 INFO:tasks.workunit.client.0.smithi136.stderr:177552: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T20:56:13.694 INFO:tasks.workunit.client.0.smithi136.stderr:177552: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T20:56:13.694 INFO:tasks.workunit.client.0.smithi136.stderr:177552: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T20:56:13.694 INFO:tasks.workunit.client.0.smithi136.stderr:177552: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T20:56:13.694 INFO:tasks.workunit.client.0.smithi136.stderr:177552: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=5 2024-09-15T20:56:13.694 INFO:tasks.workunit.client.0.smithi136.stderr:177552: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-15T20:56:13.694 INFO:tasks.workunit.client.0.smithi136.stderr:177552: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T20:56:13.694 INFO:tasks.workunit.client.0.smithi136.stderr:177552: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T20:56:13.694 INFO:tasks.workunit.client.0.smithi136.stderr:177552: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 = 5 2024-09-15T20:56:13.694 INFO:tasks.workunit.client.0.smithi136.stderr:177552: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-15T20:56:13.695 INFO:tasks.workunit.client.0.smithi136.stderr:177552: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-15T20:56:13.695 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: run_in_background: return 0 2024-09-15T20:56:13.695 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: wait_background: for pid in $pids 2024-09-15T20:56:13.695 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2175: wait_background: wait 177551 2024-09-15T20:56:13.986 INFO:tasks.workunit.client.0.smithi136.stderr:on status 2024-09-15T20:56:13.987 INFO:tasks.workunit.client.0.smithi136.stderr:177554: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-15T20:56:13.987 INFO:tasks.workunit.client.0.smithi136.stderr:177554: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=null 2024-09-15T20:56:13.987 INFO:tasks.workunit.client.0.smithi136.stderr:177554: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test null '!=' null 2024-09-15T20:56:13.987 INFO:tasks.workunit.client.0.smithi136.stderr:177554: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1655: wait_for_clean: (( 1 >= 13 )) 2024-09-15T20:56:13.987 INFO:tasks.workunit.client.0.smithi136.stderr:177554: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-15T20:56:13.987 INFO:tasks.workunit.client.0.smithi136.stderr:177554: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 0.2 2024-09-15T20:56:13.987 INFO:tasks.workunit.client.0.smithi136.stderr:177554: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-15T20:56:13.987 INFO:tasks.workunit.client.0.smithi136.stderr:177554: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-15T20:56:13.987 INFO:tasks.workunit.client.0.smithi136.stderr:177554: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-15T20:56:13.987 INFO:tasks.workunit.client.0.smithi136.stderr:177554: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-15T20:56:13.987 INFO:tasks.workunit.client.0.smithi136.stderr:177554: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T20:56:13.987 INFO:tasks.workunit.client.0.smithi136.stderr:177554: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T20:56:13.988 INFO:tasks.workunit.client.0.smithi136.stderr:177554: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T20:56:13.988 INFO:tasks.workunit.client.0.smithi136.stderr:177554: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T20:56:13.988 INFO:tasks.workunit.client.0.smithi136.stderr:177554: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=5 2024-09-15T20:56:13.988 INFO:tasks.workunit.client.0.smithi136.stderr:177554: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-15T20:56:13.988 INFO:tasks.workunit.client.0.smithi136.stderr:177554: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T20:56:13.988 INFO:tasks.workunit.client.0.smithi136.stderr:177554: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T20:56:13.988 INFO:tasks.workunit.client.0.smithi136.stderr:177554: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 = 5 2024-09-15T20:56:13.988 INFO:tasks.workunit.client.0.smithi136.stderr:177554: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-15T20:56:13.988 INFO:tasks.workunit.client.0.smithi136.stderr:177554: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-15T20:56:13.988 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: run_in_background: return 0 2024-09-15T20:56:13.988 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: wait_background: eval 'pids='\'''\''' 2024-09-15T20:56:13.988 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: wait_background: pids= 2024-09-15T20:56:13.988 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: wait_background: return 0 2024-09-15T20:56:13.989 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:212: corrupt_and_repair_two: return_code=0 2024-09-15T20:56:13.989 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:213: corrupt_and_repair_two: '[' 0 -ne 0 ']' 2024-09-15T20:56:13.989 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:215: corrupt_and_repair_two: rados --pool ecpool get SOMETHING td/osd-scrub-repair/COPY 2024-09-15T20:56:14.017 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:216: corrupt_and_repair_two: diff td/osd-scrub-repair/ORIGINAL td/osd-scrub-repair/COPY 2024-09-15T20:56:14.019 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:263: corrupt_and_repair_erasure_coded: corrupt_and_repair_two td/osd-scrub-repair ecpool 3 1 2024-09-15T20:56:14.020 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:185: corrupt_and_repair_two: local dir=td/osd-scrub-repair 2024-09-15T20:56:14.020 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:186: corrupt_and_repair_two: local poolname=ecpool 2024-09-15T20:56:14.020 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:187: corrupt_and_repair_two: local first=3 2024-09-15T20:56:14.020 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:188: corrupt_and_repair_two: local second=1 2024-09-15T20:56:14.020 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:193: corrupt_and_repair_two: pids= 2024-09-15T20:56:14.020 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:194: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 3 SOMETHING remove 2024-09-15T20:56:14.020 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2135: run_in_background: local pid_variable=pids 2024-09-15T20:56:14.020 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2136: run_in_background: shift 2024-09-15T20:56:14.020 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2140: run_in_background: eval 'pids+=" 180125"' 2024-09-15T20:56:14.020 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2140: run_in_background: pids+=' 180125' 2024-09-15T20:56:14.020 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:195: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 1 SOMETHING remove 2024-09-15T20:56:14.020 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2135: run_in_background: local pid_variable=pids 2024-09-15T20:56:14.021 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2136: run_in_background: shift 2024-09-15T20:56:14.021 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2140: run_in_background: eval 'pids+=" 180127"' 2024-09-15T20:56:14.021 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2140: run_in_background: pids+=' 180127' 2024-09-15T20:56:14.021 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: run_in_background: objectstore_tool td/osd-scrub-repair 3 SOMETHING remove 2024-09-15T20:56:14.021 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:196: corrupt_and_repair_two: wait_background pids 2024-09-15T20:56:14.021 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: wait_background: pids=' 180125 180127' 2024-09-15T20:56:14.021 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: wait_background: return_code=0 2024-09-15T20:56:14.021 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: wait_background: for pid in $pids 2024-09-15T20:56:14.021 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2175: wait_background: wait 180125 2024-09-15T20:56:14.021 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: run_in_background: sed 's/^/180128: /' 2024-09-15T20:56:14.021 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: run_in_background: objectstore_tool td/osd-scrub-repair 1 SOMETHING remove 2024-09-15T20:56:14.021 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: run_in_background: sed 's/^/180130: /' 2024-09-15T20:56:17.418 INFO:tasks.workunit.client.0.smithi136.stderr:180128: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2024-09-15T20:56:17.418 INFO:tasks.workunit.client.0.smithi136.stderr:180128: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2024-09-15T20:56:17.418 INFO:tasks.workunit.client.0.smithi136.stderr:180128: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=3 2024-09-15T20:56:17.418 INFO:tasks.workunit.client.0.smithi136.stderr:180128: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2024-09-15T20:56:17.418 INFO:tasks.workunit.client.0.smithi136.stderr:180128: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 3 SOMETHING remove 2024-09-15T20:56:17.419 INFO:tasks.workunit.client.0.smithi136.stderr:180128: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-09-15T20:56:17.419 INFO:tasks.workunit.client.0.smithi136.stderr:180128: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2024-09-15T20:56:17.419 INFO:tasks.workunit.client.0.smithi136.stderr:180128: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=3 2024-09-15T20:56:17.419 INFO:tasks.workunit.client.0.smithi136.stderr:180128: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2024-09-15T20:56:17.419 INFO:tasks.workunit.client.0.smithi136.stderr:180128: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.3 2024-09-15T20:56:17.419 INFO:tasks.workunit.client.0.smithi136.stderr:180128: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-15T20:56:17.419 INFO:tasks.workunit.client.0.smithi136.stderr:180128: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-15T20:56:17.419 INFO:tasks.workunit.client.0.smithi136.stderr:180128: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-15T20:56:17.419 INFO:tasks.workunit.client.0.smithi136.stderr:180128: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-15T20:56:17.419 INFO:tasks.workunit.client.0.smithi136.stderr:180128: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-15T20:56:17.419 INFO:tasks.workunit.client.0.smithi136.stderr:180128: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-15T20:56:17.419 INFO:tasks.workunit.client.0.smithi136.stderr:180128: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 3 SOMETHING remove 2024-09-15T20:56:17.420 INFO:tasks.workunit.client.0.smithi136.stderr:180128: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-15T20:56:17.420 INFO:tasks.workunit.client.0.smithi136.stderr:180128: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-15T20:56:17.420 INFO:tasks.workunit.client.0.smithi136.stderr:180128: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=3 2024-09-15T20:56:17.420 INFO:tasks.workunit.client.0.smithi136.stderr:180128: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-15T20:56:17.420 INFO:tasks.workunit.client.0.smithi136.stderr:180128: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/3 2024-09-15T20:56:17.420 INFO:tasks.workunit.client.0.smithi136.stderr:180128: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/3 SOMETHING remove 2024-09-15T20:56:17.420 INFO:tasks.workunit.client.0.smithi136.stderr:180128: remove 0#2:eb822e21:::SOMETHING:head# 2024-09-15T20:56:17.420 INFO:tasks.workunit.client.0.smithi136.stderr:180128: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 3 2024-09-15T20:56:17.420 INFO:tasks.workunit.client.0.smithi136.stderr:180128: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2024-09-15T20:56:17.420 INFO:tasks.workunit.client.0.smithi136.stderr:180128: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-15T20:56:17.420 INFO:tasks.workunit.client.0.smithi136.stderr:180128: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=3 2024-09-15T20:56:17.420 INFO:tasks.workunit.client.0.smithi136.stderr:180128: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-15T20:56:17.420 INFO:tasks.workunit.client.0.smithi136.stderr:180128: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/3 2024-09-15T20:56:17.420 INFO:tasks.workunit.client.0.smithi136.stderr:180128: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=ce4892c7-6c6d-4b4f-8e6f-07a3e11133e5 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-15T20:56:17.421 INFO:tasks.workunit.client.0.smithi136.stderr:180128: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T20:56:17.421 INFO:tasks.workunit.client.0.smithi136.stderr:180128: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T20:56:17.421 INFO:tasks.workunit.client.0.smithi136.stderr:180128: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T20:56:17.421 INFO:tasks.workunit.client.0.smithi136.stderr:180128: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/3' 2024-09-15T20:56:17.422 INFO:tasks.workunit.client.0.smithi136.stderr:180128: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/3/journal' 2024-09-15T20:56:17.422 INFO:tasks.workunit.client.0.smithi136.stderr:180128: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-15T20:56:17.422 INFO:tasks.workunit.client.0.smithi136.stderr:180128: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-15T20:56:17.422 INFO:tasks.workunit.client.0.smithi136.stderr:180128: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-15T20:56:17.422 INFO:tasks.workunit.client.0.smithi136.stderr:180128: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-15T20:56:17.422 INFO:tasks.workunit.client.0.smithi136.stderr:180128: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-15T20:56:17.422 INFO:tasks.workunit.client.0.smithi136.stderr:180128: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-15T20:56:17.422 INFO:tasks.workunit.client.0.smithi136.stderr:180128: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-15T20:56:17.422 INFO:tasks.workunit.client.0.smithi136.stderr:180128: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T20:56:17.422 INFO:tasks.workunit.client.0.smithi136.stderr:180128: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52602 2024-09-15T20:56:17.422 INFO:tasks.workunit.client.0.smithi136.stderr:180128: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.52602/$cluster-$name.asok' 2024-09-15T20:56:17.422 INFO:tasks.workunit.client.0.smithi136.stderr:180128: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.52602/$cluster-$name.asok' 2024-09-15T20:56:17.422 INFO:tasks.workunit.client.0.smithi136.stderr:180128: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-15T20:56:17.422 INFO:tasks.workunit.client.0.smithi136.stderr:180128: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-15T20:56:17.423 INFO:tasks.workunit.client.0.smithi136.stderr:180128: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-15T20:56:17.423 INFO:tasks.workunit.client.0.smithi136.stderr:180128: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T20:56:17.423 INFO:tasks.workunit.client.0.smithi136.stderr:180128: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T20:56:17.423 INFO:tasks.workunit.client.0.smithi136.stderr:180128: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T20:56:17.423 INFO:tasks.workunit.client.0.smithi136.stderr:180128: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T20:56:17.423 INFO:tasks.workunit.client.0.smithi136.stderr:180128: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-15T20:56:17.423 INFO:tasks.workunit.client.0.smithi136.stderr:180128: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-15T20:56:17.423 INFO:tasks.workunit.client.0.smithi136.stderr:180128: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/3 2024-09-15T20:56:17.423 INFO:tasks.workunit.client.0.smithi136.stderr:180128: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.3 2024-09-15T20:56:17.423 INFO:tasks.workunit.client.0.smithi136.stderr:180128: start osd.3 2024-09-15T20:56:17.423 INFO:tasks.workunit.client.0.smithi136.stderr:180128: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 3 --fsid=ce4892c7-6c6d-4b4f-8e6f-07a3e11133e5 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/3 --osd-journal=td/osd-scrub-repair/3/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.52602/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T20:56:17.539 INFO:tasks.workunit.client.0.smithi136.stderr:1180130: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2024-09-15T20:56:17.539 INFO:tasks.workunit.client.0.smithi136.stderr:180130: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2024-09-15T20:56:17.540 INFO:tasks.workunit.client.0.smithi136.stderr:180130: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=1 2024-09-15T20:56:17.540 INFO:tasks.workunit.client.0.smithi136.stderr:180130: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2024-09-15T20:56:17.540 INFO:tasks.workunit.client.0.smithi136.stderr:180130: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 SOMETHING remove 2024-09-15T20:56:17.540 INFO:tasks.workunit.client.0.smithi136.stderr:180130: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-09-15T20:56:17.540 INFO:tasks.workunit.client.0.smithi136.stderr:180130: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2024-09-15T20:56:17.540 INFO:tasks.workunit.client.0.smithi136.stderr:180130: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=1 2024-09-15T20:56:17.540 INFO:tasks.workunit.client.0.smithi136.stderr:180130: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2024-09-15T20:56:17.540 INFO:tasks.workunit.client.0.smithi136.stderr:180130: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-09-15T20:56:17.540 INFO:tasks.workunit.client.0.smithi136.stderr:180130: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-15T20:56:17.540 INFO:tasks.workunit.client.0.smithi136.stderr:180130: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-15T20:56:17.540 INFO:tasks.workunit.client.0.smithi136.stderr:180130: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-15T20:56:17.540 INFO:tasks.workunit.client.0.smithi136.stderr:180130: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-15T20:56:17.541 INFO:tasks.workunit.client.0.smithi136.stderr:180130: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-15T20:56:17.541 INFO:tasks.workunit.client.0.smithi136.stderr:180130: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-15T20:56:17.541 INFO:tasks.workunit.client.0.smithi136.stderr:180130: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 SOMETHING remove 2024-09-15T20:56:17.541 INFO:tasks.workunit.client.0.smithi136.stderr:180130: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-15T20:56:17.541 INFO:tasks.workunit.client.0.smithi136.stderr:180130: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-15T20:56:17.541 INFO:tasks.workunit.client.0.smithi136.stderr:180130: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=1 2024-09-15T20:56:17.541 INFO:tasks.workunit.client.0.smithi136.stderr:180130: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-15T20:56:17.541 INFO:tasks.workunit.client.0.smithi136.stderr:180130: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-09-15T20:56:17.541 INFO:tasks.workunit.client.0.smithi136.stderr:180130: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 SOMETHING remove 2024-09-15T20:56:17.541 INFO:tasks.workunit.client.0.smithi136.stderr:180130: remove 1#2:eb822e21:::SOMETHING:head# 2024-09-15T20:56:17.541 INFO:tasks.workunit.client.0.smithi136.stderr:180130: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2024-09-15T20:56:17.541 INFO:tasks.workunit.client.0.smithi136.stderr:180130: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2024-09-15T20:56:17.541 INFO:tasks.workunit.client.0.smithi136.stderr:180130: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-15T20:56:17.541 INFO:tasks.workunit.client.0.smithi136.stderr:180130: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=1 2024-09-15T20:56:17.542 INFO:tasks.workunit.client.0.smithi136.stderr:180130: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-15T20:56:17.542 INFO:tasks.workunit.client.0.smithi136.stderr:180130: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-09-15T20:56:17.542 INFO:tasks.workunit.client.0.smithi136.stderr:180130: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=ce4892c7-6c6d-4b4f-8e6f-07a3e11133e5 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-15T20:56:17.542 INFO:tasks.workunit.client.0.smithi136.stderr:180130: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T20:56:17.542 INFO:tasks.workunit.client.0.smithi136.stderr:180130: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T20:56:17.542 INFO:tasks.workunit.client.0.smithi136.stderr:180130: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T20:56:17.542 INFO:tasks.workunit.client.0.smithi136.stderr:180130: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-09-15T20:56:17.542 INFO:tasks.workunit.client.0.smithi136.stderr:180130: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-09-15T20:56:17.542 INFO:tasks.workunit.client.0.smithi136.stderr:180130: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-15T20:56:17.543 INFO:tasks.workunit.client.0.smithi136.stderr:180130: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-15T20:56:17.543 INFO:tasks.workunit.client.0.smithi136.stderr:180130: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-15T20:56:17.543 INFO:tasks.workunit.client.0.smithi136.stderr:180130: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-15T20:56:17.543 INFO:tasks.workunit.client.0.smithi136.stderr:180130: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-15T20:56:17.543 INFO:tasks.workunit.client.0.smithi136.stderr:180130: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-15T20:56:17.543 INFO:tasks.workunit.client.0.smithi136.stderr:180130: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-15T20:56:17.543 INFO:tasks.workunit.client.0.smithi136.stderr:180130: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T20:56:17.543 INFO:tasks.workunit.client.0.smithi136.stderr:180130: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52602 2024-09-15T20:56:17.543 INFO:tasks.workunit.client.0.smithi136.stderr:180130: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.52602/$cluster-$name.asok' 2024-09-15T20:56:17.543 INFO:tasks.workunit.client.0.smithi136.stderr:180130: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.52602/$cluster-$name.asok' 2024-09-15T20:56:17.543 INFO:tasks.workunit.client.0.smithi136.stderr:180130: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-15T20:56:17.543 INFO:tasks.workunit.client.0.smithi136.stderr:180130: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-15T20:56:17.544 INFO:tasks.workunit.client.0.smithi136.stderr:180130: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-15T20:56:17.544 INFO:tasks.workunit.client.0.smithi136.stderr:180130: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T20:56:17.545 INFO:tasks.workunit.client.0.smithi136.stderr:180130: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T20:56:17.545 INFO:tasks.workunit.client.0.smithi136.stderr:180130: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T20:56:17.545 INFO:tasks.workunit.client.0.smithi136.stderr:180130: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T20:56:17.545 INFO:tasks.workunit.client.0.smithi136.stderr:180130: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-15T20:56:17.545 INFO:tasks.workunit.client.0.smithi136.stderr:180130: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-15T20:56:17.545 INFO:tasks.workunit.client.0.smithi136.stderr:180130: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-09-15T20:56:17.546 INFO:tasks.workunit.client.0.smithi136.stderr:180130: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.1 2024-09-15T20:56:17.546 INFO:tasks.workunit.client.0.smithi136.stderr:180130: start osd.1 2024-09-15T20:56:17.546 INFO:tasks.workunit.client.0.smithi136.stderr:180130: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 1 --fsid=ce4892c7-6c6d-4b4f-8e6f-07a3e11133e5 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.52602/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T20:56:22.102 INFO:tasks.workunit.client.0.smithi136.stderr:180130: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-09-15T20:56:22.102 INFO:tasks.workunit.client.0.smithi136.stderr:180130: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 1 = 1 ']' 2024-09-15T20:56:22.102 INFO:tasks.workunit.client.0.smithi136.stderr:180130: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-15T20:56:22.102 INFO:tasks.workunit.client.0.smithi136.stderr:180130: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-15T20:56:22.102 INFO:tasks.workunit.client.0.smithi136.stderr:180130: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-15T20:56:22.102 INFO:tasks.workunit.client.0.smithi136.stderr:180130: 2024-09-15T20:56:17.567+0000 7ff04700d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T20:56:22.103 INFO:tasks.workunit.client.0.smithi136.stderr:180130: 2024-09-15T20:56:17.569+0000 7ff04700d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T20:56:22.103 INFO:tasks.workunit.client.0.smithi136.stderr:180130: 2024-09-15T20:56:17.571+0000 7ff04700d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T20:56:22.103 INFO:tasks.workunit.client.0.smithi136.stderr:180130: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 1 2024-09-15T20:56:22.103 INFO:tasks.workunit.client.0.smithi136.stderr:180130: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-15T20:56:22.103 INFO:tasks.workunit.client.0.smithi136.stderr:180130: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2024-09-15T20:56:22.103 INFO:tasks.workunit.client.0.smithi136.stderr:180130: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-15T20:56:22.103 INFO:tasks.workunit.client.0.smithi136.stderr:180130: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-15T20:56:22.103 INFO:tasks.workunit.client.0.smithi136.stderr:180130: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:56:22.103 INFO:tasks.workunit.client.0.smithi136.stderr:180130: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-15T20:56:22.103 INFO:tasks.workunit.client.0.smithi136.stderr:180130: 0 2024-09-15T20:56:22.103 INFO:tasks.workunit.client.0.smithi136.stderr:180130: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:56:22.103 INFO:tasks.workunit.client.0.smithi136.stderr:180130: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T20:56:22.103 INFO:tasks.workunit.client.0.smithi136.stderr:180130: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:56:22.104 INFO:tasks.workunit.client.0.smithi136.stderr:180130: 2024-09-15T20:56:18.915+0000 7ff04700d640 -1 Falling back to public interface 2024-09-15T20:56:22.104 INFO:tasks.workunit.client.0.smithi136.stderr:180130: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:56:22.104 INFO:tasks.workunit.client.0.smithi136.stderr:180130: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:56:22.104 INFO:tasks.workunit.client.0.smithi136.stderr:180130: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-15T20:56:22.104 INFO:tasks.workunit.client.0.smithi136.stderr:180130: 1 2024-09-15T20:56:22.104 INFO:tasks.workunit.client.0.smithi136.stderr:180130: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:56:22.104 INFO:tasks.workunit.client.0.smithi136.stderr:180130: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T20:56:22.104 INFO:tasks.workunit.client.0.smithi136.stderr:180130: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:56:22.104 INFO:tasks.workunit.client.0.smithi136.stderr:180130: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:56:22.104 INFO:tasks.workunit.client.0.smithi136.stderr:180130: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:56:22.104 INFO:tasks.workunit.client.0.smithi136.stderr:180130: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-15T20:56:22.104 INFO:tasks.workunit.client.0.smithi136.stderr:180130: 2 2024-09-15T20:56:22.104 INFO:tasks.workunit.client.0.smithi136.stderr:180130: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:56:22.104 INFO:tasks.workunit.client.0.smithi136.stderr:180130: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T20:56:22.105 INFO:tasks.workunit.client.0.smithi136.stderr:180130: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:56:22.105 INFO:tasks.workunit.client.0.smithi136.stderr:180130: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:56:22.105 INFO:tasks.workunit.client.0.smithi136.stderr:180130: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:56:22.105 INFO:tasks.workunit.client.0.smithi136.stderr:180130: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-15T20:56:22.105 INFO:tasks.workunit.client.0.smithi136.stderr:180130: 3 2024-09-15T20:56:22.105 INFO:tasks.workunit.client.0.smithi136.stderr:180130: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:56:22.105 INFO:tasks.workunit.client.0.smithi136.stderr:180130: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T20:56:22.105 INFO:tasks.workunit.client.0.smithi136.stderr:180130: 2024-09-15T20:56:21.827+0000 7ff04700d640 -1 osd.1 87 log_to_monitors true 2024-09-15T20:56:22.170 INFO:tasks.workunit.client.0.smithi136.stderr:180130: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers80128: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/3/whoami 2024-09-15T20:56:22.170 INFO:tasks.workunit.client.0.smithi136.stderr:180128: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 3 = 3 ']' 2024-09-15T20:56:22.170 INFO:tasks.workunit.client.0.smithi136.stderr:180128: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-15T20:56:22.171 INFO:tasks.workunit.client.0.smithi136.stderr:180128: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-15T20:56:22.171 INFO:tasks.workunit.client.0.smithi136.stderr:180128: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-15T20:56:22.171 INFO:tasks.workunit.client.0.smithi136.stderr:180128: 2024-09-15T20:56:17.449+0000 7f3ddcd58640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T20:56:22.171 INFO:tasks.workunit.client.0.smithi136.stderr:180128: 2024-09-15T20:56:17.451+0000 7f3ddcd58640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T20:56:22.171 INFO:tasks.workunit.client.0.smithi136.stderr:180128: 2024-09-15T20:56:17.452+0000 7f3ddcd58640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T20:56:22.171 INFO:tasks.workunit.client.0.smithi136.stderr:180128: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 3 2024-09-15T20:56:22.171 INFO:tasks.workunit.client.0.smithi136.stderr:180128: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-15T20:56:22.171 INFO:tasks.workunit.client.0.smithi136.stderr:180128: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=3 2024-09-15T20:56:22.171 INFO:tasks.workunit.client.0.smithi136.stderr:180128: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-15T20:56:22.171 INFO:tasks.workunit.client.0.smithi136.stderr:180128: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-15T20:56:22.171 INFO:tasks.workunit.client.0.smithi136.stderr:180128: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:56:22.171 INFO:tasks.workunit.client.0.smithi136.stderr:180128: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-15T20:56:22.171 INFO:tasks.workunit.client.0.smithi136.stderr:180128: 0 2024-09-15T20:56:22.171 INFO:tasks.workunit.client.0.smithi136.stderr:180128: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:56:22.172 INFO:tasks.workunit.client.0.smithi136.stderr:180128: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-15T20:56:22.172 INFO:tasks.workunit.client.0.smithi136.stderr:180128: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:56:22.172 INFO:tasks.workunit.client.0.smithi136.stderr:180128: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:56:22.172 INFO:tasks.workunit.client.0.smithi136.stderr:180128: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:56:22.172 INFO:tasks.workunit.client.0.smithi136.stderr:180128: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-15T20:56:22.172 INFO:tasks.workunit.client.0.smithi136.stderr:180128: 1 2024-09-15T20:56:22.172 INFO:tasks.workunit.client.0.smithi136.stderr:180128: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:56:22.172 INFO:tasks.workunit.client.0.smithi136.stderr:180128: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-15T20:56:22.172 INFO:tasks.workunit.client.0.smithi136.stderr:180128: 2024-09-15T20:56:19.068+0000 7f3ddcd58640 -1 Falling back to public interface 2024-09-15T20:56:22.172 INFO:tasks.workunit.client.0.smithi136.stderr:180128: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:56:22.172 INFO:tasks.workunit.client.0.smithi136.stderr:180128: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:56:22.172 INFO:tasks.workunit.client.0.smithi136.stderr:180128: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:56:22.172 INFO:tasks.workunit.client.0.smithi136.stderr:180128: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-15T20:56:22.173 INFO:tasks.workunit.client.0.smithi136.stderr:180128: 2 2024-09-15T20:56:22.173 INFO:tasks.workunit.client.0.smithi136.stderr:180128: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:56:22.173 INFO:tasks.workunit.client.0.smithi136.stderr:180128: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-15T20:56:22.173 INFO:tasks.workunit.client.0.smithi136.stderr:180128: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:56:22.173 INFO:tasks.workunit.client.0.smithi136.stderr:180128: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:56:22.173 INFO:tasks.workunit.client.0.smithi136.stderr:180128: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:56:22.173 INFO:tasks.workunit.client.0.smithi136.stderr:180128: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-15T20:56:22.173 INFO:tasks.workunit.client.0.smithi136.stderr:180128: 3 2024-09-15T20:56:22.173 INFO:tasks.workunit.client.0.smithi136.stderr:180128: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:56:22.173 INFO:tasks.workunit.client.0.smithi136.stderr:180128: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-15T20:56:22.173 INFO:tasks.workunit.client.0.smithi136.stderr:180128: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:56:24.168 INFO:tasks.workunit.client.0.smithi136.stderr:180128: 2024-09-15T20:56:22.167+0000 7f3ddcd58640 -.sh:976: wait_for_osd: sleep 1 2024-09-15T20:56:24.168 INFO:tasks.workunit.client.0.smithi136.stderr:180130: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:56:24.168 INFO:tasks.workunit.client.0.smithi136.stderr:180130: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:56:24.168 INFO:tasks.workunit.client.0.smithi136.stderr:180130: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-15T20:56:24.168 INFO:tasks.workunit.client.0.smithi136.stderr:180130: 4 2024-09-15T20:56:24.168 INFO:tasks.workunit.client.0.smithi136.stderr:180130: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:56:24.168 INFO:tasks.workunit.client.0.smithi136.stderr:180130: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T20:56:24.168 INFO:tasks.workunit.client.0.smithi136.stderr:180130: osd.1 up in weight 1 up_from 92 up_thru 92 down_at 88 last_clean_interval [85,87) [v2:127.0.0.1:6818/720537448,v1:127.0.0.1:6819/720537448] [v2:127.0.0.1:6820/720537448,v1:127.0.0.1:6821/720537448] exists,up 4e733256-a0ac-49e1-b991-7e6e4a060afc 2024-09-15T20:56:24.168 INFO:tasks.workunit.client.0.smithi136.stderr:180130: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-15T20:56:24.169 INFO:tasks.workunit.client.0.smithi136.stderr:180130: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-15T20:56:24.169 INFO:tasks.workunit.client.0.smithi136.stderr:180130: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-15T20:56:24.169 INFO:tasks.workunit.client.0.smithi136.stderr:180130: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2024-09-15T20:56:24.169 INFO:tasks.workunit.client.0.smithi136.stderr:180130: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-15T20:56:24.169 INFO:tasks.workunit.client.0.smithi136.stderr:180130: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-15T20:56:24.169 INFO:tasks.workunit.client.0.smithi136.stderr:180130: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-15T20:56:24.169 INFO:tasks.workunit.client.0.smithi136.stderr:180130: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-15T20:56:24.169 INFO:tasks.workunit.client.0.smithi136.stderr:180130: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-15T20:56:24.169 INFO:tasks.workunit.client.0.smithi136.stderr:180130: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-15T20:56:24.169 INFO:tasks.workunit.client.0.smithi136.stderr:180130: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-15T20:56:24.169 INFO:tasks.workunit.client.0.smithi136.stderr:180130: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-15T20:56:24.169 INFO:tasks.workunit.client.0.smithi136.stderr:180130: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-15T20:56:24.169 INFO:tasks.workunit.client.0.smithi136.stderr:180130: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-15T20:56:24.169 INFO:tasks.workunit.client.0.smithi136.stderr:180130: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-15T20:56:24.170 INFO:tasks.workunit.client.0.smithi136.stderr:180130: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-15T20:56:24.170 INFO:tasks.workunit.client.0.smithi136.stderr:180130: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-15T20:56:24.170 INFO:tasks.workunit.client.0.smithi136.stderr:180130: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-15T20:56:24.170 INFO:tasks.workunit.client.0.smithi136.stderr:180130: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-15T20:56:24.170 INFO:tasks.workunit.client.0.smithi136.stderr:180130: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-15T20:56:24.170 INFO:tasks.workunit.client.0.smithi136.stderr:180130: 1 2024-09-15T20:56:24.170 INFO:tasks.workunit.client.0.smithi136.stderr:180130: 2 2024-09-15T20:56:24.170 INFO:tasks.workunit.client.0.smithi136.stderr:180130: 3' 2024-09-15T20:56:24.170 INFO:tasks.workunit.client.0.smithi136.stderr:180130: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-15T20:56:24.170 INFO:tasks.workunit.client.0.smithi136.stderr:180130: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T20:56:24.170 INFO:tasks.workunit.client.0.smithi136.stderr:180130: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-15T20:56:24.170 INFO:tasks.workunit.client.0.smithi136.stderr:180130: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836541 2024-09-15T20:56:24.170 INFO:tasks.workunit.client.0.smithi136.stderr:180130: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836541 2024-09-15T20:56:24.170 INFO:tasks.workunit.client.0.smithi136.stderr:180130: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836541' 2024-09-15T20:56:24.171 INFO:tasks.workunit.client.0.smithi136.stderr:180130: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T20:56:25.100 INFO:tasks.workunit.client.0.smithi136.stderr:180130: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.1 osd.3 87 log_to_monitors true 2024-09-15T20:56:25.100 INFO:tasks.workunit.client.0.smithi136.stderr:180128: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:56:25.100 INFO:tasks.workunit.client.0.smithi136.stderr:180128: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:56:25.100 INFO:tasks.workunit.client.0.smithi136.stderr:180128: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-15T20:56:25.101 INFO:tasks.workunit.client.0.smithi136.stderr:180128: 4 2024-09-15T20:56:25.101 INFO:tasks.workunit.client.0.smithi136.stderr:180128: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:56:25.101 INFO:tasks.workunit.client.0.smithi136.stderr:180128: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-15T20:56:25.101 INFO:tasks.workunit.client.0.smithi136.stderr:180128: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:56:25.101 INFO:tasks.workunit.client.0.smithi136.stderr:180128: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:56:25.101 INFO:tasks.workunit.client.0.smithi136.stderr:180128: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:56:25.101 INFO:tasks.workunit.client.0.smithi136.stderr:180128: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-15T20:56:25.101 INFO:tasks.workunit.client.0.smithi136.stderr:180128: 5 2024-09-15T20:56:25.102 INFO:tasks.workunit.client.0.smithi136.stderr:180128: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:56:25.102 INFO:tasks.workunit.client.0.smithi136.stderr:180128: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-15T20:56:25.102 INFO:tasks.workunit.client.0.smithi136.stderr:180128: osd.3 up in weight 1 up_from 96 up_thru 85 down_at 88 last_clean_interval [64,87) [v2:127.0.0.1:6826/1171415262,v1:127.0.0.1:6827/1171415262] [v2:127.0.0.1:6828/1171415262,v1:127.0.0.1:6829/1171415262] exists,up b72fdf29-d86a-404d-a216-75e64b9c077b 2024-09-15T20:56:25.102 INFO:tasks.workunit.client.0.smithi136.stderr:180128: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-15T20:56:25.102 INFO:tasks.workunit.client.0.smithi136.stderr:180128: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-15T20:56:25.102 INFO:tasks.workunit.client.0.smithi136.stderr:180128: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-15T20:56:25.102 INFO:tasks.workunit.client.0.smithi136.stderr:180128: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2024-09-15T20:56:25.102 INFO:tasks.workunit.client.0.smithi136.stderr:180128: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-15T20:56:25.102 INFO:tasks.workunit.client.0.smithi136.stderr:180128: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-15T20:56:25.102 INFO:tasks.workunit.client.0.smithi136.stderr:180128: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-15T20:56:25.102 INFO:tasks.workunit.client.0.smithi136.stderr:180128: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-15T20:56:25.103 INFO:tasks.workunit.client.0.smithi136.stderr:180128: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-15T20:56:25.103 INFO:tasks.workunit.client.0.smithi136.stderr:180128: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-15T20:56:25.103 INFO:tasks.workunit.client.0.smithi136.stderr:180128: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-15T20:56:25.103 INFO:tasks.workunit.client.0.smithi136.stderr:180128: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-15T20:56:25.103 INFO:tasks.workunit.client.0.smithi136.stderr:180128: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-15T20:56:25.103 INFO:tasks.workunit.client.0.smithi136.stderr:180128: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-15T20:56:25.103 INFO:tasks.workunit.client.0.smithi136.stderr:180128: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-15T20:56:25.103 INFO:tasks.workunit.client.0.smithi136.stderr:180128: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-15T20:56:25.103 INFO:tasks.workunit.client.0.smithi136.stderr:180128: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-15T20:56:25.103 INFO:tasks.workunit.client.0.smithi136.stderr:180128: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-15T20:56:25.103 INFO:tasks.workunit.client.0.smithi136.stderr:180128: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-15T20:56:25.103 INFO:tasks.workunit.client.0.smithi136.stderr:180128: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-15T20:56:25.103 INFO:tasks.workunit.client.0.smithi136.stderr:180128: 1 2024-09-15T20:56:25.104 INFO:tasks.workunit.client.0.smithi136.stderr:180128: 2 2024-09-15T20:56:25.104 INFO:tasks.workunit.client.0.smithi136.stderr:180128: 3' 2024-09-15T20:56:25.104 INFO:tasks.workunit.client.0.smithi136.stderr:180128: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-15T20:56:25.104 INFO:tasks.workunit.client.0.smithi136.stderr:180128: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T20:56:25.629 INFO:tasks.workunit.client.0.smithi136.stderr:180128: //home/ush:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-15T20:56:25.629 INFO:tasks.workunit.client.0.smithi136.stderr:180130: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=395136991234 2024-09-15T20:56:25.629 INFO:tasks.workunit.client.0.smithi136.stderr:180130: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 395136991234 2024-09-15T20:56:25.629 INFO:tasks.workunit.client.0.smithi136.stderr:180130: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836541 1-395136991234' 2024-09-15T20:56:25.629 INFO:tasks.workunit.client.0.smithi136.stderr:180130: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T20:56:25.629 INFO:tasks.workunit.client.0.smithi136.stderr:180130: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-15T20:56:25.629 INFO:tasks.workunit.client.0.smithi136.stderr:180130: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=347892350983 2024-09-15T20:56:25.630 INFO:tasks.workunit.client.0.smithi136.stderr:180130: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 347892350983 2024-09-15T20:56:25.630 INFO:tasks.workunit.client.0.smithi136.stderr:180130: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836541 1-395136991234 2-347892350983' 2024-09-15T20:56:25.630 INFO:tasks.workunit.client.0.smithi136.stderr:180130: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T20:56:25.630 INFO:tasks.workunit.client.0.smithi136.stderr:180130: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-15T20:56:25.630 INFO:tasks.workunit.client.0.smithi136.stderr:180130: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=412316860418 2024-09-15T20:56:25.630 INFO:tasks.workunit.client.0.smithi136.stderr:180130: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 412316860418 2024-09-15T20:56:25.630 INFO:tasks.workunit.client.0.smithi136.stderr:180130: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836541 1-395136991234 2-347892350983 3-412316860418' 2024-09-15T20:56:25.630 INFO:tasks.workunit.client.0.smithi136.stderr:180130: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T20:56:25.630 INFO:tasks.workunit.client.0.smithi136.stderr:180130: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836541 2024-09-15T20:56:25.630 INFO:tasks.workunit.client.0.smithi136.stderr:180130: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T20:56:25.630 INFO:tasks.workunit.client.0.smithi136.stderr:180130: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-15T20:56:25.630 INFO:tasks.workunit.client.0.smithi136.stderr:180130: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836541 2024-09-15T20:56:25.630 INFO:tasks.workunit.client.0.smithi136.stderr:180130: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T20:56:25.630 INFO:tasks.workunit.client.0.smithi136.stderr:180130: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836541 2024-09-15T20:56:25.631 INFO:tasks.workunit.client.0.smithi136.stderr:180130: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836541' 2024-09-15T20:56:25.631 INFO:tasks.workunit.client.0.smithi136.stderr:180130: waiting osd.0 seq 21474836541 2024-09-15T20:56:25.631 INFO:tasks.workunit.client.0.smithi136.stderr:180130: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T20:56:25.631 INFO:tasks.workunit.client.0.smithi136.stderr:180130: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836541 -lt 21474836541 2024-09-15T20:56:25.631 INFO:tasks.workunit.client.0.smithi136.stderr:180130: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T20:56:25.631 INFO:tasks.workunit.client.0.smithi136.stderr:180130: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-395136991234 2024-09-15T20:56:25.631 INFO:tasks.workunit.client.0.smithi136.stderr:180130: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T20:56:25.631 INFO:tasks.workunit.client.0.smithi136.stderr:180130: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-15T20:56:25.631 INFO:tasks.workunit.client.0.smithi136.stderr:180130: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-395136991234 2024-09-15T20:56:25.631 INFO:tasks.workunit.client.0.smithi136.stderr:180130: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T20:56:25.631 INFO:tasks.workunit.client.0.smithi136.stderr:180130: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=395136991234 2024-09-15T20:56:25.631 INFO:tasks.workunit.client.0.smithi136.stderr:180130: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 395136991234' 2024-09-15T20:56:25.631 INFO:tasks.workunit.client.0.smithi136.stderr:180130: waiting osd.1 seq 395136991234 2024-09-15T20:56:25.631 INFO:tasks.workunit.client.0.smithi136.stderr:180130: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-15T20:56:27.365 INFO:tasks.workunit.client.0.smithi136.stderr:180130: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flbuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-15T20:56:27.366 INFO:tasks.workunit.client.0.smithi136.stderr:180128: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836542 2024-09-15T20:56:27.366 INFO:tasks.workunit.client.0.smithi136.stderr:180128: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836542 2024-09-15T20:56:27.366 INFO:tasks.workunit.client.0.smithi136.stderr:180128: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836542' 2024-09-15T20:56:27.366 INFO:tasks.workunit.client.0.smithi136.stderr:180128: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T20:56:27.366 INFO:tasks.workunit.client.0.smithi136.stderr:180128: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-15T20:56:27.366 INFO:tasks.workunit.client.0.smithi136.stderr:180128: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=395136991235 2024-09-15T20:56:27.366 INFO:tasks.workunit.client.0.smithi136.stderr:180128: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 395136991235 2024-09-15T20:56:27.366 INFO:tasks.workunit.client.0.smithi136.stderr:180128: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836542 1-395136991235' 2024-09-15T20:56:27.366 INFO:tasks.workunit.client.0.smithi136.stderr:180128: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T20:56:27.367 INFO:tasks.workunit.client.0.smithi136.stderr:180128: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-15T20:56:27.367 INFO:tasks.workunit.client.0.smithi136.stderr:180128: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=347892350985 2024-09-15T20:56:27.367 INFO:tasks.workunit.client.0.smithi136.stderr:180128: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 347892350985 2024-09-15T20:56:27.367 INFO:tasks.workunit.client.0.smithi136.stderr:180128: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836542 1-395136991235 2-347892350985' 2024-09-15T20:56:27.367 INFO:tasks.workunit.client.0.smithi136.stderr:180128: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T20:56:27.367 INFO:tasks.workunit.client.0.smithi136.stderr:180128: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-15T20:56:27.367 INFO:tasks.workunit.client.0.smithi136.stderr:180128: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=412316860419 2024-09-15T20:56:27.367 INFO:tasks.workunit.client.0.smithi136.stderr:180128: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 412316860419 2024-09-15T20:56:27.367 INFO:tasks.workunit.client.0.smithi136.stderr:180128: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836542 1-395136991235 2-347892350985 3-412316860419' 2024-09-15T20:56:27.367 INFO:tasks.workunit.client.0.smithi136.stderr:180128: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T20:56:27.367 INFO:tasks.workunit.client.0.smithi136.stderr:180128: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836542 2024-09-15T20:56:27.367 INFO:tasks.workunit.client.0.smithi136.stderr:180128: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T20:56:27.368 INFO:tasks.workunit.client.0.smithi136.stderr:180128: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-15T20:56:27.368 INFO:tasks.workunit.client.0.smithi136.stderr:180128: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836542 2024-09-15T20:56:27.368 INFO:tasks.workunit.client.0.smithi136.stderr:180128: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T20:56:27.368 INFO:tasks.workunit.client.0.smithi136.stderr:180128: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836542 2024-09-15T20:56:27.368 INFO:tasks.workunit.client.0.smithi136.stderr:180128: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836542' 2024-09-15T20:56:27.368 INFO:tasks.workunit.client.0.smithi136.stderr:180128: waiting osd.0 seq 21474836542 2024-09-15T20:56:27.368 INFO:tasks.workunit.client.0.smithi136.stderr:180128: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T20:56:27.368 INFO:tasks.workunit.client.0.smithi136.stderr:180128: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836541 -lt 21474836542 2024-09-15T20:56:27.368 INFO:tasks.workunit.client.0.smithi136.stderr:180128: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-15T20:56:27.368 INFO:tasks.workunit.client.0.smithi136.stderr:180128: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-15T20:56:27.368 INFO:tasks.workunit.client.0.smithi136.stderr:180128: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T20:56:27.368 INFO:tasks.workunit.client.0.smithi136.stderr:180128: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836542 -lt 21474836542 2024-09-15T20:56:27.890 INFO:tasks.workunit.client.0.smithi136.stderr:180128: /home/ubuntu/cephtest/clone.client.0/qa/ush_pg_stats: test 395136991234 -lt 395136991234 2024-09-15T20:56:27.891 INFO:tasks.workunit.client.0.smithi136.stderr:180130: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T20:56:27.891 INFO:tasks.workunit.client.0.smithi136.stderr:180130: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-347892350983 2024-09-15T20:56:27.891 INFO:tasks.workunit.client.0.smithi136.stderr:180130: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T20:56:27.891 INFO:tasks.workunit.client.0.smithi136.stderr:180130: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-15T20:56:27.891 INFO:tasks.workunit.client.0.smithi136.stderr:180130: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-347892350983 2024-09-15T20:56:27.891 INFO:tasks.workunit.client.0.smithi136.stderr:180130: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T20:56:27.891 INFO:tasks.workunit.client.0.smithi136.stderr:180130: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=347892350983 2024-09-15T20:56:27.891 INFO:tasks.workunit.client.0.smithi136.stderr:180130: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 347892350983' 2024-09-15T20:56:27.891 INFO:tasks.workunit.client.0.smithi136.stderr:180130: waiting osd.2 seq 347892350983 2024-09-15T20:56:27.891 INFO:tasks.workunit.client.0.smithi136.stderr:180130: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-15T20:56:27.891 INFO:tasks.workunit.client.0.smithi136.stderr:180130: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 347892350983 -lt 347892350983 2024-09-15T20:56:27.891 INFO:tasks.workunit.client.0.smithi136.stderr:180130: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T20:56:27.891 INFO:tasks.workunit.client.0.smithi136.stderr:180130: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 3-412316860418 2024-09-15T20:56:27.892 INFO:tasks.workunit.client.0.smithi136.stderr:180130: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T20:56:27.892 INFO:tasks.workunit.client.0.smithi136.stderr:180130: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=3 2024-09-15T20:56:27.892 INFO:tasks.workunit.client.0.smithi136.stderr:180130: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 3-412316860418 2024-09-15T20:56:27.892 INFO:tasks.workunit.client.0.smithi136.stderr:180130: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T20:56:27.892 INFO:tasks.workunit.client.0.smithi136.stderr:180130: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=412316860418 2024-09-15T20:56:27.892 INFO:tasks.workunit.client.0.smithi136.stderr:180130: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.3 seq 412316860418' 2024-09-15T20:56:27.892 INFO:tasks.workunit.client.0.smithi136.stderr:180130: waiting osd.3 seq 412316860418 2024-09-15T20:56:27.892 INFO:tasks.workunit.client.0.smithi136.stderr:180130: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-15T20:56:27.892 INFO:tasks.workunit.client.0.smithi136.stderr:180130: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 1 -lt 412316860418 2024-09-15T20:56:27.892 INFO:tasks.workunit.client.0.smithi136.stderr:180130: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-15T20:56:27.892 INFO:tasks.workunit.client.0.smithi136.stderr:180130: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-15T20:56:27.892 INFO:tasks.workunit.client.0.smithi136.stderr:180130: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-15T20:56:27.892 INFO:tasks.workunit.client.0.smithi136.stderr:180130: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 412316860419 -lt 412316860418 2024-09-15T20:56:27.893 INFO:tasks.workunit.client.0.smithi136.stderr:180130: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-15T20:56:27.893 INFO:tasks.workunit.client.0.smithi136.stderr:180130: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T20:56:27.893 INFO:tasks.workunit.client.0.smithi136.stderr:180130: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T20:56:27.893 INFO:tasks.workunit.client.0.smithi136.stderr:180130: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 5 == 0 2024-09-15T20:56:27.893 INFO:tasks.workunit.client.0.smithi136.stderr:180130: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-15T20:56:27.893 INFO:tasks.workunit.client.0.smithi136.stderr:180130: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-15T20:56:27.893 INFO:tasks.workunit.client.0.smithi136.stderr:180130: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-15T20:56:27.893 INFO:tasks.workunit.client.0.smithi136.stderr:180130: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T20:56:28.621 INFO:tasks.workunit.client.0.smithi136.stderr:180130: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T20:56:28.622 INFO:tasks.workunit.client.0.smithi136.stderr:180128: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-395136991235 2024-09-15T20:56:28.622 INFO:tasks.workunit.client.0.smithi136.stderr:180128: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T20:56:28.622 INFO:tasks.workunit.client.0.smithi136.stderr:180128: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-15T20:56:28.622 INFO:tasks.workunit.client.0.smithi136.stderr:180128: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-395136991235 2024-09-15T20:56:28.622 INFO:tasks.workunit.client.0.smithi136.stderr:180128: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T20:56:28.622 INFO:tasks.workunit.client.0.smithi136.stderr:180128: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=395136991235 2024-09-15T20:56:28.622 INFO:tasks.workunit.client.0.smithi136.stderr:180128: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 395136991235' 2024-09-15T20:56:28.622 INFO:tasks.workunit.client.0.smithi136.stderr:180128: waiting osd.1 seq 395136991235 2024-09-15T20:56:28.622 INFO:tasks.workunit.client.0.smithi136.stderr:180128: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-15T20:56:28.622 INFO:tasks.workunit.client.0.smithi136.stderr:180128: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 395136991235 -lt 395136991235 2024-09-15T20:56:28.622 INFO:tasks.workunit.client.0.smithi136.stderr:180128: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T20:56:28.622 INFO:tasks.workunit.client.0.smithi136.stderr:180128: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-347892350985 2024-09-15T20:56:28.622 INFO:tasks.workunit.client.0.smithi136.stderr:180128: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T20:56:28.623 INFO:tasks.workunit.client.0.smithi136.stderr:180128: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-15T20:56:28.623 INFO:tasks.workunit.client.0.smithi136.stderr:180128: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-347892350985 2024-09-15T20:56:28.623 INFO:tasks.workunit.client.0.smithi136.stderr:180128: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T20:56:28.623 INFO:tasks.workunit.client.0.smithi136.stderr:180128: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=347892350985 2024-09-15T20:56:28.623 INFO:tasks.workunit.client.0.smithi136.stderr:180128: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 347892350985' 2024-09-15T20:56:28.623 INFO:tasks.workunit.client.0.smithi136.stderr:180128: waiting osd.2 seq 347892350985 2024-09-15T20:56:28.623 INFO:tasks.workunit.client.0.smithi136.stderr:180128: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-15T20:56:28.623 INFO:tasks.workunit.client.0.smithi136.stderr:180128: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 347892350985 -lt 347892350985 2024-09-15T20:56:28.623 INFO:tasks.workunit.client.0.smithi136.stderr:180128: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T20:56:28.623 INFO:tasks.workunit.client.0.smithi136.stderr:180128: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 3-412316860419 2024-09-15T20:56:28.623 INFO:tasks.workunit.client.0.smithi136.stderr:180128: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T20:56:28.623 INFO:tasks.workunit.client.0.smithi136.stderr:180128: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=3 2024-09-15T20:56:28.623 INFO:tasks.workunit.client.0.smithi136.stderr:180128: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 3-412316860419 2024-09-15T20:56:28.623 INFO:tasks.workunit.client.0.smithi136.stderr:180128: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T20:56:28.624 INFO:tasks.workunit.client.0.smithi136.stderr:180128: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=412316860419 2024-09-15T20:56:28.624 INFO:tasks.workunit.client.0.smithi136.stderr:180128: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.3 seq 412316860419' 2024-09-15T20:56:28.624 INFO:tasks.workunit.client.0.smithi136.stderr:180128: waiting osd.3 seq 412316860419 2024-09-15T20:56:28.624 INFO:tasks.workunit.client.0.smithi136.stderr:180128: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-15T20:56:28.624 INFO:tasks.workunit.client.0.smithi136.stderr:180128: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 412316860419 -lt 412316860419 2024-09-15T20:56:28.624 INFO:tasks.workunit.client.0.smithi136.stderr:180128: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-15T20:56:28.624 INFO:tasks.workunit.client.0.smithi136.stderr:180128: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T20:56:28.624 INFO:tasks.workunit.client.0.smithi136.stderr:180128: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T20:56:28.624 INFO:tasks.workunit.client.0.smithi136.stderr:180128: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 5 == 0 2024-09-15T20:56:29.268 INFO:tasks.workunit.client.0.smithi136.stderr:180128: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-15T20:56:29.269 INFO:tasks.workunit.client.0.smithi136.stderr:180128: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-15T20:56:29.269 INFO:tasks.workunit.client.0.smithi136.stderr:180128: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-15T20:56:29.269 INFO:tasks.workunit.client.0.smithi136.stderr:180128: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T20:56:29.269 INFO:tasks.workunit.client.0.smithi136.stderr:180128: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T20:56:29.269 INFO:tasks.workunit.client.0.smithi136.stderr:180128: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T20:56:29.269 INFO:tasks.workunit.client.0.smithi136.stderr:180128: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T20:56:29.269 INFO:tasks.workunit.client.0.smithi136.stderr:180128: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=5 2024-09-15T20:56:29.269 INFO:tasks.workunit.client.0.smithi136.stderr:180128: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-15T20:56:29.269 INFO:tasks.workunit.client.0.smithi136.stderr:180128: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T20:56:29.269 INFO:tasks.workunit.client.0.smithi136.stderr:180128: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T20:56:29.269 INFO:tasks.workunit.client.0.smithi136.stderr:180128: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 = 5 2024-09-15T20:56:29.269 INFO:tasks.workunit.client.0.smithi136.stderr:180128: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-15T20:56:29.269 INFO:tasks.workunit.client.0.smithi136.stderr:180128: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-15T20:56:29.270 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: run_in_background: return 0 2024-09-15T20:56:29.270 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: wait_background: for pid in $pids 2024-09-15T20:56:29.270 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2175: wait_background: wait 180127 2024-09-15T20:56:29.311 INFO:tasks.workunit.client.0.smithi136.stderr:' 2024-09-15T20:56:29.311 INFO:tasks.workunit.client.0.smithi136.stderr:180130: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T20:56:29.311 INFO:tasks.workunit.client.0.smithi136.stderr:180130: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T20:56:29.311 INFO:tasks.workunit.client.0.smithi136.stderr:180130: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=4 2024-09-15T20:56:29.311 INFO:tasks.workunit.client.0.smithi136.stderr:180130: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-15T20:56:29.311 INFO:tasks.workunit.client.0.smithi136.stderr:180130: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T20:56:29.311 INFO:tasks.workunit.client.0.smithi136.stderr:180130: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T20:56:29.311 INFO:tasks.workunit.client.0.smithi136.stderr:180130: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 = 5 2024-09-15T20:56:29.311 INFO:tasks.workunit.client.0.smithi136.stderr:180130: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 4 '!=' -1 2024-09-15T20:56:29.312 INFO:tasks.workunit.client.0.smithi136.stderr:180130: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: loop=0 2024-09-15T20:56:29.312 INFO:tasks.workunit.client.0.smithi136.stderr:180130: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1652: wait_for_clean: num_active_clean=4 2024-09-15T20:56:29.312 INFO:tasks.workunit.client.0.smithi136.stderr:180130: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-15T20:56:29.312 INFO:tasks.workunit.client.0.smithi136.stderr:180130: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 0.1 2024-09-15T20:56:29.312 INFO:tasks.workunit.client.0.smithi136.stderr:180130: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-15T20:56:29.312 INFO:tasks.workunit.client.0.smithi136.stderr:180130: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-15T20:56:29.312 INFO:tasks.workunit.client.0.smithi136.stderr:180130: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-15T20:56:29.312 INFO:tasks.workunit.client.0.smithi136.stderr:180130: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-15T20:56:29.312 INFO:tasks.workunit.client.0.smithi136.stderr:180130: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T20:56:29.312 INFO:tasks.workunit.client.0.smithi136.stderr:180130: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T20:56:29.312 INFO:tasks.workunit.client.0.smithi136.stderr:180130: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T20:56:29.312 INFO:tasks.workunit.client.0.smithi136.stderr:180130: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T20:56:29.312 INFO:tasks.workunit.client.0.smithi136.stderr:180130: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=5 2024-09-15T20:56:29.312 INFO:tasks.workunit.client.0.smithi136.stderr:180130: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-15T20:56:29.313 INFO:tasks.workunit.client.0.smithi136.stderr:180130: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T20:56:29.313 INFO:tasks.workunit.client.0.smithi136.stderr:180130: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T20:56:29.313 INFO:tasks.workunit.client.0.smithi136.stderr:180130: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 = 5 2024-09-15T20:56:29.313 INFO:tasks.workunit.client.0.smithi136.stderr:180130: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-15T20:56:29.313 INFO:tasks.workunit.client.0.smithi136.stderr:180130: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-15T20:56:29.313 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: run_in_background: return 0 2024-09-15T20:56:29.313 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: wait_background: eval 'pids='\'''\''' 2024-09-15T20:56:29.313 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: wait_background: pids= 2024-09-15T20:56:29.313 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: wait_background: return 0 2024-09-15T20:56:29.313 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:197: corrupt_and_repair_two: return_code=0 2024-09-15T20:56:29.313 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:198: corrupt_and_repair_two: '[' 0 -ne 0 ']' 2024-09-15T20:56:29.313 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:203: corrupt_and_repair_two: get_pg ecpool SOMETHING 2024-09-15T20:56:29.313 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1084: get_pg: local poolname=ecpool 2024-09-15T20:56:29.313 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1085: get_pg: local objectname=SOMETHING 2024-09-15T20:56:29.313 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1087: get_pg: ceph --format json osd map ecpool SOMETHING 2024-09-15T20:56:29.314 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1087: get_pg: jq -r .pgid 2024-09-15T20:56:29.641 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:203: corrupt_and_repair_two: local pg=2.0 2024-09-15T20:56:29.641 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:204: corrupt_and_repair_two: repair 2.0 2024-09-15T20:56:29.641 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1846: repair: local pgid=2.0 2024-09-15T20:56:29.641 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1847: repair: get_last_scrub_stamp 2.0 2024-09-15T20:56:29.641 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2024-09-15T20:56:29.641 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-15T20:56:29.641 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-15T20:56:29.641 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-09-15T20:56:29.914 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1847: repair: local last_scrub=2024-09-15T20:55:50.632381+0000 2024-09-15T20:56:29.915 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1848: repair: ceph pg repair 2.0 2024-09-15T20:56:30.174 INFO:tasks.workunit.client.0.smithi136.stderr:instructing pg 2.0s0 on osd.3 to repair 2024-09-15T20:56:30.187 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1849: repair: wait_for_scrub 2.0 2024-09-15T20:55:50.632381+0000 2024-09-15T20:56:30.187 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: local pgid=2.0 2024-09-15T20:56:30.187 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2018: wait_for_scrub: local last_scrub=2024-09-15T20:55:50.632381+0000 2024-09-15T20:56:30.187 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2019: wait_for_scrub: local sname=last_scrub_stamp 2024-09-15T20:56:30.188 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i=0 )) 2024-09-15T20:56:30.188 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-15T20:56:30.188 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-09-15T20:56:30.188 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2024-09-15T20:56:30.188 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-15T20:56:30.188 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-15T20:56:30.188 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-09-15T20:56:30.457 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-15T20:55:50.632381+0000 '>' 2024-09-15T20:55:50.632381+0000 2024-09-15T20:56:30.458 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-15T20:56:31.458 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-15T20:56:31.459 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-15T20:56:31.460 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-09-15T20:56:31.460 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2024-09-15T20:56:31.460 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-15T20:56:31.460 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-15T20:56:31.460 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-09-15T20:56:31.743 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-15T20:55:50.632381+0000 '>' 2024-09-15T20:55:50.632381+0000 2024-09-15T20:56:31.743 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-15T20:56:32.745 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-15T20:56:32.745 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-15T20:56:32.745 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-09-15T20:56:32.746 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2024-09-15T20:56:32.746 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-15T20:56:32.746 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-15T20:56:32.746 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-09-15T20:56:33.037 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-15T20:55:50.632381+0000 '>' 2024-09-15T20:55:50.632381+0000 2024-09-15T20:56:33.037 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-15T20:56:34.038 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-15T20:56:34.038 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-15T20:56:34.039 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-09-15T20:56:34.039 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2024-09-15T20:56:34.039 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-15T20:56:34.039 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-15T20:56:34.039 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-09-15T20:56:34.319 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-15T20:55:50.632381+0000 '>' 2024-09-15T20:55:50.632381+0000 2024-09-15T20:56:34.319 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-15T20:56:35.320 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-15T20:56:35.320 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-15T20:56:35.320 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-09-15T20:56:35.321 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2024-09-15T20:56:35.321 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-15T20:56:35.321 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-15T20:56:35.321 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-09-15T20:56:35.603 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-15T20:56:31.115062+0000 '>' 2024-09-15T20:55:50.632381+0000 2024-09-15T20:56:35.603 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2023: wait_for_scrub: return 0 2024-09-15T20:56:35.603 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1850: repair: sleep 2 2024-09-15T20:56:37.605 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:208: corrupt_and_repair_two: pids= 2024-09-15T20:56:37.605 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:209: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 3 SOMETHING list-attrs 2024-09-15T20:56:37.606 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2135: run_in_background: local pid_variable=pids 2024-09-15T20:56:37.606 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2136: run_in_background: shift 2024-09-15T20:56:37.606 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2140: run_in_background: eval 'pids+=" 183119"' 2024-09-15T20:56:37.606 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2140: run_in_background: pids+=' 183119' 2024-09-15T20:56:37.606 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:210: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 1 SOMETHING list-attrs 2024-09-15T20:56:37.606 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2135: run_in_background: local pid_variable=pids 2024-09-15T20:56:37.606 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2136: run_in_background: shift 2024-09-15T20:56:37.606 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2140: run_in_background: eval 'pids+=" 183121"' 2024-09-15T20:56:37.606 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: run_in_background: objectstore_tool td/osd-scrub-repair 3 SOMETHING list-attrs 2024-09-15T20:56:37.606 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2140: run_in_background: pids+=' 183121' 2024-09-15T20:56:37.606 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:211: corrupt_and_repair_two: wait_background pids 2024-09-15T20:56:37.606 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: wait_background: pids=' 183119 183121' 2024-09-15T20:56:37.606 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: wait_background: return_code=0 2024-09-15T20:56:37.607 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: wait_background: for pid in $pids 2024-09-15T20:56:37.607 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2175: wait_background: wait 183119 2024-09-15T20:56:37.607 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: run_in_background: sed 's/^/183122: /' 2024-09-15T20:56:37.607 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: run_in_background: objectstore_tool td/osd-scrub-repair 1 SOMETHING list-attrs 2024-09-15T20:56:37.607 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: run_in_background: sed 's/^/183124: /' 2024-09-15T20:56:38.740 INFO:tasks.workunit.client.0.smithi136.stderr:183124: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2024-09-15T20:56:38.740 INFO:tasks.workunit.client.0.smithi136.stderr:183124: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2024-09-15T20:56:38.740 INFO:tasks.workunit.client.0.smithi136.stderr:183124: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=1 2024-09-15T20:56:38.740 INFO:tasks.workunit.client.0.smithi136.stderr:183124: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2024-09-15T20:56:38.741 INFO:tasks.workunit.client.0.smithi136.stderr:183124: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 SOMETHING list-attrs 2024-09-15T20:56:38.741 INFO:tasks.workunit.client.0.smithi136.stderr:183124: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-09-15T20:56:38.741 INFO:tasks.workunit.client.0.smithi136.stderr:183124: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2024-09-15T20:56:38.741 INFO:tasks.workunit.client.0.smithi136.stderr:183124: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=1 2024-09-15T20:56:38.741 INFO:tasks.workunit.client.0.smithi136.stderr:183124: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2024-09-15T20:56:38.741 INFO:tasks.workunit.client.0.smithi136.stderr:183124: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-09-15T20:56:38.741 INFO:tasks.workunit.client.0.smithi136.stderr:183124: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-15T20:56:38.741 INFO:tasks.workunit.client.0.smithi136.stderr:183124: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-15T20:56:38.741 INFO:tasks.workunit.client.0.smithi136.stderr:183124: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-15T20:56:38.741 INFO:tasks.workunit.client.0.smithi136.stderr:183124: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-15T20:56:38.742 INFO:tasks.workunit.client.0.smithi136.stderr:183124: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-15T20:56:38.742 INFO:tasks.workunit.client.0.smithi136.stderr:183124: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-15T20:56:38.742 INFO:tasks.workunit.client.0.smithi136.stderr:183124: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 SOMETHING list-attrs 2024-09-15T20:56:38.742 INFO:tasks.workunit.client.0.smithi136.stderr:183124: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-15T20:56:38.742 INFO:tasks.workunit.client.0.smithi136.stderr:183124: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-15T20:56:38.742 INFO:tasks.workunit.client.0.smithi136.stderr:183124: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=1 2024-09-15T20:56:38.742 INFO:tasks.workunit.client.0.smithi136.stderr:183124: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-15T20:56:38.742 INFO:tasks.workunit.client.0.smithi136.stderr:183124: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-09-15T20:56:38.742 INFO:tasks.workunit.client.0.smithi136.stderr:183124: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 SOMETHING list-attrs 2024-09-15T20:56:38.742 INFO:tasks.workunit.client.0.smithi136.stderr:183124: _ 2024-09-15T20:56:38.742 INFO:tasks.workunit.client.0.smithi136.stderr:183124: hinfo_key 2024-09-15T20:56:38.742 INFO:tasks.workunit.client.0.smithi136.stderr:183124: snapset 2024-09-15T20:56:38.742 INFO:tasks.workunit.client.0.smithi136.stderr:183124: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2024-09-15T20:56:38.742 INFO:tasks.workunit.client.0.smithi136.stderr:183124: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2024-09-15T20:56:38.743 INFO:tasks.workunit.client.0.smithi136.stderr:183124: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-15T20:56:38.743 INFO:tasks.workunit.client.0.smithi136.stderr:183124: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=1 2024-09-15T20:56:38.743 INFO:tasks.workunit.client.0.smithi136.stderr:183124: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-15T20:56:38.743 INFO:tasks.workunit.client.0.smithi136.stderr:183124: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-09-15T20:56:38.743 INFO:tasks.workunit.client.0.smithi136.stderr:183124: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=ce4892c7-6c6d-4b4f-8e6f-07a3e11133e5 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-15T20:56:38.743 INFO:tasks.workunit.client.0.smithi136.stderr:183124: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T20:56:38.743 INFO:tasks.workunit.client.0.smithi136.stderr:183124: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T20:56:38.743 INFO:tasks.workunit.client.0.smithi136.stderr:183124: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T20:56:38.744 INFO:tasks.workunit.client.0.smithi136.stderr:183124: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-09-15T20:56:38.744 INFO:tasks.workunit.client.0.smithi136.stderr:183124: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-09-15T20:56:38.744 INFO:tasks.workunit.client.0.smithi136.stderr:183124: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-15T20:56:38.744 INFO:tasks.workunit.client.0.smithi136.stderr:183124: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-15T20:56:38.744 INFO:tasks.workunit.client.0.smithi136.stderr:183124: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-15T20:56:38.744 INFO:tasks.workunit.client.0.smithi136.stderr:183124: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-15T20:56:38.744 INFO:tasks.workunit.client.0.smithi136.stderr:183124: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-15T20:56:38.744 INFO:tasks.workunit.client.0.smithi136.stderr:183124: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-15T20:56:38.744 INFO:tasks.workunit.client.0.smithi136.stderr:183124: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-15T20:56:38.744 INFO:tasks.workunit.client.0.smithi136.stderr:183124: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T20:56:38.744 INFO:tasks.workunit.client.0.smithi136.stderr:183124: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52602 2024-09-15T20:56:38.744 INFO:tasks.workunit.client.0.smithi136.stderr:183124: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.52602/$cluster-$name.asok' 2024-09-15T20:56:38.744 INFO:tasks.workunit.client.0.smithi136.stderr:183124: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.52602/$cluster-$name.asok' 2024-09-15T20:56:38.744 INFO:tasks.workunit.client.0.smithi136.stderr:183124: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-15T20:56:38.745 INFO:tasks.workunit.client.0.smithi136.stderr:183124: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-15T20:56:38.745 INFO:tasks.workunit.client.0.smithi136.stderr:183124: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-15T20:56:38.745 INFO:tasks.workunit.client.0.smithi136.stderr:183124: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T20:56:38.745 INFO:tasks.workunit.client.0.smithi136.stderr:183124: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T20:56:38.745 INFO:tasks.workunit.client.0.smithi136.stderr:183124: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T20:56:38.745 INFO:tasks.workunit.client.0.smithi136.stderr:183124: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T20:56:38.745 INFO:tasks.workunit.client.0.smithi136.stderr:183124: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-15T20:56:38.745 INFO:tasks.workunit.client.0.smithi136.stderr:183124: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-15T20:56:38.745 INFO:tasks.workunit.client.0.smithi136.stderr:183124: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-09-15T20:56:38.745 INFO:tasks.workunit.client.0.smithi136.stderr:183124: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.1 2024-09-15T20:56:38.745 INFO:tasks.workunit.client.0.smithi136.stderr:183124: start osd.1 2024-09-15T20:56:38.758 INFO:tasks.workunit.client.0.smithi136.stderr:183124: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 1 --fsid=ce4892c7-6c6d-4b4f-8e6f-07a3e11133e5 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.52602/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_reco183122: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2024-09-15T20:56:38.759 INFO:tasks.workunit.client.0.smithi136.stderr:183122: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2024-09-15T20:56:38.759 INFO:tasks.workunit.client.0.smithi136.stderr:183122: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=3 2024-09-15T20:56:38.759 INFO:tasks.workunit.client.0.smithi136.stderr:183122: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2024-09-15T20:56:38.759 INFO:tasks.workunit.client.0.smithi136.stderr:183122: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 3 SOMETHING list-attrs 2024-09-15T20:56:38.759 INFO:tasks.workunit.client.0.smithi136.stderr:183122: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-09-15T20:56:38.759 INFO:tasks.workunit.client.0.smithi136.stderr:183122: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2024-09-15T20:56:38.759 INFO:tasks.workunit.client.0.smithi136.stderr:183122: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=3 2024-09-15T20:56:38.759 INFO:tasks.workunit.client.0.smithi136.stderr:183122: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2024-09-15T20:56:38.759 INFO:tasks.workunit.client.0.smithi136.stderr:183122: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.3 2024-09-15T20:56:38.759 INFO:tasks.workunit.client.0.smithi136.stderr:183122: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-15T20:56:38.759 INFO:tasks.workunit.client.0.smithi136.stderr:183122: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-15T20:56:38.759 INFO:tasks.workunit.client.0.smithi136.stderr:183122: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-15T20:56:38.759 INFO:tasks.workunit.client.0.smithi136.stderr:183122: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-15T20:56:38.760 INFO:tasks.workunit.client.0.smithi136.stderr:183122: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-15T20:56:38.760 INFO:tasks.workunit.client.0.smithi136.stderr:183122: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-15T20:56:38.760 INFO:tasks.workunit.client.0.smithi136.stderr:183122: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 3 SOMETHING list-attrs 2024-09-15T20:56:38.760 INFO:tasks.workunit.client.0.smithi136.stderr:183122: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-15T20:56:38.760 INFO:tasks.workunit.client.0.smithi136.stderr:183122: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-15T20:56:38.760 INFO:tasks.workunit.client.0.smithi136.stderr:183122: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=3 2024-09-15T20:56:38.760 INFO:tasks.workunit.client.0.smithi136.stderr:183122: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-15T20:56:38.760 INFO:tasks.workunit.client.0.smithi136.stderr:183122: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/3 2024-09-15T20:56:38.760 INFO:tasks.workunit.client.0.smithi136.stderr:183122: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/3 SOMETHING list-attrs 2024-09-15T20:56:38.760 INFO:tasks.workunit.client.0.smithi136.stderr:183122: Error getting attr on : 2.0s0_head,0#-4:00000000:::scrub_2.0s0:head#, (61) No data available 2024-09-15T20:56:38.760 INFO:tasks.workunit.client.0.smithi136.stderr:183122: _ 2024-09-15T20:56:38.760 INFO:tasks.workunit.client.0.smithi136.stderr:183122: hinfo_key 2024-09-15T20:56:38.760 INFO:tasks.workunit.client.0.smithi136.stderr:183122: snapset 2024-09-15T20:56:38.760 INFO:tasks.workunit.client.0.smithi136.stderr:183122: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 3 2024-09-15T20:56:38.761 INFO:tasks.workunit.client.0.smithi136.stderr:183122: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2024-09-15T20:56:38.761 INFO:tasks.workunit.client.0.smithi136.stderr:183122: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-15T20:56:38.761 INFO:tasks.workunit.client.0.smithi136.stderr:183122: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=3 2024-09-15T20:56:38.761 INFO:tasks.workunit.client.0.smithi136.stderr:183122: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-15T20:56:38.761 INFO:tasks.workunit.client.0.smithi136.stderr:183122: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/3 2024-09-15T20:56:38.761 INFO:tasks.workunit.client.0.smithi136.stderr:183122: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=ce4892c7-6c6d-4b4f-8e6f-07a3e11133e5 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-15T20:56:38.761 INFO:tasks.workunit.client.0.smithi136.stderr:183122: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T20:56:38.762 INFO:tasks.workunit.client.0.smithi136.stderr:183122: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T20:56:38.762 INFO:tasks.workunit.client.0.smithi136.stderr:183122: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T20:56:38.762 INFO:tasks.workunit.client.0.smithi136.stderr:183122: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/3' 2024-09-15T20:56:38.762 INFO:tasks.workunit.client.0.smithi136.stderr:183122: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/3/journal' 2024-09-15T20:56:38.762 INFO:tasks.workunit.client.0.smithi136.stderr:183122: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-15T20:56:38.762 INFO:tasks.workunit.client.0.smithi136.stderr:183122: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-15T20:56:38.762 INFO:tasks.workunit.client.0.smithi136.stderr:183122: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-15T20:56:38.762 INFO:tasks.workunit.client.0.smithi136.stderr:183122: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-15T20:56:38.762 INFO:tasks.workunit.client.0.smithi136.stderr:183122: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-15T20:56:38.762 INFO:tasks.workunit.client.0.smithi136.stderr:183122: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-15T20:56:38.762 INFO:tasks.workunit.client.0.smithi136.stderr:183122: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-15T20:56:38.762 INFO:tasks.workunit.client.0.smithi136.stderr:183122: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T20:56:38.762 INFO:tasks.workunit.client.0.smithi136.stderr:183122: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52602 2024-09-15T20:56:38.763 INFO:tasks.workunit.client.0.smithi136.stderr:183122: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.52602/$cluster-$name.asok' 2024-09-15T20:56:38.763 INFO:tasks.workunit.client.0.smithi136.stderr:183122: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.52602/$cluster-$name.asok' 2024-09-15T20:56:38.763 INFO:tasks.workunit.client.0.smithi136.stderr:183122: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-15T20:56:38.763 INFO:tasks.workunit.client.0.smithi136.stderr:183122: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-15T20:56:38.763 INFO:tasks.workunit.client.0.smithi136.stderr:183122: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-15T20:56:38.763 INFO:tasks.workunit.client.0.smithi136.stderr:183122: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T20:56:38.763 INFO:tasks.workunit.client.0.smithi136.stderr:183122: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T20:56:38.763 INFO:tasks.workunit.client.0.smithi136.stderr:183122: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T20:56:38.763 INFO:tasks.workunit.client.0.smithi136.stderr:183122: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T20:56:38.763 INFO:tasks.workunit.client.0.smithi136.stderr:183122: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-15T20:56:38.763 INFO:tasks.workunit.client.0.smithi136.stderr:183122: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-15T20:56:38.763 INFO:tasks.workunit.client.0.smithi136.stderr:183122: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/3 2024-09-15T20:56:38.763 INFO:tasks.workunit.client.0.smithi136.stderr:183122: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.3 2024-09-15T20:56:38.763 INFO:tasks.workunit.client.0.smithi136.stderr:183122: start osd.3 2024-09-15T20:56:43.233 INFO:tasks.workunit.client.0.smithi136.stderr:183122: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 3 --fsid=ce4892c7-6c6d-4b4f-8e6f-07a3e11133e5 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/3 --osd-journal=td/osd-scrub-repair/3/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.52602/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T20:56:43.233 INFO:tasks.workunit.client.0.smithi136.stderr:183122: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/3/whoami 2024-09-15T20:56:43.233 INFO:tasks.workunit.client.0.smithi136.stderr:183122: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 3 = 3 ']' 2024-09-15T20:56:43.233 INFO:tasks.workunit.client.0.smithi136.stderr:183122: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-15T20:56:43.234 INFO:tasks.workunit.client.0.smithi136.stderr:183122: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-15T20:56:43.234 INFO:tasks.workunit.client.0.smithi136.stderr:183122: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-15T20:56:43.234 INFO:tasks.workunit.client.0.smithi136.stderr:183122: 2024-09-15T20:56:38.789+0000 7f2bca40d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T20:56:43.234 INFO:tasks.workunit.client.0.smithi136.stderr:183122: 2024-09-15T20:56:38.792+0000 7f2bca40d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T20:56:43.234 INFO:tasks.workunit.client.0.smithi136.stderr:183122: 2024-09-15T20:56:38.793+0000 7f2bca40d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T20:56:43.234 INFO:tasks.workunit.client.0.smithi136.stderr:183122: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 3 2024-09-15T20:56:43.234 INFO:tasks.workunit.client.0.smithi136.stderr:183122: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-15T20:56:43.234 INFO:tasks.workunit.client.0.smithi136.stderr:183122: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=3 2024-09-15T20:56:43.234 INFO:tasks.workunit.client.0.smithi136.stderr:183122: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-15T20:56:43.234 INFO:tasks.workunit.client.0.smithi136.stderr:183122: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-15T20:56:43.234 INFO:tasks.workunit.client.0.smithi136.stderr:183122: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:56:43.235 INFO:tasks.workunit.client.0.smithi136.stderr:183122: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-15T20:56:43.235 INFO:tasks.workunit.client.0.smithi136.stderr:183122: 0 2024-09-15T20:56:43.235 INFO:tasks.workunit.client.0.smithi136.stderr:183122: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:56:43.235 INFO:tasks.workunit.client.0.smithi136.stderr:183122: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-15T20:56:43.235 INFO:tasks.workunit.client.0.smithi136.stderr:183122: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:56:43.235 INFO:tasks.workunit.client.0.smithi136.stderr:183122: 2024-09-15T20:56:39.656+0000 7f2bca40d640 -1 Falling back to public interface 2024-09-15T20:56:43.235 INFO:tasks.workunit.client.0.smithi136.stderr:183122: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:56:43.235 INFO:tasks.workunit.client.0.smithi136.stderr:183122: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:56:43.235 INFO:tasks.workunit.client.0.smithi136.stderr:183122: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-15T20:56:43.235 INFO:tasks.workunit.client.0.smithi136.stderr:183122: 1 2024-09-15T20:56:43.236 INFO:tasks.workunit.client.0.smithi136.stderr:183122: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:56:43.236 INFO:tasks.workunit.client.0.smithi136.stderr:183122: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-15T20:56:43.236 INFO:tasks.workunit.client.0.smithi136.stderr:183122: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:56:43.236 INFO:tasks.workunit.client.0.smithi136.stderr:183122: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:56:43.236 INFO:tasks.workunit.client.0.smithi136.stderr:183122: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:56:43.236 INFO:tasks.workunit.client.0.smithi136.stderr:183122: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-15T20:56:43.236 INFO:tasks.workunit.client.0.smithi136.stderr:183122: 2 2024-09-15T20:56:43.236 INFO:tasks.workunit.client.0.smithi136.stderr:183122: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:56:43.236 INFO:tasks.workunit.client.0.smithi136.stderr:183122: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-15T20:56:43.236 INFO:tasks.workunit.client.0.smithi136.stderr:183122: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:56:43.236 INFO:tasks.workunit.client.0.smithi136.stderr:183122: 2024-09-15T20:56:42.517+0000 7f2bca40d640 -1 osd.3 97 log_to_monitors true 2024-09-15T20:56:43.236 INFO:tasks.workunit.client.0.smithi136.stderr:183122: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:56:43.236 INFO:tasks.workunit.client.0.smithi136.stderr:183122: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:56:43.237 INFO:tasks.workunit.client.0.smithi136.stderr:183122: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-15T20:56:43.237 INFO:tasks.workunit.client.0.smithi136.stderr:183122: 3 2024-09-15T20:56:43.237 INFO:tasks.workunit.client.0.smithi136.stderr:183122: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:56:44.546 INFO:tasks.workunit.client.0.smithi136.stderr:183122: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-15T20:56:44.547 INFO:tasks.workunit.client.0.smithi136.stderr:183122: osd.3 up in weight 1 up_from 102 up_thru 96 down_at 98 last_clean_interval [96,97) [v2:127.0.0.1:6818/3529608941,v1:127.0.0.1:6819/3529608941] [v2:127.0.0.1:6820/3529608941,v1:127.0.0.1:6821/3529608941] exists,up b72fdf29-d86a-404d-a216-75e64b9c077b 2024-09-15T20:56:44.547 INFO:tasks.workunit.client.0.smithi136.stderr:183122: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-15T20:56:44.547 INFO:tasks.workunit.client.0.smithi136.stderr:183122: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-15T20:56:44.547 INFO:tasks.workunit.client.0.smithi136.stderr:183122: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-15T20:56:44.547 INFO:tasks.workunit.client.0.smithi136.stderr:183122: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2024-09-15T20:56:44.547 INFO:tasks.workunit.client.0.smithi136.stderr:183122: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-15T20:56:44.547 INFO:tasks.workunit.client.0.smithi136.stderr:183122: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-15T20:56:44.547 INFO:tasks.workunit.client.0.smithi136.stderr:183122: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-15T20:56:44.547 INFO:tasks.workunit.client.0.smithi136.stderr:183122: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-15T20:56:44.547 INFO:tasks.workunit.client.0.smithi136.stderr:183122: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-15T20:56:44.548 INFO:tasks.workunit.client.0.smithi136.stderr:183122: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-15T20:56:44.548 INFO:tasks.workunit.client.0.smithi136.stderr:183122: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-15T20:56:44.548 INFO:tasks.workunit.client.0.smithi136.stderr:183122: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-15T20:56:44.548 INFO:tasks.workunit.client.0.smithi136.stderr:183122: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-15T20:56:44.548 INFO:tasks.workunit.client.0.smithi136.stderr:183122: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-15T20:56:44.548 INFO:tasks.workunit.client.0.smithi136.stderr:183122: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-15T20:56:44.548 INFO:tasks.workunit.client.0.smithi136.stderr:183122: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-15T20:56:44.548 INFO:tasks.workunit.client.0.smithi136.stderr:183122: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-15T20:56:44.548 INFO:tasks.workunit.client.0.smithi136.stderr:183122: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-15T20:56:44.548 INFO:tasks.workunit.client.0.smithi136.stderr:183122: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-15T20:56:44.548 INFO:tasks.workunit.client.0.smithi136.stderr:183122: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-15T20:56:44.548 INFO:tasks.workunit.client.0.smithi136.stderr:183122: 1 2024-09-15T20:56:44.548 INFO:tasks.workunit.client.0.smithi136.stderr:183122: 2 2024-09-15T20:56:44.548 INFO:tasks.workunit.client.0.smithi136.stderr:183122: 3' 2024-09-15T20:56:44.549 INFO:tasks.workunit.client.0.smithi136.stderr:183122: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-15T20:56:44.549 INFO:tasks.workunit.client.0.smithi136.stderr:183122: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T20:56:44.549 INFO:tasks.workunit.client.0.smithi136.stderr:183122: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-15T20:56:44.549 INFO:tasks.workunit.client.0.smithi136.stderr:183122: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836547 2024-09-15T20:56:44.549 INFO:tasks.workunit.client.0.smithi136.stderr:183122: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836547 2024-09-15T20:56:44.549 INFO:tasks.workunit.client.0.smithi136.stderr:183122: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836547' 2024-09-15T20:56:44.549 INFO:tasks.workunit.client.0.smithi136.stderr:183122: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T20:56:44.549 INFO:tasks.workunit.client.0.smithi136.stderr:183122: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-15T20:56:44.549 INFO:tasks.workunit.client.0.smithi136.stderr:183122: Error ENXIO: problem getting command descriptions from osd.1 2024-09-15T20:56:44.549 INFO:tasks.workunit.client.0.smithi136.stderr:183122: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq= 2024-09-15T20:56:44.549 INFO:tasks.workunit.client.0.smithi136.stderr:183122: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z '' 2024-09-15T20:56:44.549 INFO:tasks.workunit.client.0.smithi136.stderr:183122: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: continue 2024-09-15T20:56:44.554 INFO:tasks.workunit.client.0.smithi136.stderr:183122: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pvery_ops 2024-09-15T20:56:44.554 INFO:tasks.workunit.client.0.smithi136.stderr:183124: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-09-15T20:56:44.554 INFO:tasks.workunit.client.0.smithi136.stderr:183124: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 1 = 1 ']' 2024-09-15T20:56:44.554 INFO:tasks.workunit.client.0.smithi136.stderr:183124: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-15T20:56:44.555 INFO:tasks.workunit.client.0.smithi136.stderr:183124: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-15T20:56:44.555 INFO:tasks.workunit.client.0.smithi136.stderr:183124: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-15T20:56:44.555 INFO:tasks.workunit.client.0.smithi136.stderr:183124: 2024-09-15T20:56:38.774+0000 7fe2fa22e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T20:56:44.555 INFO:tasks.workunit.client.0.smithi136.stderr:183124: 2024-09-15T20:56:38.777+0000 7fe2fa22e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T20:56:44.555 INFO:tasks.workunit.client.0.smithi136.stderr:183124: 2024-09-15T20:56:38.779+0000 7fe2fa22e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T20:56:44.555 INFO:tasks.workunit.client.0.smithi136.stderr:183124: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 1 2024-09-15T20:56:44.555 INFO:tasks.workunit.client.0.smithi136.stderr:183124: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-15T20:56:44.555 INFO:tasks.workunit.client.0.smithi136.stderr:183124: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2024-09-15T20:56:44.555 INFO:tasks.workunit.client.0.smithi136.stderr:183124: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-15T20:56:44.555 INFO:tasks.workunit.client.0.smithi136.stderr:183124: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-15T20:56:44.555 INFO:tasks.workunit.client.0.smithi136.stderr:183124: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:56:44.555 INFO:tasks.workunit.client.0.smithi136.stderr:183124: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-15T20:56:44.555 INFO:tasks.workunit.client.0.smithi136.stderr:183124: 0 2024-09-15T20:56:44.556 INFO:tasks.workunit.client.0.smithi136.stderr:183124: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:56:44.556 INFO:tasks.workunit.client.0.smithi136.stderr:183124: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T20:56:44.556 INFO:tasks.workunit.client.0.smithi136.stderr:183124: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:56:44.556 INFO:tasks.workunit.client.0.smithi136.stderr:183124: 2024-09-15T20:56:40.148+0000 7fe2fa22e640 -1 Falling back to public interface 2024-09-15T20:56:44.556 INFO:tasks.workunit.client.0.smithi136.stderr:183124: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:56:44.556 INFO:tasks.workunit.client.0.smithi136.stderr:183124: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:56:44.556 INFO:tasks.workunit.client.0.smithi136.stderr:183124: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-15T20:56:44.556 INFO:tasks.workunit.client.0.smithi136.stderr:183124: 1 2024-09-15T20:56:44.556 INFO:tasks.workunit.client.0.smithi136.stderr:183124: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:56:44.556 INFO:tasks.workunit.client.0.smithi136.stderr:183124: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T20:56:44.556 INFO:tasks.workunit.client.0.smithi136.stderr:183124: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:56:44.556 INFO:tasks.workunit.client.0.smithi136.stderr:183124: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:56:44.556 INFO:tasks.workunit.client.0.smithi136.stderr:183124: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:56:44.557 INFO:tasks.workunit.client.0.smithi136.stderr:183124: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-15T20:56:44.557 INFO:tasks.workunit.client.0.smithi136.stderr:183124: 2 2024-09-15T20:56:44.557 INFO:tasks.workunit.client.0.smithi136.stderr:183124: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:56:44.557 INFO:tasks.workunit.client.0.smithi136.stderr:183124: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T20:56:44.557 INFO:tasks.workunit.client.0.smithi136.stderr:183124: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:56:44.557 INFO:tasks.workunit.client.0.smithi136.stderr:183124: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:56:44.557 INFO:tasks.workunit.client.0.smithi136.stderr:183124: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:56:44.557 INFO:tasks.workunit.client.0.smithi136.stderr:183124: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-15T20:56:44.557 INFO:tasks.workunit.client.0.smithi136.stderr:183124: 3 2024-09-15T20:56:44.557 INFO:tasks.workunit.client.0.smithi136.stderr:183124: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:56:44.557 INFO:tasks.workunit.client.0.smithi136.stderr:183124: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T20:56:44.557 INFO:tasks.workunit.client.0.smithi136.stderr:183124: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:56:46.841 INFO:tasks.workunit.client.0.smithi136.stderr:183124: 2024-09-15T20:56:44.552+0000 7fe2fa22e640 -1 osd.1 97 log_to_monitors true 2024-09-15T20:56:46.841 INFO:tasks.workunit.client.0.smithi136.stderr:183124: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:56:46.841 INFO:tasks.workunit.client.0.smithi136.stderr:183124: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:56:46.841 INFO:tasks.workunit.client.0.smithi136.stderr:183124: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-15T20:56:46.841 INFO:tasks.workunit.client.0.smithi136.stderr:183124: 4 2024-09-15T20:56:46.842 INFO:tasks.workunit.client.0.smithi136.stderr:183124: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:56:46.842 INFO:tasks.workunit.client.0.smithi136.stderr:183124: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T20:56:46.842 INFO:tasks.workunit.client.0.smithi136.stderr:183124: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:56:46.842 INFO:tasks.workunit.client.0.smithi136.stderr:183124: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:56:46.842 INFO:tasks.workunit.client.0.smithi136.stderr:183124: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:56:46.842 INFO:tasks.workunit.client.0.smithi136.stderr:183124: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-15T20:56:46.842 INFO:tasks.workunit.client.0.smithi136.stderr:183124: 5 2024-09-15T20:56:46.842 INFO:tasks.workunit.client.0.smithi136.stderr:183124: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:56:46.842 INFO:tasks.workunit.client.0.smithi136.stderr:183124: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T20:56:46.842 INFO:tasks.workunit.client.0.smithi136.stderr:183124: osd.1 up in weight 1 up_from 106 up_thru 106 down_at 98 last_clean_interval [92,97) [v2:127.0.0.1:6826/569658771,v1:127.0.0.1:6827/569658771] [v2:127.0.0.1:6828/569658771,v1:127.0.0.1:6829/569658771] exists,up 4e733256-a0ac-49e1-b991-7e6e4a060afc 2024-09-15T20:56:46.842 INFO:tasks.workunit.client.0.smithi136.stderr:183124: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-15T20:56:46.842 INFO:tasks.workunit.client.0.smithi136.stderr:183124: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-15T20:56:46.842 INFO:tasks.workunit.client.0.smithi136.stderr:183124: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-15T20:56:46.843 INFO:tasks.workunit.client.0.smithi136.stderr:183124: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2024-09-15T20:56:46.843 INFO:tasks.workunit.client.0.smithi136.stderr:183124: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-15T20:56:46.843 INFO:tasks.workunit.client.0.smithi136.stderr:183124: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-15T20:56:46.843 INFO:tasks.workunit.client.0.smithi136.stderr:183124: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-15T20:56:46.843 INFO:tasks.workunit.client.0.smithi136.stderr:183124: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-15T20:56:46.843 INFO:tasks.workunit.client.0.smithi136.stderr:183124: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-15T20:56:46.843 INFO:tasks.workunit.client.0.smithi136.stderr:183124: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-15T20:56:46.843 INFO:tasks.workunit.client.0.smithi136.stderr:183124: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-15T20:56:46.843 INFO:tasks.workunit.client.0.smithi136.stderr:183124: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-15T20:56:46.843 INFO:tasks.workunit.client.0.smithi136.stderr:183124: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-15T20:56:46.843 INFO:tasks.workunit.client.0.smithi136.stderr:183124: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-15T20:56:46.843 INFO:tasks.workunit.client.0.smithi136.stderr:183124: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-15T20:56:46.843 INFO:tasks.workunit.client.0.smithi136.stderr:183124: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-15T20:56:46.843 INFO:tasks.workunit.client.0.smithi136.stderr:183124: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-15T20:56:46.844 INFO:tasks.workunit.client.0.smithi136.stderr:183124: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-15T20:56:46.844 INFO:tasks.workunit.client.0.smithi136.stderr:183124: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-15T20:56:46.844 INFO:tasks.workunit.client.0.smithi136.stderr:183124: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-15T20:56:46.844 INFO:tasks.workunit.client.0.smithi136.stderr:183124: 1 2024-09-15T20:56:46.844 INFO:tasks.workunit.client.0.smithi136.stderr:183124: 2 2024-09-15T20:56:46.844 INFO:tasks.workunit.client.0.smithi136.stderr:183124: 3' 2024-09-15T20:56:46.844 INFO:tasks.workunit.client.0.smithi136.stderr:183124: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-15T20:56:46.844 INFO:tasks.workunit.client.0.smithi136.stderr:183124: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T20:56:47.208 INFO:tasks.workunit.client.0.smithi136.stderr:183124: g_stats: for osd in $ids 2024-09-15T20:56:47.208 INFO:tasks.workunit.client.0.smithi136.stderr:183122: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-15T20:56:47.208 INFO:tasks.workunit.client.0.smithi136.stderr:183122: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=347892350989 2024-09-15T20:56:47.208 INFO:tasks.workunit.client.0.smithi136.stderr:183122: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 347892350989 2024-09-15T20:56:47.208 INFO:tasks.workunit.client.0.smithi136.stderr:183122: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836547 2-347892350989' 2024-09-15T20:56:47.208 INFO:tasks.workunit.client.0.smithi136.stderr:183122: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T20:56:47.208 INFO:tasks.workunit.client.0.smithi136.stderr:183122: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-15T20:56:47.208 INFO:tasks.workunit.client.0.smithi136.stderr:183122: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=438086664194 2024-09-15T20:56:47.208 INFO:tasks.workunit.client.0.smithi136.stderr:183122: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 438086664194 2024-09-15T20:56:47.209 INFO:tasks.workunit.client.0.smithi136.stderr:183122: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836547 2-347892350989 3-438086664194' 2024-09-15T20:56:47.209 INFO:tasks.workunit.client.0.smithi136.stderr:183122: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T20:56:47.209 INFO:tasks.workunit.client.0.smithi136.stderr:183122: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836547 2024-09-15T20:56:47.209 INFO:tasks.workunit.client.0.smithi136.stderr:183122: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T20:56:47.209 INFO:tasks.workunit.client.0.smithi136.stderr:183122: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-15T20:56:47.209 INFO:tasks.workunit.client.0.smithi136.stderr:183122: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836547 2024-09-15T20:56:47.209 INFO:tasks.workunit.client.0.smithi136.stderr:183122: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T20:56:47.209 INFO:tasks.workunit.client.0.smithi136.stderr:183122: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836547 2024-09-15T20:56:47.209 INFO:tasks.workunit.client.0.smithi136.stderr:183122: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836547' 2024-09-15T20:56:47.209 INFO:tasks.workunit.client.0.smithi136.stderr:183122: waiting osd.0 seq 21474836547 2024-09-15T20:56:47.209 INFO:tasks.workunit.client.0.smithi136.stderr:183122: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T20:56:47.209 INFO:tasks.workunit.client.0.smithi136.stderr:183122: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836547 -lt 21474836547 2024-09-15T20:56:47.209 INFO:tasks.workunit.client.0.smithi136.stderr:183122: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T20:56:47.209 INFO:tasks.workunit.client.0.smithi136.stderr:183122: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-347892350989 2024-09-15T20:56:47.210 INFO:tasks.workunit.client.0.smithi136.stderr:183122: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T20:56:47.210 INFO:tasks.workunit.client.0.smithi136.stderr:183122: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-15T20:56:47.210 INFO:tasks.workunit.client.0.smithi136.stderr:183122: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-347892350989 2024-09-15T20:56:47.210 INFO:tasks.workunit.client.0.smithi136.stderr:183122: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T20:56:47.210 INFO:tasks.workunit.client.0.smithi136.stderr:183122: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=347892350989 2024-09-15T20:56:47.210 INFO:tasks.workunit.client.0.smithi136.stderr:183122: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 347892350989' 2024-09-15T20:56:47.210 INFO:tasks.workunit.client.0.smithi136.stderr:183122: waiting osd.2 seq 347892350989 2024-09-15T20:56:47.210 INFO:tasks.workunit.client.0.smithi136.stderr:183122: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-15T20:56:47.210 INFO:tasks.workunit.client.0.smithi136.stderr:183122: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 347892350988 -lt 347892350989 2024-09-15T20:56:47.210 INFO:tasks.workunit.client.0.smithi136.stderr:183122: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-15T20:56:47.210 INFO:tasks.workunit.client.0.smithi136.stderr:183122: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-15T20:56:47.210 INFO:tasks.workunit.client.0.smithi136.stderr:183122: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-15T20:56:47.210 INFO:tasks.workunit.client.0.smithi136.stderr:183122: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 347892350988 -lt 347892350989 2024-09-15T20:56:49.010 INFO:tasks.workunit.client.0.smithi136.stderr:183122: /home/u//home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-15T20:56:49.010 INFO:tasks.workunit.client.0.smithi136.stderr:183124: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836548 2024-09-15T20:56:49.011 INFO:tasks.workunit.client.0.smithi136.stderr:183124: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836548 2024-09-15T20:56:49.011 INFO:tasks.workunit.client.0.smithi136.stderr:183124: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836548' 2024-09-15T20:56:49.011 INFO:tasks.workunit.client.0.smithi136.stderr:183124: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T20:56:49.011 INFO:tasks.workunit.client.0.smithi136.stderr:183124: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-15T20:56:49.011 INFO:tasks.workunit.client.0.smithi136.stderr:183124: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=455266533378 2024-09-15T20:56:49.011 INFO:tasks.workunit.client.0.smithi136.stderr:183124: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 455266533378 2024-09-15T20:56:49.011 INFO:tasks.workunit.client.0.smithi136.stderr:183124: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836548 1-455266533378' 2024-09-15T20:56:49.011 INFO:tasks.workunit.client.0.smithi136.stderr:183124: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T20:56:49.011 INFO:tasks.workunit.client.0.smithi136.stderr:183124: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-15T20:56:49.011 INFO:tasks.workunit.client.0.smithi136.stderr:183124: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=347892350991 2024-09-15T20:56:49.011 INFO:tasks.workunit.client.0.smithi136.stderr:183124: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 347892350991 2024-09-15T20:56:49.011 INFO:tasks.workunit.client.0.smithi136.stderr:183124: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836548 1-455266533378 2-347892350991' 2024-09-15T20:56:49.011 INFO:tasks.workunit.client.0.smithi136.stderr:183124: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T20:56:49.012 INFO:tasks.workunit.client.0.smithi136.stderr:183124: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-15T20:56:49.012 INFO:tasks.workunit.client.0.smithi136.stderr:183124: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=438086664195 2024-09-15T20:56:49.012 INFO:tasks.workunit.client.0.smithi136.stderr:183124: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 438086664195 2024-09-15T20:56:49.012 INFO:tasks.workunit.client.0.smithi136.stderr:183124: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836548 1-455266533378 2-347892350991 3-438086664195' 2024-09-15T20:56:49.012 INFO:tasks.workunit.client.0.smithi136.stderr:183124: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T20:56:49.012 INFO:tasks.workunit.client.0.smithi136.stderr:183124: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836548 2024-09-15T20:56:49.012 INFO:tasks.workunit.client.0.smithi136.stderr:183124: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T20:56:49.012 INFO:tasks.workunit.client.0.smithi136.stderr:183124: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-15T20:56:49.012 INFO:tasks.workunit.client.0.smithi136.stderr:183124: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836548 2024-09-15T20:56:49.012 INFO:tasks.workunit.client.0.smithi136.stderr:183124: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T20:56:49.012 INFO:tasks.workunit.client.0.smithi136.stderr:183124: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836548 2024-09-15T20:56:49.012 INFO:tasks.workunit.client.0.smithi136.stderr:183124: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836548' 2024-09-15T20:56:49.012 INFO:tasks.workunit.client.0.smithi136.stderr:183124: waiting osd.0 seq 21474836548 2024-09-15T20:56:49.012 INFO:tasks.workunit.client.0.smithi136.stderr:183124: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T20:56:49.013 INFO:tasks.workunit.client.0.smithi136.stderr:183124: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836547 -lt 21474836548 2024-09-15T20:56:49.013 INFO:tasks.workunit.client.0.smithi136.stderr:183124: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-15T20:56:49.013 INFO:tasks.workunit.client.0.smithi136.stderr:183124: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-15T20:56:49.013 INFO:tasks.workunit.client.0.smithi136.stderr:183124: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T20:56:49.013 INFO:tasks.workunit.client.0.smithi136.stderr:183124: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836549 -lt 21474836548 2024-09-15T20:56:49.864 INFO:tasks.workunit.client.0.smithi136.stderr:183124: /home/ubuntu/cephtest/clone.cliebuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-15T20:56:49.864 INFO:tasks.workunit.client.0.smithi136.stderr:183122: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 299 -eq 0 ']' 2024-09-15T20:56:49.864 INFO:tasks.workunit.client.0.smithi136.stderr:183122: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-15T20:56:49.865 INFO:tasks.workunit.client.0.smithi136.stderr:183122: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 347892350990 -lt 347892350989 2024-09-15T20:56:49.865 INFO:tasks.workunit.client.0.smithi136.stderr:183122: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T20:56:49.865 INFO:tasks.workunit.client.0.smithi136.stderr:183122: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 3-438086664194 2024-09-15T20:56:49.865 INFO:tasks.workunit.client.0.smithi136.stderr:183122: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T20:56:49.865 INFO:tasks.workunit.client.0.smithi136.stderr:183122: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=3 2024-09-15T20:56:49.865 INFO:tasks.workunit.client.0.smithi136.stderr:183122: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 3-438086664194 2024-09-15T20:56:49.865 INFO:tasks.workunit.client.0.smithi136.stderr:183122: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T20:56:49.865 INFO:tasks.workunit.client.0.smithi136.stderr:183122: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=438086664194 2024-09-15T20:56:49.865 INFO:tasks.workunit.client.0.smithi136.stderr:183122: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.3 seq 438086664194' 2024-09-15T20:56:49.865 INFO:tasks.workunit.client.0.smithi136.stderr:183122: waiting osd.3 seq 438086664194 2024-09-15T20:56:49.865 INFO:tasks.workunit.client.0.smithi136.stderr:183122: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-15T20:56:49.865 INFO:tasks.workunit.client.0.smithi136.stderr:183122: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 438086664194 -lt 438086664194 2024-09-15T20:56:49.866 INFO:tasks.workunit.client.0.smithi136.stderr:183122: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-15T20:56:49.866 INFO:tasks.workunit.client.0.smithi136.stderr:183122: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T20:56:49.866 INFO:tasks.workunit.client.0.smithi136.stderr:183122: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T20:56:49.866 INFO:tasks.workunit.client.0.smithi136.stderr:183122: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 5 == 0 2024-09-15T20:56:49.866 INFO:tasks.workunit.client.0.smithi136.stderr:183122: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-15T20:56:49.866 INFO:tasks.workunit.client.0.smithi136.stderr:183122: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-15T20:56:49.866 INFO:tasks.workunit.client.0.smithi136.stderr:183122: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-15T20:56:49.866 INFO:tasks.workunit.client.0.smithi136.stderr:183122: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T20:56:49.866 INFO:tasks.workunit.client.0.smithi136.stderr:183122: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T20:56:49.866 INFO:tasks.workunit.client.0.smithi136.stderr:183122: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T20:56:49.866 INFO:tasks.workunit.client.0.smithi136.stderr:183122: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T20:56:49.866 INFO:tasks.workunit.client.0.smithi136.stderr:183122: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=5 2024-09-15T20:56:49.866 INFO:tasks.workunit.client.0.smithi136.stderr:183122: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-15T20:56:49.867 INFO:tasks.workunit.client.0.smithi136.stderr:183122: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T20:56:49.867 INFO:tasks.workunit.client.0.smithi136.stderr:183122: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T20:56:49.867 INFO:tasks.workunit.client.0.smithi136.stderr:183122: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 = 5 2024-09-15T20:56:49.867 INFO:tasks.workunit.client.0.smithi136.stderr:183122: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-15T20:56:49.867 INFO:tasks.workunit.client.0.smithi136.stderr:183122: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-15T20:56:49.867 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: run_in_background: return 0 2024-09-15T20:56:49.867 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: wait_background: for pid in $pids 2024-09-15T20:56:49.867 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2175: wait_background: wait 183121 2024-09-15T20:56:50.277 INFO:tasks.workunit.client.0.smithi136.stderr:nt.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T20:56:50.277 INFO:tasks.workunit.client.0.smithi136.stderr:183124: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-455266533378 2024-09-15T20:56:50.278 INFO:tasks.workunit.client.0.smithi136.stderr:183124: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T20:56:50.278 INFO:tasks.workunit.client.0.smithi136.stderr:183124: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-15T20:56:50.278 INFO:tasks.workunit.client.0.smithi136.stderr:183124: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-455266533378 2024-09-15T20:56:50.278 INFO:tasks.workunit.client.0.smithi136.stderr:183124: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T20:56:50.278 INFO:tasks.workunit.client.0.smithi136.stderr:183124: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=455266533378 2024-09-15T20:56:50.278 INFO:tasks.workunit.client.0.smithi136.stderr:183124: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 455266533378' 2024-09-15T20:56:50.278 INFO:tasks.workunit.client.0.smithi136.stderr:183124: waiting osd.1 seq 455266533378 2024-09-15T20:56:50.278 INFO:tasks.workunit.client.0.smithi136.stderr:183124: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-15T20:56:50.278 INFO:tasks.workunit.client.0.smithi136.stderr:183124: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 455266533378 -lt 455266533378 2024-09-15T20:56:50.278 INFO:tasks.workunit.client.0.smithi136.stderr:183124: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T20:56:50.278 INFO:tasks.workunit.client.0.smithi136.stderr:183124: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-347892350991 2024-09-15T20:56:50.278 INFO:tasks.workunit.client.0.smithi136.stderr:183124: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T20:56:50.278 INFO:tasks.workunit.client.0.smithi136.stderr:183124: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-15T20:56:50.279 INFO:tasks.workunit.client.0.smithi136.stderr:183124: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-347892350991 2024-09-15T20:56:50.279 INFO:tasks.workunit.client.0.smithi136.stderr:183124: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T20:56:50.279 INFO:tasks.workunit.client.0.smithi136.stderr:183124: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=347892350991 2024-09-15T20:56:50.279 INFO:tasks.workunit.client.0.smithi136.stderr:183124: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 347892350991' 2024-09-15T20:56:50.279 INFO:tasks.workunit.client.0.smithi136.stderr:183124: waiting osd.2 seq 347892350991 2024-09-15T20:56:50.279 INFO:tasks.workunit.client.0.smithi136.stderr:183124: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-15T20:56:50.279 INFO:tasks.workunit.client.0.smithi136.stderr:183124: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 347892350991 -lt 347892350991 2024-09-15T20:56:50.279 INFO:tasks.workunit.client.0.smithi136.stderr:183124: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T20:56:50.279 INFO:tasks.workunit.client.0.smithi136.stderr:183124: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 3-438086664195 2024-09-15T20:56:50.279 INFO:tasks.workunit.client.0.smithi136.stderr:183124: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T20:56:50.279 INFO:tasks.workunit.client.0.smithi136.stderr:183124: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=3 2024-09-15T20:56:50.279 INFO:tasks.workunit.client.0.smithi136.stderr:183124: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 3-438086664195 2024-09-15T20:56:50.279 INFO:tasks.workunit.client.0.smithi136.stderr:183124: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T20:56:50.280 INFO:tasks.workunit.client.0.smithi136.stderr:183124: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=438086664195 2024-09-15T20:56:50.280 INFO:tasks.workunit.client.0.smithi136.stderr:183124: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.3 seq 438086664195' 2024-09-15T20:56:50.280 INFO:tasks.workunit.client.0.smithi136.stderr:183124: waiting osd.3 seq 438086664195 2024-09-15T20:56:50.280 INFO:tasks.workunit.client.0.smithi136.stderr:183124: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-15T20:56:50.280 INFO:tasks.workunit.client.0.smithi136.stderr:183124: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 438086664196 -lt 438086664195 2024-09-15T20:56:50.280 INFO:tasks.workunit.client.0.smithi136.stderr:183124: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-15T20:56:50.280 INFO:tasks.workunit.client.0.smithi136.stderr:183124: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T20:56:50.280 INFO:tasks.workunit.client.0.smithi136.stderr:183124: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T20:56:50.935 INFO:tasks.workunit.client.0.smithi136.stderr:183124: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 5 == 0 2024-09-15T20:56:50.935 INFO:tasks.workunit.client.0.smithi136.stderr:183124: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-15T20:56:50.936 INFO:tasks.workunit.client.0.smithi136.stderr:183124: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-15T20:56:50.936 INFO:tasks.workunit.client.0.smithi136.stderr:183124: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-15T20:56:50.936 INFO:tasks.workunit.client.0.smithi136.stderr:183124: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T20:56:50.936 INFO:tasks.workunit.client.0.smithi136.stderr:183124: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T20:56:50.936 INFO:tasks.workunit.client.0.smithi136.stderr:183124: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T20:56:50.936 INFO:tasks.workunit.client.0.smithi136.stderr:183124: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T20:56:50.937 INFO:tasks.workunit.client.0.smithi136.stderr:183124: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=5 2024-09-15T20:56:50.937 INFO:tasks.workunit.client.0.smithi136.stderr:183124: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-15T20:56:50.937 INFO:tasks.workunit.client.0.smithi136.stderr:183124: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T20:56:50.937 INFO:tasks.workunit.client.0.smithi136.stderr:183124: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T20:56:50.937 INFO:tasks.workunit.client.0.smithi136.stderr:183124: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 = 5 2024-09-15T20:56:50.937 INFO:tasks.workunit.client.0.smithi136.stderr:183124: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-15T20:56:50.937 INFO:tasks.workunit.client.0.smithi136.stderr:183124: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-15T20:56:50.938 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: run_in_background: return 0 2024-09-15T20:56:50.938 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: wait_background: eval 'pids='\'''\''' 2024-09-15T20:56:50.938 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: wait_background: pids= 2024-09-15T20:56:50.938 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: wait_background: return 0 2024-09-15T20:56:50.938 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:212: corrupt_and_repair_two: return_code=0 2024-09-15T20:56:50.938 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:213: corrupt_and_repair_two: '[' 0 -ne 0 ']' 2024-09-15T20:56:50.939 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:215: corrupt_and_repair_two: rados --pool ecpool get SOMETHING td/osd-scrub-repair/COPY 2024-09-15T20:56:50.971 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:216: corrupt_and_repair_two: diff td/osd-scrub-repair/ORIGINAL td/osd-scrub-repair/COPY 2024-09-15T20:56:50.972 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:66: run: teardown td/osd-scrub-repair 2024-09-15T20:56:50.972 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/osd-scrub-repair 2024-09-15T20:56:50.973 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-15T20:56:50.973 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/osd-scrub-repair KILL 2024-09-15T20:56:50.973 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-15T20:56:50.973 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-15T20:56:50.973 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-15T20:56:50.973 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-15T20:56:50.973 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-15T20:56:51.081 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-15T20:56:51.081 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-15T20:56:51.082 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-15T20:56:51.082 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-15T20:56:51.083 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-15T20:56:51.083 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-15T20:56:51.084 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-15T20:56:51.084 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T20:56:51.084 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-15T20:56:51.085 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-15T20:56:51.085 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T20:56:51.086 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-15T20:56:51.087 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-15T20:56:51.087 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/osd-scrub-repair 2024-09-15T20:56:51.118 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-15T20:56:51.118 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T20:56:51.118 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52602 2024-09-15T20:56:51.118 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.52602 2024-09-15T20:56:51.120 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-15T20:56:51.120 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-15T20:56:51.120 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:63: run: for func in $funcs 2024-09-15T20:56:51.120 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:64: run: setup td/osd-scrub-repair 2024-09-15T20:56:51.120 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/osd-scrub-repair 2024-09-15T20:56:51.120 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/osd-scrub-repair 2024-09-15T20:56:51.120 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/osd-scrub-repair 2024-09-15T20:56:51.120 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-15T20:56:51.120 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/osd-scrub-repair KILL 2024-09-15T20:56:51.120 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-15T20:56:51.121 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-15T20:56:51.121 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-15T20:56:51.121 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-15T20:56:51.121 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-15T20:56:51.121 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-15T20:56:51.121 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-15T20:56:51.122 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-15T20:56:51.123 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-15T20:56:51.123 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-15T20:56:51.124 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-15T20:56:51.124 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-15T20:56:51.125 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T20:56:51.125 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-15T20:56:51.125 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-15T20:56:51.125 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T20:56:51.126 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-15T20:56:51.127 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-15T20:56:51.127 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/osd-scrub-repair 2024-09-15T20:56:51.128 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-15T20:56:51.128 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T20:56:51.128 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52602 2024-09-15T20:56:51.128 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.52602 2024-09-15T20:56:51.129 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-15T20:56:51.129 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-15T20:56:51.129 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/osd-scrub-repair 2024-09-15T20:56:51.130 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2024-09-15T20:56:51.131 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T20:56:51.131 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52602 2024-09-15T20:56:51.131 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.52602 2024-09-15T20:56:51.133 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2024-09-15T20:56:51.133 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 4096 -le 1024 ']' 2024-09-15T20:56:51.133 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2024-09-15T20:56:51.133 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/osd-scrub-repair 1' TERM HUP INT 2024-09-15T20:56:51.133 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:65: run: TEST_corrupt_and_repair_lrc_appends td/osd-scrub-repair 2024-09-15T20:56:51.134 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:842: TEST_corrupt_and_repair_lrc_appends: corrupt_and_repair_lrc td/osd-scrub-repair false 2024-09-15T20:56:51.134 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:825: corrupt_and_repair_lrc: local dir=td/osd-scrub-repair 2024-09-15T20:56:51.134 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:826: corrupt_and_repair_lrc: local allow_overwrites=false 2024-09-15T20:56:51.134 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:827: corrupt_and_repair_lrc: local poolname=ecpool 2024-09-15T20:56:51.134 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:829: corrupt_and_repair_lrc: run_mon td/osd-scrub-repair a 2024-09-15T20:56:51.134 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/osd-scrub-repair 2024-09-15T20:56:51.134 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2024-09-15T20:56:51.134 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2024-09-15T20:56:51.134 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2024-09-15T20:56:51.134 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/osd-scrub-repair/a 2024-09-15T20:56:51.134 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-scrub-repair/a --run-dir=td/osd-scrub-repair 2024-09-15T20:56:51.586 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2024-09-15T20:56:51.587 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-15T20:56:51.587 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-15T20:56:51.587 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-15T20:56:51.587 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T20:56:51.587 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52602 2024-09-15T20:56:51.587 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.52602/$cluster-$name.asok' 2024-09-15T20:56:51.588 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-scrub-repair/a '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.52602/$cluster-$name.asok' --mon-cluster-log-file=td/osd-scrub-repair/log --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-09-15T20:56:51.787 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2024-09-15T20:56:51.789 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2024-09-15T20:56:51.789 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-15T20:56:51.789 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-15T20:56:51.789 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2024-09-15T20:56:51.789 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2024-09-15T20:56:51.789 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-15T20:56:51.789 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-15T20:56:51.789 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-15T20:56:51.790 INFO:tasks.workunit.client.0.smithi136.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-15T20:56:51.790 INFO:tasks.workunit.client.0.smithi136.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T20:56:51.790 INFO:tasks.workunit.client.0.smithi136.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52602 2024-09-15T20:56:51.790 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.52602/ceph-mon.a.asok 2024-09-15T20:56:51.790 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-15T20:56:51.790 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.52602/ceph-mon.a.asok config get fsid 2024-09-15T20:56:51.870 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2024-09-15T20:56:51.870 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-15T20:56:51.870 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-15T20:56:51.870 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2024-09-15T20:56:51.871 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2024-09-15T20:56:51.871 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-15T20:56:51.871 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-15T20:56:51.871 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-15T20:56:51.871 INFO:tasks.workunit.client.0.smithi136.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-15T20:56:51.871 INFO:tasks.workunit.client.0.smithi136.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T20:56:51.871 INFO:tasks.workunit.client.0.smithi136.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52602 2024-09-15T20:56:51.871 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.52602/ceph-mon.a.asok 2024-09-15T20:56:51.871 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-15T20:56:51.871 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.52602/ceph-mon.a.asok config get mon_host 2024-09-15T20:56:51.948 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:830: corrupt_and_repair_lrc: run_mgr td/osd-scrub-repair x 2024-09-15T20:56:51.948 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: run_mgr: local dir=td/osd-scrub-repair 2024-09-15T20:56:51.948 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: run_mgr: shift 2024-09-15T20:56:51.949 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: run_mgr: local id=x 2024-09-15T20:56:51.949 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: run_mgr: shift 2024-09-15T20:56:51.949 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:548: run_mgr: local data=td/osd-scrub-repair/x 2024-09-15T20:56:51.949 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:550: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-15T20:56:52.755 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: get_asok_path 2024-09-15T20:56:52.755 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-15T20:56:52.755 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-15T20:56:52.755 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-15T20:56:52.756 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T20:56:52.756 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52602 2024-09-15T20:56:52.756 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.52602/$cluster-$name.asok' 2024-09-15T20:56:52.756 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-15T20:56:52.757 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-scrub-repair/x '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.52602/$cluster-$name.asok' --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-15T20:56:52.785 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:831: corrupt_and_repair_lrc: seq 0 9 2024-09-15T20:56:52.788 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:831: corrupt_and_repair_lrc: for id in $(seq 0 9) 2024-09-15T20:56:52.789 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:832: corrupt_and_repair_lrc: run_osd td/osd-scrub-repair 0 2024-09-15T20:56:52.789 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-scrub-repair 2024-09-15T20:56:52.789 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-15T20:56:52.789 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=0 2024-09-15T20:56:52.789 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-15T20:56:52.789 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-scrub-repair/0 2024-09-15T20:56:52.789 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=ce4892c7-6c6d-4b4f-8e6f-07a3e11133e5 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-15T20:56:52.789 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T20:56:52.789 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T20:56:52.789 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T20:56:52.789 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-09-15T20:56:52.789 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-09-15T20:56:52.789 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-15T20:56:52.790 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-15T20:56:52.790 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-15T20:56:52.790 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-15T20:56:52.790 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-15T20:56:52.790 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-15T20:56:52.790 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-15T20:56:52.790 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T20:56:52.790 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52602 2024-09-15T20:56:52.790 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.52602/$cluster-$name.asok' 2024-09-15T20:56:52.790 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.52602/$cluster-$name.asok' 2024-09-15T20:56:52.790 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-15T20:56:52.790 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-15T20:56:52.790 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-15T20:56:52.791 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-15T20:56:52.791 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-15T20:56:52.791 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T20:56:52.791 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T20:56:52.791 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T20:56:52.791 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T20:56:52.791 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-15T20:56:52.791 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-15T20:56:52.791 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-scrub-repair/0 2024-09-15T20:56:52.791 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-15T20:56:52.792 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=b786e26d-301c-4029-aca5-1b2aac3b673f 2024-09-15T20:56:52.792 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd0 b786e26d-301c-4029-aca5-1b2aac3b673f' 2024-09-15T20:56:52.792 INFO:tasks.workunit.client.0.smithi136.stdout:add osd0 b786e26d-301c-4029-aca5-1b2aac3b673f 2024-09-15T20:56:52.793 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-15T20:56:52.808 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQAUSudmaF4FMBAAyx+teQTVJm6EtOxtvHOY2g== 2024-09-15T20:56:52.808 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQAUSudmaF4FMBAAyx+teQTVJm6EtOxtvHOY2g=="}' 2024-09-15T20:56:52.809 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new b786e26d-301c-4029-aca5-1b2aac3b673f -i td/osd-scrub-repair/0/new.json 2024-09-15T20:56:53.011 INFO:tasks.workunit.client.0.smithi136.stdout:0 2024-09-15T20:56:53.023 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-scrub-repair/0/new.json 2024-09-15T20:56:53.023 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 0 --fsid=ce4892c7-6c6d-4b4f-8e6f-07a3e11133e5 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.52602/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAUSudmaF4FMBAAyx+teQTVJm6EtOxtvHOY2g== --osd-uuid b786e26d-301c-4029-aca5-1b2aac3b673f 2024-09-15T20:56:53.052 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:56:53.049+0000 7ff703e10640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T20:56:53.062 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:56:53.060+0000 7ff703e10640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T20:56:53.063 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:56:53.061+0000 7ff703e10640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T20:56:53.064 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:56:53.061+0000 7ff703e10640 -1 bdev(0x5575c1db3c00 td/osd-scrub-repair/0/block) open stat got: (1) Operation not permitted 2024-09-15T20:56:53.064 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:56:53.061+0000 7ff703e10640 -1 bluestore(td/osd-scrub-repair/0) _read_fsid unparsable uuid 2024-09-15T20:56:58.778 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-scrub-repair/0/keyring 2024-09-15T20:56:58.778 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-15T20:56:58.779 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd0 key to auth repository 2024-09-15T20:56:58.779 INFO:tasks.workunit.client.0.smithi136.stdout:adding osd0 key to auth repository 2024-09-15T20:56:58.779 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-scrub-repair/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-15T20:56:59.172 INFO:tasks.workunit.client.0.smithi136.stdout:start osd.0 2024-09-15T20:56:59.172 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.0 2024-09-15T20:56:59.173 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 0 --fsid=ce4892c7-6c6d-4b4f-8e6f-07a3e11133e5 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.52602/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T20:56:59.173 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-15T20:56:59.173 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-15T20:56:59.173 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-15T20:56:59.199 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:56:59.196+0000 7f5485644640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T20:56:59.200 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:56:59.198+0000 7f5485644640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T20:56:59.201 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:56:59.199+0000 7f5485644640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T20:56:59.447 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 0 2024-09-15T20:56:59.447 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-15T20:56:59.447 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2024-09-15T20:56:59.447 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-15T20:56:59.447 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-15T20:56:59.447 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:56:59.447 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-15T20:56:59.447 INFO:tasks.workunit.client.0.smithi136.stdout:0 2024-09-15T20:56:59.448 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:56:59.448 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T20:56:59.726 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:57:00.569 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:57:00.567+0000 7f5485644640 -1 Falling back to public interface 2024-09-15T20:57:00.728 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:57:00.728 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:57:00.728 INFO:tasks.workunit.client.0.smithi136.stdout:1 2024-09-15T20:57:00.728 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-15T20:57:00.729 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:57:00.729 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T20:57:01.013 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:57:02.014 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:57:02.014 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:57:02.015 INFO:tasks.workunit.client.0.smithi136.stdout:2 2024-09-15T20:57:02.015 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-15T20:57:02.015 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:57:02.015 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T20:57:02.292 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:57:03.190 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:57:03.188+0000 7f5485644640 -1 osd.0 0 log_to_monitors true 2024-09-15T20:57:03.293 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:57:03.293 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:57:03.293 INFO:tasks.workunit.client.0.smithi136.stdout:3 2024-09-15T20:57:03.294 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-15T20:57:03.294 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:57:03.294 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T20:57:03.623 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:57:04.625 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:57:04.625 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:57:04.625 INFO:tasks.workunit.client.0.smithi136.stdout:4 2024-09-15T20:57:04.626 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-15T20:57:04.626 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:57:04.626 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T20:57:04.990 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:57:05.992 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:57:05.992 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:57:05.992 INFO:tasks.workunit.client.0.smithi136.stdout:5 2024-09-15T20:57:05.993 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-15T20:57:05.993 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:57:05.993 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T20:57:06.278 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:57:07.280 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:57:07.280 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:57:07.280 INFO:tasks.workunit.client.0.smithi136.stdout:6 2024-09-15T20:57:07.281 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-15T20:57:07.281 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:57:07.281 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T20:57:07.570 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:57:08.571 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:57:08.571 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:57:08.571 INFO:tasks.workunit.client.0.smithi136.stdout:7 2024-09-15T20:57:08.572 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-15T20:57:08.572 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:57:08.572 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T20:57:08.859 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:57:09.861 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:57:09.861 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:57:09.861 INFO:tasks.workunit.client.0.smithi136.stdout:8 2024-09-15T20:57:09.861 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-15T20:57:09.862 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:57:09.862 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T20:57:10.152 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:57:11.154 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:57:11.154 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:57:11.154 INFO:tasks.workunit.client.0.smithi136.stdout:9 2024-09-15T20:57:11.155 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-15T20:57:11.155 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:57:11.155 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T20:57:11.440 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:57:12.441 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:57:12.441 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:57:12.442 INFO:tasks.workunit.client.0.smithi136.stdout:10 2024-09-15T20:57:12.442 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-15T20:57:12.442 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:57:12.442 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T20:57:12.729 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:57:13.730 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:57:13.730 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:57:13.731 INFO:tasks.workunit.client.0.smithi136.stdout:11 2024-09-15T20:57:13.731 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-15T20:57:13.731 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:57:13.731 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T20:57:14.151 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:57:15.152 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:57:15.153 INFO:tasks.workunit.client.0.smithi136.stdout:12 2024-09-15T20:57:15.153 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:57:15.153 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-15T20:57:15.153 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:57:15.153 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T20:57:15.516 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:57:16.518 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:57:16.518 INFO:tasks.workunit.client.0.smithi136.stdout:13 2024-09-15T20:57:16.518 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:57:16.519 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-15T20:57:16.519 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:57:16.519 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T20:57:17.298 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:57:18.299 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:57:18.300 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:57:18.300 INFO:tasks.workunit.client.0.smithi136.stdout:14 2024-09-15T20:57:18.301 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-15T20:57:18.301 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:57:18.301 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T20:57:18.845 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:57:19.847 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:57:19.847 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:57:19.847 INFO:tasks.workunit.client.0.smithi136.stdout:15 2024-09-15T20:57:19.848 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-15T20:57:19.848 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:57:19.848 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T20:57:20.454 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:57:21.456 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:57:21.456 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:57:21.456 INFO:tasks.workunit.client.0.smithi136.stdout:16 2024-09-15T20:57:21.457 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-15T20:57:21.457 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:57:21.457 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T20:57:22.159 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:57:23.160 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:57:23.161 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:57:23.161 INFO:tasks.workunit.client.0.smithi136.stdout:17 2024-09-15T20:57:23.161 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-15T20:57:23.161 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:57:23.161 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T20:57:23.891 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:57:24.893 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:57:24.893 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:57:24.893 INFO:tasks.workunit.client.0.smithi136.stdout:18 2024-09-15T20:57:24.894 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 18 2024-09-15T20:57:24.894 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:57:24.894 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T20:57:25.227 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:57:26.229 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:57:26.229 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:57:26.229 INFO:tasks.workunit.client.0.smithi136.stdout:19 2024-09-15T20:57:26.230 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 19 2024-09-15T20:57:26.230 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:57:26.230 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T20:57:26.538 INFO:tasks.workunit.client.0.smithi136.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/3269456345,v1:127.0.0.1:6803/3269456345] [v2:127.0.0.1:6804/3269456345,v1:127.0.0.1:6805/3269456345] exists,up b786e26d-301c-4029-aca5-1b2aac3b673f 2024-09-15T20:57:26.542 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-15T20:57:26.542 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-15T20:57:26.543 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-15T20:57:26.543 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:831: corrupt_and_repair_lrc: for id in $(seq 0 9) 2024-09-15T20:57:26.543 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:832: corrupt_and_repair_lrc: run_osd td/osd-scrub-repair 1 2024-09-15T20:57:26.543 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-scrub-repair 2024-09-15T20:57:26.543 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-15T20:57:26.543 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=1 2024-09-15T20:57:26.543 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-15T20:57:26.543 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-scrub-repair/1 2024-09-15T20:57:26.543 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=ce4892c7-6c6d-4b4f-8e6f-07a3e11133e5 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-15T20:57:26.543 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T20:57:26.543 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T20:57:26.543 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T20:57:26.543 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-09-15T20:57:26.544 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-09-15T20:57:26.544 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-15T20:57:26.544 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-15T20:57:26.544 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-15T20:57:26.544 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-15T20:57:26.544 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-15T20:57:26.544 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-15T20:57:26.544 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-15T20:57:26.544 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T20:57:26.544 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52602 2024-09-15T20:57:26.544 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.52602/$cluster-$name.asok' 2024-09-15T20:57:26.544 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.52602/$cluster-$name.asok' 2024-09-15T20:57:26.544 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-15T20:57:26.545 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-15T20:57:26.545 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-15T20:57:26.545 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-15T20:57:26.545 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-15T20:57:26.545 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T20:57:26.545 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T20:57:26.545 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T20:57:26.545 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T20:57:26.545 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-15T20:57:26.545 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-15T20:57:26.545 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-scrub-repair/1 2024-09-15T20:57:26.545 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-15T20:57:26.545 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=e564f693-b36e-4681-9d47-7337135ca59c 2024-09-15T20:57:26.545 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd1 e564f693-b36e-4681-9d47-7337135ca59c' 2024-09-15T20:57:26.546 INFO:tasks.workunit.client.0.smithi136.stdout:add osd1 e564f693-b36e-4681-9d47-7337135ca59c 2024-09-15T20:57:26.546 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-15T20:57:26.561 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQA2SudmlvhMIRAAFo4vRpKy20YPOqoIrsmn/g== 2024-09-15T20:57:26.561 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQA2SudmlvhMIRAAFo4vRpKy20YPOqoIrsmn/g=="}' 2024-09-15T20:57:26.561 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new e564f693-b36e-4681-9d47-7337135ca59c -i td/osd-scrub-repair/1/new.json 2024-09-15T20:57:26.903 INFO:tasks.workunit.client.0.smithi136.stdout:1 2024-09-15T20:57:26.915 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-scrub-repair/1/new.json 2024-09-15T20:57:26.916 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 1 --fsid=ce4892c7-6c6d-4b4f-8e6f-07a3e11133e5 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.52602/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQA2SudmlvhMIRAAFo4vRpKy20YPOqoIrsmn/g== --osd-uuid e564f693-b36e-4681-9d47-7337135ca59c 2024-09-15T20:57:26.945 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:57:26.942+0000 7f125e80d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T20:57:27.009 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:57:27.007+0000 7f125e80d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T20:57:27.010 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:57:27.008+0000 7f125e80d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T20:57:27.011 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:57:27.008+0000 7f125e80d640 -1 bdev(0x562a425bdc00 td/osd-scrub-repair/1/block) open stat got: (1) Operation not permitted 2024-09-15T20:57:27.011 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:57:27.008+0000 7f125e80d640 -1 bluestore(td/osd-scrub-repair/1) _read_fsid unparsable uuid 2024-09-15T20:57:33.214 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-scrub-repair/1/keyring 2024-09-15T20:57:33.214 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-15T20:57:33.215 INFO:tasks.workunit.client.0.smithi136.stdout:adding osd1 key to auth repository 2024-09-15T20:57:33.216 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd1 key to auth repository 2024-09-15T20:57:33.216 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-scrub-repair/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-15T20:57:33.577 INFO:tasks.workunit.client.0.smithi136.stdout:start osd.1 2024-09-15T20:57:33.578 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.1 2024-09-15T20:57:33.578 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 1 --fsid=ce4892c7-6c6d-4b4f-8e6f-07a3e11133e5 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.52602/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T20:57:33.578 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-15T20:57:33.578 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-15T20:57:33.578 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-15T20:57:33.604 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:57:33.602+0000 7f1722d1e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T20:57:33.606 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:57:33.604+0000 7f1722d1e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T20:57:33.607 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:57:33.605+0000 7f1722d1e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T20:57:33.852 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 1 2024-09-15T20:57:33.852 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-15T20:57:33.852 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2024-09-15T20:57:33.852 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-15T20:57:33.853 INFO:tasks.workunit.client.0.smithi136.stdout:0 2024-09-15T20:57:33.853 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-15T20:57:33.853 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:57:33.853 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-15T20:57:33.853 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:57:33.854 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T20:57:34.129 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:57:35.130 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:57:35.131 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:57:35.131 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-15T20:57:35.131 INFO:tasks.workunit.client.0.smithi136.stdout:1 2024-09-15T20:57:35.131 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:57:35.131 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T20:57:35.214 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:57:35.212+0000 7f1722d1e640 -1 Falling back to public interface 2024-09-15T20:57:35.415 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:57:36.417 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:57:36.417 INFO:tasks.workunit.client.0.smithi136.stdout:2 2024-09-15T20:57:36.418 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:57:36.418 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-15T20:57:36.418 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:57:36.418 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T20:57:36.702 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:57:37.704 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:57:37.704 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:57:37.704 INFO:tasks.workunit.client.0.smithi136.stdout:3 2024-09-15T20:57:37.705 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-15T20:57:37.705 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:57:37.705 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T20:57:37.885 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:57:37.882+0000 7f1722d1e640 -1 osd.1 0 log_to_monitors true 2024-09-15T20:57:37.994 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:57:38.995 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:57:38.995 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:57:38.996 INFO:tasks.workunit.client.0.smithi136.stdout:4 2024-09-15T20:57:38.996 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-15T20:57:38.996 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:57:38.996 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T20:57:39.292 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:57:40.293 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:57:40.293 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:57:40.294 INFO:tasks.workunit.client.0.smithi136.stdout:5 2024-09-15T20:57:40.294 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-15T20:57:40.294 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:57:40.294 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T20:57:40.585 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:57:41.586 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:57:41.586 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:57:41.587 INFO:tasks.workunit.client.0.smithi136.stdout:6 2024-09-15T20:57:41.587 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-15T20:57:41.587 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:57:41.587 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T20:57:41.968 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:57:42.969 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:57:42.969 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:57:42.970 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-15T20:57:42.970 INFO:tasks.workunit.client.0.smithi136.stdout:7 2024-09-15T20:57:42.970 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:57:42.970 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T20:57:43.366 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:57:44.367 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:57:44.367 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:57:44.368 INFO:tasks.workunit.client.0.smithi136.stdout:8 2024-09-15T20:57:44.368 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-15T20:57:44.368 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:57:44.368 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T20:57:44.705 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:57:45.707 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:57:45.707 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:57:45.707 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-15T20:57:45.707 INFO:tasks.workunit.client.0.smithi136.stdout:9 2024-09-15T20:57:45.707 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:57:45.708 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T20:57:45.997 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:57:46.999 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:57:46.999 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:57:46.999 INFO:tasks.workunit.client.0.smithi136.stdout:10 2024-09-15T20:57:47.000 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-15T20:57:47.000 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:57:47.000 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T20:57:47.390 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:57:48.391 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:57:48.391 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:57:48.391 INFO:tasks.workunit.client.0.smithi136.stdout:11 2024-09-15T20:57:48.392 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-15T20:57:48.392 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:57:48.392 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T20:57:48.675 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:57:49.677 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:57:49.677 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:57:49.677 INFO:tasks.workunit.client.0.smithi136.stdout:12 2024-09-15T20:57:49.678 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-15T20:57:49.678 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:57:49.678 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T20:57:49.965 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:57:50.966 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:57:50.967 INFO:tasks.workunit.client.0.smithi136.stdout:13 2024-09-15T20:57:50.967 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:57:50.967 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-15T20:57:50.967 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:57:50.967 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T20:57:51.662 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:57:52.663 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:57:52.663 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:57:52.664 INFO:tasks.workunit.client.0.smithi136.stdout:14 2024-09-15T20:57:52.664 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-15T20:57:52.664 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:57:52.664 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T20:57:52.995 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:57:53.997 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:57:53.997 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:57:53.997 INFO:tasks.workunit.client.0.smithi136.stdout:15 2024-09-15T20:57:53.998 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-15T20:57:53.998 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:57:53.998 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T20:57:54.498 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:57:55.499 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:57:55.500 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:57:55.500 INFO:tasks.workunit.client.0.smithi136.stdout:16 2024-09-15T20:57:55.500 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-15T20:57:55.500 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:57:55.501 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T20:57:56.362 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:57:57.364 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:57:57.364 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:57:57.364 INFO:tasks.workunit.client.0.smithi136.stdout:17 2024-09-15T20:57:57.365 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-15T20:57:57.365 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:57:57.365 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T20:57:57.803 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:57:58.805 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:57:58.805 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:57:58.805 INFO:tasks.workunit.client.0.smithi136.stdout:18 2024-09-15T20:57:58.806 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 18 2024-09-15T20:57:58.806 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:57:58.806 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T20:57:59.614 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:58:00.616 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:58:00.616 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:58:00.616 INFO:tasks.workunit.client.0.smithi136.stdout:19 2024-09-15T20:58:00.616 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 19 2024-09-15T20:58:00.617 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:58:00.617 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T20:58:01.020 INFO:tasks.workunit.client.0.smithi136.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/2334538585,v1:127.0.0.1:6811/2334538585] [v2:127.0.0.1:6812/2334538585,v1:127.0.0.1:6813/2334538585] exists,up e564f693-b36e-4681-9d47-7337135ca59c 2024-09-15T20:58:01.024 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-15T20:58:01.024 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-15T20:58:01.024 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-15T20:58:01.024 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:831: corrupt_and_repair_lrc: for id in $(seq 0 9) 2024-09-15T20:58:01.024 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:832: corrupt_and_repair_lrc: run_osd td/osd-scrub-repair 2 2024-09-15T20:58:01.024 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-scrub-repair 2024-09-15T20:58:01.024 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-15T20:58:01.024 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=2 2024-09-15T20:58:01.024 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-15T20:58:01.025 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-scrub-repair/2 2024-09-15T20:58:01.025 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=ce4892c7-6c6d-4b4f-8e6f-07a3e11133e5 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-15T20:58:01.025 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T20:58:01.025 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T20:58:01.025 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T20:58:01.025 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/2' 2024-09-15T20:58:01.025 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/2/journal' 2024-09-15T20:58:01.025 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-15T20:58:01.025 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-15T20:58:01.025 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-15T20:58:01.025 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-15T20:58:01.025 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-15T20:58:01.025 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-15T20:58:01.025 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-15T20:58:01.026 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T20:58:01.026 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52602 2024-09-15T20:58:01.026 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.52602/$cluster-$name.asok' 2024-09-15T20:58:01.026 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.52602/$cluster-$name.asok' 2024-09-15T20:58:01.026 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-15T20:58:01.026 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-15T20:58:01.026 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-15T20:58:01.026 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-15T20:58:01.026 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-15T20:58:01.026 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T20:58:01.026 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T20:58:01.026 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T20:58:01.026 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T20:58:01.026 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-15T20:58:01.027 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-15T20:58:01.027 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-scrub-repair/2 2024-09-15T20:58:01.027 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-15T20:58:01.027 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=011b5fdc-797f-48cd-9a7a-b9c0852b3220 2024-09-15T20:58:01.027 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd2 011b5fdc-797f-48cd-9a7a-b9c0852b3220' 2024-09-15T20:58:01.027 INFO:tasks.workunit.client.0.smithi136.stdout:add osd2 011b5fdc-797f-48cd-9a7a-b9c0852b3220 2024-09-15T20:58:01.027 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-15T20:58:01.043 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQBZSudmp5ZkAhAACrJnq5b+Kjht4u6NbdSXQQ== 2024-09-15T20:58:01.043 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQBZSudmp5ZkAhAACrJnq5b+Kjht4u6NbdSXQQ=="}' 2024-09-15T20:58:01.043 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 011b5fdc-797f-48cd-9a7a-b9c0852b3220 -i td/osd-scrub-repair/2/new.json 2024-09-15T20:58:01.466 INFO:tasks.workunit.client.0.smithi136.stdout:2 2024-09-15T20:58:01.478 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-scrub-repair/2/new.json 2024-09-15T20:58:01.479 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 2 --fsid=ce4892c7-6c6d-4b4f-8e6f-07a3e11133e5 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.52602/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBZSudmp5ZkAhAACrJnq5b+Kjht4u6NbdSXQQ== --osd-uuid 011b5fdc-797f-48cd-9a7a-b9c0852b3220 2024-09-15T20:58:01.509 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:58:01.506+0000 7f470dd54640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T20:58:01.551 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:58:01.548+0000 7f470dd54640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T20:58:01.552 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:58:01.549+0000 7f470dd54640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T20:58:01.552 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:58:01.550+0000 7f470dd54640 -1 bdev(0x56536fbce800 td/osd-scrub-repair/2/block) open stat got: (1) Operation not permitted 2024-09-15T20:58:01.552 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:58:01.550+0000 7f470dd54640 -1 bluestore(td/osd-scrub-repair/2) _read_fsid unparsable uuid 2024-09-15T20:58:07.633 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-scrub-repair/2/keyring 2024-09-15T20:58:07.633 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-15T20:58:07.634 INFO:tasks.workunit.client.0.smithi136.stdout:adding osd2 key to auth repository 2024-09-15T20:58:07.634 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd2 key to auth repository 2024-09-15T20:58:07.635 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-scrub-repair/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-15T20:58:07.994 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.2 2024-09-15T20:58:07.995 INFO:tasks.workunit.client.0.smithi136.stdout:start osd.2 2024-09-15T20:58:07.996 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 2 --fsid=ce4892c7-6c6d-4b4f-8e6f-07a3e11133e5 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.52602/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T20:58:07.996 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-15T20:58:07.996 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-15T20:58:07.996 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-15T20:58:08.022 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:58:08.019+0000 7fe376c0b640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T20:58:08.024 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:58:08.021+0000 7fe376c0b640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T20:58:08.025 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:58:08.022+0000 7fe376c0b640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T20:58:08.291 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 2 2024-09-15T20:58:08.291 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-15T20:58:08.291 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=2 2024-09-15T20:58:08.291 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-15T20:58:08.291 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-15T20:58:08.291 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:58:08.291 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-15T20:58:08.292 INFO:tasks.workunit.client.0.smithi136.stdout:0 2024-09-15T20:58:08.292 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:58:08.292 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T20:58:08.565 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:58:09.566 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:58:09.567 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:58:09.567 INFO:tasks.workunit.client.0.smithi136.stdout:1 2024-09-15T20:58:09.567 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-15T20:58:09.567 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:58:09.567 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T20:58:09.632 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:58:09.630+0000 7fe376c0b640 -1 Falling back to public interface 2024-09-15T20:58:09.855 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:58:10.856 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:58:10.856 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:58:10.857 INFO:tasks.workunit.client.0.smithi136.stdout:2 2024-09-15T20:58:10.857 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-15T20:58:10.857 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:58:10.857 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T20:58:11.153 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:58:12.155 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:58:12.155 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:58:12.155 INFO:tasks.workunit.client.0.smithi136.stdout:3 2024-09-15T20:58:12.156 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-15T20:58:12.156 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:58:12.156 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T20:58:12.441 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:58:12.518 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:58:12.515+0000 7fe376c0b640 -1 osd.2 0 log_to_monitors true 2024-09-15T20:58:13.442 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:58:13.442 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:58:13.442 INFO:tasks.workunit.client.0.smithi136.stdout:4 2024-09-15T20:58:13.443 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-15T20:58:13.443 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:58:13.443 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T20:58:13.883 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:58:14.884 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:58:14.884 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:58:14.884 INFO:tasks.workunit.client.0.smithi136.stdout:5 2024-09-15T20:58:14.885 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-15T20:58:14.885 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:58:14.885 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T20:58:15.206 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:58:16.208 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:58:16.208 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:58:16.208 INFO:tasks.workunit.client.0.smithi136.stdout:6 2024-09-15T20:58:16.209 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-15T20:58:16.209 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:58:16.209 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T20:58:16.606 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:58:17.608 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:58:17.608 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:58:17.608 INFO:tasks.workunit.client.0.smithi136.stdout:7 2024-09-15T20:58:17.609 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-15T20:58:17.609 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:58:17.609 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T20:58:17.897 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:58:18.899 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:58:18.899 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:58:18.899 INFO:tasks.workunit.client.0.smithi136.stdout:8 2024-09-15T20:58:18.900 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-15T20:58:18.900 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:58:18.900 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T20:58:19.231 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:58:20.232 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:58:20.232 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:58:20.233 INFO:tasks.workunit.client.0.smithi136.stdout:9 2024-09-15T20:58:20.233 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-15T20:58:20.233 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:58:20.233 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T20:58:20.582 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:58:21.583 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:58:21.583 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:58:21.583 INFO:tasks.workunit.client.0.smithi136.stdout:10 2024-09-15T20:58:21.584 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-15T20:58:21.584 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:58:21.584 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T20:58:21.870 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:58:22.871 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:58:22.871 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:58:22.872 INFO:tasks.workunit.client.0.smithi136.stdout:11 2024-09-15T20:58:22.872 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-15T20:58:22.872 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:58:22.872 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T20:58:23.187 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:58:24.189 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:58:24.189 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:58:24.189 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-15T20:58:24.189 INFO:tasks.workunit.client.0.smithi136.stdout:12 2024-09-15T20:58:24.189 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:58:24.189 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T20:58:24.477 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:58:25.478 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:58:25.478 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:58:25.479 INFO:tasks.workunit.client.0.smithi136.stdout:13 2024-09-15T20:58:25.479 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-15T20:58:25.479 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:58:25.479 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T20:58:26.431 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:58:27.433 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:58:27.433 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:58:27.433 INFO:tasks.workunit.client.0.smithi136.stdout:14 2024-09-15T20:58:27.434 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-15T20:58:27.434 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:58:27.434 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T20:58:28.183 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:58:29.185 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:58:29.185 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:58:29.186 INFO:tasks.workunit.client.0.smithi136.stdout:15 2024-09-15T20:58:29.186 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-15T20:58:29.186 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:58:29.187 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T20:58:29.751 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:58:30.752 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:58:30.752 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:58:30.753 INFO:tasks.workunit.client.0.smithi136.stdout:16 2024-09-15T20:58:30.753 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-15T20:58:30.753 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:58:30.753 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T20:58:31.475 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:58:32.476 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:58:32.476 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:58:32.477 INFO:tasks.workunit.client.0.smithi136.stdout:17 2024-09-15T20:58:32.478 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-15T20:58:32.478 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:58:32.478 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T20:58:32.836 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:58:33.840 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:58:33.841 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:58:33.841 INFO:tasks.workunit.client.0.smithi136.stdout:18 2024-09-15T20:58:33.841 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 18 2024-09-15T20:58:33.842 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:58:33.842 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T20:58:34.709 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:58:35.710 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:58:35.710 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:58:35.710 INFO:tasks.workunit.client.0.smithi136.stdout:19 2024-09-15T20:58:35.711 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 19 2024-09-15T20:58:35.711 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:58:35.711 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T20:58:36.094 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:58:37.095 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:58:37.095 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:58:37.095 INFO:tasks.workunit.client.0.smithi136.stdout:20 2024-09-15T20:58:37.096 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 20 2024-09-15T20:58:37.096 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:58:37.096 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T20:58:37.378 INFO:tasks.workunit.client.0.smithi136.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/1610376452,v1:127.0.0.1:6819/1610376452] [v2:127.0.0.1:6820/1610376452,v1:127.0.0.1:6821/1610376452] exists,up 011b5fdc-797f-48cd-9a7a-b9c0852b3220 2024-09-15T20:58:37.382 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-15T20:58:37.382 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-15T20:58:37.382 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-15T20:58:37.382 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:831: corrupt_and_repair_lrc: for id in $(seq 0 9) 2024-09-15T20:58:37.382 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:832: corrupt_and_repair_lrc: run_osd td/osd-scrub-repair 3 2024-09-15T20:58:37.382 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-scrub-repair 2024-09-15T20:58:37.382 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-15T20:58:37.383 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=3 2024-09-15T20:58:37.383 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-15T20:58:37.383 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-scrub-repair/3 2024-09-15T20:58:37.383 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=ce4892c7-6c6d-4b4f-8e6f-07a3e11133e5 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-15T20:58:37.383 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T20:58:37.383 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T20:58:37.383 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T20:58:37.383 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/3' 2024-09-15T20:58:37.383 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/3/journal' 2024-09-15T20:58:37.383 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-15T20:58:37.383 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-15T20:58:37.383 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-15T20:58:37.383 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-15T20:58:37.383 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-15T20:58:37.384 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-15T20:58:37.384 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-15T20:58:37.384 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T20:58:37.384 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52602 2024-09-15T20:58:37.384 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.52602/$cluster-$name.asok' 2024-09-15T20:58:37.384 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.52602/$cluster-$name.asok' 2024-09-15T20:58:37.384 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-15T20:58:37.384 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-15T20:58:37.384 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-15T20:58:37.384 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-15T20:58:37.384 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-15T20:58:37.384 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T20:58:37.384 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T20:58:37.385 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T20:58:37.385 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T20:58:37.385 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-15T20:58:37.385 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-15T20:58:37.385 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-scrub-repair/3 2024-09-15T20:58:37.385 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-15T20:58:37.385 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=0eed652b-57cc-457d-986e-cdd453c584f5 2024-09-15T20:58:37.385 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd3 0eed652b-57cc-457d-986e-cdd453c584f5' 2024-09-15T20:58:37.385 INFO:tasks.workunit.client.0.smithi136.stdout:add osd3 0eed652b-57cc-457d-986e-cdd453c584f5 2024-09-15T20:58:37.385 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-15T20:58:37.401 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQB9SudmDvTIFxAAybZ31ME6GlwoBNa+6UVFeQ== 2024-09-15T20:58:37.402 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQB9SudmDvTIFxAAybZ31ME6GlwoBNa+6UVFeQ=="}' 2024-09-15T20:58:37.402 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 0eed652b-57cc-457d-986e-cdd453c584f5 -i td/osd-scrub-repair/3/new.json 2024-09-15T20:58:37.769 INFO:tasks.workunit.client.0.smithi136.stdout:3 2024-09-15T20:58:37.780 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-scrub-repair/3/new.json 2024-09-15T20:58:37.781 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 3 --fsid=ce4892c7-6c6d-4b4f-8e6f-07a3e11133e5 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/3 --osd-journal=td/osd-scrub-repair/3/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.52602/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQB9SudmDvTIFxAAybZ31ME6GlwoBNa+6UVFeQ== --osd-uuid 0eed652b-57cc-457d-986e-cdd453c584f5 2024-09-15T20:58:37.810 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:58:37.807+0000 7efd1a134640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T20:58:37.870 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:58:37.868+0000 7efd1a134640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T20:58:37.871 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:58:37.869+0000 7efd1a134640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T20:58:37.872 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:58:37.869+0000 7efd1a134640 -1 bdev(0x564c97939c00 td/osd-scrub-repair/3/block) open stat got: (1) Operation not permitted 2024-09-15T20:58:37.872 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:58:37.869+0000 7efd1a134640 -1 bluestore(td/osd-scrub-repair/3) _read_fsid unparsable uuid 2024-09-15T20:58:44.174 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-scrub-repair/3/keyring 2024-09-15T20:58:44.174 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-15T20:58:44.175 INFO:tasks.workunit.client.0.smithi136.stdout:adding osd3 key to auth repository 2024-09-15T20:58:44.175 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd3 key to auth repository 2024-09-15T20:58:44.175 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-scrub-repair/3/keyring auth add osd.3 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-15T20:58:44.536 INFO:tasks.workunit.client.0.smithi136.stdout:start osd.3 2024-09-15T20:58:44.536 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.3 2024-09-15T20:58:44.537 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 3 --fsid=ce4892c7-6c6d-4b4f-8e6f-07a3e11133e5 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/3 --osd-journal=td/osd-scrub-repair/3/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.52602/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T20:58:44.537 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-15T20:58:44.537 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-15T20:58:44.537 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-15T20:58:44.563 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:58:44.559+0000 7f96228ef640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T20:58:44.565 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:58:44.562+0000 7f96228ef640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T20:58:44.566 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:58:44.563+0000 7f96228ef640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T20:58:44.834 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 3 2024-09-15T20:58:44.834 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-15T20:58:44.835 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=3 2024-09-15T20:58:44.835 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-15T20:58:44.835 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-15T20:58:44.835 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:58:44.835 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-15T20:58:44.836 INFO:tasks.workunit.client.0.smithi136.stdout:0 2024-09-15T20:58:44.836 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:58:44.836 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-15T20:58:45.109 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:58:45.925 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:58:45.921+0000 7f96228ef640 -1 Falling back to public interface 2024-09-15T20:58:46.110 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:58:46.110 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:58:46.111 INFO:tasks.workunit.client.0.smithi136.stdout:1 2024-09-15T20:58:46.111 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-15T20:58:46.111 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:58:46.111 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-15T20:58:46.396 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:58:47.397 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:58:47.398 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:58:47.398 INFO:tasks.workunit.client.0.smithi136.stdout:2 2024-09-15T20:58:47.398 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-15T20:58:47.398 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:58:47.399 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-15T20:58:47.674 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:58:48.675 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:58:48.676 INFO:tasks.workunit.client.0.smithi136.stdout:3 2024-09-15T20:58:48.676 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:58:48.676 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-15T20:58:48.676 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:58:48.676 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-15T20:58:48.881 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:58:48.878+0000 7f96228ef640 -1 osd.3 0 log_to_monitors true 2024-09-15T20:58:48.964 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:58:49.965 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:58:49.965 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:58:49.966 INFO:tasks.workunit.client.0.smithi136.stdout:4 2024-09-15T20:58:49.966 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-15T20:58:49.966 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:58:49.966 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-15T20:58:50.246 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:58:51.248 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:58:51.248 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:58:51.248 INFO:tasks.workunit.client.0.smithi136.stdout:5 2024-09-15T20:58:51.249 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-15T20:58:51.249 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:58:51.249 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-15T20:58:51.536 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:58:52.538 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:58:52.538 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:58:52.538 INFO:tasks.workunit.client.0.smithi136.stdout:6 2024-09-15T20:58:52.539 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-15T20:58:52.539 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:58:52.539 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-15T20:58:53.041 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:58:54.042 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:58:54.043 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:58:54.043 INFO:tasks.workunit.client.0.smithi136.stdout:7 2024-09-15T20:58:54.043 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-15T20:58:54.043 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:58:54.043 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-15T20:58:54.332 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:58:55.333 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:58:55.333 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:58:55.333 INFO:tasks.workunit.client.0.smithi136.stdout:8 2024-09-15T20:58:55.334 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-15T20:58:55.334 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:58:55.334 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-15T20:58:55.643 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:58:56.645 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:58:56.645 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:58:56.645 INFO:tasks.workunit.client.0.smithi136.stdout:9 2024-09-15T20:58:56.646 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-15T20:58:56.646 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:58:56.646 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-15T20:58:57.027 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:58:58.028 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:58:58.028 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:58:58.029 INFO:tasks.workunit.client.0.smithi136.stdout:10 2024-09-15T20:58:58.029 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-15T20:58:58.029 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:58:58.029 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-15T20:58:58.319 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:58:59.321 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:58:59.321 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:58:59.321 INFO:tasks.workunit.client.0.smithi136.stdout:11 2024-09-15T20:58:59.322 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-15T20:58:59.322 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:58:59.322 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-15T20:58:59.636 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:59:00.637 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:59:00.637 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:59:00.637 INFO:tasks.workunit.client.0.smithi136.stdout:12 2024-09-15T20:59:00.638 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-15T20:59:00.638 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:59:00.638 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-15T20:59:01.050 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:59:02.051 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:59:02.054 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:59:02.054 INFO:tasks.workunit.client.0.smithi136.stdout:13 2024-09-15T20:59:02.055 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-15T20:59:02.055 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:59:02.055 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-15T20:59:02.790 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:59:03.792 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:59:03.792 INFO:tasks.workunit.client.0.smithi136.stdout:14 2024-09-15T20:59:03.793 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:59:03.793 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-15T20:59:03.794 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:59:03.794 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-15T20:59:04.564 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:59:05.565 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:59:05.565 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:59:05.565 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-15T20:59:05.566 INFO:tasks.workunit.client.0.smithi136.stdout:15 2024-09-15T20:59:05.566 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:59:05.566 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-15T20:59:05.880 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:59:06.881 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:59:06.881 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:59:06.881 INFO:tasks.workunit.client.0.smithi136.stdout:16 2024-09-15T20:59:06.882 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-15T20:59:06.882 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:59:06.882 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-15T20:59:07.307 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:59:08.309 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:59:08.309 INFO:tasks.workunit.client.0.smithi136.stdout:17 2024-09-15T20:59:08.310 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:59:08.310 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-15T20:59:08.310 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:59:08.310 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-15T20:59:08.777 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:59:09.778 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:59:09.779 INFO:tasks.workunit.client.0.smithi136.stdout:18 2024-09-15T20:59:09.779 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:59:09.779 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 18 2024-09-15T20:59:09.779 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:59:09.780 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-15T20:59:10.256 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:59:11.257 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:59:11.257 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:59:11.258 INFO:tasks.workunit.client.0.smithi136.stdout:19 2024-09-15T20:59:11.258 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 19 2024-09-15T20:59:11.258 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:59:11.258 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-15T20:59:12.040 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:59:13.042 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:59:13.042 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:59:13.042 INFO:tasks.workunit.client.0.smithi136.stdout:20 2024-09-15T20:59:13.043 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 20 2024-09-15T20:59:13.043 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:59:13.043 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-15T20:59:13.335 INFO:tasks.workunit.client.0.smithi136.stdout:osd.3 up in weight 1 up_from 20 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6826/615347575,v1:127.0.0.1:6827/615347575] [v2:127.0.0.1:6828/615347575,v1:127.0.0.1:6829/615347575] exists,up 0eed652b-57cc-457d-986e-cdd453c584f5 2024-09-15T20:59:13.337 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-15T20:59:13.337 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-15T20:59:13.337 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-15T20:59:13.337 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:831: corrupt_and_repair_lrc: for id in $(seq 0 9) 2024-09-15T20:59:13.337 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:832: corrupt_and_repair_lrc: run_osd td/osd-scrub-repair 4 2024-09-15T20:59:13.337 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-scrub-repair 2024-09-15T20:59:13.337 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-15T20:59:13.337 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=4 2024-09-15T20:59:13.337 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-15T20:59:13.338 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-scrub-repair/4 2024-09-15T20:59:13.338 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=ce4892c7-6c6d-4b4f-8e6f-07a3e11133e5 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-15T20:59:13.338 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T20:59:13.338 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T20:59:13.338 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T20:59:13.338 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/4' 2024-09-15T20:59:13.338 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/4/journal' 2024-09-15T20:59:13.340 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-15T20:59:13.341 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-15T20:59:13.341 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-15T20:59:13.341 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-15T20:59:13.341 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-15T20:59:13.341 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-15T20:59:13.341 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-15T20:59:13.341 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T20:59:13.341 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52602 2024-09-15T20:59:13.341 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.52602/$cluster-$name.asok' 2024-09-15T20:59:13.341 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.52602/$cluster-$name.asok' 2024-09-15T20:59:13.341 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-15T20:59:13.342 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-15T20:59:13.342 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-15T20:59:13.342 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-15T20:59:13.342 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-15T20:59:13.342 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T20:59:13.342 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T20:59:13.342 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T20:59:13.342 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T20:59:13.342 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-15T20:59:13.342 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-15T20:59:13.342 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-scrub-repair/4 2024-09-15T20:59:13.342 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-15T20:59:13.342 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=efc7afae-88a4-411c-bd12-386c1aa780f9 2024-09-15T20:59:13.343 INFO:tasks.workunit.client.0.smithi136.stdout:add osd4 efc7afae-88a4-411c-bd12-386c1aa780f9 2024-09-15T20:59:13.343 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd4 efc7afae-88a4-411c-bd12-386c1aa780f9' 2024-09-15T20:59:13.343 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-15T20:59:13.356 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQChSudmnGi8FBAAi81wjhwvG0Cnqpny/0bxqQ== 2024-09-15T20:59:13.357 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQChSudmnGi8FBAAi81wjhwvG0Cnqpny/0bxqQ=="}' 2024-09-15T20:59:13.357 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new efc7afae-88a4-411c-bd12-386c1aa780f9 -i td/osd-scrub-repair/4/new.json 2024-09-15T20:59:13.717 INFO:tasks.workunit.client.0.smithi136.stdout:4 2024-09-15T20:59:13.729 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-scrub-repair/4/new.json 2024-09-15T20:59:13.730 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 4 --fsid=ce4892c7-6c6d-4b4f-8e6f-07a3e11133e5 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/4 --osd-journal=td/osd-scrub-repair/4/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.52602/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQChSudmnGi8FBAAi81wjhwvG0Cnqpny/0bxqQ== --osd-uuid efc7afae-88a4-411c-bd12-386c1aa780f9 2024-09-15T20:59:13.759 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:59:13.757+0000 7f260b40e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T20:59:13.801 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:59:13.799+0000 7f260b40e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T20:59:13.802 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:59:13.800+0000 7f260b40e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T20:59:13.803 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:59:13.800+0000 7f260b40e640 -1 bdev(0x558d2b6bfc00 td/osd-scrub-repair/4/block) open stat got: (1) Operation not permitted 2024-09-15T20:59:13.803 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:59:13.800+0000 7f260b40e640 -1 bluestore(td/osd-scrub-repair/4) _read_fsid unparsable uuid 2024-09-15T20:59:20.064 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-scrub-repair/4/keyring 2024-09-15T20:59:20.064 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-15T20:59:20.065 INFO:tasks.workunit.client.0.smithi136.stdout:adding osd4 key to auth repository 2024-09-15T20:59:20.065 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd4 key to auth repository 2024-09-15T20:59:20.065 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-scrub-repair/4/keyring auth add osd.4 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-15T20:59:20.427 INFO:tasks.workunit.client.0.smithi136.stdout:start osd.4 2024-09-15T20:59:20.428 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.4 2024-09-15T20:59:20.428 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 4 --fsid=ce4892c7-6c6d-4b4f-8e6f-07a3e11133e5 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/4 --osd-journal=td/osd-scrub-repair/4/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.52602/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T20:59:20.428 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-15T20:59:20.428 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-15T20:59:20.428 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-15T20:59:20.454 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:59:20.452+0000 7ffa2a102640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T20:59:20.457 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:59:20.454+0000 7ffa2a102640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T20:59:20.458 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:59:20.455+0000 7ffa2a102640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T20:59:20.704 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 4 2024-09-15T20:59:20.704 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-15T20:59:20.704 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=4 2024-09-15T20:59:20.704 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-15T20:59:20.704 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-15T20:59:20.704 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:59:20.704 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-15T20:59:20.705 INFO:tasks.workunit.client.0.smithi136.stdout:0 2024-09-15T20:59:20.705 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:59:20.705 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-15T20:59:21.003 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:59:22.004 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:59:22.004 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:59:22.004 INFO:tasks.workunit.client.0.smithi136.stdout:1 2024-09-15T20:59:22.005 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-15T20:59:22.005 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:59:22.005 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-15T20:59:22.093 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:59:22.091+0000 7ffa2a102640 -1 Falling back to public interface 2024-09-15T20:59:22.283 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:59:23.284 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:59:23.285 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:59:23.285 INFO:tasks.workunit.client.0.smithi136.stdout:2 2024-09-15T20:59:23.285 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-15T20:59:23.286 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:59:23.286 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-15T20:59:23.572 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:59:24.573 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:59:24.573 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:59:24.573 INFO:tasks.workunit.client.0.smithi136.stdout:3 2024-09-15T20:59:24.574 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-15T20:59:24.574 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:59:24.574 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-15T20:59:24.742 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:59:24.739+0000 7ffa2a102640 -1 osd.4 0 log_to_monitors true 2024-09-15T20:59:24.942 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:59:25.943 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:59:25.943 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:59:25.943 INFO:tasks.workunit.client.0.smithi136.stdout:4 2024-09-15T20:59:25.944 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-15T20:59:25.944 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:59:25.944 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-15T20:59:26.229 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:59:27.230 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:59:27.230 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:59:27.231 INFO:tasks.workunit.client.0.smithi136.stdout:5 2024-09-15T20:59:27.231 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-15T20:59:27.231 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:59:27.231 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-15T20:59:27.570 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:59:28.572 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:59:28.572 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:59:28.572 INFO:tasks.workunit.client.0.smithi136.stdout:6 2024-09-15T20:59:28.573 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-15T20:59:28.573 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:59:28.573 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-15T20:59:28.892 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:59:29.894 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:59:29.894 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:59:29.894 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-15T20:59:29.894 INFO:tasks.workunit.client.0.smithi136.stdout:7 2024-09-15T20:59:29.895 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:59:29.895 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-15T20:59:30.184 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:59:31.186 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:59:31.186 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:59:31.186 INFO:tasks.workunit.client.0.smithi136.stdout:8 2024-09-15T20:59:31.187 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-15T20:59:31.187 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:59:31.187 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-15T20:59:31.541 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:59:32.543 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:59:32.543 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:59:32.543 INFO:tasks.workunit.client.0.smithi136.stdout:9 2024-09-15T20:59:32.544 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-15T20:59:32.544 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:59:32.544 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-15T20:59:32.884 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:59:33.886 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:59:33.886 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:59:33.887 INFO:tasks.workunit.client.0.smithi136.stdout:10 2024-09-15T20:59:33.887 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-15T20:59:33.887 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:59:33.887 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-15T20:59:34.173 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:59:35.174 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:59:35.174 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:59:35.174 INFO:tasks.workunit.client.0.smithi136.stdout:11 2024-09-15T20:59:35.175 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-15T20:59:35.175 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:59:35.175 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-15T20:59:35.565 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:59:36.566 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:59:36.566 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:59:36.567 INFO:tasks.workunit.client.0.smithi136.stdout:12 2024-09-15T20:59:36.567 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-15T20:59:36.567 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:59:36.567 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-15T20:59:37.336 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:59:38.337 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:59:38.338 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:59:38.338 INFO:tasks.workunit.client.0.smithi136.stdout:13 2024-09-15T20:59:38.338 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-15T20:59:38.338 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:59:38.339 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-15T20:59:38.957 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:59:39.958 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:59:39.958 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:59:39.959 INFO:tasks.workunit.client.0.smithi136.stdout:14 2024-09-15T20:59:39.959 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-15T20:59:39.959 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:59:39.959 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-15T20:59:40.754 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:59:41.755 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:59:41.755 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:59:41.755 INFO:tasks.workunit.client.0.smithi136.stdout:15 2024-09-15T20:59:41.756 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-15T20:59:41.756 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:59:41.756 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-15T20:59:42.405 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:59:43.406 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:59:43.406 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:59:43.406 INFO:tasks.workunit.client.0.smithi136.stdout:16 2024-09-15T20:59:43.407 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-15T20:59:43.407 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:59:43.407 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-15T20:59:43.813 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:59:44.814 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:59:44.814 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:59:44.814 INFO:tasks.workunit.client.0.smithi136.stdout:17 2024-09-15T20:59:44.815 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-15T20:59:44.815 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:59:44.815 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-15T20:59:45.290 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:59:46.292 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:59:46.292 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:59:46.292 INFO:tasks.workunit.client.0.smithi136.stdout:18 2024-09-15T20:59:46.293 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 18 2024-09-15T20:59:46.293 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:59:46.293 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-15T20:59:47.038 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:59:48.039 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:59:48.040 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:59:48.040 INFO:tasks.workunit.client.0.smithi136.stdout:19 2024-09-15T20:59:48.040 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 19 2024-09-15T20:59:48.041 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:59:48.041 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-15T20:59:48.328 INFO:tasks.workunit.client.0.smithi136.stdout:osd.4 up in weight 1 up_from 25 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6834/2214056343,v1:127.0.0.1:6835/2214056343] [v2:127.0.0.1:6836/2214056343,v1:127.0.0.1:6837/2214056343] exists,up efc7afae-88a4-411c-bd12-386c1aa780f9 2024-09-15T20:59:48.332 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-15T20:59:48.332 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-15T20:59:48.332 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-15T20:59:48.332 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:831: corrupt_and_repair_lrc: for id in $(seq 0 9) 2024-09-15T20:59:48.332 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:832: corrupt_and_repair_lrc: run_osd td/osd-scrub-repair 5 2024-09-15T20:59:48.332 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-scrub-repair 2024-09-15T20:59:48.332 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-15T20:59:48.333 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=5 2024-09-15T20:59:48.333 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-15T20:59:48.333 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-scrub-repair/5 2024-09-15T20:59:48.333 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=ce4892c7-6c6d-4b4f-8e6f-07a3e11133e5 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-15T20:59:48.333 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T20:59:48.333 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T20:59:48.333 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T20:59:48.333 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/5' 2024-09-15T20:59:48.333 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/5/journal' 2024-09-15T20:59:48.333 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-15T20:59:48.333 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-15T20:59:48.333 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-15T20:59:48.333 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-15T20:59:48.334 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-15T20:59:48.334 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-15T20:59:48.334 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-15T20:59:48.334 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T20:59:48.334 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52602 2024-09-15T20:59:48.334 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.52602/$cluster-$name.asok' 2024-09-15T20:59:48.334 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.52602/$cluster-$name.asok' 2024-09-15T20:59:48.334 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-15T20:59:48.334 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-15T20:59:48.334 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-15T20:59:48.334 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-15T20:59:48.334 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-15T20:59:48.334 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T20:59:48.335 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T20:59:48.335 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T20:59:48.335 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T20:59:48.335 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-15T20:59:48.335 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-15T20:59:48.335 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-scrub-repair/5 2024-09-15T20:59:48.335 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-15T20:59:48.335 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=c3187161-e486-4b18-8257-d69c125e9792 2024-09-15T20:59:48.335 INFO:tasks.workunit.client.0.smithi136.stdout:add osd5 c3187161-e486-4b18-8257-d69c125e9792 2024-09-15T20:59:48.336 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd5 c3187161-e486-4b18-8257-d69c125e9792' 2024-09-15T20:59:48.336 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-15T20:59:48.351 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQDESudmty7AFBAAsq04TrIsKGpxXEYSYDXfWQ== 2024-09-15T20:59:48.351 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQDESudmty7AFBAAsq04TrIsKGpxXEYSYDXfWQ=="}' 2024-09-15T20:59:48.351 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new c3187161-e486-4b18-8257-d69c125e9792 -i td/osd-scrub-repair/5/new.json 2024-09-15T20:59:48.713 INFO:tasks.workunit.client.0.smithi136.stdout:5 2024-09-15T20:59:48.725 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-scrub-repair/5/new.json 2024-09-15T20:59:48.726 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 5 --fsid=ce4892c7-6c6d-4b4f-8e6f-07a3e11133e5 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/5 --osd-journal=td/osd-scrub-repair/5/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.52602/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDESudmty7AFBAAsq04TrIsKGpxXEYSYDXfWQ== --osd-uuid c3187161-e486-4b18-8257-d69c125e9792 2024-09-15T20:59:48.755 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:59:48.752+0000 7f7670e42640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T20:59:48.763 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:59:48.761+0000 7f7670e42640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T20:59:48.764 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:59:48.762+0000 7f7670e42640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T20:59:48.765 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:59:48.762+0000 7f7670e42640 -1 bdev(0x562e25201c00 td/osd-scrub-repair/5/block) open stat got: (1) Operation not permitted 2024-09-15T20:59:48.765 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:59:48.762+0000 7f7670e42640 -1 bluestore(td/osd-scrub-repair/5) _read_fsid unparsable uuid 2024-09-15T20:59:54.985 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-scrub-repair/5/keyring 2024-09-15T20:59:54.986 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-15T20:59:54.987 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd5 key to auth repository 2024-09-15T20:59:54.987 INFO:tasks.workunit.client.0.smithi136.stdout:adding osd5 key to auth repository 2024-09-15T20:59:54.987 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-scrub-repair/5/keyring auth add osd.5 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-15T20:59:55.345 INFO:tasks.workunit.client.0.smithi136.stdout:start osd.5 2024-09-15T20:59:55.346 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.5 2024-09-15T20:59:55.346 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 5 --fsid=ce4892c7-6c6d-4b4f-8e6f-07a3e11133e5 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/5 --osd-journal=td/osd-scrub-repair/5/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.52602/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T20:59:55.346 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-15T20:59:55.346 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-15T20:59:55.346 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-15T20:59:55.373 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:59:55.370+0000 7f03d3f29640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T20:59:55.375 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:59:55.372+0000 7f03d3f29640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T20:59:55.376 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:59:55.373+0000 7f03d3f29640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T20:59:55.623 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 5 2024-09-15T20:59:55.623 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-15T20:59:55.623 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=5 2024-09-15T20:59:55.623 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-15T20:59:55.623 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-15T20:59:55.624 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:59:55.624 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-15T20:59:55.624 INFO:tasks.workunit.client.0.smithi136.stdout:0 2024-09-15T20:59:55.624 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:59:55.624 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-15T20:59:55.900 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:59:56.740 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:59:56.737+0000 7f03d3f29640 -1 Falling back to public interface 2024-09-15T20:59:56.901 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:59:56.901 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:59:56.901 INFO:tasks.workunit.client.0.smithi136.stdout:1 2024-09-15T20:59:56.902 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-15T20:59:56.902 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:59:56.902 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-15T20:59:57.185 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:59:58.186 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:59:58.187 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:59:58.187 INFO:tasks.workunit.client.0.smithi136.stdout:2 2024-09-15T20:59:58.187 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-15T20:59:58.187 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:59:58.187 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-15T20:59:58.475 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:59:59.464 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T20:59:59.462+0000 7f03d3f29640 -1 osd.5 0 log_to_monitors true 2024-09-15T20:59:59.476 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:59:59.476 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:59:59.476 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-15T20:59:59.477 INFO:tasks.workunit.client.0.smithi136.stdout:3 2024-09-15T20:59:59.477 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:59:59.477 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-15T20:59:59.780 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:00:00.781 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:00:00.781 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:00:00.781 INFO:tasks.workunit.client.0.smithi136.stdout:4 2024-09-15T21:00:00.782 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-15T21:00:00.782 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:00:00.782 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-15T21:00:01.119 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:00:02.120 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:00:02.120 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:00:02.120 INFO:tasks.workunit.client.0.smithi136.stdout:5 2024-09-15T21:00:02.121 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-15T21:00:02.121 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:00:02.121 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-15T21:00:02.544 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:00:03.545 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:00:03.545 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:00:03.545 INFO:tasks.workunit.client.0.smithi136.stdout:6 2024-09-15T21:00:03.546 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-15T21:00:03.546 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:00:03.546 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-15T21:00:03.831 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:00:04.833 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:00:04.833 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:00:04.833 INFO:tasks.workunit.client.0.smithi136.stdout:7 2024-09-15T21:00:04.834 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-15T21:00:04.834 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:00:04.834 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-15T21:00:05.119 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:00:06.120 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:00:06.120 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:00:06.120 INFO:tasks.workunit.client.0.smithi136.stdout:8 2024-09-15T21:00:06.121 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-15T21:00:06.121 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:00:06.121 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-15T21:00:06.491 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:00:07.492 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:00:07.493 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:00:07.493 INFO:tasks.workunit.client.0.smithi136.stdout:9 2024-09-15T21:00:07.493 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-15T21:00:07.493 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:00:07.494 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-15T21:00:07.887 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:00:08.888 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:00:08.888 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:00:08.888 INFO:tasks.workunit.client.0.smithi136.stdout:10 2024-09-15T21:00:08.889 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-15T21:00:08.889 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:00:08.889 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-15T21:00:09.240 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:00:10.241 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:00:10.241 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:00:10.242 INFO:tasks.workunit.client.0.smithi136.stdout:11 2024-09-15T21:00:10.242 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-15T21:00:10.242 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:00:10.242 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-15T21:00:10.528 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:00:11.530 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:00:11.530 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:00:11.530 INFO:tasks.workunit.client.0.smithi136.stdout:12 2024-09-15T21:00:11.531 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-15T21:00:11.531 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:00:11.531 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-15T21:00:11.814 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:00:12.815 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:00:12.815 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:00:12.815 INFO:tasks.workunit.client.0.smithi136.stdout:13 2024-09-15T21:00:12.816 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-15T21:00:12.816 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:00:12.816 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-15T21:00:13.814 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:00:14.816 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:00:14.816 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:00:14.816 INFO:tasks.workunit.client.0.smithi136.stdout:14 2024-09-15T21:00:14.817 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-15T21:00:14.817 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:00:14.817 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-15T21:00:15.199 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:00:16.201 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:00:16.201 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:00:16.201 INFO:tasks.workunit.client.0.smithi136.stdout:15 2024-09-15T21:00:16.202 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-15T21:00:16.202 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:00:16.202 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-15T21:00:16.787 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:00:17.788 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:00:17.789 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:00:17.789 INFO:tasks.workunit.client.0.smithi136.stdout:16 2024-09-15T21:00:17.789 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-15T21:00:17.789 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:00:17.790 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-15T21:00:18.077 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:00:19.078 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:00:19.078 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:00:19.078 INFO:tasks.workunit.client.0.smithi136.stdout:17 2024-09-15T21:00:19.079 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-15T21:00:19.079 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:00:19.079 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-15T21:00:19.831 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:00:20.832 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:00:20.833 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:00:20.833 INFO:tasks.workunit.client.0.smithi136.stdout:18 2024-09-15T21:00:20.833 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 18 2024-09-15T21:00:20.833 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:00:20.833 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-15T21:00:21.295 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:00:22.296 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:00:22.296 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:00:22.296 INFO:tasks.workunit.client.0.smithi136.stdout:19 2024-09-15T21:00:22.297 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 19 2024-09-15T21:00:22.297 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:00:22.297 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-15T21:00:22.581 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:00:23.582 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:00:23.583 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:00:23.583 INFO:tasks.workunit.client.0.smithi136.stdout:20 2024-09-15T21:00:23.583 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 20 2024-09-15T21:00:23.583 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:00:23.583 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-15T21:00:23.861 INFO:tasks.workunit.client.0.smithi136.stdout:osd.5 up in weight 1 up_from 30 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6842/3848876406,v1:127.0.0.1:6843/3848876406] [v2:127.0.0.1:6844/3848876406,v1:127.0.0.1:6845/3848876406] exists,up c3187161-e486-4b18-8257-d69c125e9792 2024-09-15T21:00:23.864 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-15T21:00:23.865 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-15T21:00:23.865 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-15T21:00:23.865 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:831: corrupt_and_repair_lrc: for id in $(seq 0 9) 2024-09-15T21:00:23.865 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:832: corrupt_and_repair_lrc: run_osd td/osd-scrub-repair 6 2024-09-15T21:00:23.865 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-scrub-repair 2024-09-15T21:00:23.865 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-15T21:00:23.865 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=6 2024-09-15T21:00:23.865 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-15T21:00:23.865 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-scrub-repair/6 2024-09-15T21:00:23.865 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=ce4892c7-6c6d-4b4f-8e6f-07a3e11133e5 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-15T21:00:23.865 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T21:00:23.865 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T21:00:23.865 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T21:00:23.866 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/6' 2024-09-15T21:00:23.866 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/6/journal' 2024-09-15T21:00:23.866 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-15T21:00:23.866 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-15T21:00:23.866 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-15T21:00:23.866 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-15T21:00:23.866 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-15T21:00:23.866 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-15T21:00:23.866 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-15T21:00:23.866 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T21:00:23.866 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52602 2024-09-15T21:00:23.866 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.52602/$cluster-$name.asok' 2024-09-15T21:00:23.866 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.52602/$cluster-$name.asok' 2024-09-15T21:00:23.867 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-15T21:00:23.867 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-15T21:00:23.867 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-15T21:00:23.867 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-15T21:00:23.867 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-15T21:00:23.867 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T21:00:23.867 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T21:00:23.867 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T21:00:23.867 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T21:00:23.867 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-15T21:00:23.867 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-15T21:00:23.868 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-scrub-repair/6 2024-09-15T21:00:23.868 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-15T21:00:23.868 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=d6abb0e9-e70c-4ab8-8cd3-955816563eb4 2024-09-15T21:00:23.868 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd6 d6abb0e9-e70c-4ab8-8cd3-955816563eb4' 2024-09-15T21:00:23.868 INFO:tasks.workunit.client.0.smithi136.stdout:add osd6 d6abb0e9-e70c-4ab8-8cd3-955816563eb4 2024-09-15T21:00:23.869 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-15T21:00:23.884 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQDnSudmk8SCNBAABsFSxF6BMQoXpX6jK3kSXg== 2024-09-15T21:00:23.884 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQDnSudmk8SCNBAABsFSxF6BMQoXpX6jK3kSXg=="}' 2024-09-15T21:00:23.884 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new d6abb0e9-e70c-4ab8-8cd3-955816563eb4 -i td/osd-scrub-repair/6/new.json 2024-09-15T21:00:24.239 INFO:tasks.workunit.client.0.smithi136.stdout:6 2024-09-15T21:00:24.251 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-scrub-repair/6/new.json 2024-09-15T21:00:24.252 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 6 --fsid=ce4892c7-6c6d-4b4f-8e6f-07a3e11133e5 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/6 --osd-journal=td/osd-scrub-repair/6/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.52602/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDnSudmk8SCNBAABsFSxF6BMQoXpX6jK3kSXg== --osd-uuid d6abb0e9-e70c-4ab8-8cd3-955816563eb4 2024-09-15T21:00:24.281 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:00:24.278+0000 7fd241c0f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:00:24.299 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:00:24.296+0000 7fd241c0f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:00:24.300 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:00:24.297+0000 7fd241c0f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:00:24.300 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:00:24.298+0000 7fd241c0f640 -1 bdev(0x56430c3b5c00 td/osd-scrub-repair/6/block) open stat got: (1) Operation not permitted 2024-09-15T21:00:24.301 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:00:24.298+0000 7fd241c0f640 -1 bluestore(td/osd-scrub-repair/6) _read_fsid unparsable uuid 2024-09-15T21:00:31.031 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-scrub-repair/6/keyring 2024-09-15T21:00:31.031 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-15T21:00:31.032 INFO:tasks.workunit.client.0.smithi136.stdout:adding osd6 key to auth repository 2024-09-15T21:00:31.032 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd6 key to auth repository 2024-09-15T21:00:31.032 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-scrub-repair/6/keyring auth add osd.6 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-15T21:00:31.393 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.6 2024-09-15T21:00:31.393 INFO:tasks.workunit.client.0.smithi136.stdout:start osd.6 2024-09-15T21:00:31.394 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 6 --fsid=ce4892c7-6c6d-4b4f-8e6f-07a3e11133e5 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/6 --osd-journal=td/osd-scrub-repair/6/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.52602/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T21:00:31.394 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-15T21:00:31.394 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-15T21:00:31.394 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-15T21:00:31.420 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:00:31.418+0000 7ffa71f29640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:00:31.422 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:00:31.420+0000 7ffa71f29640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:00:31.423 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:00:31.421+0000 7ffa71f29640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:00:31.671 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 6 2024-09-15T21:00:31.672 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-15T21:00:31.672 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=6 2024-09-15T21:00:31.672 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-15T21:00:31.672 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-15T21:00:31.672 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:00:31.672 INFO:tasks.workunit.client.0.smithi136.stdout:0 2024-09-15T21:00:31.673 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-15T21:00:31.673 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:00:31.673 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.6 up' 2024-09-15T21:00:31.948 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:00:32.540 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:00:32.537+0000 7ffa71f29640 -1 Falling back to public interface 2024-09-15T21:00:32.950 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:00:32.950 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:00:32.950 INFO:tasks.workunit.client.0.smithi136.stdout:1 2024-09-15T21:00:32.951 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-15T21:00:32.951 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:00:32.951 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.6 up' 2024-09-15T21:00:33.237 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:00:34.239 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:00:34.239 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:00:34.239 INFO:tasks.workunit.client.0.smithi136.stdout:2 2024-09-15T21:00:34.240 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-15T21:00:34.240 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:00:34.240 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.6 up' 2024-09-15T21:00:34.531 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:00:35.361 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:00:35.358+0000 7ffa71f29640 -1 osd.6 0 log_to_monitors true 2024-09-15T21:00:35.533 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:00:35.533 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:00:35.533 INFO:tasks.workunit.client.0.smithi136.stdout:3 2024-09-15T21:00:35.534 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-15T21:00:35.534 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:00:35.534 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.6 up' 2024-09-15T21:00:35.920 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:00:36.922 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:00:36.922 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:00:36.922 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-15T21:00:36.922 INFO:tasks.workunit.client.0.smithi136.stdout:4 2024-09-15T21:00:36.923 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:00:36.923 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.6 up' 2024-09-15T21:00:37.238 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:00:38.239 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:00:38.240 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:00:38.240 INFO:tasks.workunit.client.0.smithi136.stdout:5 2024-09-15T21:00:38.241 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-15T21:00:38.241 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:00:38.241 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.6 up' 2024-09-15T21:00:38.529 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:00:39.530 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:00:39.530 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:00:39.530 INFO:tasks.workunit.client.0.smithi136.stdout:6 2024-09-15T21:00:39.531 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-15T21:00:39.531 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:00:39.531 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.6 up' 2024-09-15T21:00:39.884 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:00:40.885 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:00:40.885 INFO:tasks.workunit.client.0.smithi136.stdout:7 2024-09-15T21:00:40.886 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:00:40.886 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-15T21:00:40.886 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:00:40.886 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.6 up' 2024-09-15T21:00:41.219 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:00:42.221 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:00:42.221 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:00:42.221 INFO:tasks.workunit.client.0.smithi136.stdout:8 2024-09-15T21:00:42.222 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-15T21:00:42.222 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:00:42.222 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.6 up' 2024-09-15T21:00:42.507 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:00:43.508 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:00:43.508 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:00:43.509 INFO:tasks.workunit.client.0.smithi136.stdout:9 2024-09-15T21:00:43.509 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-15T21:00:43.509 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:00:43.509 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.6 up' 2024-09-15T21:00:43.855 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:00:44.857 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:00:44.857 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:00:44.857 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-15T21:00:44.857 INFO:tasks.workunit.client.0.smithi136.stdout:10 2024-09-15T21:00:44.857 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:00:44.858 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.6 up' 2024-09-15T21:00:45.247 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:00:46.249 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:00:46.249 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:00:46.249 INFO:tasks.workunit.client.0.smithi136.stdout:11 2024-09-15T21:00:46.249 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-15T21:00:46.250 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:00:46.250 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.6 up' 2024-09-15T21:00:46.535 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:00:47.536 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:00:47.536 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:00:47.537 INFO:tasks.workunit.client.0.smithi136.stdout:12 2024-09-15T21:00:47.537 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-15T21:00:47.537 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:00:47.537 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.6 up' 2024-09-15T21:00:48.137 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:00:49.139 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:00:49.139 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:00:49.139 INFO:tasks.workunit.client.0.smithi136.stdout:13 2024-09-15T21:00:49.140 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-15T21:00:49.140 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:00:49.140 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.6 up' 2024-09-15T21:00:49.460 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:00:50.461 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:00:50.461 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:00:50.461 INFO:tasks.workunit.client.0.smithi136.stdout:14 2024-09-15T21:00:50.462 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-15T21:00:50.462 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:00:50.462 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.6 up' 2024-09-15T21:00:50.943 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:00:51.944 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:00:51.944 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:00:51.944 INFO:tasks.workunit.client.0.smithi136.stdout:15 2024-09-15T21:00:51.945 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-15T21:00:51.945 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:00:51.945 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.6 up' 2024-09-15T21:00:52.506 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:00:53.508 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:00:53.508 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:00:53.508 INFO:tasks.workunit.client.0.smithi136.stdout:16 2024-09-15T21:00:53.509 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-15T21:00:53.509 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:00:53.509 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.6 up' 2024-09-15T21:00:54.292 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:00:55.294 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:00:55.294 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:00:55.294 INFO:tasks.workunit.client.0.smithi136.stdout:17 2024-09-15T21:00:55.295 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-15T21:00:55.295 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:00:55.295 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.6 up' 2024-09-15T21:00:55.734 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:00:56.736 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:00:56.736 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:00:56.736 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 18 2024-09-15T21:00:56.737 INFO:tasks.workunit.client.0.smithi136.stdout:18 2024-09-15T21:00:56.737 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:00:56.737 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.6 up' 2024-09-15T21:00:57.366 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:00:58.368 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:00:58.368 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:00:58.368 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 19 2024-09-15T21:00:58.368 INFO:tasks.workunit.client.0.smithi136.stdout:19 2024-09-15T21:00:58.368 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:00:58.369 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.6 up' 2024-09-15T21:00:58.701 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:00:59.703 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:00:59.703 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:00:59.703 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 20 2024-09-15T21:00:59.704 INFO:tasks.workunit.client.0.smithi136.stdout:20 2024-09-15T21:00:59.704 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:00:59.704 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.6 up' 2024-09-15T21:01:00.017 INFO:tasks.workunit.client.0.smithi136.stdout:osd.6 up in weight 1 up_from 35 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6850/4294875861,v1:127.0.0.1:6851/4294875861] [v2:127.0.0.1:6852/4294875861,v1:127.0.0.1:6853/4294875861] exists,up d6abb0e9-e70c-4ab8-8cd3-955816563eb4 2024-09-15T21:01:00.021 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-15T21:01:00.021 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-15T21:01:00.021 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-15T21:01:00.021 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:831: corrupt_and_repair_lrc: for id in $(seq 0 9) 2024-09-15T21:01:00.021 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:832: corrupt_and_repair_lrc: run_osd td/osd-scrub-repair 7 2024-09-15T21:01:00.021 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-scrub-repair 2024-09-15T21:01:00.022 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-15T21:01:00.022 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=7 2024-09-15T21:01:00.022 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-15T21:01:00.022 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-scrub-repair/7 2024-09-15T21:01:00.022 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=ce4892c7-6c6d-4b4f-8e6f-07a3e11133e5 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-15T21:01:00.022 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T21:01:00.022 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T21:01:00.022 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T21:01:00.022 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/7' 2024-09-15T21:01:00.022 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/7/journal' 2024-09-15T21:01:00.022 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-15T21:01:00.022 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-15T21:01:00.022 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-15T21:01:00.023 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-15T21:01:00.023 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-15T21:01:00.023 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-15T21:01:00.023 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-15T21:01:00.023 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T21:01:00.023 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52602 2024-09-15T21:01:00.023 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.52602/$cluster-$name.asok' 2024-09-15T21:01:00.023 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.52602/$cluster-$name.asok' 2024-09-15T21:01:00.023 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-15T21:01:00.023 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-15T21:01:00.023 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-15T21:01:00.023 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-15T21:01:00.023 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-15T21:01:00.023 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T21:01:00.024 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T21:01:00.024 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T21:01:00.024 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T21:01:00.024 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-15T21:01:00.024 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-15T21:01:00.024 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-scrub-repair/7 2024-09-15T21:01:00.024 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-15T21:01:00.024 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=584d2413-79a2-48ed-870b-96ec53e13708 2024-09-15T21:01:00.025 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd7 584d2413-79a2-48ed-870b-96ec53e13708' 2024-09-15T21:01:00.025 INFO:tasks.workunit.client.0.smithi136.stdout:add osd7 584d2413-79a2-48ed-870b-96ec53e13708 2024-09-15T21:01:00.025 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-15T21:01:00.041 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQAMS+dmY4dOAhAATDYB5WrofGAVEgVDVvuI8g== 2024-09-15T21:01:00.041 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQAMS+dmY4dOAhAATDYB5WrofGAVEgVDVvuI8g=="}' 2024-09-15T21:01:00.041 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 584d2413-79a2-48ed-870b-96ec53e13708 -i td/osd-scrub-repair/7/new.json 2024-09-15T21:01:00.383 INFO:tasks.workunit.client.0.smithi136.stdout:7 2024-09-15T21:01:00.395 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-scrub-repair/7/new.json 2024-09-15T21:01:00.395 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 7 --fsid=ce4892c7-6c6d-4b4f-8e6f-07a3e11133e5 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/7 --osd-journal=td/osd-scrub-repair/7/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.52602/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAMS+dmY4dOAhAATDYB5WrofGAVEgVDVvuI8g== --osd-uuid 584d2413-79a2-48ed-870b-96ec53e13708 2024-09-15T21:01:00.425 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:01:00.421+0000 7f02f480b640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:01:00.492 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:01:00.488+0000 7f02f480b640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:01:00.492 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:01:00.489+0000 7f02f480b640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:01:00.493 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:01:00.489+0000 7f02f480b640 -1 bdev(0x5608a75a1c00 td/osd-scrub-repair/7/block) open stat got: (1) Operation not permitted 2024-09-15T21:01:00.493 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:01:00.489+0000 7f02f480b640 -1 bluestore(td/osd-scrub-repair/7) _read_fsid unparsable uuid 2024-09-15T21:01:06.677 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-scrub-repair/7/keyring 2024-09-15T21:01:06.678 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-15T21:01:06.678 INFO:tasks.workunit.client.0.smithi136.stdout:adding osd7 key to auth repository 2024-09-15T21:01:06.679 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd7 key to auth repository 2024-09-15T21:01:06.679 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-scrub-repair/7/keyring auth add osd.7 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-15T21:01:07.041 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.7 2024-09-15T21:01:07.042 INFO:tasks.workunit.client.0.smithi136.stdout:start osd.7 2024-09-15T21:01:07.042 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 7 --fsid=ce4892c7-6c6d-4b4f-8e6f-07a3e11133e5 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/7 --osd-journal=td/osd-scrub-repair/7/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.52602/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T21:01:07.042 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-15T21:01:07.042 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-15T21:01:07.043 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-15T21:01:07.069 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:01:07.066+0000 7f6502db2640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:01:07.103 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:01:07.100+0000 7f6502db2640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:01:07.104 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:01:07.101+0000 7f6502db2640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:01:07.319 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 7 2024-09-15T21:01:07.320 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-15T21:01:07.320 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=7 2024-09-15T21:01:07.320 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-15T21:01:07.320 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-15T21:01:07.320 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:01:07.320 INFO:tasks.workunit.client.0.smithi136.stdout:0 2024-09-15T21:01:07.321 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-15T21:01:07.321 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:01:07.321 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.7 up' 2024-09-15T21:01:07.596 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:01:08.598 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:01:08.598 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:01:08.598 INFO:tasks.workunit.client.0.smithi136.stdout:1 2024-09-15T21:01:08.599 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-15T21:01:08.599 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:01:08.599 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.7 up' 2024-09-15T21:01:08.701 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:01:08.699+0000 7f6502db2640 -1 Falling back to public interface 2024-09-15T21:01:08.885 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:01:09.887 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:01:09.887 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:01:09.887 INFO:tasks.workunit.client.0.smithi136.stdout:2 2024-09-15T21:01:09.888 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-15T21:01:09.888 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:01:09.888 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.7 up' 2024-09-15T21:01:10.172 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:01:11.174 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:01:11.174 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:01:11.174 INFO:tasks.workunit.client.0.smithi136.stdout:3 2024-09-15T21:01:11.175 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-15T21:01:11.175 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:01:11.175 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.7 up' 2024-09-15T21:01:11.192 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:01:11.189+0000 7f6502db2640 -1 osd.7 0 log_to_monitors true 2024-09-15T21:01:11.509 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:01:12.511 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:01:12.511 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:01:12.511 INFO:tasks.workunit.client.0.smithi136.stdout:4 2024-09-15T21:01:12.512 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-15T21:01:12.512 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:01:12.512 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.7 up' 2024-09-15T21:01:12.804 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:01:13.806 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:01:13.806 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:01:13.807 INFO:tasks.workunit.client.0.smithi136.stdout:5 2024-09-15T21:01:13.807 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-15T21:01:13.807 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:01:13.808 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.7 up' 2024-09-15T21:01:14.097 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:01:15.098 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:01:15.098 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:01:15.098 INFO:tasks.workunit.client.0.smithi136.stdout:6 2024-09-15T21:01:15.099 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-15T21:01:15.099 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:01:15.099 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.7 up' 2024-09-15T21:01:15.496 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:01:16.497 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:01:16.498 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:01:16.498 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-15T21:01:16.498 INFO:tasks.workunit.client.0.smithi136.stdout:7 2024-09-15T21:01:16.498 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:01:16.498 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.7 up' 2024-09-15T21:01:16.854 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:01:17.855 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:01:17.855 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:01:17.856 INFO:tasks.workunit.client.0.smithi136.stdout:8 2024-09-15T21:01:17.856 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-15T21:01:17.856 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:01:17.856 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.7 up' 2024-09-15T21:01:18.242 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:01:19.244 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:01:19.244 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:01:19.245 INFO:tasks.workunit.client.0.smithi136.stdout:9 2024-09-15T21:01:19.245 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-15T21:01:19.246 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:01:19.246 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.7 up' 2024-09-15T21:01:19.599 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:01:20.601 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:01:20.601 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:01:20.601 INFO:tasks.workunit.client.0.smithi136.stdout:10 2024-09-15T21:01:20.602 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-15T21:01:20.602 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:01:20.602 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.7 up' 2024-09-15T21:01:20.891 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:01:21.892 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:01:21.892 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:01:21.892 INFO:tasks.workunit.client.0.smithi136.stdout:11 2024-09-15T21:01:21.893 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-15T21:01:21.893 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:01:21.893 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.7 up' 2024-09-15T21:01:22.211 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:01:23.212 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:01:23.213 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:01:23.213 INFO:tasks.workunit.client.0.smithi136.stdout:12 2024-09-15T21:01:23.213 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-15T21:01:23.213 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:01:23.214 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.7 up' 2024-09-15T21:01:23.741 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:01:24.742 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:01:24.743 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:01:24.743 INFO:tasks.workunit.client.0.smithi136.stdout:13 2024-09-15T21:01:24.743 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-15T21:01:24.743 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:01:24.744 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.7 up' 2024-09-15T21:01:25.686 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:01:26.688 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:01:26.688 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:01:26.689 INFO:tasks.workunit.client.0.smithi136.stdout:14 2024-09-15T21:01:26.689 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-15T21:01:26.689 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:01:26.690 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.7 up' 2024-09-15T21:01:27.373 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:01:28.374 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:01:28.374 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:01:28.375 INFO:tasks.workunit.client.0.smithi136.stdout:15 2024-09-15T21:01:28.375 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-15T21:01:28.375 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:01:28.375 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.7 up' 2024-09-15T21:01:28.666 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:01:29.667 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:01:29.668 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:01:29.668 INFO:tasks.workunit.client.0.smithi136.stdout:16 2024-09-15T21:01:29.668 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-15T21:01:29.668 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:01:29.668 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.7 up' 2024-09-15T21:01:30.378 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:01:31.380 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:01:31.380 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:01:31.380 INFO:tasks.workunit.client.0.smithi136.stdout:17 2024-09-15T21:01:31.381 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-15T21:01:31.381 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:01:31.381 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.7 up' 2024-09-15T21:01:32.223 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:01:33.225 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:01:33.225 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:01:33.225 INFO:tasks.workunit.client.0.smithi136.stdout:18 2024-09-15T21:01:33.226 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 18 2024-09-15T21:01:33.226 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:01:33.226 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.7 up' 2024-09-15T21:01:33.749 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:01:34.754 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:01:34.754 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:01:34.754 INFO:tasks.workunit.client.0.smithi136.stdout:19 2024-09-15T21:01:34.754 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 19 2024-09-15T21:01:34.755 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:01:34.755 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.7 up' 2024-09-15T21:01:35.047 INFO:tasks.workunit.client.0.smithi136.stdout:osd.7 up in weight 1 up_from 40 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6858/1154094091,v1:127.0.0.1:6859/1154094091] [v2:127.0.0.1:6860/1154094091,v1:127.0.0.1:6861/1154094091] exists,up 584d2413-79a2-48ed-870b-96ec53e13708 2024-09-15T21:01:35.051 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-15T21:01:35.051 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-15T21:01:35.052 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-15T21:01:35.052 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:831: corrupt_and_repair_lrc: for id in $(seq 0 9) 2024-09-15T21:01:35.052 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:832: corrupt_and_repair_lrc: run_osd td/osd-scrub-repair 8 2024-09-15T21:01:35.052 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-scrub-repair 2024-09-15T21:01:35.052 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-15T21:01:35.052 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=8 2024-09-15T21:01:35.052 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-15T21:01:35.052 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-scrub-repair/8 2024-09-15T21:01:35.052 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=ce4892c7-6c6d-4b4f-8e6f-07a3e11133e5 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-15T21:01:35.052 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T21:01:35.052 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T21:01:35.052 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T21:01:35.052 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/8' 2024-09-15T21:01:35.053 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/8/journal' 2024-09-15T21:01:35.053 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-15T21:01:35.053 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-15T21:01:35.053 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-15T21:01:35.053 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-15T21:01:35.053 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-15T21:01:35.053 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-15T21:01:35.053 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-15T21:01:35.053 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T21:01:35.053 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52602 2024-09-15T21:01:35.053 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.52602/$cluster-$name.asok' 2024-09-15T21:01:35.053 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.52602/$cluster-$name.asok' 2024-09-15T21:01:35.053 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-15T21:01:35.053 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-15T21:01:35.054 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-15T21:01:35.054 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-15T21:01:35.054 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-15T21:01:35.054 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T21:01:35.054 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T21:01:35.054 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T21:01:35.054 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T21:01:35.054 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-15T21:01:35.054 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-15T21:01:35.054 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-scrub-repair/8 2024-09-15T21:01:35.054 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-15T21:01:35.054 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=36462dee-d99e-4a32-8ab4-77a5fc67eaa6 2024-09-15T21:01:35.054 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd8 36462dee-d99e-4a32-8ab4-77a5fc67eaa6' 2024-09-15T21:01:35.055 INFO:tasks.workunit.client.0.smithi136.stdout:add osd8 36462dee-d99e-4a32-8ab4-77a5fc67eaa6 2024-09-15T21:01:35.055 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-15T21:01:35.071 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQAvS+dmk5YKBBAAv98co8h+baGzFmkgnYwQLw== 2024-09-15T21:01:35.071 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQAvS+dmk5YKBBAAv98co8h+baGzFmkgnYwQLw=="}' 2024-09-15T21:01:35.071 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 36462dee-d99e-4a32-8ab4-77a5fc67eaa6 -i td/osd-scrub-repair/8/new.json 2024-09-15T21:01:35.423 INFO:tasks.workunit.client.0.smithi136.stdout:8 2024-09-15T21:01:35.434 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-scrub-repair/8/new.json 2024-09-15T21:01:35.435 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 8 --fsid=ce4892c7-6c6d-4b4f-8e6f-07a3e11133e5 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/8 --osd-journal=td/osd-scrub-repair/8/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.52602/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAvS+dmk5YKBBAAv98co8h+baGzFmkgnYwQLw== --osd-uuid 36462dee-d99e-4a32-8ab4-77a5fc67eaa6 2024-09-15T21:01:35.465 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:01:35.461+0000 7f6903615640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:01:35.474 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:01:35.471+0000 7f6903615640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:01:35.475 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:01:35.472+0000 7f6903615640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:01:35.475 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:01:35.473+0000 7f6903615640 -1 bdev(0x557f17c3dc00 td/osd-scrub-repair/8/block) open stat got: (1) Operation not permitted 2024-09-15T21:01:35.476 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:01:35.473+0000 7f6903615640 -1 bluestore(td/osd-scrub-repair/8) _read_fsid unparsable uuid 2024-09-15T21:01:41.528 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-scrub-repair/8/keyring 2024-09-15T21:01:41.528 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-15T21:01:41.529 INFO:tasks.workunit.client.0.smithi136.stdout:adding osd8 key to auth repository 2024-09-15T21:01:41.530 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd8 key to auth repository 2024-09-15T21:01:41.530 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-scrub-repair/8/keyring auth add osd.8 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-15T21:01:41.892 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.8 2024-09-15T21:01:41.893 INFO:tasks.workunit.client.0.smithi136.stdout:start osd.8 2024-09-15T21:01:41.893 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 8 --fsid=ce4892c7-6c6d-4b4f-8e6f-07a3e11133e5 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/8 --osd-journal=td/osd-scrub-repair/8/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.52602/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T21:01:41.893 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-15T21:01:41.894 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-15T21:01:41.894 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-15T21:01:41.920 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:01:41.917+0000 7ffaaa8e6640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:01:41.922 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:01:41.919+0000 7ffaaa8e6640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:01:41.923 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:01:41.921+0000 7ffaaa8e6640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:01:42.170 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 8 2024-09-15T21:01:42.171 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-15T21:01:42.171 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=8 2024-09-15T21:01:42.171 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-15T21:01:42.171 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-15T21:01:42.171 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:01:42.171 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-15T21:01:42.171 INFO:tasks.workunit.client.0.smithi136.stdout:0 2024-09-15T21:01:42.172 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:01:42.172 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.8 up' 2024-09-15T21:01:42.451 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:01:43.023 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:01:43.021+0000 7ffaaa8e6640 -1 Falling back to public interface 2024-09-15T21:01:43.453 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:01:43.453 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:01:43.453 INFO:tasks.workunit.client.0.smithi136.stdout:1 2024-09-15T21:01:43.454 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-15T21:01:43.454 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:01:43.455 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.8 up' 2024-09-15T21:01:43.743 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:01:44.744 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:01:44.744 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:01:44.744 INFO:tasks.workunit.client.0.smithi136.stdout:2 2024-09-15T21:01:44.745 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-15T21:01:44.745 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:01:44.745 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.8 up' 2024-09-15T21:01:45.060 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:01:45.924 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:01:45.921+0000 7ffaaa8e6640 -1 osd.8 0 log_to_monitors true 2024-09-15T21:01:46.062 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:01:46.062 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:01:46.062 INFO:tasks.workunit.client.0.smithi136.stdout:3 2024-09-15T21:01:46.062 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-15T21:01:46.063 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:01:46.063 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.8 up' 2024-09-15T21:01:46.408 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:01:47.409 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:01:47.409 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:01:47.409 INFO:tasks.workunit.client.0.smithi136.stdout:4 2024-09-15T21:01:47.410 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-15T21:01:47.410 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:01:47.410 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.8 up' 2024-09-15T21:01:47.721 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:01:48.722 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:01:48.722 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:01:48.723 INFO:tasks.workunit.client.0.smithi136.stdout:5 2024-09-15T21:01:48.723 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-15T21:01:48.723 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:01:48.723 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.8 up' 2024-09-15T21:01:49.055 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:01:50.057 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:01:50.057 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:01:50.057 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-15T21:01:50.057 INFO:tasks.workunit.client.0.smithi136.stdout:6 2024-09-15T21:01:50.057 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:01:50.057 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.8 up' 2024-09-15T21:01:50.346 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:01:51.348 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:01:51.348 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:01:51.348 INFO:tasks.workunit.client.0.smithi136.stdout:7 2024-09-15T21:01:51.349 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-15T21:01:51.349 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:01:51.349 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.8 up' 2024-09-15T21:01:51.636 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:01:52.638 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:01:52.638 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:01:52.638 INFO:tasks.workunit.client.0.smithi136.stdout:8 2024-09-15T21:01:52.639 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-15T21:01:52.639 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:01:52.639 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.8 up' 2024-09-15T21:01:52.929 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:01:53.930 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:01:53.930 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:01:53.931 INFO:tasks.workunit.client.0.smithi136.stdout:9 2024-09-15T21:01:53.931 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-15T21:01:53.931 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:01:53.931 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.8 up' 2024-09-15T21:01:54.218 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:01:55.219 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:01:55.220 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:01:55.220 INFO:tasks.workunit.client.0.smithi136.stdout:10 2024-09-15T21:01:55.220 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-15T21:01:55.220 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:01:55.220 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.8 up' 2024-09-15T21:01:55.532 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:01:56.533 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:01:56.533 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:01:56.533 INFO:tasks.workunit.client.0.smithi136.stdout:11 2024-09-15T21:01:56.534 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-15T21:01:56.534 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:01:56.534 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.8 up' 2024-09-15T21:01:56.901 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:01:57.903 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:01:57.903 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:01:57.903 INFO:tasks.workunit.client.0.smithi136.stdout:12 2024-09-15T21:01:57.904 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-15T21:01:57.904 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:01:57.904 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.8 up' 2024-09-15T21:01:58.189 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:01:59.190 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:01:59.190 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:01:59.191 INFO:tasks.workunit.client.0.smithi136.stdout:13 2024-09-15T21:01:59.191 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-15T21:01:59.191 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:01:59.191 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.8 up' 2024-09-15T21:01:59.749 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:02:00.750 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:02:00.751 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:02:00.751 INFO:tasks.workunit.client.0.smithi136.stdout:14 2024-09-15T21:02:00.751 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-15T21:02:00.752 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:02:00.752 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.8 up' 2024-09-15T21:02:01.157 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:02:02.158 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:02:02.158 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:02:02.159 INFO:tasks.workunit.client.0.smithi136.stdout:15 2024-09-15T21:02:02.159 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-15T21:02:02.159 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:02:02.159 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.8 up' 2024-09-15T21:02:02.729 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:02:03.730 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:02:03.730 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:02:03.730 INFO:tasks.workunit.client.0.smithi136.stdout:16 2024-09-15T21:02:03.731 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-15T21:02:03.731 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:02:03.731 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.8 up' 2024-09-15T21:02:04.364 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:02:05.365 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:02:05.366 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:02:05.366 INFO:tasks.workunit.client.0.smithi136.stdout:17 2024-09-15T21:02:05.366 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-15T21:02:05.366 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:02:05.366 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.8 up' 2024-09-15T21:02:05.668 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:02:06.669 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:02:06.670 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:02:06.670 INFO:tasks.workunit.client.0.smithi136.stdout:18 2024-09-15T21:02:06.670 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 18 2024-09-15T21:02:06.670 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:02:06.671 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.8 up' 2024-09-15T21:02:07.062 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:02:08.064 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:02:08.064 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:02:08.064 INFO:tasks.workunit.client.0.smithi136.stdout:19 2024-09-15T21:02:08.065 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 19 2024-09-15T21:02:08.065 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:02:08.065 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.8 up' 2024-09-15T21:02:08.759 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:02:09.760 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:02:09.761 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:02:09.761 INFO:tasks.workunit.client.0.smithi136.stdout:20 2024-09-15T21:02:09.762 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 20 2024-09-15T21:02:09.762 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:02:09.762 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.8 up' 2024-09-15T21:02:10.050 INFO:tasks.workunit.client.0.smithi136.stdout:osd.8 up in weight 1 up_from 45 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6866/4216066537,v1:127.0.0.1:6867/4216066537] [v2:127.0.0.1:6868/4216066537,v1:127.0.0.1:6869/4216066537] exists,up 36462dee-d99e-4a32-8ab4-77a5fc67eaa6 2024-09-15T21:02:10.053 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-15T21:02:10.054 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-15T21:02:10.054 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-15T21:02:10.054 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:831: corrupt_and_repair_lrc: for id in $(seq 0 9) 2024-09-15T21:02:10.054 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:832: corrupt_and_repair_lrc: run_osd td/osd-scrub-repair 9 2024-09-15T21:02:10.054 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-scrub-repair 2024-09-15T21:02:10.054 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-15T21:02:10.054 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=9 2024-09-15T21:02:10.054 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-15T21:02:10.054 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-scrub-repair/9 2024-09-15T21:02:10.054 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=ce4892c7-6c6d-4b4f-8e6f-07a3e11133e5 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-15T21:02:10.054 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T21:02:10.054 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T21:02:10.055 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T21:02:10.055 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/9' 2024-09-15T21:02:10.055 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/9/journal' 2024-09-15T21:02:10.055 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-15T21:02:10.055 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-15T21:02:10.055 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-15T21:02:10.055 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-15T21:02:10.055 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-15T21:02:10.055 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-15T21:02:10.055 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-15T21:02:10.055 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T21:02:10.055 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52602 2024-09-15T21:02:10.056 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.52602/$cluster-$name.asok' 2024-09-15T21:02:10.056 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.52602/$cluster-$name.asok' 2024-09-15T21:02:10.056 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-15T21:02:10.056 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-15T21:02:10.056 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-15T21:02:10.056 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-15T21:02:10.056 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-15T21:02:10.056 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T21:02:10.056 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T21:02:10.056 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T21:02:10.056 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T21:02:10.057 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-15T21:02:10.057 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-15T21:02:10.057 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-scrub-repair/9 2024-09-15T21:02:10.057 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-15T21:02:10.057 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=ccfb463f-a1cc-4f3c-a7db-ae126efbeaa5 2024-09-15T21:02:10.057 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd9 ccfb463f-a1cc-4f3c-a7db-ae126efbeaa5' 2024-09-15T21:02:10.057 INFO:tasks.workunit.client.0.smithi136.stdout:add osd9 ccfb463f-a1cc-4f3c-a7db-ae126efbeaa5 2024-09-15T21:02:10.058 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-15T21:02:10.072 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQBSS+dm1FEnBBAAFtfj2ngOnEExMy9GeLNUBA== 2024-09-15T21:02:10.072 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQBSS+dm1FEnBBAAFtfj2ngOnEExMy9GeLNUBA=="}' 2024-09-15T21:02:10.072 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new ccfb463f-a1cc-4f3c-a7db-ae126efbeaa5 -i td/osd-scrub-repair/9/new.json 2024-09-15T21:02:10.418 INFO:tasks.workunit.client.0.smithi136.stdout:9 2024-09-15T21:02:10.430 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-scrub-repair/9/new.json 2024-09-15T21:02:10.431 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 9 --fsid=ce4892c7-6c6d-4b4f-8e6f-07a3e11133e5 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/9 --osd-journal=td/osd-scrub-repair/9/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.52602/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBSS+dm1FEnBBAAFtfj2ngOnEExMy9GeLNUBA== --osd-uuid ccfb463f-a1cc-4f3c-a7db-ae126efbeaa5 2024-09-15T21:02:10.460 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:02:10.457+0000 7f8a67b4b640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:02:10.503 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:02:10.501+0000 7f8a67b4b640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:02:10.504 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:02:10.502+0000 7f8a67b4b640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:02:10.505 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:02:10.502+0000 7f8a67b4b640 -1 bdev(0x55ee5d829c00 td/osd-scrub-repair/9/block) open stat got: (1) Operation not permitted 2024-09-15T21:02:10.505 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:02:10.502+0000 7f8a67b4b640 -1 bluestore(td/osd-scrub-repair/9) _read_fsid unparsable uuid 2024-09-15T21:02:16.681 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-scrub-repair/9/keyring 2024-09-15T21:02:16.682 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-15T21:02:16.683 INFO:tasks.workunit.client.0.smithi136.stdout:adding osd9 key to auth repository 2024-09-15T21:02:16.683 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd9 key to auth repository 2024-09-15T21:02:16.683 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-scrub-repair/9/keyring auth add osd.9 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-15T21:02:17.049 INFO:tasks.workunit.client.0.smithi136.stdout:start osd.9 2024-09-15T21:02:17.050 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.9 2024-09-15T21:02:17.050 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 9 --fsid=ce4892c7-6c6d-4b4f-8e6f-07a3e11133e5 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/9 --osd-journal=td/osd-scrub-repair/9/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.52602/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T21:02:17.050 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-15T21:02:17.050 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-15T21:02:17.050 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-15T21:02:17.076 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:02:17.073+0000 7faf6620f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:02:17.078 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:02:17.075+0000 7faf6620f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:02:17.079 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:02:17.076+0000 7faf6620f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:02:17.328 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 9 2024-09-15T21:02:17.328 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-15T21:02:17.328 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=9 2024-09-15T21:02:17.328 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-15T21:02:17.328 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-15T21:02:17.328 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:02:17.328 INFO:tasks.workunit.client.0.smithi136.stdout:0 2024-09-15T21:02:17.329 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-15T21:02:17.329 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:02:17.329 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.9 up' 2024-09-15T21:02:17.605 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:02:18.196 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:02:18.193+0000 7faf6620f640 -1 Falling back to public interface 2024-09-15T21:02:18.606 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:02:18.607 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:02:18.607 INFO:tasks.workunit.client.0.smithi136.stdout:1 2024-09-15T21:02:18.607 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-15T21:02:18.608 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:02:18.608 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.9 up' 2024-09-15T21:02:18.897 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:02:19.898 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:02:19.898 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:02:19.898 INFO:tasks.workunit.client.0.smithi136.stdout:2 2024-09-15T21:02:19.899 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-15T21:02:19.899 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:02:19.899 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.9 up' 2024-09-15T21:02:20.189 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:02:20.842 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:02:20.839+0000 7faf6620f640 -1 osd.9 0 log_to_monitors true 2024-09-15T21:02:21.190 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:02:21.190 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:02:21.191 INFO:tasks.workunit.client.0.smithi136.stdout:3 2024-09-15T21:02:21.191 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-15T21:02:21.191 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:02:21.191 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.9 up' 2024-09-15T21:02:21.555 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:02:22.557 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:02:22.557 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:02:22.557 INFO:tasks.workunit.client.0.smithi136.stdout:4 2024-09-15T21:02:22.558 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-15T21:02:22.558 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:02:22.558 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.9 up' 2024-09-15T21:02:22.889 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:02:23.890 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:02:23.890 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:02:23.891 INFO:tasks.workunit.client.0.smithi136.stdout:5 2024-09-15T21:02:23.891 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-15T21:02:23.891 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:02:23.891 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.9 up' 2024-09-15T21:02:24.181 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:02:25.182 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:02:25.182 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:02:25.183 INFO:tasks.workunit.client.0.smithi136.stdout:6 2024-09-15T21:02:25.183 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-15T21:02:25.183 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:02:25.183 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.9 up' 2024-09-15T21:02:25.543 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:02:26.544 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:02:26.544 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:02:26.544 INFO:tasks.workunit.client.0.smithi136.stdout:7 2024-09-15T21:02:26.545 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-15T21:02:26.545 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:02:26.545 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.9 up' 2024-09-15T21:02:26.890 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:02:27.891 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:02:27.891 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:02:27.892 INFO:tasks.workunit.client.0.smithi136.stdout:8 2024-09-15T21:02:27.892 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-15T21:02:27.892 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:02:27.892 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.9 up' 2024-09-15T21:02:28.183 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:02:29.184 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:02:29.184 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:02:29.184 INFO:tasks.workunit.client.0.smithi136.stdout:9 2024-09-15T21:02:29.185 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-15T21:02:29.185 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:02:29.185 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.9 up' 2024-09-15T21:02:29.896 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:02:30.898 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:02:30.898 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:02:30.898 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-15T21:02:30.898 INFO:tasks.workunit.client.0.smithi136.stdout:10 2024-09-15T21:02:30.899 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:02:30.899 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.9 up' 2024-09-15T21:02:31.219 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:02:32.221 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:02:32.221 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:02:32.221 INFO:tasks.workunit.client.0.smithi136.stdout:11 2024-09-15T21:02:32.221 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-15T21:02:32.222 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:02:32.222 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.9 up' 2024-09-15T21:02:32.509 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:02:33.510 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:02:33.510 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:02:33.510 INFO:tasks.workunit.client.0.smithi136.stdout:12 2024-09-15T21:02:33.511 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-15T21:02:33.511 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:02:33.511 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.9 up' 2024-09-15T21:02:33.794 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:02:34.796 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:02:34.796 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:02:34.796 INFO:tasks.workunit.client.0.smithi136.stdout:13 2024-09-15T21:02:34.797 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-15T21:02:34.797 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:02:34.797 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.9 up' 2024-09-15T21:02:35.195 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:02:36.196 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:02:36.196 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:02:36.197 INFO:tasks.workunit.client.0.smithi136.stdout:14 2024-09-15T21:02:36.197 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-15T21:02:36.197 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:02:36.197 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.9 up' 2024-09-15T21:02:36.577 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:02:37.579 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:02:37.579 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:02:37.579 INFO:tasks.workunit.client.0.smithi136.stdout:15 2024-09-15T21:02:37.579 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-15T21:02:37.580 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:02:37.580 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.9 up' 2024-09-15T21:02:37.924 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:02:38.926 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:02:38.926 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:02:38.926 INFO:tasks.workunit.client.0.smithi136.stdout:16 2024-09-15T21:02:38.927 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-15T21:02:38.927 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:02:38.927 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.9 up' 2024-09-15T21:02:39.614 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:02:40.615 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:02:40.615 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:02:40.616 INFO:tasks.workunit.client.0.smithi136.stdout:17 2024-09-15T21:02:40.616 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-15T21:02:40.616 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:02:40.616 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.9 up' 2024-09-15T21:02:41.493 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:02:42.495 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:02:42.495 INFO:tasks.workunit.client.0.smithi136.stdout:18 2024-09-15T21:02:42.496 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:02:42.496 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 18 2024-09-15T21:02:42.496 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:02:42.496 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.9 up' 2024-09-15T21:02:43.131 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:02:44.133 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:02:44.133 INFO:tasks.workunit.client.0.smithi136.stdout:19 2024-09-15T21:02:44.134 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:02:44.134 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 19 2024-09-15T21:02:44.134 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:02:44.134 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.9 up' 2024-09-15T21:02:44.522 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:02:45.523 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:02:45.523 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:02:45.523 INFO:tasks.workunit.client.0.smithi136.stdout:20 2024-09-15T21:02:45.524 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 20 2024-09-15T21:02:45.524 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:02:45.524 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.9 up' 2024-09-15T21:02:45.843 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:02:46.845 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:02:46.845 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:02:46.845 INFO:tasks.workunit.client.0.smithi136.stdout:21 2024-09-15T21:02:46.846 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 21 2024-09-15T21:02:46.846 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:02:46.846 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.9 up' 2024-09-15T21:02:47.130 INFO:tasks.workunit.client.0.smithi136.stdout:osd.9 up in weight 1 up_from 50 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6874/369181211,v1:127.0.0.1:6875/369181211] [v2:127.0.0.1:6876/369181211,v1:127.0.0.1:6877/369181211] exists,up ccfb463f-a1cc-4f3c-a7db-ae126efbeaa5 2024-09-15T21:02:47.131 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-15T21:02:47.131 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-15T21:02:47.131 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-15T21:02:47.131 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:834: corrupt_and_repair_lrc: create_rbd_pool 2024-09-15T21:02:47.131 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:526: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2024-09-15T21:02:47.402 INFO:tasks.workunit.client.0.smithi136.stderr:pool 'rbd' does not exist 2024-09-15T21:02:47.414 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:527: create_rbd_pool: create_pool rbd 4 2024-09-15T21:02:47.414 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create rbd 4 2024-09-15T21:02:47.892 INFO:tasks.workunit.client.0.smithi136.stderr:pool 'rbd' created 2024-09-15T21:02:47.904 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2024-09-15T21:02:48.905 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:528: create_rbd_pool: rbd pool init rbd 2024-09-15T21:02:52.604 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:835: corrupt_and_repair_lrc: wait_for_clean 2024-09-15T21:02:52.604 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-15T21:02:52.604 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-15T21:02:52.604 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-15T21:02:52.605 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-15T21:02:52.605 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-15T21:02:52.605 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-15T21:02:52.605 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-15T21:02:52.605 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-15T21:02:52.606 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-15T21:02:52.706 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-15T21:02:52.706 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-15T21:02:52.706 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-15T21:02:52.706 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-15T21:02:52.706 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-15T21:02:52.706 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-15T21:02:53.096 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-15T21:02:53.096 INFO:tasks.workunit.client.0.smithi136.stderr:1 2024-09-15T21:02:53.096 INFO:tasks.workunit.client.0.smithi136.stderr:2 2024-09-15T21:02:53.096 INFO:tasks.workunit.client.0.smithi136.stderr:3 2024-09-15T21:02:53.097 INFO:tasks.workunit.client.0.smithi136.stderr:4 2024-09-15T21:02:53.097 INFO:tasks.workunit.client.0.smithi136.stderr:5 2024-09-15T21:02:53.097 INFO:tasks.workunit.client.0.smithi136.stderr:6 2024-09-15T21:02:53.097 INFO:tasks.workunit.client.0.smithi136.stderr:7 2024-09-15T21:02:53.097 INFO:tasks.workunit.client.0.smithi136.stderr:8 2024-09-15T21:02:53.097 INFO:tasks.workunit.client.0.smithi136.stderr:9' 2024-09-15T21:02:53.097 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-15T21:02:53.097 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T21:02:53.097 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-15T21:02:53.207 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836553 2024-09-15T21:02:53.207 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836553 2024-09-15T21:02:53.207 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836553' 2024-09-15T21:02:53.207 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T21:02:53.207 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-15T21:02:53.341 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=42949673023 2024-09-15T21:02:53.341 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 42949673023 2024-09-15T21:02:53.342 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836553 1-42949673023' 2024-09-15T21:02:53.342 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T21:02:53.342 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-15T21:02:53.459 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=64424509498 2024-09-15T21:02:53.459 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 64424509498 2024-09-15T21:02:53.459 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836553 1-42949673023 2-64424509498' 2024-09-15T21:02:53.459 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T21:02:53.459 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-15T21:02:53.561 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=85899345971 2024-09-15T21:02:53.561 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 85899345971 2024-09-15T21:02:53.561 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836553 1-42949673023 2-64424509498 3-85899345971' 2024-09-15T21:02:53.561 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T21:02:53.561 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-09-15T21:02:53.664 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=107374182443 2024-09-15T21:02:53.664 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 107374182443 2024-09-15T21:02:53.665 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836553 1-42949673023 2-64424509498 3-85899345971 4-107374182443' 2024-09-15T21:02:53.665 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T21:02:53.665 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-09-15T21:02:53.767 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=128849018921 2024-09-15T21:02:53.767 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 128849018921 2024-09-15T21:02:53.767 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836553 1-42949673023 2-64424509498 3-85899345971 4-107374182443 5-128849018921' 2024-09-15T21:02:53.767 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T21:02:53.767 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-09-15T21:02:53.871 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=150323855391 2024-09-15T21:02:53.871 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 150323855391 2024-09-15T21:02:53.871 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836553 1-42949673023 2-64424509498 3-85899345971 4-107374182443 5-128849018921 6-150323855391' 2024-09-15T21:02:53.871 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T21:02:53.871 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2024-09-15T21:02:53.973 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=171798691863 2024-09-15T21:02:53.973 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 171798691863 2024-09-15T21:02:53.973 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836553 1-42949673023 2-64424509498 3-85899345971 4-107374182443 5-128849018921 6-150323855391 7-171798691863' 2024-09-15T21:02:53.973 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T21:02:53.974 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2024-09-15T21:02:54.077 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=193273528339 2024-09-15T21:02:54.078 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 193273528339 2024-09-15T21:02:54.078 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836553 1-42949673023 2-64424509498 3-85899345971 4-107374182443 5-128849018921 6-150323855391 7-171798691863 8-193273528339' 2024-09-15T21:02:54.078 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T21:02:54.078 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2024-09-15T21:02:54.179 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=214748364814 2024-09-15T21:02:54.179 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 214748364814 2024-09-15T21:02:54.180 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836553 1-42949673023 2-64424509498 3-85899345971 4-107374182443 5-128849018921 6-150323855391 7-171798691863 8-193273528339 9-214748364814' 2024-09-15T21:02:54.180 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T21:02:54.181 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836553 2024-09-15T21:02:54.182 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T21:02:54.182 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-15T21:02:54.182 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836553 2024-09-15T21:02:54.182 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T21:02:54.182 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836553 2024-09-15T21:02:54.182 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836553' 2024-09-15T21:02:54.183 INFO:tasks.workunit.client.0.smithi136.stdout:waiting osd.0 seq 21474836553 2024-09-15T21:02:54.183 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T21:02:54.460 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836552 -lt 21474836553 2024-09-15T21:02:54.460 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-15T21:02:55.461 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-15T21:02:55.462 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T21:02:55.748 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836553 -lt 21474836553 2024-09-15T21:02:55.748 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T21:02:55.749 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-42949673023 2024-09-15T21:02:55.749 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T21:02:55.750 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-15T21:02:55.750 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-42949673023 2024-09-15T21:02:55.750 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T21:02:55.751 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=42949673023 2024-09-15T21:02:55.752 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 42949673023' 2024-09-15T21:02:55.752 INFO:tasks.workunit.client.0.smithi136.stdout:waiting osd.1 seq 42949673023 2024-09-15T21:02:55.752 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-15T21:02:56.033 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 42949673023 -lt 42949673023 2024-09-15T21:02:56.033 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T21:02:56.034 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-64424509498 2024-09-15T21:02:56.034 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T21:02:56.034 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-15T21:02:56.035 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-64424509498 2024-09-15T21:02:56.035 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T21:02:56.036 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=64424509498 2024-09-15T21:02:56.036 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 64424509498' 2024-09-15T21:02:56.036 INFO:tasks.workunit.client.0.smithi136.stdout:waiting osd.2 seq 64424509498 2024-09-15T21:02:56.037 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-15T21:02:56.314 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 64424509498 -lt 64424509498 2024-09-15T21:02:56.314 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T21:02:56.315 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 3-85899345971 2024-09-15T21:02:56.315 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T21:02:56.316 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=3 2024-09-15T21:02:56.316 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 3-85899345971 2024-09-15T21:02:56.317 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T21:02:56.317 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=85899345971 2024-09-15T21:02:56.318 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.3 seq 85899345971' 2024-09-15T21:02:56.318 INFO:tasks.workunit.client.0.smithi136.stdout:waiting osd.3 seq 85899345971 2024-09-15T21:02:56.318 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-15T21:02:56.593 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 85899345971 -lt 85899345971 2024-09-15T21:02:56.594 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T21:02:56.594 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 4-107374182443 2024-09-15T21:02:56.594 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T21:02:56.595 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=4 2024-09-15T21:02:56.596 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 4-107374182443 2024-09-15T21:02:56.596 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T21:02:56.596 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=107374182443 2024-09-15T21:02:56.597 INFO:tasks.workunit.client.0.smithi136.stdout:waiting osd.4 seq 107374182443 2024-09-15T21:02:56.597 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.4 seq 107374182443' 2024-09-15T21:02:56.597 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 4 2024-09-15T21:02:56.884 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 107374182443 -lt 107374182443 2024-09-15T21:02:56.885 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T21:02:56.885 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 5-128849018921 2024-09-15T21:02:56.885 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T21:02:56.886 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=5 2024-09-15T21:02:56.886 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 5-128849018921 2024-09-15T21:02:56.887 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T21:02:56.887 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=128849018921 2024-09-15T21:02:56.888 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.5 seq 128849018921' 2024-09-15T21:02:56.888 INFO:tasks.workunit.client.0.smithi136.stdout:waiting osd.5 seq 128849018921 2024-09-15T21:02:56.888 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 5 2024-09-15T21:02:57.205 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 128849018922 -lt 128849018921 2024-09-15T21:02:57.205 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T21:02:57.206 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 6-150323855391 2024-09-15T21:02:57.206 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T21:02:57.206 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=6 2024-09-15T21:02:57.207 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 6-150323855391 2024-09-15T21:02:57.207 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T21:02:57.208 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=150323855391 2024-09-15T21:02:57.208 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.6 seq 150323855391' 2024-09-15T21:02:57.208 INFO:tasks.workunit.client.0.smithi136.stdout:waiting osd.6 seq 150323855391 2024-09-15T21:02:57.209 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 6 2024-09-15T21:02:57.485 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 150323855392 -lt 150323855391 2024-09-15T21:02:57.485 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T21:02:57.486 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 7-171798691863 2024-09-15T21:02:57.486 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T21:02:57.487 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=7 2024-09-15T21:02:57.487 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 7-171798691863 2024-09-15T21:02:57.487 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T21:02:57.488 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=171798691863 2024-09-15T21:02:57.488 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.7 seq 171798691863' 2024-09-15T21:02:57.489 INFO:tasks.workunit.client.0.smithi136.stdout:waiting osd.7 seq 171798691863 2024-09-15T21:02:57.489 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 7 2024-09-15T21:02:57.779 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 171798691864 -lt 171798691863 2024-09-15T21:02:57.779 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T21:02:57.780 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 8-193273528339 2024-09-15T21:02:57.780 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T21:02:57.780 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=8 2024-09-15T21:02:57.781 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 8-193273528339 2024-09-15T21:02:57.781 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T21:02:57.782 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=193273528339 2024-09-15T21:02:57.782 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.8 seq 193273528339' 2024-09-15T21:02:57.782 INFO:tasks.workunit.client.0.smithi136.stdout:waiting osd.8 seq 193273528339 2024-09-15T21:02:57.783 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 8 2024-09-15T21:02:58.057 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 193273528340 -lt 193273528339 2024-09-15T21:02:58.057 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T21:02:58.058 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 9-214748364814 2024-09-15T21:02:58.058 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T21:02:58.059 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=9 2024-09-15T21:02:58.059 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 9-214748364814 2024-09-15T21:02:58.059 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T21:02:58.060 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=214748364814 2024-09-15T21:02:58.061 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.9 seq 214748364814' 2024-09-15T21:02:58.061 INFO:tasks.workunit.client.0.smithi136.stdout:waiting osd.9 seq 214748364814 2024-09-15T21:02:58.061 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 9 2024-09-15T21:02:58.339 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 214748364815 -lt 214748364814 2024-09-15T21:02:58.340 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-15T21:02:58.340 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T21:02:58.340 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T21:02:58.689 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 4 == 0 2024-09-15T21:02:58.690 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-15T21:02:58.690 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-15T21:02:58.690 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-15T21:02:58.690 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T21:02:58.690 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T21:02:58.690 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T21:02:58.690 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T21:02:58.969 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=4 2024-09-15T21:02:58.969 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-15T21:02:58.969 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T21:02:58.970 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T21:02:59.311 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 = 4 2024-09-15T21:02:59.312 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-15T21:02:59.312 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-15T21:02:59.312 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:837: corrupt_and_repair_lrc: create_ec_pool ecpool false k=4 m=2 l=3 plugin=lrc 2024-09-15T21:02:59.312 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2443: create_ec_pool: local pool_name=ecpool 2024-09-15T21:02:59.312 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2444: create_ec_pool: shift 2024-09-15T21:02:59.312 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2445: create_ec_pool: local allow_overwrites=false 2024-09-15T21:02:59.312 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2446: create_ec_pool: shift 2024-09-15T21:02:59.312 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2448: create_ec_pool: ceph osd erasure-code-profile set myprofile crush-failure-domain=osd k=4 m=2 l=3 plugin=lrc 2024-09-15T21:02:59.710 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2450: create_ec_pool: create_pool ecpool 1 1 erasure myprofile 2024-09-15T21:02:59.711 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create ecpool 1 1 erasure myprofile 2024-09-15T21:03:00.595 INFO:tasks.workunit.client.0.smithi136.stderr:pool 'ecpool' created 2024-09-15T21:03:00.608 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2024-09-15T21:03:01.610 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2452: create_ec_pool: '[' false = true ']' 2024-09-15T21:03:01.610 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2456: create_ec_pool: wait_for_clean 2024-09-15T21:03:01.610 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-15T21:03:01.610 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-15T21:03:01.610 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-15T21:03:01.611 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-15T21:03:01.611 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-15T21:03:01.611 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-15T21:03:01.611 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-15T21:03:01.611 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-15T21:03:01.612 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-15T21:03:01.712 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-15T21:03:01.713 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-15T21:03:01.713 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-15T21:03:01.713 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-15T21:03:01.713 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-15T21:03:01.713 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-15T21:03:01.993 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-15T21:03:01.993 INFO:tasks.workunit.client.0.smithi136.stderr:1 2024-09-15T21:03:01.993 INFO:tasks.workunit.client.0.smithi136.stderr:2 2024-09-15T21:03:01.993 INFO:tasks.workunit.client.0.smithi136.stderr:3 2024-09-15T21:03:01.993 INFO:tasks.workunit.client.0.smithi136.stderr:4 2024-09-15T21:03:01.993 INFO:tasks.workunit.client.0.smithi136.stderr:5 2024-09-15T21:03:01.993 INFO:tasks.workunit.client.0.smithi136.stderr:6 2024-09-15T21:03:01.993 INFO:tasks.workunit.client.0.smithi136.stderr:7 2024-09-15T21:03:01.993 INFO:tasks.workunit.client.0.smithi136.stderr:8 2024-09-15T21:03:01.993 INFO:tasks.workunit.client.0.smithi136.stderr:9' 2024-09-15T21:03:01.994 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-15T21:03:01.994 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T21:03:01.994 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-15T21:03:02.098 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836556 2024-09-15T21:03:02.099 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836556 2024-09-15T21:03:02.099 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836556' 2024-09-15T21:03:02.099 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T21:03:02.099 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-15T21:03:02.235 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=42949673026 2024-09-15T21:03:02.235 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 42949673026 2024-09-15T21:03:02.235 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836556 1-42949673026' 2024-09-15T21:03:02.236 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T21:03:02.236 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-15T21:03:02.339 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=64424509501 2024-09-15T21:03:02.339 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 64424509501 2024-09-15T21:03:02.339 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836556 1-42949673026 2-64424509501' 2024-09-15T21:03:02.339 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T21:03:02.339 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-15T21:03:02.443 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=85899345973 2024-09-15T21:03:02.443 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 85899345973 2024-09-15T21:03:02.444 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836556 1-42949673026 2-64424509501 3-85899345973' 2024-09-15T21:03:02.444 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T21:03:02.444 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-09-15T21:03:02.546 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=107374182445 2024-09-15T21:03:02.546 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 107374182445 2024-09-15T21:03:02.546 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836556 1-42949673026 2-64424509501 3-85899345973 4-107374182445' 2024-09-15T21:03:02.546 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T21:03:02.546 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-09-15T21:03:02.777 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=128849018924 2024-09-15T21:03:02.777 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 128849018924 2024-09-15T21:03:02.778 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836556 1-42949673026 2-64424509501 3-85899345973 4-107374182445 5-128849018924' 2024-09-15T21:03:02.778 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T21:03:02.778 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-09-15T21:03:02.881 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=150323855394 2024-09-15T21:03:02.882 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 150323855394 2024-09-15T21:03:02.882 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836556 1-42949673026 2-64424509501 3-85899345973 4-107374182445 5-128849018924 6-150323855394' 2024-09-15T21:03:02.882 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T21:03:02.882 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2024-09-15T21:03:03.054 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=171798691866 2024-09-15T21:03:03.054 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 171798691866 2024-09-15T21:03:03.055 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836556 1-42949673026 2-64424509501 3-85899345973 4-107374182445 5-128849018924 6-150323855394 7-171798691866' 2024-09-15T21:03:03.055 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T21:03:03.055 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2024-09-15T21:03:03.182 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=193273528342 2024-09-15T21:03:03.182 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 193273528342 2024-09-15T21:03:03.182 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836556 1-42949673026 2-64424509501 3-85899345973 4-107374182445 5-128849018924 6-150323855394 7-171798691866 8-193273528342' 2024-09-15T21:03:03.182 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T21:03:03.182 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2024-09-15T21:03:03.333 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=214748364817 2024-09-15T21:03:03.333 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 214748364817 2024-09-15T21:03:03.333 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836556 1-42949673026 2-64424509501 3-85899345973 4-107374182445 5-128849018924 6-150323855394 7-171798691866 8-193273528342 9-214748364817' 2024-09-15T21:03:03.333 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T21:03:03.333 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836556 2024-09-15T21:03:03.333 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T21:03:03.334 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-15T21:03:03.334 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836556 2024-09-15T21:03:03.334 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T21:03:03.335 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836556 2024-09-15T21:03:03.335 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836556' 2024-09-15T21:03:03.336 INFO:tasks.workunit.client.0.smithi136.stdout:waiting osd.0 seq 21474836556 2024-09-15T21:03:03.336 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T21:03:03.610 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836556 -lt 21474836556 2024-09-15T21:03:03.610 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T21:03:03.610 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-42949673026 2024-09-15T21:03:03.611 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T21:03:03.611 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-15T21:03:03.612 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-42949673026 2024-09-15T21:03:03.612 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T21:03:03.613 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=42949673026 2024-09-15T21:03:03.613 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 42949673026' 2024-09-15T21:03:03.613 INFO:tasks.workunit.client.0.smithi136.stdout:waiting osd.1 seq 42949673026 2024-09-15T21:03:03.613 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-15T21:03:03.891 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 42949673026 -lt 42949673026 2024-09-15T21:03:03.891 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T21:03:03.892 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-64424509501 2024-09-15T21:03:03.892 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T21:03:03.893 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-15T21:03:03.893 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-64424509501 2024-09-15T21:03:03.893 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T21:03:03.894 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=64424509501 2024-09-15T21:03:03.895 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 64424509501' 2024-09-15T21:03:03.895 INFO:tasks.workunit.client.0.smithi136.stdout:waiting osd.2 seq 64424509501 2024-09-15T21:03:03.895 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-15T21:03:04.191 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 64424509501 -lt 64424509501 2024-09-15T21:03:04.191 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T21:03:04.191 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 3-85899345973 2024-09-15T21:03:04.192 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T21:03:04.192 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=3 2024-09-15T21:03:04.193 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 3-85899345973 2024-09-15T21:03:04.193 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T21:03:04.194 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=85899345973 2024-09-15T21:03:04.194 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.3 seq 85899345973' 2024-09-15T21:03:04.194 INFO:tasks.workunit.client.0.smithi136.stdout:waiting osd.3 seq 85899345973 2024-09-15T21:03:04.194 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-15T21:03:04.473 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 85899345973 -lt 85899345973 2024-09-15T21:03:04.473 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T21:03:04.473 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 4-107374182445 2024-09-15T21:03:04.473 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T21:03:04.474 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=4 2024-09-15T21:03:04.475 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 4-107374182445 2024-09-15T21:03:04.475 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T21:03:04.476 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=107374182445 2024-09-15T21:03:04.476 INFO:tasks.workunit.client.0.smithi136.stdout:waiting osd.4 seq 107374182445 2024-09-15T21:03:04.476 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.4 seq 107374182445' 2024-09-15T21:03:04.477 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 4 2024-09-15T21:03:04.754 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 107374182445 -lt 107374182445 2024-09-15T21:03:04.754 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T21:03:04.755 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 5-128849018924 2024-09-15T21:03:04.755 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T21:03:04.755 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=5 2024-09-15T21:03:04.756 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 5-128849018924 2024-09-15T21:03:04.756 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T21:03:04.757 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=128849018924 2024-09-15T21:03:04.757 INFO:tasks.workunit.client.0.smithi136.stdout:waiting osd.5 seq 128849018924 2024-09-15T21:03:04.757 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.5 seq 128849018924' 2024-09-15T21:03:04.758 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 5 2024-09-15T21:03:05.073 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 128849018925 -lt 128849018924 2024-09-15T21:03:05.073 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T21:03:05.073 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 6-150323855394 2024-09-15T21:03:05.073 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T21:03:05.074 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=6 2024-09-15T21:03:05.075 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 6-150323855394 2024-09-15T21:03:05.075 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T21:03:05.075 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=150323855394 2024-09-15T21:03:05.076 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.6 seq 150323855394' 2024-09-15T21:03:05.076 INFO:tasks.workunit.client.0.smithi136.stdout:waiting osd.6 seq 150323855394 2024-09-15T21:03:05.076 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 6 2024-09-15T21:03:05.387 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 150323855394 -lt 150323855394 2024-09-15T21:03:05.387 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T21:03:05.388 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 7-171798691866 2024-09-15T21:03:05.388 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T21:03:05.388 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=7 2024-09-15T21:03:05.389 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 7-171798691866 2024-09-15T21:03:05.389 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T21:03:05.390 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=171798691866 2024-09-15T21:03:05.390 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.7 seq 171798691866' 2024-09-15T21:03:05.390 INFO:tasks.workunit.client.0.smithi136.stdout:waiting osd.7 seq 171798691866 2024-09-15T21:03:05.391 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 7 2024-09-15T21:03:05.672 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 171798691867 -lt 171798691866 2024-09-15T21:03:05.673 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T21:03:05.673 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 8-193273528342 2024-09-15T21:03:05.673 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T21:03:05.674 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=8 2024-09-15T21:03:05.676 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 8-193273528342 2024-09-15T21:03:05.676 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T21:03:05.676 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=193273528342 2024-09-15T21:03:05.676 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.8 seq 193273528342' 2024-09-15T21:03:05.677 INFO:tasks.workunit.client.0.smithi136.stdout:waiting osd.8 seq 193273528342 2024-09-15T21:03:05.677 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 8 2024-09-15T21:03:05.955 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 193273528342 -lt 193273528342 2024-09-15T21:03:05.955 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T21:03:05.955 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 9-214748364817 2024-09-15T21:03:05.956 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T21:03:05.957 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=9 2024-09-15T21:03:05.957 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 9-214748364817 2024-09-15T21:03:05.957 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T21:03:05.958 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=214748364817 2024-09-15T21:03:05.958 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.9 seq 214748364817' 2024-09-15T21:03:05.958 INFO:tasks.workunit.client.0.smithi136.stdout:waiting osd.9 seq 214748364817 2024-09-15T21:03:05.959 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 9 2024-09-15T21:03:06.237 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 214748364817 -lt 214748364817 2024-09-15T21:03:06.237 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-15T21:03:06.237 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T21:03:06.237 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T21:03:06.583 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 5 == 0 2024-09-15T21:03:06.583 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-15T21:03:06.583 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-15T21:03:06.583 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-15T21:03:06.583 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T21:03:06.584 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T21:03:06.584 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T21:03:06.584 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T21:03:06.859 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=5 2024-09-15T21:03:06.860 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-15T21:03:06.860 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T21:03:06.860 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T21:03:07.211 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 = 5 2024-09-15T21:03:07.211 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-15T21:03:07.211 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-15T21:03:07.211 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2457: create_ec_pool: return 0 2024-09-15T21:03:07.211 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:838: corrupt_and_repair_lrc: corrupt_and_repair_erasure_coded td/osd-scrub-repair ecpool 2024-09-15T21:03:07.211 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:248: corrupt_and_repair_erasure_coded: local dir=td/osd-scrub-repair 2024-09-15T21:03:07.211 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:249: corrupt_and_repair_erasure_coded: local poolname=ecpool 2024-09-15T21:03:07.211 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:251: corrupt_and_repair_erasure_coded: add_something td/osd-scrub-repair ecpool 2024-09-15T21:03:07.211 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-09-15T21:03:07.211 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=ecpool 2024-09-15T21:03:07.212 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=SOMETHING 2024-09-15T21:03:07.212 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-09-15T21:03:07.212 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-09-15T21:03:07.212 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-09-15T21:03:07.585 INFO:tasks.workunit.client.0.smithi136.stderr:noscrub is set 2024-09-15T21:03:07.596 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-09-15T21:03:07.986 INFO:tasks.workunit.client.0.smithi136.stderr:nodeep-scrub is set 2024-09-15T21:03:07.998 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-09-15T21:03:07.999 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-09-15T21:03:07.999 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool ecpool put SOMETHING td/osd-scrub-repair/ORIGINAL 2024-09-15T21:03:08.486 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:253: corrupt_and_repair_erasure_coded: get_primary ecpool SOMETHING 2024-09-15T21:03:08.486 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1187: get_primary: local poolname=ecpool 2024-09-15T21:03:08.486 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1188: get_primary: local objectname=SOMETHING 2024-09-15T21:03:08.486 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1190: get_primary: ceph --format json osd map ecpool SOMETHING 2024-09-15T21:03:08.486 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1191: get_primary: jq .acting_primary 2024-09-15T21:03:08.841 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:253: corrupt_and_repair_erasure_coded: local primary=3 2024-09-15T21:03:08.842 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:254: corrupt_and_repair_erasure_coded: get_osds ecpool SOMETHING 2024-09-15T21:03:08.843 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1012: get_osds: local poolname=ecpool 2024-09-15T21:03:08.843 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1013: get_osds: local objectname=SOMETHING 2024-09-15T21:03:08.843 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:254: corrupt_and_repair_erasure_coded: sed -e s/3// 2024-09-15T21:03:08.843 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: ceph --format json osd map ecpool SOMETHING 2024-09-15T21:03:08.843 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: jq '.acting | .[]' 2024-09-15T21:03:09.133 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: local 'osds=3 2024-09-15T21:03:09.133 INFO:tasks.workunit.client.0.smithi136.stderr:5 2024-09-15T21:03:09.133 INFO:tasks.workunit.client.0.smithi136.stderr:9 2024-09-15T21:03:09.133 INFO:tasks.workunit.client.0.smithi136.stderr:0 2024-09-15T21:03:09.134 INFO:tasks.workunit.client.0.smithi136.stderr:6 2024-09-15T21:03:09.134 INFO:tasks.workunit.client.0.smithi136.stderr:2 2024-09-15T21:03:09.134 INFO:tasks.workunit.client.0.smithi136.stderr:1 2024-09-15T21:03:09.134 INFO:tasks.workunit.client.0.smithi136.stderr:7' 2024-09-15T21:03:09.134 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1018: get_osds: echo 3 5 9 0 6 2 1 7 2024-09-15T21:03:09.134 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:254: corrupt_and_repair_erasure_coded: osds=('5' '9' '0' '6' '2' '1' '7') 2024-09-15T21:03:09.135 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:254: corrupt_and_repair_erasure_coded: local -a osds 2024-09-15T21:03:09.135 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:255: corrupt_and_repair_erasure_coded: local not_primary_first=5 2024-09-15T21:03:09.135 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:256: corrupt_and_repair_erasure_coded: local not_primary_second=9 2024-09-15T21:03:09.135 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:259: corrupt_and_repair_erasure_coded: corrupt_and_repair_one td/osd-scrub-repair ecpool 3 2024-09-15T21:03:09.135 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:226: corrupt_and_repair_one: local dir=td/osd-scrub-repair 2024-09-15T21:03:09.135 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:227: corrupt_and_repair_one: local poolname=ecpool 2024-09-15T21:03:09.136 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:228: corrupt_and_repair_one: local osd=3 2024-09-15T21:03:09.136 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:233: corrupt_and_repair_one: objectstore_tool td/osd-scrub-repair 3 SOMETHING remove 2024-09-15T21:03:09.136 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2024-09-15T21:03:09.136 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2024-09-15T21:03:09.136 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=3 2024-09-15T21:03:09.137 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2024-09-15T21:03:09.137 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 3 SOMETHING remove 2024-09-15T21:03:09.137 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-09-15T21:03:09.137 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2024-09-15T21:03:09.137 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=3 2024-09-15T21:03:09.137 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2024-09-15T21:03:09.137 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.3 2024-09-15T21:03:09.138 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-15T21:03:09.138 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-15T21:03:09.138 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-15T21:03:09.138 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-15T21:03:09.138 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-15T21:03:09.435 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-15T21:03:09.435 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 3 SOMETHING remove 2024-09-15T21:03:09.435 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-15T21:03:09.436 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-15T21:03:09.436 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=3 2024-09-15T21:03:09.436 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-15T21:03:09.436 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/3 2024-09-15T21:03:09.436 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/3 SOMETHING remove 2024-09-15T21:03:11.419 INFO:tasks.workunit.client.0.smithi136.stdout:remove 0#2:eb822e21:::SOMETHING:head# 2024-09-15T21:03:11.936 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 3 2024-09-15T21:03:11.936 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2024-09-15T21:03:11.936 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-15T21:03:11.936 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=3 2024-09-15T21:03:11.936 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-15T21:03:11.936 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/3 2024-09-15T21:03:11.936 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=ce4892c7-6c6d-4b4f-8e6f-07a3e11133e5 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-15T21:03:11.936 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T21:03:11.936 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T21:03:11.937 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T21:03:11.937 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/3' 2024-09-15T21:03:11.937 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/3/journal' 2024-09-15T21:03:11.937 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-15T21:03:11.937 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-15T21:03:11.937 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-15T21:03:11.937 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-15T21:03:11.937 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-15T21:03:11.937 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-15T21:03:11.937 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-15T21:03:11.937 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T21:03:11.937 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52602 2024-09-15T21:03:11.937 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.52602/$cluster-$name.asok' 2024-09-15T21:03:11.938 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.52602/$cluster-$name.asok' 2024-09-15T21:03:11.938 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-15T21:03:11.938 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-15T21:03:11.938 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-15T21:03:11.938 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T21:03:11.938 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T21:03:11.938 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T21:03:11.938 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T21:03:11.938 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-15T21:03:11.938 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-15T21:03:11.938 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/3 2024-09-15T21:03:11.938 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.3 2024-09-15T21:03:11.938 INFO:tasks.workunit.client.0.smithi136.stderr:start osd.3 2024-09-15T21:03:11.940 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 3 --fsid=ce4892c7-6c6d-4b4f-8e6f-07a3e11133e5 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/3 --osd-journal=td/osd-scrub-repair/3/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.52602/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T21:03:11.940 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/3/whoami 2024-09-15T21:03:11.940 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 3 = 3 ']' 2024-09-15T21:03:11.940 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-15T21:03:11.940 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-15T21:03:11.940 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-15T21:03:11.967 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:03:11.964+0000 7fbb4e4ae640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:03:11.969 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:03:11.967+0000 7fbb4e4ae640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:03:11.971 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:03:11.968+0000 7fbb4e4ae640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:03:12.230 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 3 2024-09-15T21:03:12.230 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-15T21:03:12.230 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=3 2024-09-15T21:03:12.230 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-15T21:03:12.230 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-15T21:03:12.230 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:03:12.230 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-15T21:03:12.230 INFO:tasks.workunit.client.0.smithi136.stderr:0 2024-09-15T21:03:12.231 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:03:12.231 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-15T21:03:12.506 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:03:13.507 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:03:13.508 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:03:13.508 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-15T21:03:13.508 INFO:tasks.workunit.client.0.smithi136.stderr:1 2024-09-15T21:03:13.508 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:03:13.508 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-15T21:03:13.559 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:03:13.556+0000 7fbb4e4ae640 -1 Falling back to public interface 2024-09-15T21:03:13.794 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:03:14.795 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:03:14.795 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:03:14.796 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-15T21:03:14.796 INFO:tasks.workunit.client.0.smithi136.stderr:2 2024-09-15T21:03:14.796 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:03:14.796 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-15T21:03:15.085 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:03:16.086 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:03:16.086 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:03:16.086 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-15T21:03:16.087 INFO:tasks.workunit.client.0.smithi136.stderr:3 2024-09-15T21:03:16.087 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:03:16.087 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-15T21:03:16.278 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:03:16.275+0000 7fbb4e4ae640 -1 osd.3 69 log_to_monitors true 2024-09-15T21:03:16.379 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:03:17.381 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:03:17.381 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:03:17.381 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-15T21:03:17.381 INFO:tasks.workunit.client.0.smithi136.stderr:4 2024-09-15T21:03:17.381 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:03:17.381 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-15T21:03:17.733 INFO:tasks.workunit.client.0.smithi136.stderr:osd.3 up in weight 1 up_from 73 up_thru 73 down_at 70 last_clean_interval [20,69) [v2:127.0.0.1:6826/1093219286,v1:127.0.0.1:6827/1093219286] [v2:127.0.0.1:6828/1093219286,v1:127.0.0.1:6829/1093219286] exists,up 0eed652b-57cc-457d-986e-cdd453c584f5 2024-09-15T21:03:17.733 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-15T21:03:17.733 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-15T21:03:17.733 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-15T21:03:17.733 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2024-09-15T21:03:17.733 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-15T21:03:17.733 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-15T21:03:17.733 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-15T21:03:17.734 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-15T21:03:17.734 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-15T21:03:17.734 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-15T21:03:17.734 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-15T21:03:17.734 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-15T21:03:17.734 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-15T21:03:17.823 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-15T21:03:17.823 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-15T21:03:17.823 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-15T21:03:17.823 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-15T21:03:17.823 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-15T21:03:17.823 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-15T21:03:18.109 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-15T21:03:18.109 INFO:tasks.workunit.client.0.smithi136.stderr:1 2024-09-15T21:03:18.109 INFO:tasks.workunit.client.0.smithi136.stderr:2 2024-09-15T21:03:18.109 INFO:tasks.workunit.client.0.smithi136.stderr:3 2024-09-15T21:03:18.109 INFO:tasks.workunit.client.0.smithi136.stderr:4 2024-09-15T21:03:18.110 INFO:tasks.workunit.client.0.smithi136.stderr:5 2024-09-15T21:03:18.110 INFO:tasks.workunit.client.0.smithi136.stderr:6 2024-09-15T21:03:18.110 INFO:tasks.workunit.client.0.smithi136.stderr:7 2024-09-15T21:03:18.110 INFO:tasks.workunit.client.0.smithi136.stderr:8 2024-09-15T21:03:18.110 INFO:tasks.workunit.client.0.smithi136.stderr:9' 2024-09-15T21:03:18.110 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-15T21:03:18.110 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T21:03:18.110 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-15T21:03:18.213 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836560 2024-09-15T21:03:18.213 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836560 2024-09-15T21:03:18.213 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836560' 2024-09-15T21:03:18.213 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T21:03:18.213 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-15T21:03:18.393 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=42949673030 2024-09-15T21:03:18.393 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 42949673030 2024-09-15T21:03:18.393 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836560 1-42949673030' 2024-09-15T21:03:18.393 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T21:03:18.393 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-15T21:03:18.552 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=64424509505 2024-09-15T21:03:18.552 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 64424509505 2024-09-15T21:03:18.552 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836560 1-42949673030 2-64424509505' 2024-09-15T21:03:18.552 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T21:03:18.552 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-15T21:03:18.655 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=313532612610 2024-09-15T21:03:18.655 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 313532612610 2024-09-15T21:03:18.655 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836560 1-42949673030 2-64424509505 3-313532612610' 2024-09-15T21:03:18.655 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T21:03:18.655 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-09-15T21:03:18.757 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=107374182450 2024-09-15T21:03:18.757 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 107374182450 2024-09-15T21:03:18.757 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836560 1-42949673030 2-64424509505 3-313532612610 4-107374182450' 2024-09-15T21:03:18.757 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T21:03:18.757 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-09-15T21:03:18.859 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=128849018928 2024-09-15T21:03:18.860 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 128849018928 2024-09-15T21:03:18.860 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836560 1-42949673030 2-64424509505 3-313532612610 4-107374182450 5-128849018928' 2024-09-15T21:03:18.860 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T21:03:18.860 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-09-15T21:03:18.967 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=150323855398 2024-09-15T21:03:18.967 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 150323855398 2024-09-15T21:03:18.968 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836560 1-42949673030 2-64424509505 3-313532612610 4-107374182450 5-128849018928 6-150323855398' 2024-09-15T21:03:18.968 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T21:03:18.968 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2024-09-15T21:03:19.071 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=171798691870 2024-09-15T21:03:19.072 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 171798691870 2024-09-15T21:03:19.072 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836560 1-42949673030 2-64424509505 3-313532612610 4-107374182450 5-128849018928 6-150323855398 7-171798691870' 2024-09-15T21:03:19.072 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T21:03:19.072 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2024-09-15T21:03:19.175 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=193273528346 2024-09-15T21:03:19.175 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 193273528346 2024-09-15T21:03:19.175 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836560 1-42949673030 2-64424509505 3-313532612610 4-107374182450 5-128849018928 6-150323855398 7-171798691870 8-193273528346' 2024-09-15T21:03:19.175 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T21:03:19.175 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2024-09-15T21:03:19.277 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=214748364821 2024-09-15T21:03:19.277 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 214748364821 2024-09-15T21:03:19.277 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836560 1-42949673030 2-64424509505 3-313532612610 4-107374182450 5-128849018928 6-150323855398 7-171798691870 8-193273528346 9-214748364821' 2024-09-15T21:03:19.277 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T21:03:19.277 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836560 2024-09-15T21:03:19.278 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T21:03:19.278 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-15T21:03:19.279 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836560 2024-09-15T21:03:19.279 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T21:03:19.280 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836560 2024-09-15T21:03:19.280 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836560' 2024-09-15T21:03:19.280 INFO:tasks.workunit.client.0.smithi136.stderr:waiting osd.0 seq 21474836560 2024-09-15T21:03:19.280 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T21:03:19.555 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836560 -lt 21474836560 2024-09-15T21:03:19.555 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T21:03:19.556 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-42949673030 2024-09-15T21:03:19.556 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T21:03:19.557 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-15T21:03:19.557 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-42949673030 2024-09-15T21:03:19.557 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T21:03:19.558 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=42949673030 2024-09-15T21:03:19.558 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 42949673030' 2024-09-15T21:03:19.559 INFO:tasks.workunit.client.0.smithi136.stderr:waiting osd.1 seq 42949673030 2024-09-15T21:03:19.559 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-15T21:03:19.839 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 42949673030 -lt 42949673030 2024-09-15T21:03:19.839 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T21:03:19.839 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-64424509505 2024-09-15T21:03:19.839 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T21:03:19.840 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-15T21:03:19.841 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-64424509505 2024-09-15T21:03:19.841 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T21:03:19.842 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=64424509505 2024-09-15T21:03:19.842 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 64424509505' 2024-09-15T21:03:19.842 INFO:tasks.workunit.client.0.smithi136.stderr:waiting osd.2 seq 64424509505 2024-09-15T21:03:19.842 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-15T21:03:20.120 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 64424509505 -lt 64424509505 2024-09-15T21:03:20.120 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T21:03:20.121 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 3-313532612610 2024-09-15T21:03:20.121 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T21:03:20.121 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=3 2024-09-15T21:03:20.122 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 3-313532612610 2024-09-15T21:03:20.122 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T21:03:20.123 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=313532612610 2024-09-15T21:03:20.123 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.3 seq 313532612610' 2024-09-15T21:03:20.123 INFO:tasks.workunit.client.0.smithi136.stderr:waiting osd.3 seq 313532612610 2024-09-15T21:03:20.123 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-15T21:03:20.446 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 313532612610 -lt 313532612610 2024-09-15T21:03:20.446 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T21:03:20.446 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 4-107374182450 2024-09-15T21:03:20.447 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T21:03:20.448 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=4 2024-09-15T21:03:20.449 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 4-107374182450 2024-09-15T21:03:20.449 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T21:03:20.449 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=107374182450 2024-09-15T21:03:20.449 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.4 seq 107374182450' 2024-09-15T21:03:20.449 INFO:tasks.workunit.client.0.smithi136.stderr:waiting osd.4 seq 107374182450 2024-09-15T21:03:20.450 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 4 2024-09-15T21:03:20.724 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 107374182450 -lt 107374182450 2024-09-15T21:03:20.725 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T21:03:20.725 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 5-128849018928 2024-09-15T21:03:20.725 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T21:03:20.726 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=5 2024-09-15T21:03:20.726 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 5-128849018928 2024-09-15T21:03:20.727 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T21:03:20.728 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=128849018928 2024-09-15T21:03:20.728 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.5 seq 128849018928' 2024-09-15T21:03:20.728 INFO:tasks.workunit.client.0.smithi136.stderr:waiting osd.5 seq 128849018928 2024-09-15T21:03:20.728 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 5 2024-09-15T21:03:21.018 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 128849018929 -lt 128849018928 2024-09-15T21:03:21.018 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T21:03:21.019 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 6-150323855398 2024-09-15T21:03:21.019 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T21:03:21.020 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=6 2024-09-15T21:03:21.020 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 6-150323855398 2024-09-15T21:03:21.021 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T21:03:21.021 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=150323855398 2024-09-15T21:03:21.022 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.6 seq 150323855398' 2024-09-15T21:03:21.022 INFO:tasks.workunit.client.0.smithi136.stderr:waiting osd.6 seq 150323855398 2024-09-15T21:03:21.022 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 6 2024-09-15T21:03:21.300 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 150323855399 -lt 150323855398 2024-09-15T21:03:21.301 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T21:03:21.301 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 7-171798691870 2024-09-15T21:03:21.301 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T21:03:21.302 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=7 2024-09-15T21:03:21.302 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 7-171798691870 2024-09-15T21:03:21.303 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T21:03:21.303 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=171798691870 2024-09-15T21:03:21.304 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.7 seq 171798691870' 2024-09-15T21:03:21.304 INFO:tasks.workunit.client.0.smithi136.stderr:waiting osd.7 seq 171798691870 2024-09-15T21:03:21.304 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 7 2024-09-15T21:03:21.580 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 171798691871 -lt 171798691870 2024-09-15T21:03:21.580 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T21:03:21.581 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 8-193273528346 2024-09-15T21:03:21.581 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T21:03:21.582 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=8 2024-09-15T21:03:21.582 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 8-193273528346 2024-09-15T21:03:21.582 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T21:03:21.583 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=193273528346 2024-09-15T21:03:21.584 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.8 seq 193273528346' 2024-09-15T21:03:21.584 INFO:tasks.workunit.client.0.smithi136.stderr:waiting osd.8 seq 193273528346 2024-09-15T21:03:21.584 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 8 2024-09-15T21:03:21.864 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 193273528347 -lt 193273528346 2024-09-15T21:03:21.864 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T21:03:21.864 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 9-214748364821 2024-09-15T21:03:21.864 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T21:03:21.865 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=9 2024-09-15T21:03:21.866 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 9-214748364821 2024-09-15T21:03:21.866 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T21:03:21.867 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=214748364821 2024-09-15T21:03:21.867 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.9 seq 214748364821' 2024-09-15T21:03:21.867 INFO:tasks.workunit.client.0.smithi136.stderr:waiting osd.9 seq 214748364821 2024-09-15T21:03:21.867 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 9 2024-09-15T21:03:22.144 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 214748364821 -lt 214748364821 2024-09-15T21:03:22.144 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-15T21:03:22.144 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T21:03:22.144 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T21:03:22.494 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 5 == 0 2024-09-15T21:03:22.494 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-15T21:03:22.494 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-15T21:03:22.494 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-15T21:03:22.494 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T21:03:22.494 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T21:03:22.494 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T21:03:22.494 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T21:03:22.767 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=5 2024-09-15T21:03:22.768 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-15T21:03:22.768 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T21:03:22.768 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T21:03:23.141 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 = 5 2024-09-15T21:03:23.142 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-15T21:03:23.142 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-15T21:03:23.142 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:237: corrupt_and_repair_one: get_pg ecpool SOMETHING 2024-09-15T21:03:23.142 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1084: get_pg: local poolname=ecpool 2024-09-15T21:03:23.142 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1085: get_pg: local objectname=SOMETHING 2024-09-15T21:03:23.142 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1087: get_pg: ceph --format json osd map ecpool SOMETHING 2024-09-15T21:03:23.142 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1087: get_pg: jq -r .pgid 2024-09-15T21:03:23.422 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:237: corrupt_and_repair_one: local pg=2.0 2024-09-15T21:03:23.423 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:238: corrupt_and_repair_one: repair 2.0 2024-09-15T21:03:23.423 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1846: repair: local pgid=2.0 2024-09-15T21:03:23.423 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1847: repair: get_last_scrub_stamp 2.0 2024-09-15T21:03:23.423 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2024-09-15T21:03:23.423 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-15T21:03:23.423 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-15T21:03:23.423 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-09-15T21:03:23.698 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1847: repair: local last_scrub=2024-09-15T21:03:00.441579+0000 2024-09-15T21:03:23.698 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1848: repair: ceph pg repair 2.0 2024-09-15T21:03:23.962 INFO:tasks.workunit.client.0.smithi136.stderr:instructing pg 2.0s0 on osd.3 to repair 2024-09-15T21:03:23.974 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1849: repair: wait_for_scrub 2.0 2024-09-15T21:03:00.441579+0000 2024-09-15T21:03:23.974 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: local pgid=2.0 2024-09-15T21:03:23.974 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2018: wait_for_scrub: local last_scrub=2024-09-15T21:03:00.441579+0000 2024-09-15T21:03:23.975 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2019: wait_for_scrub: local sname=last_scrub_stamp 2024-09-15T21:03:23.975 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i=0 )) 2024-09-15T21:03:23.975 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-15T21:03:23.975 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-09-15T21:03:23.975 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2024-09-15T21:03:23.975 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-15T21:03:23.975 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-15T21:03:23.975 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-09-15T21:03:24.249 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-15T21:03:00.441579+0000 '>' 2024-09-15T21:03:00.441579+0000 2024-09-15T21:03:24.249 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-15T21:03:25.250 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-15T21:03:25.250 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-15T21:03:25.251 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-09-15T21:03:25.251 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2024-09-15T21:03:25.251 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-15T21:03:25.251 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-15T21:03:25.251 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-09-15T21:03:25.562 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-15T21:03:00.441579+0000 '>' 2024-09-15T21:03:00.441579+0000 2024-09-15T21:03:25.840 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-15T21:03:26.563 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-15T21:03:26.563 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-15T21:03:26.564 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-09-15T21:03:26.564 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2024-09-15T21:03:26.564 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-15T21:03:26.564 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-15T21:03:26.564 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-09-15T21:03:26.851 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-15T21:03:24.334290+0000 '>' 2024-09-15T21:03:00.441579+0000 2024-09-15T21:03:26.851 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2023: wait_for_scrub: return 0 2024-09-15T21:03:26.852 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1850: repair: sleep 2 2024-09-15T21:03:28.854 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:242: corrupt_and_repair_one: objectstore_tool td/osd-scrub-repair 9 SOMETHING list-attrs 2024-09-15T21:03:28.854 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2024-09-15T21:03:28.854 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2024-09-15T21:03:28.854 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=9 2024-09-15T21:03:28.854 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2024-09-15T21:03:28.854 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 9 SOMETHING list-attrs 2024-09-15T21:03:28.854 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-09-15T21:03:28.854 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2024-09-15T21:03:28.854 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=9 2024-09-15T21:03:28.854 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2024-09-15T21:03:28.854 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.9 2024-09-15T21:03:28.854 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-15T21:03:28.854 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-15T21:03:28.855 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-15T21:03:28.855 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-15T21:03:28.855 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-15T21:03:29.162 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-15T21:03:29.162 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 9 SOMETHING list-attrs 2024-09-15T21:03:29.162 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-15T21:03:29.162 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-15T21:03:29.162 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=9 2024-09-15T21:03:29.163 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-15T21:03:29.163 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/9 2024-09-15T21:03:29.163 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/9 SOMETHING list-attrs 2024-09-15T21:03:29.646 INFO:tasks.workunit.client.0.smithi136.stdout:_ 2024-09-15T21:03:29.647 INFO:tasks.workunit.client.0.smithi136.stdout:hinfo_key 2024-09-15T21:03:29.647 INFO:tasks.workunit.client.0.smithi136.stdout:snapset 2024-09-15T21:03:29.915 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 9 2024-09-15T21:03:29.916 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2024-09-15T21:03:29.916 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-15T21:03:29.916 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=9 2024-09-15T21:03:29.916 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-15T21:03:29.916 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/9 2024-09-15T21:03:29.916 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=ce4892c7-6c6d-4b4f-8e6f-07a3e11133e5 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-15T21:03:29.916 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T21:03:29.916 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T21:03:29.916 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T21:03:29.916 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/9' 2024-09-15T21:03:29.916 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/9/journal' 2024-09-15T21:03:29.916 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-15T21:03:29.916 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-15T21:03:29.917 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-15T21:03:29.917 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-15T21:03:29.917 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-15T21:03:29.917 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-15T21:03:29.917 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-15T21:03:29.917 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T21:03:29.917 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52602 2024-09-15T21:03:29.917 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.52602/$cluster-$name.asok' 2024-09-15T21:03:29.917 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.52602/$cluster-$name.asok' 2024-09-15T21:03:29.917 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-15T21:03:29.917 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-15T21:03:29.917 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-15T21:03:29.917 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T21:03:29.917 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T21:03:29.918 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T21:03:29.918 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T21:03:29.918 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-15T21:03:29.918 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-15T21:03:29.918 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/9 2024-09-15T21:03:29.918 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.9 2024-09-15T21:03:29.918 INFO:tasks.workunit.client.0.smithi136.stderr:start osd.9 2024-09-15T21:03:29.919 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 9 --fsid=ce4892c7-6c6d-4b4f-8e6f-07a3e11133e5 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/9 --osd-journal=td/osd-scrub-repair/9/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.52602/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T21:03:29.919 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/9/whoami 2024-09-15T21:03:29.919 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 9 = 9 ']' 2024-09-15T21:03:29.919 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-15T21:03:29.919 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-15T21:03:29.919 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-15T21:03:29.946 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:03:29.943+0000 7f76a16ae640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:03:29.976 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:03:29.973+0000 7f76a16ae640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:03:29.978 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:03:29.976+0000 7f76a16ae640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:03:30.293 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 9 2024-09-15T21:03:30.293 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-15T21:03:30.294 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=9 2024-09-15T21:03:30.294 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-15T21:03:30.294 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-15T21:03:30.294 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:03:30.294 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-15T21:03:30.294 INFO:tasks.workunit.client.0.smithi136.stderr:0 2024-09-15T21:03:30.294 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:03:30.294 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.9 up' 2024-09-15T21:03:30.622 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:03:31.071 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:03:31.068+0000 7f76a16ae640 -1 Falling back to public interface 2024-09-15T21:03:31.624 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:03:31.624 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:03:31.624 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-15T21:03:31.624 INFO:tasks.workunit.client.0.smithi136.stderr:1 2024-09-15T21:03:31.624 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:03:31.624 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.9 up' 2024-09-15T21:03:31.905 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:03:32.906 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:03:32.906 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:03:32.907 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-15T21:03:32.907 INFO:tasks.workunit.client.0.smithi136.stderr:2 2024-09-15T21:03:32.907 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:03:32.907 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.9 up' 2024-09-15T21:03:33.195 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:03:34.197 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:03:34.197 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:03:34.197 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-15T21:03:34.197 INFO:tasks.workunit.client.0.smithi136.stderr:3 2024-09-15T21:03:34.197 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:03:34.197 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.9 up' 2024-09-15T21:03:34.288 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:03:34.285+0000 7f76a16ae640 -1 osd.9 74 log_to_monitors true 2024-09-15T21:03:34.520 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:03:35.521 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:03:35.521 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:03:35.521 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-15T21:03:35.521 INFO:tasks.workunit.client.0.smithi136.stderr:4 2024-09-15T21:03:35.521 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:03:35.522 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.9 up' 2024-09-15T21:03:35.922 INFO:tasks.workunit.client.0.smithi136.stderr:osd.9 up in weight 1 up_from 78 up_thru 0 down_at 75 last_clean_interval [50,74) [v2:127.0.0.1:6874/484787951,v1:127.0.0.1:6875/484787951] [v2:127.0.0.1:6876/484787951,v1:127.0.0.1:6877/484787951] exists,up ccfb463f-a1cc-4f3c-a7db-ae126efbeaa5 2024-09-15T21:03:35.922 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-15T21:03:35.923 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-15T21:03:35.923 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-15T21:03:35.923 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2024-09-15T21:03:35.923 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-15T21:03:35.923 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-15T21:03:35.923 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-15T21:03:35.924 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-15T21:03:35.924 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-15T21:03:35.924 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-15T21:03:35.924 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-15T21:03:35.924 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-15T21:03:35.924 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-15T21:03:36.014 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-15T21:03:36.014 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-15T21:03:36.014 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-15T21:03:36.014 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-15T21:03:36.014 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-15T21:03:36.014 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-15T21:03:36.426 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-15T21:03:36.427 INFO:tasks.workunit.client.0.smithi136.stderr:1 2024-09-15T21:03:36.427 INFO:tasks.workunit.client.0.smithi136.stderr:2 2024-09-15T21:03:36.427 INFO:tasks.workunit.client.0.smithi136.stderr:3 2024-09-15T21:03:36.427 INFO:tasks.workunit.client.0.smithi136.stderr:4 2024-09-15T21:03:36.427 INFO:tasks.workunit.client.0.smithi136.stderr:5 2024-09-15T21:03:36.427 INFO:tasks.workunit.client.0.smithi136.stderr:6 2024-09-15T21:03:36.427 INFO:tasks.workunit.client.0.smithi136.stderr:7 2024-09-15T21:03:36.427 INFO:tasks.workunit.client.0.smithi136.stderr:8 2024-09-15T21:03:36.427 INFO:tasks.workunit.client.0.smithi136.stderr:9' 2024-09-15T21:03:36.427 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-15T21:03:36.427 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T21:03:36.427 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-15T21:03:36.531 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836564 2024-09-15T21:03:36.531 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836564 2024-09-15T21:03:36.531 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836564' 2024-09-15T21:03:36.531 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T21:03:36.531 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-15T21:03:36.635 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=42949673035 2024-09-15T21:03:36.635 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 42949673035 2024-09-15T21:03:36.635 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836564 1-42949673035' 2024-09-15T21:03:36.635 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T21:03:36.635 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-15T21:03:36.739 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=64424509509 2024-09-15T21:03:36.740 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 64424509509 2024-09-15T21:03:36.740 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836564 1-42949673035 2-64424509509' 2024-09-15T21:03:36.740 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T21:03:36.740 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-15T21:03:36.843 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=313532612615 2024-09-15T21:03:36.843 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 313532612615 2024-09-15T21:03:36.843 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836564 1-42949673035 2-64424509509 3-313532612615' 2024-09-15T21:03:36.843 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T21:03:36.843 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-09-15T21:03:36.949 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=107374182454 2024-09-15T21:03:36.949 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 107374182454 2024-09-15T21:03:36.949 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836564 1-42949673035 2-64424509509 3-313532612615 4-107374182454' 2024-09-15T21:03:36.949 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T21:03:36.949 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-09-15T21:03:37.053 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=128849018933 2024-09-15T21:03:37.053 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 128849018933 2024-09-15T21:03:37.053 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836564 1-42949673035 2-64424509509 3-313532612615 4-107374182454 5-128849018933' 2024-09-15T21:03:37.053 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T21:03:37.053 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-09-15T21:03:37.155 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=150323855403 2024-09-15T21:03:37.155 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 150323855403 2024-09-15T21:03:37.155 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836564 1-42949673035 2-64424509509 3-313532612615 4-107374182454 5-128849018933 6-150323855403' 2024-09-15T21:03:37.155 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T21:03:37.155 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2024-09-15T21:03:37.258 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=171798691875 2024-09-15T21:03:37.259 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 171798691875 2024-09-15T21:03:37.259 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836564 1-42949673035 2-64424509509 3-313532612615 4-107374182454 5-128849018933 6-150323855403 7-171798691875' 2024-09-15T21:03:37.259 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T21:03:37.259 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2024-09-15T21:03:37.363 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=193273528351 2024-09-15T21:03:37.363 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 193273528351 2024-09-15T21:03:37.364 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836564 1-42949673035 2-64424509509 3-313532612615 4-107374182454 5-128849018933 6-150323855403 7-171798691875 8-193273528351' 2024-09-15T21:03:37.364 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T21:03:37.364 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2024-09-15T21:03:37.468 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=335007449090 2024-09-15T21:03:37.469 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 335007449090 2024-09-15T21:03:37.469 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836564 1-42949673035 2-64424509509 3-313532612615 4-107374182454 5-128849018933 6-150323855403 7-171798691875 8-193273528351 9-335007449090' 2024-09-15T21:03:37.469 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T21:03:37.469 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836564 2024-09-15T21:03:37.469 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T21:03:37.470 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-15T21:03:37.470 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836564 2024-09-15T21:03:37.470 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T21:03:37.471 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836564 2024-09-15T21:03:37.471 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836564' 2024-09-15T21:03:37.471 INFO:tasks.workunit.client.0.smithi136.stderr:waiting osd.0 seq 21474836564 2024-09-15T21:03:37.472 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T21:03:37.749 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836565 -lt 21474836564 2024-09-15T21:03:37.749 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T21:03:37.749 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-42949673035 2024-09-15T21:03:37.749 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T21:03:37.750 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-15T21:03:37.751 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-42949673035 2024-09-15T21:03:37.751 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T21:03:37.752 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=42949673035 2024-09-15T21:03:37.752 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 42949673035' 2024-09-15T21:03:37.752 INFO:tasks.workunit.client.0.smithi136.stderr:waiting osd.1 seq 42949673035 2024-09-15T21:03:37.752 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-15T21:03:38.030 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 42949673035 -lt 42949673035 2024-09-15T21:03:38.031 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T21:03:38.031 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-64424509509 2024-09-15T21:03:38.031 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T21:03:38.032 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-15T21:03:38.033 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-64424509509 2024-09-15T21:03:38.033 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T21:03:38.033 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=64424509509 2024-09-15T21:03:38.034 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 64424509509' 2024-09-15T21:03:38.034 INFO:tasks.workunit.client.0.smithi136.stderr:waiting osd.2 seq 64424509509 2024-09-15T21:03:38.034 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-15T21:03:38.316 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 64424509509 -lt 64424509509 2024-09-15T21:03:38.317 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T21:03:38.317 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 3-313532612615 2024-09-15T21:03:38.317 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T21:03:38.318 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=3 2024-09-15T21:03:38.318 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 3-313532612615 2024-09-15T21:03:38.318 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T21:03:38.319 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=313532612615 2024-09-15T21:03:38.319 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.3 seq 313532612615' 2024-09-15T21:03:38.320 INFO:tasks.workunit.client.0.smithi136.stderr:waiting osd.3 seq 313532612615 2024-09-15T21:03:38.320 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-15T21:03:38.639 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 313532612614 -lt 313532612615 2024-09-15T21:03:38.640 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-15T21:03:39.640 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-15T21:03:39.641 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-15T21:03:39.930 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 313532612615 -lt 313532612615 2024-09-15T21:03:39.930 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T21:03:39.930 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 4-107374182454 2024-09-15T21:03:39.930 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T21:03:39.931 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=4 2024-09-15T21:03:39.932 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 4-107374182454 2024-09-15T21:03:39.932 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T21:03:39.932 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=107374182454 2024-09-15T21:03:39.933 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.4 seq 107374182454' 2024-09-15T21:03:39.933 INFO:tasks.workunit.client.0.smithi136.stderr:waiting osd.4 seq 107374182454 2024-09-15T21:03:39.933 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 4 2024-09-15T21:03:40.213 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 107374182455 -lt 107374182454 2024-09-15T21:03:40.213 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T21:03:40.213 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 5-128849018933 2024-09-15T21:03:40.213 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T21:03:40.214 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=5 2024-09-15T21:03:40.215 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 5-128849018933 2024-09-15T21:03:40.215 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T21:03:40.216 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=128849018933 2024-09-15T21:03:40.216 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.5 seq 128849018933' 2024-09-15T21:03:40.216 INFO:tasks.workunit.client.0.smithi136.stderr:waiting osd.5 seq 128849018933 2024-09-15T21:03:40.216 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 5 2024-09-15T21:03:40.585 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 128849018933 -lt 128849018933 2024-09-15T21:03:40.586 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T21:03:40.586 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 6-150323855403 2024-09-15T21:03:40.586 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T21:03:40.587 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=6 2024-09-15T21:03:40.588 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 6-150323855403 2024-09-15T21:03:40.588 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T21:03:40.589 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=150323855403 2024-09-15T21:03:40.589 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.6 seq 150323855403' 2024-09-15T21:03:40.589 INFO:tasks.workunit.client.0.smithi136.stderr:waiting osd.6 seq 150323855403 2024-09-15T21:03:40.589 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 6 2024-09-15T21:03:40.882 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 150323855403 -lt 150323855403 2024-09-15T21:03:40.882 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T21:03:40.882 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 7-171798691875 2024-09-15T21:03:40.882 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T21:03:40.883 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=7 2024-09-15T21:03:40.884 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 7-171798691875 2024-09-15T21:03:40.884 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T21:03:40.885 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=171798691875 2024-09-15T21:03:40.885 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.7 seq 171798691875' 2024-09-15T21:03:40.885 INFO:tasks.workunit.client.0.smithi136.stderr:waiting osd.7 seq 171798691875 2024-09-15T21:03:40.885 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 7 2024-09-15T21:03:41.191 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 171798691876 -lt 171798691875 2024-09-15T21:03:41.191 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T21:03:41.192 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 8-193273528351 2024-09-15T21:03:41.192 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T21:03:41.193 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=8 2024-09-15T21:03:41.193 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 8-193273528351 2024-09-15T21:03:41.193 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T21:03:41.194 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=193273528351 2024-09-15T21:03:41.194 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.8 seq 193273528351' 2024-09-15T21:03:41.195 INFO:tasks.workunit.client.0.smithi136.stderr:waiting osd.8 seq 193273528351 2024-09-15T21:03:41.195 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 8 2024-09-15T21:03:41.477 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 193273528352 -lt 193273528351 2024-09-15T21:03:41.477 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T21:03:41.478 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 9-335007449090 2024-09-15T21:03:41.478 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T21:03:41.478 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=9 2024-09-15T21:03:41.479 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 9-335007449090 2024-09-15T21:03:41.479 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T21:03:41.480 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=335007449090 2024-09-15T21:03:41.480 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.9 seq 335007449090' 2024-09-15T21:03:41.480 INFO:tasks.workunit.client.0.smithi136.stderr:waiting osd.9 seq 335007449090 2024-09-15T21:03:41.481 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 9 2024-09-15T21:03:41.756 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 335007449091 -lt 335007449090 2024-09-15T21:03:41.756 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-15T21:03:41.756 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T21:03:41.756 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T21:03:42.103 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 5 == 0 2024-09-15T21:03:42.103 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-15T21:03:42.103 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-15T21:03:42.103 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-15T21:03:42.103 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T21:03:42.103 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T21:03:42.104 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T21:03:42.104 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T21:03:42.392 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=5 2024-09-15T21:03:42.393 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-15T21:03:42.393 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T21:03:42.394 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T21:03:42.738 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 = 5 2024-09-15T21:03:42.738 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-15T21:03:42.738 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-15T21:03:42.738 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:243: corrupt_and_repair_one: rados --pool ecpool get SOMETHING td/osd-scrub-repair/COPY 2024-09-15T21:03:42.774 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:244: corrupt_and_repair_one: diff td/osd-scrub-repair/ORIGINAL td/osd-scrub-repair/COPY 2024-09-15T21:03:42.776 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:261: corrupt_and_repair_erasure_coded: corrupt_and_repair_one td/osd-scrub-repair ecpool 5 2024-09-15T21:03:42.776 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:226: corrupt_and_repair_one: local dir=td/osd-scrub-repair 2024-09-15T21:03:42.776 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:227: corrupt_and_repair_one: local poolname=ecpool 2024-09-15T21:03:42.776 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:228: corrupt_and_repair_one: local osd=5 2024-09-15T21:03:42.777 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:233: corrupt_and_repair_one: objectstore_tool td/osd-scrub-repair 5 SOMETHING remove 2024-09-15T21:03:42.777 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2024-09-15T21:03:42.777 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2024-09-15T21:03:42.777 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=5 2024-09-15T21:03:42.777 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2024-09-15T21:03:42.777 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 5 SOMETHING remove 2024-09-15T21:03:42.777 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-09-15T21:03:42.777 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2024-09-15T21:03:42.777 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=5 2024-09-15T21:03:42.777 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2024-09-15T21:03:42.777 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.5 2024-09-15T21:03:42.777 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-15T21:03:42.777 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-15T21:03:42.777 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-15T21:03:42.778 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-15T21:03:42.778 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-15T21:03:43.082 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-15T21:03:43.082 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 5 SOMETHING remove 2024-09-15T21:03:43.083 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-15T21:03:43.083 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-15T21:03:43.083 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=5 2024-09-15T21:03:43.083 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-15T21:03:43.083 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/5 2024-09-15T21:03:43.083 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/5 SOMETHING remove 2024-09-15T21:03:45.155 INFO:tasks.workunit.client.0.smithi136.stdout:remove 1#2:eb822e21:::SOMETHING:head# 2024-09-15T21:03:45.672 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 5 2024-09-15T21:03:45.672 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2024-09-15T21:03:45.672 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-15T21:03:45.672 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=5 2024-09-15T21:03:45.672 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-15T21:03:45.672 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/5 2024-09-15T21:03:45.672 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=ce4892c7-6c6d-4b4f-8e6f-07a3e11133e5 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-15T21:03:45.672 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T21:03:45.672 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T21:03:45.672 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T21:03:45.672 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/5' 2024-09-15T21:03:45.673 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/5/journal' 2024-09-15T21:03:45.673 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-15T21:03:45.673 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-15T21:03:45.673 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-15T21:03:45.673 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-15T21:03:45.673 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-15T21:03:45.673 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-15T21:03:45.673 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-15T21:03:45.673 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T21:03:45.673 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52602 2024-09-15T21:03:45.673 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.52602/$cluster-$name.asok' 2024-09-15T21:03:45.674 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.52602/$cluster-$name.asok' 2024-09-15T21:03:45.674 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-15T21:03:45.674 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-15T21:03:45.674 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-15T21:03:45.674 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T21:03:45.674 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T21:03:45.674 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T21:03:45.674 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T21:03:45.674 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-15T21:03:45.674 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-15T21:03:45.674 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/5 2024-09-15T21:03:45.675 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.5 2024-09-15T21:03:45.675 INFO:tasks.workunit.client.0.smithi136.stderr:start osd.5 2024-09-15T21:03:45.675 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 5 --fsid=ce4892c7-6c6d-4b4f-8e6f-07a3e11133e5 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/5 --osd-journal=td/osd-scrub-repair/5/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.52602/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T21:03:45.675 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/5/whoami 2024-09-15T21:03:45.676 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 5 = 5 ']' 2024-09-15T21:03:45.676 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-15T21:03:45.676 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-15T21:03:45.676 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-15T21:03:45.703 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:03:45.700+0000 7f8713f58640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:03:45.705 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:03:45.703+0000 7f8713f58640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:03:45.706 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:03:45.704+0000 7f8713f58640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:03:45.961 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 5 2024-09-15T21:03:45.962 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-15T21:03:45.962 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=5 2024-09-15T21:03:45.962 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-15T21:03:45.962 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-15T21:03:45.962 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:03:45.962 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-15T21:03:45.962 INFO:tasks.workunit.client.0.smithi136.stderr:0 2024-09-15T21:03:45.962 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:03:45.962 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-15T21:03:46.236 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:03:47.043 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:03:47.040+0000 7f8713f58640 -1 Falling back to public interface 2024-09-15T21:03:47.238 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:03:47.238 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:03:47.238 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-15T21:03:47.238 INFO:tasks.workunit.client.0.smithi136.stderr:1 2024-09-15T21:03:47.238 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:03:47.238 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-15T21:03:47.524 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:03:48.525 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:03:48.525 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:03:48.526 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-15T21:03:48.526 INFO:tasks.workunit.client.0.smithi136.stderr:2 2024-09-15T21:03:48.526 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:03:48.526 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-15T21:03:48.814 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:03:49.730 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:03:49.726+0000 7f8713f58640 -1 osd.5 79 log_to_monitors true 2024-09-15T21:03:49.816 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:03:49.816 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:03:49.816 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-15T21:03:49.816 INFO:tasks.workunit.client.0.smithi136.stderr:3 2024-09-15T21:03:49.816 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:03:49.816 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-15T21:03:50.146 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:03:51.148 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:03:51.148 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:03:51.148 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-15T21:03:51.148 INFO:tasks.workunit.client.0.smithi136.stderr:4 2024-09-15T21:03:51.148 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:03:51.148 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-15T21:03:51.592 INFO:tasks.workunit.client.0.smithi136.stderr:osd.5 up in weight 1 up_from 83 up_thru 83 down_at 80 last_clean_interval [30,79) [v2:127.0.0.1:6842/4003426185,v1:127.0.0.1:6843/4003426185] [v2:127.0.0.1:6844/4003426185,v1:127.0.0.1:6845/4003426185] exists,up c3187161-e486-4b18-8257-d69c125e9792 2024-09-15T21:03:51.592 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-15T21:03:51.592 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-15T21:03:51.592 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-15T21:03:51.592 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2024-09-15T21:03:51.592 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-15T21:03:51.592 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-15T21:03:51.592 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-15T21:03:51.593 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-15T21:03:51.593 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-15T21:03:51.593 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-15T21:03:51.593 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-15T21:03:51.594 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-15T21:03:51.594 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-15T21:03:51.682 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-15T21:03:51.682 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-15T21:03:51.682 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-15T21:03:51.682 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-15T21:03:51.682 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-15T21:03:51.682 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-15T21:03:52.119 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-15T21:03:52.119 INFO:tasks.workunit.client.0.smithi136.stderr:1 2024-09-15T21:03:52.119 INFO:tasks.workunit.client.0.smithi136.stderr:2 2024-09-15T21:03:52.119 INFO:tasks.workunit.client.0.smithi136.stderr:3 2024-09-15T21:03:52.119 INFO:tasks.workunit.client.0.smithi136.stderr:4 2024-09-15T21:03:52.119 INFO:tasks.workunit.client.0.smithi136.stderr:5 2024-09-15T21:03:52.119 INFO:tasks.workunit.client.0.smithi136.stderr:6 2024-09-15T21:03:52.120 INFO:tasks.workunit.client.0.smithi136.stderr:7 2024-09-15T21:03:52.120 INFO:tasks.workunit.client.0.smithi136.stderr:8 2024-09-15T21:03:52.120 INFO:tasks.workunit.client.0.smithi136.stderr:9' 2024-09-15T21:03:52.120 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-15T21:03:52.120 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T21:03:52.120 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-15T21:03:52.268 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836569 2024-09-15T21:03:52.268 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836569 2024-09-15T21:03:52.268 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836569' 2024-09-15T21:03:52.268 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T21:03:52.268 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-15T21:03:52.370 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=42949673039 2024-09-15T21:03:52.371 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 42949673039 2024-09-15T21:03:52.371 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836569 1-42949673039' 2024-09-15T21:03:52.371 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T21:03:52.371 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-15T21:03:52.474 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=64424509514 2024-09-15T21:03:52.474 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 64424509514 2024-09-15T21:03:52.475 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836569 1-42949673039 2-64424509514' 2024-09-15T21:03:52.475 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T21:03:52.475 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-15T21:03:52.577 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=313532612619 2024-09-15T21:03:52.577 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 313532612619 2024-09-15T21:03:52.577 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836569 1-42949673039 2-64424509514 3-313532612619' 2024-09-15T21:03:52.577 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T21:03:52.578 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-09-15T21:03:52.681 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=107374182458 2024-09-15T21:03:52.681 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 107374182458 2024-09-15T21:03:52.681 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836569 1-42949673039 2-64424509514 3-313532612619 4-107374182458' 2024-09-15T21:03:52.681 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T21:03:52.681 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-09-15T21:03:52.784 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=356482285570 2024-09-15T21:03:52.785 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 356482285570 2024-09-15T21:03:52.785 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836569 1-42949673039 2-64424509514 3-313532612619 4-107374182458 5-356482285570' 2024-09-15T21:03:52.785 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T21:03:52.785 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-09-15T21:03:52.900 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=150323855407 2024-09-15T21:03:52.900 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 150323855407 2024-09-15T21:03:52.900 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836569 1-42949673039 2-64424509514 3-313532612619 4-107374182458 5-356482285570 6-150323855407' 2024-09-15T21:03:52.900 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T21:03:52.900 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2024-09-15T21:03:53.020 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=171798691879 2024-09-15T21:03:53.020 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 171798691879 2024-09-15T21:03:53.020 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836569 1-42949673039 2-64424509514 3-313532612619 4-107374182458 5-356482285570 6-150323855407 7-171798691879' 2024-09-15T21:03:53.020 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T21:03:53.020 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2024-09-15T21:03:53.123 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=193273528355 2024-09-15T21:03:53.123 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 193273528355 2024-09-15T21:03:53.123 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836569 1-42949673039 2-64424509514 3-313532612619 4-107374182458 5-356482285570 6-150323855407 7-171798691879 8-193273528355' 2024-09-15T21:03:53.123 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T21:03:53.123 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2024-09-15T21:03:53.226 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=335007449094 2024-09-15T21:03:53.226 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 335007449094 2024-09-15T21:03:53.227 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836569 1-42949673039 2-64424509514 3-313532612619 4-107374182458 5-356482285570 6-150323855407 7-171798691879 8-193273528355 9-335007449094' 2024-09-15T21:03:53.227 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T21:03:53.227 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836569 2024-09-15T21:03:53.227 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T21:03:53.228 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-15T21:03:53.228 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836569 2024-09-15T21:03:53.228 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T21:03:53.229 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836569 2024-09-15T21:03:53.229 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836569' 2024-09-15T21:03:53.229 INFO:tasks.workunit.client.0.smithi136.stderr:waiting osd.0 seq 21474836569 2024-09-15T21:03:53.230 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T21:03:53.506 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836569 -lt 21474836569 2024-09-15T21:03:53.506 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T21:03:53.506 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-42949673039 2024-09-15T21:03:53.506 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T21:03:53.507 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-15T21:03:53.508 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-42949673039 2024-09-15T21:03:53.508 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T21:03:53.509 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=42949673039 2024-09-15T21:03:53.509 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 42949673039' 2024-09-15T21:03:53.509 INFO:tasks.workunit.client.0.smithi136.stderr:waiting osd.1 seq 42949673039 2024-09-15T21:03:53.509 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-15T21:03:53.787 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 42949673039 -lt 42949673039 2024-09-15T21:03:53.787 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T21:03:53.788 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-64424509514 2024-09-15T21:03:53.788 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T21:03:53.789 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-15T21:03:53.789 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-64424509514 2024-09-15T21:03:53.789 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T21:03:53.790 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=64424509514 2024-09-15T21:03:53.790 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 64424509514' 2024-09-15T21:03:53.790 INFO:tasks.workunit.client.0.smithi136.stderr:waiting osd.2 seq 64424509514 2024-09-15T21:03:53.791 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-15T21:03:54.068 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 64424509514 -lt 64424509514 2024-09-15T21:03:54.068 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T21:03:54.068 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 3-313532612619 2024-09-15T21:03:54.068 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T21:03:54.069 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=3 2024-09-15T21:03:54.070 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 3-313532612619 2024-09-15T21:03:54.070 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T21:03:54.071 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=313532612619 2024-09-15T21:03:54.071 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.3 seq 313532612619' 2024-09-15T21:03:54.071 INFO:tasks.workunit.client.0.smithi136.stderr:waiting osd.3 seq 313532612619 2024-09-15T21:03:54.071 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-15T21:03:54.350 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 313532612619 -lt 313532612619 2024-09-15T21:03:54.350 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T21:03:54.350 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 4-107374182458 2024-09-15T21:03:54.350 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T21:03:54.351 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=4 2024-09-15T21:03:54.352 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 4-107374182458 2024-09-15T21:03:54.352 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T21:03:54.353 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=107374182458 2024-09-15T21:03:54.353 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.4 seq 107374182458' 2024-09-15T21:03:54.353 INFO:tasks.workunit.client.0.smithi136.stderr:waiting osd.4 seq 107374182458 2024-09-15T21:03:54.354 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 4 2024-09-15T21:03:54.628 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 107374182458 -lt 107374182458 2024-09-15T21:03:54.628 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T21:03:54.628 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 5-356482285570 2024-09-15T21:03:54.628 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T21:03:54.629 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=5 2024-09-15T21:03:54.630 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 5-356482285570 2024-09-15T21:03:54.630 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T21:03:54.631 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=356482285570 2024-09-15T21:03:54.631 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.5 seq 356482285570' 2024-09-15T21:03:54.631 INFO:tasks.workunit.client.0.smithi136.stderr:waiting osd.5 seq 356482285570 2024-09-15T21:03:54.631 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 5 2024-09-15T21:03:54.932 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 356482285570 -lt 356482285570 2024-09-15T21:03:54.932 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T21:03:54.932 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 6-150323855407 2024-09-15T21:03:54.932 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T21:03:54.933 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=6 2024-09-15T21:03:54.934 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 6-150323855407 2024-09-15T21:03:54.934 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T21:03:54.935 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=150323855407 2024-09-15T21:03:54.935 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.6 seq 150323855407' 2024-09-15T21:03:54.935 INFO:tasks.workunit.client.0.smithi136.stderr:waiting osd.6 seq 150323855407 2024-09-15T21:03:54.935 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 6 2024-09-15T21:03:55.213 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 150323855407 -lt 150323855407 2024-09-15T21:03:55.214 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T21:03:55.214 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 7-171798691879 2024-09-15T21:03:55.214 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T21:03:55.215 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=7 2024-09-15T21:03:55.215 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 7-171798691879 2024-09-15T21:03:55.216 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T21:03:55.216 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=171798691879 2024-09-15T21:03:55.217 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.7 seq 171798691879' 2024-09-15T21:03:55.217 INFO:tasks.workunit.client.0.smithi136.stderr:waiting osd.7 seq 171798691879 2024-09-15T21:03:55.217 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 7 2024-09-15T21:03:55.533 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 171798691880 -lt 171798691879 2024-09-15T21:03:55.534 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T21:03:55.534 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 8-193273528355 2024-09-15T21:03:55.534 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T21:03:55.534 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=8 2024-09-15T21:03:55.534 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 8-193273528355 2024-09-15T21:03:55.534 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T21:03:55.534 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=193273528355 2024-09-15T21:03:55.535 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.8 seq 193273528355' 2024-09-15T21:03:55.535 INFO:tasks.workunit.client.0.smithi136.stderr:waiting osd.8 seq 193273528355 2024-09-15T21:03:55.535 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 8 2024-09-15T21:03:55.812 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 193273528355 -lt 193273528355 2024-09-15T21:03:55.812 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T21:03:55.813 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 9-335007449094 2024-09-15T21:03:55.813 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T21:03:55.814 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=9 2024-09-15T21:03:55.814 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 9-335007449094 2024-09-15T21:03:55.814 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T21:03:55.816 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=335007449094 2024-09-15T21:03:55.816 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.9 seq 335007449094' 2024-09-15T21:03:55.816 INFO:tasks.workunit.client.0.smithi136.stderr:waiting osd.9 seq 335007449094 2024-09-15T21:03:55.816 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 9 2024-09-15T21:03:56.094 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 335007449095 -lt 335007449094 2024-09-15T21:03:56.094 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-15T21:03:56.094 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T21:03:56.095 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T21:03:56.439 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 5 == 0 2024-09-15T21:03:56.503 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-15T21:03:56.504 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-15T21:03:56.504 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-15T21:03:56.504 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T21:03:56.504 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T21:03:56.504 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T21:03:56.504 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T21:03:56.714 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=5 2024-09-15T21:03:56.715 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-15T21:03:56.715 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T21:03:56.715 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T21:03:57.065 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 = 5 2024-09-15T21:03:57.065 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-15T21:03:57.065 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-15T21:03:57.065 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:237: corrupt_and_repair_one: get_pg ecpool SOMETHING 2024-09-15T21:03:57.065 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1084: get_pg: local poolname=ecpool 2024-09-15T21:03:57.065 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1085: get_pg: local objectname=SOMETHING 2024-09-15T21:03:57.065 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1087: get_pg: ceph --format json osd map ecpool SOMETHING 2024-09-15T21:03:57.065 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1087: get_pg: jq -r .pgid 2024-09-15T21:03:57.350 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:237: corrupt_and_repair_one: local pg=2.0 2024-09-15T21:03:57.350 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:238: corrupt_and_repair_one: repair 2.0 2024-09-15T21:03:57.350 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1846: repair: local pgid=2.0 2024-09-15T21:03:57.350 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1847: repair: get_last_scrub_stamp 2.0 2024-09-15T21:03:57.350 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2024-09-15T21:03:57.350 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-15T21:03:57.350 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-15T21:03:57.350 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-09-15T21:03:57.627 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1847: repair: local last_scrub=2024-09-15T21:03:24.334290+0000 2024-09-15T21:03:57.627 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1848: repair: ceph pg repair 2.0 2024-09-15T21:03:57.891 INFO:tasks.workunit.client.0.smithi136.stderr:instructing pg 2.0s0 on osd.3 to repair 2024-09-15T21:03:57.904 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1849: repair: wait_for_scrub 2.0 2024-09-15T21:03:24.334290+0000 2024-09-15T21:03:57.904 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: local pgid=2.0 2024-09-15T21:03:57.904 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2018: wait_for_scrub: local last_scrub=2024-09-15T21:03:24.334290+0000 2024-09-15T21:03:57.904 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2019: wait_for_scrub: local sname=last_scrub_stamp 2024-09-15T21:03:57.904 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i=0 )) 2024-09-15T21:03:57.904 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-15T21:03:57.904 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-09-15T21:03:57.904 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2024-09-15T21:03:57.904 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-15T21:03:57.904 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-15T21:03:57.904 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-09-15T21:03:58.178 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-15T21:03:24.334290+0000 '>' 2024-09-15T21:03:24.334290+0000 2024-09-15T21:03:58.178 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-15T21:03:59.179 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-15T21:03:59.179 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-15T21:03:59.180 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-09-15T21:03:59.181 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2024-09-15T21:03:59.181 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-15T21:03:59.181 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-15T21:03:59.181 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-09-15T21:03:59.467 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-15T21:03:24.334290+0000 '>' 2024-09-15T21:03:24.334290+0000 2024-09-15T21:03:59.467 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-15T21:04:00.468 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-15T21:04:00.468 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-15T21:04:00.469 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-09-15T21:04:00.469 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2024-09-15T21:04:00.469 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-15T21:04:00.469 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-15T21:04:00.469 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-09-15T21:04:00.749 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-15T21:03:24.334290+0000 '>' 2024-09-15T21:03:24.334290+0000 2024-09-15T21:04:00.750 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-15T21:04:01.751 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-15T21:04:01.751 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-15T21:04:01.752 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-09-15T21:04:01.752 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2024-09-15T21:04:01.752 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-15T21:04:01.752 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-15T21:04:01.752 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-09-15T21:04:02.038 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-15T21:03:24.334290+0000 '>' 2024-09-15T21:03:24.334290+0000 2024-09-15T21:04:02.038 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-15T21:04:03.039 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-15T21:04:03.039 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-15T21:04:03.039 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-09-15T21:04:03.040 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2024-09-15T21:04:03.040 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-15T21:04:03.040 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-15T21:04:03.040 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-09-15T21:04:03.325 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-15T21:03:58.250680+0000 '>' 2024-09-15T21:03:24.334290+0000 2024-09-15T21:04:03.326 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2023: wait_for_scrub: return 0 2024-09-15T21:04:03.326 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1850: repair: sleep 2 2024-09-15T21:04:05.327 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:242: corrupt_and_repair_one: objectstore_tool td/osd-scrub-repair 9 SOMETHING list-attrs 2024-09-15T21:04:05.327 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2024-09-15T21:04:05.328 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2024-09-15T21:04:05.328 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=9 2024-09-15T21:04:05.328 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2024-09-15T21:04:05.328 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 9 SOMETHING list-attrs 2024-09-15T21:04:05.328 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-09-15T21:04:05.328 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2024-09-15T21:04:05.328 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=9 2024-09-15T21:04:05.328 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2024-09-15T21:04:05.328 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.9 2024-09-15T21:04:05.328 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-15T21:04:05.328 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-15T21:04:05.329 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-15T21:04:05.329 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-15T21:04:05.329 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-15T21:04:05.637 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-15T21:04:05.638 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 9 SOMETHING list-attrs 2024-09-15T21:04:05.638 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-15T21:04:05.638 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-15T21:04:05.638 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=9 2024-09-15T21:04:05.638 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-15T21:04:05.638 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/9 2024-09-15T21:04:05.638 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/9 SOMETHING list-attrs 2024-09-15T21:04:06.288 INFO:tasks.workunit.client.0.smithi136.stdout:_ 2024-09-15T21:04:06.289 INFO:tasks.workunit.client.0.smithi136.stdout:hinfo_key 2024-09-15T21:04:06.289 INFO:tasks.workunit.client.0.smithi136.stdout:snapset 2024-09-15T21:04:06.556 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 9 2024-09-15T21:04:06.556 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2024-09-15T21:04:06.556 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-15T21:04:06.556 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=9 2024-09-15T21:04:06.556 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-15T21:04:06.556 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/9 2024-09-15T21:04:06.556 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=ce4892c7-6c6d-4b4f-8e6f-07a3e11133e5 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-15T21:04:06.556 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T21:04:06.556 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T21:04:06.556 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T21:04:06.556 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/9' 2024-09-15T21:04:06.557 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/9/journal' 2024-09-15T21:04:06.557 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-15T21:04:06.557 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-15T21:04:06.557 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-15T21:04:06.557 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-15T21:04:06.557 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-15T21:04:06.557 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-15T21:04:06.557 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-15T21:04:06.557 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T21:04:06.557 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52602 2024-09-15T21:04:06.557 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.52602/$cluster-$name.asok' 2024-09-15T21:04:06.558 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.52602/$cluster-$name.asok' 2024-09-15T21:04:06.558 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-15T21:04:06.558 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-15T21:04:06.558 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-15T21:04:06.558 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T21:04:06.558 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T21:04:06.558 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T21:04:06.558 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T21:04:06.558 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-15T21:04:06.558 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-15T21:04:06.558 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/9 2024-09-15T21:04:06.559 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.9 2024-09-15T21:04:06.559 INFO:tasks.workunit.client.0.smithi136.stderr:start osd.9 2024-09-15T21:04:06.559 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 9 --fsid=ce4892c7-6c6d-4b4f-8e6f-07a3e11133e5 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/9 --osd-journal=td/osd-scrub-repair/9/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.52602/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T21:04:06.559 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/9/whoami 2024-09-15T21:04:06.559 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 9 = 9 ']' 2024-09-15T21:04:06.560 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-15T21:04:06.560 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-15T21:04:06.560 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-15T21:04:06.587 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:04:06.583+0000 7fd4c04f1640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:04:06.625 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:04:06.621+0000 7fd4c04f1640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:04:06.626 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:04:06.623+0000 7fd4c04f1640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:04:06.850 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 9 2024-09-15T21:04:06.850 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-15T21:04:06.850 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=9 2024-09-15T21:04:06.850 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-15T21:04:06.850 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-15T21:04:06.850 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:04:06.850 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-15T21:04:06.850 INFO:tasks.workunit.client.0.smithi136.stderr:0 2024-09-15T21:04:06.850 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:04:06.850 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.9 up' 2024-09-15T21:04:07.153 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:04:07.733 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:04:07.729+0000 7fd4c04f1640 -1 Falling back to public interface 2024-09-15T21:04:08.154 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:04:08.154 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:04:08.155 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-15T21:04:08.155 INFO:tasks.workunit.client.0.smithi136.stderr:1 2024-09-15T21:04:08.155 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:04:08.155 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.9 up' 2024-09-15T21:04:08.442 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:04:09.444 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:04:09.444 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:04:09.444 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-15T21:04:09.445 INFO:tasks.workunit.client.0.smithi136.stderr:2 2024-09-15T21:04:09.445 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:04:09.445 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.9 up' 2024-09-15T21:04:09.730 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:04:10.485 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:04:10.481+0000 7fd4c04f1640 -1 osd.9 84 log_to_monitors true 2024-09-15T21:04:10.731 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:04:10.732 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:04:10.732 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-15T21:04:10.732 INFO:tasks.workunit.client.0.smithi136.stderr:3 2024-09-15T21:04:10.732 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:04:10.732 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.9 up' 2024-09-15T21:04:11.086 INFO:tasks.workunit.client.0.smithi136.stderr:osd.9 up in weight 1 up_from 88 up_thru 0 down_at 85 last_clean_interval [78,84) [v2:127.0.0.1:6874/3564149753,v1:127.0.0.1:6875/3564149753] [v2:127.0.0.1:6876/3564149753,v1:127.0.0.1:6877/3564149753] exists,up ccfb463f-a1cc-4f3c-a7db-ae126efbeaa5 2024-09-15T21:04:11.086 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-15T21:04:11.086 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-15T21:04:11.086 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-15T21:04:11.086 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2024-09-15T21:04:11.086 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-15T21:04:11.086 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-15T21:04:11.086 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-15T21:04:11.086 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-15T21:04:11.087 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-15T21:04:11.087 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-15T21:04:11.087 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-15T21:04:11.087 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-15T21:04:11.087 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-15T21:04:11.177 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-15T21:04:11.178 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-15T21:04:11.178 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-15T21:04:11.178 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-15T21:04:11.178 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-15T21:04:11.178 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-15T21:04:11.458 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-15T21:04:11.458 INFO:tasks.workunit.client.0.smithi136.stderr:1 2024-09-15T21:04:11.458 INFO:tasks.workunit.client.0.smithi136.stderr:2 2024-09-15T21:04:11.458 INFO:tasks.workunit.client.0.smithi136.stderr:3 2024-09-15T21:04:11.458 INFO:tasks.workunit.client.0.smithi136.stderr:4 2024-09-15T21:04:11.458 INFO:tasks.workunit.client.0.smithi136.stderr:5 2024-09-15T21:04:11.459 INFO:tasks.workunit.client.0.smithi136.stderr:6 2024-09-15T21:04:11.459 INFO:tasks.workunit.client.0.smithi136.stderr:7 2024-09-15T21:04:11.459 INFO:tasks.workunit.client.0.smithi136.stderr:8 2024-09-15T21:04:11.459 INFO:tasks.workunit.client.0.smithi136.stderr:9' 2024-09-15T21:04:11.459 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-15T21:04:11.459 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T21:04:11.460 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-15T21:04:11.560 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836573 2024-09-15T21:04:11.561 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836573 2024-09-15T21:04:11.561 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836573' 2024-09-15T21:04:11.561 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T21:04:11.561 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-15T21:04:11.664 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=42949673044 2024-09-15T21:04:11.664 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 42949673044 2024-09-15T21:04:11.665 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836573 1-42949673044' 2024-09-15T21:04:11.665 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T21:04:11.665 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-15T21:04:11.768 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=64424509518 2024-09-15T21:04:11.768 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 64424509518 2024-09-15T21:04:11.768 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836573 1-42949673044 2-64424509518' 2024-09-15T21:04:11.768 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T21:04:11.768 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-15T21:04:11.872 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=313532612624 2024-09-15T21:04:11.872 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 313532612624 2024-09-15T21:04:11.872 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836573 1-42949673044 2-64424509518 3-313532612624' 2024-09-15T21:04:11.872 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T21:04:11.872 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-09-15T21:04:11.976 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=107374182463 2024-09-15T21:04:11.976 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 107374182463 2024-09-15T21:04:11.977 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836573 1-42949673044 2-64424509518 3-313532612624 4-107374182463' 2024-09-15T21:04:11.977 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T21:04:11.977 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-09-15T21:04:12.120 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=356482285575 2024-09-15T21:04:12.121 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 356482285575 2024-09-15T21:04:12.121 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836573 1-42949673044 2-64424509518 3-313532612624 4-107374182463 5-356482285575' 2024-09-15T21:04:12.121 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T21:04:12.121 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-09-15T21:04:12.226 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=150323855412 2024-09-15T21:04:12.226 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 150323855412 2024-09-15T21:04:12.226 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836573 1-42949673044 2-64424509518 3-313532612624 4-107374182463 5-356482285575 6-150323855412' 2024-09-15T21:04:12.226 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T21:04:12.226 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2024-09-15T21:04:12.375 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=171798691884 2024-09-15T21:04:12.375 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 171798691884 2024-09-15T21:04:12.375 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836573 1-42949673044 2-64424509518 3-313532612624 4-107374182463 5-356482285575 6-150323855412 7-171798691884' 2024-09-15T21:04:12.375 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T21:04:12.375 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2024-09-15T21:04:12.533 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=193273528360 2024-09-15T21:04:12.533 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 193273528360 2024-09-15T21:04:12.533 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836573 1-42949673044 2-64424509518 3-313532612624 4-107374182463 5-356482285575 6-150323855412 7-171798691884 8-193273528360' 2024-09-15T21:04:12.533 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T21:04:12.533 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2024-09-15T21:04:12.637 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=377957122050 2024-09-15T21:04:12.637 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 377957122050 2024-09-15T21:04:12.637 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836573 1-42949673044 2-64424509518 3-313532612624 4-107374182463 5-356482285575 6-150323855412 7-171798691884 8-193273528360 9-377957122050' 2024-09-15T21:04:12.638 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T21:04:12.638 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836573 2024-09-15T21:04:12.638 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T21:04:12.638 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-15T21:04:12.639 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836573 2024-09-15T21:04:12.639 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T21:04:12.640 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836573 2024-09-15T21:04:12.640 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836573' 2024-09-15T21:04:12.640 INFO:tasks.workunit.client.0.smithi136.stderr:waiting osd.0 seq 21474836573 2024-09-15T21:04:12.641 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T21:04:13.214 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836572 -lt 21474836573 2024-09-15T21:04:13.215 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-15T21:04:14.216 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-15T21:04:14.216 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T21:04:14.504 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836574 -lt 21474836573 2024-09-15T21:04:14.504 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T21:04:14.505 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-42949673044 2024-09-15T21:04:14.505 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T21:04:14.506 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-15T21:04:14.506 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-42949673044 2024-09-15T21:04:14.506 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T21:04:14.507 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=42949673044 2024-09-15T21:04:14.507 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 42949673044' 2024-09-15T21:04:14.507 INFO:tasks.workunit.client.0.smithi136.stderr:waiting osd.1 seq 42949673044 2024-09-15T21:04:14.508 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-15T21:04:14.791 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 42949673044 -lt 42949673044 2024-09-15T21:04:14.791 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T21:04:14.791 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-64424509518 2024-09-15T21:04:14.791 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T21:04:14.792 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-15T21:04:14.793 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-64424509518 2024-09-15T21:04:14.793 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T21:04:14.794 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=64424509518 2024-09-15T21:04:14.794 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 64424509518' 2024-09-15T21:04:14.794 INFO:tasks.workunit.client.0.smithi136.stderr:waiting osd.2 seq 64424509518 2024-09-15T21:04:14.794 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-15T21:04:15.082 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 64424509519 -lt 64424509518 2024-09-15T21:04:15.082 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T21:04:15.082 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 3-313532612624 2024-09-15T21:04:15.082 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T21:04:15.083 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=3 2024-09-15T21:04:15.084 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 3-313532612624 2024-09-15T21:04:15.084 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T21:04:15.085 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=313532612624 2024-09-15T21:04:15.085 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.3 seq 313532612624' 2024-09-15T21:04:15.085 INFO:tasks.workunit.client.0.smithi136.stderr:waiting osd.3 seq 313532612624 2024-09-15T21:04:15.085 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-15T21:04:15.364 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 313532612624 -lt 313532612624 2024-09-15T21:04:15.364 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T21:04:15.364 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 4-107374182463 2024-09-15T21:04:15.364 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T21:04:15.365 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=4 2024-09-15T21:04:15.366 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 4-107374182463 2024-09-15T21:04:15.366 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T21:04:15.367 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=107374182463 2024-09-15T21:04:15.367 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.4 seq 107374182463' 2024-09-15T21:04:15.367 INFO:tasks.workunit.client.0.smithi136.stderr:waiting osd.4 seq 107374182463 2024-09-15T21:04:15.367 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 4 2024-09-15T21:04:15.647 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 107374182464 -lt 107374182463 2024-09-15T21:04:15.647 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T21:04:15.648 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 5-356482285575 2024-09-15T21:04:15.648 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T21:04:15.648 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=5 2024-09-15T21:04:15.649 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 5-356482285575 2024-09-15T21:04:15.649 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T21:04:15.650 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=356482285575 2024-09-15T21:04:15.650 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.5 seq 356482285575' 2024-09-15T21:04:15.650 INFO:tasks.workunit.client.0.smithi136.stderr:waiting osd.5 seq 356482285575 2024-09-15T21:04:15.650 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 5 2024-09-15T21:04:15.931 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 356482285575 -lt 356482285575 2024-09-15T21:04:15.932 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T21:04:15.932 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 6-150323855412 2024-09-15T21:04:15.932 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T21:04:15.932 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=6 2024-09-15T21:04:15.933 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 6-150323855412 2024-09-15T21:04:15.933 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T21:04:15.934 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=150323855412 2024-09-15T21:04:15.934 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.6 seq 150323855412' 2024-09-15T21:04:15.935 INFO:tasks.workunit.client.0.smithi136.stderr:waiting osd.6 seq 150323855412 2024-09-15T21:04:15.935 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 6 2024-09-15T21:04:16.209 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 150323855412 -lt 150323855412 2024-09-15T21:04:16.209 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T21:04:16.209 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 7-171798691884 2024-09-15T21:04:16.209 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T21:04:16.210 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=7 2024-09-15T21:04:16.211 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 7-171798691884 2024-09-15T21:04:16.211 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T21:04:16.212 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=171798691884 2024-09-15T21:04:16.212 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.7 seq 171798691884' 2024-09-15T21:04:16.212 INFO:tasks.workunit.client.0.smithi136.stderr:waiting osd.7 seq 171798691884 2024-09-15T21:04:16.212 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 7 2024-09-15T21:04:16.490 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 171798691885 -lt 171798691884 2024-09-15T21:04:16.490 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T21:04:16.490 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 8-193273528360 2024-09-15T21:04:16.490 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T21:04:16.491 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=8 2024-09-15T21:04:16.492 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 8-193273528360 2024-09-15T21:04:16.492 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T21:04:16.493 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=193273528360 2024-09-15T21:04:16.493 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.8 seq 193273528360' 2024-09-15T21:04:16.493 INFO:tasks.workunit.client.0.smithi136.stderr:waiting osd.8 seq 193273528360 2024-09-15T21:04:16.493 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 8 2024-09-15T21:04:16.769 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 193273528360 -lt 193273528360 2024-09-15T21:04:16.769 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T21:04:16.770 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 9-377957122050 2024-09-15T21:04:16.770 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T21:04:16.770 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=9 2024-09-15T21:04:16.771 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 9-377957122050 2024-09-15T21:04:16.771 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T21:04:16.772 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=377957122050 2024-09-15T21:04:16.772 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.9 seq 377957122050' 2024-09-15T21:04:16.772 INFO:tasks.workunit.client.0.smithi136.stderr:waiting osd.9 seq 377957122050 2024-09-15T21:04:16.772 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 9 2024-09-15T21:04:17.082 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 377957122051 -lt 377957122050 2024-09-15T21:04:17.082 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-15T21:04:17.082 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T21:04:17.083 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T21:04:17.431 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 5 == 0 2024-09-15T21:04:17.432 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-15T21:04:17.432 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-15T21:04:17.432 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-15T21:04:17.432 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T21:04:17.432 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T21:04:17.432 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T21:04:17.432 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T21:04:17.705 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=5 2024-09-15T21:04:17.706 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-15T21:04:17.706 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T21:04:17.706 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T21:04:18.051 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 = 5 2024-09-15T21:04:18.052 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-15T21:04:18.052 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-15T21:04:18.052 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:243: corrupt_and_repair_one: rados --pool ecpool get SOMETHING td/osd-scrub-repair/COPY 2024-09-15T21:04:18.082 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:244: corrupt_and_repair_one: diff td/osd-scrub-repair/ORIGINAL td/osd-scrub-repair/COPY 2024-09-15T21:04:18.084 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:262: corrupt_and_repair_erasure_coded: corrupt_and_repair_two td/osd-scrub-repair ecpool 5 9 2024-09-15T21:04:18.084 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:185: corrupt_and_repair_two: local dir=td/osd-scrub-repair 2024-09-15T21:04:18.084 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:186: corrupt_and_repair_two: local poolname=ecpool 2024-09-15T21:04:18.084 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:187: corrupt_and_repair_two: local first=5 2024-09-15T21:04:18.085 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:188: corrupt_and_repair_two: local second=9 2024-09-15T21:04:18.085 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:193: corrupt_and_repair_two: pids= 2024-09-15T21:04:18.085 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:194: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 5 SOMETHING remove 2024-09-15T21:04:18.085 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2135: run_in_background: local pid_variable=pids 2024-09-15T21:04:18.085 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2136: run_in_background: shift 2024-09-15T21:04:18.085 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2140: run_in_background: eval 'pids+=" 211855"' 2024-09-15T21:04:18.085 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2140: run_in_background: pids+=' 211855' 2024-09-15T21:04:18.085 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:195: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 9 SOMETHING remove 2024-09-15T21:04:18.085 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2135: run_in_background: local pid_variable=pids 2024-09-15T21:04:18.085 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2136: run_in_background: shift 2024-09-15T21:04:18.085 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2140: run_in_background: eval 'pids+=" 211857"' 2024-09-15T21:04:18.085 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: run_in_background: objectstore_tool td/osd-scrub-repair 5 SOMETHING remove 2024-09-15T21:04:18.085 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2140: run_in_background: pids+=' 211857' 2024-09-15T21:04:18.085 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:196: corrupt_and_repair_two: wait_background pids 2024-09-15T21:04:18.086 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: wait_background: pids=' 211855 211857' 2024-09-15T21:04:18.086 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: wait_background: return_code=0 2024-09-15T21:04:18.086 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: wait_background: for pid in $pids 2024-09-15T21:04:18.086 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: run_in_background: sed 's/^/211858: /' 2024-09-15T21:04:18.086 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2175: wait_background: wait 211855 2024-09-15T21:04:18.086 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: run_in_background: objectstore_tool td/osd-scrub-repair 9 SOMETHING remove 2024-09-15T21:04:18.086 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: run_in_background: sed 's/^/211860: /' 2024-09-15T21:04:21.673 INFO:tasks.workunit.client.0.smithi136.stderr:211860: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2024-09-15T21:04:21.673 INFO:tasks.workunit.client.0.smithi136.stderr:211860: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2024-09-15T21:04:21.673 INFO:tasks.workunit.client.0.smithi136.stderr:211860: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=9 2024-09-15T21:04:21.674 INFO:tasks.workunit.client.0.smithi136.stderr:211860: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2024-09-15T21:04:21.674 INFO:tasks.workunit.client.0.smithi136.stderr:211860: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 9 SOMETHING remove 2024-09-15T21:04:21.674 INFO:tasks.workunit.client.0.smithi136.stderr:211860: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-09-15T21:04:21.674 INFO:tasks.workunit.client.0.smithi136.stderr:211860: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2024-09-15T21:04:21.674 INFO:tasks.workunit.client.0.smithi136.stderr:211860: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=9 2024-09-15T21:04:21.674 INFO:tasks.workunit.client.0.smithi136.stderr:211860: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2024-09-15T21:04:21.674 INFO:tasks.workunit.client.0.smithi136.stderr:211860: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.9 2024-09-15T21:04:21.674 INFO:tasks.workunit.client.0.smithi136.stderr:211860: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-15T21:04:21.674 INFO:tasks.workunit.client.0.smithi136.stderr:211860: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-15T21:04:21.674 INFO:tasks.workunit.client.0.smithi136.stderr:211860: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-15T21:04:21.674 INFO:tasks.workunit.client.0.smithi136.stderr:211860: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-15T21:04:21.674 INFO:tasks.workunit.client.0.smithi136.stderr:211860: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-15T21:04:21.674 INFO:tasks.workunit.client.0.smithi136.stderr:211860: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-15T21:04:21.674 INFO:tasks.workunit.client.0.smithi136.stderr:211860: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 9 SOMETHING remove 2024-09-15T21:04:21.675 INFO:tasks.workunit.client.0.smithi136.stderr:211860: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-15T21:04:21.675 INFO:tasks.workunit.client.0.smithi136.stderr:211860: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-15T21:04:21.675 INFO:tasks.workunit.client.0.smithi136.stderr:211860: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=9 2024-09-15T21:04:21.675 INFO:tasks.workunit.client.0.smithi136.stderr:211860: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-15T21:04:21.675 INFO:tasks.workunit.client.0.smithi136.stderr:211860: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/9 2024-09-15T21:04:21.675 INFO:tasks.workunit.client.0.smithi136.stderr:211860: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/9 SOMETHING remove 2024-09-15T21:04:21.675 INFO:tasks.workunit.client.0.smithi136.stderr:211860: remove 2#2:eb822e21:::SOMETHING:head# 2024-09-15T21:04:21.675 INFO:tasks.workunit.client.0.smithi136.stderr:211860: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 9 2024-09-15T21:04:21.675 INFO:tasks.workunit.client.0.smithi136.stderr:211860: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2024-09-15T21:04:21.675 INFO:tasks.workunit.client.0.smithi136.stderr:211860: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-15T21:04:21.675 INFO:tasks.workunit.client.0.smithi136.stderr:211860: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=9 2024-09-15T21:04:21.675 INFO:tasks.workunit.client.0.smithi136.stderr:211860: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-15T21:04:21.675 INFO:tasks.workunit.client.0.smithi136.stderr:211860: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/9 2024-09-15T21:04:21.675 INFO:tasks.workunit.client.0.smithi136.stderr:211860: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=ce4892c7-6c6d-4b4f-8e6f-07a3e11133e5 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-15T21:04:21.676 INFO:tasks.workunit.client.0.smithi136.stderr:211860: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T21:04:21.677 INFO:tasks.workunit.client.0.smithi136.stderr:211860: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T21:04:21.677 INFO:tasks.workunit.client.0.smithi136.stderr:211860: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T21:04:21.677 INFO:tasks.workunit.client.0.smithi136.stderr:211860: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/9' 2024-09-15T21:04:21.677 INFO:tasks.workunit.client.0.smithi136.stderr:211860: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/9/journal' 2024-09-15T21:04:21.677 INFO:tasks.workunit.client.0.smithi136.stderr:211860: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-15T21:04:21.677 INFO:tasks.workunit.client.0.smithi136.stderr:211860: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-15T21:04:21.677 INFO:tasks.workunit.client.0.smithi136.stderr:211860: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-15T21:04:21.677 INFO:tasks.workunit.client.0.smithi136.stderr:211860: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-15T21:04:21.677 INFO:tasks.workunit.client.0.smithi136.stderr:211860: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-15T21:04:21.677 INFO:tasks.workunit.client.0.smithi136.stderr:211860: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-15T21:04:21.677 INFO:tasks.workunit.client.0.smithi136.stderr:211860: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-15T21:04:21.677 INFO:tasks.workunit.client.0.smithi136.stderr:211860: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T21:04:21.678 INFO:tasks.workunit.client.0.smithi136.stderr:211860: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52602 2024-09-15T21:04:21.678 INFO:tasks.workunit.client.0.smithi136.stderr:211860: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.52602/$cluster-$name.asok' 2024-09-15T21:04:21.678 INFO:tasks.workunit.client.0.smithi136.stderr:211860: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.52602/$cluster-$name.asok' 2024-09-15T21:04:21.678 INFO:tasks.workunit.client.0.smithi136.stderr:211860: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-15T21:04:21.678 INFO:tasks.workunit.client.0.smithi136.stderr:211860: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-15T21:04:21.678 INFO:tasks.workunit.client.0.smithi136.stderr:211860: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-15T21:04:21.678 INFO:tasks.workunit.client.0.smithi136.stderr:211860: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T21:04:21.678 INFO:tasks.workunit.client.0.smithi136.stderr:211860: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T21:04:21.678 INFO:tasks.workunit.client.0.smithi136.stderr:211860: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T21:04:21.678 INFO:tasks.workunit.client.0.smithi136.stderr:211860: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T21:04:21.678 INFO:tasks.workunit.client.0.smithi136.stderr:211860: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-15T21:04:21.678 INFO:tasks.workunit.client.0.smithi136.stderr:211860: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-15T21:04:21.678 INFO:tasks.workunit.client.0.smithi136.stderr:211860: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/9 2024-09-15T21:04:21.678 INFO:tasks.workunit.client.0.smithi136.stderr:211860: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.9 2024-09-15T21:04:21.679 INFO:tasks.workunit.client.0.smithi136.stderr:211860: start osd.9 2024-09-15T21:04:21.679 INFO:tasks.workunit.client.0.smithi136.stderr:211860: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 9 --fsid=ce4892c7-6c6d-4b4f-8e6f-07a3e11133e5 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/9 --osd-journal=td/osd-scrub-repair/9/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.52602/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T21:04:21.710 INFO:tasks.workunit.client.0.smithi136.stderr:2211858: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2024-09-15T21:04:21.710 INFO:tasks.workunit.client.0.smithi136.stderr:211858: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2024-09-15T21:04:21.710 INFO:tasks.workunit.client.0.smithi136.stderr:211858: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=5 2024-09-15T21:04:21.710 INFO:tasks.workunit.client.0.smithi136.stderr:211858: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2024-09-15T21:04:21.711 INFO:tasks.workunit.client.0.smithi136.stderr:211858: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 5 SOMETHING remove 2024-09-15T21:04:21.711 INFO:tasks.workunit.client.0.smithi136.stderr:211858: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-09-15T21:04:21.711 INFO:tasks.workunit.client.0.smithi136.stderr:211858: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2024-09-15T21:04:21.711 INFO:tasks.workunit.client.0.smithi136.stderr:211858: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=5 2024-09-15T21:04:21.711 INFO:tasks.workunit.client.0.smithi136.stderr:211858: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2024-09-15T21:04:21.711 INFO:tasks.workunit.client.0.smithi136.stderr:211858: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.5 2024-09-15T21:04:21.711 INFO:tasks.workunit.client.0.smithi136.stderr:211858: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-15T21:04:21.711 INFO:tasks.workunit.client.0.smithi136.stderr:211858: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-15T21:04:21.711 INFO:tasks.workunit.client.0.smithi136.stderr:211858: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-15T21:04:21.711 INFO:tasks.workunit.client.0.smithi136.stderr:211858: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-15T21:04:21.711 INFO:tasks.workunit.client.0.smithi136.stderr:211858: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-15T21:04:21.711 INFO:tasks.workunit.client.0.smithi136.stderr:211858: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-15T21:04:21.712 INFO:tasks.workunit.client.0.smithi136.stderr:211858: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 5 SOMETHING remove 2024-09-15T21:04:21.712 INFO:tasks.workunit.client.0.smithi136.stderr:211858: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-15T21:04:21.712 INFO:tasks.workunit.client.0.smithi136.stderr:211858: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-15T21:04:21.712 INFO:tasks.workunit.client.0.smithi136.stderr:211858: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=5 2024-09-15T21:04:21.712 INFO:tasks.workunit.client.0.smithi136.stderr:211858: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-15T21:04:21.712 INFO:tasks.workunit.client.0.smithi136.stderr:211858: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/5 2024-09-15T21:04:21.712 INFO:tasks.workunit.client.0.smithi136.stderr:211858: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/5 SOMETHING remove 2024-09-15T21:04:21.712 INFO:tasks.workunit.client.0.smithi136.stderr:211858: remove 1#2:eb822e21:::SOMETHING:head# 2024-09-15T21:04:21.712 INFO:tasks.workunit.client.0.smithi136.stderr:211858: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 5 2024-09-15T21:04:21.712 INFO:tasks.workunit.client.0.smithi136.stderr:211858: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2024-09-15T21:04:21.712 INFO:tasks.workunit.client.0.smithi136.stderr:211858: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-15T21:04:21.712 INFO:tasks.workunit.client.0.smithi136.stderr:211858: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=5 2024-09-15T21:04:21.712 INFO:tasks.workunit.client.0.smithi136.stderr:211858: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-15T21:04:21.713 INFO:tasks.workunit.client.0.smithi136.stderr:211858: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/5 2024-09-15T21:04:21.713 INFO:tasks.workunit.client.0.smithi136.stderr:211858: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=ce4892c7-6c6d-4b4f-8e6f-07a3e11133e5 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-15T21:04:21.713 INFO:tasks.workunit.client.0.smithi136.stderr:211858: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T21:04:21.713 INFO:tasks.workunit.client.0.smithi136.stderr:211858: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T21:04:21.713 INFO:tasks.workunit.client.0.smithi136.stderr:211858: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T21:04:21.713 INFO:tasks.workunit.client.0.smithi136.stderr:211858: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/5' 2024-09-15T21:04:21.714 INFO:tasks.workunit.client.0.smithi136.stderr:211858: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/5/journal' 2024-09-15T21:04:21.714 INFO:tasks.workunit.client.0.smithi136.stderr:211858: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-15T21:04:21.714 INFO:tasks.workunit.client.0.smithi136.stderr:211858: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-15T21:04:21.714 INFO:tasks.workunit.client.0.smithi136.stderr:211858: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-15T21:04:21.714 INFO:tasks.workunit.client.0.smithi136.stderr:211858: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-15T21:04:21.714 INFO:tasks.workunit.client.0.smithi136.stderr:211858: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-15T21:04:21.714 INFO:tasks.workunit.client.0.smithi136.stderr:211858: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-15T21:04:21.714 INFO:tasks.workunit.client.0.smithi136.stderr:211858: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-15T21:04:21.714 INFO:tasks.workunit.client.0.smithi136.stderr:211858: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T21:04:21.714 INFO:tasks.workunit.client.0.smithi136.stderr:211858: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52602 2024-09-15T21:04:21.714 INFO:tasks.workunit.client.0.smithi136.stderr:211858: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.52602/$cluster-$name.asok' 2024-09-15T21:04:21.714 INFO:tasks.workunit.client.0.smithi136.stderr:211858: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.52602/$cluster-$name.asok' 2024-09-15T21:04:21.715 INFO:tasks.workunit.client.0.smithi136.stderr:211858: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-15T21:04:21.715 INFO:tasks.workunit.client.0.smithi136.stderr:211858: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-15T21:04:21.715 INFO:tasks.workunit.client.0.smithi136.stderr:211858: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-15T21:04:21.715 INFO:tasks.workunit.client.0.smithi136.stderr:211858: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T21:04:21.715 INFO:tasks.workunit.client.0.smithi136.stderr:211858: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T21:04:21.715 INFO:tasks.workunit.client.0.smithi136.stderr:211858: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T21:04:21.715 INFO:tasks.workunit.client.0.smithi136.stderr:211858: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T21:04:21.715 INFO:tasks.workunit.client.0.smithi136.stderr:211858: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-15T21:04:21.715 INFO:tasks.workunit.client.0.smithi136.stderr:211858: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-15T21:04:21.715 INFO:tasks.workunit.client.0.smithi136.stderr:211858: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/5 2024-09-15T21:04:21.715 INFO:tasks.workunit.client.0.smithi136.stderr:211858: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.5 2024-09-15T21:04:21.715 INFO:tasks.workunit.client.0.smithi136.stderr:211858: start osd.5 2024-09-15T21:04:21.715 INFO:tasks.workunit.client.0.smithi136.stderr:211858: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 5 --fsid=ce4892c7-6c6d-4b4f-8e6f-07a3e11133e5 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/5 --osd-journal=td/osd-scrub-repair/5/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.52602/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T21:04:26.306 INFO:tasks.workunit.client.0.smithi136.stderr:211858: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/5/whoami 2024-09-15T21:04:26.306 INFO:tasks.workunit.client.0.smithi136.stderr:211858: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 5 = 5 ']' 2024-09-15T21:04:26.306 INFO:tasks.workunit.client.0.smithi136.stderr:211858: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-15T21:04:26.306 INFO:tasks.workunit.client.0.smithi136.stderr:211858: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-15T21:04:26.307 INFO:tasks.workunit.client.0.smithi136.stderr:211858: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-15T21:04:26.307 INFO:tasks.workunit.client.0.smithi136.stderr:211858: 2024-09-15T21:04:21.740+0000 7f07a7ddb640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:04:26.307 INFO:tasks.workunit.client.0.smithi136.stderr:211858: 2024-09-15T21:04:21.742+0000 7f07a7ddb640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:04:26.307 INFO:tasks.workunit.client.0.smithi136.stderr:211858: 2024-09-15T21:04:21.743+0000 7f07a7ddb640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:04:26.307 INFO:tasks.workunit.client.0.smithi136.stderr:211858: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 5 2024-09-15T21:04:26.307 INFO:tasks.workunit.client.0.smithi136.stderr:211858: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-15T21:04:26.307 INFO:tasks.workunit.client.0.smithi136.stderr:211858: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=5 2024-09-15T21:04:26.307 INFO:tasks.workunit.client.0.smithi136.stderr:211858: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-15T21:04:26.307 INFO:tasks.workunit.client.0.smithi136.stderr:211858: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-15T21:04:26.307 INFO:tasks.workunit.client.0.smithi136.stderr:211858: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:04:26.307 INFO:tasks.workunit.client.0.smithi136.stderr:211858: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-15T21:04:26.307 INFO:tasks.workunit.client.0.smithi136.stderr:211858: 0 2024-09-15T21:04:26.307 INFO:tasks.workunit.client.0.smithi136.stderr:211858: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:04:26.308 INFO:tasks.workunit.client.0.smithi136.stderr:211858: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-15T21:04:26.308 INFO:tasks.workunit.client.0.smithi136.stderr:211858: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:04:26.308 INFO:tasks.workunit.client.0.smithi136.stderr:211858: 2024-09-15T21:04:23.115+0000 7f07a7ddb640 -1 Falling back to public interface 2024-09-15T21:04:26.308 INFO:tasks.workunit.client.0.smithi136.stderr:211858: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:04:26.308 INFO:tasks.workunit.client.0.smithi136.stderr:211858: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:04:26.308 INFO:tasks.workunit.client.0.smithi136.stderr:211858: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-15T21:04:26.308 INFO:tasks.workunit.client.0.smithi136.stderr:211858: 1 2024-09-15T21:04:26.308 INFO:tasks.workunit.client.0.smithi136.stderr:211858: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:04:26.308 INFO:tasks.workunit.client.0.smithi136.stderr:211858: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-15T21:04:26.308 INFO:tasks.workunit.client.0.smithi136.stderr:211858: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:04:26.308 INFO:tasks.workunit.client.0.smithi136.stderr:211858: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:04:26.308 INFO:tasks.workunit.client.0.smithi136.stderr:211858: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:04:26.308 INFO:tasks.workunit.client.0.smithi136.stderr:211858: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-15T21:04:26.309 INFO:tasks.workunit.client.0.smithi136.stderr:211858: 2 2024-09-15T21:04:26.309 INFO:tasks.workunit.client.0.smithi136.stderr:211858: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:04:26.309 INFO:tasks.workunit.client.0.smithi136.stderr:211858: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-15T21:04:26.309 INFO:tasks.workunit.client.0.smithi136.stderr:211858: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:04:26.309 INFO:tasks.workunit.client.0.smithi136.stderr:211858: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:04:26.309 INFO:tasks.workunit.client.0.smithi136.stderr:211858: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:04:26.309 INFO:tasks.workunit.client.0.smithi136.stderr:211858: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-15T21:04:26.309 INFO:tasks.workunit.client.0.smithi136.stderr:211858: 3 2024-09-15T21:04:26.309 INFO:tasks.workunit.client.0.smithi136.stderr:211858: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:04:26.309 INFO:tasks.workunit.client.0.smithi136.stderr:211858: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-15T21:04:26.309 INFO:tasks.workunit.client.0.smithi136.stderr:211858: 2024-09-15T21:04:26.106+0000 7f07a7ddb640 -1 osd.5 90 log_to_monitors true 2024-09-15T21:04:26.614 INFO:tasks.workunit.client.0.smithi136.stderr:211858: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers11860: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/9/whoami 2024-09-15T21:04:26.615 INFO:tasks.workunit.client.0.smithi136.stderr:211860: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 9 = 9 ']' 2024-09-15T21:04:26.615 INFO:tasks.workunit.client.0.smithi136.stderr:211860: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-15T21:04:26.615 INFO:tasks.workunit.client.0.smithi136.stderr:211860: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-15T21:04:26.615 INFO:tasks.workunit.client.0.smithi136.stderr:211860: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-15T21:04:26.615 INFO:tasks.workunit.client.0.smithi136.stderr:211860: 2024-09-15T21:04:21.704+0000 7fee6920c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:04:26.615 INFO:tasks.workunit.client.0.smithi136.stderr:211860: 2024-09-15T21:04:21.707+0000 7fee6920c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:04:26.615 INFO:tasks.workunit.client.0.smithi136.stderr:211860: 2024-09-15T21:04:21.708+0000 7fee6920c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:04:26.615 INFO:tasks.workunit.client.0.smithi136.stderr:211860: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 9 2024-09-15T21:04:26.615 INFO:tasks.workunit.client.0.smithi136.stderr:211860: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-15T21:04:26.615 INFO:tasks.workunit.client.0.smithi136.stderr:211860: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=9 2024-09-15T21:04:26.615 INFO:tasks.workunit.client.0.smithi136.stderr:211860: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-15T21:04:26.615 INFO:tasks.workunit.client.0.smithi136.stderr:211860: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-15T21:04:26.615 INFO:tasks.workunit.client.0.smithi136.stderr:211860: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:04:26.616 INFO:tasks.workunit.client.0.smithi136.stderr:211860: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-15T21:04:26.616 INFO:tasks.workunit.client.0.smithi136.stderr:211860: 0 2024-09-15T21:04:26.616 INFO:tasks.workunit.client.0.smithi136.stderr:211860: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:04:26.616 INFO:tasks.workunit.client.0.smithi136.stderr:211860: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.9 up' 2024-09-15T21:04:26.616 INFO:tasks.workunit.client.0.smithi136.stderr:211860: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:04:26.616 INFO:tasks.workunit.client.0.smithi136.stderr:211860: 2024-09-15T21:04:23.295+0000 7fee6920c640 -1 Falling back to public interface 2024-09-15T21:04:26.616 INFO:tasks.workunit.client.0.smithi136.stderr:211860: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:04:26.616 INFO:tasks.workunit.client.0.smithi136.stderr:211860: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:04:26.616 INFO:tasks.workunit.client.0.smithi136.stderr:211860: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-15T21:04:26.616 INFO:tasks.workunit.client.0.smithi136.stderr:211860: 1 2024-09-15T21:04:26.616 INFO:tasks.workunit.client.0.smithi136.stderr:211860: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:04:26.616 INFO:tasks.workunit.client.0.smithi136.stderr:211860: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.9 up' 2024-09-15T21:04:26.616 INFO:tasks.workunit.client.0.smithi136.stderr:211860: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:04:26.616 INFO:tasks.workunit.client.0.smithi136.stderr:211860: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:04:26.617 INFO:tasks.workunit.client.0.smithi136.stderr:211860: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:04:26.617 INFO:tasks.workunit.client.0.smithi136.stderr:211860: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-15T21:04:26.617 INFO:tasks.workunit.client.0.smithi136.stderr:211860: 2 2024-09-15T21:04:26.617 INFO:tasks.workunit.client.0.smithi136.stderr:211860: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:04:26.617 INFO:tasks.workunit.client.0.smithi136.stderr:211860: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.9 up' 2024-09-15T21:04:26.617 INFO:tasks.workunit.client.0.smithi136.stderr:211860: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:04:26.617 INFO:tasks.workunit.client.0.smithi136.stderr:211860: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:04:26.617 INFO:tasks.workunit.client.0.smithi136.stderr:211860: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:04:26.617 INFO:tasks.workunit.client.0.smithi136.stderr:211860: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-15T21:04:26.617 INFO:tasks.workunit.client.0.smithi136.stderr:211860: 3 2024-09-15T21:04:26.617 INFO:tasks.workunit.client.0.smithi136.stderr:211860: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:04:26.617 INFO:tasks.workunit.client.0.smithi136.stderr:211860: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.9 up' 2024-09-15T21:04:26.617 INFO:tasks.workunit.client.0.smithi136.stderr:211860: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:04:28.508 INFO:tasks.workunit.client.0.smithi136.stderr:211860: 2024-09-15T21:04:26.611+0000 7fee6920c640 -1 osd.9 90 log_to_monitors true 2024-09-15T21:04:28.509 INFO:tasks.workunit.client.0.smithi136.stderr:211860: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:04:28.509 INFO:tasks.workunit.client.0.smithi136.stderr:211860: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:04:28.509 INFO:tasks.workunit.client.0.smithi136.stderr:211860: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-15T21:04:28.509 INFO:tasks.workunit.client.0.smithi136.stderr:211860: 4 2024-09-15T21:04:28.509 INFO:tasks.workunit.client.0.smithi136.stderr:211860: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:04:28.509 INFO:tasks.workunit.client.0.smithi136.stderr:211860: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.9 up' 2024-09-15T21:04:28.509 INFO:tasks.workunit.client.0.smithi136.stderr:211860: osd.9 up in weight 1 up_from 96 up_thru 0 down_at 91 last_clean_interval [88,90) [v2:127.0.0.1:6874/1281269103,v1:127.0.0.1:6875/1281269103] [v2:127.0.0.1:6876/1281269103,v1:127.0.0.1:6877/1281269103] exists,up ccfb463f-a1cc-4f3c-a7db-ae126efbeaa5 2024-09-15T21:04:28.509 INFO:tasks.workunit.client.0.smithi136.stderr:211860: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-15T21:04:28.509 INFO:tasks.workunit.client.0.smithi136.stderr:211860: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-15T21:04:28.509 INFO:tasks.workunit.client.0.smithi136.stderr:211860: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-15T21:04:28.509 INFO:tasks.workunit.client.0.smithi136.stderr:211860: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2024-09-15T21:04:28.509 INFO:tasks.workunit.client.0.smithi136.stderr:211860: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-15T21:04:28.510 INFO:tasks.workunit.client.0.smithi136.stderr:211860: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-15T21:04:28.510 INFO:tasks.workunit.client.0.smithi136.stderr:211860: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-15T21:04:28.510 INFO:tasks.workunit.client.0.smithi136.stderr:211860: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-15T21:04:28.510 INFO:tasks.workunit.client.0.smithi136.stderr:211860: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-15T21:04:28.510 INFO:tasks.workunit.client.0.smithi136.stderr:211860: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-15T21:04:28.510 INFO:tasks.workunit.client.0.smithi136.stderr:211860: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-15T21:04:28.510 INFO:tasks.workunit.client.0.smithi136.stderr:211860: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-15T21:04:28.510 INFO:tasks.workunit.client.0.smithi136.stderr:211860: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-15T21:04:28.510 INFO:tasks.workunit.client.0.smithi136.stderr:211860: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-15T21:04:28.510 INFO:tasks.workunit.client.0.smithi136.stderr:211860: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-15T21:04:28.510 INFO:tasks.workunit.client.0.smithi136.stderr:211860: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-15T21:04:28.511 INFO:tasks.workunit.client.0.smithi136.stderr:211860: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-15T21:04:28.511 INFO:tasks.workunit.client.0.smithi136.stderr:211860: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-15T21:04:28.511 INFO:tasks.workunit.client.0.smithi136.stderr:211860: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-15T21:04:28.511 INFO:tasks.workunit.client.0.smithi136.stderr:211860: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-15T21:04:28.511 INFO:tasks.workunit.client.0.smithi136.stderr:211860: 1 2024-09-15T21:04:28.511 INFO:tasks.workunit.client.0.smithi136.stderr:211860: 2 2024-09-15T21:04:28.511 INFO:tasks.workunit.client.0.smithi136.stderr:211860: 3 2024-09-15T21:04:28.511 INFO:tasks.workunit.client.0.smithi136.stderr:211860: 4 2024-09-15T21:04:28.511 INFO:tasks.workunit.client.0.smithi136.stderr:211860: 5 2024-09-15T21:04:28.511 INFO:tasks.workunit.client.0.smithi136.stderr:211860: 6 2024-09-15T21:04:28.511 INFO:tasks.workunit.client.0.smithi136.stderr:211860: 7 2024-09-15T21:04:28.511 INFO:tasks.workunit.client.0.smithi136.stderr:211860: 8 2024-09-15T21:04:28.511 INFO:tasks.workunit.client.0.smithi136.stderr:211860: 9' 2024-09-15T21:04:28.512 INFO:tasks.workunit.client.0.smithi136.stderr:211860: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-15T21:04:28.512 INFO:tasks.workunit.client.0.smithi136.stderr:211860: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T21:04:28.512 INFO:tasks.workunit.client.0.smithi136.stderr:211860: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-15T21:04:28.512 INFO:tasks.workunit.client.0.smithi136.stderr:211860: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836579 2024-09-15T21:04:28.512 INFO:tasks.workunit.client.0.smithi136.stderr:211860: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836579 2024-09-15T21:04:28.512 INFO:tasks.workunit.client.0.smithi136.stderr:211860: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836579' 2024-09-15T21:04:28.512 INFO:tasks.workunit.client.0.smithi136.stderr:211860: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T21:04:28.512 INFO:tasks.workunit.client.0.smithi136.stderr:211860: //.sh:976: wait_for_osd: sleep 1 2024-09-15T21:04:28.512 INFO:tasks.workunit.client.0.smithi136.stderr:211858: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:04:28.512 INFO:tasks.workunit.client.0.smithi136.stderr:211858: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:04:28.512 INFO:tasks.workunit.client.0.smithi136.stderr:211858: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-15T21:04:28.512 INFO:tasks.workunit.client.0.smithi136.stderr:211858: 4 2024-09-15T21:04:28.512 INFO:tasks.workunit.client.0.smithi136.stderr:211858: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:04:28.513 INFO:tasks.workunit.client.0.smithi136.stderr:211858: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-15T21:04:28.513 INFO:tasks.workunit.client.0.smithi136.stderr:211858: osd.5 up in weight 1 up_from 95 up_thru 95 down_at 91 last_clean_interval [83,90) [v2:127.0.0.1:6842/3988459449,v1:127.0.0.1:6843/3988459449] [v2:127.0.0.1:6844/3988459449,v1:127.0.0.1:6845/3988459449] exists,up c3187161-e486-4b18-8257-d69c125e9792 2024-09-15T21:04:28.513 INFO:tasks.workunit.client.0.smithi136.stderr:211858: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-15T21:04:28.513 INFO:tasks.workunit.client.0.smithi136.stderr:211858: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-15T21:04:28.513 INFO:tasks.workunit.client.0.smithi136.stderr:211858: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-15T21:04:28.513 INFO:tasks.workunit.client.0.smithi136.stderr:211858: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2024-09-15T21:04:28.513 INFO:tasks.workunit.client.0.smithi136.stderr:211858: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-15T21:04:28.513 INFO:tasks.workunit.client.0.smithi136.stderr:211858: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-15T21:04:28.513 INFO:tasks.workunit.client.0.smithi136.stderr:211858: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-15T21:04:28.513 INFO:tasks.workunit.client.0.smithi136.stderr:211858: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-15T21:04:28.513 INFO:tasks.workunit.client.0.smithi136.stderr:211858: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-15T21:04:28.513 INFO:tasks.workunit.client.0.smithi136.stderr:211858: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-15T21:04:28.514 INFO:tasks.workunit.client.0.smithi136.stderr:211858: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-15T21:04:28.514 INFO:tasks.workunit.client.0.smithi136.stderr:211858: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-15T21:04:28.514 INFO:tasks.workunit.client.0.smithi136.stderr:211858: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-15T21:04:28.514 INFO:tasks.workunit.client.0.smithi136.stderr:211858: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-15T21:04:28.514 INFO:tasks.workunit.client.0.smithi136.stderr:211858: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-15T21:04:28.514 INFO:tasks.workunit.client.0.smithi136.stderr:211858: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-15T21:04:28.514 INFO:tasks.workunit.client.0.smithi136.stderr:211858: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-15T21:04:28.514 INFO:tasks.workunit.client.0.smithi136.stderr:211858: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-15T21:04:28.514 INFO:tasks.workunit.client.0.smithi136.stderr:211858: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-15T21:04:28.514 INFO:tasks.workunit.client.0.smithi136.stderr:211858: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-15T21:04:28.514 INFO:tasks.workunit.client.0.smithi136.stderr:211858: 1 2024-09-15T21:04:28.514 INFO:tasks.workunit.client.0.smithi136.stderr:211858: 2 2024-09-15T21:04:28.514 INFO:tasks.workunit.client.0.smithi136.stderr:211858: 3 2024-09-15T21:04:28.514 INFO:tasks.workunit.client.0.smithi136.stderr:211858: 4 2024-09-15T21:04:28.515 INFO:tasks.workunit.client.0.smithi136.stderr:211858: 5 2024-09-15T21:04:28.515 INFO:tasks.workunit.client.0.smithi136.stderr:211858: 6 2024-09-15T21:04:28.515 INFO:tasks.workunit.client.0.smithi136.stderr:211858: 7 2024-09-15T21:04:28.515 INFO:tasks.workunit.client.0.smithi136.stderr:211858: 8 2024-09-15T21:04:28.515 INFO:tasks.workunit.client.0.smithi136.stderr:211858: 9' 2024-09-15T21:04:28.515 INFO:tasks.workunit.client.0.smithi136.stderr:211858: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-15T21:04:28.515 INFO:tasks.workunit.client.0.smithi136.stderr:211858: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T21:04:28.515 INFO:tasks.workunit.client.0.smithi136.stderr:211858: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-15T21:04:28.515 INFO:tasks.workunit.client.0.smithi136.stderr:211858: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836578 2024-09-15T21:04:28.515 INFO:tasks.workunit.client.0.smithi136.stderr:211858: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836578 2024-09-15T21:04:28.515 INFO:tasks.workunit.client.0.smithi136.stderr:211858: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836578' 2024-09-15T21:04:28.515 INFO:tasks.workunit.client.0.smithi136.stderr:211858: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T21:04:29.783 INFO:tasks.workunit.client.0.smithi136.stderr:211858: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-15T21:04:29.783 INFO:tasks.workunit.client.0.smithi136.stderr:211858: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=42949673049 2024-09-15T21:04:29.783 INFO:tasks.workunit.client.0.smithi136.stderr:211858: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 42949673049 2024-09-15T21:04:29.783 INFO:tasks.workunit.client.0.smithi136.stderr:211858: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836578 1-42949673049' 2024-09-15T21:04:29.783 INFO:tasks.workunit.client.0.smithi136.stderr:211858: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T21:04:29.783 INFO:tasks.workunit.client.0.smithi136.stderr:211858: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-15T21:04:29.783 INFO:tasks.workunit.client.0.smithi136.stderr:211858: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=64424509523 2024-09-15T21:04:29.783 INFO:tasks.workunit.client.0.smithi136.stderr:211858: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 64424509523 2024-09-15T21:04:29.784 INFO:tasks.workunit.client.0.smithi136.stderr:211858: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836578 1-42949673049 2-64424509523' 2024-09-15T21:04:29.784 INFO:tasks.workunit.client.0.smithi136.stderr:211858: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T21:04:29.784 INFO:tasks.workunit.client.0.smithi136.stderr:211858: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-15T21:04:29.784 INFO:tasks.workunit.client.0.smithi136.stderr:211858: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=313532612629 2024-09-15T21:04:29.784 INFO:tasks.workunit.client.0.smithi136.stderr:211858: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 313532612629 2024-09-15T21:04:29.784 INFO:tasks.workunit.client.0.smithi136.stderr:211858: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836578 1-42949673049 2-64424509523 3-313532612629' 2024-09-15T21:04:29.784 INFO:tasks.workunit.client.0.smithi136.stderr:211858: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T21:04:29.784 INFO:tasks.workunit.client.0.smithi136.stderr:211858: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-09-15T21:04:29.784 INFO:tasks.workunit.client.0.smithi136.stderr:211858: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=107374182469 2024-09-15T21:04:29.784 INFO:tasks.workunit.client.0.smithi136.stderr:211858: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 107374182469 2024-09-15T21:04:29.784 INFO:tasks.workunit.client.0.smithi136.stderr:211858: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836578 1-42949673049 2-64424509523 3-313532612629 4-107374182469' 2024-09-15T21:04:29.784 INFO:tasks.workunit.client.0.smithi136.stderr:211858: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T21:04:29.784 INFO:tasks.workunit.client.0.smithi136.stderr:211858: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-09-15T21:04:29.785 INFO:tasks.workunit.client.0.smithi136.stderr:211858: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=408021893122 2024-09-15T21:04:29.785 INFO:tasks.workunit.client.0.smithi136.stderr:211858: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 408021893122 2024-09-15T21:04:29.785 INFO:tasks.workunit.client.0.smithi136.stderr:211858: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836578 1-42949673049 2-64424509523 3-313532612629 4-107374182469 5-408021893122' 2024-09-15T21:04:29.785 INFO:tasks.workunit.client.0.smithi136.stderr:211858: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T21:04:29.785 INFO:tasks.workunit.client.0.smithi136.stderr:211858: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-09-15T21:04:29.785 INFO:tasks.workunit.client.0.smithi136.stderr:211858: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=150323855417 2024-09-15T21:04:29.785 INFO:tasks.workunit.client.0.smithi136.stderr:211858: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 150323855417 2024-09-15T21:04:29.785 INFO:tasks.workunit.client.0.smithi136.stderr:211858: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836578 1-42949673049 2-64424509523 3-313532612629 4-107374182469 5-408021893122 6-150323855417' 2024-09-15T21:04:29.785 INFO:tasks.workunit.client.0.smithi136.stderr:211858: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T21:04:29.785 INFO:tasks.workunit.client.0.smithi136.stderr:211858: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2024-09-15T21:04:29.785 INFO:tasks.workunit.client.0.smithi136.stderr:211858: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-15T21:04:29.785 INFO:tasks.workunit.client.0.smithi136.stderr:211860: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=42949673048 2024-09-15T21:04:29.785 INFO:tasks.workunit.client.0.smithi136.stderr:211860: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 42949673048 2024-09-15T21:04:29.786 INFO:tasks.workunit.client.0.smithi136.stderr:211860: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836579 1-42949673048' 2024-09-15T21:04:29.786 INFO:tasks.workunit.client.0.smithi136.stderr:211860: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T21:04:29.786 INFO:tasks.workunit.client.0.smithi136.stderr:211860: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-15T21:04:29.786 INFO:tasks.workunit.client.0.smithi136.stderr:211860: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=64424509524 2024-09-15T21:04:29.786 INFO:tasks.workunit.client.0.smithi136.stderr:211860: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 64424509524 2024-09-15T21:04:29.786 INFO:tasks.workunit.client.0.smithi136.stderr:211860: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836579 1-42949673048 2-64424509524' 2024-09-15T21:04:29.786 INFO:tasks.workunit.client.0.smithi136.stderr:211860: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T21:04:29.786 INFO:tasks.workunit.client.0.smithi136.stderr:211860: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-15T21:04:29.786 INFO:tasks.workunit.client.0.smithi136.stderr:211860: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=313532612628 2024-09-15T21:04:29.786 INFO:tasks.workunit.client.0.smithi136.stderr:211860: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 313532612628 2024-09-15T21:04:29.786 INFO:tasks.workunit.client.0.smithi136.stderr:211860: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836579 1-42949673048 2-64424509524 3-313532612628' 2024-09-15T21:04:29.786 INFO:tasks.workunit.client.0.smithi136.stderr:211860: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T21:04:29.786 INFO:tasks.workunit.client.0.smithi136.stderr:211860: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-09-15T21:04:29.786 INFO:tasks.workunit.client.0.smithi136.stderr:211860: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=107374182468 2024-09-15T21:04:29.787 INFO:tasks.workunit.client.0.smithi136.stderr:211860: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 107374182468 2024-09-15T21:04:29.787 INFO:tasks.workunit.client.0.smithi136.stderr:211860: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836579 1-42949673048 2-64424509524 3-313532612628 4-107374182468' 2024-09-15T21:04:29.787 INFO:tasks.workunit.client.0.smithi136.stderr:211860: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T21:04:29.787 INFO:tasks.workunit.client.0.smithi136.stderr:211860: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-09-15T21:04:29.787 INFO:tasks.workunit.client.0.smithi136.stderr:211860: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=408021893123 2024-09-15T21:04:29.787 INFO:tasks.workunit.client.0.smithi136.stderr:211860: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 408021893123 2024-09-15T21:04:29.787 INFO:tasks.workunit.client.0.smithi136.stderr:211860: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836579 1-42949673048 2-64424509524 3-313532612628 4-107374182468 5-408021893123' 2024-09-15T21:04:29.787 INFO:tasks.workunit.client.0.smithi136.stderr:211860: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T21:04:29.787 INFO:tasks.workunit.client.0.smithi136.stderr:211860: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-09-15T21:04:29.787 INFO:tasks.workunit.client.0.smithi136.stderr:211860: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=150323855416 2024-09-15T21:04:29.787 INFO:tasks.workunit.client.0.smithi136.stderr:211860: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 150323855416 2024-09-15T21:04:29.787 INFO:tasks.workunit.client.0.smithi136.stderr:211860: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836579 1-42949673048 2-64424509524 3-313532612628 4-107374182468 5-408021893123 6-150323855416' 2024-09-15T21:04:29.787 INFO:tasks.workunit.client.0.smithi136.stderr:211860: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T21:04:29.788 INFO:tasks.workunit.client.0.smithi136.stderr:211860: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2024-09-15T21:04:30.309 INFO:tasks.workunit.client.0.smithi136.stderr:211860: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_ppg_stats: seq=171798691890 2024-09-15T21:04:30.309 INFO:tasks.workunit.client.0.smithi136.stderr:211858: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 171798691890 2024-09-15T21:04:30.309 INFO:tasks.workunit.client.0.smithi136.stderr:211858: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836578 1-42949673049 2-64424509523 3-313532612629 4-107374182469 5-408021893122 6-150323855417 7-171798691890' 2024-09-15T21:04:30.309 INFO:tasks.workunit.client.0.smithi136.stderr:211858: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T21:04:30.309 INFO:tasks.workunit.client.0.smithi136.stderr:211858: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2024-09-15T21:04:30.310 INFO:tasks.workunit.client.0.smithi136.stderr:211858: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=193273528365 2024-09-15T21:04:30.310 INFO:tasks.workunit.client.0.smithi136.stderr:211858: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 193273528365 2024-09-15T21:04:30.310 INFO:tasks.workunit.client.0.smithi136.stderr:211858: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836578 1-42949673049 2-64424509523 3-313532612629 4-107374182469 5-408021893122 6-150323855417 7-171798691890 8-193273528365' 2024-09-15T21:04:30.310 INFO:tasks.workunit.client.0.smithi136.stderr:211858: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T21:04:30.310 INFO:tasks.workunit.client.0.smithi136.stderr:211858: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2024-09-15T21:04:30.310 INFO:tasks.workunit.client.0.smithi136.stderr:211858: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=412316860418 2024-09-15T21:04:30.310 INFO:tasks.workunit.client.0.smithi136.stderr:211858: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 412316860418 2024-09-15T21:04:30.310 INFO:tasks.workunit.client.0.smithi136.stderr:211858: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836578 1-42949673049 2-64424509523 3-313532612629 4-107374182469 5-408021893122 6-150323855417 7-171798691890 8-193273528365 9-412316860418' 2024-09-15T21:04:30.310 INFO:tasks.workunit.client.0.smithi136.stderr:211858: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T21:04:30.310 INFO:tasks.workunit.client.0.smithi136.stderr:211858: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836578 2024-09-15T21:04:30.310 INFO:tasks.workunit.client.0.smithi136.stderr:211858: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T21:04:30.311 INFO:tasks.workunit.client.0.smithi136.stderr:211858: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-15T21:04:30.311 INFO:tasks.workunit.client.0.smithi136.stderr:211858: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836578 2024-09-15T21:04:30.311 INFO:tasks.workunit.client.0.smithi136.stderr:211858: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T21:04:30.311 INFO:tasks.workunit.client.0.smithi136.stderr:211858: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836578 2024-09-15T21:04:30.311 INFO:tasks.workunit.client.0.smithi136.stderr:211858: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836578' 2024-09-15T21:04:30.311 INFO:tasks.workunit.client.0.smithi136.stderr:211858: waiting osd.0 seq 21474836578 2024-09-15T21:04:30.311 INFO:tasks.workunit.client.0.smithi136.stderr:211858: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T21:04:30.311 INFO:tasks.workunit.client.0.smithi136.stderr:211858: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836579 -lt 21474836578 2024-09-15T21:04:30.311 INFO:tasks.workunit.client.0.smithi136.stderr:211858: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T21:04:30.311 INFO:tasks.workunit.client.0.smithi136.stderr:211858: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-42949673049 2024-09-15T21:04:30.311 INFO:tasks.workunit.client.0.smithi136.stderr:211858: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T21:04:30.311 INFO:tasks.workunit.client.0.smithi136.stderr:211858: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-15T21:04:30.312 INFO:tasks.workunit.client.0.smithi136.stderr:211858: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-42949673049 2024-09-15T21:04:30.312 INFO:tasks.workunit.client.0.smithi136.stderr:211858: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T21:04:30.312 INFO:tasks.workunit.client.0.smithi136.stderr:211858: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=42949673049 2024-09-15T21:04:30.312 INFO:tasks.workunit.client.0.smithi136.stderr:211858: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 42949673049' 2024-09-15T21:04:30.312 INFO:tasks.workunit.client.0.smithi136.stderr:211858: waiting osd.1 seq 42949673049 2024-09-15T21:04:30.312 INFO:tasks.workunit.client.0.smithi136.stderr:211858: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_sg_stats: seq=171798691889 2024-09-15T21:04:30.312 INFO:tasks.workunit.client.0.smithi136.stderr:211860: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 171798691889 2024-09-15T21:04:30.313 INFO:tasks.workunit.client.0.smithi136.stderr:211860: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836579 1-42949673048 2-64424509524 3-313532612628 4-107374182468 5-408021893123 6-150323855416 7-171798691889' 2024-09-15T21:04:30.313 INFO:tasks.workunit.client.0.smithi136.stderr:211860: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T21:04:30.313 INFO:tasks.workunit.client.0.smithi136.stderr:211860: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2024-09-15T21:04:30.313 INFO:tasks.workunit.client.0.smithi136.stderr:211860: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=193273528364 2024-09-15T21:04:30.313 INFO:tasks.workunit.client.0.smithi136.stderr:211860: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 193273528364 2024-09-15T21:04:30.313 INFO:tasks.workunit.client.0.smithi136.stderr:211860: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836579 1-42949673048 2-64424509524 3-313532612628 4-107374182468 5-408021893123 6-150323855416 7-171798691889 8-193273528364' 2024-09-15T21:04:30.313 INFO:tasks.workunit.client.0.smithi136.stderr:211860: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T21:04:30.313 INFO:tasks.workunit.client.0.smithi136.stderr:211860: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2024-09-15T21:04:30.313 INFO:tasks.workunit.client.0.smithi136.stderr:211860: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=412316860419 2024-09-15T21:04:30.313 INFO:tasks.workunit.client.0.smithi136.stderr:211860: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 412316860419 2024-09-15T21:04:30.313 INFO:tasks.workunit.client.0.smithi136.stderr:211860: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836579 1-42949673048 2-64424509524 3-313532612628 4-107374182468 5-408021893123 6-150323855416 7-171798691889 8-193273528364 9-412316860419' 2024-09-15T21:04:30.313 INFO:tasks.workunit.client.0.smithi136.stderr:211860: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T21:04:30.313 INFO:tasks.workunit.client.0.smithi136.stderr:211860: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836579 2024-09-15T21:04:30.314 INFO:tasks.workunit.client.0.smithi136.stderr:211860: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T21:04:30.314 INFO:tasks.workunit.client.0.smithi136.stderr:211860: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-15T21:04:30.314 INFO:tasks.workunit.client.0.smithi136.stderr:211860: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836579 2024-09-15T21:04:30.314 INFO:tasks.workunit.client.0.smithi136.stderr:211860: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T21:04:30.314 INFO:tasks.workunit.client.0.smithi136.stderr:211860: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836579 2024-09-15T21:04:30.314 INFO:tasks.workunit.client.0.smithi136.stderr:211860: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836579' 2024-09-15T21:04:30.314 INFO:tasks.workunit.client.0.smithi136.stderr:211860: waiting osd.0 seq 21474836579 2024-09-15T21:04:30.314 INFO:tasks.workunit.client.0.smithi136.stderr:211860: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T21:04:30.314 INFO:tasks.workunit.client.0.smithi136.stderr:211860: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836579 -lt 21474836579 2024-09-15T21:04:30.314 INFO:tasks.workunit.client.0.smithi136.stderr:211860: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T21:04:30.314 INFO:tasks.workunit.client.0.smithi136.stderr:211860: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-42949673048 2024-09-15T21:04:30.314 INFO:tasks.workunit.client.0.smithi136.stderr:211860: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T21:04:30.315 INFO:tasks.workunit.client.0.smithi136.stderr:211860: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-15T21:04:30.315 INFO:tasks.workunit.client.0.smithi136.stderr:211860: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-42949673048 2024-09-15T21:04:30.315 INFO:tasks.workunit.client.0.smithi136.stderr:211860: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T21:04:30.315 INFO:tasks.workunit.client.0.smithi136.stderr:211860: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=42949673048 2024-09-15T21:04:30.315 INFO:tasks.workunit.client.0.smithi136.stderr:211860: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 42949673048' 2024-09-15T21:04:30.315 INFO:tasks.workunit.client.0.smithi136.stderr:211860: waiting osd.1 seq 42949673048 2024-09-15T21:04:31.522 INFO:tasks.workunit.client.0.smithi136.stderr:211860: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_sttats: ceph osd last-stat-seq 1 2024-09-15T21:04:31.523 INFO:tasks.workunit.client.0.smithi136.stderr:211858: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 42949673049 -lt 42949673049 2024-09-15T21:04:31.523 INFO:tasks.workunit.client.0.smithi136.stderr:211858: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T21:04:31.523 INFO:tasks.workunit.client.0.smithi136.stderr:211858: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-64424509523 2024-09-15T21:04:31.523 INFO:tasks.workunit.client.0.smithi136.stderr:211858: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T21:04:31.523 INFO:tasks.workunit.client.0.smithi136.stderr:211858: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-15T21:04:31.523 INFO:tasks.workunit.client.0.smithi136.stderr:211858: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-64424509523 2024-09-15T21:04:31.523 INFO:tasks.workunit.client.0.smithi136.stderr:211858: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T21:04:31.523 INFO:tasks.workunit.client.0.smithi136.stderr:211858: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=64424509523 2024-09-15T21:04:31.523 INFO:tasks.workunit.client.0.smithi136.stderr:211858: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 64424509523' 2024-09-15T21:04:31.523 INFO:tasks.workunit.client.0.smithi136.stderr:211858: waiting osd.2 seq 64424509523 2024-09-15T21:04:31.523 INFO:tasks.workunit.client.0.smithi136.stderr:211858: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-15T21:04:31.523 INFO:tasks.workunit.client.0.smithi136.stderr:211858: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 64424509524 -lt 64424509523 2024-09-15T21:04:31.523 INFO:tasks.workunit.client.0.smithi136.stderr:211858: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T21:04:31.524 INFO:tasks.workunit.client.0.smithi136.stderr:211858: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 3-313532612629 2024-09-15T21:04:31.524 INFO:tasks.workunit.client.0.smithi136.stderr:211858: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T21:04:31.524 INFO:tasks.workunit.client.0.smithi136.stderr:211858: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=3 2024-09-15T21:04:31.524 INFO:tasks.workunit.client.0.smithi136.stderr:211858: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 3-313532612629 2024-09-15T21:04:31.524 INFO:tasks.workunit.client.0.smithi136.stderr:211858: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T21:04:31.524 INFO:tasks.workunit.client.0.smithi136.stderr:211858: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=313532612629 2024-09-15T21:04:31.524 INFO:tasks.workunit.client.0.smithi136.stderr:211858: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.3 seq 313532612629' 2024-09-15T21:04:31.524 INFO:tasks.workunit.client.0.smithi136.stderr:211858: waiting osd.3 seq 313532612629 2024-09-15T21:04:31.524 INFO:tasks.workunit.client.0.smithi136.stderr:211858: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-15T21:04:31.524 INFO:tasks.workunit.client.0.smithi136.stderr:211858: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 313532612629 -lt 313532612629 2024-09-15T21:04:31.524 INFO:tasks.workunit.client.0.smithi136.stderr:211858: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T21:04:31.524 INFO:tasks.workunit.client.0.smithi136.stderr:211858: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 4-107374182469 2024-09-15T21:04:31.524 INFO:tasks.workunit.client.0.smithi136.stderr:211858: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T21:04:31.525 INFO:tasks.workunit.client.0.smithi136.stderr:211858: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=4 2024-09-15T21:04:31.525 INFO:tasks.workunit.client.0.smithi136.stderr:211858: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 4-107374182469 2024-09-15T21:04:31.525 INFO:tasks.workunit.client.0.smithi136.stderr:211858: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T21:04:31.525 INFO:tasks.workunit.client.0.smithi136.stderr:211858: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=107374182469 2024-09-15T21:04:31.525 INFO:tasks.workunit.client.0.smithi136.stderr:211858: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.4 seq 107374182469' 2024-09-15T21:04:31.525 INFO:tasks.workunit.client.0.smithi136.stderr:211858: waiting osd.4 seq 107374182469 2024-09-15T21:04:31.525 INFO:tasks.workunit.client.0.smithi136.stderr:211858: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 4 2024-09-15T21:04:31.525 INFO:tasks.workunit.client.0.smithi136.stderr:211858: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 107374182469 -lt 107374182469 2024-09-15T21:04:31.525 INFO:tasks.workunit.client.0.smithi136.stderr:211858: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T21:04:31.525 INFO:tasks.workunit.client.0.smithi136.stderr:211858: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 5-408021893122 2024-09-15T21:04:31.605 INFO:tasks.workunit.client.0.smithi136.stderr:211858: //home/ubuntu/ceats: ceph osd last-stat-seq 1 2024-09-15T21:04:31.605 INFO:tasks.workunit.client.0.smithi136.stderr:211860: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 42949673049 -lt 42949673048 2024-09-15T21:04:31.605 INFO:tasks.workunit.client.0.smithi136.stderr:211860: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T21:04:31.605 INFO:tasks.workunit.client.0.smithi136.stderr:211860: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-64424509524 2024-09-15T21:04:31.605 INFO:tasks.workunit.client.0.smithi136.stderr:211860: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T21:04:31.605 INFO:tasks.workunit.client.0.smithi136.stderr:211860: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-15T21:04:31.605 INFO:tasks.workunit.client.0.smithi136.stderr:211860: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-64424509524 2024-09-15T21:04:31.605 INFO:tasks.workunit.client.0.smithi136.stderr:211860: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T21:04:31.605 INFO:tasks.workunit.client.0.smithi136.stderr:211860: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=64424509524 2024-09-15T21:04:31.605 INFO:tasks.workunit.client.0.smithi136.stderr:211860: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 64424509524' 2024-09-15T21:04:31.605 INFO:tasks.workunit.client.0.smithi136.stderr:211860: waiting osd.2 seq 64424509524 2024-09-15T21:04:31.606 INFO:tasks.workunit.client.0.smithi136.stderr:211860: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-15T21:04:31.606 INFO:tasks.workunit.client.0.smithi136.stderr:211860: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 64424509524 -lt 64424509524 2024-09-15T21:04:31.606 INFO:tasks.workunit.client.0.smithi136.stderr:211860: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T21:04:31.606 INFO:tasks.workunit.client.0.smithi136.stderr:211860: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 3-313532612628 2024-09-15T21:04:31.606 INFO:tasks.workunit.client.0.smithi136.stderr:211860: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T21:04:31.606 INFO:tasks.workunit.client.0.smithi136.stderr:211860: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=3 2024-09-15T21:04:31.606 INFO:tasks.workunit.client.0.smithi136.stderr:211860: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 3-313532612628 2024-09-15T21:04:31.606 INFO:tasks.workunit.client.0.smithi136.stderr:211860: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T21:04:31.606 INFO:tasks.workunit.client.0.smithi136.stderr:211860: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=313532612628 2024-09-15T21:04:31.606 INFO:tasks.workunit.client.0.smithi136.stderr:211860: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.3 seq 313532612628' 2024-09-15T21:04:31.606 INFO:tasks.workunit.client.0.smithi136.stderr:211860: waiting osd.3 seq 313532612628 2024-09-15T21:04:31.606 INFO:tasks.workunit.client.0.smithi136.stderr:211860: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-15T21:04:31.607 INFO:tasks.workunit.client.0.smithi136.stderr:211860: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 313532612629 -lt 313532612628 2024-09-15T21:04:31.607 INFO:tasks.workunit.client.0.smithi136.stderr:211860: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T21:04:31.607 INFO:tasks.workunit.client.0.smithi136.stderr:211860: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 4-107374182468 2024-09-15T21:04:31.607 INFO:tasks.workunit.client.0.smithi136.stderr:211860: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T21:04:31.607 INFO:tasks.workunit.client.0.smithi136.stderr:211860: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=4 2024-09-15T21:04:31.607 INFO:tasks.workunit.client.0.smithi136.stderr:211860: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 4-107374182468 2024-09-15T21:04:31.607 INFO:tasks.workunit.client.0.smithi136.stderr:211860: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T21:04:31.607 INFO:tasks.workunit.client.0.smithi136.stderr:211860: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=107374182468 2024-09-15T21:04:31.607 INFO:tasks.workunit.client.0.smithi136.stderr:211860: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.4 seq 107374182468' 2024-09-15T21:04:31.607 INFO:tasks.workunit.client.0.smithi136.stderr:211860: waiting osd.4 seq 107374182468 2024-09-15T21:04:31.607 INFO:tasks.workunit.client.0.smithi136.stderr:211860: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 4 2024-09-15T21:04:31.607 INFO:tasks.workunit.client.0.smithi136.stderr:211860: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 107374182469 -lt 107374182468 2024-09-15T21:04:31.607 INFO:tasks.workunit.client.0.smithi136.stderr:211860: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T21:04:31.607 INFO:tasks.workunit.client.0.smithi136.stderr:211860: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 5-408021893123 2024-09-15T21:04:32.494 INFO:tasks.workunit.client.0.smithi136.stderr:211860: //home/ubuntu/cepphtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T21:04:32.494 INFO:tasks.workunit.client.0.smithi136.stderr:211858: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=5 2024-09-15T21:04:32.495 INFO:tasks.workunit.client.0.smithi136.stderr:211858: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 5-408021893122 2024-09-15T21:04:32.495 INFO:tasks.workunit.client.0.smithi136.stderr:211858: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T21:04:32.495 INFO:tasks.workunit.client.0.smithi136.stderr:211858: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=408021893122 2024-09-15T21:04:32.495 INFO:tasks.workunit.client.0.smithi136.stderr:211858: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.5 seq 408021893122' 2024-09-15T21:04:32.495 INFO:tasks.workunit.client.0.smithi136.stderr:211858: waiting osd.5 seq 408021893122 2024-09-15T21:04:32.495 INFO:tasks.workunit.client.0.smithi136.stderr:211858: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 5 2024-09-15T21:04:32.495 INFO:tasks.workunit.client.0.smithi136.stderr:211858: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 408021893123 -lt 408021893122 2024-09-15T21:04:32.495 INFO:tasks.workunit.client.0.smithi136.stderr:211858: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T21:04:32.495 INFO:tasks.workunit.client.0.smithi136.stderr:211858: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 6-150323855417 2024-09-15T21:04:32.495 INFO:tasks.workunit.client.0.smithi136.stderr:211858: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T21:04:32.495 INFO:tasks.workunit.client.0.smithi136.stderr:211858: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=6 2024-09-15T21:04:32.495 INFO:tasks.workunit.client.0.smithi136.stderr:211858: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 6-150323855417 2024-09-15T21:04:32.495 INFO:tasks.workunit.client.0.smithi136.stderr:211858: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T21:04:32.496 INFO:tasks.workunit.client.0.smithi136.stderr:211858: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=150323855417 2024-09-15T21:04:32.496 INFO:tasks.workunit.client.0.smithi136.stderr:211858: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.6 seq 150323855417' 2024-09-15T21:04:32.496 INFO:tasks.workunit.client.0.smithi136.stderr:211858: waiting osd.6 seq 150323855417 2024-09-15T21:04:32.496 INFO:tasks.workunit.client.0.smithi136.stderr:211858: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 6 2024-09-15T21:04:32.496 INFO:tasks.workunit.client.0.smithi136.stderr:211858: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 150323855418 -lt 150323855417 2024-09-15T21:04:32.496 INFO:tasks.workunit.client.0.smithi136.stderr:211858: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T21:04:32.496 INFO:tasks.workunit.client.0.smithi136.stderr:211858: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 7-171798691890 2024-09-15T21:04:32.496 INFO:tasks.workunit.client.0.smithi136.stderr:211858: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T21:04:32.496 INFO:tasks.workunit.client.0.smithi136.stderr:211858: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=7 2024-09-15T21:04:32.496 INFO:tasks.workunit.client.0.smithi136.stderr:211858: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 7-171798691890 2024-09-15T21:04:32.496 INFO:tasks.workunit.client.0.smithi136.stderr:211858: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T21:04:32.497 INFO:tasks.workunit.client.0.smithi136.stderr:211858: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=171798691890 2024-09-15T21:04:32.497 INFO:tasks.workunit.client.0.smithi136.stderr:211858: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.7 seq 171798691890' 2024-09-15T21:04:32.497 INFO:tasks.workunit.client.0.smithi136.stderr:211858: waiting osd.7 seq 171798691890 2024-09-15T21:04:32.497 INFO:tasks.workunit.client.0.smithi136.stderr:211858: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 7 2024-09-15T21:04:32.497 INFO:tasks.workunit.client.0.smithi136.stderr:211858: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 171798691890 -lt 171798691890 2024-09-15T21:04:32.497 INFO:tasks.workunit.client.0.smithi136.stderr:211858: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T21:04:32.497 INFO:tasks.workunit.client.0.smithi136.stderr:211858: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 8-193273528365 2024-09-15T21:04:32.497 INFO:tasks.workunit.client.0.smithi136.stderr:211858: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T21:04:32.497 INFO:tasks.workunit.client.0.smithi136.stderr:211858: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=8 2024-09-15T21:04:32.497 INFO:tasks.workunit.client.0.smithi136.stderr:211858: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 8-193273528365 2024-09-15T21:04:32.532 INFO:tasks.workunit.client.0.smithi136.stderr:211858: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: chtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T21:04:32.533 INFO:tasks.workunit.client.0.smithi136.stderr:211860: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=5 2024-09-15T21:04:32.533 INFO:tasks.workunit.client.0.smithi136.stderr:211860: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 5-408021893123 2024-09-15T21:04:32.533 INFO:tasks.workunit.client.0.smithi136.stderr:211860: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T21:04:32.533 INFO:tasks.workunit.client.0.smithi136.stderr:211860: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=408021893123 2024-09-15T21:04:32.533 INFO:tasks.workunit.client.0.smithi136.stderr:211860: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.5 seq 408021893123' 2024-09-15T21:04:32.533 INFO:tasks.workunit.client.0.smithi136.stderr:211860: waiting osd.5 seq 408021893123 2024-09-15T21:04:32.533 INFO:tasks.workunit.client.0.smithi136.stderr:211860: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 5 2024-09-15T21:04:32.533 INFO:tasks.workunit.client.0.smithi136.stderr:211860: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 408021893123 -lt 408021893123 2024-09-15T21:04:32.533 INFO:tasks.workunit.client.0.smithi136.stderr:211860: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T21:04:32.533 INFO:tasks.workunit.client.0.smithi136.stderr:211860: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 6-150323855416 2024-09-15T21:04:32.533 INFO:tasks.workunit.client.0.smithi136.stderr:211860: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T21:04:32.534 INFO:tasks.workunit.client.0.smithi136.stderr:211860: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=6 2024-09-15T21:04:32.534 INFO:tasks.workunit.client.0.smithi136.stderr:211860: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 6-150323855416 2024-09-15T21:04:32.534 INFO:tasks.workunit.client.0.smithi136.stderr:211860: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T21:04:32.534 INFO:tasks.workunit.client.0.smithi136.stderr:211860: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=150323855416 2024-09-15T21:04:32.534 INFO:tasks.workunit.client.0.smithi136.stderr:211860: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.6 seq 150323855416' 2024-09-15T21:04:32.534 INFO:tasks.workunit.client.0.smithi136.stderr:211860: waiting osd.6 seq 150323855416 2024-09-15T21:04:32.534 INFO:tasks.workunit.client.0.smithi136.stderr:211860: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 6 2024-09-15T21:04:32.534 INFO:tasks.workunit.client.0.smithi136.stderr:211860: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 150323855418 -lt 150323855416 2024-09-15T21:04:32.534 INFO:tasks.workunit.client.0.smithi136.stderr:211860: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T21:04:32.534 INFO:tasks.workunit.client.0.smithi136.stderr:211860: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 7-171798691889 2024-09-15T21:04:32.534 INFO:tasks.workunit.client.0.smithi136.stderr:211860: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T21:04:32.534 INFO:tasks.workunit.client.0.smithi136.stderr:211860: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=7 2024-09-15T21:04:32.534 INFO:tasks.workunit.client.0.smithi136.stderr:211860: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 7-171798691889 2024-09-15T21:04:32.534 INFO:tasks.workunit.client.0.smithi136.stderr:211860: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T21:04:32.534 INFO:tasks.workunit.client.0.smithi136.stderr:211860: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=171798691889 2024-09-15T21:04:32.535 INFO:tasks.workunit.client.0.smithi136.stderr:211860: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.7 seq 171798691889' 2024-09-15T21:04:32.535 INFO:tasks.workunit.client.0.smithi136.stderr:211860: waiting osd.7 seq 171798691889 2024-09-15T21:04:32.535 INFO:tasks.workunit.client.0.smithi136.stderr:211860: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 7 2024-09-15T21:04:32.535 INFO:tasks.workunit.client.0.smithi136.stderr:211860: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 171798691890 -lt 171798691889 2024-09-15T21:04:32.535 INFO:tasks.workunit.client.0.smithi136.stderr:211860: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T21:04:32.535 INFO:tasks.workunit.client.0.smithi136.stderr:211860: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 8-193273528364 2024-09-15T21:04:32.535 INFO:tasks.workunit.client.0.smithi136.stderr:211860: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T21:04:32.535 INFO:tasks.workunit.client.0.smithi136.stderr:211860: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=8 2024-09-15T21:04:32.536 INFO:tasks.workunit.client.0.smithi136.stderr:211860: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 8-193273528364 2024-09-15T21:04:34.137 INFO:tasks.workunit.client.0.smithi136.stderr:211860: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cuut -d - -f 2 2024-09-15T21:04:34.137 INFO:tasks.workunit.client.0.smithi136.stderr:211858: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=193273528365 2024-09-15T21:04:34.137 INFO:tasks.workunit.client.0.smithi136.stderr:211858: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.8 seq 193273528365' 2024-09-15T21:04:34.137 INFO:tasks.workunit.client.0.smithi136.stderr:211858: waiting osd.8 seq 193273528365 2024-09-15T21:04:34.137 INFO:tasks.workunit.client.0.smithi136.stderr:211858: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 8 2024-09-15T21:04:34.137 INFO:tasks.workunit.client.0.smithi136.stderr:211858: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 193273528366 -lt 193273528365 2024-09-15T21:04:34.137 INFO:tasks.workunit.client.0.smithi136.stderr:211858: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T21:04:34.138 INFO:tasks.workunit.client.0.smithi136.stderr:211858: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 9-412316860418 2024-09-15T21:04:34.138 INFO:tasks.workunit.client.0.smithi136.stderr:211858: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T21:04:34.138 INFO:tasks.workunit.client.0.smithi136.stderr:211858: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=9 2024-09-15T21:04:34.138 INFO:tasks.workunit.client.0.smithi136.stderr:211858: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 9-412316860418 2024-09-15T21:04:34.138 INFO:tasks.workunit.client.0.smithi136.stderr:211858: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T21:04:34.138 INFO:tasks.workunit.client.0.smithi136.stderr:211858: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=412316860418 2024-09-15T21:04:34.138 INFO:tasks.workunit.client.0.smithi136.stderr:211858: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.9 seq 412316860418' 2024-09-15T21:04:34.138 INFO:tasks.workunit.client.0.smithi136.stderr:211858: waiting osd.9 seq 412316860418 2024-09-15T21:04:34.138 INFO:tasks.workunit.client.0.smithi136.stderr:211858: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 9 2024-09-15T21:04:34.138 INFO:tasks.workunit.client.0.smithi136.stderr:211858: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 412316860420 -lt 412316860418 2024-09-15T21:04:34.138 INFO:tasks.workunit.client.0.smithi136.stderr:211858: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-15T21:04:34.139 INFO:tasks.workunit.client.0.smithi136.stderr:211858: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T21:04:34.139 INFO:tasks.workunit.client.0.smithi136.stderr:211858: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T21:04:34.139 INFO:tasks.workunit.client.0.smithi136.stderr:211858: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 5 == 0 2024-09-15T21:04:34.139 INFO:tasks.workunit.client.0.smithi136.stderr:211858: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-15T21:04:34.139 INFO:tasks.workunit.client.0.smithi136.stderr:211858: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-15T21:04:34.139 INFO:tasks.workunit.client.0.smithi136.stderr:211858: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-15T21:04:34.139 INFO:tasks.workunit.client.0.smithi136.stderr:211858: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T21:04:34.139 INFO:tasks.workunit.client.0.smithi136.stderr:211858: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T21:04:34.139 INFO:tasks.workunit.client.0.smithi136.stderr:211858: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T21:04:34.140 INFO:tasks.workunit.client.0.smithi136.stderr:211858: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T21:04:34.140 INFO:tasks.workunit.client.0.smithi136.stderr:211858: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=5 2024-09-15T21:04:34.140 INFO:tasks.workunit.client.0.smithi136.stderr:211858: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-15T21:04:34.140 INFO:tasks.workunit.client.0.smithi136.stderr:211858: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T21:04:34.140 INFO:tasks.workunit.client.0.smithi136.stderr:211858: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T21:04:34.140 INFO:tasks.workunit.client.0.smithi136.stderr:211858: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 = 5 2024-09-15T21:04:34.140 INFO:tasks.workunit.client.0.smithi136.stderr:211858: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-15T21:04:34.140 INFO:tasks.workunit.client.0.smithi136.stderr:211858: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-15T21:04:34.140 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: run_in_background: return 0 2024-09-15T21:04:34.140 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: wait_background: for pid in $pids 2024-09-15T21:04:34.141 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2175: wait_background: wait 211857 2024-09-15T21:04:34.217 INFO:tasks.workunit.client.0.smithi136.stderr:t -d - -f 2 2024-09-15T21:04:34.218 INFO:tasks.workunit.client.0.smithi136.stderr:211860: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=193273528364 2024-09-15T21:04:34.218 INFO:tasks.workunit.client.0.smithi136.stderr:211860: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.8 seq 193273528364' 2024-09-15T21:04:34.218 INFO:tasks.workunit.client.0.smithi136.stderr:211860: waiting osd.8 seq 193273528364 2024-09-15T21:04:34.218 INFO:tasks.workunit.client.0.smithi136.stderr:211860: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 8 2024-09-15T21:04:34.218 INFO:tasks.workunit.client.0.smithi136.stderr:211860: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 193273528366 -lt 193273528364 2024-09-15T21:04:34.218 INFO:tasks.workunit.client.0.smithi136.stderr:211860: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T21:04:34.218 INFO:tasks.workunit.client.0.smithi136.stderr:211860: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 9-412316860419 2024-09-15T21:04:34.218 INFO:tasks.workunit.client.0.smithi136.stderr:211860: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T21:04:34.218 INFO:tasks.workunit.client.0.smithi136.stderr:211860: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=9 2024-09-15T21:04:34.218 INFO:tasks.workunit.client.0.smithi136.stderr:211860: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 9-412316860419 2024-09-15T21:04:34.218 INFO:tasks.workunit.client.0.smithi136.stderr:211860: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T21:04:34.218 INFO:tasks.workunit.client.0.smithi136.stderr:211860: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=412316860419 2024-09-15T21:04:34.218 INFO:tasks.workunit.client.0.smithi136.stderr:211860: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.9 seq 412316860419' 2024-09-15T21:04:34.219 INFO:tasks.workunit.client.0.smithi136.stderr:211860: waiting osd.9 seq 412316860419 2024-09-15T21:04:34.219 INFO:tasks.workunit.client.0.smithi136.stderr:211860: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 9 2024-09-15T21:04:34.219 INFO:tasks.workunit.client.0.smithi136.stderr:211860: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 412316860420 -lt 412316860419 2024-09-15T21:04:34.219 INFO:tasks.workunit.client.0.smithi136.stderr:211860: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-15T21:04:34.219 INFO:tasks.workunit.client.0.smithi136.stderr:211860: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T21:04:34.219 INFO:tasks.workunit.client.0.smithi136.stderr:211860: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T21:04:34.219 INFO:tasks.workunit.client.0.smithi136.stderr:211860: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 5 == 0 2024-09-15T21:04:34.219 INFO:tasks.workunit.client.0.smithi136.stderr:211860: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-15T21:04:34.219 INFO:tasks.workunit.client.0.smithi136.stderr:211860: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-15T21:04:34.219 INFO:tasks.workunit.client.0.smithi136.stderr:211860: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-15T21:04:34.219 INFO:tasks.workunit.client.0.smithi136.stderr:211860: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T21:04:34.219 INFO:tasks.workunit.client.0.smithi136.stderr:211860: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T21:04:34.219 INFO:tasks.workunit.client.0.smithi136.stderr:211860: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T21:04:34.220 INFO:tasks.workunit.client.0.smithi136.stderr:211860: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T21:04:34.220 INFO:tasks.workunit.client.0.smithi136.stderr:211860: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=5 2024-09-15T21:04:34.220 INFO:tasks.workunit.client.0.smithi136.stderr:211860: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-15T21:04:34.220 INFO:tasks.workunit.client.0.smithi136.stderr:211860: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T21:04:34.220 INFO:tasks.workunit.client.0.smithi136.stderr:211860: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T21:04:34.220 INFO:tasks.workunit.client.0.smithi136.stderr:211860: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 = 5 2024-09-15T21:04:34.220 INFO:tasks.workunit.client.0.smithi136.stderr:211860: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-15T21:04:34.220 INFO:tasks.workunit.client.0.smithi136.stderr:211860: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-15T21:04:34.220 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: run_in_background: return 0 2024-09-15T21:04:34.220 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: wait_background: eval 'pids='\'''\''' 2024-09-15T21:04:34.220 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: wait_background: pids= 2024-09-15T21:04:34.220 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: wait_background: return 0 2024-09-15T21:04:34.221 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:197: corrupt_and_repair_two: return_code=0 2024-09-15T21:04:34.221 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:198: corrupt_and_repair_two: '[' 0 -ne 0 ']' 2024-09-15T21:04:34.221 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:203: corrupt_and_repair_two: get_pg ecpool SOMETHING 2024-09-15T21:04:34.221 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1084: get_pg: local poolname=ecpool 2024-09-15T21:04:34.221 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1085: get_pg: local objectname=SOMETHING 2024-09-15T21:04:34.221 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1087: get_pg: ceph --format json osd map ecpool SOMETHING 2024-09-15T21:04:34.221 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1087: get_pg: jq -r .pgid 2024-09-15T21:04:34.502 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:203: corrupt_and_repair_two: local pg=2.0 2024-09-15T21:04:34.502 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:204: corrupt_and_repair_two: repair 2.0 2024-09-15T21:04:34.502 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1846: repair: local pgid=2.0 2024-09-15T21:04:34.502 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1847: repair: get_last_scrub_stamp 2.0 2024-09-15T21:04:34.502 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2024-09-15T21:04:34.502 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-15T21:04:34.502 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-15T21:04:34.502 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-09-15T21:04:34.777 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1847: repair: local last_scrub=2024-09-15T21:03:58.250680+0000 2024-09-15T21:04:34.777 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1848: repair: ceph pg repair 2.0 2024-09-15T21:04:35.068 INFO:tasks.workunit.client.0.smithi136.stderr:instructing pg 2.0s0 on osd.3 to repair 2024-09-15T21:04:35.081 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1849: repair: wait_for_scrub 2.0 2024-09-15T21:03:58.250680+0000 2024-09-15T21:04:35.081 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: local pgid=2.0 2024-09-15T21:04:35.081 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2018: wait_for_scrub: local last_scrub=2024-09-15T21:03:58.250680+0000 2024-09-15T21:04:35.081 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2019: wait_for_scrub: local sname=last_scrub_stamp 2024-09-15T21:04:35.081 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i=0 )) 2024-09-15T21:04:35.081 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-15T21:04:35.081 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-09-15T21:04:35.081 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2024-09-15T21:04:35.081 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-15T21:04:35.081 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-15T21:04:35.082 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-09-15T21:04:35.358 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-15T21:03:58.250680+0000 '>' 2024-09-15T21:03:58.250680+0000 2024-09-15T21:04:35.358 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-15T21:04:36.359 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-15T21:04:36.359 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-15T21:04:36.360 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-09-15T21:04:36.360 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2024-09-15T21:04:36.360 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-15T21:04:36.361 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-15T21:04:36.361 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-09-15T21:04:36.643 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-15T21:03:58.250680+0000 '>' 2024-09-15T21:03:58.250680+0000 2024-09-15T21:04:36.644 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-15T21:04:37.645 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-15T21:04:37.645 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-15T21:04:37.645 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-09-15T21:04:37.645 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2024-09-15T21:04:37.646 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-15T21:04:37.646 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-15T21:04:37.646 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-09-15T21:04:37.929 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-15T21:04:35.276431+0000 '>' 2024-09-15T21:03:58.250680+0000 2024-09-15T21:04:37.929 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2023: wait_for_scrub: return 0 2024-09-15T21:04:37.929 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1850: repair: sleep 2 2024-09-15T21:04:39.932 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:208: corrupt_and_repair_two: pids= 2024-09-15T21:04:39.932 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:209: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 5 SOMETHING list-attrs 2024-09-15T21:04:39.932 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2135: run_in_background: local pid_variable=pids 2024-09-15T21:04:39.932 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2136: run_in_background: shift 2024-09-15T21:04:39.932 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2140: run_in_background: eval 'pids+=" 215281"' 2024-09-15T21:04:39.932 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2140: run_in_background: pids+=' 215281' 2024-09-15T21:04:39.932 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:210: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 9 SOMETHING list-attrs 2024-09-15T21:04:39.932 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2135: run_in_background: local pid_variable=pids 2024-09-15T21:04:39.932 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2136: run_in_background: shift 2024-09-15T21:04:39.933 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2140: run_in_background: eval 'pids+=" 215283"' 2024-09-15T21:04:39.933 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: run_in_background: objectstore_tool td/osd-scrub-repair 5 SOMETHING list-attrs 2024-09-15T21:04:39.933 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2140: run_in_background: pids+=' 215283' 2024-09-15T21:04:39.933 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: run_in_background: sed 's/^/215284: /' 2024-09-15T21:04:39.933 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:211: corrupt_and_repair_two: wait_background pids 2024-09-15T21:04:39.933 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: wait_background: pids=' 215281 215283' 2024-09-15T21:04:39.933 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: wait_background: return_code=0 2024-09-15T21:04:39.933 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: wait_background: for pid in $pids 2024-09-15T21:04:39.933 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2175: wait_background: wait 215281 2024-09-15T21:04:39.933 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: run_in_background: objectstore_tool td/osd-scrub-repair 9 SOMETHING list-attrs 2024-09-15T21:04:39.933 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: run_in_background: sed 's/^/215286: /' 2024-09-15T21:04:41.329 INFO:tasks.workunit.client.0.smithi136.stderr:215284: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2024-09-15T21:04:41.329 INFO:tasks.workunit.client.0.smithi136.stderr:215284: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2024-09-15T21:04:41.329 INFO:tasks.workunit.client.0.smithi136.stderr:215284: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=5 2024-09-15T21:04:41.329 INFO:tasks.workunit.client.0.smithi136.stderr:215284: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2024-09-15T21:04:41.329 INFO:tasks.workunit.client.0.smithi136.stderr:215284: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 5 SOMETHING list-attrs 2024-09-15T21:04:41.329 INFO:tasks.workunit.client.0.smithi136.stderr:215284: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-09-15T21:04:41.329 INFO:tasks.workunit.client.0.smithi136.stderr:215284: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2024-09-15T21:04:41.330 INFO:tasks.workunit.client.0.smithi136.stderr:215284: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=5 2024-09-15T21:04:41.330 INFO:tasks.workunit.client.0.smithi136.stderr:215284: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2024-09-15T21:04:41.330 INFO:tasks.workunit.client.0.smithi136.stderr:215284: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.5 2024-09-15T21:04:41.330 INFO:tasks.workunit.client.0.smithi136.stderr:215284: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-15T21:04:41.330 INFO:tasks.workunit.client.0.smithi136.stderr:215284: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-15T21:04:41.330 INFO:tasks.workunit.client.0.smithi136.stderr:215284: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-15T21:04:41.330 INFO:tasks.workunit.client.0.smithi136.stderr:215284: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-15T21:04:41.330 INFO:tasks.workunit.client.0.smithi136.stderr:215284: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-15T21:04:41.330 INFO:tasks.workunit.client.0.smithi136.stderr:215284: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-15T21:04:41.330 INFO:tasks.workunit.client.0.smithi136.stderr:215284: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 5 SOMETHING list-attrs 2024-09-15T21:04:41.330 INFO:tasks.workunit.client.0.smithi136.stderr:215284: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-15T21:04:41.330 INFO:tasks.workunit.client.0.smithi136.stderr:215284: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-15T21:04:41.330 INFO:tasks.workunit.client.0.smithi136.stderr:215284: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=5 2024-09-15T21:04:41.330 INFO:tasks.workunit.client.0.smithi136.stderr:215284: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-15T21:04:41.330 INFO:tasks.workunit.client.0.smithi136.stderr:215284: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/5 2024-09-15T21:04:41.331 INFO:tasks.workunit.client.0.smithi136.stderr:215284: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/5 SOMETHING list-attrs 2024-09-15T21:04:41.331 INFO:tasks.workunit.client.0.smithi136.stderr:215284: _ 2024-09-15T21:04:41.331 INFO:tasks.workunit.client.0.smithi136.stderr:215284: hinfo_key 2024-09-15T21:04:41.331 INFO:tasks.workunit.client.0.smithi136.stderr:215284: snapset 2024-09-15T21:04:41.331 INFO:tasks.workunit.client.0.smithi136.stderr:215284: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 5 2024-09-15T21:04:41.331 INFO:tasks.workunit.client.0.smithi136.stderr:215284: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2024-09-15T21:04:41.331 INFO:tasks.workunit.client.0.smithi136.stderr:215284: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-15T21:04:41.331 INFO:tasks.workunit.client.0.smithi136.stderr:215284: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=5 2024-09-15T21:04:41.331 INFO:tasks.workunit.client.0.smithi136.stderr:215284: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-15T21:04:41.331 INFO:tasks.workunit.client.0.smithi136.stderr:215284: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/5 2024-09-15T21:04:41.331 INFO:tasks.workunit.client.0.smithi136.stderr:215284: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=ce4892c7-6c6d-4b4f-8e6f-07a3e11133e5 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-15T21:04:41.331 INFO:tasks.workunit.client.0.smithi136.stderr:215284: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T21:04:41.332 INFO:tasks.workunit.client.0.smithi136.stderr:215284: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T21:04:41.332 INFO:tasks.workunit.client.0.smithi136.stderr:215284: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T21:04:41.332 INFO:tasks.workunit.client.0.smithi136.stderr:215284: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/5' 2024-09-15T21:04:41.332 INFO:tasks.workunit.client.0.smithi136.stderr:215284: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/5/journal' 2024-09-15T21:04:41.333 INFO:tasks.workunit.client.0.smithi136.stderr:215284: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-15T21:04:41.333 INFO:tasks.workunit.client.0.smithi136.stderr:215284: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-15T21:04:41.333 INFO:tasks.workunit.client.0.smithi136.stderr:215284: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-15T21:04:41.333 INFO:tasks.workunit.client.0.smithi136.stderr:215284: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-15T21:04:41.333 INFO:tasks.workunit.client.0.smithi136.stderr:215284: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-15T21:04:41.333 INFO:tasks.workunit.client.0.smithi136.stderr:215284: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-15T21:04:41.333 INFO:tasks.workunit.client.0.smithi136.stderr:215284: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-15T21:04:41.333 INFO:tasks.workunit.client.0.smithi136.stderr:215284: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T21:04:41.333 INFO:tasks.workunit.client.0.smithi136.stderr:215284: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52602 2024-09-15T21:04:41.333 INFO:tasks.workunit.client.0.smithi136.stderr:215284: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.52602/$cluster-$name.asok' 2024-09-15T21:04:41.333 INFO:tasks.workunit.client.0.smithi136.stderr:215284: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.52602/$cluster-$name.asok' 2024-09-15T21:04:41.333 INFO:tasks.workunit.client.0.smithi136.stderr:215284: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-15T21:04:41.333 INFO:tasks.workunit.client.0.smithi136.stderr:215284: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-15T21:04:41.334 INFO:tasks.workunit.client.0.smithi136.stderr:215284: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-15T21:04:41.334 INFO:tasks.workunit.client.0.smithi136.stderr:215284: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T21:04:41.334 INFO:tasks.workunit.client.0.smithi136.stderr:215284: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T21:04:41.334 INFO:tasks.workunit.client.0.smithi136.stderr:215284: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T21:04:41.334 INFO:tasks.workunit.client.0.smithi136.stderr:215284: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T21:04:41.334 INFO:tasks.workunit.client.0.smithi136.stderr:215284: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-15T21:04:41.334 INFO:tasks.workunit.client.0.smithi136.stderr:215284: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-15T21:04:41.334 INFO:tasks.workunit.client.0.smithi136.stderr:215284: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/5 2024-09-15T21:04:41.334 INFO:tasks.workunit.client.0.smithi136.stderr:215284: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.5 2024-09-15T21:04:41.334 INFO:tasks.workunit.client.0.smithi136.stderr:215284: start osd.5 2024-09-15T21:04:41.345 INFO:tasks.workunit.client.0.smithi136.stderr:215284: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 5 --fsid=ce4892c7-6c6d-4b4f-8e6f-07a3e11133e5 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/5 --osd-journal=td/osd-scrub-repair/5/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.52602/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_reco215286: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2024-09-15T21:04:41.345 INFO:tasks.workunit.client.0.smithi136.stderr:215286: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2024-09-15T21:04:41.345 INFO:tasks.workunit.client.0.smithi136.stderr:215286: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=9 2024-09-15T21:04:41.345 INFO:tasks.workunit.client.0.smithi136.stderr:215286: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2024-09-15T21:04:41.345 INFO:tasks.workunit.client.0.smithi136.stderr:215286: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 9 SOMETHING list-attrs 2024-09-15T21:04:41.345 INFO:tasks.workunit.client.0.smithi136.stderr:215286: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-09-15T21:04:41.345 INFO:tasks.workunit.client.0.smithi136.stderr:215286: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2024-09-15T21:04:41.346 INFO:tasks.workunit.client.0.smithi136.stderr:215286: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=9 2024-09-15T21:04:41.346 INFO:tasks.workunit.client.0.smithi136.stderr:215286: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2024-09-15T21:04:41.346 INFO:tasks.workunit.client.0.smithi136.stderr:215286: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.9 2024-09-15T21:04:41.346 INFO:tasks.workunit.client.0.smithi136.stderr:215286: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-15T21:04:41.346 INFO:tasks.workunit.client.0.smithi136.stderr:215286: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-15T21:04:41.346 INFO:tasks.workunit.client.0.smithi136.stderr:215286: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-15T21:04:41.346 INFO:tasks.workunit.client.0.smithi136.stderr:215286: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-15T21:04:41.346 INFO:tasks.workunit.client.0.smithi136.stderr:215286: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-15T21:04:41.346 INFO:tasks.workunit.client.0.smithi136.stderr:215286: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-15T21:04:41.346 INFO:tasks.workunit.client.0.smithi136.stderr:215286: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 9 SOMETHING list-attrs 2024-09-15T21:04:41.346 INFO:tasks.workunit.client.0.smithi136.stderr:215286: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-15T21:04:41.346 INFO:tasks.workunit.client.0.smithi136.stderr:215286: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-15T21:04:41.346 INFO:tasks.workunit.client.0.smithi136.stderr:215286: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=9 2024-09-15T21:04:41.347 INFO:tasks.workunit.client.0.smithi136.stderr:215286: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-15T21:04:41.347 INFO:tasks.workunit.client.0.smithi136.stderr:215286: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/9 2024-09-15T21:04:41.347 INFO:tasks.workunit.client.0.smithi136.stderr:215286: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/9 SOMETHING list-attrs 2024-09-15T21:04:41.347 INFO:tasks.workunit.client.0.smithi136.stderr:215286: _ 2024-09-15T21:04:41.347 INFO:tasks.workunit.client.0.smithi136.stderr:215286: hinfo_key 2024-09-15T21:04:41.347 INFO:tasks.workunit.client.0.smithi136.stderr:215286: snapset 2024-09-15T21:04:41.347 INFO:tasks.workunit.client.0.smithi136.stderr:215286: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 9 2024-09-15T21:04:41.347 INFO:tasks.workunit.client.0.smithi136.stderr:215286: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2024-09-15T21:04:41.347 INFO:tasks.workunit.client.0.smithi136.stderr:215286: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-15T21:04:41.347 INFO:tasks.workunit.client.0.smithi136.stderr:215286: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=9 2024-09-15T21:04:41.347 INFO:tasks.workunit.client.0.smithi136.stderr:215286: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-15T21:04:41.347 INFO:tasks.workunit.client.0.smithi136.stderr:215286: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/9 2024-09-15T21:04:41.347 INFO:tasks.workunit.client.0.smithi136.stderr:215286: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=ce4892c7-6c6d-4b4f-8e6f-07a3e11133e5 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-15T21:04:41.347 INFO:tasks.workunit.client.0.smithi136.stderr:215286: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T21:04:41.348 INFO:tasks.workunit.client.0.smithi136.stderr:215286: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T21:04:41.348 INFO:tasks.workunit.client.0.smithi136.stderr:215286: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T21:04:41.348 INFO:tasks.workunit.client.0.smithi136.stderr:215286: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/9' 2024-09-15T21:04:41.348 INFO:tasks.workunit.client.0.smithi136.stderr:215286: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/9/journal' 2024-09-15T21:04:41.348 INFO:tasks.workunit.client.0.smithi136.stderr:215286: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-15T21:04:41.349 INFO:tasks.workunit.client.0.smithi136.stderr:215286: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-15T21:04:41.349 INFO:tasks.workunit.client.0.smithi136.stderr:215286: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-15T21:04:41.349 INFO:tasks.workunit.client.0.smithi136.stderr:215286: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-15T21:04:41.349 INFO:tasks.workunit.client.0.smithi136.stderr:215286: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-15T21:04:41.349 INFO:tasks.workunit.client.0.smithi136.stderr:215286: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-15T21:04:41.349 INFO:tasks.workunit.client.0.smithi136.stderr:215286: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-15T21:04:41.349 INFO:tasks.workunit.client.0.smithi136.stderr:215286: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T21:04:41.349 INFO:tasks.workunit.client.0.smithi136.stderr:215286: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52602 2024-09-15T21:04:41.349 INFO:tasks.workunit.client.0.smithi136.stderr:215286: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.52602/$cluster-$name.asok' 2024-09-15T21:04:41.349 INFO:tasks.workunit.client.0.smithi136.stderr:215286: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.52602/$cluster-$name.asok' 2024-09-15T21:04:41.349 INFO:tasks.workunit.client.0.smithi136.stderr:215286: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-15T21:04:41.349 INFO:tasks.workunit.client.0.smithi136.stderr:215286: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-15T21:04:41.350 INFO:tasks.workunit.client.0.smithi136.stderr:215286: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-15T21:04:41.350 INFO:tasks.workunit.client.0.smithi136.stderr:215286: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T21:04:41.350 INFO:tasks.workunit.client.0.smithi136.stderr:215286: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T21:04:41.350 INFO:tasks.workunit.client.0.smithi136.stderr:215286: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T21:04:41.350 INFO:tasks.workunit.client.0.smithi136.stderr:215286: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T21:04:41.350 INFO:tasks.workunit.client.0.smithi136.stderr:215286: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-15T21:04:41.350 INFO:tasks.workunit.client.0.smithi136.stderr:215286: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-15T21:04:41.350 INFO:tasks.workunit.client.0.smithi136.stderr:215286: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/9 2024-09-15T21:04:41.350 INFO:tasks.workunit.client.0.smithi136.stderr:215286: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.9 2024-09-15T21:04:41.350 INFO:tasks.workunit.client.0.smithi136.stderr:215286: start osd.9 2024-09-15T21:04:46.965 INFO:tasks.workunit.client.0.smithi136.stderr:215286: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 9 --fsid=ce4892c7-6c6d-4b4f-8e6f-07a3e11133e5 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/9 --osd-journal=td/osd-scrub-repair/9/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.52602/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T21:04:46.965 INFO:tasks.workunit.client.0.smithi136.stderr:215284: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/5/whoami 2024-09-15T21:04:46.965 INFO:tasks.workunit.client.0.smithi136.stderr:215284: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 5 = 5 ']' 2024-09-15T21:04:46.965 INFO:tasks.workunit.client.0.smithi136.stderr:215284: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-15T21:04:46.965 INFO:tasks.workunit.client.0.smithi136.stderr:215284: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-15T21:04:46.965 INFO:tasks.workunit.client.0.smithi136.stderr:215284: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-15T21:04:46.965 INFO:tasks.workunit.client.0.smithi136.stderr:215284: 2024-09-15T21:04:41.362+0000 7f0e47013640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:04:46.965 INFO:tasks.workunit.client.0.smithi136.stderr:215284: 2024-09-15T21:04:41.438+0000 7f0e47013640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:04:46.965 INFO:tasks.workunit.client.0.smithi136.stderr:215284: 2024-09-15T21:04:41.440+0000 7f0e47013640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:04:46.965 INFO:tasks.workunit.client.0.smithi136.stderr:215284: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 5 2024-09-15T21:04:46.966 INFO:tasks.workunit.client.0.smithi136.stderr:215284: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-15T21:04:46.966 INFO:tasks.workunit.client.0.smithi136.stderr:215284: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=5 2024-09-15T21:04:46.966 INFO:tasks.workunit.client.0.smithi136.stderr:215284: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-15T21:04:46.966 INFO:tasks.workunit.client.0.smithi136.stderr:215284: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-15T21:04:46.966 INFO:tasks.workunit.client.0.smithi136.stderr:215284: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:04:46.966 INFO:tasks.workunit.client.0.smithi136.stderr:215284: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-15T21:04:46.966 INFO:tasks.workunit.client.0.smithi136.stderr:215284: 0 2024-09-15T21:04:46.966 INFO:tasks.workunit.client.0.smithi136.stderr:215284: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:04:46.966 INFO:tasks.workunit.client.0.smithi136.stderr:215284: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-15T21:04:46.966 INFO:tasks.workunit.client.0.smithi136.stderr:215284: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:04:46.966 INFO:tasks.workunit.client.0.smithi136.stderr:215284: 2024-09-15T21:04:43.039+0000 7f0e47013640 -1 Falling back to public interface 2024-09-15T21:04:46.966 INFO:tasks.workunit.client.0.smithi136.stderr:215284: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:04:46.966 INFO:tasks.workunit.client.0.smithi136.stderr:215284: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:04:46.967 INFO:tasks.workunit.client.0.smithi136.stderr:215284: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-15T21:04:46.967 INFO:tasks.workunit.client.0.smithi136.stderr:215284: 1 2024-09-15T21:04:46.967 INFO:tasks.workunit.client.0.smithi136.stderr:215284: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:04:46.967 INFO:tasks.workunit.client.0.smithi136.stderr:215284: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-15T21:04:46.967 INFO:tasks.workunit.client.0.smithi136.stderr:215284: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:04:46.967 INFO:tasks.workunit.client.0.smithi136.stderr:215284: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:04:46.967 INFO:tasks.workunit.client.0.smithi136.stderr:215284: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:04:46.967 INFO:tasks.workunit.client.0.smithi136.stderr:215284: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-15T21:04:46.967 INFO:tasks.workunit.client.0.smithi136.stderr:215284: 2 2024-09-15T21:04:46.967 INFO:tasks.workunit.client.0.smithi136.stderr:215284: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:04:46.967 INFO:tasks.workunit.client.0.smithi136.stderr:215284: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-15T21:04:46.967 INFO:tasks.workunit.client.0.smithi136.stderr:215284: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:04:46.967 INFO:tasks.workunit.client.0.smithi136.stderr:215284: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:04:46.967 INFO:tasks.workunit.client.0.smithi136.stderr:215284: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:04:46.968 INFO:tasks.workunit.client.0.smithi136.stderr:215284: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-15T21:04:46.968 INFO:tasks.workunit.client.0.smithi136.stderr:215284: 3 2024-09-15T21:04:46.968 INFO:tasks.workunit.client.0.smithi136.stderr:215284: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:04:46.968 INFO:tasks.workunit.client.0.smithi136.stderr:215284: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-15T21:04:46.968 INFO:tasks.workunit.client.0.smithi136.stderr:215284: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:04:46.994 INFO:tasks.workunit.client.0.smithi136.stderr:215284: 2024-09-15T21:04:46.961+0000 7f0every_ops 2024-09-15T21:04:46.995 INFO:tasks.workunit.client.0.smithi136.stderr:215286: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/9/whoami 2024-09-15T21:04:46.995 INFO:tasks.workunit.client.0.smithi136.stderr:215286: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 9 = 9 ']' 2024-09-15T21:04:46.995 INFO:tasks.workunit.client.0.smithi136.stderr:215286: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-15T21:04:46.995 INFO:tasks.workunit.client.0.smithi136.stderr:215286: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-15T21:04:46.995 INFO:tasks.workunit.client.0.smithi136.stderr:215286: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-15T21:04:46.995 INFO:tasks.workunit.client.0.smithi136.stderr:215286: 2024-09-15T21:04:41.375+0000 7fae7c418640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:04:46.995 INFO:tasks.workunit.client.0.smithi136.stderr:215286: 2024-09-15T21:04:41.439+0000 7fae7c418640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:04:46.995 INFO:tasks.workunit.client.0.smithi136.stderr:215286: 2024-09-15T21:04:41.440+0000 7fae7c418640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:04:46.995 INFO:tasks.workunit.client.0.smithi136.stderr:215286: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 9 2024-09-15T21:04:46.995 INFO:tasks.workunit.client.0.smithi136.stderr:215286: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-15T21:04:46.995 INFO:tasks.workunit.client.0.smithi136.stderr:215286: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=9 2024-09-15T21:04:46.995 INFO:tasks.workunit.client.0.smithi136.stderr:215286: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-15T21:04:46.996 INFO:tasks.workunit.client.0.smithi136.stderr:215286: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-15T21:04:46.996 INFO:tasks.workunit.client.0.smithi136.stderr:215286: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:04:46.996 INFO:tasks.workunit.client.0.smithi136.stderr:215286: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-15T21:04:46.996 INFO:tasks.workunit.client.0.smithi136.stderr:215286: 0 2024-09-15T21:04:46.996 INFO:tasks.workunit.client.0.smithi136.stderr:215286: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:04:46.996 INFO:tasks.workunit.client.0.smithi136.stderr:215286: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.9 up' 2024-09-15T21:04:46.996 INFO:tasks.workunit.client.0.smithi136.stderr:215286: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:04:46.996 INFO:tasks.workunit.client.0.smithi136.stderr:215286: 2024-09-15T21:04:42.801+0000 7fae7c418640 -1 Falling back to public interface 2024-09-15T21:04:46.996 INFO:tasks.workunit.client.0.smithi136.stderr:215286: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:04:46.996 INFO:tasks.workunit.client.0.smithi136.stderr:215286: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:04:46.996 INFO:tasks.workunit.client.0.smithi136.stderr:215286: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-15T21:04:46.996 INFO:tasks.workunit.client.0.smithi136.stderr:215286: 1 2024-09-15T21:04:46.997 INFO:tasks.workunit.client.0.smithi136.stderr:215286: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:04:46.997 INFO:tasks.workunit.client.0.smithi136.stderr:215286: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.9 up' 2024-09-15T21:04:46.997 INFO:tasks.workunit.client.0.smithi136.stderr:215286: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:04:46.997 INFO:tasks.workunit.client.0.smithi136.stderr:215286: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:04:46.997 INFO:tasks.workunit.client.0.smithi136.stderr:215286: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:04:46.997 INFO:tasks.workunit.client.0.smithi136.stderr:215286: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-15T21:04:46.997 INFO:tasks.workunit.client.0.smithi136.stderr:215286: 2 2024-09-15T21:04:46.997 INFO:tasks.workunit.client.0.smithi136.stderr:215286: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:04:46.997 INFO:tasks.workunit.client.0.smithi136.stderr:215286: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.9 up' 2024-09-15T21:04:46.997 INFO:tasks.workunit.client.0.smithi136.stderr:215286: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:04:46.997 INFO:tasks.workunit.client.0.smithi136.stderr:215286: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:04:46.997 INFO:tasks.workunit.client.0.smithi136.stderr:215286: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:04:46.997 INFO:tasks.workunit.client.0.smithi136.stderr:215286: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-15T21:04:46.997 INFO:tasks.workunit.client.0.smithi136.stderr:215286: 3 2024-09-15T21:04:46.998 INFO:tasks.workunit.client.0.smithi136.stderr:215286: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:04:46.998 INFO:tasks.workunit.client.0.smithi136.stderr:215286: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.9 up' 2024-09-15T21:04:46.998 INFO:tasks.workunit.client.0.smithi136.stderr:215286: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:04:49.150 INFO:tasks.workunit.client.0.smithi136.stderr:215286: /home/ubuntu/cephtest/clone.clien47013640 -1 osd.5 98 log_to_monitors true 2024-09-15T21:04:49.150 INFO:tasks.workunit.client.0.smithi136.stderr:215284: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:04:49.150 INFO:tasks.workunit.client.0.smithi136.stderr:215284: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:04:49.150 INFO:tasks.workunit.client.0.smithi136.stderr:215284: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-15T21:04:49.150 INFO:tasks.workunit.client.0.smithi136.stderr:215284: 4 2024-09-15T21:04:49.150 INFO:tasks.workunit.client.0.smithi136.stderr:215284: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:04:49.150 INFO:tasks.workunit.client.0.smithi136.stderr:215284: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-15T21:04:49.150 INFO:tasks.workunit.client.0.smithi136.stderr:215284: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:04:49.151 INFO:tasks.workunit.client.0.smithi136.stderr:215284: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:04:49.151 INFO:tasks.workunit.client.0.smithi136.stderr:215284: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:04:49.151 INFO:tasks.workunit.client.0.smithi136.stderr:215284: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-15T21:04:49.151 INFO:tasks.workunit.client.0.smithi136.stderr:215284: 5 2024-09-15T21:04:49.151 INFO:tasks.workunit.client.0.smithi136.stderr:215284: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:04:49.151 INFO:tasks.workunit.client.0.smithi136.stderr:215284: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-15T21:04:49.151 INFO:tasks.workunit.client.0.smithi136.stderr:215284: osd.5 up in weight 1 up_from 104 up_thru 95 down_at 99 last_clean_interval [95,98) [v2:127.0.0.1:6874/3747108546,v1:127.0.0.1:6875/3747108546] [v2:127.0.0.1:6876/3747108546,v1:127.0.0.1:6877/3747108546] exists,up c3187161-e486-4b18-8257-d69c125e9792 2024-09-15T21:04:49.151 INFO:tasks.workunit.client.0.smithi136.stderr:215284: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-15T21:04:49.151 INFO:tasks.workunit.client.0.smithi136.stderr:215284: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-15T21:04:49.151 INFO:tasks.workunit.client.0.smithi136.stderr:215284: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-15T21:04:49.151 INFO:tasks.workunit.client.0.smithi136.stderr:215284: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2024-09-15T21:04:49.151 INFO:tasks.workunit.client.0.smithi136.stderr:215284: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-15T21:04:49.151 INFO:tasks.workunit.client.0.smithi136.stderr:215284: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-15T21:04:49.151 INFO:tasks.workunit.client.0.smithi136.stderr:215284: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-15T21:04:49.151 INFO:tasks.workunit.client.0.smithi136.stderr:215284: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-15T21:04:49.152 INFO:tasks.workunit.client.0.smithi136.stderr:215284: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-15T21:04:49.152 INFO:tasks.workunit.client.0.smithi136.stderr:215284: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-15T21:04:49.152 INFO:tasks.workunit.client.0.smithi136.stderr:215284: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-15T21:04:49.152 INFO:tasks.workunit.client.0.smithi136.stderr:215284: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-15T21:04:49.152 INFO:tasks.workunit.client.0.smithi136.stderr:215284: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-15T21:04:49.152 INFO:tasks.workunit.client.0.smithi136.stderr:215284: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-15T21:04:49.152 INFO:tasks.workunit.client.0.smithi136.stderr:215284: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-15T21:04:49.152 INFO:tasks.workunit.client.0.smithi136.stderr:215284: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-15T21:04:49.152 INFO:tasks.workunit.client.0.smithi136.stderr:215284: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-15T21:04:49.152 INFO:tasks.workunit.client.0.smithi136.stderr:215284: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-15T21:04:49.152 INFO:tasks.workunit.client.0.smithi136.stderr:215284: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-15T21:04:49.152 INFO:tasks.workunit.client.0.smithi136.stderr:215284: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-15T21:04:49.152 INFO:tasks.workunit.client.0.smithi136.stderr:215284: 1 2024-09-15T21:04:49.152 INFO:tasks.workunit.client.0.smithi136.stderr:215284: 2 2024-09-15T21:04:49.153 INFO:tasks.workunit.client.0.smithi136.stderr:215284: 3 2024-09-15T21:04:49.153 INFO:tasks.workunit.client.0.smithi136.stderr:215284: 4 2024-09-15T21:04:49.153 INFO:tasks.workunit.client.0.smithi136.stderr:215284: 5 2024-09-15T21:04:49.153 INFO:tasks.workunit.client.0.smithi136.stderr:215284: 6 2024-09-15T21:04:49.153 INFO:tasks.workunit.client.0.smithi136.stderr:215284: 7 2024-09-15T21:04:49.153 INFO:tasks.workunit.client.0.smithi136.stderr:215284: 8 2024-09-15T21:04:49.153 INFO:tasks.workunit.client.0.smithi136.stderr:215284: 9' 2024-09-15T21:04:49.153 INFO:tasks.workunit.client.0.smithi136.stderr:215284: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-15T21:04:49.153 INFO:tasks.workunit.client.0.smithi136.stderr:215284: /home/ubuntu/cephtest/clone.client.0/qa/standalone/t.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:04:49.153 INFO:tasks.workunit.client.0.smithi136.stderr:215286: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:04:49.153 INFO:tasks.workunit.client.0.smithi136.stderr:215286: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-15T21:04:49.153 INFO:tasks.workunit.client.0.smithi136.stderr:215286: 4 2024-09-15T21:04:49.153 INFO:tasks.workunit.client.0.smithi136.stderr:215286: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:04:49.153 INFO:tasks.workunit.client.0.smithi136.stderr:215286: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.9 up' 2024-09-15T21:04:49.153 INFO:tasks.workunit.client.0.smithi136.stderr:215286: 2024-09-15T21:04:47.109+0000 7fae7c418640 -1 osd.9 98 log_to_monitors true 2024-09-15T21:04:49.154 INFO:tasks.workunit.client.0.smithi136.stderr:215286: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:04:49.154 INFO:tasks.workunit.client.0.smithi136.stderr:215286: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:04:49.154 INFO:tasks.workunit.client.0.smithi136.stderr:215286: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:04:49.154 INFO:tasks.workunit.client.0.smithi136.stderr:215286: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-15T21:04:49.154 INFO:tasks.workunit.client.0.smithi136.stderr:215286: 5 2024-09-15T21:04:49.154 INFO:tasks.workunit.client.0.smithi136.stderr:215286: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:04:49.154 INFO:tasks.workunit.client.0.smithi136.stderr:215286: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.9 up' 2024-09-15T21:04:49.154 INFO:tasks.workunit.client.0.smithi136.stderr:215286: osd.9 up in weight 1 up_from 105 up_thru 0 down_at 99 last_clean_interval [96,98) [v2:127.0.0.1:6842/572731292,v1:127.0.0.1:6843/572731292] [v2:127.0.0.1:6844/572731292,v1:127.0.0.1:6845/572731292] exists,up ccfb463f-a1cc-4f3c-a7db-ae126efbeaa5 2024-09-15T21:04:49.154 INFO:tasks.workunit.client.0.smithi136.stderr:215286: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-15T21:04:49.154 INFO:tasks.workunit.client.0.smithi136.stderr:215286: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-15T21:04:49.154 INFO:tasks.workunit.client.0.smithi136.stderr:215286: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-15T21:04:49.154 INFO:tasks.workunit.client.0.smithi136.stderr:215286: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2024-09-15T21:04:49.154 INFO:tasks.workunit.client.0.smithi136.stderr:215286: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-15T21:04:49.154 INFO:tasks.workunit.client.0.smithi136.stderr:215286: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-15T21:04:49.154 INFO:tasks.workunit.client.0.smithi136.stderr:215286: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-15T21:04:49.155 INFO:tasks.workunit.client.0.smithi136.stderr:215286: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-15T21:04:49.155 INFO:tasks.workunit.client.0.smithi136.stderr:215286: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-15T21:04:49.155 INFO:tasks.workunit.client.0.smithi136.stderr:215286: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-15T21:04:49.155 INFO:tasks.workunit.client.0.smithi136.stderr:215286: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-15T21:04:49.155 INFO:tasks.workunit.client.0.smithi136.stderr:215286: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-15T21:04:49.155 INFO:tasks.workunit.client.0.smithi136.stderr:215286: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-15T21:04:49.155 INFO:tasks.workunit.client.0.smithi136.stderr:215286: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-15T21:04:49.155 INFO:tasks.workunit.client.0.smithi136.stderr:215286: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-15T21:04:49.155 INFO:tasks.workunit.client.0.smithi136.stderr:215286: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-15T21:04:49.155 INFO:tasks.workunit.client.0.smithi136.stderr:215286: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-15T21:04:49.155 INFO:tasks.workunit.client.0.smithi136.stderr:215286: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-15T21:04:49.155 INFO:tasks.workunit.client.0.smithi136.stderr:215286: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-15T21:04:49.155 INFO:tasks.workunit.client.0.smithi136.stderr:215286: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-15T21:04:49.155 INFO:tasks.workunit.client.0.smithi136.stderr:215286: 1 2024-09-15T21:04:49.155 INFO:tasks.workunit.client.0.smithi136.stderr:215286: 2 2024-09-15T21:04:49.156 INFO:tasks.workunit.client.0.smithi136.stderr:215286: 3 2024-09-15T21:04:49.156 INFO:tasks.workunit.client.0.smithi136.stderr:215286: 4 2024-09-15T21:04:49.156 INFO:tasks.workunit.client.0.smithi136.stderr:215286: 5 2024-09-15T21:04:49.156 INFO:tasks.workunit.client.0.smithi136.stderr:215286: 6 2024-09-15T21:04:49.156 INFO:tasks.workunit.client.0.smithi136.stderr:215286: 7 2024-09-15T21:04:49.156 INFO:tasks.workunit.client.0.smithi136.stderr:215286: 8 2024-09-15T21:04:49.156 INFO:tasks.workunit.client.0.smithi136.stderr:215286: 9' 2024-09-15T21:04:49.156 INFO:tasks.workunit.client.0.smithi136.stderr:215286: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-15T21:04:50.762 INFO:tasks.workunit.client.0.smithi136.stderr:215286: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T21:04:50.762 INFO:tasks.workunit.client.0.smithi136.stderr:215284: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-15T21:04:50.762 INFO:tasks.workunit.client.0.smithi136.stderr:215284: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836585 2024-09-15T21:04:50.762 INFO:tasks.workunit.client.0.smithi136.stderr:215284: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836585 2024-09-15T21:04:50.763 INFO:tasks.workunit.client.0.smithi136.stderr:215284: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836585' 2024-09-15T21:04:50.763 INFO:tasks.workunit.client.0.smithi136.stderr:215284: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T21:04:50.763 INFO:tasks.workunit.client.0.smithi136.stderr:215284: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-15T21:04:50.763 INFO:tasks.workunit.client.0.smithi136.stderr:215284: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=42949673054 2024-09-15T21:04:50.763 INFO:tasks.workunit.client.0.smithi136.stderr:215284: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 42949673054 2024-09-15T21:04:50.763 INFO:tasks.workunit.client.0.smithi136.stderr:215284: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836585 1-42949673054' 2024-09-15T21:04:50.763 INFO:tasks.workunit.client.0.smithi136.stderr:215284: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T21:04:50.763 INFO:tasks.workunit.client.0.smithi136.stderr:215284: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-15T21:04:50.763 INFO:tasks.workunit.client.0.smithi136.stderr:215284: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=64424509529 2024-09-15T21:04:50.763 INFO:tasks.workunit.client.0.smithi136.stderr:215284: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 64424509529 2024-09-15T21:04:50.763 INFO:tasks.workunit.client.0.smithi136.stderr:215284: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836585 1-42949673054 2-64424509529' 2024-09-15T21:04:50.763 INFO:tasks.workunit.client.0.smithi136.stderr:215284: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T21:04:50.763 INFO:tasks.workunit.client.0.smithi136.stderr:215284: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-15T21:04:50.764 INFO:tasks.workunit.client.0.smithi136.stderr:215284: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=313532612634 2024-09-15T21:04:50.764 INFO:tasks.workunit.client.0.smithi136.stderr:215284: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 313532612634 2024-09-15T21:04:50.764 INFO:tasks.workunit.client.0.smithi136.stderr:215284: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836585 1-42949673054 2-64424509529 3-313532612634' 2024-09-15T21:04:50.764 INFO:tasks.workunit.client.0.smithi136.stderr:215284: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T21:04:50.764 INFO:tasks.workunit.client.0.smithi136.stderr:215284: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-09-15T21:04:50.764 INFO:tasks.workunit.client.0.smithi136.stderr:215284: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=107374182474 2024-09-15T21:04:50.764 INFO:tasks.workunit.client.0.smithi136.stderr:215284: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 107374182474 2024-09-15T21:04:50.764 INFO:tasks.workunit.client.0.smithi136.stderr:215284: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836585 1-42949673054 2-64424509529 3-313532612634 4-107374182474' 2024-09-15T21:04:50.764 INFO:tasks.workunit.client.0.smithi136.stderr:215284: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T21:04:50.764 INFO:tasks.workunit.client.0.smithi136.stderr:215284: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-09-15T21:04:50.764 INFO:tasks.workunit.client.0.smithi136.stderr:215284: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=446676598786 2024-09-15T21:04:50.764 INFO:tasks.workunit.client.0.smithi136.stderr:215284: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 446676598786 2024-09-15T21:04:50.764 INFO:tasks.workunit.client.0.smithi136.stderr:215284: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836585 1-42949673054 2-64424509529 3-313532612634 4-107374182474 5-446676598786' 2024-09-15T21:04:50.764 INFO:tasks.workunit.client.0.smithi136.stderr:215284: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T21:04:50.765 INFO:tasks.workunit.client.0.smithi136.stderr:215284: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-09-15T21:04:50.765 INFO:tasks.workunit.client.0.smithi136.stderr:215284: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=15032385542helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T21:04:50.765 INFO:tasks.workunit.client.0.smithi136.stderr:215286: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-15T21:04:50.765 INFO:tasks.workunit.client.0.smithi136.stderr:215286: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836584 2024-09-15T21:04:50.765 INFO:tasks.workunit.client.0.smithi136.stderr:215286: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836584 2024-09-15T21:04:50.765 INFO:tasks.workunit.client.0.smithi136.stderr:215286: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836584' 2024-09-15T21:04:50.765 INFO:tasks.workunit.client.0.smithi136.stderr:215286: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T21:04:50.765 INFO:tasks.workunit.client.0.smithi136.stderr:215286: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-15T21:04:50.765 INFO:tasks.workunit.client.0.smithi136.stderr:215286: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=42949673055 2024-09-15T21:04:50.766 INFO:tasks.workunit.client.0.smithi136.stderr:215286: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 42949673055 2024-09-15T21:04:50.766 INFO:tasks.workunit.client.0.smithi136.stderr:215286: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836584 1-42949673055' 2024-09-15T21:04:50.766 INFO:tasks.workunit.client.0.smithi136.stderr:215286: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T21:04:50.766 INFO:tasks.workunit.client.0.smithi136.stderr:215286: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-15T21:04:50.766 INFO:tasks.workunit.client.0.smithi136.stderr:215286: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=64424509530 2024-09-15T21:04:50.766 INFO:tasks.workunit.client.0.smithi136.stderr:215286: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 64424509530 2024-09-15T21:04:50.766 INFO:tasks.workunit.client.0.smithi136.stderr:215286: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836584 1-42949673055 2-64424509530' 2024-09-15T21:04:50.766 INFO:tasks.workunit.client.0.smithi136.stderr:215286: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T21:04:50.766 INFO:tasks.workunit.client.0.smithi136.stderr:215286: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-15T21:04:50.766 INFO:tasks.workunit.client.0.smithi136.stderr:215286: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=313532612635 2024-09-15T21:04:50.766 INFO:tasks.workunit.client.0.smithi136.stderr:215286: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 313532612635 2024-09-15T21:04:50.766 INFO:tasks.workunit.client.0.smithi136.stderr:215286: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836584 1-42949673055 2-64424509530 3-313532612635' 2024-09-15T21:04:50.766 INFO:tasks.workunit.client.0.smithi136.stderr:215286: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T21:04:50.766 INFO:tasks.workunit.client.0.smithi136.stderr:215286: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-09-15T21:04:50.767 INFO:tasks.workunit.client.0.smithi136.stderr:215286: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=107374182475 2024-09-15T21:04:50.767 INFO:tasks.workunit.client.0.smithi136.stderr:215286: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 107374182475 2024-09-15T21:04:50.767 INFO:tasks.workunit.client.0.smithi136.stderr:215286: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836584 1-42949673055 2-64424509530 3-313532612635 4-107374182475' 2024-09-15T21:04:50.767 INFO:tasks.workunit.client.0.smithi136.stderr:215286: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T21:04:50.767 INFO:tasks.workunit.client.0.smithi136.stderr:215286: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-09-15T21:04:50.767 INFO:tasks.workunit.client.0.smithi136.stderr:215286: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=446676598787 2024-09-15T21:04:50.767 INFO:tasks.workunit.client.0.smithi136.stderr:215286: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 446676598787 2024-09-15T21:04:50.767 INFO:tasks.workunit.client.0.smithi136.stderr:215286: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836584 1-42949673055 2-64424509530 3-313532612635 4-107374182475 5-446676598787' 2024-09-15T21:04:50.767 INFO:tasks.workunit.client.0.smithi136.stderr:215286: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T21:04:50.767 INFO:tasks.workunit.client.0.smithi136.stderr:215286: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-09-15T21:04:50.767 INFO:tasks.workunit.client.0.smithi136.stderr:215286: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=150323855424 2024-09-15T21:04:51.621 INFO:tasks.workunit.client.0.smithi136.stderr:215286: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 150323855424 2024-09-15T21:04:51.621 INFO:tasks.workunit.client.0.smithi136.stderr:215286: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836584 1-42949673055 2-64424509530 3-313532612635 4-107374182475 5-446676598787 6-150323855424' 2024-09-15T21:04:51.621 INFO:tasks.workunit.client.0.smithi136.stderr:215286: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T21:04:51.621 INFO:tasks.workunit.client.0.smithi136.stderr:215286: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2024-09-15T21:04:51.622 INFO:tasks.workunit.client.0.smithi136.stderr:215286: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=171798691896 2024-09-15T21:04:51.622 INFO:tasks.workunit.client.0.smithi136.stderr:215286: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 171798691896 2024-09-15T21:04:51.622 INFO:tasks.workunit.client.0.smithi136.stderr:215286: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836584 1-42949673055 2-64424509530 3-313532612635 4-107374182475 5-446676598787 6-150323855424 7-171798691896' 2024-09-15T21:04:51.622 INFO:tasks.workunit.client.0.smithi136.stderr:215286: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T21:04:51.622 INFO:tasks.workunit.client.0.smithi136.stderr:215286: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2024-09-15T21:04:51.622 INFO:tasks.workunit.client.0.smithi136.stderr:215286: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=193273528371 2024-09-15T21:04:51.622 INFO:tasks.workunit.client.0.smithi136.stderr:215286: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 193273528371 2024-09-15T21:04:51.622 INFO:tasks.workunit.client.0.smithi136.stderr:215286: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836584 1-42949673055 2-64424509530 3-313532612635 4-107374182475 5-446676598787 6-150323855424 7-171798691896 8-193273528371' 2024-09-15T21:04:51.622 INFO:tasks.workunit.client.0.smithi136.stderr:215286: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T21:04:51.622 INFO:tasks.workunit.client.0.smithi136.stderr:215286: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2024-09-15T21:04:51.622 INFO:tasks.workunit.client.0.smithi136.stderr:215286: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=450971566083 2024-09-15T21:04:51.622 INFO:tasks.workunit.client.0.smithi136.stderr:215286: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 450971566083 2024-09-15T21:04:51.622 INFO:tasks.workunit.client.0.smithi136.stderr:215286: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836584 1-42949673055 2-64424509530 3-313532612635 4-107374182475 5-446676598787 6-150323855424 7-171798691896 8-193273528371 9-450971566083' 2024-09-15T21:04:51.622 INFO:tasks.workunit.client.0.smithi136.stderr:215286: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T21:04:51.623 INFO:tasks.workunit.client.0.smithi136.stderr:215286: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836584 2024-09-15T21:04:51.623 INFO:tasks.workunit.client.0.smithi136.stderr:215286: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T21:04:51.623 INFO:tasks.workunit.client.0.smithi136.stderr:215286: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-15T21:04:51.623 INFO:tasks.workunit.client.0.smithi136.stderr:215286: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836584 2024-09-15T21:04:51.623 INFO:tasks.workunit.client.0.smithi136.stderr:215286: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T21:04:51.623 INFO:tasks.workunit.client.0.smithi136.stderr:215286: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836584 2024-09-15T21:04:51.623 INFO:tasks.workunit.client.0.smithi136.stderr:215286: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836584' 2024-09-15T21:04:51.623 INFO:tasks.workunit.client.0.smithi136.stderr:215286: waiting osd.0 seq 21474836584 2024-09-15T21:04:51.623 INFO:tasks.workunit.client.0.smithi136.stderr:215286: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T21:04:51.623 INFO:tasks.workunit.client.0.smithi136.stderr:215286: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836585 -lt 21474836584 2024-09-15T21:04:51.623 INFO:tasks.workunit.client.0.smithi136.stderr:215286: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T21:04:51.623 INFO:tasks.workunit.client.0.smithi136.stderr:215286: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-42949673055 2024-09-15T21:04:51.623 INFO:tasks.workunit.client.0.smithi136.stderr:215286: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T21:04:51.624 INFO:tasks.workunit.client.0.smithi136.stderr:215286: /home/ubuntu/cephtest/clone.client.0/qa/standalone/3 2024-09-15T21:04:51.624 INFO:tasks.workunit.client.0.smithi136.stderr:215284: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 150323855423 2024-09-15T21:04:51.624 INFO:tasks.workunit.client.0.smithi136.stderr:215284: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836585 1-42949673054 2-64424509529 3-313532612634 4-107374182474 5-446676598786 6-150323855423' 2024-09-15T21:04:51.624 INFO:tasks.workunit.client.0.smithi136.stderr:215284: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T21:04:51.624 INFO:tasks.workunit.client.0.smithi136.stderr:215284: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2024-09-15T21:04:51.624 INFO:tasks.workunit.client.0.smithi136.stderr:215284: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=171798691895 2024-09-15T21:04:51.624 INFO:tasks.workunit.client.0.smithi136.stderr:215284: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 171798691895 2024-09-15T21:04:51.624 INFO:tasks.workunit.client.0.smithi136.stderr:215284: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836585 1-42949673054 2-64424509529 3-313532612634 4-107374182474 5-446676598786 6-150323855423 7-171798691895' 2024-09-15T21:04:51.624 INFO:tasks.workunit.client.0.smithi136.stderr:215284: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T21:04:51.624 INFO:tasks.workunit.client.0.smithi136.stderr:215284: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2024-09-15T21:04:51.625 INFO:tasks.workunit.client.0.smithi136.stderr:215284: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=193273528372 2024-09-15T21:04:51.625 INFO:tasks.workunit.client.0.smithi136.stderr:215284: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 193273528372 2024-09-15T21:04:51.625 INFO:tasks.workunit.client.0.smithi136.stderr:215284: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836585 1-42949673054 2-64424509529 3-313532612634 4-107374182474 5-446676598786 6-150323855423 7-171798691895 8-193273528372' 2024-09-15T21:04:51.625 INFO:tasks.workunit.client.0.smithi136.stderr:215284: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T21:04:51.625 INFO:tasks.workunit.client.0.smithi136.stderr:215284: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2024-09-15T21:04:51.625 INFO:tasks.workunit.client.0.smithi136.stderr:215284: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=450971566082 2024-09-15T21:04:51.625 INFO:tasks.workunit.client.0.smithi136.stderr:215284: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 450971566082 2024-09-15T21:04:51.625 INFO:tasks.workunit.client.0.smithi136.stderr:215284: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836585 1-42949673054 2-64424509529 3-313532612634 4-107374182474 5-446676598786 6-150323855423 7-171798691895 8-193273528372 9-450971566082' 2024-09-15T21:04:51.625 INFO:tasks.workunit.client.0.smithi136.stderr:215284: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T21:04:51.625 INFO:tasks.workunit.client.0.smithi136.stderr:215284: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836585 2024-09-15T21:04:51.625 INFO:tasks.workunit.client.0.smithi136.stderr:215284: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T21:04:51.625 INFO:tasks.workunit.client.0.smithi136.stderr:215284: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-15T21:04:51.625 INFO:tasks.workunit.client.0.smithi136.stderr:215284: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836585 2024-09-15T21:04:51.626 INFO:tasks.workunit.client.0.smithi136.stderr:215284: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T21:04:51.626 INFO:tasks.workunit.client.0.smithi136.stderr:215284: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836585 2024-09-15T21:04:51.626 INFO:tasks.workunit.client.0.smithi136.stderr:215284: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836585' 2024-09-15T21:04:51.626 INFO:tasks.workunit.client.0.smithi136.stderr:215284: waiting osd.0 seq 21474836585 2024-09-15T21:04:51.626 INFO:tasks.workunit.client.0.smithi136.stderr:215284: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T21:04:51.626 INFO:tasks.workunit.client.0.smithi136.stderr:215284: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836585 -lt 21474836585 2024-09-15T21:04:51.626 INFO:tasks.workunit.client.0.smithi136.stderr:215284: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T21:04:51.626 INFO:tasks.workunit.client.0.smithi136.stderr:215284: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-42949673054 2024-09-15T21:04:51.626 INFO:tasks.workunit.client.0.smithi136.stderr:215284: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T21:04:52.497 INFO:tasks.workunit.client.0.smithi136.stderr:215284: /home/ubuntu/cephtest/clone.client.0/qa/standaceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-15T21:04:52.497 INFO:tasks.workunit.client.0.smithi136.stderr:215286: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-42949673055 2024-09-15T21:04:52.497 INFO:tasks.workunit.client.0.smithi136.stderr:215286: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T21:04:52.497 INFO:tasks.workunit.client.0.smithi136.stderr:215286: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=42949673055 2024-09-15T21:04:52.497 INFO:tasks.workunit.client.0.smithi136.stderr:215286: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 42949673055' 2024-09-15T21:04:52.497 INFO:tasks.workunit.client.0.smithi136.stderr:215286: waiting osd.1 seq 42949673055 2024-09-15T21:04:52.497 INFO:tasks.workunit.client.0.smithi136.stderr:215286: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-15T21:04:52.497 INFO:tasks.workunit.client.0.smithi136.stderr:215286: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 42949673055 -lt 42949673055 2024-09-15T21:04:52.497 INFO:tasks.workunit.client.0.smithi136.stderr:215286: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T21:04:52.497 INFO:tasks.workunit.client.0.smithi136.stderr:215286: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-64424509530 2024-09-15T21:04:52.498 INFO:tasks.workunit.client.0.smithi136.stderr:215286: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T21:04:52.498 INFO:tasks.workunit.client.0.smithi136.stderr:215286: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-15T21:04:52.498 INFO:tasks.workunit.client.0.smithi136.stderr:215286: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-64424509530 2024-09-15T21:04:52.498 INFO:tasks.workunit.client.0.smithi136.stderr:215286: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T21:04:52.498 INFO:tasks.workunit.client.0.smithi136.stderr:215286: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=64424509530 2024-09-15T21:04:52.498 INFO:tasks.workunit.client.0.smithi136.stderr:215286: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 64424509530' 2024-09-15T21:04:52.498 INFO:tasks.workunit.client.0.smithi136.stderr:215286: waiting osd.2 seq 64424509530 2024-09-15T21:04:52.498 INFO:tasks.workunit.client.0.smithi136.stderr:215286: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-15T21:04:52.498 INFO:tasks.workunit.client.0.smithi136.stderr:215286: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 64424509530 -lt 64424509530 2024-09-15T21:04:52.498 INFO:tasks.workunit.client.0.smithi136.stderr:215286: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T21:04:52.498 INFO:tasks.workunit.client.0.smithi136.stderr:215286: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 3-313532612635 2024-09-15T21:04:52.498 INFO:tasks.workunit.client.0.smithi136.stderr:215286: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T21:04:52.499 INFO:tasks.workunit.client.0.smithi136.stderr:215286: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=3 2024-09-15T21:04:52.499 INFO:tasks.workunit.client.0.smithi136.stderr:215286: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 3-313532612635 2024-09-15T21:04:52.499 INFO:tasks.workunit.client.0.smithi136.stderr:215286: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T21:04:52.499 INFO:tasks.workunit.client.0.smithi136.stderr:215286: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=313532612635 2024-09-15T21:04:52.499 INFO:tasks.workunit.client.0.smithi136.stderr:215286: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.3 seq 313532612635' 2024-09-15T21:04:52.499 INFO:tasks.workunit.client.0.smithi136.stderr:215286: waiting osd.3 seq 313532612635 2024-09-15T21:04:52.499 INFO:tasks.workunit.client.0.smithi136.stderr:215286: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-15T21:04:52.499 INFO:tasks.workunit.client.0.smithi136.stderr:215286: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 313532612635 -lt 313532612635 2024-09-15T21:04:52.499 INFO:tasks.workunit.client.0.smithi136.stderr:215286: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T21:04:52.499 INFO:tasks.workunit.client.0.smithi136.stderr:215286: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 4-107374182475 2024-09-15T21:04:52.499 INFO:tasks.workunit.client.0.smithi136.stderr:215286: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T21:04:52.499 INFO:tasks.workunit.client.0.smithi136.stderr:215286: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=4 2024-09-15T21:04:52.499 INFO:tasks.workunit.client.0.smithi136.stderr:215286: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 4-107374182475 2024-09-15T21:04:52.499 INFO:tasks.workunit.client.0.smithi136.stderr:215286: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T21:04:52.500 INFO:tasks.workunit.client.0.smithi136.stderr:215286: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=107374182475 2024-09-15T21:04:52.574 INFO:tasks.workunit.client.0.smithi136.stderr:215286: /home/ubuntu/cephtest/clolone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-15T21:04:52.574 INFO:tasks.workunit.client.0.smithi136.stderr:215284: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-42949673054 2024-09-15T21:04:52.574 INFO:tasks.workunit.client.0.smithi136.stderr:215284: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T21:04:52.574 INFO:tasks.workunit.client.0.smithi136.stderr:215284: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=42949673054 2024-09-15T21:04:52.574 INFO:tasks.workunit.client.0.smithi136.stderr:215284: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 42949673054' 2024-09-15T21:04:52.574 INFO:tasks.workunit.client.0.smithi136.stderr:215284: waiting osd.1 seq 42949673054 2024-09-15T21:04:52.574 INFO:tasks.workunit.client.0.smithi136.stderr:215284: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-15T21:04:52.575 INFO:tasks.workunit.client.0.smithi136.stderr:215284: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 42949673055 -lt 42949673054 2024-09-15T21:04:52.575 INFO:tasks.workunit.client.0.smithi136.stderr:215284: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T21:04:52.575 INFO:tasks.workunit.client.0.smithi136.stderr:215284: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-64424509529 2024-09-15T21:04:52.575 INFO:tasks.workunit.client.0.smithi136.stderr:215284: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T21:04:52.575 INFO:tasks.workunit.client.0.smithi136.stderr:215284: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-15T21:04:52.575 INFO:tasks.workunit.client.0.smithi136.stderr:215284: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-64424509529 2024-09-15T21:04:52.575 INFO:tasks.workunit.client.0.smithi136.stderr:215284: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T21:04:52.575 INFO:tasks.workunit.client.0.smithi136.stderr:215284: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=64424509529 2024-09-15T21:04:52.575 INFO:tasks.workunit.client.0.smithi136.stderr:215284: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 64424509529' 2024-09-15T21:04:52.575 INFO:tasks.workunit.client.0.smithi136.stderr:215284: waiting osd.2 seq 64424509529 2024-09-15T21:04:52.575 INFO:tasks.workunit.client.0.smithi136.stderr:215284: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-15T21:04:52.575 INFO:tasks.workunit.client.0.smithi136.stderr:215284: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 64424509530 -lt 64424509529 2024-09-15T21:04:52.575 INFO:tasks.workunit.client.0.smithi136.stderr:215284: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T21:04:52.575 INFO:tasks.workunit.client.0.smithi136.stderr:215284: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 3-313532612634 2024-09-15T21:04:52.576 INFO:tasks.workunit.client.0.smithi136.stderr:215284: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T21:04:52.576 INFO:tasks.workunit.client.0.smithi136.stderr:215284: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=3 2024-09-15T21:04:52.576 INFO:tasks.workunit.client.0.smithi136.stderr:215284: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 3-313532612634 2024-09-15T21:04:52.576 INFO:tasks.workunit.client.0.smithi136.stderr:215284: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T21:04:52.576 INFO:tasks.workunit.client.0.smithi136.stderr:215284: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=313532612634 2024-09-15T21:04:52.576 INFO:tasks.workunit.client.0.smithi136.stderr:215284: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.3 seq 313532612634' 2024-09-15T21:04:52.576 INFO:tasks.workunit.client.0.smithi136.stderr:215284: waiting osd.3 seq 313532612634 2024-09-15T21:04:52.576 INFO:tasks.workunit.client.0.smithi136.stderr:215284: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-15T21:04:52.576 INFO:tasks.workunit.client.0.smithi136.stderr:215284: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 313532612635 -lt 313532612634 2024-09-15T21:04:52.576 INFO:tasks.workunit.client.0.smithi136.stderr:215284: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T21:04:52.576 INFO:tasks.workunit.client.0.smithi136.stderr:215284: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 4-107374182474 2024-09-15T21:04:52.576 INFO:tasks.workunit.client.0.smithi136.stderr:215284: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T21:04:52.576 INFO:tasks.workunit.client.0.smithi136.stderr:215284: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=4 2024-09-15T21:04:52.577 INFO:tasks.workunit.client.0.smithi136.stderr:215284: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 4-107374182474 2024-09-15T21:04:52.577 INFO:tasks.workunit.client.0.smithi136.stderr:215284: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T21:04:52.577 INFO:tasks.workunit.client.0.smithi136.stderr:215284: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=107374182474 2024-09-15T21:04:53.737 INFO:tasks.workunit.client.0.smithi136.stderr:215284: /home/ubuntu/cephtesne.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.4 seq 107374182475' 2024-09-15T21:04:53.737 INFO:tasks.workunit.client.0.smithi136.stderr:215286: waiting osd.4 seq 107374182475 2024-09-15T21:04:53.737 INFO:tasks.workunit.client.0.smithi136.stderr:215286: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 4 2024-09-15T21:04:53.737 INFO:tasks.workunit.client.0.smithi136.stderr:215286: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 107374182475 -lt 107374182475 2024-09-15T21:04:53.737 INFO:tasks.workunit.client.0.smithi136.stderr:215286: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T21:04:53.737 INFO:tasks.workunit.client.0.smithi136.stderr:215286: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 5-446676598787 2024-09-15T21:04:53.737 INFO:tasks.workunit.client.0.smithi136.stderr:215286: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T21:04:53.737 INFO:tasks.workunit.client.0.smithi136.stderr:215286: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=5 2024-09-15T21:04:53.737 INFO:tasks.workunit.client.0.smithi136.stderr:215286: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 5-446676598787 2024-09-15T21:04:53.737 INFO:tasks.workunit.client.0.smithi136.stderr:215286: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T21:04:53.738 INFO:tasks.workunit.client.0.smithi136.stderr:215286: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=446676598787 2024-09-15T21:04:53.738 INFO:tasks.workunit.client.0.smithi136.stderr:215286: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.5 seq 446676598787' 2024-09-15T21:04:53.738 INFO:tasks.workunit.client.0.smithi136.stderr:215286: waiting osd.5 seq 446676598787 2024-09-15T21:04:53.738 INFO:tasks.workunit.client.0.smithi136.stderr:215286: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 5 2024-09-15T21:04:53.738 INFO:tasks.workunit.client.0.smithi136.stderr:215286: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 446676598788 -lt 446676598787 2024-09-15T21:04:53.738 INFO:tasks.workunit.client.0.smithi136.stderr:215286: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T21:04:53.738 INFO:tasks.workunit.client.0.smithi136.stderr:215286: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 6-150323855424 2024-09-15T21:04:53.738 INFO:tasks.workunit.client.0.smithi136.stderr:215286: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T21:04:53.738 INFO:tasks.workunit.client.0.smithi136.stderr:215286: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=6 2024-09-15T21:04:53.738 INFO:tasks.workunit.client.0.smithi136.stderr:215286: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 6-150323855424 2024-09-15T21:04:53.738 INFO:tasks.workunit.client.0.smithi136.stderr:215286: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T21:04:53.738 INFO:tasks.workunit.client.0.smithi136.stderr:215286: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=150323855424 2024-09-15T21:04:53.738 INFO:tasks.workunit.client.0.smithi136.stderr:215286: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.6 seq 150323855424' 2024-09-15T21:04:53.738 INFO:tasks.workunit.client.0.smithi136.stderr:215286: waiting osd.6 seq 150323855424 2024-09-15T21:04:53.739 INFO:tasks.workunit.client.0.smithi136.stderr:215286: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 6 2024-09-15T21:04:53.739 INFO:tasks.workunit.client.0.smithi136.stderr:215286: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 150323855424 -lt 150323855424 2024-09-15T21:04:53.739 INFO:tasks.workunit.client.0.smithi136.stderr:215286: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T21:04:53.739 INFO:tasks.workunit.client.0.smithi136.stderr:215286: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 7-171798691896 2024-09-15T21:04:53.739 INFO:tasks.workunit.client.0.smithi136.stderr:215286: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T21:04:53.739 INFO:tasks.workunit.client.0.smithi136.stderr:215286: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=7 2024-09-15T21:04:53.739 INFO:tasks.workunit.client.0.smithi136.stderr:215286: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 7-171798691896 2024-09-15T21:04:53.739 INFO:tasks.workunit.client.0.smithi136.stderr:215286: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T21:04:53.739 INFO:tasks.workunit.client.0.smithi136.stderr:215286: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=171798691896 2024-09-15T21:04:53.739 INFO:tasks.workunit.client.0.smithi136.stderr:215286: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.7 seq 171798691896' 2024-09-15T21:04:53.739 INFO:tasks.workunit.client.0.smithi136.stderr:215286: waiting osd.7 seq 171798691896 2024-09-15T21:04:53.739 INFO:tasks.workunit.client.0.smithi136.stderr:215286: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 7 2024-09-15T21:04:53.739 INFO:tasks.workunit.client.0.smithi136.stderr:215286: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 171798691896 -lt 171798691896 2024-09-15T21:04:53.880 INFO:tasks.workunit.client.0.smithi136.stderr:215286: /homet/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.4 seq 107374182474' 2024-09-15T21:04:53.880 INFO:tasks.workunit.client.0.smithi136.stderr:215284: waiting osd.4 seq 107374182474 2024-09-15T21:04:53.880 INFO:tasks.workunit.client.0.smithi136.stderr:215284: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 4 2024-09-15T21:04:53.880 INFO:tasks.workunit.client.0.smithi136.stderr:215284: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 107374182475 -lt 107374182474 2024-09-15T21:04:53.880 INFO:tasks.workunit.client.0.smithi136.stderr:215284: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T21:04:53.880 INFO:tasks.workunit.client.0.smithi136.stderr:215284: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 5-446676598786 2024-09-15T21:04:53.881 INFO:tasks.workunit.client.0.smithi136.stderr:215284: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T21:04:53.881 INFO:tasks.workunit.client.0.smithi136.stderr:215284: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=5 2024-09-15T21:04:53.881 INFO:tasks.workunit.client.0.smithi136.stderr:215284: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 5-446676598786 2024-09-15T21:04:53.881 INFO:tasks.workunit.client.0.smithi136.stderr:215284: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T21:04:53.881 INFO:tasks.workunit.client.0.smithi136.stderr:215284: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=446676598786 2024-09-15T21:04:53.881 INFO:tasks.workunit.client.0.smithi136.stderr:215284: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.5 seq 446676598786' 2024-09-15T21:04:53.881 INFO:tasks.workunit.client.0.smithi136.stderr:215284: waiting osd.5 seq 446676598786 2024-09-15T21:04:53.881 INFO:tasks.workunit.client.0.smithi136.stderr:215284: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 5 2024-09-15T21:04:53.881 INFO:tasks.workunit.client.0.smithi136.stderr:215284: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 446676598788 -lt 446676598786 2024-09-15T21:04:53.881 INFO:tasks.workunit.client.0.smithi136.stderr:215284: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T21:04:53.881 INFO:tasks.workunit.client.0.smithi136.stderr:215284: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 6-150323855423 2024-09-15T21:04:53.881 INFO:tasks.workunit.client.0.smithi136.stderr:215284: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T21:04:53.881 INFO:tasks.workunit.client.0.smithi136.stderr:215284: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=6 2024-09-15T21:04:53.881 INFO:tasks.workunit.client.0.smithi136.stderr:215284: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 6-150323855423 2024-09-15T21:04:53.882 INFO:tasks.workunit.client.0.smithi136.stderr:215284: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T21:04:53.882 INFO:tasks.workunit.client.0.smithi136.stderr:215284: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=150323855423 2024-09-15T21:04:53.882 INFO:tasks.workunit.client.0.smithi136.stderr:215284: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.6 seq 150323855423' 2024-09-15T21:04:53.882 INFO:tasks.workunit.client.0.smithi136.stderr:215284: waiting osd.6 seq 150323855423 2024-09-15T21:04:53.882 INFO:tasks.workunit.client.0.smithi136.stderr:215284: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 6 2024-09-15T21:04:53.882 INFO:tasks.workunit.client.0.smithi136.stderr:215284: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 150323855424 -lt 150323855423 2024-09-15T21:04:53.882 INFO:tasks.workunit.client.0.smithi136.stderr:215284: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T21:04:53.882 INFO:tasks.workunit.client.0.smithi136.stderr:215284: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 7-171798691895 2024-09-15T21:04:53.882 INFO:tasks.workunit.client.0.smithi136.stderr:215284: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T21:04:53.882 INFO:tasks.workunit.client.0.smithi136.stderr:215284: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=7 2024-09-15T21:04:53.882 INFO:tasks.workunit.client.0.smithi136.stderr:215284: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 7-171798691895 2024-09-15T21:04:53.882 INFO:tasks.workunit.client.0.smithi136.stderr:215284: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T21:04:53.882 INFO:tasks.workunit.client.0.smithi136.stderr:215284: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=171798691895 2024-09-15T21:04:53.883 INFO:tasks.workunit.client.0.smithi136.stderr:215284: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.7 seq 171798691895' 2024-09-15T21:04:53.883 INFO:tasks.workunit.client.0.smithi136.stderr:215284: waiting osd.7 seq 171798691895 2024-09-15T21:04:53.883 INFO:tasks.workunit.client.0.smithi136.stderr:215284: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 7 2024-09-15T21:04:53.883 INFO:tasks.workunit.client.0.smithi136.stderr:215284: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 171798691896 -lt 171798691895 2024-09-15T21:04:55.000 INFO:tasks.workunit.client.0.smithi136.stderr:215284: /ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T21:04:55.000 INFO:tasks.workunit.client.0.smithi136.stderr:215286: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 8-193273528371 2024-09-15T21:04:55.000 INFO:tasks.workunit.client.0.smithi136.stderr:215286: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T21:04:55.000 INFO:tasks.workunit.client.0.smithi136.stderr:215286: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=8 2024-09-15T21:04:55.000 INFO:tasks.workunit.client.0.smithi136.stderr:215286: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 8-193273528371 2024-09-15T21:04:55.000 INFO:tasks.workunit.client.0.smithi136.stderr:215286: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T21:04:55.000 INFO:tasks.workunit.client.0.smithi136.stderr:215286: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=193273528371 2024-09-15T21:04:55.000 INFO:tasks.workunit.client.0.smithi136.stderr:215286: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.8 seq 193273528371' 2024-09-15T21:04:55.000 INFO:tasks.workunit.client.0.smithi136.stderr:215286: waiting osd.8 seq 193273528371 2024-09-15T21:04:55.001 INFO:tasks.workunit.client.0.smithi136.stderr:215286: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 8 2024-09-15T21:04:55.001 INFO:tasks.workunit.client.0.smithi136.stderr:215286: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 193273528372 -lt 193273528371 2024-09-15T21:04:55.001 INFO:tasks.workunit.client.0.smithi136.stderr:215286: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T21:04:55.001 INFO:tasks.workunit.client.0.smithi136.stderr:215286: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 9-450971566083 2024-09-15T21:04:55.001 INFO:tasks.workunit.client.0.smithi136.stderr:215286: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T21:04:55.001 INFO:tasks.workunit.client.0.smithi136.stderr:215286: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=9 2024-09-15T21:04:55.001 INFO:tasks.workunit.client.0.smithi136.stderr:215286: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 9-450971566083 2024-09-15T21:04:55.001 INFO:tasks.workunit.client.0.smithi136.stderr:215286: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T21:04:55.001 INFO:tasks.workunit.client.0.smithi136.stderr:215286: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=450971566083 2024-09-15T21:04:55.001 INFO:tasks.workunit.client.0.smithi136.stderr:215286: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.9 seq 450971566083' 2024-09-15T21:04:55.001 INFO:tasks.workunit.client.0.smithi136.stderr:215286: waiting osd.9 seq 450971566083 2024-09-15T21:04:55.001 INFO:tasks.workunit.client.0.smithi136.stderr:215286: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 9 2024-09-15T21:04:55.001 INFO:tasks.workunit.client.0.smithi136.stderr:215286: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 450971566083 -lt 450971566083 2024-09-15T21:04:55.002 INFO:tasks.workunit.client.0.smithi136.stderr:215286: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-15T21:04:55.002 INFO:tasks.workunit.client.0.smithi136.stderr:215286: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T21:04:55.002 INFO:tasks.workunit.client.0.smithi136.stderr:215286: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T21:04:55.002 INFO:tasks.workunit.client.0.smithi136.stderr:215286: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 5 == 0 2024-09-15T21:04:55.002 INFO:tasks.workunit.client.0.smithi136.stderr:215286: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-15T21:04:55.002 INFO:tasks.workunit.client.0.smithi136.stderr:215286: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-15T21:04:55.002 INFO:tasks.workunit.client.0.smithi136.stderr:215286: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-15T21:04:55.002 INFO:tasks.workunit.client.0.smithi136.stderr:215286: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T21:04:55.002 INFO:tasks.workunit.client.0.smithi136.stderr:215286: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T21:04:55.002 INFO:tasks.workunit.client.0.smithi136.stderr:215286: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T21:04:55.002 INFO:tasks.workunit.client.0.smithi136.stderr:215286: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T21:04:55.002 INFO:tasks.workunit.client.0.smithi136.stderr:215286: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=5 2024-09-15T21:04:55.137 INFO:tasks.workunit.client.0.smithi136.stderr:215286: //home/ubuntu/cephtest/clone.client.0/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T21:04:55.137 INFO:tasks.workunit.client.0.smithi136.stderr:215284: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 8-193273528372 2024-09-15T21:04:55.137 INFO:tasks.workunit.client.0.smithi136.stderr:215284: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T21:04:55.138 INFO:tasks.workunit.client.0.smithi136.stderr:215284: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=8 2024-09-15T21:04:55.139 INFO:tasks.workunit.client.0.smithi136.stderr:215284: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 8-193273528372 2024-09-15T21:04:55.139 INFO:tasks.workunit.client.0.smithi136.stderr:215284: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T21:04:55.139 INFO:tasks.workunit.client.0.smithi136.stderr:215284: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=193273528372 2024-09-15T21:04:55.140 INFO:tasks.workunit.client.0.smithi136.stderr:215284: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.8 seq 193273528372' 2024-09-15T21:04:55.140 INFO:tasks.workunit.client.0.smithi136.stderr:215284: waiting osd.8 seq 193273528372 2024-09-15T21:04:55.140 INFO:tasks.workunit.client.0.smithi136.stderr:215284: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 8 2024-09-15T21:04:55.140 INFO:tasks.workunit.client.0.smithi136.stderr:215284: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 193273528372 -lt 193273528372 2024-09-15T21:04:55.140 INFO:tasks.workunit.client.0.smithi136.stderr:215284: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T21:04:55.140 INFO:tasks.workunit.client.0.smithi136.stderr:215284: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 9-450971566082 2024-09-15T21:04:55.140 INFO:tasks.workunit.client.0.smithi136.stderr:215284: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T21:04:55.140 INFO:tasks.workunit.client.0.smithi136.stderr:215284: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=9 2024-09-15T21:04:55.140 INFO:tasks.workunit.client.0.smithi136.stderr:215284: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 9-450971566082 2024-09-15T21:04:55.141 INFO:tasks.workunit.client.0.smithi136.stderr:215284: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T21:04:55.141 INFO:tasks.workunit.client.0.smithi136.stderr:215284: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=450971566082 2024-09-15T21:04:55.141 INFO:tasks.workunit.client.0.smithi136.stderr:215284: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.9 seq 450971566082' 2024-09-15T21:04:55.141 INFO:tasks.workunit.client.0.smithi136.stderr:215284: waiting osd.9 seq 450971566082 2024-09-15T21:04:55.141 INFO:tasks.workunit.client.0.smithi136.stderr:215284: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 9 2024-09-15T21:04:55.141 INFO:tasks.workunit.client.0.smithi136.stderr:215284: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 450971566083 -lt 450971566082 2024-09-15T21:04:55.141 INFO:tasks.workunit.client.0.smithi136.stderr:215284: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-15T21:04:55.141 INFO:tasks.workunit.client.0.smithi136.stderr:215284: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T21:04:55.142 INFO:tasks.workunit.client.0.smithi136.stderr:215284: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T21:04:55.142 INFO:tasks.workunit.client.0.smithi136.stderr:215284: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 5 == 0 2024-09-15T21:04:55.142 INFO:tasks.workunit.client.0.smithi136.stderr:215284: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-15T21:04:55.142 INFO:tasks.workunit.client.0.smithi136.stderr:215284: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-15T21:04:55.142 INFO:tasks.workunit.client.0.smithi136.stderr:215284: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-15T21:04:55.142 INFO:tasks.workunit.client.0.smithi136.stderr:215284: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T21:04:55.142 INFO:tasks.workunit.client.0.smithi136.stderr:215284: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T21:04:55.143 INFO:tasks.workunit.client.0.smithi136.stderr:215284: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T21:04:55.143 INFO:tasks.workunit.client.0.smithi136.stderr:215284: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T21:04:55.143 INFO:tasks.workunit.client.0.smithi136.stderr:215284: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=5 2024-09-15T21:04:55.360 INFO:tasks.workunit.client.0.smithi136.stderr:215284: //home/ubuntu/cephtest/clone.cli/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-15T21:04:55.360 INFO:tasks.workunit.client.0.smithi136.stderr:215286: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T21:04:55.360 INFO:tasks.workunit.client.0.smithi136.stderr:215286: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T21:04:55.360 INFO:tasks.workunit.client.0.smithi136.stderr:215286: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 = 5 2024-09-15T21:04:55.360 INFO:tasks.workunit.client.0.smithi136.stderr:215286: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-15T21:04:55.360 INFO:tasks.workunit.client.0.smithi136.stderr:215286: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-15T21:04:55.360 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: run_in_background: return 0 2024-09-15T21:04:55.498 INFO:tasks.workunit.client.0.smithi136.stderr:ent.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-15T21:04:55.498 INFO:tasks.workunit.client.0.smithi136.stderr:215284: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T21:04:55.498 INFO:tasks.workunit.client.0.smithi136.stderr:215284: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T21:04:55.498 INFO:tasks.workunit.client.0.smithi136.stderr:215284: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 = 5 2024-09-15T21:04:55.498 INFO:tasks.workunit.client.0.smithi136.stderr:215284: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-15T21:04:55.498 INFO:tasks.workunit.client.0.smithi136.stderr:215284: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-15T21:04:55.498 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: run_in_background: return 0 2024-09-15T21:04:55.498 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: wait_background: for pid in $pids 2024-09-15T21:04:55.498 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2175: wait_background: wait 215283 2024-09-15T21:04:55.499 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: wait_background: eval 'pids='\'''\''' 2024-09-15T21:04:55.499 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: wait_background: pids= 2024-09-15T21:04:55.499 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: wait_background: return 0 2024-09-15T21:04:55.499 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:212: corrupt_and_repair_two: return_code=0 2024-09-15T21:04:55.499 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:213: corrupt_and_repair_two: '[' 0 -ne 0 ']' 2024-09-15T21:04:55.499 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:215: corrupt_and_repair_two: rados --pool ecpool get SOMETHING td/osd-scrub-repair/COPY 2024-09-15T21:04:55.528 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:216: corrupt_and_repair_two: diff td/osd-scrub-repair/ORIGINAL td/osd-scrub-repair/COPY 2024-09-15T21:04:55.531 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:263: corrupt_and_repair_erasure_coded: corrupt_and_repair_two td/osd-scrub-repair ecpool 3 5 2024-09-15T21:04:55.531 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:185: corrupt_and_repair_two: local dir=td/osd-scrub-repair 2024-09-15T21:04:55.531 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:186: corrupt_and_repair_two: local poolname=ecpool 2024-09-15T21:04:55.531 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:187: corrupt_and_repair_two: local first=3 2024-09-15T21:04:55.531 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:188: corrupt_and_repair_two: local second=5 2024-09-15T21:04:55.531 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:193: corrupt_and_repair_two: pids= 2024-09-15T21:04:55.531 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:194: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 3 SOMETHING remove 2024-09-15T21:04:55.531 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2135: run_in_background: local pid_variable=pids 2024-09-15T21:04:55.531 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2136: run_in_background: shift 2024-09-15T21:04:55.531 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2140: run_in_background: eval 'pids+=" 218264"' 2024-09-15T21:04:55.532 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2140: run_in_background: pids+=' 218264' 2024-09-15T21:04:55.532 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:195: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 5 SOMETHING remove 2024-09-15T21:04:55.532 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2135: run_in_background: local pid_variable=pids 2024-09-15T21:04:55.532 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2136: run_in_background: shift 2024-09-15T21:04:55.532 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2140: run_in_background: eval 'pids+=" 218266"' 2024-09-15T21:04:55.532 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2140: run_in_background: pids+=' 218266' 2024-09-15T21:04:55.532 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: run_in_background: objectstore_tool td/osd-scrub-repair 3 SOMETHING remove 2024-09-15T21:04:55.532 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:196: corrupt_and_repair_two: wait_background pids 2024-09-15T21:04:55.532 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: wait_background: pids=' 218264 218266' 2024-09-15T21:04:55.532 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: wait_background: return_code=0 2024-09-15T21:04:55.532 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: wait_background: for pid in $pids 2024-09-15T21:04:55.532 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2175: wait_background: wait 218264 2024-09-15T21:04:55.533 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: run_in_background: sed 's/^/218267: /' 2024-09-15T21:04:55.533 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: run_in_background: objectstore_tool td/osd-scrub-repair 5 SOMETHING remove 2024-09-15T21:04:55.533 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: run_in_background: sed 's/^/218269: /' 2024-09-15T21:04:58.986 INFO:tasks.workunit.client.0.smithi136.stderr:218267: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2024-09-15T21:04:58.986 INFO:tasks.workunit.client.0.smithi136.stderr:218267: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2024-09-15T21:04:58.986 INFO:tasks.workunit.client.0.smithi136.stderr:218267: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=3 2024-09-15T21:04:58.986 INFO:tasks.workunit.client.0.smithi136.stderr:218267: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2024-09-15T21:04:58.986 INFO:tasks.workunit.client.0.smithi136.stderr:218267: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 3 SOMETHING remove 2024-09-15T21:04:58.987 INFO:tasks.workunit.client.0.smithi136.stderr:218267: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-09-15T21:04:58.987 INFO:tasks.workunit.client.0.smithi136.stderr:218267: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2024-09-15T21:04:58.987 INFO:tasks.workunit.client.0.smithi136.stderr:218267: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=3 2024-09-15T21:04:58.987 INFO:tasks.workunit.client.0.smithi136.stderr:218267: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2024-09-15T21:04:58.987 INFO:tasks.workunit.client.0.smithi136.stderr:218267: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.3 2024-09-15T21:04:58.987 INFO:tasks.workunit.client.0.smithi136.stderr:218267: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-15T21:04:58.987 INFO:tasks.workunit.client.0.smithi136.stderr:218267: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-15T21:04:58.987 INFO:tasks.workunit.client.0.smithi136.stderr:218267: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-15T21:04:58.987 INFO:tasks.workunit.client.0.smithi136.stderr:218267: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-15T21:04:58.987 INFO:tasks.workunit.client.0.smithi136.stderr:218267: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-15T21:04:58.987 INFO:tasks.workunit.client.0.smithi136.stderr:218267: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-15T21:04:58.987 INFO:tasks.workunit.client.0.smithi136.stderr:218267: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 3 SOMETHING remove 2024-09-15T21:04:58.987 INFO:tasks.workunit.client.0.smithi136.stderr:218267: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-15T21:04:58.987 INFO:tasks.workunit.client.0.smithi136.stderr:218267: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-15T21:04:58.988 INFO:tasks.workunit.client.0.smithi136.stderr:218267: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=3 2024-09-15T21:04:58.988 INFO:tasks.workunit.client.0.smithi136.stderr:218267: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-15T21:04:58.988 INFO:tasks.workunit.client.0.smithi136.stderr:218267: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/3 2024-09-15T21:04:58.988 INFO:tasks.workunit.client.0.smithi136.stderr:218267: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/3 SOMETHING remove 2024-09-15T21:04:58.988 INFO:tasks.workunit.client.0.smithi136.stderr:218267: remove 0#2:eb822e21:::SOMETHING:head# 2024-09-15T21:04:58.988 INFO:tasks.workunit.client.0.smithi136.stderr:218267: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 3 2024-09-15T21:04:58.988 INFO:tasks.workunit.client.0.smithi136.stderr:218267: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2024-09-15T21:04:58.988 INFO:tasks.workunit.client.0.smithi136.stderr:218267: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-15T21:04:58.988 INFO:tasks.workunit.client.0.smithi136.stderr:218267: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=3 2024-09-15T21:04:58.988 INFO:tasks.workunit.client.0.smithi136.stderr:218267: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-15T21:04:58.988 INFO:tasks.workunit.client.0.smithi136.stderr:218267: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/3 2024-09-15T21:04:58.988 INFO:tasks.workunit.client.0.smithi136.stderr:218267: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=ce4892c7-6c6d-4b4f-8e6f-07a3e11133e5 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-15T21:04:58.988 INFO:tasks.workunit.client.0.smithi136.stderr:218267: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T21:04:58.989 INFO:tasks.workunit.client.0.smithi136.stderr:218267: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T21:04:58.990 INFO:tasks.workunit.client.0.smithi136.stderr:218267: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T21:04:58.990 INFO:tasks.workunit.client.0.smithi136.stderr:218267: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/3' 2024-09-15T21:04:58.990 INFO:tasks.workunit.client.0.smithi136.stderr:218267: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/3/journal' 2024-09-15T21:04:58.990 INFO:tasks.workunit.client.0.smithi136.stderr:218267: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-15T21:04:58.990 INFO:tasks.workunit.client.0.smithi136.stderr:218267: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-15T21:04:58.990 INFO:tasks.workunit.client.0.smithi136.stderr:218267: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-15T21:04:58.990 INFO:tasks.workunit.client.0.smithi136.stderr:218267: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-15T21:04:58.990 INFO:tasks.workunit.client.0.smithi136.stderr:218267: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-15T21:04:58.990 INFO:tasks.workunit.client.0.smithi136.stderr:218267: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-15T21:04:58.990 INFO:tasks.workunit.client.0.smithi136.stderr:218267: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-15T21:04:58.990 INFO:tasks.workunit.client.0.smithi136.stderr:218267: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T21:04:58.990 INFO:tasks.workunit.client.0.smithi136.stderr:218267: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52602 2024-09-15T21:04:58.990 INFO:tasks.workunit.client.0.smithi136.stderr:218267: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.52602/$cluster-$name.asok' 2024-09-15T21:04:58.990 INFO:tasks.workunit.client.0.smithi136.stderr:218267: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.52602/$cluster-$name.asok' 2024-09-15T21:04:58.991 INFO:tasks.workunit.client.0.smithi136.stderr:218267: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-15T21:04:58.991 INFO:tasks.workunit.client.0.smithi136.stderr:218267: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-15T21:04:58.991 INFO:tasks.workunit.client.0.smithi136.stderr:218267: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-15T21:04:58.991 INFO:tasks.workunit.client.0.smithi136.stderr:218267: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T21:04:58.991 INFO:tasks.workunit.client.0.smithi136.stderr:218267: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T21:04:58.991 INFO:tasks.workunit.client.0.smithi136.stderr:218267: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T21:04:58.991 INFO:tasks.workunit.client.0.smithi136.stderr:218267: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T21:04:58.991 INFO:tasks.workunit.client.0.smithi136.stderr:218267: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-15T21:04:58.991 INFO:tasks.workunit.client.0.smithi136.stderr:218267: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-15T21:04:58.991 INFO:tasks.workunit.client.0.smithi136.stderr:218267: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/3 2024-09-15T21:04:58.991 INFO:tasks.workunit.client.0.smithi136.stderr:218267: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.3 2024-09-15T21:04:58.991 INFO:tasks.workunit.client.0.smithi136.stderr:218267: start osd.3 2024-09-15T21:04:58.991 INFO:tasks.workunit.client.0.smithi136.stderr:218267: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 3 --fsid=ce4892c7-6c6d-4b4f-8e6f-07a3e11133e5 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/3 --osd-journal=td/osd-scrub-repair/3/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.52602/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T21:04:59.092 INFO:tasks.workunit.client.0.smithi136.stderr:2218269: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2024-09-15T21:04:59.093 INFO:tasks.workunit.client.0.smithi136.stderr:218269: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2024-09-15T21:04:59.093 INFO:tasks.workunit.client.0.smithi136.stderr:218269: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=5 2024-09-15T21:04:59.093 INFO:tasks.workunit.client.0.smithi136.stderr:218269: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2024-09-15T21:04:59.093 INFO:tasks.workunit.client.0.smithi136.stderr:218269: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 5 SOMETHING remove 2024-09-15T21:04:59.093 INFO:tasks.workunit.client.0.smithi136.stderr:218269: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-09-15T21:04:59.093 INFO:tasks.workunit.client.0.smithi136.stderr:218269: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2024-09-15T21:04:59.093 INFO:tasks.workunit.client.0.smithi136.stderr:218269: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=5 2024-09-15T21:04:59.093 INFO:tasks.workunit.client.0.smithi136.stderr:218269: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2024-09-15T21:04:59.093 INFO:tasks.workunit.client.0.smithi136.stderr:218269: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.5 2024-09-15T21:04:59.093 INFO:tasks.workunit.client.0.smithi136.stderr:218269: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-15T21:04:59.093 INFO:tasks.workunit.client.0.smithi136.stderr:218269: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-15T21:04:59.093 INFO:tasks.workunit.client.0.smithi136.stderr:218269: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-15T21:04:59.093 INFO:tasks.workunit.client.0.smithi136.stderr:218269: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-15T21:04:59.094 INFO:tasks.workunit.client.0.smithi136.stderr:218269: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-15T21:04:59.094 INFO:tasks.workunit.client.0.smithi136.stderr:218269: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-15T21:04:59.094 INFO:tasks.workunit.client.0.smithi136.stderr:218269: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 5 SOMETHING remove 2024-09-15T21:04:59.094 INFO:tasks.workunit.client.0.smithi136.stderr:218269: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-15T21:04:59.094 INFO:tasks.workunit.client.0.smithi136.stderr:218269: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-15T21:04:59.094 INFO:tasks.workunit.client.0.smithi136.stderr:218269: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=5 2024-09-15T21:04:59.094 INFO:tasks.workunit.client.0.smithi136.stderr:218269: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-15T21:04:59.094 INFO:tasks.workunit.client.0.smithi136.stderr:218269: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/5 2024-09-15T21:04:59.094 INFO:tasks.workunit.client.0.smithi136.stderr:218269: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/5 SOMETHING remove 2024-09-15T21:04:59.094 INFO:tasks.workunit.client.0.smithi136.stderr:218269: remove 1#2:eb822e21:::SOMETHING:head# 2024-09-15T21:04:59.094 INFO:tasks.workunit.client.0.smithi136.stderr:218269: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 5 2024-09-15T21:04:59.094 INFO:tasks.workunit.client.0.smithi136.stderr:218269: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2024-09-15T21:04:59.094 INFO:tasks.workunit.client.0.smithi136.stderr:218269: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-15T21:04:59.094 INFO:tasks.workunit.client.0.smithi136.stderr:218269: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=5 2024-09-15T21:04:59.095 INFO:tasks.workunit.client.0.smithi136.stderr:218269: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-15T21:04:59.095 INFO:tasks.workunit.client.0.smithi136.stderr:218269: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/5 2024-09-15T21:04:59.095 INFO:tasks.workunit.client.0.smithi136.stderr:218269: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=ce4892c7-6c6d-4b4f-8e6f-07a3e11133e5 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-15T21:04:59.095 INFO:tasks.workunit.client.0.smithi136.stderr:218269: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T21:04:59.095 INFO:tasks.workunit.client.0.smithi136.stderr:218269: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T21:04:59.095 INFO:tasks.workunit.client.0.smithi136.stderr:218269: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T21:04:59.095 INFO:tasks.workunit.client.0.smithi136.stderr:218269: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/5' 2024-09-15T21:04:59.095 INFO:tasks.workunit.client.0.smithi136.stderr:218269: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/5/journal' 2024-09-15T21:04:59.096 INFO:tasks.workunit.client.0.smithi136.stderr:218269: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-15T21:04:59.096 INFO:tasks.workunit.client.0.smithi136.stderr:218269: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-15T21:04:59.096 INFO:tasks.workunit.client.0.smithi136.stderr:218269: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-15T21:04:59.096 INFO:tasks.workunit.client.0.smithi136.stderr:218269: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-15T21:04:59.096 INFO:tasks.workunit.client.0.smithi136.stderr:218269: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-15T21:04:59.096 INFO:tasks.workunit.client.0.smithi136.stderr:218269: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-15T21:04:59.096 INFO:tasks.workunit.client.0.smithi136.stderr:218269: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-15T21:04:59.096 INFO:tasks.workunit.client.0.smithi136.stderr:218269: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T21:04:59.096 INFO:tasks.workunit.client.0.smithi136.stderr:218269: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52602 2024-09-15T21:04:59.096 INFO:tasks.workunit.client.0.smithi136.stderr:218269: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.52602/$cluster-$name.asok' 2024-09-15T21:04:59.096 INFO:tasks.workunit.client.0.smithi136.stderr:218269: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.52602/$cluster-$name.asok' 2024-09-15T21:04:59.096 INFO:tasks.workunit.client.0.smithi136.stderr:218269: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-15T21:04:59.096 INFO:tasks.workunit.client.0.smithi136.stderr:218269: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-15T21:04:59.097 INFO:tasks.workunit.client.0.smithi136.stderr:218269: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-15T21:04:59.097 INFO:tasks.workunit.client.0.smithi136.stderr:218269: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T21:04:59.097 INFO:tasks.workunit.client.0.smithi136.stderr:218269: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T21:04:59.097 INFO:tasks.workunit.client.0.smithi136.stderr:218269: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T21:04:59.097 INFO:tasks.workunit.client.0.smithi136.stderr:218269: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T21:04:59.097 INFO:tasks.workunit.client.0.smithi136.stderr:218269: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-15T21:04:59.097 INFO:tasks.workunit.client.0.smithi136.stderr:218269: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-15T21:04:59.097 INFO:tasks.workunit.client.0.smithi136.stderr:218269: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/5 2024-09-15T21:04:59.097 INFO:tasks.workunit.client.0.smithi136.stderr:218269: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.5 2024-09-15T21:04:59.097 INFO:tasks.workunit.client.0.smithi136.stderr:218269: start osd.5 2024-09-15T21:04:59.097 INFO:tasks.workunit.client.0.smithi136.stderr:218269: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 5 --fsid=ce4892c7-6c6d-4b4f-8e6f-07a3e11133e5 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/5 --osd-journal=td/osd-scrub-repair/5/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.52602/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T21:05:04.062 INFO:tasks.workunit.client.0.smithi136.stderr:218269: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/5/whoami 2024-09-15T21:05:04.062 INFO:tasks.workunit.client.0.smithi136.stderr:218269: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 5 = 5 ']' 2024-09-15T21:05:04.063 INFO:tasks.workunit.client.0.smithi136.stderr:218269: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-15T21:05:04.063 INFO:tasks.workunit.client.0.smithi136.stderr:218269: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-15T21:05:04.063 INFO:tasks.workunit.client.0.smithi136.stderr:218269: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-15T21:05:04.063 INFO:tasks.workunit.client.0.smithi136.stderr:218269: 2024-09-15T21:04:59.120+0000 7fa099844640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:05:04.063 INFO:tasks.workunit.client.0.smithi136.stderr:218269: 2024-09-15T21:04:59.130+0000 7fa099844640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:05:04.063 INFO:tasks.workunit.client.0.smithi136.stderr:218269: 2024-09-15T21:04:59.131+0000 7fa099844640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:05:04.063 INFO:tasks.workunit.client.0.smithi136.stderr:218269: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 5 2024-09-15T21:05:04.064 INFO:tasks.workunit.client.0.smithi136.stderr:218269: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-15T21:05:04.064 INFO:tasks.workunit.client.0.smithi136.stderr:218269: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=5 2024-09-15T21:05:04.064 INFO:tasks.workunit.client.0.smithi136.stderr:218269: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-15T21:05:04.064 INFO:tasks.workunit.client.0.smithi136.stderr:218269: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-15T21:05:04.064 INFO:tasks.workunit.client.0.smithi136.stderr:218269: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:05:04.064 INFO:tasks.workunit.client.0.smithi136.stderr:218269: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-15T21:05:04.064 INFO:tasks.workunit.client.0.smithi136.stderr:218269: 0 2024-09-15T21:05:04.064 INFO:tasks.workunit.client.0.smithi136.stderr:218269: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:05:04.065 INFO:tasks.workunit.client.0.smithi136.stderr:218269: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-15T21:05:04.065 INFO:tasks.workunit.client.0.smithi136.stderr:218269: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:05:04.065 INFO:tasks.workunit.client.0.smithi136.stderr:218269: 2024-09-15T21:05:00.008+0000 7fa099844640 -1 Falling back to public interface 2024-09-15T21:05:04.065 INFO:tasks.workunit.client.0.smithi136.stderr:218269: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:05:04.065 INFO:tasks.workunit.client.0.smithi136.stderr:218269: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:05:04.065 INFO:tasks.workunit.client.0.smithi136.stderr:218269: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-15T21:05:04.066 INFO:tasks.workunit.client.0.smithi136.stderr:218269: 1 2024-09-15T21:05:04.066 INFO:tasks.workunit.client.0.smithi136.stderr:218269: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:05:04.066 INFO:tasks.workunit.client.0.smithi136.stderr:218269: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-15T21:05:04.066 INFO:tasks.workunit.client.0.smithi136.stderr:218269: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:05:04.066 INFO:tasks.workunit.client.0.smithi136.stderr:218269: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:05:04.066 INFO:tasks.workunit.client.0.smithi136.stderr:218269: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:05:04.066 INFO:tasks.workunit.client.0.smithi136.stderr:218269: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-15T21:05:04.067 INFO:tasks.workunit.client.0.smithi136.stderr:218269: 2 2024-09-15T21:05:04.067 INFO:tasks.workunit.client.0.smithi136.stderr:218269: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:05:04.067 INFO:tasks.workunit.client.0.smithi136.stderr:218269: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-15T21:05:04.067 INFO:tasks.workunit.client.0.smithi136.stderr:218269: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:05:04.067 INFO:tasks.workunit.client.0.smithi136.stderr:218269: 2024-09-15T21:05:02.770+0000 7fa099844640 -1 osd.5 108 log_to_monitors true 2024-09-15T21:05:04.067 INFO:tasks.workunit.client.0.smithi136.stderr:218269: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:05:04.067 INFO:tasks.workunit.client.0.smithi136.stderr:218269: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:05:04.067 INFO:tasks.workunit.client.0.smithi136.stderr:218269: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-15T21:05:04.067 INFO:tasks.workunit.client.0.smithi136.stderr:218269: 3 2024-09-15T21:05:04.068 INFO:tasks.workunit.client.0.smithi136.stderr:218269: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:05:04.068 INFO:tasks.workunit.client.0.smithi136.stderr:218269: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-15T21:05:04.344 INFO:tasks.workunit.client.0.smithi136.stderr:218269: osd.5 up in weight 1 up_from 112 up_thru 104 down_at 109 la18267: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/3/whoami 2024-09-15T21:05:04.344 INFO:tasks.workunit.client.0.smithi136.stderr:218267: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 3 = 3 ']' 2024-09-15T21:05:04.345 INFO:tasks.workunit.client.0.smithi136.stderr:218267: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-15T21:05:04.345 INFO:tasks.workunit.client.0.smithi136.stderr:218267: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-15T21:05:04.345 INFO:tasks.workunit.client.0.smithi136.stderr:218267: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-15T21:05:04.345 INFO:tasks.workunit.client.0.smithi136.stderr:218267: 2024-09-15T21:04:59.017+0000 7f024061c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:05:04.345 INFO:tasks.workunit.client.0.smithi136.stderr:218267: 2024-09-15T21:04:59.040+0000 7f024061c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:05:04.345 INFO:tasks.workunit.client.0.smithi136.stderr:218267: 2024-09-15T21:04:59.041+0000 7f024061c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:05:04.345 INFO:tasks.workunit.client.0.smithi136.stderr:218267: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 3 2024-09-15T21:05:04.345 INFO:tasks.workunit.client.0.smithi136.stderr:218267: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-15T21:05:04.345 INFO:tasks.workunit.client.0.smithi136.stderr:218267: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=3 2024-09-15T21:05:04.345 INFO:tasks.workunit.client.0.smithi136.stderr:218267: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-15T21:05:04.346 INFO:tasks.workunit.client.0.smithi136.stderr:218267: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-15T21:05:04.346 INFO:tasks.workunit.client.0.smithi136.stderr:218267: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:05:04.346 INFO:tasks.workunit.client.0.smithi136.stderr:218267: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-15T21:05:04.346 INFO:tasks.workunit.client.0.smithi136.stderr:218267: 0 2024-09-15T21:05:04.346 INFO:tasks.workunit.client.0.smithi136.stderr:218267: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:05:04.346 INFO:tasks.workunit.client.0.smithi136.stderr:218267: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-15T21:05:04.346 INFO:tasks.workunit.client.0.smithi136.stderr:218267: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:05:04.346 INFO:tasks.workunit.client.0.smithi136.stderr:218267: 2024-09-15T21:05:00.407+0000 7f024061c640 -1 Falling back to public interface 2024-09-15T21:05:04.346 INFO:tasks.workunit.client.0.smithi136.stderr:218267: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:05:04.346 INFO:tasks.workunit.client.0.smithi136.stderr:218267: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:05:04.346 INFO:tasks.workunit.client.0.smithi136.stderr:218267: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-15T21:05:04.347 INFO:tasks.workunit.client.0.smithi136.stderr:218267: 1 2024-09-15T21:05:04.347 INFO:tasks.workunit.client.0.smithi136.stderr:218267: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:05:04.347 INFO:tasks.workunit.client.0.smithi136.stderr:218267: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-15T21:05:04.347 INFO:tasks.workunit.client.0.smithi136.stderr:218267: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:05:04.347 INFO:tasks.workunit.client.0.smithi136.stderr:218267: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:05:04.347 INFO:tasks.workunit.client.0.smithi136.stderr:218267: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:05:04.347 INFO:tasks.workunit.client.0.smithi136.stderr:218267: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-15T21:05:04.347 INFO:tasks.workunit.client.0.smithi136.stderr:218267: 2 2024-09-15T21:05:04.347 INFO:tasks.workunit.client.0.smithi136.stderr:218267: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:05:04.347 INFO:tasks.workunit.client.0.smithi136.stderr:218267: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-15T21:05:04.347 INFO:tasks.workunit.client.0.smithi136.stderr:218267: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:05:04.347 INFO:tasks.workunit.client.0.smithi136.stderr:218267: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:05:04.347 INFO:tasks.workunit.client.0.smithi136.stderr:218267: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:05:04.347 INFO:tasks.workunit.client.0.smithi136.stderr:218267: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-15T21:05:04.348 INFO:tasks.workunit.client.0.smithi136.stderr:218267: 3 2024-09-15T21:05:04.348 INFO:tasks.workunit.client.0.smithi136.stderr:218267: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:05:04.348 INFO:tasks.workunit.client.0.smithi136.stderr:218267: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-15T21:05:04.348 INFO:tasks.workunit.client.0.smithi136.stderr:218267: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:05:05.375 INFO:tasks.workunit.client.0.smithi136.stderr:218267: 2024-09-15T21:05:04.341+0000 7f024061c640 -st_clean_interval [104,108) [v2:127.0.0.1:6826/2071297435,v1:127.0.0.1:6827/2071297435] [v2:127.0.0.1:6828/2071297435,v1:127.0.0.1:6829/2071297435] exists,up c3187161-e486-4b18-8257-d69c125e9792 2024-09-15T21:05:05.376 INFO:tasks.workunit.client.0.smithi136.stderr:218269: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-15T21:05:05.376 INFO:tasks.workunit.client.0.smithi136.stderr:218269: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-15T21:05:05.376 INFO:tasks.workunit.client.0.smithi136.stderr:218269: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-15T21:05:05.376 INFO:tasks.workunit.client.0.smithi136.stderr:218269: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2024-09-15T21:05:05.376 INFO:tasks.workunit.client.0.smithi136.stderr:218269: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-15T21:05:05.376 INFO:tasks.workunit.client.0.smithi136.stderr:218269: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-15T21:05:05.376 INFO:tasks.workunit.client.0.smithi136.stderr:218269: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-15T21:05:05.376 INFO:tasks.workunit.client.0.smithi136.stderr:218269: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-15T21:05:05.376 INFO:tasks.workunit.client.0.smithi136.stderr:218269: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-15T21:05:05.376 INFO:tasks.workunit.client.0.smithi136.stderr:218269: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-15T21:05:05.376 INFO:tasks.workunit.client.0.smithi136.stderr:218269: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-15T21:05:05.376 INFO:tasks.workunit.client.0.smithi136.stderr:218269: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-15T21:05:05.377 INFO:tasks.workunit.client.0.smithi136.stderr:218269: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-15T21:05:05.377 INFO:tasks.workunit.client.0.smithi136.stderr:218269: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-15T21:05:05.377 INFO:tasks.workunit.client.0.smithi136.stderr:218269: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-15T21:05:05.377 INFO:tasks.workunit.client.0.smithi136.stderr:218269: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-15T21:05:05.377 INFO:tasks.workunit.client.0.smithi136.stderr:218269: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-15T21:05:05.377 INFO:tasks.workunit.client.0.smithi136.stderr:218269: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-15T21:05:05.377 INFO:tasks.workunit.client.0.smithi136.stderr:218269: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-15T21:05:05.377 INFO:tasks.workunit.client.0.smithi136.stderr:218269: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-15T21:05:05.377 INFO:tasks.workunit.client.0.smithi136.stderr:218269: 1 2024-09-15T21:05:05.377 INFO:tasks.workunit.client.0.smithi136.stderr:218269: 2 2024-09-15T21:05:05.377 INFO:tasks.workunit.client.0.smithi136.stderr:218269: 3 2024-09-15T21:05:05.377 INFO:tasks.workunit.client.0.smithi136.stderr:218269: 4 2024-09-15T21:05:05.377 INFO:tasks.workunit.client.0.smithi136.stderr:218269: 5 2024-09-15T21:05:05.377 INFO:tasks.workunit.client.0.smithi136.stderr:218269: 6 2024-09-15T21:05:05.378 INFO:tasks.workunit.client.0.smithi136.stderr:218269: 7 2024-09-15T21:05:05.378 INFO:tasks.workunit.client.0.smithi136.stderr:218269: 8 2024-09-15T21:05:05.378 INFO:tasks.workunit.client.0.smithi136.stderr:218269: 9' 2024-09-15T21:05:05.378 INFO:tasks.workunit.client.0.smithi136.stderr:218269: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-15T21:05:05.378 INFO:tasks.workunit.client.0.smithi136.stderr:218269: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T21:05:05.378 INFO:tasks.workunit.client.0.smithi136.stderr:218269: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-15T21:05:05.378 INFO:tasks.workunit.client.0.smithi136.stderr:218269: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836589 2024-09-15T21:05:05.378 INFO:tasks.workunit.client.0.smithi136.stderr:218269: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836589 2024-09-15T21:05:05.378 INFO:tasks.workunit.client.0.smithi136.stderr:218269: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836589' 2024-09-15T21:05:05.378 INFO:tasks.workunit.client.0.smithi136.stderr:218269: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T21:05:05.378 INFO:tasks.workunit.client.0.smithi136.stderr:218269: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-15T21:05:05.378 INFO:tasks.workunit.client.0.smithi136.stderr:218269: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=42949673059 2024-09-15T21:05:05.378 INFO:tasks.workunit.client.0.smithi136.stderr:218269: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 42949673059 2024-09-15T21:05:05.378 INFO:tasks.workunit.client.0.smithi136.stderr:218269: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836589 1-42949673059' 2024-09-15T21:05:05.379 INFO:tasks.workunit.client.0.smithi136.stderr:218269: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T21:05:06.725 INFO:tasks.workunit.client.0.smithi136.stderr:218269: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-15T21:05:06.725 INFO:tasks.workunit.client.0.smithi136.stderr:218269: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=64424509534 2024-09-15T21:05:06.725 INFO:tasks.workunit.client.0.smithi136.stderr:218269: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 64424509534 2024-09-15T21:05:06.725 INFO:tasks.workunit.client.0.smithi136.stderr:218269: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836589 1-42949673059 2-64424509534' 2024-09-15T21:05:06.725 INFO:tasks.workunit.client.0.smithi136.stderr:218269: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T21:05:06.725 INFO:tasks.workunit.client.0.smithi136.stderr:218269: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-15T21:05:06.725 INFO:tasks.workunit.client.0.smithi136.stderr:218269: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=493921239042 2024-09-15T21:05:06.726 INFO:tasks.workunit.client.0.smithi136.stderr:218269: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 493921239042 2024-09-15T21:05:06.726 INFO:tasks.workunit.client.0.smithi136.stderr:218269: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836589 1-42949673059 2-64424509534 3-493921239042' 2024-09-15T21:05:06.726 INFO:tasks.workunit.client.0.smithi136.stderr:218269: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T21:05:06.726 INFO:tasks.workunit.client.0.smithi136.stderr:218269: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-09-15T21:05:06.726 INFO:tasks.workunit.client.0.smithi136.stderr:218269: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=107374182479 2024-09-15T21:05:06.726 INFO:tasks.workunit.client.0.smithi136.stderr:218269: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 107374182479 2024-09-15T21:05:06.726 INFO:tasks.workunit.client.0.smithi136.stderr:218269: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836589 1-42949673059 2-64424509534 3-493921239042 4-107374182479' 2024-09-15T21:05:06.726 INFO:tasks.workunit.client.0.smithi136.stderr:218269: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T21:05:06.726 INFO:tasks.workunit.client.0.smithi136.stderr:218269: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-09-15T21:05:06.726 INFO:tasks.workunit.client.0.smithi136.stderr:218269: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=481036337154 2024-09-15T21:05:06.726 INFO:tasks.workunit.client.0.smithi136.stderr:218269: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 481036337154 2024-09-15T21:05:06.726 INFO:tasks.workunit.client.0.smithi136.stderr:218269: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836589 1-42949673059 2-64424509534 3-493921239042 4-107374182479 5-481036337154' 2024-09-15T21:05:06.726 INFO:tasks.workunit.client.0.smithi136.stderr:218269: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T21:05:06.726 INFO:tasks.workunit.client.0.smithi136.stderr:218269: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-09-15T21:05:06.727 INFO:tasks.workunit.client.0.smithi136.stderr:218269: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=150323855428 2024-09-15T21:05:06.727 INFO:tasks.workunit.client.0.smithi136.stderr:218269: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 150323855428 2024-09-15T21:05:06.727 INFO:tasks.workunit.client.0.smithi136.stderr:218269: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836589 1-42949673059 2-64424509534 3-493921239042 4-107374182479 5-481036337154 6-150323855428' 2024-09-15T21:05:06.727 INFO:tasks.workunit.client.0.smithi136.stderr:218269: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T21:05:06.727 INFO:tasks.workunit.client.0.smithi136.stderr:218269: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2024-09-15T21:05:06.727 INFO:tasks.workunit.client.0.smithi136.stderr:218269: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=171798691900 2024-09-15T21:05:06.727 INFO:tasks.workunit.client.0.smithi136.stderr:218269: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 171798691900 2024-09-15T21:05:06.727 INFO:tasks.workunit.client.0.smithi136.stderr:218269: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836589 1-42949673059 2-64424509534 3-493921239042 4-107374182479 5-481036337154 6-150323855428 7-171798691900' 2024-09-15T21:05:06.727 INFO:tasks.workunit.client.0.smithi136.stderr:218269: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T21:05:06.727 INFO:tasks.workunit.client.0.smithi136.stderr:218269: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2024-09-15T21:05:07.202 INFO:tasks.workunit.client.0.smithi136.stderr:218269: /home/ubuntu/cephtest1 osd.3 108 log_to_monitors true 2024-09-15T21:05:07.203 INFO:tasks.workunit.client.0.smithi136.stderr:218267: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:05:07.203 INFO:tasks.workunit.client.0.smithi136.stderr:218267: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:05:07.203 INFO:tasks.workunit.client.0.smithi136.stderr:218267: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-15T21:05:07.203 INFO:tasks.workunit.client.0.smithi136.stderr:218267: 4 2024-09-15T21:05:07.203 INFO:tasks.workunit.client.0.smithi136.stderr:218267: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:05:07.203 INFO:tasks.workunit.client.0.smithi136.stderr:218267: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-15T21:05:07.203 INFO:tasks.workunit.client.0.smithi136.stderr:218267: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:05:07.203 INFO:tasks.workunit.client.0.smithi136.stderr:218267: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:05:07.203 INFO:tasks.workunit.client.0.smithi136.stderr:218267: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:05:07.203 INFO:tasks.workunit.client.0.smithi136.stderr:218267: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-15T21:05:07.203 INFO:tasks.workunit.client.0.smithi136.stderr:218267: 5 2024-09-15T21:05:07.203 INFO:tasks.workunit.client.0.smithi136.stderr:218267: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:05:07.204 INFO:tasks.workunit.client.0.smithi136.stderr:218267: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-15T21:05:07.204 INFO:tasks.workunit.client.0.smithi136.stderr:218267: osd.3 up in weight 1 up_from 115 up_thru 115 down_at 109 last_clean_interval [73,108) [v2:127.0.0.1:6874/193328311,v1:127.0.0.1:6875/193328311] [v2:127.0.0.1:6876/193328311,v1:127.0.0.1:6877/193328311] exists,up 0eed652b-57cc-457d-986e-cdd453c584f5 2024-09-15T21:05:07.204 INFO:tasks.workunit.client.0.smithi136.stderr:218267: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-15T21:05:07.204 INFO:tasks.workunit.client.0.smithi136.stderr:218267: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-15T21:05:07.204 INFO:tasks.workunit.client.0.smithi136.stderr:218267: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-15T21:05:07.204 INFO:tasks.workunit.client.0.smithi136.stderr:218267: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2024-09-15T21:05:07.204 INFO:tasks.workunit.client.0.smithi136.stderr:218267: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-15T21:05:07.204 INFO:tasks.workunit.client.0.smithi136.stderr:218267: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-15T21:05:07.204 INFO:tasks.workunit.client.0.smithi136.stderr:218267: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-15T21:05:07.204 INFO:tasks.workunit.client.0.smithi136.stderr:218267: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-15T21:05:07.204 INFO:tasks.workunit.client.0.smithi136.stderr:218267: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-15T21:05:07.204 INFO:tasks.workunit.client.0.smithi136.stderr:218267: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-15T21:05:07.204 INFO:tasks.workunit.client.0.smithi136.stderr:218267: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-15T21:05:07.205 INFO:tasks.workunit.client.0.smithi136.stderr:218267: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-15T21:05:07.205 INFO:tasks.workunit.client.0.smithi136.stderr:218267: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-15T21:05:07.205 INFO:tasks.workunit.client.0.smithi136.stderr:218267: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-15T21:05:07.205 INFO:tasks.workunit.client.0.smithi136.stderr:218267: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-15T21:05:07.205 INFO:tasks.workunit.client.0.smithi136.stderr:218267: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-15T21:05:07.205 INFO:tasks.workunit.client.0.smithi136.stderr:218267: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-15T21:05:07.205 INFO:tasks.workunit.client.0.smithi136.stderr:218267: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-15T21:05:07.205 INFO:tasks.workunit.client.0.smithi136.stderr:218267: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-15T21:05:07.205 INFO:tasks.workunit.client.0.smithi136.stderr:218267: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-15T21:05:07.205 INFO:tasks.workunit.client.0.smithi136.stderr:218267: 1 2024-09-15T21:05:07.205 INFO:tasks.workunit.client.0.smithi136.stderr:218267: 2 2024-09-15T21:05:07.205 INFO:tasks.workunit.client.0.smithi136.stderr:218267: 3 2024-09-15T21:05:07.205 INFO:tasks.workunit.client.0.smithi136.stderr:218267: 4 2024-09-15T21:05:07.205 INFO:tasks.workunit.client.0.smithi136.stderr:218267: 5 2024-09-15T21:05:07.206 INFO:tasks.workunit.client.0.smithi136.stderr:218267: 6 2024-09-15T21:05:07.206 INFO:tasks.workunit.client.0.smithi136.stderr:218267: 7 2024-09-15T21:05:07.206 INFO:tasks.workunit.client.0.smithi136.stderr:218267: 8 2024-09-15T21:05:07.206 INFO:tasks.workunit.client.0.smithi136.stderr:218267: 9' 2024-09-15T21:05:07.206 INFO:tasks.workunit.client.0.smithi136.stderr:218267: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-15T21:05:07.949 INFO:tasks.workunit.client.0.smithi136.stderr:218267: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T21:05:07.949 INFO:tasks.workunit.client.0.smithi136.stderr:218267: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-15T21:05:07.950 INFO:tasks.workunit.client.0.smithi136.stderr:218267: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836591 2024-09-15T21:05:07.950 INFO:tasks.workunit.client.0.smithi136.stderr:218267: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836591 2024-09-15T21:05:07.950 INFO:tasks.workunit.client.0.smithi136.stderr:218267: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836591' 2024-09-15T21:05:07.950 INFO:tasks.workunit.client.0.smithi136.stderr:218267: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T21:05:07.950 INFO:tasks.workunit.client.0.smithi136.stderr:218267: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-15T21:05:07.950 INFO:tasks.workunit.client.0.smithi136.stderr:218267: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=42949673061 2024-09-15T21:05:07.950 INFO:tasks.workunit.client.0.smithi136.stderr:218267: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 42949673061 2024-09-15T21:05:07.950 INFO:tasks.workunit.client.0.smithi136.stderr:218267: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836591 1-42949673061' 2024-09-15T21:05:07.950 INFO:tasks.workunit.client.0.smithi136.stderr:218267: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T21:05:07.950 INFO:tasks.workunit.client.0.smithi136.stderr:218267: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-15T21:05:07.950 INFO:tasks.workunit.client.0.smithi136.stderr:218267: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=64424509536 2024-09-15T21:05:07.950 INFO:tasks.workunit.client.0.smithi136.stderr:218267: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 64424509536 2024-09-15T21:05:07.950 INFO:tasks.workunit.client.0.smithi136.stderr:218267: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836591 1-42949673061 2-64424509536' 2024-09-15T21:05:07.951 INFO:tasks.workunit.client.0.smithi136.stderr:218267: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T21:05:07.951 INFO:tasks.workunit.client.0.smithi136.stderr:218267: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-15T21:05:07.951 INFO:tasks.workunit.client.0.smithi136.stderr:218267: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=493921239043 2024-09-15T21:05:07.951 INFO:tasks.workunit.client.0.smithi136.stderr:218267: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 493921239043 2024-09-15T21:05:07.951 INFO:tasks.workunit.client.0.smithi136.stderr:218267: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836591 1-42949673061 2-64424509536 3-493921239043' 2024-09-15T21:05:07.951 INFO:tasks.workunit.client.0.smithi136.stderr:218267: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T21:05:07.951 INFO:tasks.workunit.client.0.smithi136.stderr:218267: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-09-15T21:05:07.951 INFO:tasks.workunit.client.0.smithi136.stderr:218267: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=107374182480 2024-09-15T21:05:07.951 INFO:tasks.workunit.client.0.smithi136.stderr:218267: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 107374182480 2024-09-15T21:05:07.951 INFO:tasks.workunit.client.0.smithi136.stderr:218267: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836591 1-42949673061 2-64424509536 3-493921239043 4-107374182480' 2024-09-15T21:05:07.951 INFO:tasks.workunit.client.0.smithi136.stderr:218267: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T21:05:07.951 INFO:tasks.workunit.client.0.smithi136.stderr:218267: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-09-15T21:05:07.951 INFO:tasks.workunit.client.0.smithi136.stderr:218267: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=481036337155 2024-09-15T21:05:07.951 INFO:tasks.workunit.client.0.smithi136.stderr:218267: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 481036337155 2024-09-15T21:05:07.952 INFO:tasks.workunit.client.0.smithi136.stderr:218267: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836591 1-42949673061 2-64424509536 3-493921239043 4-107374182480 5-481036337155' 2024-09-15T21:05:07.952 INFO:tasks.workunit.client.0.smithi136.stderr:218267: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T21:05:07.952 INFO:tasks.workunit.client.0.smithi136.stderr:218267: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-09-15T21:05:07.952 INFO:tasks.workunit.client.0.smithi136.stderr:218267: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=150323855429 2024-09-15T21:05:08.741 INFO:tasks.workunit.client.0.smithi136.stderr:218267: /clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=193273528376 2024-09-15T21:05:08.741 INFO:tasks.workunit.client.0.smithi136.stderr:218269: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 193273528376 2024-09-15T21:05:08.741 INFO:tasks.workunit.client.0.smithi136.stderr:218269: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836589 1-42949673059 2-64424509534 3-493921239042 4-107374182479 5-481036337154 6-150323855428 7-171798691900 8-193273528376' 2024-09-15T21:05:08.741 INFO:tasks.workunit.client.0.smithi136.stderr:218269: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T21:05:08.741 INFO:tasks.workunit.client.0.smithi136.stderr:218269: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2024-09-15T21:05:08.741 INFO:tasks.workunit.client.0.smithi136.stderr:218269: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=450971566087 2024-09-15T21:05:08.741 INFO:tasks.workunit.client.0.smithi136.stderr:218269: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 450971566087 2024-09-15T21:05:08.741 INFO:tasks.workunit.client.0.smithi136.stderr:218269: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836589 1-42949673059 2-64424509534 3-493921239042 4-107374182479 5-481036337154 6-150323855428 7-171798691900 8-193273528376 9-450971566087' 2024-09-15T21:05:08.741 INFO:tasks.workunit.client.0.smithi136.stderr:218269: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T21:05:08.742 INFO:tasks.workunit.client.0.smithi136.stderr:218269: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836589 2024-09-15T21:05:08.742 INFO:tasks.workunit.client.0.smithi136.stderr:218269: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T21:05:08.742 INFO:tasks.workunit.client.0.smithi136.stderr:218269: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-15T21:05:08.742 INFO:tasks.workunit.client.0.smithi136.stderr:218269: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836589 2024-09-15T21:05:08.742 INFO:tasks.workunit.client.0.smithi136.stderr:218269: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T21:05:08.742 INFO:tasks.workunit.client.0.smithi136.stderr:218269: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836589 2024-09-15T21:05:08.742 INFO:tasks.workunit.client.0.smithi136.stderr:218269: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836589' 2024-09-15T21:05:08.742 INFO:tasks.workunit.client.0.smithi136.stderr:218269: waiting osd.0 seq 21474836589 2024-09-15T21:05:08.742 INFO:tasks.workunit.client.0.smithi136.stderr:218269: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T21:05:08.742 INFO:tasks.workunit.client.0.smithi136.stderr:218269: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836588 -lt 21474836589 2024-09-15T21:05:08.742 INFO:tasks.workunit.client.0.smithi136.stderr:218269: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-15T21:05:08.742 INFO:tasks.workunit.client.0.smithi136.stderr:218269: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-15T21:05:08.743 INFO:tasks.workunit.client.0.smithi136.stderr:218269: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T21:05:08.743 INFO:tasks.workunit.client.0.smithi136.stderr:218269: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836590 -lt 21474836589 2024-09-15T21:05:08.743 INFO:tasks.workunit.client.0.smithi136.stderr:218269: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T21:05:08.743 INFO:tasks.workunit.client.0.smithi136.stderr:218269: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-42949673059 2024-09-15T21:05:08.743 INFO:tasks.workunit.client.0.smithi136.stderr:218269: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T21:05:08.743 INFO:tasks.workunit.client.0.smithi136.stderr:218269: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-15T21:05:08.743 INFO:tasks.workunit.client.0.smithi136.stderr:218269: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-42949673059 2024-09-15T21:05:08.743 INFO:tasks.workunit.client.0.smithi136.stderr:218269: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T21:05:08.743 INFO:tasks.workunit.client.0.smithi136.stderr:218269: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=42949673059 2024-09-15T21:05:08.743 INFO:tasks.workunit.client.0.smithi136.stderr:218269: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 42949673059' 2024-09-15T21:05:08.743 INFO:tasks.workunit.client.0.smithi136.stderr:218269: waiting osd.1 seq 42949673059 2024-09-15T21:05:08.743 INFO:tasks.workunit.client.0.smithi136.stderr:218269: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-15T21:05:08.743 INFO:tasks.workunit.client.0.smithi136.stderr:218269: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 42949673060 -lt 42949673059 2024-09-15T21:05:09.657 INFO:tasks.workunit.client.0.smithi136.stderr:218269: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T21:05:09.657 INFO:tasks.workunit.client.0.smithi136.stderr:218269: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-64424509534 2024-09-15T21:05:09.657 INFO:tasks.workunit.client.0.smithi136.stderr:218269: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T21:05:09.657 INFO:tasks.workunit.client.0.smithi136.stderr:218269: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-15T21:05:09.657 INFO:tasks.workunit.client.0.smithi136.stderr:218269: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-64424509534 2024-09-15T21:05:09.657 INFO:tasks.workunit.client.0.smithi136.stderr:218269: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T21:05:09.658 INFO:tasks.workunit.client.0.smithi136.stderr:218269: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=64424509534 2024-09-15T21:05:09.658 INFO:tasks.workunit.client.0.smithi136.stderr:218269: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 64424509534' 2024-09-15T21:05:09.658 INFO:tasks.workunit.client.0.smithi136.stderr:218269: waiting osd.2 seq 64424509534 2024-09-15T21:05:09.658 INFO:tasks.workunit.client.0.smithi136.stderr:218269: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-15T21:05:09.658 INFO:tasks.workunit.client.0.smithi136.stderr:218269: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 64424509536 -lt 64424509534 2024-09-15T21:05:09.658 INFO:tasks.workunit.client.0.smithi136.stderr:218269: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T21:05:09.658 INFO:tasks.workunit.client.0.smithi136.stderr:218269: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 3-493921239042 2024-09-15T21:05:09.658 INFO:tasks.workunit.client.0.smithi136.stderr:218269: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T21:05:09.658 INFO:tasks.workunit.client.0.smithi136.stderr:218269: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=3 2024-09-15T21:05:09.658 INFO:tasks.workunit.client.0.smithi136.stderr:218269: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 3-493921239042 2024-09-15T21:05:09.658 INFO:tasks.workunit.client.0.smithi136.stderr:218269: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T21:05:09.659 INFO:tasks.workunit.client.0.smithi136.stderr:218269: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=493921239042 2024-09-15T21:05:09.659 INFO:tasks.workunit.client.0.smithi136.stderr:218269: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.3 seq 493921239042' 2024-09-15T21:05:09.659 INFO:tasks.workunit.client.0.smithi136.stderr:218269: waiting osd.3 seq 493921239042 2024-09-15T21:05:09.659 INFO:tasks.workunit.client.0.smithi136.stderr:218269: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-15T21:05:09.659 INFO:tasks.workunit.client.0.smithi136.stderr:218269: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 493921239043 -lt 493921239042 2024-09-15T21:05:09.659 INFO:tasks.workunit.client.0.smithi136.stderr:218269: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T21:05:09.659 INFO:tasks.workunit.client.0.smithi136.stderr:218269: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 4-107374182479 2024-09-15T21:05:09.659 INFO:tasks.workunit.client.0.smithi136.stderr:218269: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T21:05:09.659 INFO:tasks.workunit.client.0.smithi136.stderr:218269: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=4 2024-09-15T21:05:09.659 INFO:tasks.workunit.client.0.smithi136.stderr:218269: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 4-107374182479 2024-09-15T21:05:09.660 INFO:tasks.workunit.client.0.smithi136.stderr:218269: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T21:05:09.660 INFO:tasks.workunit.client.0.smithi136.stderr:218269: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=107374182479 2024-09-15T21:05:09.660 INFO:tasks.workunit.client.0.smithi136.stderr:218269: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.4 seq 107374182479' 2024-09-15T21:05:09.660 INFO:tasks.workunit.client.0.smithi136.stderr:218269: waiting osd.4 seq 107374182479 2024-09-15T21:05:09.660 INFO:tasks.workunit.client.0.smithi136.stderr:218269: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 4 2024-09-15T21:05:09.660 INFO:tasks.workunit.client.0.smithi136.stderr:218269: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 107374182481 -lt 107374182479 2024-09-15T21:05:09.660 INFO:tasks.workunit.client.0.smithi136.stderr:218269: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T21:05:09.660 INFO:tasks.workunit.client.0.smithi136.stderr:218269: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 5-481036337154 2024-09-15T21:05:09.660 INFO:tasks.workunit.client.0.smithi136.stderr:218269: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T21:05:09.888 INFO:tasks.workunit.client.0.smithi136.stderr:218269: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:22/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 150323855429 2024-09-15T21:05:09.888 INFO:tasks.workunit.client.0.smithi136.stderr:218267: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836591 1-42949673061 2-64424509536 3-493921239043 4-107374182480 5-481036337155 6-150323855429' 2024-09-15T21:05:09.888 INFO:tasks.workunit.client.0.smithi136.stderr:218267: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T21:05:09.889 INFO:tasks.workunit.client.0.smithi136.stderr:218267: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2024-09-15T21:05:09.889 INFO:tasks.workunit.client.0.smithi136.stderr:218267: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=171798691901 2024-09-15T21:05:09.889 INFO:tasks.workunit.client.0.smithi136.stderr:218267: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 171798691901 2024-09-15T21:05:09.889 INFO:tasks.workunit.client.0.smithi136.stderr:218267: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836591 1-42949673061 2-64424509536 3-493921239043 4-107374182480 5-481036337155 6-150323855429 7-171798691901' 2024-09-15T21:05:09.889 INFO:tasks.workunit.client.0.smithi136.stderr:218267: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T21:05:09.889 INFO:tasks.workunit.client.0.smithi136.stderr:218267: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2024-09-15T21:05:09.889 INFO:tasks.workunit.client.0.smithi136.stderr:218267: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=193273528377 2024-09-15T21:05:09.889 INFO:tasks.workunit.client.0.smithi136.stderr:218267: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 193273528377 2024-09-15T21:05:09.889 INFO:tasks.workunit.client.0.smithi136.stderr:218267: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836591 1-42949673061 2-64424509536 3-493921239043 4-107374182480 5-481036337155 6-150323855429 7-171798691901 8-193273528377' 2024-09-15T21:05:09.889 INFO:tasks.workunit.client.0.smithi136.stderr:218267: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T21:05:09.889 INFO:tasks.workunit.client.0.smithi136.stderr:218267: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2024-09-15T21:05:09.889 INFO:tasks.workunit.client.0.smithi136.stderr:218267: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=450971566089 2024-09-15T21:05:09.889 INFO:tasks.workunit.client.0.smithi136.stderr:218267: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 450971566089 2024-09-15T21:05:09.890 INFO:tasks.workunit.client.0.smithi136.stderr:218267: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836591 1-42949673061 2-64424509536 3-493921239043 4-107374182480 5-481036337155 6-150323855429 7-171798691901 8-193273528377 9-450971566089' 2024-09-15T21:05:09.890 INFO:tasks.workunit.client.0.smithi136.stderr:218267: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T21:05:09.890 INFO:tasks.workunit.client.0.smithi136.stderr:218267: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836591 2024-09-15T21:05:09.890 INFO:tasks.workunit.client.0.smithi136.stderr:218267: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T21:05:09.890 INFO:tasks.workunit.client.0.smithi136.stderr:218267: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-15T21:05:09.890 INFO:tasks.workunit.client.0.smithi136.stderr:218267: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836591 2024-09-15T21:05:09.890 INFO:tasks.workunit.client.0.smithi136.stderr:218267: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T21:05:09.890 INFO:tasks.workunit.client.0.smithi136.stderr:218267: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836591 2024-09-15T21:05:09.890 INFO:tasks.workunit.client.0.smithi136.stderr:218267: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836591' 2024-09-15T21:05:09.890 INFO:tasks.workunit.client.0.smithi136.stderr:218267: waiting osd.0 seq 21474836591 2024-09-15T21:05:09.890 INFO:tasks.workunit.client.0.smithi136.stderr:218267: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T21:05:09.890 INFO:tasks.workunit.client.0.smithi136.stderr:218267: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836590 -lt 21474836591 2024-09-15T21:05:09.891 INFO:tasks.workunit.client.0.smithi136.stderr:218267: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-15T21:05:09.891 INFO:tasks.workunit.client.0.smithi136.stderr:218267: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-15T21:05:09.891 INFO:tasks.workunit.client.0.smithi136.stderr:218267: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T21:05:10.576 INFO:tasks.workunit.client.0.smithi136.stderr:218267: /home/ubuntu/cephtest/clone.client.0/qa/standalone/cep19: flush_pg_stats: osd=5 2024-09-15T21:05:10.577 INFO:tasks.workunit.client.0.smithi136.stderr:218269: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 5-481036337154 2024-09-15T21:05:10.577 INFO:tasks.workunit.client.0.smithi136.stderr:218269: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T21:05:10.577 INFO:tasks.workunit.client.0.smithi136.stderr:218269: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=481036337154 2024-09-15T21:05:10.577 INFO:tasks.workunit.client.0.smithi136.stderr:218269: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.5 seq 481036337154' 2024-09-15T21:05:10.578 INFO:tasks.workunit.client.0.smithi136.stderr:218269: waiting osd.5 seq 481036337154 2024-09-15T21:05:10.578 INFO:tasks.workunit.client.0.smithi136.stderr:218269: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 5 2024-09-15T21:05:10.578 INFO:tasks.workunit.client.0.smithi136.stderr:218269: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 481036337156 -lt 481036337154 2024-09-15T21:05:10.578 INFO:tasks.workunit.client.0.smithi136.stderr:218269: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T21:05:10.578 INFO:tasks.workunit.client.0.smithi136.stderr:218269: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 6-150323855428 2024-09-15T21:05:10.578 INFO:tasks.workunit.client.0.smithi136.stderr:218269: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T21:05:10.579 INFO:tasks.workunit.client.0.smithi136.stderr:218269: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=6 2024-09-15T21:05:10.579 INFO:tasks.workunit.client.0.smithi136.stderr:218269: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 6-150323855428 2024-09-15T21:05:10.579 INFO:tasks.workunit.client.0.smithi136.stderr:218269: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T21:05:10.579 INFO:tasks.workunit.client.0.smithi136.stderr:218269: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=150323855428 2024-09-15T21:05:10.579 INFO:tasks.workunit.client.0.smithi136.stderr:218269: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.6 seq 150323855428' 2024-09-15T21:05:10.579 INFO:tasks.workunit.client.0.smithi136.stderr:218269: waiting osd.6 seq 150323855428 2024-09-15T21:05:10.580 INFO:tasks.workunit.client.0.smithi136.stderr:218269: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 6 2024-09-15T21:05:10.580 INFO:tasks.workunit.client.0.smithi136.stderr:218269: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 150323855429 -lt 150323855428 2024-09-15T21:05:10.580 INFO:tasks.workunit.client.0.smithi136.stderr:218269: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T21:05:10.580 INFO:tasks.workunit.client.0.smithi136.stderr:218269: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 7-171798691900 2024-09-15T21:05:10.580 INFO:tasks.workunit.client.0.smithi136.stderr:218269: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T21:05:10.580 INFO:tasks.workunit.client.0.smithi136.stderr:218269: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=7 2024-09-15T21:05:10.581 INFO:tasks.workunit.client.0.smithi136.stderr:218269: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 7-171798691900 2024-09-15T21:05:10.581 INFO:tasks.workunit.client.0.smithi136.stderr:218269: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T21:05:10.581 INFO:tasks.workunit.client.0.smithi136.stderr:218269: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=171798691900 2024-09-15T21:05:10.581 INFO:tasks.workunit.client.0.smithi136.stderr:218269: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.7 seq 171798691900' 2024-09-15T21:05:10.581 INFO:tasks.workunit.client.0.smithi136.stderr:218269: waiting osd.7 seq 171798691900 2024-09-15T21:05:10.581 INFO:tasks.workunit.client.0.smithi136.stderr:218269: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 7 2024-09-15T21:05:10.581 INFO:tasks.workunit.client.0.smithi136.stderr:218269: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 171798691901 -lt 171798691900 2024-09-15T21:05:10.581 INFO:tasks.workunit.client.0.smithi136.stderr:218269: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T21:05:10.581 INFO:tasks.workunit.client.0.smithi136.stderr:218269: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 8-193273528376 2024-09-15T21:05:10.581 INFO:tasks.workunit.client.0.smithi136.stderr:218269: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T21:05:10.581 INFO:tasks.workunit.client.0.smithi136.stderr:218269: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=8 2024-09-15T21:05:10.582 INFO:tasks.workunit.client.0.smithi136.stderr:218269: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 8-193273528376 2024-09-15T21:05:10.582 INFO:tasks.workunit.client.0.smithi136.stderr:218269: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T21:05:10.582 INFO:tasks.workunit.client.0.smithi136.stderr:218269: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=193273528376 2024-09-15T21:05:10.786 INFO:tasks.workunit.client.0.smithi136.stderr:218269: /home/ubuntu/cephtest/clone.clh-helpers.sh:2222: flush_pg_stats: test 21474836591 -lt 21474836591 2024-09-15T21:05:10.786 INFO:tasks.workunit.client.0.smithi136.stderr:218267: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T21:05:10.786 INFO:tasks.workunit.client.0.smithi136.stderr:218267: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-42949673061 2024-09-15T21:05:10.786 INFO:tasks.workunit.client.0.smithi136.stderr:218267: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T21:05:10.786 INFO:tasks.workunit.client.0.smithi136.stderr:218267: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-15T21:05:10.786 INFO:tasks.workunit.client.0.smithi136.stderr:218267: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-42949673061 2024-09-15T21:05:10.786 INFO:tasks.workunit.client.0.smithi136.stderr:218267: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T21:05:10.786 INFO:tasks.workunit.client.0.smithi136.stderr:218267: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=42949673061 2024-09-15T21:05:10.786 INFO:tasks.workunit.client.0.smithi136.stderr:218267: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 42949673061' 2024-09-15T21:05:10.786 INFO:tasks.workunit.client.0.smithi136.stderr:218267: waiting osd.1 seq 42949673061 2024-09-15T21:05:10.787 INFO:tasks.workunit.client.0.smithi136.stderr:218267: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-15T21:05:10.787 INFO:tasks.workunit.client.0.smithi136.stderr:218267: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 42949673061 -lt 42949673061 2024-09-15T21:05:10.787 INFO:tasks.workunit.client.0.smithi136.stderr:218267: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T21:05:10.787 INFO:tasks.workunit.client.0.smithi136.stderr:218267: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-64424509536 2024-09-15T21:05:10.787 INFO:tasks.workunit.client.0.smithi136.stderr:218267: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T21:05:10.787 INFO:tasks.workunit.client.0.smithi136.stderr:218267: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-15T21:05:10.787 INFO:tasks.workunit.client.0.smithi136.stderr:218267: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-64424509536 2024-09-15T21:05:10.787 INFO:tasks.workunit.client.0.smithi136.stderr:218267: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T21:05:10.787 INFO:tasks.workunit.client.0.smithi136.stderr:218267: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=64424509536 2024-09-15T21:05:10.787 INFO:tasks.workunit.client.0.smithi136.stderr:218267: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 64424509536' 2024-09-15T21:05:10.787 INFO:tasks.workunit.client.0.smithi136.stderr:218267: waiting osd.2 seq 64424509536 2024-09-15T21:05:10.787 INFO:tasks.workunit.client.0.smithi136.stderr:218267: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-15T21:05:10.787 INFO:tasks.workunit.client.0.smithi136.stderr:218267: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 64424509536 -lt 64424509536 2024-09-15T21:05:10.787 INFO:tasks.workunit.client.0.smithi136.stderr:218267: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T21:05:10.787 INFO:tasks.workunit.client.0.smithi136.stderr:218267: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 3-493921239043 2024-09-15T21:05:10.788 INFO:tasks.workunit.client.0.smithi136.stderr:218267: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T21:05:10.788 INFO:tasks.workunit.client.0.smithi136.stderr:218267: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=3 2024-09-15T21:05:10.788 INFO:tasks.workunit.client.0.smithi136.stderr:218267: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 3-493921239043 2024-09-15T21:05:10.788 INFO:tasks.workunit.client.0.smithi136.stderr:218267: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T21:05:10.788 INFO:tasks.workunit.client.0.smithi136.stderr:218267: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=493921239043 2024-09-15T21:05:10.788 INFO:tasks.workunit.client.0.smithi136.stderr:218267: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.3 seq 493921239043' 2024-09-15T21:05:10.788 INFO:tasks.workunit.client.0.smithi136.stderr:218267: waiting osd.3 seq 493921239043 2024-09-15T21:05:10.788 INFO:tasks.workunit.client.0.smithi136.stderr:218267: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-15T21:05:10.788 INFO:tasks.workunit.client.0.smithi136.stderr:218267: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 493921239043 -lt 493921239043 2024-09-15T21:05:10.788 INFO:tasks.workunit.client.0.smithi136.stderr:218267: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T21:05:10.788 INFO:tasks.workunit.client.0.smithi136.stderr:218267: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 4-107374182480 2024-09-15T21:05:10.788 INFO:tasks.workunit.client.0.smithi136.stderr:218267: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T21:05:11.696 INFO:tasks.workunit.client.0.smithi136.stderr:218267: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=4 2024-09-15T21:05:11.697 INFO:tasks.workunit.client.0.smithi136.stderr:218267: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 4-107374182480 2024-09-15T21:05:11.697 INFO:tasks.workunit.client.0.smithi136.stderr:218267: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T21:05:11.697 INFO:tasks.workunit.client.0.smithi136.stderr:218267: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=107374182480 2024-09-15T21:05:11.697 INFO:tasks.workunit.client.0.smithi136.stderr:218267: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.4 seq 107374182480' 2024-09-15T21:05:11.697 INFO:tasks.workunit.client.0.smithi136.stderr:218267: waiting osd.4 seq 107374182480 2024-09-15T21:05:11.697 INFO:tasks.workunit.client.0.smithi136.stderr:218267: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 4 2024-09-15T21:05:11.697 INFO:tasks.workunit.client.0.smithi136.stderr:218267: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 107374182481 -lt 107374182480 2024-09-15T21:05:11.697 INFO:tasks.workunit.client.0.smithi136.stderr:218267: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T21:05:11.697 INFO:tasks.workunit.client.0.smithi136.stderr:218267: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 5-481036337155 2024-09-15T21:05:11.697 INFO:tasks.workunit.client.0.smithi136.stderr:218267: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T21:05:11.697 INFO:tasks.workunit.client.0.smithi136.stderr:218267: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=5 2024-09-15T21:05:11.697 INFO:tasks.workunit.client.0.smithi136.stderr:218267: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 5-481036337155 2024-09-15T21:05:11.698 INFO:tasks.workunit.client.0.smithi136.stderr:218267: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T21:05:11.698 INFO:tasks.workunit.client.0.smithi136.stderr:218267: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=481036337155 2024-09-15T21:05:11.698 INFO:tasks.workunit.client.0.smithi136.stderr:218267: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.5 seq 481036337155' 2024-09-15T21:05:11.698 INFO:tasks.workunit.client.0.smithi136.stderr:218267: waiting osd.5 seq 481036337155 2024-09-15T21:05:11.698 INFO:tasks.workunit.client.0.smithi136.stderr:218267: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 5 2024-09-15T21:05:11.698 INFO:tasks.workunit.client.0.smithi136.stderr:218267: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 481036337156 -lt 481036337155 2024-09-15T21:05:11.698 INFO:tasks.workunit.client.0.smithi136.stderr:218267: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T21:05:11.698 INFO:tasks.workunit.client.0.smithi136.stderr:218267: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 6-150323855429 2024-09-15T21:05:11.698 INFO:tasks.workunit.client.0.smithi136.stderr:218267: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T21:05:11.698 INFO:tasks.workunit.client.0.smithi136.stderr:218267: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=6 2024-09-15T21:05:11.698 INFO:tasks.workunit.client.0.smithi136.stderr:218267: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 6-150323855429 2024-09-15T21:05:11.698 INFO:tasks.workunit.client.0.smithi136.stderr:218267: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T21:05:11.699 INFO:tasks.workunit.client.0.smithi136.stderr:218267: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=150323855429 2024-09-15T21:05:11.699 INFO:tasks.workunit.client.0.smithi136.stderr:218267: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.6 seq 150323855429' 2024-09-15T21:05:11.699 INFO:tasks.workunit.client.0.smithi136.stderr:218267: waiting osd.6 seq 150323855429 2024-09-15T21:05:11.699 INFO:tasks.workunit.client.0.smithi136.stderr:218267: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 6 2024-09-15T21:05:11.699 INFO:tasks.workunit.client.0.smithi136.stderr:218267: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 150323855430 -lt 150323855429 2024-09-15T21:05:11.699 INFO:tasks.workunit.client.0.smithi136.stderr:218267: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T21:05:11.699 INFO:tasks.workunit.client.0.smithi136.stderr:218267: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 7-171798691901 2024-09-15T21:05:11.699 INFO:tasks.workunit.client.0.smithi136.stderr:218267: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T21:05:11.699 INFO:tasks.workunit.client.0.smithi136.stderr:218267: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=7 2024-09-15T21:05:11.699 INFO:tasks.workunit.client.0.smithi136.stderr:218267: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 7-171798691901 2024-09-15T21:05:11.699 INFO:tasks.workunit.client.0.smithi136.stderr:218267: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T21:05:12.263 INFO:tasks.workunit.client.0.smithi136.stderr:218267: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pient.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.8 seq 193273528376' 2024-09-15T21:05:12.263 INFO:tasks.workunit.client.0.smithi136.stderr:218269: waiting osd.8 seq 193273528376 2024-09-15T21:05:12.263 INFO:tasks.workunit.client.0.smithi136.stderr:218269: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 8 2024-09-15T21:05:12.263 INFO:tasks.workunit.client.0.smithi136.stderr:218269: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 193273528377 -lt 193273528376 2024-09-15T21:05:12.263 INFO:tasks.workunit.client.0.smithi136.stderr:218269: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T21:05:12.263 INFO:tasks.workunit.client.0.smithi136.stderr:218269: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 9-450971566087 2024-09-15T21:05:12.263 INFO:tasks.workunit.client.0.smithi136.stderr:218269: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T21:05:12.263 INFO:tasks.workunit.client.0.smithi136.stderr:218269: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=9 2024-09-15T21:05:12.264 INFO:tasks.workunit.client.0.smithi136.stderr:218269: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 9-450971566087 2024-09-15T21:05:12.264 INFO:tasks.workunit.client.0.smithi136.stderr:218269: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T21:05:12.264 INFO:tasks.workunit.client.0.smithi136.stderr:218269: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=450971566087 2024-09-15T21:05:12.264 INFO:tasks.workunit.client.0.smithi136.stderr:218269: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.9 seq 450971566087' 2024-09-15T21:05:12.264 INFO:tasks.workunit.client.0.smithi136.stderr:218269: waiting osd.9 seq 450971566087 2024-09-15T21:05:12.264 INFO:tasks.workunit.client.0.smithi136.stderr:218269: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 9 2024-09-15T21:05:12.264 INFO:tasks.workunit.client.0.smithi136.stderr:218269: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 450971566089 -lt 450971566087 2024-09-15T21:05:12.264 INFO:tasks.workunit.client.0.smithi136.stderr:218269: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-15T21:05:12.264 INFO:tasks.workunit.client.0.smithi136.stderr:218269: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T21:05:12.264 INFO:tasks.workunit.client.0.smithi136.stderr:218269: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T21:05:12.264 INFO:tasks.workunit.client.0.smithi136.stderr:218269: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 5 == 0 2024-09-15T21:05:12.264 INFO:tasks.workunit.client.0.smithi136.stderr:218269: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-15T21:05:12.265 INFO:tasks.workunit.client.0.smithi136.stderr:218269: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-15T21:05:12.265 INFO:tasks.workunit.client.0.smithi136.stderr:218269: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-15T21:05:12.265 INFO:tasks.workunit.client.0.smithi136.stderr:218269: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T21:05:12.265 INFO:tasks.workunit.client.0.smithi136.stderr:218269: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T21:05:12.265 INFO:tasks.workunit.client.0.smithi136.stderr:218269: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T21:05:12.265 INFO:tasks.workunit.client.0.smithi136.stderr:218269: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T21:05:12.265 INFO:tasks.workunit.client.0.smithi136.stderr:218269: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=5 2024-09-15T21:05:12.265 INFO:tasks.workunit.client.0.smithi136.stderr:218269: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-15T21:05:12.265 INFO:tasks.workunit.client.0.smithi136.stderr:218269: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T21:05:12.265 INFO:tasks.workunit.client.0.smithi136.stderr:218269: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T21:05:12.265 INFO:tasks.workunit.client.0.smithi136.stderr:218269: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 = 5 2024-09-15T21:05:12.265 INFO:tasks.workunit.client.0.smithi136.stderr:218269: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-15T21:05:12.266 INFO:tasks.workunit.client.0.smithi136.stderr:218269: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-15T21:05:12.266 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: run_in_background: return 0 2024-09-15T21:05:12.950 INFO:tasks.workunit.client.0.smithi136.stderr:g_stats: seq=171798691901 2024-09-15T21:05:12.951 INFO:tasks.workunit.client.0.smithi136.stderr:218267: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.7 seq 171798691901' 2024-09-15T21:05:12.951 INFO:tasks.workunit.client.0.smithi136.stderr:218267: waiting osd.7 seq 171798691901 2024-09-15T21:05:12.951 INFO:tasks.workunit.client.0.smithi136.stderr:218267: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 7 2024-09-15T21:05:12.951 INFO:tasks.workunit.client.0.smithi136.stderr:218267: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 171798691902 -lt 171798691901 2024-09-15T21:05:12.951 INFO:tasks.workunit.client.0.smithi136.stderr:218267: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T21:05:12.951 INFO:tasks.workunit.client.0.smithi136.stderr:218267: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 8-193273528377 2024-09-15T21:05:12.951 INFO:tasks.workunit.client.0.smithi136.stderr:218267: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T21:05:12.951 INFO:tasks.workunit.client.0.smithi136.stderr:218267: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=8 2024-09-15T21:05:12.951 INFO:tasks.workunit.client.0.smithi136.stderr:218267: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 8-193273528377 2024-09-15T21:05:12.951 INFO:tasks.workunit.client.0.smithi136.stderr:218267: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T21:05:12.951 INFO:tasks.workunit.client.0.smithi136.stderr:218267: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=193273528377 2024-09-15T21:05:12.952 INFO:tasks.workunit.client.0.smithi136.stderr:218267: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.8 seq 193273528377' 2024-09-15T21:05:12.952 INFO:tasks.workunit.client.0.smithi136.stderr:218267: waiting osd.8 seq 193273528377 2024-09-15T21:05:12.952 INFO:tasks.workunit.client.0.smithi136.stderr:218267: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 8 2024-09-15T21:05:12.952 INFO:tasks.workunit.client.0.smithi136.stderr:218267: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 193273528378 -lt 193273528377 2024-09-15T21:05:12.952 INFO:tasks.workunit.client.0.smithi136.stderr:218267: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T21:05:12.952 INFO:tasks.workunit.client.0.smithi136.stderr:218267: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 9-450971566089 2024-09-15T21:05:12.952 INFO:tasks.workunit.client.0.smithi136.stderr:218267: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T21:05:12.952 INFO:tasks.workunit.client.0.smithi136.stderr:218267: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=9 2024-09-15T21:05:12.952 INFO:tasks.workunit.client.0.smithi136.stderr:218267: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 9-450971566089 2024-09-15T21:05:12.952 INFO:tasks.workunit.client.0.smithi136.stderr:218267: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T21:05:12.952 INFO:tasks.workunit.client.0.smithi136.stderr:218267: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=450971566089 2024-09-15T21:05:12.952 INFO:tasks.workunit.client.0.smithi136.stderr:218267: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.9 seq 450971566089' 2024-09-15T21:05:12.953 INFO:tasks.workunit.client.0.smithi136.stderr:218267: waiting osd.9 seq 450971566089 2024-09-15T21:05:12.953 INFO:tasks.workunit.client.0.smithi136.stderr:218267: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 9 2024-09-15T21:05:12.953 INFO:tasks.workunit.client.0.smithi136.stderr:218267: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 450971566089 -lt 450971566089 2024-09-15T21:05:12.953 INFO:tasks.workunit.client.0.smithi136.stderr:218267: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-15T21:05:12.953 INFO:tasks.workunit.client.0.smithi136.stderr:218267: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T21:05:12.953 INFO:tasks.workunit.client.0.smithi136.stderr:218267: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T21:05:12.953 INFO:tasks.workunit.client.0.smithi136.stderr:218267: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 5 == 0 2024-09-15T21:05:12.953 INFO:tasks.workunit.client.0.smithi136.stderr:218267: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-15T21:05:12.953 INFO:tasks.workunit.client.0.smithi136.stderr:218267: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-15T21:05:12.953 INFO:tasks.workunit.client.0.smithi136.stderr:218267: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-15T21:05:12.953 INFO:tasks.workunit.client.0.smithi136.stderr:218267: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T21:05:12.953 INFO:tasks.workunit.client.0.smithi136.stderr:218267: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T21:05:13.567 INFO:tasks.workunit.client.0.smithi136.stderr:218267: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T21:05:13.567 INFO:tasks.workunit.client.0.smithi136.stderr:218267: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T21:05:13.567 INFO:tasks.workunit.client.0.smithi136.stderr:218267: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=5 2024-09-15T21:05:13.567 INFO:tasks.workunit.client.0.smithi136.stderr:218267: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-15T21:05:13.567 INFO:tasks.workunit.client.0.smithi136.stderr:218267: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T21:05:13.567 INFO:tasks.workunit.client.0.smithi136.stderr:218267: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T21:05:13.567 INFO:tasks.workunit.client.0.smithi136.stderr:218267: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 = 5 2024-09-15T21:05:13.568 INFO:tasks.workunit.client.0.smithi136.stderr:218267: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-15T21:05:13.568 INFO:tasks.workunit.client.0.smithi136.stderr:218267: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-15T21:05:13.568 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: run_in_background: return 0 2024-09-15T21:05:13.568 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: wait_background: for pid in $pids 2024-09-15T21:05:13.568 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2175: wait_background: wait 218266 2024-09-15T21:05:13.568 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: wait_background: eval 'pids='\'''\''' 2024-09-15T21:05:13.568 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: wait_background: pids= 2024-09-15T21:05:13.568 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: wait_background: return 0 2024-09-15T21:05:13.568 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:197: corrupt_and_repair_two: return_code=0 2024-09-15T21:05:13.568 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:198: corrupt_and_repair_two: '[' 0 -ne 0 ']' 2024-09-15T21:05:13.568 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:203: corrupt_and_repair_two: get_pg ecpool SOMETHING 2024-09-15T21:05:13.568 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1084: get_pg: local poolname=ecpool 2024-09-15T21:05:13.568 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1085: get_pg: local objectname=SOMETHING 2024-09-15T21:05:13.569 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1087: get_pg: ceph --format json osd map ecpool SOMETHING 2024-09-15T21:05:13.569 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1087: get_pg: jq -r .pgid 2024-09-15T21:05:13.845 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:203: corrupt_and_repair_two: local pg=2.0 2024-09-15T21:05:13.845 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:204: corrupt_and_repair_two: repair 2.0 2024-09-15T21:05:13.846 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1846: repair: local pgid=2.0 2024-09-15T21:05:13.846 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1847: repair: get_last_scrub_stamp 2.0 2024-09-15T21:05:13.846 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2024-09-15T21:05:13.846 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-15T21:05:13.846 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-15T21:05:13.846 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-09-15T21:05:14.120 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1847: repair: local last_scrub=2024-09-15T21:04:35.276431+0000 2024-09-15T21:05:14.121 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1848: repair: ceph pg repair 2.0 2024-09-15T21:05:14.379 INFO:tasks.workunit.client.0.smithi136.stderr:instructing pg 2.0s0 on osd.3 to repair 2024-09-15T21:05:14.391 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1849: repair: wait_for_scrub 2.0 2024-09-15T21:04:35.276431+0000 2024-09-15T21:05:14.391 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: local pgid=2.0 2024-09-15T21:05:14.392 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2018: wait_for_scrub: local last_scrub=2024-09-15T21:04:35.276431+0000 2024-09-15T21:05:14.392 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2019: wait_for_scrub: local sname=last_scrub_stamp 2024-09-15T21:05:14.392 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i=0 )) 2024-09-15T21:05:14.392 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-15T21:05:14.392 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-09-15T21:05:14.392 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2024-09-15T21:05:14.392 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-15T21:05:14.392 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-15T21:05:14.392 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-09-15T21:05:14.668 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-15T21:04:35.276431+0000 '>' 2024-09-15T21:04:35.276431+0000 2024-09-15T21:05:14.668 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-15T21:05:15.670 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-15T21:05:15.670 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-15T21:05:15.671 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-09-15T21:05:15.671 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2024-09-15T21:05:15.671 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-15T21:05:15.671 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-15T21:05:15.671 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-09-15T21:05:15.954 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-15T21:04:35.276431+0000 '>' 2024-09-15T21:04:35.276431+0000 2024-09-15T21:05:15.954 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-15T21:05:16.955 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-15T21:05:16.955 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-15T21:05:16.956 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-09-15T21:05:16.956 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2024-09-15T21:05:16.956 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-15T21:05:16.956 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-15T21:05:16.957 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-09-15T21:05:17.239 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-15T21:05:14.467863+0000 '>' 2024-09-15T21:04:35.276431+0000 2024-09-15T21:05:17.240 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2023: wait_for_scrub: return 0 2024-09-15T21:05:17.240 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1850: repair: sleep 2 2024-09-15T21:05:19.242 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:208: corrupt_and_repair_two: pids= 2024-09-15T21:05:19.242 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:209: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 3 SOMETHING list-attrs 2024-09-15T21:05:19.242 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2135: run_in_background: local pid_variable=pids 2024-09-15T21:05:19.242 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2136: run_in_background: shift 2024-09-15T21:05:19.242 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2140: run_in_background: eval 'pids+=" 221735"' 2024-09-15T21:05:19.242 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2140: run_in_background: pids+=' 221735' 2024-09-15T21:05:19.243 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:210: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 5 SOMETHING list-attrs 2024-09-15T21:05:19.243 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2135: run_in_background: local pid_variable=pids 2024-09-15T21:05:19.243 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2136: run_in_background: shift 2024-09-15T21:05:19.243 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2140: run_in_background: eval 'pids+=" 221737"' 2024-09-15T21:05:19.243 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: run_in_background: objectstore_tool td/osd-scrub-repair 3 SOMETHING list-attrs 2024-09-15T21:05:19.243 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2140: run_in_background: pids+=' 221737' 2024-09-15T21:05:19.243 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:211: corrupt_and_repair_two: wait_background pids 2024-09-15T21:05:19.243 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: wait_background: pids=' 221735 221737' 2024-09-15T21:05:19.243 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: wait_background: return_code=0 2024-09-15T21:05:19.243 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: wait_background: for pid in $pids 2024-09-15T21:05:19.243 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2175: wait_background: wait 221735 2024-09-15T21:05:19.243 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: run_in_background: sed 's/^/221738: /' 2024-09-15T21:05:19.244 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: run_in_background: objectstore_tool td/osd-scrub-repair 5 SOMETHING list-attrs 2024-09-15T21:05:19.244 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: run_in_background: sed 's/^/221740: /' 2024-09-15T21:05:20.375 INFO:tasks.workunit.client.0.smithi136.stderr:221738: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2024-09-15T21:05:20.375 INFO:tasks.workunit.client.0.smithi136.stderr:221738: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2024-09-15T21:05:20.375 INFO:tasks.workunit.client.0.smithi136.stderr:221738: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=3 2024-09-15T21:05:20.375 INFO:tasks.workunit.client.0.smithi136.stderr:221738: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2024-09-15T21:05:20.375 INFO:tasks.workunit.client.0.smithi136.stderr:221738: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 3 SOMETHING list-attrs 2024-09-15T21:05:20.375 INFO:tasks.workunit.client.0.smithi136.stderr:221738: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-09-15T21:05:20.375 INFO:tasks.workunit.client.0.smithi136.stderr:221738: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2024-09-15T21:05:20.375 INFO:tasks.workunit.client.0.smithi136.stderr:221738: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=3 2024-09-15T21:05:20.375 INFO:tasks.workunit.client.0.smithi136.stderr:221738: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2024-09-15T21:05:20.375 INFO:tasks.workunit.client.0.smithi136.stderr:221738: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.3 2024-09-15T21:05:20.376 INFO:tasks.workunit.client.0.smithi136.stderr:221738: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-15T21:05:20.376 INFO:tasks.workunit.client.0.smithi136.stderr:221738: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-15T21:05:20.376 INFO:tasks.workunit.client.0.smithi136.stderr:221738: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-15T21:05:20.376 INFO:tasks.workunit.client.0.smithi136.stderr:221738: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-15T21:05:20.376 INFO:tasks.workunit.client.0.smithi136.stderr:221738: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-15T21:05:20.376 INFO:tasks.workunit.client.0.smithi136.stderr:221738: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-15T21:05:20.376 INFO:tasks.workunit.client.0.smithi136.stderr:221738: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 3 SOMETHING list-attrs 2024-09-15T21:05:20.376 INFO:tasks.workunit.client.0.smithi136.stderr:221738: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-15T21:05:20.376 INFO:tasks.workunit.client.0.smithi136.stderr:221738: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-15T21:05:20.376 INFO:tasks.workunit.client.0.smithi136.stderr:221738: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=3 2024-09-15T21:05:20.376 INFO:tasks.workunit.client.0.smithi136.stderr:221738: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-15T21:05:20.376 INFO:tasks.workunit.client.0.smithi136.stderr:221738: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/3 2024-09-15T21:05:20.376 INFO:tasks.workunit.client.0.smithi136.stderr:221738: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/3 SOMETHING list-attrs 2024-09-15T21:05:20.376 INFO:tasks.workunit.client.0.smithi136.stderr:221738: Error getting attr on : 2.0s0_head,0#-4:00000000:::scrub_2.0s0:head#, (61) No data available 2024-09-15T21:05:20.377 INFO:tasks.workunit.client.0.smithi136.stderr:221738: _ 2024-09-15T21:05:20.377 INFO:tasks.workunit.client.0.smithi136.stderr:221738: hinfo_key 2024-09-15T21:05:20.377 INFO:tasks.workunit.client.0.smithi136.stderr:221738: snapset 2024-09-15T21:05:20.377 INFO:tasks.workunit.client.0.smithi136.stderr:221738: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 3 2024-09-15T21:05:20.377 INFO:tasks.workunit.client.0.smithi136.stderr:221738: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2024-09-15T21:05:20.377 INFO:tasks.workunit.client.0.smithi136.stderr:221738: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-15T21:05:20.377 INFO:tasks.workunit.client.0.smithi136.stderr:221738: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=3 2024-09-15T21:05:20.377 INFO:tasks.workunit.client.0.smithi136.stderr:221738: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-15T21:05:20.377 INFO:tasks.workunit.client.0.smithi136.stderr:221738: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/3 2024-09-15T21:05:20.377 INFO:tasks.workunit.client.0.smithi136.stderr:221738: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=ce4892c7-6c6d-4b4f-8e6f-07a3e11133e5 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-15T21:05:20.378 INFO:tasks.workunit.client.0.smithi136.stderr:221738: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T21:05:20.378 INFO:tasks.workunit.client.0.smithi136.stderr:221738: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T21:05:20.378 INFO:tasks.workunit.client.0.smithi136.stderr:221738: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T21:05:20.378 INFO:tasks.workunit.client.0.smithi136.stderr:221738: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/3' 2024-09-15T21:05:20.378 INFO:tasks.workunit.client.0.smithi136.stderr:221738: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/3/journal' 2024-09-15T21:05:20.378 INFO:tasks.workunit.client.0.smithi136.stderr:221738: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-15T21:05:20.378 INFO:tasks.workunit.client.0.smithi136.stderr:221738: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-15T21:05:20.378 INFO:tasks.workunit.client.0.smithi136.stderr:221738: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-15T21:05:20.378 INFO:tasks.workunit.client.0.smithi136.stderr:221738: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-15T21:05:20.378 INFO:tasks.workunit.client.0.smithi136.stderr:221738: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-15T21:05:20.379 INFO:tasks.workunit.client.0.smithi136.stderr:221738: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-15T21:05:20.379 INFO:tasks.workunit.client.0.smithi136.stderr:221738: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-15T21:05:20.379 INFO:tasks.workunit.client.0.smithi136.stderr:221738: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T21:05:20.379 INFO:tasks.workunit.client.0.smithi136.stderr:221738: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52602 2024-09-15T21:05:20.379 INFO:tasks.workunit.client.0.smithi136.stderr:221738: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.52602/$cluster-$name.asok' 2024-09-15T21:05:20.379 INFO:tasks.workunit.client.0.smithi136.stderr:221738: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.52602/$cluster-$name.asok' 2024-09-15T21:05:20.380 INFO:tasks.workunit.client.0.smithi136.stderr:221738: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-15T21:05:20.380 INFO:tasks.workunit.client.0.smithi136.stderr:221738: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-15T21:05:20.380 INFO:tasks.workunit.client.0.smithi136.stderr:221738: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-15T21:05:20.380 INFO:tasks.workunit.client.0.smithi136.stderr:221738: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T21:05:20.380 INFO:tasks.workunit.client.0.smithi136.stderr:221738: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T21:05:20.380 INFO:tasks.workunit.client.0.smithi136.stderr:221738: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T21:05:20.380 INFO:tasks.workunit.client.0.smithi136.stderr:221738: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T21:05:20.380 INFO:tasks.workunit.client.0.smithi136.stderr:221738: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-15T21:05:20.380 INFO:tasks.workunit.client.0.smithi136.stderr:221738: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-15T21:05:20.380 INFO:tasks.workunit.client.0.smithi136.stderr:221738: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/3 2024-09-15T21:05:20.380 INFO:tasks.workunit.client.0.smithi136.stderr:221738: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.3 2024-09-15T21:05:20.380 INFO:tasks.workunit.client.0.smithi136.stderr:221738: start osd.3 2024-09-15T21:05:20.400 INFO:tasks.workunit.client.0.smithi136.stderr:221738: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 3 --fsid=ce4892c7-6c6d-4b4f-8e6f-07a3e11133e5 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/3 --osd-journal=td/osd-scrub-repair/3/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.52602/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-l221740: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2024-09-15T21:05:20.400 INFO:tasks.workunit.client.0.smithi136.stderr:221740: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2024-09-15T21:05:20.400 INFO:tasks.workunit.client.0.smithi136.stderr:221740: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=5 2024-09-15T21:05:20.400 INFO:tasks.workunit.client.0.smithi136.stderr:221740: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2024-09-15T21:05:20.400 INFO:tasks.workunit.client.0.smithi136.stderr:221740: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 5 SOMETHING list-attrs 2024-09-15T21:05:20.400 INFO:tasks.workunit.client.0.smithi136.stderr:221740: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-09-15T21:05:20.400 INFO:tasks.workunit.client.0.smithi136.stderr:221740: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2024-09-15T21:05:20.400 INFO:tasks.workunit.client.0.smithi136.stderr:221740: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=5 2024-09-15T21:05:20.400 INFO:tasks.workunit.client.0.smithi136.stderr:221740: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2024-09-15T21:05:20.400 INFO:tasks.workunit.client.0.smithi136.stderr:221740: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.5 2024-09-15T21:05:20.400 INFO:tasks.workunit.client.0.smithi136.stderr:221740: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-15T21:05:20.401 INFO:tasks.workunit.client.0.smithi136.stderr:221740: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-15T21:05:20.401 INFO:tasks.workunit.client.0.smithi136.stderr:221740: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-15T21:05:20.401 INFO:tasks.workunit.client.0.smithi136.stderr:221740: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-15T21:05:20.401 INFO:tasks.workunit.client.0.smithi136.stderr:221740: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-15T21:05:20.401 INFO:tasks.workunit.client.0.smithi136.stderr:221740: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-15T21:05:20.401 INFO:tasks.workunit.client.0.smithi136.stderr:221740: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 5 SOMETHING list-attrs 2024-09-15T21:05:20.401 INFO:tasks.workunit.client.0.smithi136.stderr:221740: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-15T21:05:20.401 INFO:tasks.workunit.client.0.smithi136.stderr:221740: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-15T21:05:20.401 INFO:tasks.workunit.client.0.smithi136.stderr:221740: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=5 2024-09-15T21:05:20.401 INFO:tasks.workunit.client.0.smithi136.stderr:221740: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-15T21:05:20.401 INFO:tasks.workunit.client.0.smithi136.stderr:221740: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/5 2024-09-15T21:05:20.401 INFO:tasks.workunit.client.0.smithi136.stderr:221740: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/5 SOMETHING list-attrs 2024-09-15T21:05:20.401 INFO:tasks.workunit.client.0.smithi136.stderr:221740: _ 2024-09-15T21:05:20.401 INFO:tasks.workunit.client.0.smithi136.stderr:221740: hinfo_key 2024-09-15T21:05:20.401 INFO:tasks.workunit.client.0.smithi136.stderr:221740: snapset 2024-09-15T21:05:20.402 INFO:tasks.workunit.client.0.smithi136.stderr:221740: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 5 2024-09-15T21:05:20.402 INFO:tasks.workunit.client.0.smithi136.stderr:221740: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2024-09-15T21:05:20.402 INFO:tasks.workunit.client.0.smithi136.stderr:221740: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-15T21:05:20.402 INFO:tasks.workunit.client.0.smithi136.stderr:221740: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=5 2024-09-15T21:05:20.402 INFO:tasks.workunit.client.0.smithi136.stderr:221740: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-15T21:05:20.402 INFO:tasks.workunit.client.0.smithi136.stderr:221740: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/5 2024-09-15T21:05:20.402 INFO:tasks.workunit.client.0.smithi136.stderr:221740: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=ce4892c7-6c6d-4b4f-8e6f-07a3e11133e5 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-15T21:05:20.402 INFO:tasks.workunit.client.0.smithi136.stderr:221740: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T21:05:20.402 INFO:tasks.workunit.client.0.smithi136.stderr:221740: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T21:05:20.402 INFO:tasks.workunit.client.0.smithi136.stderr:221740: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T21:05:20.403 INFO:tasks.workunit.client.0.smithi136.stderr:221740: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/5' 2024-09-15T21:05:20.403 INFO:tasks.workunit.client.0.smithi136.stderr:221740: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/5/journal' 2024-09-15T21:05:20.403 INFO:tasks.workunit.client.0.smithi136.stderr:221740: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-15T21:05:20.403 INFO:tasks.workunit.client.0.smithi136.stderr:221740: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-15T21:05:20.403 INFO:tasks.workunit.client.0.smithi136.stderr:221740: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-15T21:05:20.403 INFO:tasks.workunit.client.0.smithi136.stderr:221740: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-15T21:05:20.403 INFO:tasks.workunit.client.0.smithi136.stderr:221740: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-15T21:05:20.403 INFO:tasks.workunit.client.0.smithi136.stderr:221740: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-15T21:05:20.403 INFO:tasks.workunit.client.0.smithi136.stderr:221740: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-15T21:05:20.403 INFO:tasks.workunit.client.0.smithi136.stderr:221740: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T21:05:20.403 INFO:tasks.workunit.client.0.smithi136.stderr:221740: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52602 2024-09-15T21:05:20.403 INFO:tasks.workunit.client.0.smithi136.stderr:221740: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.52602/$cluster-$name.asok' 2024-09-15T21:05:20.403 INFO:tasks.workunit.client.0.smithi136.stderr:221740: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.52602/$cluster-$name.asok' 2024-09-15T21:05:20.403 INFO:tasks.workunit.client.0.smithi136.stderr:221740: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-15T21:05:20.404 INFO:tasks.workunit.client.0.smithi136.stderr:221740: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-15T21:05:20.404 INFO:tasks.workunit.client.0.smithi136.stderr:221740: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-15T21:05:20.404 INFO:tasks.workunit.client.0.smithi136.stderr:221740: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T21:05:20.404 INFO:tasks.workunit.client.0.smithi136.stderr:221740: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T21:05:20.404 INFO:tasks.workunit.client.0.smithi136.stderr:221740: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T21:05:20.404 INFO:tasks.workunit.client.0.smithi136.stderr:221740: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T21:05:20.404 INFO:tasks.workunit.client.0.smithi136.stderr:221740: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-15T21:05:20.404 INFO:tasks.workunit.client.0.smithi136.stderr:221740: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-15T21:05:20.404 INFO:tasks.workunit.client.0.smithi136.stderr:221740: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/5 2024-09-15T21:05:20.404 INFO:tasks.workunit.client.0.smithi136.stderr:221740: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.5 2024-09-15T21:05:20.404 INFO:tasks.workunit.client.0.smithi136.stderr:221740: start osd.5 2024-09-15T21:05:25.244 INFO:tasks.workunit.client.0.smithi136.stderr:221740: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 5 --fsid=ce4892c7-6c6d-4b4f-8e6f-07a3e11133e5 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/5 --osd-journal=td/osd-scrub-repair/5/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.52602/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recoen=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T21:05:25.244 INFO:tasks.workunit.client.0.smithi136.stderr:221738: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/3/whoami 2024-09-15T21:05:25.244 INFO:tasks.workunit.client.0.smithi136.stderr:221738: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 3 = 3 ']' 2024-09-15T21:05:25.244 INFO:tasks.workunit.client.0.smithi136.stderr:221738: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-15T21:05:25.244 INFO:tasks.workunit.client.0.smithi136.stderr:221738: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-15T21:05:25.244 INFO:tasks.workunit.client.0.smithi136.stderr:221738: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-15T21:05:25.244 INFO:tasks.workunit.client.0.smithi136.stderr:221738: 2024-09-15T21:05:20.407+0000 7ff7f4ce8640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:05:25.244 INFO:tasks.workunit.client.0.smithi136.stderr:221738: 2024-09-15T21:05:20.421+0000 7ff7f4ce8640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:05:25.245 INFO:tasks.workunit.client.0.smithi136.stderr:221738: 2024-09-15T21:05:20.422+0000 7ff7f4ce8640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:05:25.245 INFO:tasks.workunit.client.0.smithi136.stderr:221738: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 3 2024-09-15T21:05:25.245 INFO:tasks.workunit.client.0.smithi136.stderr:221738: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-15T21:05:25.245 INFO:tasks.workunit.client.0.smithi136.stderr:221738: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=3 2024-09-15T21:05:25.245 INFO:tasks.workunit.client.0.smithi136.stderr:221738: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-15T21:05:25.245 INFO:tasks.workunit.client.0.smithi136.stderr:221738: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-15T21:05:25.245 INFO:tasks.workunit.client.0.smithi136.stderr:221738: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:05:25.245 INFO:tasks.workunit.client.0.smithi136.stderr:221738: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-15T21:05:25.245 INFO:tasks.workunit.client.0.smithi136.stderr:221738: 0 2024-09-15T21:05:25.245 INFO:tasks.workunit.client.0.smithi136.stderr:221738: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:05:25.245 INFO:tasks.workunit.client.0.smithi136.stderr:221738: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-15T21:05:25.245 INFO:tasks.workunit.client.0.smithi136.stderr:221738: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:05:25.245 INFO:tasks.workunit.client.0.smithi136.stderr:221738: 2024-09-15T21:05:21.791+0000 7ff7f4ce8640 -1 Falling back to public interface 2024-09-15T21:05:25.246 INFO:tasks.workunit.client.0.smithi136.stderr:221738: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:05:25.246 INFO:tasks.workunit.client.0.smithi136.stderr:221738: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:05:25.246 INFO:tasks.workunit.client.0.smithi136.stderr:221738: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-15T21:05:25.246 INFO:tasks.workunit.client.0.smithi136.stderr:221738: 1 2024-09-15T21:05:25.246 INFO:tasks.workunit.client.0.smithi136.stderr:221738: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:05:25.246 INFO:tasks.workunit.client.0.smithi136.stderr:221738: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-15T21:05:25.246 INFO:tasks.workunit.client.0.smithi136.stderr:221738: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:05:25.246 INFO:tasks.workunit.client.0.smithi136.stderr:221738: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:05:25.246 INFO:tasks.workunit.client.0.smithi136.stderr:221738: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:05:25.246 INFO:tasks.workunit.client.0.smithi136.stderr:221738: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-15T21:05:25.246 INFO:tasks.workunit.client.0.smithi136.stderr:221738: 2 2024-09-15T21:05:25.246 INFO:tasks.workunit.client.0.smithi136.stderr:221738: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:05:25.246 INFO:tasks.workunit.client.0.smithi136.stderr:221738: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-15T21:05:25.246 INFO:tasks.workunit.client.0.smithi136.stderr:221738: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:05:25.246 INFO:tasks.workunit.client.0.smithi136.stderr:221738: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:05:25.247 INFO:tasks.workunit.client.0.smithi136.stderr:221738: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:05:25.247 INFO:tasks.workunit.client.0.smithi136.stderr:221738: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-15T21:05:25.247 INFO:tasks.workunit.client.0.smithi136.stderr:221738: 3 2024-09-15T21:05:25.247 INFO:tasks.workunit.client.0.smithi136.stderr:221738: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:05:25.247 INFO:tasks.workunit.client.0.smithi136.stderr:221738: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-15T21:05:25.559 INFO:tasks.workunit.client.0.smithi136.stderr:221738: 2024-09-15T21:05:25.241+0000 7ff7f4very_ops 2024-09-15T21:05:25.559 INFO:tasks.workunit.client.0.smithi136.stderr:221740: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/5/whoami 2024-09-15T21:05:25.559 INFO:tasks.workunit.client.0.smithi136.stderr:221740: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 5 = 5 ']' 2024-09-15T21:05:25.559 INFO:tasks.workunit.client.0.smithi136.stderr:221740: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-15T21:05:25.559 INFO:tasks.workunit.client.0.smithi136.stderr:221740: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-15T21:05:25.559 INFO:tasks.workunit.client.0.smithi136.stderr:221740: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-15T21:05:25.559 INFO:tasks.workunit.client.0.smithi136.stderr:221740: 2024-09-15T21:05:20.430+0000 7f6a0d414640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:05:25.560 INFO:tasks.workunit.client.0.smithi136.stderr:221740: 2024-09-15T21:05:20.433+0000 7f6a0d414640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:05:25.560 INFO:tasks.workunit.client.0.smithi136.stderr:221740: 2024-09-15T21:05:20.434+0000 7f6a0d414640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:05:25.560 INFO:tasks.workunit.client.0.smithi136.stderr:221740: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 5 2024-09-15T21:05:25.560 INFO:tasks.workunit.client.0.smithi136.stderr:221740: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-15T21:05:25.560 INFO:tasks.workunit.client.0.smithi136.stderr:221740: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=5 2024-09-15T21:05:25.560 INFO:tasks.workunit.client.0.smithi136.stderr:221740: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-15T21:05:25.560 INFO:tasks.workunit.client.0.smithi136.stderr:221740: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-15T21:05:25.560 INFO:tasks.workunit.client.0.smithi136.stderr:221740: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:05:25.560 INFO:tasks.workunit.client.0.smithi136.stderr:221740: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-15T21:05:25.560 INFO:tasks.workunit.client.0.smithi136.stderr:221740: 0 2024-09-15T21:05:25.560 INFO:tasks.workunit.client.0.smithi136.stderr:221740: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:05:25.560 INFO:tasks.workunit.client.0.smithi136.stderr:221740: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-15T21:05:25.561 INFO:tasks.workunit.client.0.smithi136.stderr:221740: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:05:25.561 INFO:tasks.workunit.client.0.smithi136.stderr:221740: 2024-09-15T21:05:21.799+0000 7f6a0d414640 -1 Falling back to public interface 2024-09-15T21:05:25.561 INFO:tasks.workunit.client.0.smithi136.stderr:221740: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:05:25.561 INFO:tasks.workunit.client.0.smithi136.stderr:221740: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:05:25.561 INFO:tasks.workunit.client.0.smithi136.stderr:221740: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-15T21:05:25.561 INFO:tasks.workunit.client.0.smithi136.stderr:221740: 1 2024-09-15T21:05:25.561 INFO:tasks.workunit.client.0.smithi136.stderr:221740: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:05:25.561 INFO:tasks.workunit.client.0.smithi136.stderr:221740: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-15T21:05:25.561 INFO:tasks.workunit.client.0.smithi136.stderr:221740: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:05:25.561 INFO:tasks.workunit.client.0.smithi136.stderr:221740: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:05:25.562 INFO:tasks.workunit.client.0.smithi136.stderr:221740: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:05:25.562 INFO:tasks.workunit.client.0.smithi136.stderr:221740: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-15T21:05:25.562 INFO:tasks.workunit.client.0.smithi136.stderr:221740: 2 2024-09-15T21:05:25.562 INFO:tasks.workunit.client.0.smithi136.stderr:221740: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:05:25.562 INFO:tasks.workunit.client.0.smithi136.stderr:221740: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-15T21:05:25.562 INFO:tasks.workunit.client.0.smithi136.stderr:221740: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:05:25.562 INFO:tasks.workunit.client.0.smithi136.stderr:221740: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:05:25.562 INFO:tasks.workunit.client.0.smithi136.stderr:221740: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:05:25.562 INFO:tasks.workunit.client.0.smithi136.stderr:221740: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-15T21:05:25.562 INFO:tasks.workunit.client.0.smithi136.stderr:221740: 3 2024-09-15T21:05:25.562 INFO:tasks.workunit.client.0.smithi136.stderr:221740: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:05:25.563 INFO:tasks.workunit.client.0.smithi136.stderr:221740: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-15T21:05:25.563 INFO:tasks.workunit.client.0.smithi136.stderr:221740: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:05:27.360 INFO:tasks.workunit.client.0.smithi136.stderr:221740: 2024-09-15T21:05:25.556+0000 7f6a0d414640 -1 osd.5 116 log_to_monitors true 2024-09-15T21:05:27.360 INFO:tasks.workunit.client.0.smithi136.stderr:221740: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:05:27.360 INFO:tasks.workunit.client.0.smithi136.stderr:221740: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:05:27.361 INFO:tasks.workunit.client.0.smithi136.stderr:221740: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-15T21:05:27.361 INFO:tasks.workunit.client.0.smithi136.stderr:221740: 4 2024-09-15T21:05:27.361 INFO:tasks.workunit.client.0.smithi136.stderr:221740: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:05:27.361 INFO:tasks.workunit.client.0.smithi136.stderr:221740: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-15T21:05:27.361 INFO:tasks.workunit.client.0.smithi136.stderr:221740: osd.5 up in weight 1 up_from 122 up_thru 112 down_at 117 last_clean_interval [112,116) [v2:127.0.0.1:6874/3673033263,v1:127.0.0.1:6875/3673033263] [v2:127.0.0.1:6876/3673033263,v1:127.0.0.1:6877/3673033263] exists,up c3187161-e486-4b18-8257-d69c125e9792 2024-09-15T21:05:27.361 INFO:tasks.workunit.client.0.smithi136.stderr:221740: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-15T21:05:27.361 INFO:tasks.workunit.client.0.smithi136.stderr:221740: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-15T21:05:27.361 INFO:tasks.workunit.client.0.smithi136.stderr:221740: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-15T21:05:27.361 INFO:tasks.workunit.client.0.smithi136.stderr:221740: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2024-09-15T21:05:27.361 INFO:tasks.workunit.client.0.smithi136.stderr:221740: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-15T21:05:27.361 INFO:tasks.workunit.client.0.smithi136.stderr:221740: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-15T21:05:27.361 INFO:tasks.workunit.client.0.smithi136.stderr:221740: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-15T21:05:27.362 INFO:tasks.workunit.client.0.smithi136.stderr:221740: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-15T21:05:27.362 INFO:tasks.workunit.client.0.smithi136.stderr:221740: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-15T21:05:27.362 INFO:tasks.workunit.client.0.smithi136.stderr:221740: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-15T21:05:27.362 INFO:tasks.workunit.client.0.smithi136.stderr:221740: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-15T21:05:27.362 INFO:tasks.workunit.client.0.smithi136.stderr:221740: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-15T21:05:27.362 INFO:tasks.workunit.client.0.smithi136.stderr:221740: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-15T21:05:27.362 INFO:tasks.workunit.client.0.smithi136.stderr:221740: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-15T21:05:27.362 INFO:tasks.workunit.client.0.smithi136.stderr:221740: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-15T21:05:27.362 INFO:tasks.workunit.client.0.smithi136.stderr:221740: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-15T21:05:27.362 INFO:tasks.workunit.client.0.smithi136.stderr:221740: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-15T21:05:27.362 INFO:tasks.workunit.client.0.smithi136.stderr:221740: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-15T21:05:27.362 INFO:tasks.workunit.client.0.smithi136.stderr:221740: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-15T21:05:27.362 INFO:tasks.workunit.client.0.smithi136.stderr:221740: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-15T21:05:27.362 INFO:tasks.workunit.client.0.smithi136.stderr:221740: 1 2024-09-15T21:05:27.362 INFO:tasks.workunit.client.0.smithi136.stderr:221740: 2 2024-09-15T21:05:27.363 INFO:tasks.workunit.client.0.smithi136.stderr:221740: 3 2024-09-15T21:05:27.363 INFO:tasks.workunit.client.0.smithi136.stderr:221740: 4 2024-09-15T21:05:27.363 INFO:tasks.workunit.client.0.smithi136.stderr:221740: 5 2024-09-15T21:05:27.363 INFO:tasks.workunit.client.0.smithi136.stderr:221740: 6 2024-09-15T21:05:27.363 INFO:tasks.workunit.client.0.smithi136.stderr:221740: 7 2024-09-15T21:05:27.363 INFO:tasks.workunit.client.0.smithi136.stderr:221740: 8 2024-09-15T21:05:27.363 INFO:tasks.workunit.client.0.smithi136.stderr:221740: 9' 2024-09-15T21:05:27.363 INFO:tasks.workunit.client.0.smithi136.stderr:221740: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-15T21:05:27.363 INFO:tasks.workunit.client.0.smithi136.stderr:221740: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T21:05:27.363 INFO:tasks.workunit.client.0.smithi136.stderr:221740: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-15T21:05:27.363 INFO:tasks.workunit.client.0.smithi136.stderr:221740: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836596 2024-09-15T21:05:27.363 INFO:tasks.workunit.client.0.smithi136.stderr:221740: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836596 2024-09-15T21:05:27.363 INFO:tasks.workunit.client.0.smithi136.stderr:221740: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836596' 2024-09-15T21:05:27.364 INFO:tasks.workunit.client.0.smithi136.stderr:221740: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd ice8640 -1 osd.3 116 log_to_monitors true 2024-09-15T21:05:27.364 INFO:tasks.workunit.client.0.smithi136.stderr:221738: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:05:27.364 INFO:tasks.workunit.client.0.smithi136.stderr:221738: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:05:27.364 INFO:tasks.workunit.client.0.smithi136.stderr:221738: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:05:27.364 INFO:tasks.workunit.client.0.smithi136.stderr:221738: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-15T21:05:27.364 INFO:tasks.workunit.client.0.smithi136.stderr:221738: 4 2024-09-15T21:05:27.364 INFO:tasks.workunit.client.0.smithi136.stderr:221738: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:05:27.364 INFO:tasks.workunit.client.0.smithi136.stderr:221738: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-15T21:05:27.364 INFO:tasks.workunit.client.0.smithi136.stderr:221738: osd.3 up in weight 1 up_from 122 up_thru 115 down_at 117 last_clean_interval [115,116) [v2:127.0.0.1:6826/3235292973,v1:127.0.0.1:6827/3235292973] [v2:127.0.0.1:6828/3235292973,v1:127.0.0.1:6829/3235292973] exists,up 0eed652b-57cc-457d-986e-cdd453c584f5 2024-09-15T21:05:27.365 INFO:tasks.workunit.client.0.smithi136.stderr:221738: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-15T21:05:27.365 INFO:tasks.workunit.client.0.smithi136.stderr:221738: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-15T21:05:27.365 INFO:tasks.workunit.client.0.smithi136.stderr:221738: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-15T21:05:27.365 INFO:tasks.workunit.client.0.smithi136.stderr:221738: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2024-09-15T21:05:27.365 INFO:tasks.workunit.client.0.smithi136.stderr:221738: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-15T21:05:27.365 INFO:tasks.workunit.client.0.smithi136.stderr:221738: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-15T21:05:27.365 INFO:tasks.workunit.client.0.smithi136.stderr:221738: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-15T21:05:27.365 INFO:tasks.workunit.client.0.smithi136.stderr:221738: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-15T21:05:27.365 INFO:tasks.workunit.client.0.smithi136.stderr:221738: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-15T21:05:27.365 INFO:tasks.workunit.client.0.smithi136.stderr:221738: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-15T21:05:27.365 INFO:tasks.workunit.client.0.smithi136.stderr:221738: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-15T21:05:27.365 INFO:tasks.workunit.client.0.smithi136.stderr:221738: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-15T21:05:27.365 INFO:tasks.workunit.client.0.smithi136.stderr:221738: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-15T21:05:27.365 INFO:tasks.workunit.client.0.smithi136.stderr:221738: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-15T21:05:27.366 INFO:tasks.workunit.client.0.smithi136.stderr:221738: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-15T21:05:27.366 INFO:tasks.workunit.client.0.smithi136.stderr:221738: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-15T21:05:27.366 INFO:tasks.workunit.client.0.smithi136.stderr:221738: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-15T21:05:27.366 INFO:tasks.workunit.client.0.smithi136.stderr:221738: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-15T21:05:27.366 INFO:tasks.workunit.client.0.smithi136.stderr:221738: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-15T21:05:27.366 INFO:tasks.workunit.client.0.smithi136.stderr:221738: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-15T21:05:27.366 INFO:tasks.workunit.client.0.smithi136.stderr:221738: 1 2024-09-15T21:05:27.366 INFO:tasks.workunit.client.0.smithi136.stderr:221738: 2 2024-09-15T21:05:27.366 INFO:tasks.workunit.client.0.smithi136.stderr:221738: 3 2024-09-15T21:05:27.366 INFO:tasks.workunit.client.0.smithi136.stderr:221738: 4 2024-09-15T21:05:27.366 INFO:tasks.workunit.client.0.smithi136.stderr:221738: 5 2024-09-15T21:05:27.366 INFO:tasks.workunit.client.0.smithi136.stderr:221738: 6 2024-09-15T21:05:27.366 INFO:tasks.workunit.client.0.smithi136.stderr:221738: 7 2024-09-15T21:05:27.366 INFO:tasks.workunit.client.0.smithi136.stderr:221738: 8 2024-09-15T21:05:27.367 INFO:tasks.workunit.client.0.smithi136.stderr:221738: 9' 2024-09-15T21:05:27.367 INFO:tasks.workunit.client.0.smithi136.stderr:221738: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-15T21:05:27.367 INFO:tasks.workunit.client.0.smithi136.stderr:221738: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T21:05:27.367 INFO:tasks.workunit.client.0.smithi136.stderr:221738: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-15T21:05:27.367 INFO:tasks.workunit.client.0.smithi136.stderr:221738: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836597 2024-09-15T21:05:27.367 INFO:tasks.workunit.client.0.smithi136.stderr:221738: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836597 2024-09-15T21:05:27.367 INFO:tasks.workunit.client.0.smithi136.stderr:221738: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836597' 2024-09-15T21:05:28.548 INFO:tasks.workunit.client.0.smithi136.stderr:221738: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T21:05:28.549 INFO:tasks.workunit.client.0.smithi136.stderr:221738: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-15T21:05:28.549 INFO:tasks.workunit.client.0.smithi136.stderr:221738: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=42949673066 2024-09-15T21:05:28.549 INFO:tasks.workunit.client.0.smithi136.stderr:221738: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 42949673066 2024-09-15T21:05:28.549 INFO:tasks.workunit.client.0.smithi136.stderr:221738: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836597 1-42949673066' 2024-09-15T21:05:28.549 INFO:tasks.workunit.client.0.smithi136.stderr:221738: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T21:05:28.549 INFO:tasks.workunit.client.0.smithi136.stderr:221738: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-15T21:05:28.549 INFO:tasks.workunit.client.0.smithi136.stderr:221738: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=64424509542 2024-09-15T21:05:28.549 INFO:tasks.workunit.client.0.smithi136.stderr:221738: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 64424509542 2024-09-15T21:05:28.549 INFO:tasks.workunit.client.0.smithi136.stderr:221738: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836597 1-42949673066 2-64424509542' 2024-09-15T21:05:28.549 INFO:tasks.workunit.client.0.smithi136.stderr:221738: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T21:05:28.549 INFO:tasks.workunit.client.0.smithi136.stderr:221738: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-15T21:05:28.550 INFO:tasks.workunit.client.0.smithi136.stderr:221738: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=523986010115 2024-09-15T21:05:28.550 INFO:tasks.workunit.client.0.smithi136.stderr:221738: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 523986010115 2024-09-15T21:05:28.550 INFO:tasks.workunit.client.0.smithi136.stderr:221738: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836597 1-42949673066 2-64424509542 3-523986010115' 2024-09-15T21:05:28.550 INFO:tasks.workunit.client.0.smithi136.stderr:221738: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T21:05:28.550 INFO:tasks.workunit.client.0.smithi136.stderr:221738: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-09-15T21:05:28.550 INFO:tasks.workunit.client.0.smithi136.stderr:221738: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=107374182486 2024-09-15T21:05:28.550 INFO:tasks.workunit.client.0.smithi136.stderr:221738: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 107374182486 2024-09-15T21:05:28.550 INFO:tasks.workunit.client.0.smithi136.stderr:221738: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836597 1-42949673066 2-64424509542 3-523986010115 4-107374182486' 2024-09-15T21:05:28.550 INFO:tasks.workunit.client.0.smithi136.stderr:221738: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T21:05:28.550 INFO:tasks.workunit.client.0.smithi136.stderr:221738: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-09-15T21:05:28.550 INFO:tasks.workunit.client.0.smithi136.stderr:221738: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=523986010114 2024-09-15T21:05:28.550 INFO:tasks.workunit.client.0.smithi136.stderr:221738: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 523986010114 2024-09-15T21:05:28.550 INFO:tasks.workunit.client.0.smithi136.stderr:221738: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836597 1-42949673066 2-64424509542 3-523986010115 4-107374182486 5-523986010114' 2024-09-15T21:05:28.550 INFO:tasks.workunit.client.0.smithi136.stderr:221738: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T21:05:28.551 INFO:tasks.workunit.client.0.smithi136.stderr:221738: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-09-15T21:05:28.551 INFO:tasks.workunit.client.0.smithi136.stderr:221738: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=150323855435 2024-09-15T21:05:28.551 INFO:tasks.workunit.client.0.smithi136.stderr:221738: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 150323855435 2024-09-15T21:05:28.551 INFO:tasks.workunit.client.0.smithi136.stderr:221738: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836597 1-42949673066 2-64424509542 3-523986010115 4-107374182486 5-523986010114 6-150323855435' 2024-09-15T21:05:28.551 INFO:tasks.workunit.client.0.smithi136.stderr:221738: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T21:05:28.660 INFO:tasks.workunit.client.0.smithi136.stderr:221738: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: cn $ids 2024-09-15T21:05:28.661 INFO:tasks.workunit.client.0.smithi136.stderr:221740: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-15T21:05:28.661 INFO:tasks.workunit.client.0.smithi136.stderr:221740: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=42949673067 2024-09-15T21:05:28.661 INFO:tasks.workunit.client.0.smithi136.stderr:221740: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 42949673067 2024-09-15T21:05:28.661 INFO:tasks.workunit.client.0.smithi136.stderr:221740: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836596 1-42949673067' 2024-09-15T21:05:28.661 INFO:tasks.workunit.client.0.smithi136.stderr:221740: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T21:05:28.661 INFO:tasks.workunit.client.0.smithi136.stderr:221740: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-15T21:05:28.661 INFO:tasks.workunit.client.0.smithi136.stderr:221740: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=64424509541 2024-09-15T21:05:28.661 INFO:tasks.workunit.client.0.smithi136.stderr:221740: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 64424509541 2024-09-15T21:05:28.661 INFO:tasks.workunit.client.0.smithi136.stderr:221740: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836596 1-42949673067 2-64424509541' 2024-09-15T21:05:28.661 INFO:tasks.workunit.client.0.smithi136.stderr:221740: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T21:05:28.661 INFO:tasks.workunit.client.0.smithi136.stderr:221740: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-15T21:05:28.661 INFO:tasks.workunit.client.0.smithi136.stderr:221740: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=523986010114 2024-09-15T21:05:28.662 INFO:tasks.workunit.client.0.smithi136.stderr:221740: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 523986010114 2024-09-15T21:05:28.662 INFO:tasks.workunit.client.0.smithi136.stderr:221740: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836596 1-42949673067 2-64424509541 3-523986010114' 2024-09-15T21:05:28.662 INFO:tasks.workunit.client.0.smithi136.stderr:221740: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T21:05:28.662 INFO:tasks.workunit.client.0.smithi136.stderr:221740: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-09-15T21:05:28.662 INFO:tasks.workunit.client.0.smithi136.stderr:221740: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=107374182487 2024-09-15T21:05:28.662 INFO:tasks.workunit.client.0.smithi136.stderr:221740: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 107374182487 2024-09-15T21:05:28.662 INFO:tasks.workunit.client.0.smithi136.stderr:221740: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836596 1-42949673067 2-64424509541 3-523986010114 4-107374182487' 2024-09-15T21:05:28.662 INFO:tasks.workunit.client.0.smithi136.stderr:221740: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T21:05:28.662 INFO:tasks.workunit.client.0.smithi136.stderr:221740: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-09-15T21:05:28.662 INFO:tasks.workunit.client.0.smithi136.stderr:221740: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=523986010115 2024-09-15T21:05:28.662 INFO:tasks.workunit.client.0.smithi136.stderr:221740: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 523986010115 2024-09-15T21:05:28.662 INFO:tasks.workunit.client.0.smithi136.stderr:221740: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836596 1-42949673067 2-64424509541 3-523986010114 4-107374182487 5-523986010115' 2024-09-15T21:05:28.662 INFO:tasks.workunit.client.0.smithi136.stderr:221740: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T21:05:28.663 INFO:tasks.workunit.client.0.smithi136.stderr:221740: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-09-15T21:05:28.663 INFO:tasks.workunit.client.0.smithi136.stderr:221740: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=150323855434 2024-09-15T21:05:28.663 INFO:tasks.workunit.client.0.smithi136.stderr:221740: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 150323855434 2024-09-15T21:05:28.663 INFO:tasks.workunit.client.0.smithi136.stderr:221740: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836596 1-42949673067 2-64424509541 3-523986010114 4-107374182487 5-523986010115 6-150323855434' 2024-09-15T21:05:28.663 INFO:tasks.workunit.client.0.smithi136.stderr:221740: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T21:05:28.663 INFO:tasks.workunit.client.0.smithi136.stderr:221740: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2024-09-15T21:05:29.216 INFO:tasks.workunit.client.0.smithi136.stderr:221740: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=171798691906 2024-09-15T21:05:29.217 INFO:tasks.workunit.client.0.smithi136.stderr:221740: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 171798691906 2024-09-15T21:05:29.217 INFO:tasks.workunit.client.0.smithi136.stderr:221740: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836596 1-42949673067 2-64424509541 3-523986010114 4-107374182487 5-523986010115 6-150323855434 7-171798691906' 2024-09-15T21:05:29.217 INFO:tasks.workunit.client.0.smithi136.stderr:221740: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T21:05:29.217 INFO:tasks.workunit.client.0.smithi136.stderr:221740: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2024-09-15T21:05:29.217 INFO:tasks.workunit.client.0.smithi136.stderr:221740: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=193273528383 2024-09-15T21:05:29.217 INFO:tasks.workunit.client.0.smithi136.stderr:221740: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 193273528383 2024-09-15T21:05:29.217 INFO:tasks.workunit.client.0.smithi136.stderr:221740: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836596 1-42949673067 2-64424509541 3-523986010114 4-107374182487 5-523986010115 6-150323855434 7-171798691906 8-193273528383' 2024-09-15T21:05:29.217 INFO:tasks.workunit.client.0.smithi136.stderr:221740: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T21:05:29.217 INFO:tasks.workunit.client.0.smithi136.stderr:221740: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2024-09-15T21:05:29.217 INFO:tasks.workunit.client.0.smithi136.stderr:221740: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=450971566095 2024-09-15T21:05:29.217 INFO:tasks.workunit.client.0.smithi136.stderr:221740: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 450971566095 2024-09-15T21:05:29.217 INFO:tasks.workunit.client.0.smithi136.stderr:221740: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836596 1-42949673067 2-64424509541 3-523986010114 4-107374182487 5-523986010115 6-150323855434 7-171798691906 8-193273528383 9-450971566095' 2024-09-15T21:05:29.217 INFO:tasks.workunit.client.0.smithi136.stderr:221740: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T21:05:29.218 INFO:tasks.workunit.client.0.smithi136.stderr:221740: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836596 2024-09-15T21:05:29.218 INFO:tasks.workunit.client.0.smithi136.stderr:221740: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T21:05:29.218 INFO:tasks.workunit.client.0.smithi136.stderr:221740: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-15T21:05:29.218 INFO:tasks.workunit.client.0.smithi136.stderr:221740: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836596 2024-09-15T21:05:29.218 INFO:tasks.workunit.client.0.smithi136.stderr:221740: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T21:05:29.218 INFO:tasks.workunit.client.0.smithi136.stderr:221740: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836596 2024-09-15T21:05:29.218 INFO:tasks.workunit.client.0.smithi136.stderr:221740: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836596' 2024-09-15T21:05:29.218 INFO:tasks.workunit.client.0.smithi136.stderr:221740: waiting osd.0 seq 21474836596 2024-09-15T21:05:29.218 INFO:tasks.workunit.client.0.smithi136.stderr:221740: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T21:05:29.218 INFO:tasks.workunit.client.0.smithi136.stderr:221740: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836597 -lt 21474836596 2024-09-15T21:05:29.218 INFO:tasks.workunit.client.0.smithi136.stderr:221740: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T21:05:29.218 INFO:tasks.workunit.client.0.smithi136.stderr:221740: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-42949673067 2024-09-15T21:05:29.218 INFO:tasks.workunit.client.0.smithi136.stderr:221740: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T21:05:29.218 INFO:tasks.workunit.client.0.smithi136.stderr:221740: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-15T21:05:29.219 INFO:tasks.workunit.client.0.smithi136.stderr:221740: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-42949673067 2024-09-15T21:05:29.219 INFO:tasks.workunit.client.0.smithi136.stderr:221740: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T21:05:29.219 INFO:tasks.workunit.client.0.smithi136.stderr:221740: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=42949673067 2024-09-15T21:05:29.219 INFO:tasks.workunit.client.0.smithi136.stderr:221740: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 42949673067' 2024-09-15T21:05:29.219 INFO:tasks.workunit.client.0.smithi136.stderr:221740: waiting osd.1 seq 42949673067 2024-09-15T21:05:29.221 INFO:tasks.workunit.client.0.smithi136.stderr:221740: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:eph tell osd.7 flush_pg_stats 2024-09-15T21:05:29.221 INFO:tasks.workunit.client.0.smithi136.stderr:221738: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=171798691907 2024-09-15T21:05:29.221 INFO:tasks.workunit.client.0.smithi136.stderr:221738: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 171798691907 2024-09-15T21:05:29.221 INFO:tasks.workunit.client.0.smithi136.stderr:221738: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836597 1-42949673066 2-64424509542 3-523986010115 4-107374182486 5-523986010114 6-150323855435 7-171798691907' 2024-09-15T21:05:29.221 INFO:tasks.workunit.client.0.smithi136.stderr:221738: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T21:05:29.221 INFO:tasks.workunit.client.0.smithi136.stderr:221738: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2024-09-15T21:05:29.221 INFO:tasks.workunit.client.0.smithi136.stderr:221738: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=193273528382 2024-09-15T21:05:29.222 INFO:tasks.workunit.client.0.smithi136.stderr:221738: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 193273528382 2024-09-15T21:05:29.222 INFO:tasks.workunit.client.0.smithi136.stderr:221738: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836597 1-42949673066 2-64424509542 3-523986010115 4-107374182486 5-523986010114 6-150323855435 7-171798691907 8-193273528382' 2024-09-15T21:05:29.222 INFO:tasks.workunit.client.0.smithi136.stderr:221738: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T21:05:29.222 INFO:tasks.workunit.client.0.smithi136.stderr:221738: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2024-09-15T21:05:29.222 INFO:tasks.workunit.client.0.smithi136.stderr:221738: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=450971566094 2024-09-15T21:05:29.222 INFO:tasks.workunit.client.0.smithi136.stderr:221738: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 450971566094 2024-09-15T21:05:29.222 INFO:tasks.workunit.client.0.smithi136.stderr:221738: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836597 1-42949673066 2-64424509542 3-523986010115 4-107374182486 5-523986010114 6-150323855435 7-171798691907 8-193273528382 9-450971566094' 2024-09-15T21:05:29.222 INFO:tasks.workunit.client.0.smithi136.stderr:221738: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T21:05:29.222 INFO:tasks.workunit.client.0.smithi136.stderr:221738: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836597 2024-09-15T21:05:29.222 INFO:tasks.workunit.client.0.smithi136.stderr:221738: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T21:05:29.222 INFO:tasks.workunit.client.0.smithi136.stderr:221738: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-15T21:05:29.222 INFO:tasks.workunit.client.0.smithi136.stderr:221738: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836597 2024-09-15T21:05:29.222 INFO:tasks.workunit.client.0.smithi136.stderr:221738: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T21:05:29.223 INFO:tasks.workunit.client.0.smithi136.stderr:221738: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836597 2024-09-15T21:05:29.223 INFO:tasks.workunit.client.0.smithi136.stderr:221738: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836597' 2024-09-15T21:05:29.223 INFO:tasks.workunit.client.0.smithi136.stderr:221738: waiting osd.0 seq 21474836597 2024-09-15T21:05:29.223 INFO:tasks.workunit.client.0.smithi136.stderr:221738: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T21:05:29.223 INFO:tasks.workunit.client.0.smithi136.stderr:221738: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836597 -lt 21474836597 2024-09-15T21:05:29.223 INFO:tasks.workunit.client.0.smithi136.stderr:221738: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T21:05:29.223 INFO:tasks.workunit.client.0.smithi136.stderr:221738: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-42949673066 2024-09-15T21:05:29.223 INFO:tasks.workunit.client.0.smithi136.stderr:221738: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T21:05:29.223 INFO:tasks.workunit.client.0.smithi136.stderr:221738: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-15T21:05:29.223 INFO:tasks.workunit.client.0.smithi136.stderr:221738: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-42949673066 2024-09-15T21:05:29.223 INFO:tasks.workunit.client.0.smithi136.stderr:221738: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T21:05:29.223 INFO:tasks.workunit.client.0.smithi136.stderr:221738: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=42949673066 2024-09-15T21:05:29.223 INFO:tasks.workunit.client.0.smithi136.stderr:221738: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 42949673066' 2024-09-15T21:05:30.465 INFO:tasks.workunit.client.0.smithi136.stderr:221738: waiti2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-15T21:05:30.465 INFO:tasks.workunit.client.0.smithi136.stderr:221740: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 42949673067 -lt 42949673067 2024-09-15T21:05:30.465 INFO:tasks.workunit.client.0.smithi136.stderr:221740: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T21:05:30.465 INFO:tasks.workunit.client.0.smithi136.stderr:221740: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-64424509541 2024-09-15T21:05:30.465 INFO:tasks.workunit.client.0.smithi136.stderr:221740: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T21:05:30.466 INFO:tasks.workunit.client.0.smithi136.stderr:221740: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-15T21:05:30.466 INFO:tasks.workunit.client.0.smithi136.stderr:221740: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-64424509541 2024-09-15T21:05:30.466 INFO:tasks.workunit.client.0.smithi136.stderr:221740: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T21:05:30.466 INFO:tasks.workunit.client.0.smithi136.stderr:221740: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=64424509541 2024-09-15T21:05:30.466 INFO:tasks.workunit.client.0.smithi136.stderr:221740: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 64424509541' 2024-09-15T21:05:30.466 INFO:tasks.workunit.client.0.smithi136.stderr:221740: waiting osd.2 seq 64424509541 2024-09-15T21:05:30.466 INFO:tasks.workunit.client.0.smithi136.stderr:221740: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-15T21:05:30.466 INFO:tasks.workunit.client.0.smithi136.stderr:221740: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 64424509542 -lt 64424509541 2024-09-15T21:05:30.466 INFO:tasks.workunit.client.0.smithi136.stderr:221740: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T21:05:30.466 INFO:tasks.workunit.client.0.smithi136.stderr:221740: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 3-523986010114 2024-09-15T21:05:30.466 INFO:tasks.workunit.client.0.smithi136.stderr:221740: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T21:05:30.466 INFO:tasks.workunit.client.0.smithi136.stderr:221740: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=3 2024-09-15T21:05:30.466 INFO:tasks.workunit.client.0.smithi136.stderr:221740: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 3-523986010114 2024-09-15T21:05:30.467 INFO:tasks.workunit.client.0.smithi136.stderr:221740: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T21:05:30.467 INFO:tasks.workunit.client.0.smithi136.stderr:221740: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=523986010114 2024-09-15T21:05:30.467 INFO:tasks.workunit.client.0.smithi136.stderr:221740: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.3 seq 523986010114' 2024-09-15T21:05:30.467 INFO:tasks.workunit.client.0.smithi136.stderr:221740: waiting osd.3 seq 523986010114 2024-09-15T21:05:30.467 INFO:tasks.workunit.client.0.smithi136.stderr:221740: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-15T21:05:30.467 INFO:tasks.workunit.client.0.smithi136.stderr:221740: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 523986010115 -lt 523986010114 2024-09-15T21:05:30.467 INFO:tasks.workunit.client.0.smithi136.stderr:221740: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T21:05:30.467 INFO:tasks.workunit.client.0.smithi136.stderr:221740: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 4-107374182487 2024-09-15T21:05:30.467 INFO:tasks.workunit.client.0.smithi136.stderr:221740: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T21:05:30.467 INFO:tasks.workunit.client.0.smithi136.stderr:221740: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=4 2024-09-15T21:05:30.467 INFO:tasks.workunit.client.0.smithi136.stderr:221740: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 4-107374182487 2024-09-15T21:05:30.467 INFO:tasks.workunit.client.0.smithi136.stderr:221740: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T21:05:30.467 INFO:tasks.workunit.client.0.smithi136.stderr:221740: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=107374182487 2024-09-15T21:05:30.467 INFO:tasks.workunit.client.0.smithi136.stderr:221740: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.4 seq 107374182487' 2024-09-15T21:05:30.468 INFO:tasks.workunit.client.0.smithi136.stderr:221740: waiting osd.4 seq 107374182487 2024-09-15T21:05:30.468 INFO:tasks.workunit.client.0.smithi136.stderr:221740: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 4 2024-09-15T21:05:30.468 INFO:tasks.workunit.client.0.smithi136.stderr:221740: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 107374182487 -lt 107374182487 2024-09-15T21:05:30.468 INFO:tasks.workunit.client.0.smithi136.stderr:221740: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T21:05:30.468 INFO:tasks.workunit.client.0.smithi136.stderr:221740: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 5-523986010115 2024-09-15T21:05:30.508 INFO:tasks.workunit.client.0.smithi136.stderr:221740: ng osd.1 seq 42949673066 2024-09-15T21:05:30.508 INFO:tasks.workunit.client.0.smithi136.stderr:221738: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-15T21:05:30.508 INFO:tasks.workunit.client.0.smithi136.stderr:221738: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 42949673067 -lt 42949673066 2024-09-15T21:05:30.508 INFO:tasks.workunit.client.0.smithi136.stderr:221738: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T21:05:30.508 INFO:tasks.workunit.client.0.smithi136.stderr:221738: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-64424509542 2024-09-15T21:05:30.508 INFO:tasks.workunit.client.0.smithi136.stderr:221738: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T21:05:30.508 INFO:tasks.workunit.client.0.smithi136.stderr:221738: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-15T21:05:30.508 INFO:tasks.workunit.client.0.smithi136.stderr:221738: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-64424509542 2024-09-15T21:05:30.508 INFO:tasks.workunit.client.0.smithi136.stderr:221738: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T21:05:30.508 INFO:tasks.workunit.client.0.smithi136.stderr:221738: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=64424509542 2024-09-15T21:05:30.509 INFO:tasks.workunit.client.0.smithi136.stderr:221738: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 64424509542' 2024-09-15T21:05:30.509 INFO:tasks.workunit.client.0.smithi136.stderr:221738: waiting osd.2 seq 64424509542 2024-09-15T21:05:30.509 INFO:tasks.workunit.client.0.smithi136.stderr:221738: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-15T21:05:30.509 INFO:tasks.workunit.client.0.smithi136.stderr:221738: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 64424509542 -lt 64424509542 2024-09-15T21:05:30.509 INFO:tasks.workunit.client.0.smithi136.stderr:221738: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T21:05:30.509 INFO:tasks.workunit.client.0.smithi136.stderr:221738: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 3-523986010115 2024-09-15T21:05:30.509 INFO:tasks.workunit.client.0.smithi136.stderr:221738: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T21:05:30.509 INFO:tasks.workunit.client.0.smithi136.stderr:221738: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=3 2024-09-15T21:05:30.509 INFO:tasks.workunit.client.0.smithi136.stderr:221738: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 3-523986010115 2024-09-15T21:05:30.509 INFO:tasks.workunit.client.0.smithi136.stderr:221738: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T21:05:30.509 INFO:tasks.workunit.client.0.smithi136.stderr:221738: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=523986010115 2024-09-15T21:05:30.509 INFO:tasks.workunit.client.0.smithi136.stderr:221738: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.3 seq 523986010115' 2024-09-15T21:05:30.509 INFO:tasks.workunit.client.0.smithi136.stderr:221738: waiting osd.3 seq 523986010115 2024-09-15T21:05:30.510 INFO:tasks.workunit.client.0.smithi136.stderr:221738: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-15T21:05:30.510 INFO:tasks.workunit.client.0.smithi136.stderr:221738: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 523986010115 -lt 523986010115 2024-09-15T21:05:30.510 INFO:tasks.workunit.client.0.smithi136.stderr:221738: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T21:05:30.510 INFO:tasks.workunit.client.0.smithi136.stderr:221738: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 4-107374182486 2024-09-15T21:05:30.510 INFO:tasks.workunit.client.0.smithi136.stderr:221738: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T21:05:30.510 INFO:tasks.workunit.client.0.smithi136.stderr:221738: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=4 2024-09-15T21:05:30.510 INFO:tasks.workunit.client.0.smithi136.stderr:221738: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 4-107374182486 2024-09-15T21:05:30.510 INFO:tasks.workunit.client.0.smithi136.stderr:221738: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T21:05:30.510 INFO:tasks.workunit.client.0.smithi136.stderr:221738: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=107374182486 2024-09-15T21:05:30.510 INFO:tasks.workunit.client.0.smithi136.stderr:221738: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.4 seq 107374182486' 2024-09-15T21:05:30.510 INFO:tasks.workunit.client.0.smithi136.stderr:221738: waiting osd.4 seq 107374182486 2024-09-15T21:05:30.510 INFO:tasks.workunit.client.0.smithi136.stderr:221738: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 4 2024-09-15T21:05:30.510 INFO:tasks.workunit.client.0.smithi136.stderr:221738: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 107374182487 -lt 107374182486 2024-09-15T21:05:30.510 INFO:tasks.workunit.client.0.smithi136.stderr:221738: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T21:05:31.398 INFO:tasks.workunit.client.0.smithi136.stderr:221738: //home/ubuntu/ceph//home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T21:05:31.398 INFO:tasks.workunit.client.0.smithi136.stderr:221740: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=5 2024-09-15T21:05:31.398 INFO:tasks.workunit.client.0.smithi136.stderr:221740: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 5-523986010115 2024-09-15T21:05:31.398 INFO:tasks.workunit.client.0.smithi136.stderr:221740: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T21:05:31.398 INFO:tasks.workunit.client.0.smithi136.stderr:221740: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=523986010115 2024-09-15T21:05:31.398 INFO:tasks.workunit.client.0.smithi136.stderr:221740: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.5 seq 523986010115' 2024-09-15T21:05:31.398 INFO:tasks.workunit.client.0.smithi136.stderr:221740: waiting osd.5 seq 523986010115 2024-09-15T21:05:31.398 INFO:tasks.workunit.client.0.smithi136.stderr:221740: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 5 2024-09-15T21:05:31.398 INFO:tasks.workunit.client.0.smithi136.stderr:221740: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 523986010115 -lt 523986010115 2024-09-15T21:05:31.398 INFO:tasks.workunit.client.0.smithi136.stderr:221740: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T21:05:31.398 INFO:tasks.workunit.client.0.smithi136.stderr:221740: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 6-150323855434 2024-09-15T21:05:31.399 INFO:tasks.workunit.client.0.smithi136.stderr:221740: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T21:05:31.399 INFO:tasks.workunit.client.0.smithi136.stderr:221740: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=6 2024-09-15T21:05:31.399 INFO:tasks.workunit.client.0.smithi136.stderr:221740: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 6-150323855434 2024-09-15T21:05:31.399 INFO:tasks.workunit.client.0.smithi136.stderr:221740: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T21:05:31.399 INFO:tasks.workunit.client.0.smithi136.stderr:221740: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=150323855434 2024-09-15T21:05:31.399 INFO:tasks.workunit.client.0.smithi136.stderr:221740: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.6 seq 150323855434' 2024-09-15T21:05:31.399 INFO:tasks.workunit.client.0.smithi136.stderr:221740: waiting osd.6 seq 150323855434 2024-09-15T21:05:31.399 INFO:tasks.workunit.client.0.smithi136.stderr:221740: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 6 2024-09-15T21:05:31.399 INFO:tasks.workunit.client.0.smithi136.stderr:221740: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 150323855436 -lt 150323855434 2024-09-15T21:05:31.399 INFO:tasks.workunit.client.0.smithi136.stderr:221740: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T21:05:31.399 INFO:tasks.workunit.client.0.smithi136.stderr:221740: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 7-171798691906 2024-09-15T21:05:31.399 INFO:tasks.workunit.client.0.smithi136.stderr:221740: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T21:05:31.399 INFO:tasks.workunit.client.0.smithi136.stderr:221740: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=7 2024-09-15T21:05:31.399 INFO:tasks.workunit.client.0.smithi136.stderr:221740: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 7-171798691906 2024-09-15T21:05:31.400 INFO:tasks.workunit.client.0.smithi136.stderr:221740: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T21:05:31.400 INFO:tasks.workunit.client.0.smithi136.stderr:221740: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=171798691906 2024-09-15T21:05:31.400 INFO:tasks.workunit.client.0.smithi136.stderr:221740: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.7 seq 171798691906' 2024-09-15T21:05:31.400 INFO:tasks.workunit.client.0.smithi136.stderr:221740: waiting osd.7 seq 171798691906 2024-09-15T21:05:31.400 INFO:tasks.workunit.client.0.smithi136.stderr:221740: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 7 2024-09-15T21:05:31.400 INFO:tasks.workunit.client.0.smithi136.stderr:221740: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 171798691908 -lt 171798691906 2024-09-15T21:05:31.400 INFO:tasks.workunit.client.0.smithi136.stderr:221740: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T21:05:31.400 INFO:tasks.workunit.client.0.smithi136.stderr:221740: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 8-193273528383 2024-09-15T21:05:31.400 INFO:tasks.workunit.client.0.smithi136.stderr:221740: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T21:05:31.400 INFO:tasks.workunit.client.0.smithi136.stderr:221740: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=8 2024-09-15T21:05:31.400 INFO:tasks.workunit.client.0.smithi136.stderr:221740: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 8-193273528383 2024-09-15T21:05:31.491 INFO:tasks.workunit.client.0.smithi136.stderr:221740: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: fltest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 5-523986010114 2024-09-15T21:05:31.491 INFO:tasks.workunit.client.0.smithi136.stderr:221738: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T21:05:31.491 INFO:tasks.workunit.client.0.smithi136.stderr:221738: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=5 2024-09-15T21:05:31.491 INFO:tasks.workunit.client.0.smithi136.stderr:221738: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 5-523986010114 2024-09-15T21:05:31.491 INFO:tasks.workunit.client.0.smithi136.stderr:221738: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T21:05:31.491 INFO:tasks.workunit.client.0.smithi136.stderr:221738: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=523986010114 2024-09-15T21:05:31.491 INFO:tasks.workunit.client.0.smithi136.stderr:221738: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.5 seq 523986010114' 2024-09-15T21:05:31.491 INFO:tasks.workunit.client.0.smithi136.stderr:221738: waiting osd.5 seq 523986010114 2024-09-15T21:05:31.491 INFO:tasks.workunit.client.0.smithi136.stderr:221738: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 5 2024-09-15T21:05:31.491 INFO:tasks.workunit.client.0.smithi136.stderr:221738: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 523986010115 -lt 523986010114 2024-09-15T21:05:31.491 INFO:tasks.workunit.client.0.smithi136.stderr:221738: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T21:05:31.491 INFO:tasks.workunit.client.0.smithi136.stderr:221738: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 6-150323855435 2024-09-15T21:05:31.492 INFO:tasks.workunit.client.0.smithi136.stderr:221738: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T21:05:31.492 INFO:tasks.workunit.client.0.smithi136.stderr:221738: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=6 2024-09-15T21:05:31.492 INFO:tasks.workunit.client.0.smithi136.stderr:221738: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 6-150323855435 2024-09-15T21:05:31.492 INFO:tasks.workunit.client.0.smithi136.stderr:221738: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T21:05:31.492 INFO:tasks.workunit.client.0.smithi136.stderr:221738: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=150323855435 2024-09-15T21:05:31.492 INFO:tasks.workunit.client.0.smithi136.stderr:221738: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.6 seq 150323855435' 2024-09-15T21:05:31.492 INFO:tasks.workunit.client.0.smithi136.stderr:221738: waiting osd.6 seq 150323855435 2024-09-15T21:05:31.492 INFO:tasks.workunit.client.0.smithi136.stderr:221738: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 6 2024-09-15T21:05:31.492 INFO:tasks.workunit.client.0.smithi136.stderr:221738: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 150323855436 -lt 150323855435 2024-09-15T21:05:31.492 INFO:tasks.workunit.client.0.smithi136.stderr:221738: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T21:05:31.492 INFO:tasks.workunit.client.0.smithi136.stderr:221738: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 7-171798691907 2024-09-15T21:05:31.492 INFO:tasks.workunit.client.0.smithi136.stderr:221738: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T21:05:31.492 INFO:tasks.workunit.client.0.smithi136.stderr:221738: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=7 2024-09-15T21:05:31.493 INFO:tasks.workunit.client.0.smithi136.stderr:221738: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 7-171798691907 2024-09-15T21:05:31.493 INFO:tasks.workunit.client.0.smithi136.stderr:221738: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T21:05:31.493 INFO:tasks.workunit.client.0.smithi136.stderr:221738: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=171798691907 2024-09-15T21:05:31.493 INFO:tasks.workunit.client.0.smithi136.stderr:221738: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.7 seq 171798691907' 2024-09-15T21:05:31.493 INFO:tasks.workunit.client.0.smithi136.stderr:221738: waiting osd.7 seq 171798691907 2024-09-15T21:05:31.493 INFO:tasks.workunit.client.0.smithi136.stderr:221738: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 7 2024-09-15T21:05:31.493 INFO:tasks.workunit.client.0.smithi136.stderr:221738: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 171798691908 -lt 171798691907 2024-09-15T21:05:31.493 INFO:tasks.workunit.client.0.smithi136.stderr:221738: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T21:05:31.493 INFO:tasks.workunit.client.0.smithi136.stderr:221738: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 8-193273528382 2024-09-15T21:05:31.493 INFO:tasks.workunit.client.0.smithi136.stderr:221738: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T21:05:31.493 INFO:tasks.workunit.client.0.smithi136.stderr:221738: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=8 2024-09-15T21:05:33.052 INFO:tasks.workunit.client.0.smithi136.stderr:221738: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echush_pg_stats: cut -d - -f 2 2024-09-15T21:05:33.052 INFO:tasks.workunit.client.0.smithi136.stderr:221740: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=193273528383 2024-09-15T21:05:33.052 INFO:tasks.workunit.client.0.smithi136.stderr:221740: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.8 seq 193273528383' 2024-09-15T21:05:33.052 INFO:tasks.workunit.client.0.smithi136.stderr:221740: waiting osd.8 seq 193273528383 2024-09-15T21:05:33.052 INFO:tasks.workunit.client.0.smithi136.stderr:221740: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 8 2024-09-15T21:05:33.052 INFO:tasks.workunit.client.0.smithi136.stderr:221740: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 193273528384 -lt 193273528383 2024-09-15T21:05:33.052 INFO:tasks.workunit.client.0.smithi136.stderr:221740: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T21:05:33.053 INFO:tasks.workunit.client.0.smithi136.stderr:221740: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 9-450971566095 2024-09-15T21:05:33.053 INFO:tasks.workunit.client.0.smithi136.stderr:221740: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T21:05:33.053 INFO:tasks.workunit.client.0.smithi136.stderr:221740: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=9 2024-09-15T21:05:33.053 INFO:tasks.workunit.client.0.smithi136.stderr:221740: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 9-450971566095 2024-09-15T21:05:33.053 INFO:tasks.workunit.client.0.smithi136.stderr:221740: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T21:05:33.053 INFO:tasks.workunit.client.0.smithi136.stderr:221740: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=450971566095 2024-09-15T21:05:33.053 INFO:tasks.workunit.client.0.smithi136.stderr:221740: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.9 seq 450971566095' 2024-09-15T21:05:33.053 INFO:tasks.workunit.client.0.smithi136.stderr:221740: waiting osd.9 seq 450971566095 2024-09-15T21:05:33.053 INFO:tasks.workunit.client.0.smithi136.stderr:221740: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 9 2024-09-15T21:05:33.053 INFO:tasks.workunit.client.0.smithi136.stderr:221740: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 450971566095 -lt 450971566095 2024-09-15T21:05:33.053 INFO:tasks.workunit.client.0.smithi136.stderr:221740: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-15T21:05:33.053 INFO:tasks.workunit.client.0.smithi136.stderr:221740: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T21:05:33.053 INFO:tasks.workunit.client.0.smithi136.stderr:221740: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T21:05:33.054 INFO:tasks.workunit.client.0.smithi136.stderr:221740: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 5 == 0 2024-09-15T21:05:33.054 INFO:tasks.workunit.client.0.smithi136.stderr:221740: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-15T21:05:33.054 INFO:tasks.workunit.client.0.smithi136.stderr:221740: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-15T21:05:33.054 INFO:tasks.workunit.client.0.smithi136.stderr:221740: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-15T21:05:33.054 INFO:tasks.workunit.client.0.smithi136.stderr:221740: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T21:05:33.054 INFO:tasks.workunit.client.0.smithi136.stderr:221740: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T21:05:33.054 INFO:tasks.workunit.client.0.smithi136.stderr:221740: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T21:05:33.054 INFO:tasks.workunit.client.0.smithi136.stderr:221740: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T21:05:33.054 INFO:tasks.workunit.client.0.smithi136.stderr:221740: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=4 2024-09-15T21:05:33.054 INFO:tasks.workunit.client.0.smithi136.stderr:221740: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-15T21:05:33.054 INFO:tasks.workunit.client.0.smithi136.stderr:221740: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T21:05:33.054 INFO:tasks.workunit.client.0.smithi136.stderr:221740: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T21:05:33.054 INFO:tasks.workunit.client.0.smithi136.stderr:221740: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 = 5 2024-09-15T21:05:33.054 INFO:tasks.workunit.client.0.smithi136.stderr:221740: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 4 '!=' -1 2024-09-15T21:05:33.054 INFO:tasks.workunit.client.0.smithi136.stderr:221740: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: loop=0 2024-09-15T21:05:33.129 INFO:tasks.workunit.client.0.smithi136.stderr:221740: /ho 8-193273528382 2024-09-15T21:05:33.129 INFO:tasks.workunit.client.0.smithi136.stderr:221738: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T21:05:33.129 INFO:tasks.workunit.client.0.smithi136.stderr:221738: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=193273528382 2024-09-15T21:05:33.129 INFO:tasks.workunit.client.0.smithi136.stderr:221738: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.8 seq 193273528382' 2024-09-15T21:05:33.129 INFO:tasks.workunit.client.0.smithi136.stderr:221738: waiting osd.8 seq 193273528382 2024-09-15T21:05:33.129 INFO:tasks.workunit.client.0.smithi136.stderr:221738: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 8 2024-09-15T21:05:33.130 INFO:tasks.workunit.client.0.smithi136.stderr:221738: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 193273528384 -lt 193273528382 2024-09-15T21:05:33.130 INFO:tasks.workunit.client.0.smithi136.stderr:221738: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T21:05:33.130 INFO:tasks.workunit.client.0.smithi136.stderr:221738: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 9-450971566094 2024-09-15T21:05:33.130 INFO:tasks.workunit.client.0.smithi136.stderr:221738: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T21:05:33.130 INFO:tasks.workunit.client.0.smithi136.stderr:221738: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=9 2024-09-15T21:05:33.130 INFO:tasks.workunit.client.0.smithi136.stderr:221738: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 9-450971566094 2024-09-15T21:05:33.130 INFO:tasks.workunit.client.0.smithi136.stderr:221738: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T21:05:33.130 INFO:tasks.workunit.client.0.smithi136.stderr:221738: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=450971566094 2024-09-15T21:05:33.130 INFO:tasks.workunit.client.0.smithi136.stderr:221738: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.9 seq 450971566094' 2024-09-15T21:05:33.130 INFO:tasks.workunit.client.0.smithi136.stderr:221738: waiting osd.9 seq 450971566094 2024-09-15T21:05:33.130 INFO:tasks.workunit.client.0.smithi136.stderr:221738: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 9 2024-09-15T21:05:33.130 INFO:tasks.workunit.client.0.smithi136.stderr:221738: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 450971566095 -lt 450971566094 2024-09-15T21:05:33.130 INFO:tasks.workunit.client.0.smithi136.stderr:221738: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-15T21:05:33.131 INFO:tasks.workunit.client.0.smithi136.stderr:221738: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T21:05:33.131 INFO:tasks.workunit.client.0.smithi136.stderr:221738: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T21:05:33.131 INFO:tasks.workunit.client.0.smithi136.stderr:221738: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 5 == 0 2024-09-15T21:05:33.131 INFO:tasks.workunit.client.0.smithi136.stderr:221738: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-15T21:05:33.131 INFO:tasks.workunit.client.0.smithi136.stderr:221738: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-15T21:05:33.131 INFO:tasks.workunit.client.0.smithi136.stderr:221738: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-15T21:05:33.131 INFO:tasks.workunit.client.0.smithi136.stderr:221738: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T21:05:33.131 INFO:tasks.workunit.client.0.smithi136.stderr:221738: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T21:05:33.131 INFO:tasks.workunit.client.0.smithi136.stderr:221738: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T21:05:33.131 INFO:tasks.workunit.client.0.smithi136.stderr:221738: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T21:05:33.131 INFO:tasks.workunit.client.0.smithi136.stderr:221738: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=4 2024-09-15T21:05:33.131 INFO:tasks.workunit.client.0.smithi136.stderr:221738: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-15T21:05:33.131 INFO:tasks.workunit.client.0.smithi136.stderr:221738: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T21:05:33.131 INFO:tasks.workunit.client.0.smithi136.stderr:221738: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T21:05:33.132 INFO:tasks.workunit.client.0.smithi136.stderr:221738: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 = 5 2024-09-15T21:05:33.132 INFO:tasks.workunit.client.0.smithi136.stderr:221738: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 4 '!=' -1 2024-09-15T21:05:33.812 INFO:tasks.workunit.client.0.smithi136.stderr:221738: /home/ome/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1652: wait_for_clean: num_active_clean=4 2024-09-15T21:05:33.812 INFO:tasks.workunit.client.0.smithi136.stderr:221740: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-15T21:05:33.812 INFO:tasks.workunit.client.0.smithi136.stderr:221740: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 0.1 2024-09-15T21:05:33.812 INFO:tasks.workunit.client.0.smithi136.stderr:221740: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-15T21:05:33.812 INFO:tasks.workunit.client.0.smithi136.stderr:221740: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-15T21:05:33.812 INFO:tasks.workunit.client.0.smithi136.stderr:221740: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-15T21:05:33.812 INFO:tasks.workunit.client.0.smithi136.stderr:221740: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-15T21:05:33.812 INFO:tasks.workunit.client.0.smithi136.stderr:221740: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T21:05:33.812 INFO:tasks.workunit.client.0.smithi136.stderr:221740: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T21:05:33.813 INFO:tasks.workunit.client.0.smithi136.stderr:221740: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T21:05:33.813 INFO:tasks.workunit.client.0.smithi136.stderr:221740: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T21:05:33.813 INFO:tasks.workunit.client.0.smithi136.stderr:221740: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=5 2024-09-15T21:05:33.813 INFO:tasks.workunit.client.0.smithi136.stderr:221740: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-15T21:05:33.813 INFO:tasks.workunit.client.0.smithi136.stderr:221740: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T21:05:33.813 INFO:tasks.workunit.client.0.smithi136.stderr:221740: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T21:05:33.813 INFO:tasks.workunit.client.0.smithi136.stderr:221740: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 = 5 2024-09-15T21:05:33.813 INFO:tasks.workunit.client.0.smithi136.stderr:221740: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-15T21:05:33.813 INFO:tasks.workunit.client.0.smithi136.stderr:221740: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-15T21:05:33.813 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: run_in_background: return 0 2024-09-15T21:05:33.898 INFO:tasks.workunit.client.0.smithi136.stderr:ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: loop=0 2024-09-15T21:05:33.898 INFO:tasks.workunit.client.0.smithi136.stderr:221738: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1652: wait_for_clean: num_active_clean=4 2024-09-15T21:05:33.898 INFO:tasks.workunit.client.0.smithi136.stderr:221738: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-15T21:05:33.898 INFO:tasks.workunit.client.0.smithi136.stderr:221738: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 0.1 2024-09-15T21:05:33.898 INFO:tasks.workunit.client.0.smithi136.stderr:221738: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-15T21:05:33.899 INFO:tasks.workunit.client.0.smithi136.stderr:221738: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-15T21:05:33.899 INFO:tasks.workunit.client.0.smithi136.stderr:221738: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-15T21:05:33.899 INFO:tasks.workunit.client.0.smithi136.stderr:221738: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-15T21:05:33.899 INFO:tasks.workunit.client.0.smithi136.stderr:221738: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T21:05:33.899 INFO:tasks.workunit.client.0.smithi136.stderr:221738: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T21:05:33.899 INFO:tasks.workunit.client.0.smithi136.stderr:221738: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T21:05:33.899 INFO:tasks.workunit.client.0.smithi136.stderr:221738: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T21:05:33.899 INFO:tasks.workunit.client.0.smithi136.stderr:221738: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=5 2024-09-15T21:05:33.899 INFO:tasks.workunit.client.0.smithi136.stderr:221738: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-15T21:05:33.899 INFO:tasks.workunit.client.0.smithi136.stderr:221738: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T21:05:33.899 INFO:tasks.workunit.client.0.smithi136.stderr:221738: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T21:05:33.899 INFO:tasks.workunit.client.0.smithi136.stderr:221738: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 = 5 2024-09-15T21:05:33.899 INFO:tasks.workunit.client.0.smithi136.stderr:221738: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-15T21:05:33.900 INFO:tasks.workunit.client.0.smithi136.stderr:221738: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-15T21:05:33.900 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: run_in_background: return 0 2024-09-15T21:05:33.900 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: wait_background: for pid in $pids 2024-09-15T21:05:33.900 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2175: wait_background: wait 221737 2024-09-15T21:05:33.900 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: wait_background: eval 'pids='\'''\''' 2024-09-15T21:05:33.900 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: wait_background: pids= 2024-09-15T21:05:33.900 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: wait_background: return 0 2024-09-15T21:05:33.900 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:212: corrupt_and_repair_two: return_code=0 2024-09-15T21:05:33.900 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:213: corrupt_and_repair_two: '[' 0 -ne 0 ']' 2024-09-15T21:05:33.900 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:215: corrupt_and_repair_two: rados --pool ecpool get SOMETHING td/osd-scrub-repair/COPY 2024-09-15T21:05:33.951 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:216: corrupt_and_repair_two: diff td/osd-scrub-repair/ORIGINAL td/osd-scrub-repair/COPY 2024-09-15T21:05:33.953 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:66: run: teardown td/osd-scrub-repair 2024-09-15T21:05:33.953 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/osd-scrub-repair 2024-09-15T21:05:33.953 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-15T21:05:33.953 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/osd-scrub-repair KILL 2024-09-15T21:05:33.953 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-15T21:05:33.953 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-15T21:05:33.953 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-15T21:05:33.954 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-15T21:05:33.954 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-15T21:05:34.067 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-15T21:05:34.067 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-15T21:05:34.068 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-15T21:05:34.068 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-15T21:05:34.069 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-15T21:05:34.070 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-15T21:05:34.070 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-15T21:05:34.071 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T21:05:34.071 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-15T21:05:34.071 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-15T21:05:34.071 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T21:05:34.072 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-15T21:05:34.073 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-15T21:05:34.073 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/osd-scrub-repair 2024-09-15T21:05:34.139 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-15T21:05:34.139 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T21:05:34.139 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52602 2024-09-15T21:05:34.139 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.52602 2024-09-15T21:05:34.141 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-15T21:05:34.141 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-15T21:05:34.141 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:63: run: for func in $funcs 2024-09-15T21:05:34.141 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:64: run: setup td/osd-scrub-repair 2024-09-15T21:05:34.141 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/osd-scrub-repair 2024-09-15T21:05:34.141 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/osd-scrub-repair 2024-09-15T21:05:34.141 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/osd-scrub-repair 2024-09-15T21:05:34.141 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-15T21:05:34.141 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/osd-scrub-repair KILL 2024-09-15T21:05:34.141 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-15T21:05:34.141 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-15T21:05:34.141 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-15T21:05:34.142 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-15T21:05:34.142 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-15T21:05:34.142 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-15T21:05:34.143 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-15T21:05:34.144 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-15T21:05:34.144 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-15T21:05:34.145 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-15T21:05:34.145 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-15T21:05:34.145 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-15T21:05:34.146 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T21:05:34.146 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-15T21:05:34.147 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-15T21:05:34.147 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T21:05:34.148 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-15T21:05:34.148 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-15T21:05:34.149 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/osd-scrub-repair 2024-09-15T21:05:34.150 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-15T21:05:34.150 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T21:05:34.150 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52602 2024-09-15T21:05:34.150 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.52602 2024-09-15T21:05:34.151 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-15T21:05:34.151 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-15T21:05:34.151 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/osd-scrub-repair 2024-09-15T21:05:34.153 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2024-09-15T21:05:34.153 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T21:05:34.153 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52602 2024-09-15T21:05:34.153 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.52602 2024-09-15T21:05:34.155 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2024-09-15T21:05:34.155 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 4096 -le 1024 ']' 2024-09-15T21:05:34.155 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2024-09-15T21:05:34.156 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/osd-scrub-repair 1' TERM HUP INT 2024-09-15T21:05:34.156 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:65: run: TEST_corrupt_and_repair_lrc_overwrites td/osd-scrub-repair 2024-09-15T21:05:34.156 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:846: TEST_corrupt_and_repair_lrc_overwrites: '[' true = true ']' 2024-09-15T21:05:34.156 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:847: TEST_corrupt_and_repair_lrc_overwrites: corrupt_and_repair_lrc td/osd-scrub-repair true 2024-09-15T21:05:34.156 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:825: corrupt_and_repair_lrc: local dir=td/osd-scrub-repair 2024-09-15T21:05:34.156 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:826: corrupt_and_repair_lrc: local allow_overwrites=true 2024-09-15T21:05:34.156 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:827: corrupt_and_repair_lrc: local poolname=ecpool 2024-09-15T21:05:34.156 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:829: corrupt_and_repair_lrc: run_mon td/osd-scrub-repair a 2024-09-15T21:05:34.156 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/osd-scrub-repair 2024-09-15T21:05:34.156 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2024-09-15T21:05:34.156 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2024-09-15T21:05:34.156 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2024-09-15T21:05:34.156 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/osd-scrub-repair/a 2024-09-15T21:05:34.156 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-scrub-repair/a --run-dir=td/osd-scrub-repair 2024-09-15T21:05:34.610 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2024-09-15T21:05:34.610 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-15T21:05:34.610 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-15T21:05:34.610 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-15T21:05:34.611 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T21:05:34.611 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52602 2024-09-15T21:05:34.611 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.52602/$cluster-$name.asok' 2024-09-15T21:05:34.611 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-scrub-repair/a '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.52602/$cluster-$name.asok' --mon-cluster-log-file=td/osd-scrub-repair/log --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-09-15T21:05:34.794 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2024-09-15T21:05:34.795 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2024-09-15T21:05:34.795 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-15T21:05:34.795 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-15T21:05:34.795 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2024-09-15T21:05:34.796 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2024-09-15T21:05:34.796 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-15T21:05:34.796 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-15T21:05:34.796 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-15T21:05:34.797 INFO:tasks.workunit.client.0.smithi136.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-15T21:05:34.797 INFO:tasks.workunit.client.0.smithi136.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T21:05:34.797 INFO:tasks.workunit.client.0.smithi136.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52602 2024-09-15T21:05:34.797 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.52602/ceph-mon.a.asok 2024-09-15T21:05:34.797 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-15T21:05:34.797 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.52602/ceph-mon.a.asok config get fsid 2024-09-15T21:05:34.875 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2024-09-15T21:05:34.876 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-15T21:05:34.876 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-15T21:05:34.876 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2024-09-15T21:05:34.876 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2024-09-15T21:05:34.876 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-15T21:05:34.876 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-15T21:05:34.876 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-15T21:05:34.876 INFO:tasks.workunit.client.0.smithi136.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-15T21:05:34.876 INFO:tasks.workunit.client.0.smithi136.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T21:05:34.876 INFO:tasks.workunit.client.0.smithi136.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52602 2024-09-15T21:05:34.876 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.52602/ceph-mon.a.asok 2024-09-15T21:05:34.877 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-15T21:05:34.877 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.52602/ceph-mon.a.asok config get mon_host 2024-09-15T21:05:34.954 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:830: corrupt_and_repair_lrc: run_mgr td/osd-scrub-repair x 2024-09-15T21:05:34.954 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: run_mgr: local dir=td/osd-scrub-repair 2024-09-15T21:05:34.954 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: run_mgr: shift 2024-09-15T21:05:34.954 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: run_mgr: local id=x 2024-09-15T21:05:34.955 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: run_mgr: shift 2024-09-15T21:05:34.955 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:548: run_mgr: local data=td/osd-scrub-repair/x 2024-09-15T21:05:34.955 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:550: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-15T21:05:35.762 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: get_asok_path 2024-09-15T21:05:35.763 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-15T21:05:35.763 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-15T21:05:35.763 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-15T21:05:35.763 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T21:05:35.763 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52602 2024-09-15T21:05:35.763 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.52602/$cluster-$name.asok' 2024-09-15T21:05:35.763 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-15T21:05:35.764 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-scrub-repair/x '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.52602/$cluster-$name.asok' --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-15T21:05:35.792 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:831: corrupt_and_repair_lrc: seq 0 9 2024-09-15T21:05:35.796 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:831: corrupt_and_repair_lrc: for id in $(seq 0 9) 2024-09-15T21:05:35.796 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:832: corrupt_and_repair_lrc: run_osd td/osd-scrub-repair 0 2024-09-15T21:05:35.796 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-scrub-repair 2024-09-15T21:05:35.796 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-15T21:05:35.796 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=0 2024-09-15T21:05:35.796 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-15T21:05:35.796 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-scrub-repair/0 2024-09-15T21:05:35.796 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=ce4892c7-6c6d-4b4f-8e6f-07a3e11133e5 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-15T21:05:35.796 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T21:05:35.796 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T21:05:35.797 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T21:05:35.797 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-09-15T21:05:35.797 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-09-15T21:05:35.797 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-15T21:05:35.797 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-15T21:05:35.797 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-15T21:05:35.797 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-15T21:05:35.797 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-15T21:05:35.797 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-15T21:05:35.797 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-15T21:05:35.797 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T21:05:35.797 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52602 2024-09-15T21:05:35.798 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.52602/$cluster-$name.asok' 2024-09-15T21:05:35.798 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.52602/$cluster-$name.asok' 2024-09-15T21:05:35.798 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-15T21:05:35.798 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-15T21:05:35.798 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-15T21:05:35.798 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-15T21:05:35.798 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-15T21:05:35.798 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T21:05:35.798 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T21:05:35.798 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T21:05:35.798 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T21:05:35.798 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-15T21:05:35.798 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-15T21:05:35.798 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-scrub-repair/0 2024-09-15T21:05:35.798 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-15T21:05:35.799 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=5ce4f8f1-1115-4ef7-8663-17119106e16a 2024-09-15T21:05:35.799 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd0 5ce4f8f1-1115-4ef7-8663-17119106e16a' 2024-09-15T21:05:35.799 INFO:tasks.workunit.client.0.smithi136.stdout:add osd0 5ce4f8f1-1115-4ef7-8663-17119106e16a 2024-09-15T21:05:35.800 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-15T21:05:35.815 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQAfTOdmvRFtMBAAJmBcT4FvDh+QdJK59/buWg== 2024-09-15T21:05:35.815 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQAfTOdmvRFtMBAAJmBcT4FvDh+QdJK59/buWg=="}' 2024-09-15T21:05:35.815 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 5ce4f8f1-1115-4ef7-8663-17119106e16a -i td/osd-scrub-repair/0/new.json 2024-09-15T21:05:36.019 INFO:tasks.workunit.client.0.smithi136.stdout:0 2024-09-15T21:05:36.032 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-scrub-repair/0/new.json 2024-09-15T21:05:36.033 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 0 --fsid=ce4892c7-6c6d-4b4f-8e6f-07a3e11133e5 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.52602/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAfTOdmvRFtMBAAJmBcT4FvDh+QdJK59/buWg== --osd-uuid 5ce4f8f1-1115-4ef7-8663-17119106e16a 2024-09-15T21:05:36.063 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:05:36.060+0000 7f3849e0d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:05:36.078 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:05:36.075+0000 7f3849e0d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:05:36.079 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:05:36.076+0000 7f3849e0d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:05:36.079 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:05:36.076+0000 7f3849e0d640 -1 bdev(0x559d9dabbc00 td/osd-scrub-repair/0/block) open stat got: (1) Operation not permitted 2024-09-15T21:05:36.079 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:05:36.076+0000 7f3849e0d640 -1 bluestore(td/osd-scrub-repair/0) _read_fsid unparsable uuid 2024-09-15T21:05:42.062 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-scrub-repair/0/keyring 2024-09-15T21:05:42.062 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-15T21:05:42.063 INFO:tasks.workunit.client.0.smithi136.stdout:adding osd0 key to auth repository 2024-09-15T21:05:42.063 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd0 key to auth repository 2024-09-15T21:05:42.063 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-scrub-repair/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-15T21:05:42.422 INFO:tasks.workunit.client.0.smithi136.stdout:start osd.0 2024-09-15T21:05:42.422 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.0 2024-09-15T21:05:42.422 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 0 --fsid=ce4892c7-6c6d-4b4f-8e6f-07a3e11133e5 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.52602/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T21:05:42.422 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-15T21:05:42.423 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-15T21:05:42.423 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-15T21:05:42.449 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:05:42.445+0000 7fe5d701e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:05:42.451 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:05:42.447+0000 7fe5d701e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:05:42.452 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:05:42.448+0000 7fe5d701e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:05:42.700 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 0 2024-09-15T21:05:42.700 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-15T21:05:42.700 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2024-09-15T21:05:42.701 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-15T21:05:42.701 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-15T21:05:42.701 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:05:42.701 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-15T21:05:42.701 INFO:tasks.workunit.client.0.smithi136.stdout:0 2024-09-15T21:05:42.701 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:05:42.702 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:05:42.977 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:05:43.583 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:05:43.581+0000 7fe5d701e640 -1 Falling back to public interface 2024-09-15T21:05:43.979 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:05:43.979 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:05:43.979 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-15T21:05:43.979 INFO:tasks.workunit.client.0.smithi136.stdout:1 2024-09-15T21:05:43.979 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:05:43.980 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:05:44.267 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:05:45.268 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:05:45.268 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:05:45.268 INFO:tasks.workunit.client.0.smithi136.stdout:2 2024-09-15T21:05:45.269 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-15T21:05:45.269 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:05:45.269 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:05:45.558 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:05:46.192 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:05:46.189+0000 7fe5d701e640 -1 osd.0 0 log_to_monitors true 2024-09-15T21:05:46.559 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:05:46.559 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:05:46.559 INFO:tasks.workunit.client.0.smithi136.stdout:3 2024-09-15T21:05:46.560 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-15T21:05:46.560 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:05:46.560 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:05:46.844 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:05:47.845 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:05:47.846 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:05:47.846 INFO:tasks.workunit.client.0.smithi136.stdout:4 2024-09-15T21:05:47.846 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-15T21:05:47.846 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:05:47.846 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:05:48.168 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:05:49.169 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:05:49.170 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:05:49.170 INFO:tasks.workunit.client.0.smithi136.stdout:5 2024-09-15T21:05:49.170 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-15T21:05:49.170 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:05:49.171 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:05:49.458 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:05:50.460 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:05:50.460 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:05:50.460 INFO:tasks.workunit.client.0.smithi136.stdout:6 2024-09-15T21:05:50.460 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-15T21:05:50.461 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:05:50.461 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:05:50.777 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:05:51.778 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:05:51.778 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:05:51.779 INFO:tasks.workunit.client.0.smithi136.stdout:7 2024-09-15T21:05:51.779 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-15T21:05:51.779 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:05:51.779 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:05:52.141 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:05:53.142 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:05:53.142 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:05:53.142 INFO:tasks.workunit.client.0.smithi136.stdout:8 2024-09-15T21:05:53.143 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-15T21:05:53.143 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:05:53.143 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:05:53.426 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:05:54.427 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:05:54.427 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:05:54.428 INFO:tasks.workunit.client.0.smithi136.stdout:9 2024-09-15T21:05:54.428 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-15T21:05:54.429 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:05:54.429 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:05:54.767 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:05:55.768 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:05:55.768 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:05:55.769 INFO:tasks.workunit.client.0.smithi136.stdout:10 2024-09-15T21:05:55.769 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-15T21:05:55.769 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:05:55.769 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:05:56.183 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:05:57.184 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:05:57.184 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:05:57.184 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-15T21:05:57.185 INFO:tasks.workunit.client.0.smithi136.stdout:11 2024-09-15T21:05:57.185 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:05:57.185 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:05:57.476 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:05:58.477 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:05:58.477 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:05:58.478 INFO:tasks.workunit.client.0.smithi136.stdout:12 2024-09-15T21:05:58.478 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-15T21:05:58.478 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:05:58.478 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:05:59.046 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:06:00.047 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:06:00.047 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:06:00.048 INFO:tasks.workunit.client.0.smithi136.stdout:13 2024-09-15T21:06:00.048 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-15T21:06:00.048 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:06:00.048 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:06:00.832 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:06:01.833 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:06:01.833 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:06:01.833 INFO:tasks.workunit.client.0.smithi136.stdout:14 2024-09-15T21:06:01.834 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-15T21:06:01.834 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:06:01.834 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:06:02.291 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:06:03.293 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:06:03.293 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:06:03.293 INFO:tasks.workunit.client.0.smithi136.stdout:15 2024-09-15T21:06:03.296 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-15T21:06:03.296 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:06:03.296 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:06:03.969 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:06:04.971 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:06:04.971 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:06:04.971 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-15T21:06:04.971 INFO:tasks.workunit.client.0.smithi136.stdout:16 2024-09-15T21:06:04.972 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:06:04.972 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:06:05.480 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:06:06.481 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:06:06.481 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:06:06.482 INFO:tasks.workunit.client.0.smithi136.stdout:17 2024-09-15T21:06:06.482 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-15T21:06:06.482 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:06:06.482 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:06:07.256 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:06:08.258 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:06:08.258 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:06:08.258 INFO:tasks.workunit.client.0.smithi136.stdout:18 2024-09-15T21:06:08.259 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 18 2024-09-15T21:06:08.259 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:06:08.259 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:06:08.593 INFO:tasks.workunit.client.0.smithi136.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/267805389,v1:127.0.0.1:6803/267805389] [v2:127.0.0.1:6804/267805389,v1:127.0.0.1:6805/267805389] exists,up 5ce4f8f1-1115-4ef7-8663-17119106e16a 2024-09-15T21:06:08.597 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-15T21:06:08.597 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-15T21:06:08.597 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-15T21:06:08.597 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:831: corrupt_and_repair_lrc: for id in $(seq 0 9) 2024-09-15T21:06:08.597 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:832: corrupt_and_repair_lrc: run_osd td/osd-scrub-repair 1 2024-09-15T21:06:08.597 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-scrub-repair 2024-09-15T21:06:08.597 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-15T21:06:08.597 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=1 2024-09-15T21:06:08.597 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-15T21:06:08.597 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-scrub-repair/1 2024-09-15T21:06:08.598 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=ce4892c7-6c6d-4b4f-8e6f-07a3e11133e5 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-15T21:06:08.598 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T21:06:08.598 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T21:06:08.598 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T21:06:08.598 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-09-15T21:06:08.598 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-09-15T21:06:08.598 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-15T21:06:08.598 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-15T21:06:08.598 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-15T21:06:08.598 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-15T21:06:08.598 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-15T21:06:08.598 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-15T21:06:08.598 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-15T21:06:08.599 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T21:06:08.599 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52602 2024-09-15T21:06:08.599 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.52602/$cluster-$name.asok' 2024-09-15T21:06:08.599 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.52602/$cluster-$name.asok' 2024-09-15T21:06:08.599 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-15T21:06:08.599 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-15T21:06:08.599 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-15T21:06:08.599 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-15T21:06:08.599 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-15T21:06:08.600 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T21:06:08.600 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T21:06:08.600 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T21:06:08.600 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T21:06:08.600 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-15T21:06:08.600 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-15T21:06:08.600 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-scrub-repair/1 2024-09-15T21:06:08.600 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-15T21:06:08.601 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=f3e76d42-55e1-46ef-aea2-98e4d4078155 2024-09-15T21:06:08.601 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd1 f3e76d42-55e1-46ef-aea2-98e4d4078155' 2024-09-15T21:06:08.601 INFO:tasks.workunit.client.0.smithi136.stdout:add osd1 f3e76d42-55e1-46ef-aea2-98e4d4078155 2024-09-15T21:06:08.601 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-15T21:06:08.616 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQBATOdmGpuPJBAAAWq+YdhdV3Inym4oS1hkUA== 2024-09-15T21:06:08.616 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQBATOdmGpuPJBAAAWq+YdhdV3Inym4oS1hkUA=="}' 2024-09-15T21:06:08.616 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new f3e76d42-55e1-46ef-aea2-98e4d4078155 -i td/osd-scrub-repair/1/new.json 2024-09-15T21:06:08.942 INFO:tasks.workunit.client.0.smithi136.stdout:1 2024-09-15T21:06:08.953 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-scrub-repair/1/new.json 2024-09-15T21:06:08.954 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 1 --fsid=ce4892c7-6c6d-4b4f-8e6f-07a3e11133e5 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.52602/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBATOdmGpuPJBAAAWq+YdhdV3Inym4oS1hkUA== --osd-uuid f3e76d42-55e1-46ef-aea2-98e4d4078155 2024-09-15T21:06:08.983 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:06:08.980+0000 7ff125d02640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:06:09.020 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:06:09.015+0000 7ff125d02640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:06:09.021 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:06:09.016+0000 7ff125d02640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:06:09.021 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:06:09.016+0000 7ff125d02640 -1 bdev(0x5612d0ef5c00 td/osd-scrub-repair/1/block) open stat got: (1) Operation not permitted 2024-09-15T21:06:09.021 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:06:09.016+0000 7ff125d02640 -1 bluestore(td/osd-scrub-repair/1) _read_fsid unparsable uuid 2024-09-15T21:06:15.229 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-scrub-repair/1/keyring 2024-09-15T21:06:15.229 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-15T21:06:15.230 INFO:tasks.workunit.client.0.smithi136.stdout:adding osd1 key to auth repository 2024-09-15T21:06:15.230 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd1 key to auth repository 2024-09-15T21:06:15.230 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-scrub-repair/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-15T21:06:15.589 INFO:tasks.workunit.client.0.smithi136.stdout:start osd.1 2024-09-15T21:06:15.590 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.1 2024-09-15T21:06:15.590 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 1 --fsid=ce4892c7-6c6d-4b4f-8e6f-07a3e11133e5 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.52602/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T21:06:15.590 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-15T21:06:15.590 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-15T21:06:15.590 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-15T21:06:15.617 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:06:15.614+0000 7fc952e33640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:06:15.619 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:06:15.616+0000 7fc952e33640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:06:15.620 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:06:15.617+0000 7fc952e33640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:06:15.882 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 1 2024-09-15T21:06:15.883 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-15T21:06:15.883 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2024-09-15T21:06:15.883 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-15T21:06:15.883 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-15T21:06:15.883 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:06:15.883 INFO:tasks.workunit.client.0.smithi136.stdout:0 2024-09-15T21:06:15.884 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-15T21:06:15.884 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:06:15.884 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T21:06:16.189 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:06:16.972 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:06:16.969+0000 7fc952e33640 -1 Falling back to public interface 2024-09-15T21:06:17.191 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:06:17.191 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:06:17.191 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-15T21:06:17.191 INFO:tasks.workunit.client.0.smithi136.stdout:1 2024-09-15T21:06:17.192 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:06:17.192 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T21:06:17.476 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:06:18.477 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:06:18.477 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:06:18.478 INFO:tasks.workunit.client.0.smithi136.stdout:2 2024-09-15T21:06:18.478 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-15T21:06:18.478 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:06:18.478 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T21:06:18.765 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:06:19.639 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:06:19.636+0000 7fc952e33640 -1 osd.1 0 log_to_monitors true 2024-09-15T21:06:19.766 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:06:19.767 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:06:19.767 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-15T21:06:19.767 INFO:tasks.workunit.client.0.smithi136.stdout:3 2024-09-15T21:06:19.767 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:06:19.767 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T21:06:20.158 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:06:21.159 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:06:21.159 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:06:21.160 INFO:tasks.workunit.client.0.smithi136.stdout:4 2024-09-15T21:06:21.161 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-15T21:06:21.161 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:06:21.161 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T21:06:21.495 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:06:22.496 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:06:22.497 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:06:22.497 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-15T21:06:22.497 INFO:tasks.workunit.client.0.smithi136.stdout:5 2024-09-15T21:06:22.497 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:06:22.497 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T21:06:22.797 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:06:23.798 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:06:23.799 INFO:tasks.workunit.client.0.smithi136.stdout:6 2024-09-15T21:06:23.799 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:06:23.799 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-15T21:06:23.800 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:06:23.800 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T21:06:24.113 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:06:25.115 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:06:25.115 INFO:tasks.workunit.client.0.smithi136.stdout:7 2024-09-15T21:06:25.116 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:06:25.116 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-15T21:06:25.116 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:06:25.116 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T21:06:25.466 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:06:26.467 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:06:26.467 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:06:26.468 INFO:tasks.workunit.client.0.smithi136.stdout:8 2024-09-15T21:06:26.468 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-15T21:06:26.468 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:06:26.468 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T21:06:26.793 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:06:27.794 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:06:27.794 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:06:27.794 INFO:tasks.workunit.client.0.smithi136.stdout:9 2024-09-15T21:06:27.795 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-15T21:06:27.795 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:06:27.795 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T21:06:28.102 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:06:29.103 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:06:29.103 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:06:29.103 INFO:tasks.workunit.client.0.smithi136.stdout:10 2024-09-15T21:06:29.104 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-15T21:06:29.104 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:06:29.104 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T21:06:29.465 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:06:30.467 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:06:30.467 INFO:tasks.workunit.client.0.smithi136.stdout:11 2024-09-15T21:06:30.468 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:06:30.468 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-15T21:06:30.468 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:06:30.468 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T21:06:30.755 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:06:31.756 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:06:31.756 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:06:31.756 INFO:tasks.workunit.client.0.smithi136.stdout:12 2024-09-15T21:06:31.757 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-15T21:06:31.757 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:06:31.757 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T21:06:32.398 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:06:33.399 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:06:33.399 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:06:33.400 INFO:tasks.workunit.client.0.smithi136.stdout:13 2024-09-15T21:06:33.400 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-15T21:06:33.400 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:06:33.400 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T21:06:33.858 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:06:34.860 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:06:34.860 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:06:34.860 INFO:tasks.workunit.client.0.smithi136.stdout:14 2024-09-15T21:06:34.861 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-15T21:06:34.861 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:06:34.861 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T21:06:35.392 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:06:36.394 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:06:36.394 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:06:36.394 INFO:tasks.workunit.client.0.smithi136.stdout:15 2024-09-15T21:06:36.394 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-15T21:06:36.395 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:06:36.395 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T21:06:36.795 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:06:37.796 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:06:37.797 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:06:37.797 INFO:tasks.workunit.client.0.smithi136.stdout:16 2024-09-15T21:06:37.797 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-15T21:06:37.797 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:06:37.797 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T21:06:38.205 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:06:39.207 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:06:39.207 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:06:39.207 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-15T21:06:39.207 INFO:tasks.workunit.client.0.smithi136.stdout:17 2024-09-15T21:06:39.208 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:06:39.208 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T21:06:39.650 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:06:40.652 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:06:40.652 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:06:40.652 INFO:tasks.workunit.client.0.smithi136.stdout:18 2024-09-15T21:06:40.653 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 18 2024-09-15T21:06:40.653 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:06:40.653 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T21:06:41.470 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:06:42.471 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:06:42.472 INFO:tasks.workunit.client.0.smithi136.stdout:19 2024-09-15T21:06:42.472 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:06:42.472 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 19 2024-09-15T21:06:42.472 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:06:42.473 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T21:06:42.824 INFO:tasks.workunit.client.0.smithi136.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/2930893365,v1:127.0.0.1:6811/2930893365] [v2:127.0.0.1:6812/2930893365,v1:127.0.0.1:6813/2930893365] exists,up f3e76d42-55e1-46ef-aea2-98e4d4078155 2024-09-15T21:06:42.827 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-15T21:06:42.827 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-15T21:06:42.827 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-15T21:06:42.828 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:831: corrupt_and_repair_lrc: for id in $(seq 0 9) 2024-09-15T21:06:42.828 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:832: corrupt_and_repair_lrc: run_osd td/osd-scrub-repair 2 2024-09-15T21:06:42.828 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-scrub-repair 2024-09-15T21:06:42.828 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-15T21:06:42.828 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=2 2024-09-15T21:06:42.828 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-15T21:06:42.828 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-scrub-repair/2 2024-09-15T21:06:42.828 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=ce4892c7-6c6d-4b4f-8e6f-07a3e11133e5 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-15T21:06:42.828 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T21:06:42.828 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T21:06:42.828 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T21:06:42.828 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/2' 2024-09-15T21:06:42.828 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/2/journal' 2024-09-15T21:06:42.829 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-15T21:06:42.829 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-15T21:06:42.829 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-15T21:06:42.829 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-15T21:06:42.829 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-15T21:06:42.829 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-15T21:06:42.829 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-15T21:06:42.829 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T21:06:42.829 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52602 2024-09-15T21:06:42.829 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.52602/$cluster-$name.asok' 2024-09-15T21:06:42.829 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.52602/$cluster-$name.asok' 2024-09-15T21:06:42.829 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-15T21:06:42.829 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-15T21:06:42.829 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-15T21:06:42.830 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-15T21:06:42.830 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-15T21:06:42.830 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T21:06:42.830 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T21:06:42.830 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T21:06:42.830 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T21:06:42.830 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-15T21:06:42.830 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-15T21:06:42.830 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-scrub-repair/2 2024-09-15T21:06:42.830 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-15T21:06:42.831 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=3d1fb3bf-5850-4765-bf01-d8525dc804b0 2024-09-15T21:06:42.831 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd2 3d1fb3bf-5850-4765-bf01-d8525dc804b0' 2024-09-15T21:06:42.831 INFO:tasks.workunit.client.0.smithi136.stdout:add osd2 3d1fb3bf-5850-4765-bf01-d8525dc804b0 2024-09-15T21:06:42.831 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-15T21:06:42.847 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQBiTOdmvbZOMhAAurcFPvOW/SXzNNd7Zr0tFQ== 2024-09-15T21:06:42.847 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQBiTOdmvbZOMhAAurcFPvOW/SXzNNd7Zr0tFQ=="}' 2024-09-15T21:06:42.847 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 3d1fb3bf-5850-4765-bf01-d8525dc804b0 -i td/osd-scrub-repair/2/new.json 2024-09-15T21:06:43.288 INFO:tasks.workunit.client.0.smithi136.stdout:2 2024-09-15T21:06:43.299 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-scrub-repair/2/new.json 2024-09-15T21:06:43.300 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 2 --fsid=ce4892c7-6c6d-4b4f-8e6f-07a3e11133e5 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.52602/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBiTOdmvbZOMhAAurcFPvOW/SXzNNd7Zr0tFQ== --osd-uuid 3d1fb3bf-5850-4765-bf01-d8525dc804b0 2024-09-15T21:06:43.329 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:06:43.326+0000 7f7ead9bf640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:06:43.373 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:06:43.370+0000 7f7ead9bf640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:06:43.373 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:06:43.371+0000 7f7ead9bf640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:06:43.374 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:06:43.371+0000 7f7ead9bf640 -1 bdev(0x556db6d8dc00 td/osd-scrub-repair/2/block) open stat got: (1) Operation not permitted 2024-09-15T21:06:43.374 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:06:43.371+0000 7f7ead9bf640 -1 bluestore(td/osd-scrub-repair/2) _read_fsid unparsable uuid 2024-09-15T21:06:49.873 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-scrub-repair/2/keyring 2024-09-15T21:06:49.873 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-15T21:06:49.874 INFO:tasks.workunit.client.0.smithi136.stdout:adding osd2 key to auth repository 2024-09-15T21:06:49.874 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd2 key to auth repository 2024-09-15T21:06:49.874 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-scrub-repair/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-15T21:06:50.275 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.2 2024-09-15T21:06:50.275 INFO:tasks.workunit.client.0.smithi136.stdout:start osd.2 2024-09-15T21:06:50.276 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 2 --fsid=ce4892c7-6c6d-4b4f-8e6f-07a3e11133e5 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.52602/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T21:06:50.276 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-15T21:06:50.276 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-15T21:06:50.276 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-15T21:06:50.302 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:06:50.299+0000 7f0cb4b1b640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:06:50.304 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:06:50.301+0000 7f0cb4b1b640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:06:50.305 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:06:50.302+0000 7f0cb4b1b640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:06:50.551 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 2 2024-09-15T21:06:50.551 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-15T21:06:50.552 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=2 2024-09-15T21:06:50.552 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-15T21:06:50.552 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-15T21:06:50.552 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:06:50.552 INFO:tasks.workunit.client.0.smithi136.stdout:0 2024-09-15T21:06:50.552 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-15T21:06:50.553 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:06:50.553 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T21:06:50.827 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:06:51.828 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:06:51.829 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:06:51.829 INFO:tasks.workunit.client.0.smithi136.stdout:1 2024-09-15T21:06:51.829 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-15T21:06:51.829 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:06:51.829 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T21:06:51.940 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:06:51.937+0000 7f0cb4b1b640 -1 Falling back to public interface 2024-09-15T21:06:52.131 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:06:53.132 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:06:53.133 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:06:53.133 INFO:tasks.workunit.client.0.smithi136.stdout:2 2024-09-15T21:06:53.133 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-15T21:06:53.133 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:06:53.133 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T21:06:53.419 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:06:54.420 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:06:54.420 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:06:54.421 INFO:tasks.workunit.client.0.smithi136.stdout:3 2024-09-15T21:06:54.421 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-15T21:06:54.421 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:06:54.421 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T21:06:54.698 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:06:55.154 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:06:55.151+0000 7f0cb4b1b640 -1 osd.2 0 log_to_monitors true 2024-09-15T21:06:55.699 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:06:55.699 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:06:55.700 INFO:tasks.workunit.client.0.smithi136.stdout:4 2024-09-15T21:06:55.700 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-15T21:06:55.700 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:06:55.700 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T21:06:56.216 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:06:57.217 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:06:57.217 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:06:57.218 INFO:tasks.workunit.client.0.smithi136.stdout:5 2024-09-15T21:06:57.218 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-15T21:06:57.218 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:06:57.218 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T21:06:57.507 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:06:58.508 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:06:58.508 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:06:58.509 INFO:tasks.workunit.client.0.smithi136.stdout:6 2024-09-15T21:06:58.509 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-15T21:06:58.509 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:06:58.509 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T21:06:58.794 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:06:59.796 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:06:59.796 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:06:59.797 INFO:tasks.workunit.client.0.smithi136.stdout:7 2024-09-15T21:06:59.797 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-15T21:06:59.797 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:06:59.797 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T21:07:00.391 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:07:01.393 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:07:01.393 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:07:01.393 INFO:tasks.workunit.client.0.smithi136.stdout:8 2024-09-15T21:07:01.394 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-15T21:07:01.394 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:07:01.394 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T21:07:01.683 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:07:02.684 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:07:02.684 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:07:02.685 INFO:tasks.workunit.client.0.smithi136.stdout:9 2024-09-15T21:07:02.685 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-15T21:07:02.685 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:07:02.685 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T21:07:02.984 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:07:03.985 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:07:03.985 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:07:03.986 INFO:tasks.workunit.client.0.smithi136.stdout:10 2024-09-15T21:07:03.986 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-15T21:07:03.986 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:07:03.986 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T21:07:04.390 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:07:05.391 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:07:05.392 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:07:05.392 INFO:tasks.workunit.client.0.smithi136.stdout:11 2024-09-15T21:07:05.392 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-15T21:07:05.392 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:07:05.392 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T21:07:05.680 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:07:06.682 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:07:06.682 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:07:06.682 INFO:tasks.workunit.client.0.smithi136.stdout:12 2024-09-15T21:07:06.682 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-15T21:07:06.683 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:07:06.683 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T21:07:06.984 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:07:07.985 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:07:07.985 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:07:07.986 INFO:tasks.workunit.client.0.smithi136.stdout:13 2024-09-15T21:07:07.986 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-15T21:07:07.986 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:07:07.986 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T21:07:08.778 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:07:09.780 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:07:09.780 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:07:09.780 INFO:tasks.workunit.client.0.smithi136.stdout:14 2024-09-15T21:07:09.781 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-15T21:07:09.781 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:07:09.781 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T21:07:10.530 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:07:11.531 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:07:11.531 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:07:11.532 INFO:tasks.workunit.client.0.smithi136.stdout:15 2024-09-15T21:07:11.532 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-15T21:07:11.532 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:07:11.532 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T21:07:12.125 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:07:13.126 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:07:13.126 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:07:13.126 INFO:tasks.workunit.client.0.smithi136.stdout:16 2024-09-15T21:07:13.127 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-15T21:07:13.127 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:07:13.127 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T21:07:13.517 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:07:14.517 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:07:14.517 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:07:14.517 INFO:tasks.workunit.client.0.smithi136.stdout:17 2024-09-15T21:07:14.518 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-15T21:07:14.518 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:07:14.518 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T21:07:15.327 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:07:16.328 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:07:16.328 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:07:16.328 INFO:tasks.workunit.client.0.smithi136.stdout:18 2024-09-15T21:07:16.329 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 18 2024-09-15T21:07:16.329 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:07:16.329 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T21:07:17.137 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:07:18.138 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:07:18.139 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:07:18.139 INFO:tasks.workunit.client.0.smithi136.stdout:19 2024-09-15T21:07:18.139 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 19 2024-09-15T21:07:18.140 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:07:18.140 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T21:07:18.493 INFO:tasks.workunit.client.0.smithi136.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/2476202553,v1:127.0.0.1:6819/2476202553] [v2:127.0.0.1:6820/2476202553,v1:127.0.0.1:6821/2476202553] exists,up 3d1fb3bf-5850-4765-bf01-d8525dc804b0 2024-09-15T21:07:18.497 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-15T21:07:18.498 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-15T21:07:18.498 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-15T21:07:18.498 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:831: corrupt_and_repair_lrc: for id in $(seq 0 9) 2024-09-15T21:07:18.498 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:832: corrupt_and_repair_lrc: run_osd td/osd-scrub-repair 3 2024-09-15T21:07:18.498 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-scrub-repair 2024-09-15T21:07:18.498 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-15T21:07:18.498 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=3 2024-09-15T21:07:18.498 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-15T21:07:18.498 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-scrub-repair/3 2024-09-15T21:07:18.498 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=ce4892c7-6c6d-4b4f-8e6f-07a3e11133e5 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-15T21:07:18.498 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T21:07:18.498 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T21:07:18.498 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T21:07:18.498 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/3' 2024-09-15T21:07:18.499 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/3/journal' 2024-09-15T21:07:18.499 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-15T21:07:18.499 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-15T21:07:18.499 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-15T21:07:18.499 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-15T21:07:18.499 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-15T21:07:18.499 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-15T21:07:18.499 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-15T21:07:18.499 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T21:07:18.499 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52602 2024-09-15T21:07:18.499 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.52602/$cluster-$name.asok' 2024-09-15T21:07:18.499 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.52602/$cluster-$name.asok' 2024-09-15T21:07:18.499 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-15T21:07:18.500 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-15T21:07:18.500 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-15T21:07:18.500 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-15T21:07:18.500 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-15T21:07:18.500 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T21:07:18.500 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T21:07:18.500 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T21:07:18.500 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T21:07:18.500 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-15T21:07:18.500 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-15T21:07:18.500 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-scrub-repair/3 2024-09-15T21:07:18.500 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-15T21:07:18.501 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=0e7c66ad-8619-49df-8643-12288ef1d7ba 2024-09-15T21:07:18.501 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd3 0e7c66ad-8619-49df-8643-12288ef1d7ba' 2024-09-15T21:07:18.501 INFO:tasks.workunit.client.0.smithi136.stdout:add osd3 0e7c66ad-8619-49df-8643-12288ef1d7ba 2024-09-15T21:07:18.501 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-15T21:07:18.516 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQCGTOdm+HGeHhAA+k0cbMFfUMGfqJOLseb5vg== 2024-09-15T21:07:18.516 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQCGTOdm+HGeHhAA+k0cbMFfUMGfqJOLseb5vg=="}' 2024-09-15T21:07:18.516 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 0e7c66ad-8619-49df-8643-12288ef1d7ba -i td/osd-scrub-repair/3/new.json 2024-09-15T21:07:18.848 INFO:tasks.workunit.client.0.smithi136.stdout:3 2024-09-15T21:07:18.859 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-scrub-repair/3/new.json 2024-09-15T21:07:18.860 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 3 --fsid=ce4892c7-6c6d-4b4f-8e6f-07a3e11133e5 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/3 --osd-journal=td/osd-scrub-repair/3/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.52602/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCGTOdm+HGeHhAA+k0cbMFfUMGfqJOLseb5vg== --osd-uuid 0e7c66ad-8619-49df-8643-12288ef1d7ba 2024-09-15T21:07:18.889 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:07:18.886+0000 7fecbdca5640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:07:18.924 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:07:18.921+0000 7fecbdca5640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:07:18.924 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:07:18.922+0000 7fecbdca5640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:07:18.925 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:07:18.922+0000 7fecbdca5640 -1 bdev(0x55ded3c13c00 td/osd-scrub-repair/3/block) open stat got: (1) Operation not permitted 2024-09-15T21:07:18.925 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:07:18.922+0000 7fecbdca5640 -1 bluestore(td/osd-scrub-repair/3) _read_fsid unparsable uuid 2024-09-15T21:07:25.437 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-scrub-repair/3/keyring 2024-09-15T21:07:25.437 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-15T21:07:25.438 INFO:tasks.workunit.client.0.smithi136.stdout:adding osd3 key to auth repository 2024-09-15T21:07:25.439 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd3 key to auth repository 2024-09-15T21:07:25.439 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-scrub-repair/3/keyring auth add osd.3 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-15T21:07:25.798 INFO:tasks.workunit.client.0.smithi136.stdout:start osd.3 2024-09-15T21:07:25.799 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.3 2024-09-15T21:07:25.799 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 3 --fsid=ce4892c7-6c6d-4b4f-8e6f-07a3e11133e5 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/3 --osd-journal=td/osd-scrub-repair/3/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.52602/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T21:07:25.799 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-15T21:07:25.799 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-15T21:07:25.799 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-15T21:07:25.825 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:07:25.822+0000 7fcd8f619640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:07:25.827 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:07:25.824+0000 7fcd8f619640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:07:25.828 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:07:25.825+0000 7fcd8f619640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:07:26.077 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 3 2024-09-15T21:07:26.078 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-15T21:07:26.078 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=3 2024-09-15T21:07:26.078 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-15T21:07:26.078 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-15T21:07:26.078 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:07:26.078 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-15T21:07:26.078 INFO:tasks.workunit.client.0.smithi136.stdout:0 2024-09-15T21:07:26.078 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:07:26.079 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-15T21:07:26.368 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:07:27.197 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:07:27.194+0000 7fcd8f619640 -1 Falling back to public interface 2024-09-15T21:07:27.369 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:07:27.369 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:07:27.369 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-15T21:07:27.370 INFO:tasks.workunit.client.0.smithi136.stdout:1 2024-09-15T21:07:27.370 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:07:27.370 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-15T21:07:27.658 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:07:28.660 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:07:28.660 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:07:28.660 INFO:tasks.workunit.client.0.smithi136.stdout:2 2024-09-15T21:07:28.661 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-15T21:07:28.661 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:07:28.661 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-15T21:07:28.938 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:07:29.939 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:07:29.939 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:07:29.939 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-15T21:07:29.940 INFO:tasks.workunit.client.0.smithi136.stdout:3 2024-09-15T21:07:29.940 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:07:29.940 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-15T21:07:30.225 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:07:30.233 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:07:30.230+0000 7fcd8f619640 -1 osd.3 0 log_to_monitors true 2024-09-15T21:07:31.226 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:07:31.227 INFO:tasks.workunit.client.0.smithi136.stdout:4 2024-09-15T21:07:31.227 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:07:31.227 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-15T21:07:31.227 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:07:31.227 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-15T21:07:31.508 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:07:32.509 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:07:32.509 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:07:32.509 INFO:tasks.workunit.client.0.smithi136.stdout:5 2024-09-15T21:07:32.510 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-15T21:07:32.510 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:07:32.510 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-15T21:07:32.877 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:07:33.878 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:07:33.878 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:07:33.879 INFO:tasks.workunit.client.0.smithi136.stdout:6 2024-09-15T21:07:33.879 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-15T21:07:33.879 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:07:33.879 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-15T21:07:34.313 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:07:35.314 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:07:35.314 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:07:35.314 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-15T21:07:35.314 INFO:tasks.workunit.client.0.smithi136.stdout:7 2024-09-15T21:07:35.315 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:07:35.315 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-15T21:07:35.601 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:07:36.602 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:07:36.602 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:07:36.602 INFO:tasks.workunit.client.0.smithi136.stdout:8 2024-09-15T21:07:36.603 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-15T21:07:36.603 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:07:36.603 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-15T21:07:36.891 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:07:37.892 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:07:37.892 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:07:37.892 INFO:tasks.workunit.client.0.smithi136.stdout:9 2024-09-15T21:07:37.893 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-15T21:07:37.893 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:07:37.893 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-15T21:07:38.336 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:07:39.338 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:07:39.338 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:07:39.338 INFO:tasks.workunit.client.0.smithi136.stdout:10 2024-09-15T21:07:39.339 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-15T21:07:39.339 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:07:39.339 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-15T21:07:39.627 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:07:40.629 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:07:40.629 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:07:40.629 INFO:tasks.workunit.client.0.smithi136.stdout:11 2024-09-15T21:07:40.630 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-15T21:07:40.630 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:07:40.630 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-15T21:07:40.919 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:07:41.920 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:07:41.920 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:07:41.920 INFO:tasks.workunit.client.0.smithi136.stdout:12 2024-09-15T21:07:41.921 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-15T21:07:41.921 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:07:41.921 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-15T21:07:42.241 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:07:43.242 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:07:43.243 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:07:43.243 INFO:tasks.workunit.client.0.smithi136.stdout:13 2024-09-15T21:07:43.243 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-15T21:07:43.243 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:07:43.243 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-15T21:07:43.737 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:07:44.739 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:07:44.739 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:07:44.739 INFO:tasks.workunit.client.0.smithi136.stdout:14 2024-09-15T21:07:44.739 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-15T21:07:44.740 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:07:44.740 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-15T21:07:45.120 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:07:46.122 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:07:46.122 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:07:46.122 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-15T21:07:46.123 INFO:tasks.workunit.client.0.smithi136.stdout:15 2024-09-15T21:07:46.123 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:07:46.123 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-15T21:07:46.748 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:07:47.750 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:07:47.750 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:07:47.751 INFO:tasks.workunit.client.0.smithi136.stdout:16 2024-09-15T21:07:47.751 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-15T21:07:47.752 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:07:47.752 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-15T21:07:48.251 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:07:49.253 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:07:49.253 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:07:49.253 INFO:tasks.workunit.client.0.smithi136.stdout:17 2024-09-15T21:07:49.254 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-15T21:07:49.254 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:07:49.254 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-15T21:07:49.800 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:07:50.801 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:07:50.801 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:07:50.802 INFO:tasks.workunit.client.0.smithi136.stdout:18 2024-09-15T21:07:50.802 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 18 2024-09-15T21:07:50.802 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:07:50.802 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-15T21:07:51.551 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:07:52.553 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:07:52.553 INFO:tasks.workunit.client.0.smithi136.stdout:19 2024-09-15T21:07:52.554 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:07:52.554 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 19 2024-09-15T21:07:52.554 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:07:52.554 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-15T21:07:53.037 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:07:54.039 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:07:54.039 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:07:54.039 INFO:tasks.workunit.client.0.smithi136.stdout:20 2024-09-15T21:07:54.040 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 20 2024-09-15T21:07:54.040 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:07:54.040 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-15T21:07:54.347 INFO:tasks.workunit.client.0.smithi136.stdout:osd.3 up in weight 1 up_from 20 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6826/1622627228,v1:127.0.0.1:6827/1622627228] [v2:127.0.0.1:6828/1622627228,v1:127.0.0.1:6829/1622627228] exists,up 0e7c66ad-8619-49df-8643-12288ef1d7ba 2024-09-15T21:07:54.353 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-15T21:07:54.353 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-15T21:07:54.354 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-15T21:07:54.354 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:831: corrupt_and_repair_lrc: for id in $(seq 0 9) 2024-09-15T21:07:54.354 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:832: corrupt_and_repair_lrc: run_osd td/osd-scrub-repair 4 2024-09-15T21:07:54.354 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-scrub-repair 2024-09-15T21:07:54.354 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-15T21:07:54.354 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=4 2024-09-15T21:07:54.354 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-15T21:07:54.354 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-scrub-repair/4 2024-09-15T21:07:54.354 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=ce4892c7-6c6d-4b4f-8e6f-07a3e11133e5 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-15T21:07:54.354 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T21:07:54.354 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T21:07:54.354 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T21:07:54.354 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/4' 2024-09-15T21:07:54.355 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/4/journal' 2024-09-15T21:07:54.355 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-15T21:07:54.355 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-15T21:07:54.355 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-15T21:07:54.355 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-15T21:07:54.355 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-15T21:07:54.355 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-15T21:07:54.355 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-15T21:07:54.355 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T21:07:54.355 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52602 2024-09-15T21:07:54.355 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.52602/$cluster-$name.asok' 2024-09-15T21:07:54.355 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.52602/$cluster-$name.asok' 2024-09-15T21:07:54.355 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-15T21:07:54.356 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-15T21:07:54.356 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-15T21:07:54.356 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-15T21:07:54.356 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-15T21:07:54.356 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T21:07:54.356 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T21:07:54.356 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T21:07:54.356 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T21:07:54.356 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-15T21:07:54.356 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-15T21:07:54.356 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-scrub-repair/4 2024-09-15T21:07:54.356 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-15T21:07:54.356 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=caebdd7b-014a-4672-ac7e-93b3d444bca5 2024-09-15T21:07:54.356 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd4 caebdd7b-014a-4672-ac7e-93b3d444bca5' 2024-09-15T21:07:54.357 INFO:tasks.workunit.client.0.smithi136.stdout:add osd4 caebdd7b-014a-4672-ac7e-93b3d444bca5 2024-09-15T21:07:54.357 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-15T21:07:54.370 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQCqTOdm61TkFRAA7HiYoSqG7zjlPllzlxkQLg== 2024-09-15T21:07:54.370 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQCqTOdm61TkFRAA7HiYoSqG7zjlPllzlxkQLg=="}' 2024-09-15T21:07:54.370 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new caebdd7b-014a-4672-ac7e-93b3d444bca5 -i td/osd-scrub-repair/4/new.json 2024-09-15T21:07:54.788 INFO:tasks.workunit.client.0.smithi136.stdout:4 2024-09-15T21:07:54.799 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-scrub-repair/4/new.json 2024-09-15T21:07:54.800 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 4 --fsid=ce4892c7-6c6d-4b4f-8e6f-07a3e11133e5 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/4 --osd-journal=td/osd-scrub-repair/4/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.52602/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCqTOdm61TkFRAA7HiYoSqG7zjlPllzlxkQLg== --osd-uuid caebdd7b-014a-4672-ac7e-93b3d444bca5 2024-09-15T21:07:54.830 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:07:54.827+0000 7fc3672b0640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:07:54.889 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:07:54.886+0000 7fc3672b0640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:07:54.890 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:07:54.887+0000 7fc3672b0640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:07:54.891 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:07:54.888+0000 7fc3672b0640 -1 bdev(0x55598e9e7c00 td/osd-scrub-repair/4/block) open stat got: (1) Operation not permitted 2024-09-15T21:07:54.891 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:07:54.888+0000 7fc3672b0640 -1 bluestore(td/osd-scrub-repair/4) _read_fsid unparsable uuid 2024-09-15T21:08:01.317 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-scrub-repair/4/keyring 2024-09-15T21:08:01.318 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-15T21:08:01.318 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd4 key to auth repository 2024-09-15T21:08:01.319 INFO:tasks.workunit.client.0.smithi136.stdout:adding osd4 key to auth repository 2024-09-15T21:08:01.319 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-scrub-repair/4/keyring auth add osd.4 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-15T21:08:01.677 INFO:tasks.workunit.client.0.smithi136.stdout:start osd.4 2024-09-15T21:08:01.677 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.4 2024-09-15T21:08:01.678 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 4 --fsid=ce4892c7-6c6d-4b4f-8e6f-07a3e11133e5 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/4 --osd-journal=td/osd-scrub-repair/4/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.52602/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T21:08:01.678 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-15T21:08:01.678 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-15T21:08:01.678 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-15T21:08:01.704 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:08:01.700+0000 7fce02bb2640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:08:01.706 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:08:01.702+0000 7fce02bb2640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:08:01.707 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:08:01.703+0000 7fce02bb2640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:08:01.958 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 4 2024-09-15T21:08:01.958 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-15T21:08:01.958 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=4 2024-09-15T21:08:01.958 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-15T21:08:01.958 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-15T21:08:01.958 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:08:01.958 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-15T21:08:01.959 INFO:tasks.workunit.client.0.smithi136.stdout:0 2024-09-15T21:08:01.959 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:08:01.959 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-15T21:08:02.236 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:08:03.237 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:08:03.237 INFO:tasks.workunit.client.0.smithi136.stdout:1 2024-09-15T21:08:03.238 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:08:03.238 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-15T21:08:03.238 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:08:03.238 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-15T21:08:03.336 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:08:03.333+0000 7fce02bb2640 -1 Falling back to public interface 2024-09-15T21:08:03.523 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:08:04.524 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:08:04.524 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:08:04.524 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-15T21:08:04.525 INFO:tasks.workunit.client.0.smithi136.stdout:2 2024-09-15T21:08:04.525 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:08:04.525 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-15T21:08:04.813 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:08:05.814 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:08:05.814 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:08:05.814 INFO:tasks.workunit.client.0.smithi136.stdout:3 2024-09-15T21:08:05.815 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-15T21:08:05.815 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:08:05.815 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-15T21:08:06.172 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:08:06.491 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:08:06.488+0000 7fce02bb2640 -1 osd.4 0 log_to_monitors true 2024-09-15T21:08:07.174 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:08:07.174 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:08:07.174 INFO:tasks.workunit.client.0.smithi136.stdout:4 2024-09-15T21:08:07.175 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-15T21:08:07.175 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:08:07.175 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-15T21:08:07.588 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:08:08.589 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:08:08.590 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:08:08.590 INFO:tasks.workunit.client.0.smithi136.stdout:5 2024-09-15T21:08:08.590 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-15T21:08:08.591 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:08:08.591 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-15T21:08:08.962 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:08:09.963 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:08:09.963 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:08:09.963 INFO:tasks.workunit.client.0.smithi136.stdout:6 2024-09-15T21:08:09.964 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-15T21:08:09.964 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:08:09.964 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-15T21:08:10.300 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:08:11.301 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:08:11.301 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:08:11.302 INFO:tasks.workunit.client.0.smithi136.stdout:7 2024-09-15T21:08:11.302 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-15T21:08:11.302 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:08:11.302 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-15T21:08:11.592 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:08:12.594 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:08:12.594 INFO:tasks.workunit.client.0.smithi136.stdout:8 2024-09-15T21:08:12.595 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:08:12.595 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-15T21:08:12.595 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:08:12.595 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-15T21:08:12.890 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:08:13.891 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:08:13.891 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:08:13.892 INFO:tasks.workunit.client.0.smithi136.stdout:9 2024-09-15T21:08:13.892 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-15T21:08:13.892 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:08:13.892 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-15T21:08:14.180 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:08:15.181 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:08:15.182 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:08:15.182 INFO:tasks.workunit.client.0.smithi136.stdout:10 2024-09-15T21:08:15.182 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-15T21:08:15.182 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:08:15.182 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-15T21:08:15.554 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:08:16.556 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:08:16.556 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:08:16.556 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-15T21:08:16.557 INFO:tasks.workunit.client.0.smithi136.stdout:11 2024-09-15T21:08:16.558 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:08:16.558 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-15T21:08:16.869 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:08:17.870 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:08:17.871 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:08:17.871 INFO:tasks.workunit.client.0.smithi136.stdout:12 2024-09-15T21:08:17.872 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-15T21:08:17.872 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:08:17.872 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-15T21:08:18.164 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:08:19.165 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:08:19.166 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:08:19.166 INFO:tasks.workunit.client.0.smithi136.stdout:13 2024-09-15T21:08:19.166 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-15T21:08:19.166 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:08:19.166 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-15T21:08:19.453 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:08:20.455 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:08:20.455 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:08:20.455 INFO:tasks.workunit.client.0.smithi136.stdout:14 2024-09-15T21:08:20.455 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-15T21:08:20.456 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:08:20.456 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-15T21:08:21.031 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:08:22.032 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:08:22.032 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:08:22.032 INFO:tasks.workunit.client.0.smithi136.stdout:15 2024-09-15T21:08:22.033 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-15T21:08:22.033 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:08:22.033 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-15T21:08:22.661 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:08:23.662 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:08:23.663 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:08:23.663 INFO:tasks.workunit.client.0.smithi136.stdout:16 2024-09-15T21:08:23.664 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-15T21:08:23.664 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:08:23.664 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-15T21:08:24.336 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:08:25.338 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:08:25.338 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:08:25.338 INFO:tasks.workunit.client.0.smithi136.stdout:17 2024-09-15T21:08:25.338 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-15T21:08:25.339 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:08:25.339 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-15T21:08:25.967 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:08:26.969 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:08:26.969 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:08:26.969 INFO:tasks.workunit.client.0.smithi136.stdout:18 2024-09-15T21:08:26.969 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 18 2024-09-15T21:08:26.970 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:08:26.970 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-15T21:08:27.401 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:08:28.403 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:08:28.403 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:08:28.403 INFO:tasks.workunit.client.0.smithi136.stdout:19 2024-09-15T21:08:28.403 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 19 2024-09-15T21:08:28.404 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:08:28.404 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-15T21:08:28.867 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:08:29.868 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:08:29.869 INFO:tasks.workunit.client.0.smithi136.stdout:20 2024-09-15T21:08:29.869 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:08:29.869 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 20 2024-09-15T21:08:29.869 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:08:29.870 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-15T21:08:30.249 INFO:tasks.workunit.client.0.smithi136.stdout:osd.4 up in weight 1 up_from 25 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6834/103795431,v1:127.0.0.1:6835/103795431] [v2:127.0.0.1:6836/103795431,v1:127.0.0.1:6837/103795431] exists,up caebdd7b-014a-4672-ac7e-93b3d444bca5 2024-09-15T21:08:30.253 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-15T21:08:30.253 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-15T21:08:30.253 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-15T21:08:30.253 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:831: corrupt_and_repair_lrc: for id in $(seq 0 9) 2024-09-15T21:08:30.253 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:832: corrupt_and_repair_lrc: run_osd td/osd-scrub-repair 5 2024-09-15T21:08:30.253 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-scrub-repair 2024-09-15T21:08:30.253 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-15T21:08:30.253 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=5 2024-09-15T21:08:30.253 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-15T21:08:30.253 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-scrub-repair/5 2024-09-15T21:08:30.253 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=ce4892c7-6c6d-4b4f-8e6f-07a3e11133e5 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-15T21:08:30.253 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T21:08:30.254 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T21:08:30.254 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T21:08:30.254 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/5' 2024-09-15T21:08:30.254 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/5/journal' 2024-09-15T21:08:30.254 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-15T21:08:30.254 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-15T21:08:30.254 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-15T21:08:30.254 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-15T21:08:30.254 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-15T21:08:30.254 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-15T21:08:30.254 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-15T21:08:30.254 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T21:08:30.254 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52602 2024-09-15T21:08:30.254 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.52602/$cluster-$name.asok' 2024-09-15T21:08:30.255 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.52602/$cluster-$name.asok' 2024-09-15T21:08:30.255 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-15T21:08:30.255 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-15T21:08:30.255 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-15T21:08:30.255 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-15T21:08:30.255 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-15T21:08:30.255 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T21:08:30.255 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T21:08:30.255 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T21:08:30.255 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T21:08:30.255 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-15T21:08:30.255 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-15T21:08:30.255 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-scrub-repair/5 2024-09-15T21:08:30.255 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-15T21:08:30.256 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=9918af07-0bbe-4711-bea1-402a4b5b6247 2024-09-15T21:08:30.256 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd5 9918af07-0bbe-4711-bea1-402a4b5b6247' 2024-09-15T21:08:30.256 INFO:tasks.workunit.client.0.smithi136.stdout:add osd5 9918af07-0bbe-4711-bea1-402a4b5b6247 2024-09-15T21:08:30.256 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-15T21:08:30.273 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQDOTOdm1yAdEBAA3Z43AM4cbHSK/OmLAeQCHQ== 2024-09-15T21:08:30.273 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQDOTOdm1yAdEBAA3Z43AM4cbHSK/OmLAeQCHQ=="}' 2024-09-15T21:08:30.273 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 9918af07-0bbe-4711-bea1-402a4b5b6247 -i td/osd-scrub-repair/5/new.json 2024-09-15T21:08:30.724 INFO:tasks.workunit.client.0.smithi136.stdout:5 2024-09-15T21:08:30.734 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-scrub-repair/5/new.json 2024-09-15T21:08:30.735 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 5 --fsid=ce4892c7-6c6d-4b4f-8e6f-07a3e11133e5 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/5 --osd-journal=td/osd-scrub-repair/5/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.52602/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDOTOdm1yAdEBAA3Z43AM4cbHSK/OmLAeQCHQ== --osd-uuid 9918af07-0bbe-4711-bea1-402a4b5b6247 2024-09-15T21:08:30.764 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:08:30.761+0000 7f61ee41e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:08:30.815 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:08:30.812+0000 7f61ee41e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:08:30.816 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:08:30.813+0000 7f61ee41e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:08:30.817 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:08:30.814+0000 7f61ee41e640 -1 bdev(0x55d6f3cbe800 td/osd-scrub-repair/5/block) open stat got: (1) Operation not permitted 2024-09-15T21:08:30.817 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:08:30.814+0000 7f61ee41e640 -1 bluestore(td/osd-scrub-repair/5) _read_fsid unparsable uuid 2024-09-15T21:08:37.155 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-scrub-repair/5/keyring 2024-09-15T21:08:37.155 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-15T21:08:37.156 INFO:tasks.workunit.client.0.smithi136.stdout:adding osd5 key to auth repository 2024-09-15T21:08:37.157 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd5 key to auth repository 2024-09-15T21:08:37.157 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-scrub-repair/5/keyring auth add osd.5 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-15T21:08:37.518 INFO:tasks.workunit.client.0.smithi136.stdout:start osd.5 2024-09-15T21:08:37.519 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.5 2024-09-15T21:08:37.519 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 5 --fsid=ce4892c7-6c6d-4b4f-8e6f-07a3e11133e5 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/5 --osd-journal=td/osd-scrub-repair/5/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.52602/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T21:08:37.519 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-15T21:08:37.519 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-15T21:08:37.519 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-15T21:08:37.545 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:08:37.542+0000 7f9c90f23640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:08:37.547 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:08:37.544+0000 7f9c90f23640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:08:37.548 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:08:37.546+0000 7f9c90f23640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:08:37.798 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 5 2024-09-15T21:08:37.798 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-15T21:08:37.798 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=5 2024-09-15T21:08:37.798 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-15T21:08:37.798 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-15T21:08:37.798 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:08:37.798 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-15T21:08:37.799 INFO:tasks.workunit.client.0.smithi136.stdout:0 2024-09-15T21:08:37.799 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:08:37.799 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-15T21:08:38.074 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:08:39.075 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:08:39.076 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:08:39.076 INFO:tasks.workunit.client.0.smithi136.stdout:1 2024-09-15T21:08:39.077 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-15T21:08:39.077 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:08:39.077 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-15T21:08:39.164 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:08:39.161+0000 7f9c90f23640 -1 Falling back to public interface 2024-09-15T21:08:39.364 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:08:40.365 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:08:40.365 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:08:40.365 INFO:tasks.workunit.client.0.smithi136.stdout:2 2024-09-15T21:08:40.366 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-15T21:08:40.366 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:08:40.366 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-15T21:08:40.651 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:08:41.652 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:08:41.652 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:08:41.652 INFO:tasks.workunit.client.0.smithi136.stdout:3 2024-09-15T21:08:41.653 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-15T21:08:41.653 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:08:41.653 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-15T21:08:41.811 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:08:41.808+0000 7f9c90f23640 -1 osd.5 0 log_to_monitors true 2024-09-15T21:08:41.941 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:08:42.945 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:08:42.945 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:08:42.945 INFO:tasks.workunit.client.0.smithi136.stdout:4 2024-09-15T21:08:42.946 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-15T21:08:42.946 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:08:42.946 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-15T21:08:43.229 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:08:44.231 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:08:44.231 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:08:44.231 INFO:tasks.workunit.client.0.smithi136.stdout:5 2024-09-15T21:08:44.231 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-15T21:08:44.232 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:08:44.232 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-15T21:08:44.532 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:08:45.533 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:08:45.534 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:08:45.534 INFO:tasks.workunit.client.0.smithi136.stdout:6 2024-09-15T21:08:45.534 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-15T21:08:45.534 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:08:45.534 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-15T21:08:46.000 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:08:47.001 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:08:47.002 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:08:47.002 INFO:tasks.workunit.client.0.smithi136.stdout:7 2024-09-15T21:08:47.002 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-15T21:08:47.003 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:08:47.003 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-15T21:08:47.293 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:08:48.294 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:08:48.294 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:08:48.294 INFO:tasks.workunit.client.0.smithi136.stdout:8 2024-09-15T21:08:48.295 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-15T21:08:48.295 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:08:48.295 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-15T21:08:48.710 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:08:49.712 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:08:49.712 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:08:49.712 INFO:tasks.workunit.client.0.smithi136.stdout:9 2024-09-15T21:08:49.712 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-15T21:08:49.713 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:08:49.713 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-15T21:08:50.000 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:08:51.000 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:08:51.000 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:08:51.000 INFO:tasks.workunit.client.0.smithi136.stdout:10 2024-09-15T21:08:51.001 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-15T21:08:51.001 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:08:51.001 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-15T21:08:51.290 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:08:52.292 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:08:52.292 INFO:tasks.workunit.client.0.smithi136.stdout:11 2024-09-15T21:08:52.293 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:08:52.293 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-15T21:08:52.293 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:08:52.293 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-15T21:08:52.629 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:08:53.630 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:08:53.630 INFO:tasks.workunit.client.0.smithi136.stdout:12 2024-09-15T21:08:53.631 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:08:53.631 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-15T21:08:53.631 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:08:53.631 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-15T21:08:54.085 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:08:55.087 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:08:55.087 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:08:55.087 INFO:tasks.workunit.client.0.smithi136.stdout:13 2024-09-15T21:08:55.088 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-15T21:08:55.088 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:08:55.088 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-15T21:08:55.646 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:08:56.647 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:08:56.647 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:08:56.648 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-15T21:08:56.648 INFO:tasks.workunit.client.0.smithi136.stdout:14 2024-09-15T21:08:56.648 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:08:56.648 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-15T21:08:57.240 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:08:58.242 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:08:58.242 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:08:58.242 INFO:tasks.workunit.client.0.smithi136.stdout:15 2024-09-15T21:08:58.243 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-15T21:08:58.243 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:08:58.243 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-15T21:08:58.992 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:08:59.994 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:08:59.994 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:08:59.994 INFO:tasks.workunit.client.0.smithi136.stdout:16 2024-09-15T21:08:59.994 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-15T21:08:59.995 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:08:59.995 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-15T21:09:00.803 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:09:01.804 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:09:01.804 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:09:01.805 INFO:tasks.workunit.client.0.smithi136.stdout:17 2024-09-15T21:09:01.805 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-15T21:09:01.805 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:09:01.805 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-15T21:09:02.213 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:09:03.215 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:09:03.215 INFO:tasks.workunit.client.0.smithi136.stdout:18 2024-09-15T21:09:03.216 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:09:03.216 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 18 2024-09-15T21:09:03.216 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:09:03.216 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-15T21:09:03.877 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:09:04.879 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:09:04.879 INFO:tasks.workunit.client.0.smithi136.stdout:19 2024-09-15T21:09:04.879 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:09:04.880 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 19 2024-09-15T21:09:04.880 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:09:04.880 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-15T21:09:05.276 INFO:tasks.workunit.client.0.smithi136.stdout:osd.5 up in weight 1 up_from 30 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6842/622593986,v1:127.0.0.1:6843/622593986] [v2:127.0.0.1:6844/622593986,v1:127.0.0.1:6845/622593986] exists,up 9918af07-0bbe-4711-bea1-402a4b5b6247 2024-09-15T21:09:05.280 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-15T21:09:05.280 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-15T21:09:05.280 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-15T21:09:05.280 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:831: corrupt_and_repair_lrc: for id in $(seq 0 9) 2024-09-15T21:09:05.280 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:832: corrupt_and_repair_lrc: run_osd td/osd-scrub-repair 6 2024-09-15T21:09:05.280 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-scrub-repair 2024-09-15T21:09:05.281 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-15T21:09:05.281 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=6 2024-09-15T21:09:05.281 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-15T21:09:05.281 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-scrub-repair/6 2024-09-15T21:09:05.281 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=ce4892c7-6c6d-4b4f-8e6f-07a3e11133e5 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-15T21:09:05.281 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T21:09:05.281 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T21:09:05.281 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T21:09:05.281 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/6' 2024-09-15T21:09:05.281 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/6/journal' 2024-09-15T21:09:05.281 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-15T21:09:05.281 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-15T21:09:05.282 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-15T21:09:05.282 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-15T21:09:05.282 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-15T21:09:05.282 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-15T21:09:05.282 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-15T21:09:05.282 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T21:09:05.282 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52602 2024-09-15T21:09:05.282 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.52602/$cluster-$name.asok' 2024-09-15T21:09:05.282 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.52602/$cluster-$name.asok' 2024-09-15T21:09:05.282 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-15T21:09:05.282 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-15T21:09:05.282 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-15T21:09:05.283 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-15T21:09:05.283 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-15T21:09:05.283 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T21:09:05.283 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T21:09:05.283 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T21:09:05.283 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T21:09:05.283 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-15T21:09:05.283 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-15T21:09:05.283 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-scrub-repair/6 2024-09-15T21:09:05.283 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-15T21:09:05.283 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=be621ea8-c8ad-40e0-ad1c-5e029e812c9d 2024-09-15T21:09:05.283 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd6 be621ea8-c8ad-40e0-ad1c-5e029e812c9d' 2024-09-15T21:09:05.284 INFO:tasks.workunit.client.0.smithi136.stdout:add osd6 be621ea8-c8ad-40e0-ad1c-5e029e812c9d 2024-09-15T21:09:05.284 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-15T21:09:05.302 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQDxTOdm6JucERAAvsl2KulbL8nDjiSG2k3pKQ== 2024-09-15T21:09:05.303 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQDxTOdm6JucERAAvsl2KulbL8nDjiSG2k3pKQ=="}' 2024-09-15T21:09:05.303 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new be621ea8-c8ad-40e0-ad1c-5e029e812c9d -i td/osd-scrub-repair/6/new.json 2024-09-15T21:09:05.644 INFO:tasks.workunit.client.0.smithi136.stdout:6 2024-09-15T21:09:05.655 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-scrub-repair/6/new.json 2024-09-15T21:09:05.656 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 6 --fsid=ce4892c7-6c6d-4b4f-8e6f-07a3e11133e5 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/6 --osd-journal=td/osd-scrub-repair/6/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.52602/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDxTOdm6JucERAAvsl2KulbL8nDjiSG2k3pKQ== --osd-uuid be621ea8-c8ad-40e0-ad1c-5e029e812c9d 2024-09-15T21:09:05.686 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:09:05.683+0000 7f5366aa5640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:09:05.737 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:09:05.735+0000 7f5366aa5640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:09:05.738 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:09:05.736+0000 7f5366aa5640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:09:05.739 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:09:05.736+0000 7f5366aa5640 -1 bdev(0x55a94f098800 td/osd-scrub-repair/6/block) open stat got: (1) Operation not permitted 2024-09-15T21:09:05.739 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:09:05.736+0000 7f5366aa5640 -1 bluestore(td/osd-scrub-repair/6) _read_fsid unparsable uuid 2024-09-15T21:09:11.748 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-scrub-repair/6/keyring 2024-09-15T21:09:11.749 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-15T21:09:11.749 INFO:tasks.workunit.client.0.smithi136.stdout:adding osd6 key to auth repository 2024-09-15T21:09:11.750 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd6 key to auth repository 2024-09-15T21:09:11.750 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-scrub-repair/6/keyring auth add osd.6 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-15T21:09:12.111 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.6 2024-09-15T21:09:12.111 INFO:tasks.workunit.client.0.smithi136.stdout:start osd.6 2024-09-15T21:09:12.112 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 6 --fsid=ce4892c7-6c6d-4b4f-8e6f-07a3e11133e5 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/6 --osd-journal=td/osd-scrub-repair/6/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.52602/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T21:09:12.112 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-15T21:09:12.113 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-15T21:09:12.113 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-15T21:09:12.138 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:09:12.135+0000 7f396d220640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:09:12.140 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:09:12.137+0000 7f396d220640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:09:12.141 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:09:12.138+0000 7f396d220640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:09:12.390 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 6 2024-09-15T21:09:12.390 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-15T21:09:12.390 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=6 2024-09-15T21:09:12.390 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-15T21:09:12.390 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-15T21:09:12.390 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:09:12.390 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-15T21:09:12.391 INFO:tasks.workunit.client.0.smithi136.stdout:0 2024-09-15T21:09:12.391 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:09:12.391 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.6 up' 2024-09-15T21:09:12.667 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:09:13.668 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:09:13.668 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:09:13.668 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-15T21:09:13.669 INFO:tasks.workunit.client.0.smithi136.stdout:1 2024-09-15T21:09:13.669 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:09:13.669 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.6 up' 2024-09-15T21:09:13.753 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:09:13.750+0000 7f396d220640 -1 Falling back to public interface 2024-09-15T21:09:13.956 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:09:14.957 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:09:14.957 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:09:14.958 INFO:tasks.workunit.client.0.smithi136.stdout:2 2024-09-15T21:09:14.958 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-15T21:09:14.958 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:09:14.958 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.6 up' 2024-09-15T21:09:15.248 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:09:16.249 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:09:16.249 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:09:16.250 INFO:tasks.workunit.client.0.smithi136.stdout:3 2024-09-15T21:09:16.251 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-15T21:09:16.251 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:09:16.251 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.6 up' 2024-09-15T21:09:16.539 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:09:16.709 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:09:16.706+0000 7f396d220640 -1 osd.6 0 log_to_monitors true 2024-09-15T21:09:17.540 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:09:17.540 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:09:17.541 INFO:tasks.workunit.client.0.smithi136.stdout:4 2024-09-15T21:09:17.541 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-15T21:09:17.541 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:09:17.541 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.6 up' 2024-09-15T21:09:17.834 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:09:18.835 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:09:18.835 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:09:18.835 INFO:tasks.workunit.client.0.smithi136.stdout:5 2024-09-15T21:09:18.836 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-15T21:09:18.836 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:09:18.836 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.6 up' 2024-09-15T21:09:19.125 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:09:20.126 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:09:20.126 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:09:20.127 INFO:tasks.workunit.client.0.smithi136.stdout:6 2024-09-15T21:09:20.127 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-15T21:09:20.127 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:09:20.127 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.6 up' 2024-09-15T21:09:20.524 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:09:21.525 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:09:21.525 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:09:21.526 INFO:tasks.workunit.client.0.smithi136.stdout:7 2024-09-15T21:09:21.526 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-15T21:09:21.526 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:09:21.526 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.6 up' 2024-09-15T21:09:21.813 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:09:22.814 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:09:22.814 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:09:22.815 INFO:tasks.workunit.client.0.smithi136.stdout:8 2024-09-15T21:09:22.815 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-15T21:09:22.815 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:09:22.815 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.6 up' 2024-09-15T21:09:23.105 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:09:24.107 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:09:24.107 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:09:24.107 INFO:tasks.workunit.client.0.smithi136.stdout:9 2024-09-15T21:09:24.108 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-15T21:09:24.108 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:09:24.108 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.6 up' 2024-09-15T21:09:24.608 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:09:25.609 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:09:25.610 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:09:25.610 INFO:tasks.workunit.client.0.smithi136.stdout:10 2024-09-15T21:09:25.610 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-15T21:09:25.610 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:09:25.611 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.6 up' 2024-09-15T21:09:25.961 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:09:26.963 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:09:26.963 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:09:26.963 INFO:tasks.workunit.client.0.smithi136.stdout:11 2024-09-15T21:09:26.964 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-15T21:09:26.964 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:09:26.964 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.6 up' 2024-09-15T21:09:27.255 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:09:28.257 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:09:28.257 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:09:28.257 INFO:tasks.workunit.client.0.smithi136.stdout:12 2024-09-15T21:09:28.258 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-15T21:09:28.258 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:09:28.259 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.6 up' 2024-09-15T21:09:28.633 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:09:29.634 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:09:29.635 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:09:29.635 INFO:tasks.workunit.client.0.smithi136.stdout:13 2024-09-15T21:09:29.635 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-15T21:09:29.635 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:09:29.636 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.6 up' 2024-09-15T21:09:30.244 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:09:31.245 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:09:31.245 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:09:31.246 INFO:tasks.workunit.client.0.smithi136.stdout:14 2024-09-15T21:09:31.246 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-15T21:09:31.246 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:09:31.246 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.6 up' 2024-09-15T21:09:32.005 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:09:33.007 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:09:33.007 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:09:33.007 INFO:tasks.workunit.client.0.smithi136.stdout:15 2024-09-15T21:09:33.008 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-15T21:09:33.008 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:09:33.008 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.6 up' 2024-09-15T21:09:33.758 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:09:34.760 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:09:34.760 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:09:34.760 INFO:tasks.workunit.client.0.smithi136.stdout:16 2024-09-15T21:09:34.760 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-15T21:09:34.761 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:09:34.761 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.6 up' 2024-09-15T21:09:35.178 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:09:36.179 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:09:36.180 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:09:36.180 INFO:tasks.workunit.client.0.smithi136.stdout:17 2024-09-15T21:09:36.180 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-15T21:09:36.180 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:09:36.180 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.6 up' 2024-09-15T21:09:36.872 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:09:37.874 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:09:37.874 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:09:37.874 INFO:tasks.workunit.client.0.smithi136.stdout:18 2024-09-15T21:09:37.875 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 18 2024-09-15T21:09:37.875 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:09:37.875 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.6 up' 2024-09-15T21:09:38.234 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:09:39.235 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:09:39.235 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:09:39.236 INFO:tasks.workunit.client.0.smithi136.stdout:19 2024-09-15T21:09:39.236 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 19 2024-09-15T21:09:39.236 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:09:39.236 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.6 up' 2024-09-15T21:09:39.596 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:09:40.597 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:09:40.597 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:09:40.597 INFO:tasks.workunit.client.0.smithi136.stdout:20 2024-09-15T21:09:40.598 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 20 2024-09-15T21:09:40.598 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:09:40.598 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.6 up' 2024-09-15T21:09:40.874 INFO:tasks.workunit.client.0.smithi136.stdout:osd.6 up in weight 1 up_from 35 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6850/2063038166,v1:127.0.0.1:6851/2063038166] [v2:127.0.0.1:6852/2063038166,v1:127.0.0.1:6853/2063038166] exists,up be621ea8-c8ad-40e0-ad1c-5e029e812c9d 2024-09-15T21:09:40.878 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-15T21:09:40.878 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-15T21:09:40.878 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-15T21:09:40.878 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:831: corrupt_and_repair_lrc: for id in $(seq 0 9) 2024-09-15T21:09:40.878 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:832: corrupt_and_repair_lrc: run_osd td/osd-scrub-repair 7 2024-09-15T21:09:40.878 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-scrub-repair 2024-09-15T21:09:40.878 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-15T21:09:40.878 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=7 2024-09-15T21:09:40.878 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-15T21:09:40.879 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-scrub-repair/7 2024-09-15T21:09:40.879 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=ce4892c7-6c6d-4b4f-8e6f-07a3e11133e5 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-15T21:09:40.879 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T21:09:40.879 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T21:09:40.879 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T21:09:40.879 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/7' 2024-09-15T21:09:40.879 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/7/journal' 2024-09-15T21:09:40.879 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-15T21:09:40.879 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-15T21:09:40.879 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-15T21:09:40.879 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-15T21:09:40.879 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-15T21:09:40.879 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-15T21:09:40.880 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-15T21:09:40.880 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T21:09:40.880 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52602 2024-09-15T21:09:40.880 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.52602/$cluster-$name.asok' 2024-09-15T21:09:40.880 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.52602/$cluster-$name.asok' 2024-09-15T21:09:40.880 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-15T21:09:40.880 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-15T21:09:40.880 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-15T21:09:40.880 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-15T21:09:40.880 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-15T21:09:40.880 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T21:09:40.880 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T21:09:40.880 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T21:09:40.880 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T21:09:40.881 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-15T21:09:40.881 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-15T21:09:40.881 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-scrub-repair/7 2024-09-15T21:09:40.881 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-15T21:09:40.882 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=7ec87edc-9eab-4395-969d-f2299174764d 2024-09-15T21:09:40.882 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd7 7ec87edc-9eab-4395-969d-f2299174764d' 2024-09-15T21:09:40.882 INFO:tasks.workunit.client.0.smithi136.stdout:add osd7 7ec87edc-9eab-4395-969d-f2299174764d 2024-09-15T21:09:40.882 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-15T21:09:40.898 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQAUTedmbcZbNRAAbPkZ4xdrfdNuh/NiCVvmiA== 2024-09-15T21:09:40.898 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQAUTedmbcZbNRAAbPkZ4xdrfdNuh/NiCVvmiA=="}' 2024-09-15T21:09:40.898 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 7ec87edc-9eab-4395-969d-f2299174764d -i td/osd-scrub-repair/7/new.json 2024-09-15T21:09:41.358 INFO:tasks.workunit.client.0.smithi136.stdout:7 2024-09-15T21:09:41.370 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-scrub-repair/7/new.json 2024-09-15T21:09:41.370 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 7 --fsid=ce4892c7-6c6d-4b4f-8e6f-07a3e11133e5 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/7 --osd-journal=td/osd-scrub-repair/7/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.52602/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAUTedmbcZbNRAAbPkZ4xdrfdNuh/NiCVvmiA== --osd-uuid 7ec87edc-9eab-4395-969d-f2299174764d 2024-09-15T21:09:41.401 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:09:41.398+0000 7fe1a1eef640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:09:41.435 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:09:41.432+0000 7fe1a1eef640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:09:41.436 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:09:41.433+0000 7fe1a1eef640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:09:41.436 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:09:41.433+0000 7fe1a1eef640 -1 bdev(0x55e5d5c23c00 td/osd-scrub-repair/7/block) open stat got: (1) Operation not permitted 2024-09-15T21:09:41.436 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:09:41.433+0000 7fe1a1eef640 -1 bluestore(td/osd-scrub-repair/7) _read_fsid unparsable uuid 2024-09-15T21:09:48.010 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-scrub-repair/7/keyring 2024-09-15T21:09:48.011 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-15T21:09:48.012 INFO:tasks.workunit.client.0.smithi136.stdout:adding osd7 key to auth repository 2024-09-15T21:09:48.012 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd7 key to auth repository 2024-09-15T21:09:48.012 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-scrub-repair/7/keyring auth add osd.7 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-15T21:09:48.374 INFO:tasks.workunit.client.0.smithi136.stdout:start osd.7 2024-09-15T21:09:48.375 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.7 2024-09-15T21:09:48.375 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 7 --fsid=ce4892c7-6c6d-4b4f-8e6f-07a3e11133e5 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/7 --osd-journal=td/osd-scrub-repair/7/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.52602/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T21:09:48.375 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-15T21:09:48.375 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-15T21:09:48.375 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-15T21:09:48.402 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:09:48.399+0000 7f4f5ee18640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:09:48.404 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:09:48.401+0000 7f4f5ee18640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:09:48.405 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:09:48.402+0000 7f4f5ee18640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:09:48.656 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 7 2024-09-15T21:09:48.656 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-15T21:09:48.656 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=7 2024-09-15T21:09:48.656 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-15T21:09:48.656 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-15T21:09:48.656 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:09:48.657 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-15T21:09:48.657 INFO:tasks.workunit.client.0.smithi136.stdout:0 2024-09-15T21:09:48.657 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:09:48.657 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.7 up' 2024-09-15T21:09:48.950 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:09:49.951 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:09:49.951 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:09:49.952 INFO:tasks.workunit.client.0.smithi136.stdout:1 2024-09-15T21:09:49.952 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-15T21:09:49.952 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:09:49.952 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.7 up' 2024-09-15T21:09:50.238 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:09:50.278 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:09:50.275+0000 7f4f5ee18640 -1 Falling back to public interface 2024-09-15T21:09:51.240 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:09:51.240 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:09:51.240 INFO:tasks.workunit.client.0.smithi136.stdout:2 2024-09-15T21:09:51.241 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-15T21:09:51.241 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:09:51.241 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.7 up' 2024-09-15T21:09:51.525 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:09:52.526 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:09:52.526 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:09:52.527 INFO:tasks.workunit.client.0.smithi136.stdout:3 2024-09-15T21:09:52.527 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-15T21:09:52.527 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:09:52.527 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.7 up' 2024-09-15T21:09:52.816 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:09:52.981 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:09:52.978+0000 7f4f5ee18640 -1 osd.7 0 log_to_monitors true 2024-09-15T21:09:53.817 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:09:53.818 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:09:53.818 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-15T21:09:53.818 INFO:tasks.workunit.client.0.smithi136.stdout:4 2024-09-15T21:09:53.818 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:09:53.818 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.7 up' 2024-09-15T21:09:54.111 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:09:55.112 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:09:55.112 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:09:55.113 INFO:tasks.workunit.client.0.smithi136.stdout:5 2024-09-15T21:09:55.113 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-15T21:09:55.113 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:09:55.113 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.7 up' 2024-09-15T21:09:55.410 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:09:56.412 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:09:56.412 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:09:56.412 INFO:tasks.workunit.client.0.smithi136.stdout:6 2024-09-15T21:09:56.413 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-15T21:09:56.413 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:09:56.413 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.7 up' 2024-09-15T21:09:56.701 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:09:57.703 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:09:57.703 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:09:57.703 INFO:tasks.workunit.client.0.smithi136.stdout:7 2024-09-15T21:09:57.704 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-15T21:09:57.704 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:09:57.704 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.7 up' 2024-09-15T21:09:58.023 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:09:59.024 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:09:59.024 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:09:59.025 INFO:tasks.workunit.client.0.smithi136.stdout:8 2024-09-15T21:09:59.025 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-15T21:09:59.025 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:09:59.025 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.7 up' 2024-09-15T21:09:59.381 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:10:00.383 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:10:00.383 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:10:00.383 INFO:tasks.workunit.client.0.smithi136.stdout:9 2024-09-15T21:10:00.384 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-15T21:10:00.385 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:10:00.385 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.7 up' 2024-09-15T21:10:00.673 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:10:01.674 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:10:01.674 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:10:01.674 INFO:tasks.workunit.client.0.smithi136.stdout:10 2024-09-15T21:10:01.675 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-15T21:10:01.675 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:10:01.675 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.7 up' 2024-09-15T21:10:01.985 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:10:02.986 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:10:02.987 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:10:02.987 INFO:tasks.workunit.client.0.smithi136.stdout:11 2024-09-15T21:10:02.987 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-15T21:10:02.987 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:10:02.987 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.7 up' 2024-09-15T21:10:03.367 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:10:04.368 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:10:04.368 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:10:04.368 INFO:tasks.workunit.client.0.smithi136.stdout:12 2024-09-15T21:10:04.369 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-15T21:10:04.369 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:10:04.369 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.7 up' 2024-09-15T21:10:04.655 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:10:05.657 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:10:05.657 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:10:05.657 INFO:tasks.workunit.client.0.smithi136.stdout:13 2024-09-15T21:10:05.658 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-15T21:10:05.658 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:10:05.658 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.7 up' 2024-09-15T21:10:06.207 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:10:07.208 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:10:07.208 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:10:07.209 INFO:tasks.workunit.client.0.smithi136.stdout:14 2024-09-15T21:10:07.209 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-15T21:10:07.209 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:10:07.209 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.7 up' 2024-09-15T21:10:07.728 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:10:08.729 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:10:08.729 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:10:08.730 INFO:tasks.workunit.client.0.smithi136.stdout:15 2024-09-15T21:10:08.730 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-15T21:10:08.730 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:10:08.731 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.7 up' 2024-09-15T21:10:09.095 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:10:10.096 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:10:10.096 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:10:10.097 INFO:tasks.workunit.client.0.smithi136.stdout:16 2024-09-15T21:10:10.097 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-15T21:10:10.097 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:10:10.097 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.7 up' 2024-09-15T21:10:10.708 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:10:11.709 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:10:11.709 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:10:11.710 INFO:tasks.workunit.client.0.smithi136.stdout:17 2024-09-15T21:10:11.710 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-15T21:10:11.710 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:10:11.710 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.7 up' 2024-09-15T21:10:12.148 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:10:13.149 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:10:13.149 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:10:13.150 INFO:tasks.workunit.client.0.smithi136.stdout:18 2024-09-15T21:10:13.150 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 18 2024-09-15T21:10:13.150 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:10:13.150 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.7 up' 2024-09-15T21:10:13.453 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:10:14.454 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:10:14.454 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:10:14.455 INFO:tasks.workunit.client.0.smithi136.stdout:19 2024-09-15T21:10:14.455 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 19 2024-09-15T21:10:14.455 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:10:14.455 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.7 up' 2024-09-15T21:10:15.281 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:10:16.282 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:10:16.282 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:10:16.283 INFO:tasks.workunit.client.0.smithi136.stdout:20 2024-09-15T21:10:16.283 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 20 2024-09-15T21:10:16.283 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:10:16.283 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.7 up' 2024-09-15T21:10:16.580 INFO:tasks.workunit.client.0.smithi136.stdout:osd.7 up in weight 1 up_from 40 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6858/267007965,v1:127.0.0.1:6859/267007965] [v2:127.0.0.1:6860/267007965,v1:127.0.0.1:6861/267007965] exists,up 7ec87edc-9eab-4395-969d-f2299174764d 2024-09-15T21:10:16.584 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-15T21:10:16.584 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-15T21:10:16.584 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-15T21:10:16.584 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:831: corrupt_and_repair_lrc: for id in $(seq 0 9) 2024-09-15T21:10:16.584 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:832: corrupt_and_repair_lrc: run_osd td/osd-scrub-repair 8 2024-09-15T21:10:16.585 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-scrub-repair 2024-09-15T21:10:16.585 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-15T21:10:16.585 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=8 2024-09-15T21:10:16.585 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-15T21:10:16.585 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-scrub-repair/8 2024-09-15T21:10:16.585 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=ce4892c7-6c6d-4b4f-8e6f-07a3e11133e5 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-15T21:10:16.585 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T21:10:16.585 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T21:10:16.585 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T21:10:16.585 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/8' 2024-09-15T21:10:16.585 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/8/journal' 2024-09-15T21:10:16.585 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-15T21:10:16.585 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-15T21:10:16.586 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-15T21:10:16.586 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-15T21:10:16.586 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-15T21:10:16.586 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-15T21:10:16.586 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-15T21:10:16.586 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T21:10:16.586 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52602 2024-09-15T21:10:16.586 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.52602/$cluster-$name.asok' 2024-09-15T21:10:16.586 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.52602/$cluster-$name.asok' 2024-09-15T21:10:16.586 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-15T21:10:16.586 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-15T21:10:16.586 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-15T21:10:16.586 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-15T21:10:16.586 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-15T21:10:16.587 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T21:10:16.587 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T21:10:16.587 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T21:10:16.587 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T21:10:16.587 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-15T21:10:16.587 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-15T21:10:16.587 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-scrub-repair/8 2024-09-15T21:10:16.587 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-15T21:10:16.587 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=8c90a8a3-044c-4139-b6af-2330a64272f9 2024-09-15T21:10:16.587 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd8 8c90a8a3-044c-4139-b6af-2330a64272f9' 2024-09-15T21:10:16.587 INFO:tasks.workunit.client.0.smithi136.stdout:add osd8 8c90a8a3-044c-4139-b6af-2330a64272f9 2024-09-15T21:10:16.588 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-15T21:10:16.603 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQA4Tedm5wi4IxAA2E/MkBs/bw7zzwGpJWftGA== 2024-09-15T21:10:16.603 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQA4Tedm5wi4IxAA2E/MkBs/bw7zzwGpJWftGA=="}' 2024-09-15T21:10:16.603 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 8c90a8a3-044c-4139-b6af-2330a64272f9 -i td/osd-scrub-repair/8/new.json 2024-09-15T21:10:16.967 INFO:tasks.workunit.client.0.smithi136.stdout:8 2024-09-15T21:10:16.978 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-scrub-repair/8/new.json 2024-09-15T21:10:16.979 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 8 --fsid=ce4892c7-6c6d-4b4f-8e6f-07a3e11133e5 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/8 --osd-journal=td/osd-scrub-repair/8/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.52602/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQA4Tedm5wi4IxAA2E/MkBs/bw7zzwGpJWftGA== --osd-uuid 8c90a8a3-044c-4139-b6af-2330a64272f9 2024-09-15T21:10:17.010 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:10:17.006+0000 7f73d500e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:10:17.060 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:10:17.056+0000 7f73d500e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:10:17.061 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:10:17.057+0000 7f73d500e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:10:17.062 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:10:17.058+0000 7f73d500e640 -1 bdev(0x5586f8ec5c00 td/osd-scrub-repair/8/block) open stat got: (1) Operation not permitted 2024-09-15T21:10:17.062 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:10:17.058+0000 7f73d500e640 -1 bluestore(td/osd-scrub-repair/8) _read_fsid unparsable uuid 2024-09-15T21:10:23.220 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-scrub-repair/8/keyring 2024-09-15T21:10:23.220 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-15T21:10:23.221 INFO:tasks.workunit.client.0.smithi136.stdout:adding osd8 key to auth repository 2024-09-15T21:10:23.221 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd8 key to auth repository 2024-09-15T21:10:23.221 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-scrub-repair/8/keyring auth add osd.8 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-15T21:10:23.583 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.8 2024-09-15T21:10:23.583 INFO:tasks.workunit.client.0.smithi136.stdout:start osd.8 2024-09-15T21:10:23.584 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 8 --fsid=ce4892c7-6c6d-4b4f-8e6f-07a3e11133e5 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/8 --osd-journal=td/osd-scrub-repair/8/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.52602/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T21:10:23.584 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-15T21:10:23.584 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-15T21:10:23.584 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-15T21:10:23.610 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:10:23.607+0000 7f906d418640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:10:23.612 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:10:23.609+0000 7f906d418640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:10:23.613 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:10:23.610+0000 7f906d418640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:10:23.862 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 8 2024-09-15T21:10:23.862 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-15T21:10:23.862 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=8 2024-09-15T21:10:23.863 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-15T21:10:23.863 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-15T21:10:23.863 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:10:23.863 INFO:tasks.workunit.client.0.smithi136.stdout:0 2024-09-15T21:10:23.863 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-15T21:10:23.863 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:10:23.863 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.8 up' 2024-09-15T21:10:24.138 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:10:25.139 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:10:25.139 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:10:25.140 INFO:tasks.workunit.client.0.smithi136.stdout:1 2024-09-15T21:10:25.140 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-15T21:10:25.140 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:10:25.140 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.8 up' 2024-09-15T21:10:25.228 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:10:25.225+0000 7f906d418640 -1 Falling back to public interface 2024-09-15T21:10:25.428 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:10:26.430 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:10:26.430 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:10:26.430 INFO:tasks.workunit.client.0.smithi136.stdout:2 2024-09-15T21:10:26.430 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-15T21:10:26.431 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:10:26.431 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.8 up' 2024-09-15T21:10:26.717 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:10:27.719 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:10:27.719 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:10:27.719 INFO:tasks.workunit.client.0.smithi136.stdout:3 2024-09-15T21:10:27.720 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-15T21:10:27.720 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:10:27.720 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.8 up' 2024-09-15T21:10:27.867 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:10:27.864+0000 7f906d418640 -1 osd.8 0 log_to_monitors true 2024-09-15T21:10:28.014 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:10:29.015 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:10:29.015 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:10:29.015 INFO:tasks.workunit.client.0.smithi136.stdout:4 2024-09-15T21:10:29.016 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-15T21:10:29.016 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:10:29.016 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.8 up' 2024-09-15T21:10:29.465 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:10:30.467 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:10:30.467 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:10:30.467 INFO:tasks.workunit.client.0.smithi136.stdout:5 2024-09-15T21:10:30.468 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-15T21:10:30.468 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:10:30.468 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.8 up' 2024-09-15T21:10:30.867 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:10:31.868 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:10:31.868 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:10:31.868 INFO:tasks.workunit.client.0.smithi136.stdout:6 2024-09-15T21:10:31.869 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-15T21:10:31.869 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:10:31.869 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.8 up' 2024-09-15T21:10:32.168 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:10:33.169 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:10:33.170 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:10:33.170 INFO:tasks.workunit.client.0.smithi136.stdout:7 2024-09-15T21:10:33.170 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-15T21:10:33.170 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:10:33.170 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.8 up' 2024-09-15T21:10:33.459 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:10:34.460 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:10:34.460 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:10:34.461 INFO:tasks.workunit.client.0.smithi136.stdout:8 2024-09-15T21:10:34.461 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-15T21:10:34.461 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:10:34.461 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.8 up' 2024-09-15T21:10:34.841 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:10:35.842 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:10:35.842 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:10:35.843 INFO:tasks.workunit.client.0.smithi136.stdout:9 2024-09-15T21:10:35.843 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-15T21:10:35.843 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:10:35.843 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.8 up' 2024-09-15T21:10:36.134 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:10:37.135 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:10:37.135 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:10:37.136 INFO:tasks.workunit.client.0.smithi136.stdout:10 2024-09-15T21:10:37.136 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-15T21:10:37.136 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:10:37.136 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.8 up' 2024-09-15T21:10:37.476 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:10:38.477 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:10:38.478 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:10:38.478 INFO:tasks.workunit.client.0.smithi136.stdout:11 2024-09-15T21:10:38.478 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-15T21:10:38.478 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:10:38.478 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.8 up' 2024-09-15T21:10:38.867 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:10:39.869 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:10:39.869 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:10:39.869 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-15T21:10:39.869 INFO:tasks.workunit.client.0.smithi136.stdout:12 2024-09-15T21:10:39.870 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:10:39.870 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.8 up' 2024-09-15T21:10:40.245 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:10:41.246 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:10:41.246 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:10:41.246 INFO:tasks.workunit.client.0.smithi136.stdout:13 2024-09-15T21:10:41.247 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-15T21:10:41.247 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:10:41.247 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.8 up' 2024-09-15T21:10:41.981 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:10:42.983 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:10:42.983 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:10:42.983 INFO:tasks.workunit.client.0.smithi136.stdout:14 2024-09-15T21:10:42.983 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-15T21:10:42.984 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:10:42.984 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.8 up' 2024-09-15T21:10:43.565 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:10:44.566 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:10:44.566 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:10:44.567 INFO:tasks.workunit.client.0.smithi136.stdout:15 2024-09-15T21:10:44.567 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-15T21:10:44.567 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:10:44.567 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.8 up' 2024-09-15T21:10:45.296 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:10:46.297 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:10:46.298 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:10:46.298 INFO:tasks.workunit.client.0.smithi136.stdout:16 2024-09-15T21:10:46.298 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-15T21:10:46.298 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:10:46.298 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.8 up' 2024-09-15T21:10:47.108 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:10:48.110 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:10:48.110 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:10:48.110 INFO:tasks.workunit.client.0.smithi136.stdout:17 2024-09-15T21:10:48.111 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-15T21:10:48.111 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:10:48.111 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.8 up' 2024-09-15T21:10:48.771 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:10:49.772 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:10:49.773 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:10:49.773 INFO:tasks.workunit.client.0.smithi136.stdout:18 2024-09-15T21:10:49.773 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 18 2024-09-15T21:10:49.774 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:10:49.774 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.8 up' 2024-09-15T21:10:50.617 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:10:51.619 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:10:51.619 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:10:51.619 INFO:tasks.workunit.client.0.smithi136.stdout:19 2024-09-15T21:10:51.619 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 19 2024-09-15T21:10:51.619 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:10:51.620 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.8 up' 2024-09-15T21:10:51.988 INFO:tasks.workunit.client.0.smithi136.stdout:osd.8 up in weight 1 up_from 45 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6866/4097646374,v1:127.0.0.1:6867/4097646374] [v2:127.0.0.1:6868/4097646374,v1:127.0.0.1:6869/4097646374] exists,up 8c90a8a3-044c-4139-b6af-2330a64272f9 2024-09-15T21:10:51.991 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-15T21:10:51.991 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-15T21:10:51.991 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-15T21:10:51.992 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:831: corrupt_and_repair_lrc: for id in $(seq 0 9) 2024-09-15T21:10:51.992 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:832: corrupt_and_repair_lrc: run_osd td/osd-scrub-repair 9 2024-09-15T21:10:51.992 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-scrub-repair 2024-09-15T21:10:51.992 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-15T21:10:51.992 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=9 2024-09-15T21:10:51.992 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-15T21:10:51.992 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-scrub-repair/9 2024-09-15T21:10:51.992 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=ce4892c7-6c6d-4b4f-8e6f-07a3e11133e5 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-15T21:10:51.992 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T21:10:51.992 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T21:10:51.992 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T21:10:51.992 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/9' 2024-09-15T21:10:51.992 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/9/journal' 2024-09-15T21:10:51.993 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-15T21:10:51.993 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-15T21:10:51.993 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-15T21:10:51.993 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-15T21:10:51.993 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-15T21:10:51.993 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-15T21:10:51.993 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-15T21:10:51.993 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T21:10:51.993 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52602 2024-09-15T21:10:51.993 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.52602/$cluster-$name.asok' 2024-09-15T21:10:51.993 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.52602/$cluster-$name.asok' 2024-09-15T21:10:51.993 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-15T21:10:51.993 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-15T21:10:51.993 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-15T21:10:51.994 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-15T21:10:51.994 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-15T21:10:51.994 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T21:10:51.994 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T21:10:51.994 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T21:10:51.994 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T21:10:51.994 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-15T21:10:51.994 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-15T21:10:51.994 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-scrub-repair/9 2024-09-15T21:10:51.994 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-15T21:10:51.995 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=78ecdb67-c9df-44e1-b77a-4e850e67a453 2024-09-15T21:10:51.995 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd9 78ecdb67-c9df-44e1-b77a-4e850e67a453' 2024-09-15T21:10:51.995 INFO:tasks.workunit.client.0.smithi136.stdout:add osd9 78ecdb67-c9df-44e1-b77a-4e850e67a453 2024-09-15T21:10:51.995 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-15T21:10:52.011 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQBcTedmKBx3ABAASqamSgwmzgSXHz5DZask1g== 2024-09-15T21:10:52.011 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQBcTedmKBx3ABAASqamSgwmzgSXHz5DZask1g=="}' 2024-09-15T21:10:52.011 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 78ecdb67-c9df-44e1-b77a-4e850e67a453 -i td/osd-scrub-repair/9/new.json 2024-09-15T21:10:52.361 INFO:tasks.workunit.client.0.smithi136.stdout:9 2024-09-15T21:10:52.372 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-scrub-repair/9/new.json 2024-09-15T21:10:52.373 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 9 --fsid=ce4892c7-6c6d-4b4f-8e6f-07a3e11133e5 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/9 --osd-journal=td/osd-scrub-repair/9/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.52602/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBcTedmKBx3ABAASqamSgwmzgSXHz5DZask1g== --osd-uuid 78ecdb67-c9df-44e1-b77a-4e850e67a453 2024-09-15T21:10:52.403 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:10:52.400+0000 7fdc3440b640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:10:52.445 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:10:52.442+0000 7fdc3440b640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:10:52.446 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:10:52.443+0000 7fdc3440b640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:10:52.447 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:10:52.444+0000 7fdc3440b640 -1 bdev(0x55721cfd7c00 td/osd-scrub-repair/9/block) open stat got: (1) Operation not permitted 2024-09-15T21:10:52.447 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:10:52.444+0000 7fdc3440b640 -1 bluestore(td/osd-scrub-repair/9) _read_fsid unparsable uuid 2024-09-15T21:10:58.930 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-scrub-repair/9/keyring 2024-09-15T21:10:58.930 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-15T21:10:58.931 INFO:tasks.workunit.client.0.smithi136.stdout:adding osd9 key to auth repository 2024-09-15T21:10:58.931 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd9 key to auth repository 2024-09-15T21:10:58.932 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-scrub-repair/9/keyring auth add osd.9 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-15T21:10:59.292 INFO:tasks.workunit.client.0.smithi136.stdout:start osd.9 2024-09-15T21:10:59.293 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.9 2024-09-15T21:10:59.293 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 9 --fsid=ce4892c7-6c6d-4b4f-8e6f-07a3e11133e5 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/9 --osd-journal=td/osd-scrub-repair/9/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.52602/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T21:10:59.293 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-15T21:10:59.293 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-15T21:10:59.294 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-15T21:10:59.320 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:10:59.317+0000 7fdfcb9dd640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:10:59.322 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:10:59.319+0000 7fdfcb9dd640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:10:59.323 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:10:59.320+0000 7fdfcb9dd640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:10:59.570 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 9 2024-09-15T21:10:59.570 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-15T21:10:59.570 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=9 2024-09-15T21:10:59.570 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-15T21:10:59.571 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-15T21:10:59.571 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:10:59.571 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-15T21:10:59.571 INFO:tasks.workunit.client.0.smithi136.stdout:0 2024-09-15T21:10:59.571 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:10:59.571 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.9 up' 2024-09-15T21:10:59.848 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:11:00.849 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:11:00.849 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:11:00.850 INFO:tasks.workunit.client.0.smithi136.stdout:1 2024-09-15T21:11:00.850 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-15T21:11:00.850 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:11:00.850 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.9 up' 2024-09-15T21:11:00.938 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:11:00.935+0000 7fdfcb9dd640 -1 Falling back to public interface 2024-09-15T21:11:01.179 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:11:02.180 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:11:02.180 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:11:02.180 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-15T21:11:02.180 INFO:tasks.workunit.client.0.smithi136.stdout:2 2024-09-15T21:11:02.181 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:11:02.181 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.9 up' 2024-09-15T21:11:02.467 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:11:03.469 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:11:03.469 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:11:03.469 INFO:tasks.workunit.client.0.smithi136.stdout:3 2024-09-15T21:11:03.470 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-15T21:11:03.470 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:11:03.470 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.9 up' 2024-09-15T21:11:03.608 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:11:03.604+0000 7fdfcb9dd640 -1 osd.9 0 log_to_monitors true 2024-09-15T21:11:03.753 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:11:04.755 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:11:04.755 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:11:04.755 INFO:tasks.workunit.client.0.smithi136.stdout:4 2024-09-15T21:11:04.755 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-15T21:11:04.756 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:11:04.756 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.9 up' 2024-09-15T21:11:05.145 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:11:06.146 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:11:06.147 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:11:06.147 INFO:tasks.workunit.client.0.smithi136.stdout:5 2024-09-15T21:11:06.147 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-15T21:11:06.147 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:11:06.148 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.9 up' 2024-09-15T21:11:06.457 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:11:07.458 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:11:07.458 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:11:07.459 INFO:tasks.workunit.client.0.smithi136.stdout:6 2024-09-15T21:11:07.459 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-15T21:11:07.459 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:11:07.459 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.9 up' 2024-09-15T21:11:07.749 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:11:08.750 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:11:08.750 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:11:08.751 INFO:tasks.workunit.client.0.smithi136.stdout:7 2024-09-15T21:11:08.751 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-15T21:11:08.751 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:11:08.751 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.9 up' 2024-09-15T21:11:09.038 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:11:10.040 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:11:10.040 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:11:10.040 INFO:tasks.workunit.client.0.smithi136.stdout:8 2024-09-15T21:11:10.041 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-15T21:11:10.041 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:11:10.041 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.9 up' 2024-09-15T21:11:10.385 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:11:11.386 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:11:11.386 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:11:11.386 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-15T21:11:11.387 INFO:tasks.workunit.client.0.smithi136.stdout:9 2024-09-15T21:11:11.387 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:11:11.387 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.9 up' 2024-09-15T21:11:11.676 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:11:12.677 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:11:12.677 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:11:12.678 INFO:tasks.workunit.client.0.smithi136.stdout:10 2024-09-15T21:11:12.678 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-15T21:11:12.678 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:11:12.678 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.9 up' 2024-09-15T21:11:12.968 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:11:13.969 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:11:13.970 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:11:13.970 INFO:tasks.workunit.client.0.smithi136.stdout:11 2024-09-15T21:11:13.970 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-15T21:11:13.970 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:11:13.971 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.9 up' 2024-09-15T21:11:14.262 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:11:15.264 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:11:15.264 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:11:15.264 INFO:tasks.workunit.client.0.smithi136.stdout:12 2024-09-15T21:11:15.265 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-15T21:11:15.265 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:11:15.265 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.9 up' 2024-09-15T21:11:15.559 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:11:16.560 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:11:16.561 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:11:16.561 INFO:tasks.workunit.client.0.smithi136.stdout:13 2024-09-15T21:11:16.561 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-15T21:11:16.562 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:11:16.562 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.9 up' 2024-09-15T21:11:17.235 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:11:18.236 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:11:18.236 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:11:18.237 INFO:tasks.workunit.client.0.smithi136.stdout:14 2024-09-15T21:11:18.237 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-15T21:11:18.237 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:11:18.237 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.9 up' 2024-09-15T21:11:18.550 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:11:19.552 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:11:19.552 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:11:19.552 INFO:tasks.workunit.client.0.smithi136.stdout:15 2024-09-15T21:11:19.553 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-15T21:11:19.553 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:11:19.553 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.9 up' 2024-09-15T21:11:20.473 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:11:21.475 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:11:21.475 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:11:21.475 INFO:tasks.workunit.client.0.smithi136.stdout:16 2024-09-15T21:11:21.476 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-15T21:11:21.476 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:11:21.476 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.9 up' 2024-09-15T21:11:22.035 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:11:23.037 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:11:23.037 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:11:23.037 INFO:tasks.workunit.client.0.smithi136.stdout:17 2024-09-15T21:11:23.038 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-15T21:11:23.038 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:11:23.038 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.9 up' 2024-09-15T21:11:23.453 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:11:24.454 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:11:24.454 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:11:24.455 INFO:tasks.workunit.client.0.smithi136.stdout:18 2024-09-15T21:11:24.455 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 18 2024-09-15T21:11:24.455 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:11:24.455 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.9 up' 2024-09-15T21:11:24.950 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:11:25.951 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:11:25.951 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:11:25.952 INFO:tasks.workunit.client.0.smithi136.stdout:19 2024-09-15T21:11:25.952 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 19 2024-09-15T21:11:25.952 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:11:25.952 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.9 up' 2024-09-15T21:11:26.283 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:11:27.285 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:11:27.285 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:11:27.285 INFO:tasks.workunit.client.0.smithi136.stdout:20 2024-09-15T21:11:27.286 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 20 2024-09-15T21:11:27.286 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:11:27.286 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.9 up' 2024-09-15T21:11:27.594 INFO:tasks.workunit.client.0.smithi136.stdout:osd.9 up in weight 1 up_from 50 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6874/3721342003,v1:127.0.0.1:6875/3721342003] [v2:127.0.0.1:6876/3721342003,v1:127.0.0.1:6877/3721342003] exists,up 78ecdb67-c9df-44e1-b77a-4e850e67a453 2024-09-15T21:11:27.595 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-15T21:11:27.595 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-15T21:11:27.595 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-15T21:11:27.595 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:834: corrupt_and_repair_lrc: create_rbd_pool 2024-09-15T21:11:27.595 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:526: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2024-09-15T21:11:27.869 INFO:tasks.workunit.client.0.smithi136.stderr:pool 'rbd' does not exist 2024-09-15T21:11:27.880 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:527: create_rbd_pool: create_pool rbd 4 2024-09-15T21:11:27.880 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create rbd 4 2024-09-15T21:11:28.313 INFO:tasks.workunit.client.0.smithi136.stderr:pool 'rbd' created 2024-09-15T21:11:28.325 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2024-09-15T21:11:29.326 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:528: create_rbd_pool: rbd pool init rbd 2024-09-15T21:11:32.469 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:835: corrupt_and_repair_lrc: wait_for_clean 2024-09-15T21:11:32.469 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-15T21:11:32.469 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-15T21:11:32.469 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-15T21:11:32.469 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-15T21:11:32.470 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-15T21:11:32.470 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-15T21:11:32.470 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-15T21:11:32.470 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-15T21:11:32.471 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-15T21:11:32.571 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-15T21:11:32.572 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-15T21:11:32.572 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-15T21:11:32.572 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-15T21:11:32.572 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-15T21:11:32.572 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-15T21:11:32.851 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-15T21:11:32.851 INFO:tasks.workunit.client.0.smithi136.stderr:1 2024-09-15T21:11:32.851 INFO:tasks.workunit.client.0.smithi136.stderr:2 2024-09-15T21:11:32.852 INFO:tasks.workunit.client.0.smithi136.stderr:3 2024-09-15T21:11:32.852 INFO:tasks.workunit.client.0.smithi136.stderr:4 2024-09-15T21:11:32.852 INFO:tasks.workunit.client.0.smithi136.stderr:5 2024-09-15T21:11:32.852 INFO:tasks.workunit.client.0.smithi136.stderr:6 2024-09-15T21:11:32.852 INFO:tasks.workunit.client.0.smithi136.stderr:7 2024-09-15T21:11:32.852 INFO:tasks.workunit.client.0.smithi136.stderr:8 2024-09-15T21:11:32.852 INFO:tasks.workunit.client.0.smithi136.stderr:9' 2024-09-15T21:11:32.852 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-15T21:11:32.852 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T21:11:32.852 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-15T21:11:32.955 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836547 2024-09-15T21:11:32.955 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836547 2024-09-15T21:11:32.955 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836547' 2024-09-15T21:11:32.955 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T21:11:32.955 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-15T21:11:33.059 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=42949673023 2024-09-15T21:11:33.059 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 42949673023 2024-09-15T21:11:33.059 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836547 1-42949673023' 2024-09-15T21:11:33.059 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T21:11:33.059 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-15T21:11:33.162 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=64424509493 2024-09-15T21:11:33.163 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 64424509493 2024-09-15T21:11:33.163 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836547 1-42949673023 2-64424509493' 2024-09-15T21:11:33.163 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T21:11:33.163 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-15T21:11:33.266 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=85899345967 2024-09-15T21:11:33.267 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 85899345967 2024-09-15T21:11:33.267 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836547 1-42949673023 2-64424509493 3-85899345967' 2024-09-15T21:11:33.267 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T21:11:33.267 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-09-15T21:11:33.371 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=107374182442 2024-09-15T21:11:33.371 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 107374182442 2024-09-15T21:11:33.371 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836547 1-42949673023 2-64424509493 3-85899345967 4-107374182442' 2024-09-15T21:11:33.371 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T21:11:33.371 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-09-15T21:11:33.473 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=128849018913 2024-09-15T21:11:33.474 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 128849018913 2024-09-15T21:11:33.474 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836547 1-42949673023 2-64424509493 3-85899345967 4-107374182442 5-128849018913' 2024-09-15T21:11:33.474 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T21:11:33.474 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-09-15T21:11:33.577 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=150323855390 2024-09-15T21:11:33.577 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 150323855390 2024-09-15T21:11:33.577 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836547 1-42949673023 2-64424509493 3-85899345967 4-107374182442 5-128849018913 6-150323855390' 2024-09-15T21:11:33.577 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T21:11:33.577 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2024-09-15T21:11:33.680 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=171798691862 2024-09-15T21:11:33.680 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 171798691862 2024-09-15T21:11:33.680 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836547 1-42949673023 2-64424509493 3-85899345967 4-107374182442 5-128849018913 6-150323855390 7-171798691862' 2024-09-15T21:11:33.680 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T21:11:33.681 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2024-09-15T21:11:33.782 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=193273528336 2024-09-15T21:11:33.783 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 193273528336 2024-09-15T21:11:33.783 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836547 1-42949673023 2-64424509493 3-85899345967 4-107374182442 5-128849018913 6-150323855390 7-171798691862 8-193273528336' 2024-09-15T21:11:33.783 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T21:11:33.783 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2024-09-15T21:11:33.887 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=214748364808 2024-09-15T21:11:33.887 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 214748364808 2024-09-15T21:11:33.887 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836547 1-42949673023 2-64424509493 3-85899345967 4-107374182442 5-128849018913 6-150323855390 7-171798691862 8-193273528336 9-214748364808' 2024-09-15T21:11:33.888 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T21:11:33.888 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836547 2024-09-15T21:11:33.888 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T21:11:33.888 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-15T21:11:33.889 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836547 2024-09-15T21:11:33.889 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T21:11:33.890 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836547 2024-09-15T21:11:33.890 INFO:tasks.workunit.client.0.smithi136.stdout:waiting osd.0 seq 21474836547 2024-09-15T21:11:33.891 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836547' 2024-09-15T21:11:33.891 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T21:11:34.166 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836548 -lt 21474836547 2024-09-15T21:11:34.166 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T21:11:34.167 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-42949673023 2024-09-15T21:11:34.167 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T21:11:34.168 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-15T21:11:34.168 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-42949673023 2024-09-15T21:11:34.168 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T21:11:34.169 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=42949673023 2024-09-15T21:11:34.169 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 42949673023' 2024-09-15T21:11:34.170 INFO:tasks.workunit.client.0.smithi136.stdout:waiting osd.1 seq 42949673023 2024-09-15T21:11:34.170 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-15T21:11:34.447 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 42949673024 -lt 42949673023 2024-09-15T21:11:34.447 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T21:11:34.448 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-64424509493 2024-09-15T21:11:34.448 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T21:11:34.448 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-15T21:11:34.449 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-64424509493 2024-09-15T21:11:34.449 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T21:11:34.450 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=64424509493 2024-09-15T21:11:34.450 INFO:tasks.workunit.client.0.smithi136.stdout:waiting osd.2 seq 64424509493 2024-09-15T21:11:34.451 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 64424509493' 2024-09-15T21:11:34.451 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-15T21:11:34.731 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 64424509494 -lt 64424509493 2024-09-15T21:11:34.731 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T21:11:34.731 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 3-85899345967 2024-09-15T21:11:34.731 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T21:11:34.732 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=3 2024-09-15T21:11:34.733 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 3-85899345967 2024-09-15T21:11:34.733 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T21:11:34.734 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=85899345967 2024-09-15T21:11:34.734 INFO:tasks.workunit.client.0.smithi136.stdout:waiting osd.3 seq 85899345967 2024-09-15T21:11:34.734 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.3 seq 85899345967' 2024-09-15T21:11:34.734 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-15T21:11:35.093 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 85899345968 -lt 85899345967 2024-09-15T21:11:35.093 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T21:11:35.093 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 4-107374182442 2024-09-15T21:11:35.093 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T21:11:35.094 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=4 2024-09-15T21:11:35.094 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 4-107374182442 2024-09-15T21:11:35.094 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T21:11:35.094 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=107374182442 2024-09-15T21:11:35.095 INFO:tasks.workunit.client.0.smithi136.stdout:waiting osd.4 seq 107374182442 2024-09-15T21:11:35.095 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.4 seq 107374182442' 2024-09-15T21:11:35.095 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 4 2024-09-15T21:11:35.379 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 107374182442 -lt 107374182442 2024-09-15T21:11:35.379 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T21:11:35.380 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 5-128849018913 2024-09-15T21:11:35.380 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T21:11:35.380 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=5 2024-09-15T21:11:35.381 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 5-128849018913 2024-09-15T21:11:35.381 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T21:11:35.382 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=128849018913 2024-09-15T21:11:35.382 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.5 seq 128849018913' 2024-09-15T21:11:35.383 INFO:tasks.workunit.client.0.smithi136.stdout:waiting osd.5 seq 128849018913 2024-09-15T21:11:35.383 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 5 2024-09-15T21:11:35.661 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 128849018913 -lt 128849018913 2024-09-15T21:11:35.661 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T21:11:35.661 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 6-150323855390 2024-09-15T21:11:35.661 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T21:11:35.662 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=6 2024-09-15T21:11:35.663 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 6-150323855390 2024-09-15T21:11:35.663 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T21:11:35.664 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=150323855390 2024-09-15T21:11:35.664 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.6 seq 150323855390' 2024-09-15T21:11:35.664 INFO:tasks.workunit.client.0.smithi136.stdout:waiting osd.6 seq 150323855390 2024-09-15T21:11:35.664 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 6 2024-09-15T21:11:35.944 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 150323855391 -lt 150323855390 2024-09-15T21:11:35.944 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T21:11:35.944 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 7-171798691862 2024-09-15T21:11:35.944 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T21:11:35.945 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=7 2024-09-15T21:11:35.946 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 7-171798691862 2024-09-15T21:11:35.946 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T21:11:35.947 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=171798691862 2024-09-15T21:11:35.947 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.7 seq 171798691862' 2024-09-15T21:11:35.947 INFO:tasks.workunit.client.0.smithi136.stdout:waiting osd.7 seq 171798691862 2024-09-15T21:11:35.947 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 7 2024-09-15T21:11:36.238 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 171798691862 -lt 171798691862 2024-09-15T21:11:36.238 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T21:11:36.238 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 8-193273528336 2024-09-15T21:11:36.238 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T21:11:36.239 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=8 2024-09-15T21:11:36.240 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 8-193273528336 2024-09-15T21:11:36.240 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T21:11:36.241 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=193273528336 2024-09-15T21:11:36.241 INFO:tasks.workunit.client.0.smithi136.stdout:waiting osd.8 seq 193273528336 2024-09-15T21:11:36.241 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.8 seq 193273528336' 2024-09-15T21:11:36.241 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 8 2024-09-15T21:11:36.517 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 193273528336 -lt 193273528336 2024-09-15T21:11:36.518 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T21:11:36.518 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 9-214748364808 2024-09-15T21:11:36.518 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T21:11:36.519 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=9 2024-09-15T21:11:36.519 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 9-214748364808 2024-09-15T21:11:36.519 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T21:11:36.520 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=214748364808 2024-09-15T21:11:36.521 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.9 seq 214748364808' 2024-09-15T21:11:36.521 INFO:tasks.workunit.client.0.smithi136.stdout:waiting osd.9 seq 214748364808 2024-09-15T21:11:36.521 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 9 2024-09-15T21:11:36.798 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 214748364808 -lt 214748364808 2024-09-15T21:11:36.799 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-15T21:11:36.799 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T21:11:36.799 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T21:11:37.148 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 4 == 0 2024-09-15T21:11:37.148 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-15T21:11:37.149 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-15T21:11:37.149 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-15T21:11:37.149 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T21:11:37.149 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T21:11:37.150 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T21:11:37.150 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T21:11:37.461 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=4 2024-09-15T21:11:37.461 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-15T21:11:37.462 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T21:11:37.462 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T21:11:37.810 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 = 4 2024-09-15T21:11:37.811 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-15T21:11:37.811 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-15T21:11:37.811 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:837: corrupt_and_repair_lrc: create_ec_pool ecpool true k=4 m=2 l=3 plugin=lrc 2024-09-15T21:11:37.811 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2443: create_ec_pool: local pool_name=ecpool 2024-09-15T21:11:37.811 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2444: create_ec_pool: shift 2024-09-15T21:11:37.811 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2445: create_ec_pool: local allow_overwrites=true 2024-09-15T21:11:37.811 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2446: create_ec_pool: shift 2024-09-15T21:11:37.811 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2448: create_ec_pool: ceph osd erasure-code-profile set myprofile crush-failure-domain=osd k=4 m=2 l=3 plugin=lrc 2024-09-15T21:11:38.202 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2450: create_ec_pool: create_pool ecpool 1 1 erasure myprofile 2024-09-15T21:11:38.202 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create ecpool 1 1 erasure myprofile 2024-09-15T21:11:38.953 INFO:tasks.workunit.client.0.smithi136.stderr:pool 'ecpool' created 2024-09-15T21:11:38.966 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2024-09-15T21:11:39.968 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2452: create_ec_pool: '[' true = true ']' 2024-09-15T21:11:39.968 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2453: create_ec_pool: ceph osd pool set ecpool allow_ec_overwrites true 2024-09-15T21:11:40.786 INFO:tasks.workunit.client.0.smithi136.stderr:set pool 2 allow_ec_overwrites to true 2024-09-15T21:11:40.800 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2456: create_ec_pool: wait_for_clean 2024-09-15T21:11:40.800 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-15T21:11:40.800 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-15T21:11:40.800 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-15T21:11:40.801 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-15T21:11:40.801 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-15T21:11:40.801 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-15T21:11:40.801 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-15T21:11:40.801 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-15T21:11:40.801 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-15T21:11:40.896 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-15T21:11:40.896 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-15T21:11:40.896 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-15T21:11:40.896 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-15T21:11:40.896 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-15T21:11:40.896 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-15T21:11:41.244 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-15T21:11:41.244 INFO:tasks.workunit.client.0.smithi136.stderr:1 2024-09-15T21:11:41.244 INFO:tasks.workunit.client.0.smithi136.stderr:2 2024-09-15T21:11:41.244 INFO:tasks.workunit.client.0.smithi136.stderr:3 2024-09-15T21:11:41.244 INFO:tasks.workunit.client.0.smithi136.stderr:4 2024-09-15T21:11:41.244 INFO:tasks.workunit.client.0.smithi136.stderr:5 2024-09-15T21:11:41.245 INFO:tasks.workunit.client.0.smithi136.stderr:6 2024-09-15T21:11:41.245 INFO:tasks.workunit.client.0.smithi136.stderr:7 2024-09-15T21:11:41.245 INFO:tasks.workunit.client.0.smithi136.stderr:8 2024-09-15T21:11:41.245 INFO:tasks.workunit.client.0.smithi136.stderr:9' 2024-09-15T21:11:41.245 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-15T21:11:41.245 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T21:11:41.245 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-15T21:11:41.403 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836550 2024-09-15T21:11:41.403 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836550 2024-09-15T21:11:41.403 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836550' 2024-09-15T21:11:41.403 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T21:11:41.403 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-15T21:11:41.521 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=42949673026 2024-09-15T21:11:41.521 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 42949673026 2024-09-15T21:11:41.521 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836550 1-42949673026' 2024-09-15T21:11:41.521 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T21:11:41.521 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-15T21:11:41.624 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=64424509496 2024-09-15T21:11:41.624 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 64424509496 2024-09-15T21:11:41.625 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836550 1-42949673026 2-64424509496' 2024-09-15T21:11:41.625 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T21:11:41.625 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-15T21:11:41.729 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=85899345970 2024-09-15T21:11:41.729 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 85899345970 2024-09-15T21:11:41.729 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836550 1-42949673026 2-64424509496 3-85899345970' 2024-09-15T21:11:41.729 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T21:11:41.729 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-09-15T21:11:41.832 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=107374182445 2024-09-15T21:11:41.832 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 107374182445 2024-09-15T21:11:41.832 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836550 1-42949673026 2-64424509496 3-85899345970 4-107374182445' 2024-09-15T21:11:41.832 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T21:11:41.832 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-09-15T21:11:41.989 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=128849018916 2024-09-15T21:11:41.989 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 128849018916 2024-09-15T21:11:41.989 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836550 1-42949673026 2-64424509496 3-85899345970 4-107374182445 5-128849018916' 2024-09-15T21:11:41.990 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T21:11:41.990 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-09-15T21:11:42.091 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=150323855393 2024-09-15T21:11:42.092 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 150323855393 2024-09-15T21:11:42.092 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836550 1-42949673026 2-64424509496 3-85899345970 4-107374182445 5-128849018916 6-150323855393' 2024-09-15T21:11:42.092 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T21:11:42.092 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2024-09-15T21:11:42.195 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=171798691865 2024-09-15T21:11:42.195 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 171798691865 2024-09-15T21:11:42.195 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836550 1-42949673026 2-64424509496 3-85899345970 4-107374182445 5-128849018916 6-150323855393 7-171798691865' 2024-09-15T21:11:42.195 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T21:11:42.195 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2024-09-15T21:11:42.297 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=193273528339 2024-09-15T21:11:42.297 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 193273528339 2024-09-15T21:11:42.297 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836550 1-42949673026 2-64424509496 3-85899345970 4-107374182445 5-128849018916 6-150323855393 7-171798691865 8-193273528339' 2024-09-15T21:11:42.297 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T21:11:42.298 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2024-09-15T21:11:42.401 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=214748364810 2024-09-15T21:11:42.401 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 214748364810 2024-09-15T21:11:42.401 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836550 1-42949673026 2-64424509496 3-85899345970 4-107374182445 5-128849018916 6-150323855393 7-171798691865 8-193273528339 9-214748364810' 2024-09-15T21:11:42.401 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T21:11:42.401 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836550 2024-09-15T21:11:42.402 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T21:11:42.402 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-15T21:11:42.403 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836550 2024-09-15T21:11:42.403 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T21:11:42.404 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836550 2024-09-15T21:11:42.404 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836550' 2024-09-15T21:11:42.404 INFO:tasks.workunit.client.0.smithi136.stdout:waiting osd.0 seq 21474836550 2024-09-15T21:11:42.404 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T21:11:42.679 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836550 -lt 21474836550 2024-09-15T21:11:42.707 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T21:11:42.708 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-42949673026 2024-09-15T21:11:42.709 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T21:11:42.709 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-15T21:11:42.709 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-42949673026 2024-09-15T21:11:42.709 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T21:11:42.709 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=42949673026 2024-09-15T21:11:42.709 INFO:tasks.workunit.client.0.smithi136.stdout:waiting osd.1 seq 42949673026 2024-09-15T21:11:42.709 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 42949673026' 2024-09-15T21:11:42.710 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-15T21:11:42.977 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 42949673026 -lt 42949673026 2024-09-15T21:11:42.977 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T21:11:42.977 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-64424509496 2024-09-15T21:11:42.977 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T21:11:42.978 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-15T21:11:42.979 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-64424509496 2024-09-15T21:11:42.979 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T21:11:42.980 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=64424509496 2024-09-15T21:11:42.980 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 64424509496' 2024-09-15T21:11:42.980 INFO:tasks.workunit.client.0.smithi136.stdout:waiting osd.2 seq 64424509496 2024-09-15T21:11:42.980 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-15T21:11:43.267 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 64424509496 -lt 64424509496 2024-09-15T21:11:43.267 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T21:11:43.267 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 3-85899345970 2024-09-15T21:11:43.268 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T21:11:43.268 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=3 2024-09-15T21:11:43.269 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 3-85899345970 2024-09-15T21:11:43.269 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T21:11:43.270 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=85899345970 2024-09-15T21:11:43.270 INFO:tasks.workunit.client.0.smithi136.stdout:waiting osd.3 seq 85899345970 2024-09-15T21:11:43.271 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.3 seq 85899345970' 2024-09-15T21:11:43.271 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-15T21:11:43.548 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 85899345970 -lt 85899345970 2024-09-15T21:11:43.548 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T21:11:43.548 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 4-107374182445 2024-09-15T21:11:43.548 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T21:11:43.549 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=4 2024-09-15T21:11:43.549 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 4-107374182445 2024-09-15T21:11:43.550 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T21:11:43.550 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=107374182445 2024-09-15T21:11:43.551 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.4 seq 107374182445' 2024-09-15T21:11:43.551 INFO:tasks.workunit.client.0.smithi136.stdout:waiting osd.4 seq 107374182445 2024-09-15T21:11:43.551 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 4 2024-09-15T21:11:43.873 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 107374182445 -lt 107374182445 2024-09-15T21:11:43.873 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T21:11:43.873 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 5-128849018916 2024-09-15T21:11:43.873 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T21:11:43.874 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=5 2024-09-15T21:11:43.875 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 5-128849018916 2024-09-15T21:11:43.875 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T21:11:43.876 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=128849018916 2024-09-15T21:11:43.876 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.5 seq 128849018916' 2024-09-15T21:11:43.876 INFO:tasks.workunit.client.0.smithi136.stdout:waiting osd.5 seq 128849018916 2024-09-15T21:11:43.876 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 5 2024-09-15T21:11:44.170 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 128849018916 -lt 128849018916 2024-09-15T21:11:44.170 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T21:11:44.170 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 6-150323855393 2024-09-15T21:11:44.170 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T21:11:44.171 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=6 2024-09-15T21:11:44.172 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 6-150323855393 2024-09-15T21:11:44.172 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T21:11:44.173 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=150323855393 2024-09-15T21:11:44.173 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.6 seq 150323855393' 2024-09-15T21:11:44.173 INFO:tasks.workunit.client.0.smithi136.stdout:waiting osd.6 seq 150323855393 2024-09-15T21:11:44.173 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 6 2024-09-15T21:11:44.452 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 150323855393 -lt 150323855393 2024-09-15T21:11:44.452 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T21:11:44.452 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 7-171798691865 2024-09-15T21:11:44.452 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T21:11:44.453 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=7 2024-09-15T21:11:44.454 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 7-171798691865 2024-09-15T21:11:44.454 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T21:11:44.455 INFO:tasks.workunit.client.0.smithi136.stdout:waiting osd.7 seq 171798691865 2024-09-15T21:11:44.455 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=171798691865 2024-09-15T21:11:44.455 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.7 seq 171798691865' 2024-09-15T21:11:44.455 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 7 2024-09-15T21:11:44.735 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 171798691865 -lt 171798691865 2024-09-15T21:11:44.736 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T21:11:44.736 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 8-193273528339 2024-09-15T21:11:44.736 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T21:11:44.737 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=8 2024-09-15T21:11:44.738 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 8-193273528339 2024-09-15T21:11:44.738 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T21:11:44.738 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=193273528339 2024-09-15T21:11:44.739 INFO:tasks.workunit.client.0.smithi136.stdout:waiting osd.8 seq 193273528339 2024-09-15T21:11:44.739 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.8 seq 193273528339' 2024-09-15T21:11:44.739 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 8 2024-09-15T21:11:45.093 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 193273528339 -lt 193273528339 2024-09-15T21:11:45.093 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T21:11:45.094 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 9-214748364810 2024-09-15T21:11:45.094 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T21:11:45.095 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=9 2024-09-15T21:11:45.095 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 9-214748364810 2024-09-15T21:11:45.095 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T21:11:45.096 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=214748364810 2024-09-15T21:11:45.096 INFO:tasks.workunit.client.0.smithi136.stdout:waiting osd.9 seq 214748364810 2024-09-15T21:11:45.097 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.9 seq 214748364810' 2024-09-15T21:11:45.097 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 9 2024-09-15T21:11:45.375 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 214748364811 -lt 214748364810 2024-09-15T21:11:45.375 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-15T21:11:45.376 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T21:11:45.376 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T21:11:45.723 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 5 == 0 2024-09-15T21:11:45.723 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-15T21:11:45.723 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-15T21:11:45.723 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-15T21:11:45.723 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T21:11:45.723 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T21:11:45.723 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T21:11:45.723 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T21:11:45.999 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=5 2024-09-15T21:11:45.999 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-15T21:11:45.999 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T21:11:46.000 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T21:11:46.348 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 = 5 2024-09-15T21:11:46.348 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-15T21:11:46.348 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-15T21:11:46.349 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2457: create_ec_pool: return 0 2024-09-15T21:11:46.349 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:838: corrupt_and_repair_lrc: corrupt_and_repair_erasure_coded td/osd-scrub-repair ecpool 2024-09-15T21:11:46.349 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:248: corrupt_and_repair_erasure_coded: local dir=td/osd-scrub-repair 2024-09-15T21:11:46.349 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:249: corrupt_and_repair_erasure_coded: local poolname=ecpool 2024-09-15T21:11:46.349 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:251: corrupt_and_repair_erasure_coded: add_something td/osd-scrub-repair ecpool 2024-09-15T21:11:46.349 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-09-15T21:11:46.349 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=ecpool 2024-09-15T21:11:46.349 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=SOMETHING 2024-09-15T21:11:46.349 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-09-15T21:11:46.349 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-09-15T21:11:46.349 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-09-15T21:11:46.717 INFO:tasks.workunit.client.0.smithi136.stderr:noscrub is set 2024-09-15T21:11:46.728 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-09-15T21:11:47.254 INFO:tasks.workunit.client.0.smithi136.stderr:nodeep-scrub is set 2024-09-15T21:11:47.266 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-09-15T21:11:47.266 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-09-15T21:11:47.266 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool ecpool put SOMETHING td/osd-scrub-repair/ORIGINAL 2024-09-15T21:11:47.687 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:253: corrupt_and_repair_erasure_coded: get_primary ecpool SOMETHING 2024-09-15T21:11:47.687 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1187: get_primary: local poolname=ecpool 2024-09-15T21:11:47.687 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1188: get_primary: local objectname=SOMETHING 2024-09-15T21:11:47.687 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1190: get_primary: ceph --format json osd map ecpool SOMETHING 2024-09-15T21:11:47.687 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1191: get_primary: jq .acting_primary 2024-09-15T21:11:47.970 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:253: corrupt_and_repair_erasure_coded: local primary=3 2024-09-15T21:11:47.971 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:254: corrupt_and_repair_erasure_coded: get_osds ecpool SOMETHING 2024-09-15T21:11:47.971 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1012: get_osds: local poolname=ecpool 2024-09-15T21:11:47.971 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1013: get_osds: local objectname=SOMETHING 2024-09-15T21:11:47.971 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:254: corrupt_and_repair_erasure_coded: sed -e s/3// 2024-09-15T21:11:47.972 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: ceph --format json osd map ecpool SOMETHING 2024-09-15T21:11:47.972 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: jq '.acting | .[]' 2024-09-15T21:11:48.271 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: local 'osds=3 2024-09-15T21:11:48.272 INFO:tasks.workunit.client.0.smithi136.stderr:5 2024-09-15T21:11:48.272 INFO:tasks.workunit.client.0.smithi136.stderr:9 2024-09-15T21:11:48.272 INFO:tasks.workunit.client.0.smithi136.stderr:0 2024-09-15T21:11:48.272 INFO:tasks.workunit.client.0.smithi136.stderr:6 2024-09-15T21:11:48.272 INFO:tasks.workunit.client.0.smithi136.stderr:2 2024-09-15T21:11:48.272 INFO:tasks.workunit.client.0.smithi136.stderr:1 2024-09-15T21:11:48.272 INFO:tasks.workunit.client.0.smithi136.stderr:7' 2024-09-15T21:11:48.272 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1018: get_osds: echo 3 5 9 0 6 2 1 7 2024-09-15T21:11:48.275 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:254: corrupt_and_repair_erasure_coded: osds=('5' '9' '0' '6' '2' '1' '7') 2024-09-15T21:11:48.275 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:254: corrupt_and_repair_erasure_coded: local -a osds 2024-09-15T21:11:48.275 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:255: corrupt_and_repair_erasure_coded: local not_primary_first=5 2024-09-15T21:11:48.275 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:256: corrupt_and_repair_erasure_coded: local not_primary_second=9 2024-09-15T21:11:48.275 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:259: corrupt_and_repair_erasure_coded: corrupt_and_repair_one td/osd-scrub-repair ecpool 3 2024-09-15T21:11:48.275 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:226: corrupt_and_repair_one: local dir=td/osd-scrub-repair 2024-09-15T21:11:48.275 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:227: corrupt_and_repair_one: local poolname=ecpool 2024-09-15T21:11:48.275 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:228: corrupt_and_repair_one: local osd=3 2024-09-15T21:11:48.275 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:233: corrupt_and_repair_one: objectstore_tool td/osd-scrub-repair 3 SOMETHING remove 2024-09-15T21:11:48.275 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2024-09-15T21:11:48.275 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2024-09-15T21:11:48.275 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=3 2024-09-15T21:11:48.276 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2024-09-15T21:11:48.276 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 3 SOMETHING remove 2024-09-15T21:11:48.276 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-09-15T21:11:48.276 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2024-09-15T21:11:48.276 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=3 2024-09-15T21:11:48.276 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2024-09-15T21:11:48.276 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.3 2024-09-15T21:11:48.276 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-15T21:11:48.276 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-15T21:11:48.276 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-15T21:11:48.276 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-15T21:11:48.276 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-15T21:11:48.580 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-15T21:11:48.580 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 3 SOMETHING remove 2024-09-15T21:11:48.580 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-15T21:11:48.580 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-15T21:11:48.580 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=3 2024-09-15T21:11:48.580 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-15T21:11:48.580 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/3 2024-09-15T21:11:48.580 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/3 SOMETHING remove 2024-09-15T21:11:50.588 INFO:tasks.workunit.client.0.smithi136.stdout:remove 0#2:eb822e21:::SOMETHING:head# 2024-09-15T21:11:51.104 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 3 2024-09-15T21:11:51.105 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2024-09-15T21:11:51.105 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-15T21:11:51.105 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=3 2024-09-15T21:11:51.105 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-15T21:11:51.105 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/3 2024-09-15T21:11:51.105 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=ce4892c7-6c6d-4b4f-8e6f-07a3e11133e5 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-15T21:11:51.105 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T21:11:51.105 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T21:11:51.105 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T21:11:51.105 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/3' 2024-09-15T21:11:51.105 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/3/journal' 2024-09-15T21:11:51.105 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-15T21:11:51.105 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-15T21:11:51.106 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-15T21:11:51.106 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-15T21:11:51.106 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-15T21:11:51.106 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-15T21:11:51.106 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-15T21:11:51.106 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T21:11:51.106 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52602 2024-09-15T21:11:51.106 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.52602/$cluster-$name.asok' 2024-09-15T21:11:51.106 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.52602/$cluster-$name.asok' 2024-09-15T21:11:51.106 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-15T21:11:51.106 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-15T21:11:51.106 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-15T21:11:51.106 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T21:11:51.106 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T21:11:51.107 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T21:11:51.107 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T21:11:51.107 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-15T21:11:51.107 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-15T21:11:51.107 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/3 2024-09-15T21:11:51.108 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.3 2024-09-15T21:11:51.108 INFO:tasks.workunit.client.0.smithi136.stderr:start osd.3 2024-09-15T21:11:51.108 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 3 --fsid=ce4892c7-6c6d-4b4f-8e6f-07a3e11133e5 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/3 --osd-journal=td/osd-scrub-repair/3/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.52602/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T21:11:51.108 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/3/whoami 2024-09-15T21:11:51.108 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 3 = 3 ']' 2024-09-15T21:11:51.108 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-15T21:11:51.108 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-15T21:11:51.108 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-15T21:11:51.136 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:11:51.132+0000 7fd37920c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:11:51.139 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:11:51.135+0000 7fd37920c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:11:51.140 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:11:51.136+0000 7fd37920c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:11:51.395 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 3 2024-09-15T21:11:51.396 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-15T21:11:51.396 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=3 2024-09-15T21:11:51.396 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-15T21:11:51.396 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-15T21:11:51.396 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:11:51.396 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-15T21:11:51.396 INFO:tasks.workunit.client.0.smithi136.stderr:0 2024-09-15T21:11:51.396 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:11:51.396 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-15T21:11:51.675 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:11:52.676 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:11:52.677 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:11:52.677 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-15T21:11:52.677 INFO:tasks.workunit.client.0.smithi136.stderr:1 2024-09-15T21:11:52.677 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:11:52.677 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-15T21:11:52.736 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:11:52.732+0000 7fd37920c640 -1 Falling back to public interface 2024-09-15T21:11:52.988 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:11:53.990 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:11:53.991 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:11:53.991 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-15T21:11:53.991 INFO:tasks.workunit.client.0.smithi136.stderr:2 2024-09-15T21:11:53.991 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:11:53.991 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-15T21:11:54.274 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:11:55.275 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:11:55.276 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:11:55.276 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-15T21:11:55.276 INFO:tasks.workunit.client.0.smithi136.stderr:3 2024-09-15T21:11:55.276 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:11:55.276 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-15T21:11:55.443 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:11:55.440+0000 7fd37920c640 -1 osd.3 69 log_to_monitors true 2024-09-15T21:11:55.562 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:11:56.564 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:11:56.564 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:11:56.564 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-15T21:11:56.564 INFO:tasks.workunit.client.0.smithi136.stderr:4 2024-09-15T21:11:56.564 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:11:56.564 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-15T21:11:56.861 INFO:tasks.workunit.client.0.smithi136.stderr:osd.3 up in weight 1 up_from 73 up_thru 73 down_at 70 last_clean_interval [20,69) [v2:127.0.0.1:6826/3517695937,v1:127.0.0.1:6827/3517695937] [v2:127.0.0.1:6828/3517695937,v1:127.0.0.1:6829/3517695937] exists,up 0e7c66ad-8619-49df-8643-12288ef1d7ba 2024-09-15T21:11:56.861 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-15T21:11:56.861 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-15T21:11:56.861 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-15T21:11:56.861 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2024-09-15T21:11:56.861 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-15T21:11:56.861 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-15T21:11:56.861 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-15T21:11:56.861 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-15T21:11:56.862 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-15T21:11:56.862 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-15T21:11:56.862 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-15T21:11:56.862 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-15T21:11:56.862 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-15T21:11:56.951 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-15T21:11:56.952 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-15T21:11:56.952 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-15T21:11:56.952 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-15T21:11:56.952 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-15T21:11:56.952 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-15T21:11:57.236 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-15T21:11:57.236 INFO:tasks.workunit.client.0.smithi136.stderr:1 2024-09-15T21:11:57.236 INFO:tasks.workunit.client.0.smithi136.stderr:2 2024-09-15T21:11:57.236 INFO:tasks.workunit.client.0.smithi136.stderr:3 2024-09-15T21:11:57.236 INFO:tasks.workunit.client.0.smithi136.stderr:4 2024-09-15T21:11:57.237 INFO:tasks.workunit.client.0.smithi136.stderr:5 2024-09-15T21:11:57.237 INFO:tasks.workunit.client.0.smithi136.stderr:6 2024-09-15T21:11:57.237 INFO:tasks.workunit.client.0.smithi136.stderr:7 2024-09-15T21:11:57.237 INFO:tasks.workunit.client.0.smithi136.stderr:8 2024-09-15T21:11:57.237 INFO:tasks.workunit.client.0.smithi136.stderr:9' 2024-09-15T21:11:57.237 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-15T21:11:57.237 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T21:11:57.237 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-15T21:11:57.376 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836554 2024-09-15T21:11:57.376 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836554 2024-09-15T21:11:57.376 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836554' 2024-09-15T21:11:57.376 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T21:11:57.376 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-15T21:11:57.643 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=42949673030 2024-09-15T21:11:57.643 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 42949673030 2024-09-15T21:11:57.644 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836554 1-42949673030' 2024-09-15T21:11:57.644 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T21:11:57.644 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-15T21:11:57.746 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=64424509500 2024-09-15T21:11:57.746 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 64424509500 2024-09-15T21:11:57.746 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836554 1-42949673030 2-64424509500' 2024-09-15T21:11:57.746 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T21:11:57.747 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-15T21:11:57.850 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=313532612610 2024-09-15T21:11:57.850 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 313532612610 2024-09-15T21:11:57.850 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836554 1-42949673030 2-64424509500 3-313532612610' 2024-09-15T21:11:57.850 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T21:11:57.850 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-09-15T21:11:57.955 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=107374182449 2024-09-15T21:11:57.955 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 107374182449 2024-09-15T21:11:57.955 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836554 1-42949673030 2-64424509500 3-313532612610 4-107374182449' 2024-09-15T21:11:57.955 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T21:11:57.955 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-09-15T21:11:58.079 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=128849018920 2024-09-15T21:11:58.079 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 128849018920 2024-09-15T21:11:58.079 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836554 1-42949673030 2-64424509500 3-313532612610 4-107374182449 5-128849018920' 2024-09-15T21:11:58.079 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T21:11:58.079 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-09-15T21:11:58.183 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=150323855397 2024-09-15T21:11:58.183 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 150323855397 2024-09-15T21:11:58.183 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836554 1-42949673030 2-64424509500 3-313532612610 4-107374182449 5-128849018920 6-150323855397' 2024-09-15T21:11:58.183 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T21:11:58.183 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2024-09-15T21:11:58.285 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=171798691869 2024-09-15T21:11:58.286 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 171798691869 2024-09-15T21:11:58.286 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836554 1-42949673030 2-64424509500 3-313532612610 4-107374182449 5-128849018920 6-150323855397 7-171798691869' 2024-09-15T21:11:58.286 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T21:11:58.286 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2024-09-15T21:11:58.388 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=193273528343 2024-09-15T21:11:58.388 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 193273528343 2024-09-15T21:11:58.388 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836554 1-42949673030 2-64424509500 3-313532612610 4-107374182449 5-128849018920 6-150323855397 7-171798691869 8-193273528343' 2024-09-15T21:11:58.388 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T21:11:58.389 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2024-09-15T21:11:58.491 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=214748364815 2024-09-15T21:11:58.491 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 214748364815 2024-09-15T21:11:58.491 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836554 1-42949673030 2-64424509500 3-313532612610 4-107374182449 5-128849018920 6-150323855397 7-171798691869 8-193273528343 9-214748364815' 2024-09-15T21:11:58.492 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T21:11:58.492 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836554 2024-09-15T21:11:58.492 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T21:11:58.493 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-15T21:11:58.493 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836554 2024-09-15T21:11:58.493 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T21:11:58.494 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836554 2024-09-15T21:11:58.494 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836554' 2024-09-15T21:11:58.494 INFO:tasks.workunit.client.0.smithi136.stderr:waiting osd.0 seq 21474836554 2024-09-15T21:11:58.495 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T21:11:58.773 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836554 -lt 21474836554 2024-09-15T21:11:58.773 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T21:11:58.773 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-42949673030 2024-09-15T21:11:58.773 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T21:11:58.774 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-15T21:11:58.775 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-42949673030 2024-09-15T21:11:58.775 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T21:11:58.776 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=42949673030 2024-09-15T21:11:58.776 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 42949673030' 2024-09-15T21:11:58.776 INFO:tasks.workunit.client.0.smithi136.stderr:waiting osd.1 seq 42949673030 2024-09-15T21:11:58.776 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-15T21:11:59.074 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 42949673030 -lt 42949673030 2024-09-15T21:11:59.074 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T21:11:59.074 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-64424509500 2024-09-15T21:11:59.074 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T21:11:59.075 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-15T21:11:59.076 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-64424509500 2024-09-15T21:11:59.076 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T21:11:59.077 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=64424509500 2024-09-15T21:11:59.077 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 64424509500' 2024-09-15T21:11:59.077 INFO:tasks.workunit.client.0.smithi136.stderr:waiting osd.2 seq 64424509500 2024-09-15T21:11:59.077 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-15T21:11:59.353 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 64424509500 -lt 64424509500 2024-09-15T21:11:59.353 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T21:11:59.354 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 3-313532612610 2024-09-15T21:11:59.354 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T21:11:59.354 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=3 2024-09-15T21:11:59.355 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 3-313532612610 2024-09-15T21:11:59.355 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T21:11:59.356 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=313532612610 2024-09-15T21:11:59.356 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.3 seq 313532612610' 2024-09-15T21:11:59.356 INFO:tasks.workunit.client.0.smithi136.stderr:waiting osd.3 seq 313532612610 2024-09-15T21:11:59.357 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-15T21:11:59.636 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 313532612610 -lt 313532612610 2024-09-15T21:11:59.636 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T21:11:59.636 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 4-107374182449 2024-09-15T21:11:59.637 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T21:11:59.637 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=4 2024-09-15T21:11:59.638 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 4-107374182449 2024-09-15T21:11:59.638 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T21:11:59.639 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=107374182449 2024-09-15T21:11:59.639 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.4 seq 107374182449' 2024-09-15T21:11:59.639 INFO:tasks.workunit.client.0.smithi136.stderr:waiting osd.4 seq 107374182449 2024-09-15T21:11:59.639 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 4 2024-09-15T21:11:59.925 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 107374182448 -lt 107374182449 2024-09-15T21:11:59.925 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-15T21:12:00.926 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-15T21:12:00.927 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 4 2024-09-15T21:12:01.215 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 107374182449 -lt 107374182449 2024-09-15T21:12:01.215 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T21:12:01.215 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 5-128849018920 2024-09-15T21:12:01.215 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T21:12:01.216 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=5 2024-09-15T21:12:01.217 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 5-128849018920 2024-09-15T21:12:01.217 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T21:12:01.218 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=128849018920 2024-09-15T21:12:01.218 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.5 seq 128849018920' 2024-09-15T21:12:01.218 INFO:tasks.workunit.client.0.smithi136.stderr:waiting osd.5 seq 128849018920 2024-09-15T21:12:01.218 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 5 2024-09-15T21:12:01.498 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 128849018920 -lt 128849018920 2024-09-15T21:12:01.498 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T21:12:01.499 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 6-150323855397 2024-09-15T21:12:01.499 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T21:12:01.499 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=6 2024-09-15T21:12:01.500 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 6-150323855397 2024-09-15T21:12:01.500 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T21:12:01.501 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=150323855397 2024-09-15T21:12:01.501 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.6 seq 150323855397' 2024-09-15T21:12:01.501 INFO:tasks.workunit.client.0.smithi136.stderr:waiting osd.6 seq 150323855397 2024-09-15T21:12:01.502 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 6 2024-09-15T21:12:01.779 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 150323855397 -lt 150323855397 2024-09-15T21:12:01.779 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T21:12:01.780 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 7-171798691869 2024-09-15T21:12:01.780 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T21:12:01.780 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=7 2024-09-15T21:12:01.781 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 7-171798691869 2024-09-15T21:12:01.781 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T21:12:01.782 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=171798691869 2024-09-15T21:12:01.782 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.7 seq 171798691869' 2024-09-15T21:12:01.782 INFO:tasks.workunit.client.0.smithi136.stderr:waiting osd.7 seq 171798691869 2024-09-15T21:12:01.783 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 7 2024-09-15T21:12:02.063 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 171798691870 -lt 171798691869 2024-09-15T21:12:02.063 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T21:12:02.064 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 8-193273528343 2024-09-15T21:12:02.064 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T21:12:02.065 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=8 2024-09-15T21:12:02.065 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 8-193273528343 2024-09-15T21:12:02.065 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T21:12:02.066 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=193273528343 2024-09-15T21:12:02.066 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.8 seq 193273528343' 2024-09-15T21:12:02.066 INFO:tasks.workunit.client.0.smithi136.stderr:waiting osd.8 seq 193273528343 2024-09-15T21:12:02.067 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 8 2024-09-15T21:12:02.356 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 193273528344 -lt 193273528343 2024-09-15T21:12:02.356 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T21:12:02.357 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 9-214748364815 2024-09-15T21:12:02.357 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T21:12:02.357 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=9 2024-09-15T21:12:02.358 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 9-214748364815 2024-09-15T21:12:02.358 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T21:12:02.359 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=214748364815 2024-09-15T21:12:02.359 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.9 seq 214748364815' 2024-09-15T21:12:02.359 INFO:tasks.workunit.client.0.smithi136.stderr:waiting osd.9 seq 214748364815 2024-09-15T21:12:02.360 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 9 2024-09-15T21:12:02.638 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 214748364815 -lt 214748364815 2024-09-15T21:12:02.638 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-15T21:12:02.639 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T21:12:02.639 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T21:12:03.003 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 5 == 0 2024-09-15T21:12:03.003 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-15T21:12:03.003 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-15T21:12:03.003 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-15T21:12:03.003 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T21:12:03.003 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T21:12:03.003 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T21:12:03.003 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T21:12:03.279 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=5 2024-09-15T21:12:03.279 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-15T21:12:03.279 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T21:12:03.279 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T21:12:03.629 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 = 5 2024-09-15T21:12:03.629 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-15T21:12:03.629 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-15T21:12:03.630 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:237: corrupt_and_repair_one: get_pg ecpool SOMETHING 2024-09-15T21:12:03.630 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1084: get_pg: local poolname=ecpool 2024-09-15T21:12:03.630 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1085: get_pg: local objectname=SOMETHING 2024-09-15T21:12:03.630 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1087: get_pg: ceph --format json osd map ecpool SOMETHING 2024-09-15T21:12:03.630 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1087: get_pg: jq -r .pgid 2024-09-15T21:12:03.936 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:237: corrupt_and_repair_one: local pg=2.0 2024-09-15T21:12:03.936 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:238: corrupt_and_repair_one: repair 2.0 2024-09-15T21:12:03.936 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1846: repair: local pgid=2.0 2024-09-15T21:12:03.936 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1847: repair: get_last_scrub_stamp 2.0 2024-09-15T21:12:03.936 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2024-09-15T21:12:03.936 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-15T21:12:03.936 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-15T21:12:03.937 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-09-15T21:12:04.212 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1847: repair: local last_scrub=2024-09-15T21:11:38.833548+0000 2024-09-15T21:12:04.212 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1848: repair: ceph pg repair 2.0 2024-09-15T21:12:04.475 INFO:tasks.workunit.client.0.smithi136.stderr:instructing pg 2.0s0 on osd.3 to repair 2024-09-15T21:12:04.487 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1849: repair: wait_for_scrub 2.0 2024-09-15T21:11:38.833548+0000 2024-09-15T21:12:04.487 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: local pgid=2.0 2024-09-15T21:12:04.488 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2018: wait_for_scrub: local last_scrub=2024-09-15T21:11:38.833548+0000 2024-09-15T21:12:04.488 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2019: wait_for_scrub: local sname=last_scrub_stamp 2024-09-15T21:12:04.488 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i=0 )) 2024-09-15T21:12:04.488 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-15T21:12:04.488 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-09-15T21:12:04.488 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2024-09-15T21:12:04.488 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-15T21:12:04.488 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-15T21:12:04.488 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-09-15T21:12:04.760 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-15T21:11:38.833548+0000 '>' 2024-09-15T21:11:38.833548+0000 2024-09-15T21:12:04.760 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-15T21:12:05.761 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-15T21:12:05.761 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-15T21:12:05.761 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-09-15T21:12:05.761 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2024-09-15T21:12:05.762 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-15T21:12:05.762 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-15T21:12:05.762 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-09-15T21:12:06.045 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-15T21:12:05.495858+0000 '>' 2024-09-15T21:11:38.833548+0000 2024-09-15T21:12:06.045 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2023: wait_for_scrub: return 0 2024-09-15T21:12:06.045 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1850: repair: sleep 2 2024-09-15T21:12:08.047 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:242: corrupt_and_repair_one: objectstore_tool td/osd-scrub-repair 9 SOMETHING list-attrs 2024-09-15T21:12:08.048 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2024-09-15T21:12:08.048 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2024-09-15T21:12:08.048 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=9 2024-09-15T21:12:08.048 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2024-09-15T21:12:08.048 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 9 SOMETHING list-attrs 2024-09-15T21:12:08.048 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-09-15T21:12:08.048 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2024-09-15T21:12:08.048 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=9 2024-09-15T21:12:08.048 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2024-09-15T21:12:08.048 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.9 2024-09-15T21:12:08.048 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-15T21:12:08.048 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-15T21:12:08.049 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-15T21:12:08.049 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-15T21:12:08.049 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-15T21:12:08.356 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-15T21:12:08.357 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 9 SOMETHING list-attrs 2024-09-15T21:12:08.357 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-15T21:12:08.357 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-15T21:12:08.357 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=9 2024-09-15T21:12:08.357 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-15T21:12:08.357 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/9 2024-09-15T21:12:08.357 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/9 SOMETHING list-attrs 2024-09-15T21:12:08.853 INFO:tasks.workunit.client.0.smithi136.stdout:_ 2024-09-15T21:12:08.853 INFO:tasks.workunit.client.0.smithi136.stdout:hinfo_key 2024-09-15T21:12:08.853 INFO:tasks.workunit.client.0.smithi136.stdout:snapset 2024-09-15T21:12:09.123 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 9 2024-09-15T21:12:09.123 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2024-09-15T21:12:09.123 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-15T21:12:09.123 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=9 2024-09-15T21:12:09.123 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-15T21:12:09.123 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/9 2024-09-15T21:12:09.124 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=ce4892c7-6c6d-4b4f-8e6f-07a3e11133e5 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-15T21:12:09.124 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T21:12:09.124 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T21:12:09.124 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T21:12:09.124 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/9' 2024-09-15T21:12:09.124 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/9/journal' 2024-09-15T21:12:09.124 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-15T21:12:09.124 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-15T21:12:09.124 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-15T21:12:09.124 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-15T21:12:09.124 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-15T21:12:09.124 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-15T21:12:09.124 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-15T21:12:09.125 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T21:12:09.125 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52602 2024-09-15T21:12:09.125 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.52602/$cluster-$name.asok' 2024-09-15T21:12:09.125 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.52602/$cluster-$name.asok' 2024-09-15T21:12:09.125 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-15T21:12:09.125 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-15T21:12:09.125 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-15T21:12:09.125 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T21:12:09.125 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T21:12:09.125 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T21:12:09.125 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T21:12:09.126 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-15T21:12:09.126 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-15T21:12:09.126 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/9 2024-09-15T21:12:09.126 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.9 2024-09-15T21:12:09.126 INFO:tasks.workunit.client.0.smithi136.stderr:start osd.9 2024-09-15T21:12:09.126 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 9 --fsid=ce4892c7-6c6d-4b4f-8e6f-07a3e11133e5 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/9 --osd-journal=td/osd-scrub-repair/9/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.52602/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T21:12:09.127 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/9/whoami 2024-09-15T21:12:09.127 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 9 = 9 ']' 2024-09-15T21:12:09.127 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-15T21:12:09.127 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-15T21:12:09.127 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-15T21:12:09.154 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:12:09.151+0000 7f6ab2e18640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:12:09.284 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:12:09.281+0000 7f6ab2e18640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:12:09.285 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:12:09.282+0000 7f6ab2e18640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:12:09.487 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 9 2024-09-15T21:12:09.488 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-15T21:12:09.488 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=9 2024-09-15T21:12:09.488 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-15T21:12:09.488 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-15T21:12:09.488 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:12:09.488 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-15T21:12:09.488 INFO:tasks.workunit.client.0.smithi136.stderr:0 2024-09-15T21:12:09.488 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:12:09.488 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.9 up' 2024-09-15T21:12:09.806 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:12:10.808 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:12:10.808 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:12:10.808 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-15T21:12:10.808 INFO:tasks.workunit.client.0.smithi136.stderr:1 2024-09-15T21:12:10.808 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:12:10.809 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.9 up' 2024-09-15T21:12:11.121 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:12:11.152 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:12:11.149+0000 7f6ab2e18640 -1 Falling back to public interface 2024-09-15T21:12:12.123 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:12:12.123 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:12:12.123 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-15T21:12:12.123 INFO:tasks.workunit.client.0.smithi136.stderr:2 2024-09-15T21:12:12.123 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:12:12.123 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.9 up' 2024-09-15T21:12:12.408 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:12:13.410 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:12:13.410 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:12:13.410 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-15T21:12:13.411 INFO:tasks.workunit.client.0.smithi136.stderr:3 2024-09-15T21:12:13.411 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:12:13.411 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.9 up' 2024-09-15T21:12:13.699 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:12:14.167 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:12:14.164+0000 7f6ab2e18640 -1 osd.9 74 log_to_monitors true 2024-09-15T21:12:14.700 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:12:14.701 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:12:14.701 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-15T21:12:14.701 INFO:tasks.workunit.client.0.smithi136.stderr:4 2024-09-15T21:12:14.701 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:12:14.701 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.9 up' 2024-09-15T21:12:15.146 INFO:tasks.workunit.client.0.smithi136.stderr:osd.9 up in weight 1 up_from 78 up_thru 0 down_at 75 last_clean_interval [50,74) [v2:127.0.0.1:6874/556688969,v1:127.0.0.1:6875/556688969] [v2:127.0.0.1:6876/556688969,v1:127.0.0.1:6877/556688969] exists,up 78ecdb67-c9df-44e1-b77a-4e850e67a453 2024-09-15T21:12:15.147 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-15T21:12:15.147 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-15T21:12:15.147 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-15T21:12:15.147 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2024-09-15T21:12:15.147 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-15T21:12:15.147 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-15T21:12:15.147 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-15T21:12:15.147 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-15T21:12:15.148 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-15T21:12:15.148 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-15T21:12:15.148 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-15T21:12:15.148 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-15T21:12:15.148 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-15T21:12:15.238 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-15T21:12:15.238 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-15T21:12:15.238 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-15T21:12:15.238 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-15T21:12:15.238 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-15T21:12:15.238 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-15T21:12:15.695 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-15T21:12:15.695 INFO:tasks.workunit.client.0.smithi136.stderr:1 2024-09-15T21:12:15.695 INFO:tasks.workunit.client.0.smithi136.stderr:2 2024-09-15T21:12:15.695 INFO:tasks.workunit.client.0.smithi136.stderr:3 2024-09-15T21:12:15.695 INFO:tasks.workunit.client.0.smithi136.stderr:4 2024-09-15T21:12:15.695 INFO:tasks.workunit.client.0.smithi136.stderr:5 2024-09-15T21:12:15.695 INFO:tasks.workunit.client.0.smithi136.stderr:6 2024-09-15T21:12:15.695 INFO:tasks.workunit.client.0.smithi136.stderr:7 2024-09-15T21:12:15.695 INFO:tasks.workunit.client.0.smithi136.stderr:8 2024-09-15T21:12:15.695 INFO:tasks.workunit.client.0.smithi136.stderr:9' 2024-09-15T21:12:15.695 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-15T21:12:15.696 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T21:12:15.696 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-15T21:12:15.829 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836559 2024-09-15T21:12:15.829 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836559 2024-09-15T21:12:15.829 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836559' 2024-09-15T21:12:15.829 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T21:12:15.829 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-15T21:12:16.309 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=42949673035 2024-09-15T21:12:16.309 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 42949673035 2024-09-15T21:12:16.309 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836559 1-42949673035' 2024-09-15T21:12:16.309 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T21:12:16.310 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-15T21:12:16.413 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=64424509505 2024-09-15T21:12:16.413 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 64424509505 2024-09-15T21:12:16.413 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836559 1-42949673035 2-64424509505' 2024-09-15T21:12:16.413 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T21:12:16.413 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-15T21:12:16.518 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=313532612615 2024-09-15T21:12:16.518 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 313532612615 2024-09-15T21:12:16.518 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836559 1-42949673035 2-64424509505 3-313532612615' 2024-09-15T21:12:16.518 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T21:12:16.518 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-09-15T21:12:16.622 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=107374182454 2024-09-15T21:12:16.622 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 107374182454 2024-09-15T21:12:16.622 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836559 1-42949673035 2-64424509505 3-313532612615 4-107374182454' 2024-09-15T21:12:16.623 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T21:12:16.623 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-09-15T21:12:16.726 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=128849018925 2024-09-15T21:12:16.726 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 128849018925 2024-09-15T21:12:16.726 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836559 1-42949673035 2-64424509505 3-313532612615 4-107374182454 5-128849018925' 2024-09-15T21:12:16.726 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T21:12:16.726 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-09-15T21:12:16.828 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=150323855402 2024-09-15T21:12:16.829 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 150323855402 2024-09-15T21:12:16.829 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836559 1-42949673035 2-64424509505 3-313532612615 4-107374182454 5-128849018925 6-150323855402' 2024-09-15T21:12:16.829 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T21:12:16.829 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2024-09-15T21:12:16.951 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=171798691874 2024-09-15T21:12:16.951 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 171798691874 2024-09-15T21:12:16.951 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836559 1-42949673035 2-64424509505 3-313532612615 4-107374182454 5-128849018925 6-150323855402 7-171798691874' 2024-09-15T21:12:16.951 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T21:12:16.951 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2024-09-15T21:12:17.054 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=193273528348 2024-09-15T21:12:17.054 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 193273528348 2024-09-15T21:12:17.054 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836559 1-42949673035 2-64424509505 3-313532612615 4-107374182454 5-128849018925 6-150323855402 7-171798691874 8-193273528348' 2024-09-15T21:12:17.054 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T21:12:17.054 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2024-09-15T21:12:17.159 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=335007449090 2024-09-15T21:12:17.159 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 335007449090 2024-09-15T21:12:17.159 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836559 1-42949673035 2-64424509505 3-313532612615 4-107374182454 5-128849018925 6-150323855402 7-171798691874 8-193273528348 9-335007449090' 2024-09-15T21:12:17.159 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T21:12:17.159 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836559 2024-09-15T21:12:17.160 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T21:12:17.160 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-15T21:12:17.161 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836559 2024-09-15T21:12:17.161 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T21:12:17.162 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836559 2024-09-15T21:12:17.162 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836559' 2024-09-15T21:12:17.162 INFO:tasks.workunit.client.0.smithi136.stderr:waiting osd.0 seq 21474836559 2024-09-15T21:12:17.162 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T21:12:17.437 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836559 -lt 21474836559 2024-09-15T21:12:17.437 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T21:12:17.438 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-42949673035 2024-09-15T21:12:17.438 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T21:12:17.439 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-15T21:12:17.439 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-42949673035 2024-09-15T21:12:17.439 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T21:12:17.440 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=42949673035 2024-09-15T21:12:17.440 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 42949673035' 2024-09-15T21:12:17.441 INFO:tasks.workunit.client.0.smithi136.stderr:waiting osd.1 seq 42949673035 2024-09-15T21:12:17.441 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-15T21:12:17.719 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 42949673034 -lt 42949673035 2024-09-15T21:12:17.719 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-15T21:12:18.720 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-15T21:12:18.720 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-15T21:12:19.007 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 42949673035 -lt 42949673035 2024-09-15T21:12:19.008 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T21:12:19.008 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-64424509505 2024-09-15T21:12:19.008 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T21:12:19.009 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-15T21:12:19.010 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-64424509505 2024-09-15T21:12:19.010 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T21:12:19.011 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=64424509505 2024-09-15T21:12:19.011 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 64424509505' 2024-09-15T21:12:19.011 INFO:tasks.workunit.client.0.smithi136.stderr:waiting osd.2 seq 64424509505 2024-09-15T21:12:19.011 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-15T21:12:19.286 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 64424509505 -lt 64424509505 2024-09-15T21:12:19.286 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T21:12:19.286 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 3-313532612615 2024-09-15T21:12:19.286 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T21:12:19.287 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=3 2024-09-15T21:12:19.288 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 3-313532612615 2024-09-15T21:12:19.288 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T21:12:19.289 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=313532612615 2024-09-15T21:12:19.289 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.3 seq 313532612615' 2024-09-15T21:12:19.289 INFO:tasks.workunit.client.0.smithi136.stderr:waiting osd.3 seq 313532612615 2024-09-15T21:12:19.289 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-15T21:12:19.566 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 313532612615 -lt 313532612615 2024-09-15T21:12:19.566 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T21:12:19.566 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 4-107374182454 2024-09-15T21:12:19.567 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T21:12:19.567 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=4 2024-09-15T21:12:19.568 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 4-107374182454 2024-09-15T21:12:19.568 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T21:12:19.569 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=107374182454 2024-09-15T21:12:19.569 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.4 seq 107374182454' 2024-09-15T21:12:19.569 INFO:tasks.workunit.client.0.smithi136.stderr:waiting osd.4 seq 107374182454 2024-09-15T21:12:19.569 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 4 2024-09-15T21:12:19.850 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 107374182454 -lt 107374182454 2024-09-15T21:12:19.850 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T21:12:19.851 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 5-128849018925 2024-09-15T21:12:19.851 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T21:12:19.851 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=5 2024-09-15T21:12:19.852 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 5-128849018925 2024-09-15T21:12:19.852 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T21:12:19.853 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=128849018925 2024-09-15T21:12:19.854 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.5 seq 128849018925' 2024-09-15T21:12:19.854 INFO:tasks.workunit.client.0.smithi136.stderr:waiting osd.5 seq 128849018925 2024-09-15T21:12:19.854 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 5 2024-09-15T21:12:20.130 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 128849018925 -lt 128849018925 2024-09-15T21:12:20.131 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T21:12:20.131 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 6-150323855402 2024-09-15T21:12:20.131 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T21:12:20.132 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=6 2024-09-15T21:12:20.132 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 6-150323855402 2024-09-15T21:12:20.133 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T21:12:20.133 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=150323855402 2024-09-15T21:12:20.133 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.6 seq 150323855402' 2024-09-15T21:12:20.134 INFO:tasks.workunit.client.0.smithi136.stderr:waiting osd.6 seq 150323855402 2024-09-15T21:12:20.134 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 6 2024-09-15T21:12:20.421 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 150323855402 -lt 150323855402 2024-09-15T21:12:20.421 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T21:12:20.422 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 7-171798691874 2024-09-15T21:12:20.422 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T21:12:20.423 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=7 2024-09-15T21:12:20.423 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 7-171798691874 2024-09-15T21:12:20.423 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T21:12:20.425 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=171798691874 2024-09-15T21:12:20.425 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.7 seq 171798691874' 2024-09-15T21:12:20.425 INFO:tasks.workunit.client.0.smithi136.stderr:waiting osd.7 seq 171798691874 2024-09-15T21:12:20.425 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 7 2024-09-15T21:12:20.706 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 171798691874 -lt 171798691874 2024-09-15T21:12:20.706 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T21:12:20.707 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 8-193273528348 2024-09-15T21:12:20.707 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T21:12:20.707 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=8 2024-09-15T21:12:20.708 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 8-193273528348 2024-09-15T21:12:20.708 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T21:12:20.709 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=193273528348 2024-09-15T21:12:20.709 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.8 seq 193273528348' 2024-09-15T21:12:20.709 INFO:tasks.workunit.client.0.smithi136.stderr:waiting osd.8 seq 193273528348 2024-09-15T21:12:20.710 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 8 2024-09-15T21:12:21.005 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 193273528348 -lt 193273528348 2024-09-15T21:12:21.005 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T21:12:21.006 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 9-335007449090 2024-09-15T21:12:21.006 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T21:12:21.007 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=9 2024-09-15T21:12:21.008 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 9-335007449090 2024-09-15T21:12:21.008 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T21:12:21.008 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=335007449090 2024-09-15T21:12:21.009 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.9 seq 335007449090' 2024-09-15T21:12:21.009 INFO:tasks.workunit.client.0.smithi136.stderr:waiting osd.9 seq 335007449090 2024-09-15T21:12:21.009 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 9 2024-09-15T21:12:21.362 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 335007449091 -lt 335007449090 2024-09-15T21:12:21.363 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-15T21:12:21.363 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T21:12:21.363 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T21:12:21.733 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 5 == 0 2024-09-15T21:12:21.733 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-15T21:12:21.734 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-15T21:12:21.734 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-15T21:12:21.734 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T21:12:21.734 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T21:12:21.735 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T21:12:21.735 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T21:12:22.009 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=5 2024-09-15T21:12:22.010 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-15T21:12:22.010 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T21:12:22.010 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T21:12:22.353 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 = 5 2024-09-15T21:12:22.353 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-15T21:12:22.354 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-15T21:12:22.354 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:243: corrupt_and_repair_one: rados --pool ecpool get SOMETHING td/osd-scrub-repair/COPY 2024-09-15T21:12:22.384 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:244: corrupt_and_repair_one: diff td/osd-scrub-repair/ORIGINAL td/osd-scrub-repair/COPY 2024-09-15T21:12:22.387 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:261: corrupt_and_repair_erasure_coded: corrupt_and_repair_one td/osd-scrub-repair ecpool 5 2024-09-15T21:12:22.387 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:226: corrupt_and_repair_one: local dir=td/osd-scrub-repair 2024-09-15T21:12:22.387 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:227: corrupt_and_repair_one: local poolname=ecpool 2024-09-15T21:12:22.387 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:228: corrupt_and_repair_one: local osd=5 2024-09-15T21:12:22.387 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:233: corrupt_and_repair_one: objectstore_tool td/osd-scrub-repair 5 SOMETHING remove 2024-09-15T21:12:22.387 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2024-09-15T21:12:22.388 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2024-09-15T21:12:22.388 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=5 2024-09-15T21:12:22.388 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2024-09-15T21:12:22.388 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 5 SOMETHING remove 2024-09-15T21:12:22.388 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-09-15T21:12:22.388 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2024-09-15T21:12:22.388 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=5 2024-09-15T21:12:22.388 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2024-09-15T21:12:22.388 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.5 2024-09-15T21:12:22.388 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-15T21:12:22.388 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-15T21:12:22.388 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-15T21:12:22.389 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-15T21:12:22.389 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-15T21:12:22.692 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-15T21:12:22.693 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 5 SOMETHING remove 2024-09-15T21:12:22.693 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-15T21:12:22.693 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-15T21:12:22.693 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=5 2024-09-15T21:12:22.693 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-15T21:12:22.693 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/5 2024-09-15T21:12:22.693 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/5 SOMETHING remove 2024-09-15T21:12:24.867 INFO:tasks.workunit.client.0.smithi136.stdout:remove 1#2:eb822e21:::SOMETHING:head# 2024-09-15T21:12:25.386 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 5 2024-09-15T21:12:25.386 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2024-09-15T21:12:25.386 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-15T21:12:25.386 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=5 2024-09-15T21:12:25.386 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-15T21:12:25.386 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/5 2024-09-15T21:12:25.386 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=ce4892c7-6c6d-4b4f-8e6f-07a3e11133e5 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-15T21:12:25.386 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T21:12:25.386 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T21:12:25.387 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T21:12:25.387 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/5' 2024-09-15T21:12:25.387 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/5/journal' 2024-09-15T21:12:25.387 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-15T21:12:25.387 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-15T21:12:25.387 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-15T21:12:25.387 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-15T21:12:25.387 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-15T21:12:25.387 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-15T21:12:25.388 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-15T21:12:25.388 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T21:12:25.388 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52602 2024-09-15T21:12:25.388 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.52602/$cluster-$name.asok' 2024-09-15T21:12:25.388 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.52602/$cluster-$name.asok' 2024-09-15T21:12:25.388 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-15T21:12:25.388 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-15T21:12:25.388 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-15T21:12:25.388 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T21:12:25.388 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T21:12:25.388 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T21:12:25.389 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T21:12:25.389 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-15T21:12:25.389 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-15T21:12:25.389 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/5 2024-09-15T21:12:25.389 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.5 2024-09-15T21:12:25.389 INFO:tasks.workunit.client.0.smithi136.stderr:start osd.5 2024-09-15T21:12:25.390 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 5 --fsid=ce4892c7-6c6d-4b4f-8e6f-07a3e11133e5 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/5 --osd-journal=td/osd-scrub-repair/5/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.52602/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T21:12:25.390 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/5/whoami 2024-09-15T21:12:25.390 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 5 = 5 ']' 2024-09-15T21:12:25.390 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-15T21:12:25.390 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-15T21:12:25.390 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-15T21:12:25.417 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:12:25.414+0000 7f996e60d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:12:25.419 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:12:25.416+0000 7f996e60d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:12:25.420 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:12:25.417+0000 7f996e60d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:12:25.674 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 5 2024-09-15T21:12:25.674 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-15T21:12:25.675 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=5 2024-09-15T21:12:25.675 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-15T21:12:25.675 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-15T21:12:25.675 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:12:25.675 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-15T21:12:25.675 INFO:tasks.workunit.client.0.smithi136.stderr:0 2024-09-15T21:12:25.675 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:12:25.675 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-15T21:12:25.972 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:12:26.551 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:12:26.548+0000 7f996e60d640 -1 Falling back to public interface 2024-09-15T21:12:26.973 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:12:26.974 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:12:26.974 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-15T21:12:26.974 INFO:tasks.workunit.client.0.smithi136.stderr:1 2024-09-15T21:12:26.974 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:12:26.974 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-15T21:12:27.263 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:12:28.264 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:12:28.265 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:12:28.265 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-15T21:12:28.265 INFO:tasks.workunit.client.0.smithi136.stderr:2 2024-09-15T21:12:28.265 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:12:28.265 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-15T21:12:28.555 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:12:29.473 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:12:29.470+0000 7f996e60d640 -1 osd.5 79 log_to_monitors true 2024-09-15T21:12:29.557 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:12:29.557 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:12:29.557 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-15T21:12:29.557 INFO:tasks.workunit.client.0.smithi136.stderr:3 2024-09-15T21:12:29.557 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:12:29.557 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-15T21:12:29.958 INFO:tasks.workunit.client.0.smithi136.stderr:osd.5 up in weight 1 up_from 84 up_thru 70 down_at 80 last_clean_interval [30,79) [v2:127.0.0.1:6842/991035823,v1:127.0.0.1:6843/991035823] [v2:127.0.0.1:6844/991035823,v1:127.0.0.1:6845/991035823] exists,up 9918af07-0bbe-4711-bea1-402a4b5b6247 2024-09-15T21:12:29.958 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-15T21:12:29.958 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-15T21:12:29.959 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-15T21:12:29.959 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2024-09-15T21:12:29.959 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-15T21:12:29.959 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-15T21:12:29.959 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-15T21:12:29.959 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-15T21:12:29.960 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-15T21:12:29.960 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-15T21:12:29.960 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-15T21:12:29.960 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-15T21:12:29.960 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-15T21:12:30.051 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-15T21:12:30.051 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-15T21:12:30.051 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-15T21:12:30.051 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-15T21:12:30.051 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-15T21:12:30.051 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-15T21:12:30.523 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-15T21:12:30.523 INFO:tasks.workunit.client.0.smithi136.stderr:1 2024-09-15T21:12:30.523 INFO:tasks.workunit.client.0.smithi136.stderr:2 2024-09-15T21:12:30.523 INFO:tasks.workunit.client.0.smithi136.stderr:3 2024-09-15T21:12:30.523 INFO:tasks.workunit.client.0.smithi136.stderr:4 2024-09-15T21:12:30.523 INFO:tasks.workunit.client.0.smithi136.stderr:5 2024-09-15T21:12:30.523 INFO:tasks.workunit.client.0.smithi136.stderr:6 2024-09-15T21:12:30.523 INFO:tasks.workunit.client.0.smithi136.stderr:7 2024-09-15T21:12:30.523 INFO:tasks.workunit.client.0.smithi136.stderr:8 2024-09-15T21:12:30.523 INFO:tasks.workunit.client.0.smithi136.stderr:9' 2024-09-15T21:12:30.523 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-15T21:12:30.523 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T21:12:30.524 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-15T21:12:30.790 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836563 2024-09-15T21:12:30.790 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836563 2024-09-15T21:12:30.790 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836563' 2024-09-15T21:12:30.790 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T21:12:30.790 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-15T21:12:30.900 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=42949673039 2024-09-15T21:12:30.900 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 42949673039 2024-09-15T21:12:30.900 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836563 1-42949673039' 2024-09-15T21:12:30.900 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T21:12:30.901 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-15T21:12:31.085 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=64424509509 2024-09-15T21:12:31.086 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 64424509509 2024-09-15T21:12:31.086 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836563 1-42949673039 2-64424509509' 2024-09-15T21:12:31.086 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T21:12:31.086 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-15T21:12:31.361 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=313532612619 2024-09-15T21:12:31.361 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 313532612619 2024-09-15T21:12:31.361 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836563 1-42949673039 2-64424509509 3-313532612619' 2024-09-15T21:12:31.362 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T21:12:31.362 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-09-15T21:12:31.465 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=107374182458 2024-09-15T21:12:31.465 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 107374182458 2024-09-15T21:12:31.465 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836563 1-42949673039 2-64424509509 3-313532612619 4-107374182458' 2024-09-15T21:12:31.465 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T21:12:31.465 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-09-15T21:12:31.570 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=360777252866 2024-09-15T21:12:31.570 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 360777252866 2024-09-15T21:12:31.570 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836563 1-42949673039 2-64424509509 3-313532612619 4-107374182458 5-360777252866' 2024-09-15T21:12:31.570 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T21:12:31.570 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-09-15T21:12:31.823 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=150323855406 2024-09-15T21:12:31.823 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 150323855406 2024-09-15T21:12:31.823 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836563 1-42949673039 2-64424509509 3-313532612619 4-107374182458 5-360777252866 6-150323855406' 2024-09-15T21:12:31.823 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T21:12:31.823 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2024-09-15T21:12:31.972 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=171798691878 2024-09-15T21:12:31.972 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 171798691878 2024-09-15T21:12:31.972 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836563 1-42949673039 2-64424509509 3-313532612619 4-107374182458 5-360777252866 6-150323855406 7-171798691878' 2024-09-15T21:12:31.972 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T21:12:31.972 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2024-09-15T21:12:32.076 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=193273528352 2024-09-15T21:12:32.076 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 193273528352 2024-09-15T21:12:32.076 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836563 1-42949673039 2-64424509509 3-313532612619 4-107374182458 5-360777252866 6-150323855406 7-171798691878 8-193273528352' 2024-09-15T21:12:32.076 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T21:12:32.076 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2024-09-15T21:12:32.180 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=335007449094 2024-09-15T21:12:32.180 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 335007449094 2024-09-15T21:12:32.180 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836563 1-42949673039 2-64424509509 3-313532612619 4-107374182458 5-360777252866 6-150323855406 7-171798691878 8-193273528352 9-335007449094' 2024-09-15T21:12:32.180 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T21:12:32.181 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836563 2024-09-15T21:12:32.181 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T21:12:32.181 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-15T21:12:32.182 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836563 2024-09-15T21:12:32.182 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T21:12:32.183 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836563 2024-09-15T21:12:32.183 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836563' 2024-09-15T21:12:32.183 INFO:tasks.workunit.client.0.smithi136.stderr:waiting osd.0 seq 21474836563 2024-09-15T21:12:32.184 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T21:12:32.461 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836563 -lt 21474836563 2024-09-15T21:12:32.461 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T21:12:32.461 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-42949673039 2024-09-15T21:12:32.462 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T21:12:32.462 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-15T21:12:32.463 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-42949673039 2024-09-15T21:12:32.463 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T21:12:32.464 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=42949673039 2024-09-15T21:12:32.464 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 42949673039' 2024-09-15T21:12:32.464 INFO:tasks.workunit.client.0.smithi136.stderr:waiting osd.1 seq 42949673039 2024-09-15T21:12:32.464 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-15T21:12:32.743 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 42949673039 -lt 42949673039 2024-09-15T21:12:32.743 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T21:12:32.744 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-64424509509 2024-09-15T21:12:32.744 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T21:12:32.744 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-15T21:12:32.745 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-64424509509 2024-09-15T21:12:32.745 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T21:12:32.746 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=64424509509 2024-09-15T21:12:32.746 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 64424509509' 2024-09-15T21:12:32.746 INFO:tasks.workunit.client.0.smithi136.stderr:waiting osd.2 seq 64424509509 2024-09-15T21:12:32.747 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-15T21:12:33.055 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 64424509509 -lt 64424509509 2024-09-15T21:12:33.055 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T21:12:33.055 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 3-313532612619 2024-09-15T21:12:33.055 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T21:12:33.056 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=3 2024-09-15T21:12:33.057 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 3-313532612619 2024-09-15T21:12:33.057 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T21:12:33.058 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=313532612619 2024-09-15T21:12:33.058 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.3 seq 313532612619' 2024-09-15T21:12:33.058 INFO:tasks.workunit.client.0.smithi136.stderr:waiting osd.3 seq 313532612619 2024-09-15T21:12:33.058 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-15T21:12:33.337 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 313532612619 -lt 313532612619 2024-09-15T21:12:33.337 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T21:12:33.337 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 4-107374182458 2024-09-15T21:12:33.338 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T21:12:33.338 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=4 2024-09-15T21:12:33.339 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 4-107374182458 2024-09-15T21:12:33.339 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T21:12:33.340 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=107374182458 2024-09-15T21:12:33.340 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.4 seq 107374182458' 2024-09-15T21:12:33.340 INFO:tasks.workunit.client.0.smithi136.stderr:waiting osd.4 seq 107374182458 2024-09-15T21:12:33.341 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 4 2024-09-15T21:12:33.621 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 107374182458 -lt 107374182458 2024-09-15T21:12:33.621 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T21:12:33.622 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 5-360777252866 2024-09-15T21:12:33.622 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T21:12:33.623 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=5 2024-09-15T21:12:33.623 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 5-360777252866 2024-09-15T21:12:33.623 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T21:12:33.624 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=360777252866 2024-09-15T21:12:33.625 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.5 seq 360777252866' 2024-09-15T21:12:33.625 INFO:tasks.workunit.client.0.smithi136.stderr:waiting osd.5 seq 360777252866 2024-09-15T21:12:33.625 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 5 2024-09-15T21:12:33.903 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 360777252866 -lt 360777252866 2024-09-15T21:12:33.903 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T21:12:33.904 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 6-150323855406 2024-09-15T21:12:33.904 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T21:12:33.905 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=6 2024-09-15T21:12:33.905 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 6-150323855406 2024-09-15T21:12:33.905 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T21:12:33.906 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=150323855406 2024-09-15T21:12:33.907 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.6 seq 150323855406' 2024-09-15T21:12:33.907 INFO:tasks.workunit.client.0.smithi136.stderr:waiting osd.6 seq 150323855406 2024-09-15T21:12:33.907 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 6 2024-09-15T21:12:34.196 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 150323855406 -lt 150323855406 2024-09-15T21:12:34.196 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T21:12:34.196 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 7-171798691878 2024-09-15T21:12:34.197 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T21:12:34.197 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=7 2024-09-15T21:12:34.198 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 7-171798691878 2024-09-15T21:12:34.198 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T21:12:34.199 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=171798691878 2024-09-15T21:12:34.199 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.7 seq 171798691878' 2024-09-15T21:12:34.199 INFO:tasks.workunit.client.0.smithi136.stderr:waiting osd.7 seq 171798691878 2024-09-15T21:12:34.199 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 7 2024-09-15T21:12:34.477 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 171798691878 -lt 171798691878 2024-09-15T21:12:34.477 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T21:12:34.477 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 8-193273528352 2024-09-15T21:12:34.478 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T21:12:34.478 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=8 2024-09-15T21:12:34.479 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 8-193273528352 2024-09-15T21:12:34.479 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T21:12:34.480 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=193273528352 2024-09-15T21:12:34.480 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.8 seq 193273528352' 2024-09-15T21:12:34.480 INFO:tasks.workunit.client.0.smithi136.stderr:waiting osd.8 seq 193273528352 2024-09-15T21:12:34.480 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 8 2024-09-15T21:12:34.757 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 193273528352 -lt 193273528352 2024-09-15T21:12:34.757 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T21:12:34.757 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 9-335007449094 2024-09-15T21:12:34.757 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T21:12:34.758 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=9 2024-09-15T21:12:34.759 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 9-335007449094 2024-09-15T21:12:34.759 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T21:12:34.760 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=335007449094 2024-09-15T21:12:34.760 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.9 seq 335007449094' 2024-09-15T21:12:34.760 INFO:tasks.workunit.client.0.smithi136.stderr:waiting osd.9 seq 335007449094 2024-09-15T21:12:34.760 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 9 2024-09-15T21:12:35.054 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 335007449094 -lt 335007449094 2024-09-15T21:12:35.054 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-15T21:12:35.054 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T21:12:35.054 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T21:12:35.405 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 5 == 0 2024-09-15T21:12:35.405 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-15T21:12:35.405 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-15T21:12:35.405 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-15T21:12:35.405 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T21:12:35.405 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T21:12:35.405 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T21:12:35.405 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T21:12:35.678 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=3 2024-09-15T21:12:35.678 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-15T21:12:35.678 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T21:12:35.678 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T21:12:36.033 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 3 = 5 2024-09-15T21:12:36.033 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 3 '!=' -1 2024-09-15T21:12:36.033 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: loop=0 2024-09-15T21:12:36.033 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1652: wait_for_clean: num_active_clean=3 2024-09-15T21:12:36.033 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-15T21:12:36.033 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 0.1 2024-09-15T21:12:36.133 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-15T21:12:36.133 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-15T21:12:36.134 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-15T21:12:36.134 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-15T21:12:36.134 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T21:12:36.134 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T21:12:36.134 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T21:12:36.134 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T21:12:36.407 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=5 2024-09-15T21:12:36.408 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-15T21:12:36.408 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T21:12:36.408 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T21:12:36.759 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 = 5 2024-09-15T21:12:36.759 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-15T21:12:36.759 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-15T21:12:36.759 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:237: corrupt_and_repair_one: get_pg ecpool SOMETHING 2024-09-15T21:12:36.760 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1084: get_pg: local poolname=ecpool 2024-09-15T21:12:36.760 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1085: get_pg: local objectname=SOMETHING 2024-09-15T21:12:36.760 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1087: get_pg: ceph --format json osd map ecpool SOMETHING 2024-09-15T21:12:36.760 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1087: get_pg: jq -r .pgid 2024-09-15T21:12:37.062 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:237: corrupt_and_repair_one: local pg=2.0 2024-09-15T21:12:37.062 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:238: corrupt_and_repair_one: repair 2.0 2024-09-15T21:12:37.062 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1846: repair: local pgid=2.0 2024-09-15T21:12:37.062 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1847: repair: get_last_scrub_stamp 2.0 2024-09-15T21:12:37.062 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2024-09-15T21:12:37.062 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-15T21:12:37.062 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-15T21:12:37.062 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-09-15T21:12:37.337 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1847: repair: local last_scrub=2024-09-15T21:12:05.495858+0000 2024-09-15T21:12:37.337 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1848: repair: ceph pg repair 2.0 2024-09-15T21:12:37.598 INFO:tasks.workunit.client.0.smithi136.stderr:instructing pg 2.0s0 on osd.3 to repair 2024-09-15T21:12:37.610 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1849: repair: wait_for_scrub 2.0 2024-09-15T21:12:05.495858+0000 2024-09-15T21:12:37.610 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: local pgid=2.0 2024-09-15T21:12:37.610 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2018: wait_for_scrub: local last_scrub=2024-09-15T21:12:05.495858+0000 2024-09-15T21:12:37.610 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2019: wait_for_scrub: local sname=last_scrub_stamp 2024-09-15T21:12:37.610 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i=0 )) 2024-09-15T21:12:37.610 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-15T21:12:37.610 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-09-15T21:12:37.610 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2024-09-15T21:12:37.610 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-15T21:12:37.611 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-15T21:12:37.611 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-09-15T21:12:37.886 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-15T21:12:05.495858+0000 '>' 2024-09-15T21:12:05.495858+0000 2024-09-15T21:12:37.886 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-15T21:12:38.887 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-15T21:12:38.887 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-15T21:12:38.888 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-09-15T21:12:38.888 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2024-09-15T21:12:38.888 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-15T21:12:38.889 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-15T21:12:38.889 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-09-15T21:12:39.186 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-15T21:12:05.495858+0000 '>' 2024-09-15T21:12:05.495858+0000 2024-09-15T21:12:39.186 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-15T21:12:40.187 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-15T21:12:40.187 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-15T21:12:40.188 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-09-15T21:12:40.188 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2024-09-15T21:12:40.188 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-15T21:12:40.188 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-15T21:12:40.188 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-09-15T21:12:40.467 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-15T21:12:05.495858+0000 '>' 2024-09-15T21:12:05.495858+0000 2024-09-15T21:12:40.467 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-15T21:12:41.468 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-15T21:12:41.468 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-15T21:12:41.469 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-09-15T21:12:41.469 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2024-09-15T21:12:41.469 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-15T21:12:41.469 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-15T21:12:41.469 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-09-15T21:12:41.751 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-15T21:12:05.495858+0000 '>' 2024-09-15T21:12:05.495858+0000 2024-09-15T21:12:41.751 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-15T21:12:42.752 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-15T21:12:42.753 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-15T21:12:42.753 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-09-15T21:12:42.753 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2024-09-15T21:12:42.754 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-15T21:12:42.754 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-15T21:12:42.754 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-09-15T21:12:43.038 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-15T21:12:38.602968+0000 '>' 2024-09-15T21:12:05.495858+0000 2024-09-15T21:12:43.038 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2023: wait_for_scrub: return 0 2024-09-15T21:12:43.038 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1850: repair: sleep 2 2024-09-15T21:12:45.039 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:242: corrupt_and_repair_one: objectstore_tool td/osd-scrub-repair 9 SOMETHING list-attrs 2024-09-15T21:12:45.039 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2024-09-15T21:12:45.039 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2024-09-15T21:12:45.039 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=9 2024-09-15T21:12:45.040 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2024-09-15T21:12:45.040 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 9 SOMETHING list-attrs 2024-09-15T21:12:45.040 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-09-15T21:12:45.040 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2024-09-15T21:12:45.040 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=9 2024-09-15T21:12:45.040 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2024-09-15T21:12:45.040 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.9 2024-09-15T21:12:45.040 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-15T21:12:45.040 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-15T21:12:45.041 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-15T21:12:45.041 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-15T21:12:45.041 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-15T21:12:45.348 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-15T21:12:45.348 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 9 SOMETHING list-attrs 2024-09-15T21:12:45.349 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-15T21:12:45.349 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-15T21:12:45.349 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=9 2024-09-15T21:12:45.349 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-15T21:12:45.349 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/9 2024-09-15T21:12:45.349 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/9 SOMETHING list-attrs 2024-09-15T21:12:45.783 INFO:tasks.workunit.client.0.smithi136.stdout:_ 2024-09-15T21:12:45.783 INFO:tasks.workunit.client.0.smithi136.stdout:hinfo_key 2024-09-15T21:12:45.783 INFO:tasks.workunit.client.0.smithi136.stdout:snapset 2024-09-15T21:12:46.050 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 9 2024-09-15T21:12:46.050 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2024-09-15T21:12:46.050 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-15T21:12:46.050 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=9 2024-09-15T21:12:46.050 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-15T21:12:46.050 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/9 2024-09-15T21:12:46.050 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=ce4892c7-6c6d-4b4f-8e6f-07a3e11133e5 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-15T21:12:46.050 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T21:12:46.050 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T21:12:46.051 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T21:12:46.051 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/9' 2024-09-15T21:12:46.051 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/9/journal' 2024-09-15T21:12:46.051 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-15T21:12:46.051 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-15T21:12:46.051 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-15T21:12:46.051 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-15T21:12:46.051 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-15T21:12:46.051 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-15T21:12:46.051 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-15T21:12:46.051 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T21:12:46.051 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52602 2024-09-15T21:12:46.052 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.52602/$cluster-$name.asok' 2024-09-15T21:12:46.054 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.52602/$cluster-$name.asok' 2024-09-15T21:12:46.054 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-15T21:12:46.054 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-15T21:12:46.054 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-15T21:12:46.054 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T21:12:46.054 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T21:12:46.054 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T21:12:46.054 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T21:12:46.054 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-15T21:12:46.054 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-15T21:12:46.055 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/9 2024-09-15T21:12:46.055 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.9 2024-09-15T21:12:46.055 INFO:tasks.workunit.client.0.smithi136.stderr:start osd.9 2024-09-15T21:12:46.055 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 9 --fsid=ce4892c7-6c6d-4b4f-8e6f-07a3e11133e5 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/9 --osd-journal=td/osd-scrub-repair/9/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.52602/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T21:12:46.055 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/9/whoami 2024-09-15T21:12:46.055 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 9 = 9 ']' 2024-09-15T21:12:46.055 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-15T21:12:46.055 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-15T21:12:46.056 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-15T21:12:46.083 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:12:46.079+0000 7f97e3a2f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:12:46.085 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:12:46.082+0000 7f97e3a2f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:12:46.086 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:12:46.083+0000 7f97e3a2f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:12:46.463 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 9 2024-09-15T21:12:46.463 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-15T21:12:46.464 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=9 2024-09-15T21:12:46.464 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-15T21:12:46.464 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-15T21:12:46.464 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:12:46.464 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-15T21:12:46.464 INFO:tasks.workunit.client.0.smithi136.stderr:0 2024-09-15T21:12:46.464 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:12:46.464 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.9 up' 2024-09-15T21:12:46.878 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:12:47.438 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:12:47.435+0000 7f97e3a2f640 -1 Falling back to public interface 2024-09-15T21:12:47.880 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:12:47.880 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:12:47.880 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-15T21:12:47.880 INFO:tasks.workunit.client.0.smithi136.stderr:1 2024-09-15T21:12:47.880 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:12:47.880 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.9 up' 2024-09-15T21:12:48.169 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:12:49.171 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:12:49.171 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:12:49.171 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-15T21:12:49.171 INFO:tasks.workunit.client.0.smithi136.stderr:2 2024-09-15T21:12:49.171 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:12:49.171 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.9 up' 2024-09-15T21:12:49.458 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:12:50.460 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:12:50.460 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:12:50.460 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-15T21:12:50.460 INFO:tasks.workunit.client.0.smithi136.stderr:3 2024-09-15T21:12:50.460 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:12:50.460 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.9 up' 2024-09-15T21:12:50.749 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:12:50.764 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:12:50.761+0000 7f97e3a2f640 -1 osd.9 87 log_to_monitors true 2024-09-15T21:12:51.751 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:12:51.751 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:12:51.751 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-15T21:12:51.751 INFO:tasks.workunit.client.0.smithi136.stderr:4 2024-09-15T21:12:51.751 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:12:51.751 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.9 up' 2024-09-15T21:12:52.224 INFO:tasks.workunit.client.0.smithi136.stderr:osd.9 up in weight 1 up_from 90 up_thru 0 down_at 87 last_clean_interval [78,87) [v2:127.0.0.1:6874/3270436388,v1:127.0.0.1:6875/3270436388] [v2:127.0.0.1:6876/3270436388,v1:127.0.0.1:6877/3270436388] exists,up 78ecdb67-c9df-44e1-b77a-4e850e67a453 2024-09-15T21:12:52.224 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-15T21:12:52.224 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-15T21:12:52.224 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-15T21:12:52.224 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2024-09-15T21:12:52.224 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-15T21:12:52.225 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-15T21:12:52.225 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-15T21:12:52.225 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-15T21:12:52.225 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-15T21:12:52.225 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-15T21:12:52.225 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-15T21:12:52.225 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-15T21:12:52.225 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-15T21:12:52.315 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-15T21:12:52.315 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-15T21:12:52.315 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-15T21:12:52.315 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-15T21:12:52.315 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-15T21:12:52.315 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-15T21:12:52.757 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-15T21:12:52.757 INFO:tasks.workunit.client.0.smithi136.stderr:1 2024-09-15T21:12:52.757 INFO:tasks.workunit.client.0.smithi136.stderr:2 2024-09-15T21:12:52.757 INFO:tasks.workunit.client.0.smithi136.stderr:3 2024-09-15T21:12:52.758 INFO:tasks.workunit.client.0.smithi136.stderr:4 2024-09-15T21:12:52.758 INFO:tasks.workunit.client.0.smithi136.stderr:5 2024-09-15T21:12:52.758 INFO:tasks.workunit.client.0.smithi136.stderr:6 2024-09-15T21:12:52.758 INFO:tasks.workunit.client.0.smithi136.stderr:7 2024-09-15T21:12:52.758 INFO:tasks.workunit.client.0.smithi136.stderr:8 2024-09-15T21:12:52.758 INFO:tasks.workunit.client.0.smithi136.stderr:9' 2024-09-15T21:12:52.758 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-15T21:12:52.758 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T21:12:52.758 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-15T21:12:52.885 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836568 2024-09-15T21:12:52.885 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836568 2024-09-15T21:12:52.885 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836568' 2024-09-15T21:12:52.885 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T21:12:52.885 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-15T21:12:52.989 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=42949673044 2024-09-15T21:12:52.989 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 42949673044 2024-09-15T21:12:52.989 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836568 1-42949673044' 2024-09-15T21:12:52.989 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T21:12:52.989 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-15T21:12:53.120 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=64424509514 2024-09-15T21:12:53.120 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 64424509514 2024-09-15T21:12:53.120 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836568 1-42949673044 2-64424509514' 2024-09-15T21:12:53.120 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T21:12:53.120 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-15T21:12:53.391 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=313532612624 2024-09-15T21:12:53.391 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 313532612624 2024-09-15T21:12:53.391 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836568 1-42949673044 2-64424509514 3-313532612624' 2024-09-15T21:12:53.391 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T21:12:53.391 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-09-15T21:12:53.506 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=107374182463 2024-09-15T21:12:53.506 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 107374182463 2024-09-15T21:12:53.506 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836568 1-42949673044 2-64424509514 3-313532612624 4-107374182463' 2024-09-15T21:12:53.506 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T21:12:53.506 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-09-15T21:12:53.744 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=360777252871 2024-09-15T21:12:53.744 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 360777252871 2024-09-15T21:12:53.744 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836568 1-42949673044 2-64424509514 3-313532612624 4-107374182463 5-360777252871' 2024-09-15T21:12:53.744 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T21:12:53.744 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-09-15T21:12:53.849 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=150323855411 2024-09-15T21:12:53.849 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 150323855411 2024-09-15T21:12:53.849 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836568 1-42949673044 2-64424509514 3-313532612624 4-107374182463 5-360777252871 6-150323855411' 2024-09-15T21:12:53.849 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T21:12:53.849 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2024-09-15T21:12:53.967 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=171798691883 2024-09-15T21:12:53.967 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 171798691883 2024-09-15T21:12:53.967 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836568 1-42949673044 2-64424509514 3-313532612624 4-107374182463 5-360777252871 6-150323855411 7-171798691883' 2024-09-15T21:12:53.968 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T21:12:53.968 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2024-09-15T21:12:54.113 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=193273528357 2024-09-15T21:12:54.113 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 193273528357 2024-09-15T21:12:54.113 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836568 1-42949673044 2-64424509514 3-313532612624 4-107374182463 5-360777252871 6-150323855411 7-171798691883 8-193273528357' 2024-09-15T21:12:54.113 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T21:12:54.113 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2024-09-15T21:12:54.215 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=386547056642 2024-09-15T21:12:54.216 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 386547056642 2024-09-15T21:12:54.216 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836568 1-42949673044 2-64424509514 3-313532612624 4-107374182463 5-360777252871 6-150323855411 7-171798691883 8-193273528357 9-386547056642' 2024-09-15T21:12:54.216 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T21:12:54.216 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836568 2024-09-15T21:12:54.216 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T21:12:54.217 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-15T21:12:54.217 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836568 2024-09-15T21:12:54.217 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T21:12:54.218 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836568 2024-09-15T21:12:54.219 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836568' 2024-09-15T21:12:54.219 INFO:tasks.workunit.client.0.smithi136.stderr:waiting osd.0 seq 21474836568 2024-09-15T21:12:54.219 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T21:12:54.495 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836569 -lt 21474836568 2024-09-15T21:12:54.495 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T21:12:54.495 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-42949673044 2024-09-15T21:12:54.495 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T21:12:54.496 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-15T21:12:54.497 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-42949673044 2024-09-15T21:12:54.497 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T21:12:54.498 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=42949673044 2024-09-15T21:12:54.498 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 42949673044' 2024-09-15T21:12:54.498 INFO:tasks.workunit.client.0.smithi136.stderr:waiting osd.1 seq 42949673044 2024-09-15T21:12:54.498 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-15T21:12:54.811 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 42949673045 -lt 42949673044 2024-09-15T21:12:54.812 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T21:12:54.812 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-64424509514 2024-09-15T21:12:54.812 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T21:12:54.813 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-15T21:12:54.813 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-64424509514 2024-09-15T21:12:54.814 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T21:12:54.814 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=64424509514 2024-09-15T21:12:54.814 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 64424509514' 2024-09-15T21:12:54.815 INFO:tasks.workunit.client.0.smithi136.stderr:waiting osd.2 seq 64424509514 2024-09-15T21:12:54.815 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-15T21:12:55.109 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 64424509515 -lt 64424509514 2024-09-15T21:12:55.109 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T21:12:55.109 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 3-313532612624 2024-09-15T21:12:55.109 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T21:12:55.110 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=3 2024-09-15T21:12:55.111 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 3-313532612624 2024-09-15T21:12:55.111 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T21:12:55.112 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=313532612624 2024-09-15T21:12:55.112 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.3 seq 313532612624' 2024-09-15T21:12:55.112 INFO:tasks.workunit.client.0.smithi136.stderr:waiting osd.3 seq 313532612624 2024-09-15T21:12:55.112 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-15T21:12:55.392 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 313532612624 -lt 313532612624 2024-09-15T21:12:55.392 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T21:12:55.393 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 4-107374182463 2024-09-15T21:12:55.393 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T21:12:55.393 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=4 2024-09-15T21:12:55.394 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 4-107374182463 2024-09-15T21:12:55.394 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T21:12:55.395 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=107374182463 2024-09-15T21:12:55.395 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.4 seq 107374182463' 2024-09-15T21:12:55.395 INFO:tasks.workunit.client.0.smithi136.stderr:waiting osd.4 seq 107374182463 2024-09-15T21:12:55.396 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 4 2024-09-15T21:12:55.674 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 107374182463 -lt 107374182463 2024-09-15T21:12:55.674 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T21:12:55.675 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 5-360777252871 2024-09-15T21:12:55.675 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T21:12:55.676 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=5 2024-09-15T21:12:55.676 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 5-360777252871 2024-09-15T21:12:55.676 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T21:12:55.677 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=360777252871 2024-09-15T21:12:55.678 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.5 seq 360777252871' 2024-09-15T21:12:55.678 INFO:tasks.workunit.client.0.smithi136.stderr:waiting osd.5 seq 360777252871 2024-09-15T21:12:55.678 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 5 2024-09-15T21:12:55.957 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 360777252872 -lt 360777252871 2024-09-15T21:12:55.957 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T21:12:55.957 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 6-150323855411 2024-09-15T21:12:55.957 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T21:12:55.958 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=6 2024-09-15T21:12:55.959 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 6-150323855411 2024-09-15T21:12:55.959 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T21:12:55.960 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=150323855411 2024-09-15T21:12:55.960 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.6 seq 150323855411' 2024-09-15T21:12:55.960 INFO:tasks.workunit.client.0.smithi136.stderr:waiting osd.6 seq 150323855411 2024-09-15T21:12:55.960 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 6 2024-09-15T21:12:56.251 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 150323855412 -lt 150323855411 2024-09-15T21:12:56.251 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T21:12:56.252 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 7-171798691883 2024-09-15T21:12:56.252 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T21:12:56.253 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=7 2024-09-15T21:12:56.253 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 7-171798691883 2024-09-15T21:12:56.253 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T21:12:56.254 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=171798691883 2024-09-15T21:12:56.254 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.7 seq 171798691883' 2024-09-15T21:12:56.255 INFO:tasks.workunit.client.0.smithi136.stderr:waiting osd.7 seq 171798691883 2024-09-15T21:12:56.255 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 7 2024-09-15T21:12:56.534 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 171798691883 -lt 171798691883 2024-09-15T21:12:56.535 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T21:12:56.535 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 8-193273528357 2024-09-15T21:12:56.535 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T21:12:56.536 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=8 2024-09-15T21:12:56.537 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 8-193273528357 2024-09-15T21:12:56.537 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T21:12:56.538 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=193273528357 2024-09-15T21:12:56.538 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.8 seq 193273528357' 2024-09-15T21:12:56.538 INFO:tasks.workunit.client.0.smithi136.stderr:waiting osd.8 seq 193273528357 2024-09-15T21:12:56.538 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 8 2024-09-15T21:12:56.816 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 193273528357 -lt 193273528357 2024-09-15T21:12:56.817 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T21:12:56.817 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 9-386547056642 2024-09-15T21:12:56.817 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T21:12:56.818 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=9 2024-09-15T21:12:56.818 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 9-386547056642 2024-09-15T21:12:56.818 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T21:12:56.820 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=386547056642 2024-09-15T21:12:56.820 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.9 seq 386547056642' 2024-09-15T21:12:56.820 INFO:tasks.workunit.client.0.smithi136.stderr:waiting osd.9 seq 386547056642 2024-09-15T21:12:56.820 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 9 2024-09-15T21:12:57.096 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 386547056642 -lt 386547056642 2024-09-15T21:12:57.096 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-15T21:12:57.096 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T21:12:57.097 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T21:12:57.443 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 5 == 0 2024-09-15T21:12:57.443 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-15T21:12:57.443 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-15T21:12:57.443 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-15T21:12:57.444 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T21:12:57.444 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T21:12:57.444 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T21:12:57.444 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T21:12:57.720 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=5 2024-09-15T21:12:57.720 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-15T21:12:57.720 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T21:12:57.721 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T21:12:58.068 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 = 5 2024-09-15T21:12:58.068 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-15T21:12:58.068 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-15T21:12:58.069 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:243: corrupt_and_repair_one: rados --pool ecpool get SOMETHING td/osd-scrub-repair/COPY 2024-09-15T21:12:58.099 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:244: corrupt_and_repair_one: diff td/osd-scrub-repair/ORIGINAL td/osd-scrub-repair/COPY 2024-09-15T21:12:58.102 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:262: corrupt_and_repair_erasure_coded: corrupt_and_repair_two td/osd-scrub-repair ecpool 5 9 2024-09-15T21:12:58.102 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:185: corrupt_and_repair_two: local dir=td/osd-scrub-repair 2024-09-15T21:12:58.102 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:186: corrupt_and_repair_two: local poolname=ecpool 2024-09-15T21:12:58.102 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:187: corrupt_and_repair_two: local first=5 2024-09-15T21:12:58.102 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:188: corrupt_and_repair_two: local second=9 2024-09-15T21:12:58.102 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:193: corrupt_and_repair_two: pids= 2024-09-15T21:12:58.103 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:194: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 5 SOMETHING remove 2024-09-15T21:12:58.103 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2135: run_in_background: local pid_variable=pids 2024-09-15T21:12:58.103 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2136: run_in_background: shift 2024-09-15T21:12:58.103 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2140: run_in_background: eval 'pids+=" 251094"' 2024-09-15T21:12:58.103 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2140: run_in_background: pids+=' 251094' 2024-09-15T21:12:58.103 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:195: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 9 SOMETHING remove 2024-09-15T21:12:58.103 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2135: run_in_background: local pid_variable=pids 2024-09-15T21:12:58.103 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2136: run_in_background: shift 2024-09-15T21:12:58.103 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2140: run_in_background: eval 'pids+=" 251095"' 2024-09-15T21:12:58.103 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2140: run_in_background: pids+=' 251095' 2024-09-15T21:12:58.103 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:196: corrupt_and_repair_two: wait_background pids 2024-09-15T21:12:58.103 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: wait_background: pids=' 251094 251095' 2024-09-15T21:12:58.103 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: wait_background: return_code=0 2024-09-15T21:12:58.103 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: wait_background: for pid in $pids 2024-09-15T21:12:58.104 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2175: wait_background: wait 251094 2024-09-15T21:12:58.104 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: run_in_background: objectstore_tool td/osd-scrub-repair 5 SOMETHING remove 2024-09-15T21:12:58.104 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: run_in_background: sed 's/^/251097: /' 2024-09-15T21:12:58.104 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: run_in_background: objectstore_tool td/osd-scrub-repair 9 SOMETHING remove 2024-09-15T21:12:58.104 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: run_in_background: sed 's/^/251099: /' 2024-09-15T21:13:01.561 INFO:tasks.workunit.client.0.smithi136.stderr:251097: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2024-09-15T21:13:01.561 INFO:tasks.workunit.client.0.smithi136.stderr:251097: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2024-09-15T21:13:01.561 INFO:tasks.workunit.client.0.smithi136.stderr:251097: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=5 2024-09-15T21:13:01.561 INFO:tasks.workunit.client.0.smithi136.stderr:251097: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2024-09-15T21:13:01.561 INFO:tasks.workunit.client.0.smithi136.stderr:251097: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 5 SOMETHING remove 2024-09-15T21:13:01.561 INFO:tasks.workunit.client.0.smithi136.stderr:251097: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-09-15T21:13:01.561 INFO:tasks.workunit.client.0.smithi136.stderr:251097: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2024-09-15T21:13:01.562 INFO:tasks.workunit.client.0.smithi136.stderr:251097: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=5 2024-09-15T21:13:01.562 INFO:tasks.workunit.client.0.smithi136.stderr:251097: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2024-09-15T21:13:01.562 INFO:tasks.workunit.client.0.smithi136.stderr:251097: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.5 2024-09-15T21:13:01.562 INFO:tasks.workunit.client.0.smithi136.stderr:251097: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-15T21:13:01.562 INFO:tasks.workunit.client.0.smithi136.stderr:251097: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-15T21:13:01.562 INFO:tasks.workunit.client.0.smithi136.stderr:251097: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-15T21:13:01.562 INFO:tasks.workunit.client.0.smithi136.stderr:251097: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-15T21:13:01.562 INFO:tasks.workunit.client.0.smithi136.stderr:251097: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-15T21:13:01.562 INFO:tasks.workunit.client.0.smithi136.stderr:251097: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-15T21:13:01.562 INFO:tasks.workunit.client.0.smithi136.stderr:251097: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 5 SOMETHING remove 2024-09-15T21:13:01.562 INFO:tasks.workunit.client.0.smithi136.stderr:251097: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-15T21:13:01.562 INFO:tasks.workunit.client.0.smithi136.stderr:251097: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-15T21:13:01.562 INFO:tasks.workunit.client.0.smithi136.stderr:251097: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=5 2024-09-15T21:13:01.563 INFO:tasks.workunit.client.0.smithi136.stderr:251097: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-15T21:13:01.563 INFO:tasks.workunit.client.0.smithi136.stderr:251097: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/5 2024-09-15T21:13:01.563 INFO:tasks.workunit.client.0.smithi136.stderr:251097: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/5 SOMETHING remove 2024-09-15T21:13:01.563 INFO:tasks.workunit.client.0.smithi136.stderr:251097: remove 1#2:eb822e21:::SOMETHING:head# 2024-09-15T21:13:01.563 INFO:tasks.workunit.client.0.smithi136.stderr:251097: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 5 2024-09-15T21:13:01.563 INFO:tasks.workunit.client.0.smithi136.stderr:251097: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2024-09-15T21:13:01.563 INFO:tasks.workunit.client.0.smithi136.stderr:251097: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-15T21:13:01.563 INFO:tasks.workunit.client.0.smithi136.stderr:251097: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=5 2024-09-15T21:13:01.563 INFO:tasks.workunit.client.0.smithi136.stderr:251097: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-15T21:13:01.563 INFO:tasks.workunit.client.0.smithi136.stderr:251097: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/5 2024-09-15T21:13:01.563 INFO:tasks.workunit.client.0.smithi136.stderr:251097: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=ce4892c7-6c6d-4b4f-8e6f-07a3e11133e5 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-15T21:13:01.563 INFO:tasks.workunit.client.0.smithi136.stderr:251097: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T21:13:01.564 INFO:tasks.workunit.client.0.smithi136.stderr:251097: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T21:13:01.564 INFO:tasks.workunit.client.0.smithi136.stderr:251097: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T21:13:01.564 INFO:tasks.workunit.client.0.smithi136.stderr:251097: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/5' 2024-09-15T21:13:01.565 INFO:tasks.workunit.client.0.smithi136.stderr:251097: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/5/journal' 2024-09-15T21:13:01.565 INFO:tasks.workunit.client.0.smithi136.stderr:251097: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-15T21:13:01.565 INFO:tasks.workunit.client.0.smithi136.stderr:251097: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-15T21:13:01.565 INFO:tasks.workunit.client.0.smithi136.stderr:251097: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-15T21:13:01.565 INFO:tasks.workunit.client.0.smithi136.stderr:251097: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-15T21:13:01.565 INFO:tasks.workunit.client.0.smithi136.stderr:251097: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-15T21:13:01.565 INFO:tasks.workunit.client.0.smithi136.stderr:251097: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-15T21:13:01.565 INFO:tasks.workunit.client.0.smithi136.stderr:251097: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-15T21:13:01.565 INFO:tasks.workunit.client.0.smithi136.stderr:251097: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T21:13:01.565 INFO:tasks.workunit.client.0.smithi136.stderr:251097: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52602 2024-09-15T21:13:01.565 INFO:tasks.workunit.client.0.smithi136.stderr:251097: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.52602/$cluster-$name.asok' 2024-09-15T21:13:01.565 INFO:tasks.workunit.client.0.smithi136.stderr:251097: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.52602/$cluster-$name.asok' 2024-09-15T21:13:01.565 INFO:tasks.workunit.client.0.smithi136.stderr:251097: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-15T21:13:01.565 INFO:tasks.workunit.client.0.smithi136.stderr:251097: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-15T21:13:01.566 INFO:tasks.workunit.client.0.smithi136.stderr:251097: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-15T21:13:01.566 INFO:tasks.workunit.client.0.smithi136.stderr:251097: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T21:13:01.566 INFO:tasks.workunit.client.0.smithi136.stderr:251097: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T21:13:01.566 INFO:tasks.workunit.client.0.smithi136.stderr:251097: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T21:13:01.566 INFO:tasks.workunit.client.0.smithi136.stderr:251097: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T21:13:01.566 INFO:tasks.workunit.client.0.smithi136.stderr:251097: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-15T21:13:01.566 INFO:tasks.workunit.client.0.smithi136.stderr:251097: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-15T21:13:01.566 INFO:tasks.workunit.client.0.smithi136.stderr:251097: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/5 2024-09-15T21:13:01.566 INFO:tasks.workunit.client.0.smithi136.stderr:251097: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.5 2024-09-15T21:13:01.566 INFO:tasks.workunit.client.0.smithi136.stderr:251097: start osd.5 2024-09-15T21:13:01.566 INFO:tasks.workunit.client.0.smithi136.stderr:251097: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 5 --fsid=ce4892c7-6c6d-4b4f-8e6f-07a3e11133e5 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/5 --osd-journal=td/osd-scrub-repair/5/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.52602/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T21:13:01.723 INFO:tasks.workunit.client.0.smithi136.stderr:2251099: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2024-09-15T21:13:01.723 INFO:tasks.workunit.client.0.smithi136.stderr:251099: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2024-09-15T21:13:01.723 INFO:tasks.workunit.client.0.smithi136.stderr:251099: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=9 2024-09-15T21:13:01.723 INFO:tasks.workunit.client.0.smithi136.stderr:251099: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2024-09-15T21:13:01.723 INFO:tasks.workunit.client.0.smithi136.stderr:251099: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 9 SOMETHING remove 2024-09-15T21:13:01.723 INFO:tasks.workunit.client.0.smithi136.stderr:251099: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-09-15T21:13:01.724 INFO:tasks.workunit.client.0.smithi136.stderr:251099: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2024-09-15T21:13:01.724 INFO:tasks.workunit.client.0.smithi136.stderr:251099: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=9 2024-09-15T21:13:01.724 INFO:tasks.workunit.client.0.smithi136.stderr:251099: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2024-09-15T21:13:01.724 INFO:tasks.workunit.client.0.smithi136.stderr:251099: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.9 2024-09-15T21:13:01.724 INFO:tasks.workunit.client.0.smithi136.stderr:251099: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-15T21:13:01.724 INFO:tasks.workunit.client.0.smithi136.stderr:251099: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-15T21:13:01.724 INFO:tasks.workunit.client.0.smithi136.stderr:251099: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-15T21:13:01.724 INFO:tasks.workunit.client.0.smithi136.stderr:251099: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-15T21:13:01.724 INFO:tasks.workunit.client.0.smithi136.stderr:251099: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-15T21:13:01.724 INFO:tasks.workunit.client.0.smithi136.stderr:251099: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-15T21:13:01.724 INFO:tasks.workunit.client.0.smithi136.stderr:251099: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 9 SOMETHING remove 2024-09-15T21:13:01.724 INFO:tasks.workunit.client.0.smithi136.stderr:251099: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-15T21:13:01.724 INFO:tasks.workunit.client.0.smithi136.stderr:251099: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-15T21:13:01.724 INFO:tasks.workunit.client.0.smithi136.stderr:251099: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=9 2024-09-15T21:13:01.725 INFO:tasks.workunit.client.0.smithi136.stderr:251099: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-15T21:13:01.725 INFO:tasks.workunit.client.0.smithi136.stderr:251099: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/9 2024-09-15T21:13:01.725 INFO:tasks.workunit.client.0.smithi136.stderr:251099: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/9 SOMETHING remove 2024-09-15T21:13:01.725 INFO:tasks.workunit.client.0.smithi136.stderr:251099: remove 2#2:eb822e21:::SOMETHING:head# 2024-09-15T21:13:01.725 INFO:tasks.workunit.client.0.smithi136.stderr:251099: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 9 2024-09-15T21:13:01.725 INFO:tasks.workunit.client.0.smithi136.stderr:251099: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2024-09-15T21:13:01.725 INFO:tasks.workunit.client.0.smithi136.stderr:251099: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-15T21:13:01.725 INFO:tasks.workunit.client.0.smithi136.stderr:251099: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=9 2024-09-15T21:13:01.725 INFO:tasks.workunit.client.0.smithi136.stderr:251099: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-15T21:13:01.725 INFO:tasks.workunit.client.0.smithi136.stderr:251099: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/9 2024-09-15T21:13:01.725 INFO:tasks.workunit.client.0.smithi136.stderr:251099: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=ce4892c7-6c6d-4b4f-8e6f-07a3e11133e5 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-15T21:13:01.725 INFO:tasks.workunit.client.0.smithi136.stderr:251099: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T21:13:01.726 INFO:tasks.workunit.client.0.smithi136.stderr:251099: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T21:13:01.726 INFO:tasks.workunit.client.0.smithi136.stderr:251099: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T21:13:01.726 INFO:tasks.workunit.client.0.smithi136.stderr:251099: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/9' 2024-09-15T21:13:01.726 INFO:tasks.workunit.client.0.smithi136.stderr:251099: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/9/journal' 2024-09-15T21:13:01.726 INFO:tasks.workunit.client.0.smithi136.stderr:251099: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-15T21:13:01.726 INFO:tasks.workunit.client.0.smithi136.stderr:251099: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-15T21:13:01.726 INFO:tasks.workunit.client.0.smithi136.stderr:251099: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-15T21:13:01.726 INFO:tasks.workunit.client.0.smithi136.stderr:251099: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-15T21:13:01.726 INFO:tasks.workunit.client.0.smithi136.stderr:251099: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-15T21:13:01.726 INFO:tasks.workunit.client.0.smithi136.stderr:251099: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-15T21:13:01.726 INFO:tasks.workunit.client.0.smithi136.stderr:251099: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-15T21:13:01.727 INFO:tasks.workunit.client.0.smithi136.stderr:251099: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T21:13:01.727 INFO:tasks.workunit.client.0.smithi136.stderr:251099: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52602 2024-09-15T21:13:01.727 INFO:tasks.workunit.client.0.smithi136.stderr:251099: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.52602/$cluster-$name.asok' 2024-09-15T21:13:01.727 INFO:tasks.workunit.client.0.smithi136.stderr:251099: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.52602/$cluster-$name.asok' 2024-09-15T21:13:01.727 INFO:tasks.workunit.client.0.smithi136.stderr:251099: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-15T21:13:01.727 INFO:tasks.workunit.client.0.smithi136.stderr:251099: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-15T21:13:01.727 INFO:tasks.workunit.client.0.smithi136.stderr:251099: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-15T21:13:01.727 INFO:tasks.workunit.client.0.smithi136.stderr:251099: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T21:13:01.727 INFO:tasks.workunit.client.0.smithi136.stderr:251099: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T21:13:01.727 INFO:tasks.workunit.client.0.smithi136.stderr:251099: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T21:13:01.728 INFO:tasks.workunit.client.0.smithi136.stderr:251099: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T21:13:01.728 INFO:tasks.workunit.client.0.smithi136.stderr:251099: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-15T21:13:01.728 INFO:tasks.workunit.client.0.smithi136.stderr:251099: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-15T21:13:01.728 INFO:tasks.workunit.client.0.smithi136.stderr:251099: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/9 2024-09-15T21:13:01.728 INFO:tasks.workunit.client.0.smithi136.stderr:251099: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.9 2024-09-15T21:13:01.728 INFO:tasks.workunit.client.0.smithi136.stderr:251099: start osd.9 2024-09-15T21:13:01.728 INFO:tasks.workunit.client.0.smithi136.stderr:251099: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 9 --fsid=ce4892c7-6c6d-4b4f-8e6f-07a3e11133e5 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/9 --osd-journal=td/osd-scrub-repair/9/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.52602/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T21:13:06.087 INFO:tasks.workunit.client.0.smithi136.stderr:251097: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/5/whoami 2024-09-15T21:13:06.087 INFO:tasks.workunit.client.0.smithi136.stderr:251097: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 5 = 5 ']' 2024-09-15T21:13:06.087 INFO:tasks.workunit.client.0.smithi136.stderr:251097: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-15T21:13:06.087 INFO:tasks.workunit.client.0.smithi136.stderr:251097: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-15T21:13:06.087 INFO:tasks.workunit.client.0.smithi136.stderr:251097: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-15T21:13:06.087 INFO:tasks.workunit.client.0.smithi136.stderr:251097: 2024-09-15T21:13:01.591+0000 7f9d7a215640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:13:06.087 INFO:tasks.workunit.client.0.smithi136.stderr:251097: 2024-09-15T21:13:01.594+0000 7f9d7a215640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:13:06.087 INFO:tasks.workunit.client.0.smithi136.stderr:251097: 2024-09-15T21:13:01.596+0000 7f9d7a215640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:13:06.088 INFO:tasks.workunit.client.0.smithi136.stderr:251097: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 5 2024-09-15T21:13:06.088 INFO:tasks.workunit.client.0.smithi136.stderr:251097: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-15T21:13:06.088 INFO:tasks.workunit.client.0.smithi136.stderr:251097: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=5 2024-09-15T21:13:06.088 INFO:tasks.workunit.client.0.smithi136.stderr:251097: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-15T21:13:06.088 INFO:tasks.workunit.client.0.smithi136.stderr:251097: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-15T21:13:06.088 INFO:tasks.workunit.client.0.smithi136.stderr:251097: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:13:06.088 INFO:tasks.workunit.client.0.smithi136.stderr:251097: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-15T21:13:06.088 INFO:tasks.workunit.client.0.smithi136.stderr:251097: 0 2024-09-15T21:13:06.088 INFO:tasks.workunit.client.0.smithi136.stderr:251097: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:13:06.088 INFO:tasks.workunit.client.0.smithi136.stderr:251097: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-15T21:13:06.088 INFO:tasks.workunit.client.0.smithi136.stderr:251097: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:13:06.088 INFO:tasks.workunit.client.0.smithi136.stderr:251097: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:13:06.088 INFO:tasks.workunit.client.0.smithi136.stderr:251097: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:13:06.089 INFO:tasks.workunit.client.0.smithi136.stderr:251097: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-15T21:13:06.089 INFO:tasks.workunit.client.0.smithi136.stderr:251097: 1 2024-09-15T21:13:06.089 INFO:tasks.workunit.client.0.smithi136.stderr:251097: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:13:06.089 INFO:tasks.workunit.client.0.smithi136.stderr:251097: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-15T21:13:06.089 INFO:tasks.workunit.client.0.smithi136.stderr:251097: 2024-09-15T21:13:03.206+0000 7f9d7a215640 -1 Falling back to public interface 2024-09-15T21:13:06.089 INFO:tasks.workunit.client.0.smithi136.stderr:251097: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:13:06.089 INFO:tasks.workunit.client.0.smithi136.stderr:251097: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:13:06.089 INFO:tasks.workunit.client.0.smithi136.stderr:251097: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:13:06.089 INFO:tasks.workunit.client.0.smithi136.stderr:251097: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-15T21:13:06.089 INFO:tasks.workunit.client.0.smithi136.stderr:251097: 2 2024-09-15T21:13:06.089 INFO:tasks.workunit.client.0.smithi136.stderr:251097: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:13:06.089 INFO:tasks.workunit.client.0.smithi136.stderr:251097: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-15T21:13:06.089 INFO:tasks.workunit.client.0.smithi136.stderr:251097: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:13:06.089 INFO:tasks.workunit.client.0.smithi136.stderr:251097: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:13:06.089 INFO:tasks.workunit.client.0.smithi136.stderr:251097: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:13:06.090 INFO:tasks.workunit.client.0.smithi136.stderr:251097: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-15T21:13:06.090 INFO:tasks.workunit.client.0.smithi136.stderr:251097: 3 2024-09-15T21:13:06.090 INFO:tasks.workunit.client.0.smithi136.stderr:251097: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:13:06.090 INFO:tasks.workunit.client.0.smithi136.stderr:251097: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-15T21:13:06.090 INFO:tasks.workunit.client.0.smithi136.stderr:251097: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:13:06.469 INFO:tasks.workunit.client.0.smithi136.stderr:251097: 2024-09-15T21:13:06.083+0000 7f9d7a215640 -51099: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/9/whoami 2024-09-15T21:13:06.469 INFO:tasks.workunit.client.0.smithi136.stderr:251099: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 9 = 9 ']' 2024-09-15T21:13:06.469 INFO:tasks.workunit.client.0.smithi136.stderr:251099: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-15T21:13:06.469 INFO:tasks.workunit.client.0.smithi136.stderr:251099: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-15T21:13:06.469 INFO:tasks.workunit.client.0.smithi136.stderr:251099: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-15T21:13:06.470 INFO:tasks.workunit.client.0.smithi136.stderr:251099: 2024-09-15T21:13:01.749+0000 7f99f5c0d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:13:06.470 INFO:tasks.workunit.client.0.smithi136.stderr:251099: 2024-09-15T21:13:01.752+0000 7f99f5c0d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:13:06.470 INFO:tasks.workunit.client.0.smithi136.stderr:251099: 2024-09-15T21:13:01.753+0000 7f99f5c0d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:13:06.470 INFO:tasks.workunit.client.0.smithi136.stderr:251099: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 9 2024-09-15T21:13:06.470 INFO:tasks.workunit.client.0.smithi136.stderr:251099: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-15T21:13:06.470 INFO:tasks.workunit.client.0.smithi136.stderr:251099: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=9 2024-09-15T21:13:06.470 INFO:tasks.workunit.client.0.smithi136.stderr:251099: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-15T21:13:06.470 INFO:tasks.workunit.client.0.smithi136.stderr:251099: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-15T21:13:06.470 INFO:tasks.workunit.client.0.smithi136.stderr:251099: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:13:06.470 INFO:tasks.workunit.client.0.smithi136.stderr:251099: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-15T21:13:06.470 INFO:tasks.workunit.client.0.smithi136.stderr:251099: 0 2024-09-15T21:13:06.470 INFO:tasks.workunit.client.0.smithi136.stderr:251099: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:13:06.470 INFO:tasks.workunit.client.0.smithi136.stderr:251099: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.9 up' 2024-09-15T21:13:06.470 INFO:tasks.workunit.client.0.smithi136.stderr:251099: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:13:06.471 INFO:tasks.workunit.client.0.smithi136.stderr:251099: 2024-09-15T21:13:03.346+0000 7f99f5c0d640 -1 Falling back to public interface 2024-09-15T21:13:06.471 INFO:tasks.workunit.client.0.smithi136.stderr:251099: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:13:06.471 INFO:tasks.workunit.client.0.smithi136.stderr:251099: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:13:06.471 INFO:tasks.workunit.client.0.smithi136.stderr:251099: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-15T21:13:06.471 INFO:tasks.workunit.client.0.smithi136.stderr:251099: 1 2024-09-15T21:13:06.471 INFO:tasks.workunit.client.0.smithi136.stderr:251099: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:13:06.471 INFO:tasks.workunit.client.0.smithi136.stderr:251099: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.9 up' 2024-09-15T21:13:06.471 INFO:tasks.workunit.client.0.smithi136.stderr:251099: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:13:06.471 INFO:tasks.workunit.client.0.smithi136.stderr:251099: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:13:06.471 INFO:tasks.workunit.client.0.smithi136.stderr:251099: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:13:06.471 INFO:tasks.workunit.client.0.smithi136.stderr:251099: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-15T21:13:06.471 INFO:tasks.workunit.client.0.smithi136.stderr:251099: 2 2024-09-15T21:13:06.471 INFO:tasks.workunit.client.0.smithi136.stderr:251099: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:13:06.472 INFO:tasks.workunit.client.0.smithi136.stderr:251099: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.9 up' 2024-09-15T21:13:06.472 INFO:tasks.workunit.client.0.smithi136.stderr:251099: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:13:06.472 INFO:tasks.workunit.client.0.smithi136.stderr:251099: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:13:06.472 INFO:tasks.workunit.client.0.smithi136.stderr:251099: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:13:06.472 INFO:tasks.workunit.client.0.smithi136.stderr:251099: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-15T21:13:06.472 INFO:tasks.workunit.client.0.smithi136.stderr:251099: 3 2024-09-15T21:13:06.472 INFO:tasks.workunit.client.0.smithi136.stderr:251099: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:13:06.472 INFO:tasks.workunit.client.0.smithi136.stderr:251099: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.9 up' 2024-09-15T21:13:06.472 INFO:tasks.workunit.client.0.smithi136.stderr:251099: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:13:08.194 INFO:tasks.workunit.client.0.smithi136.stderr:251099: 2024-09-15T21:13:06.466+0000 7f99f5c0d640 -1 osd.5 92 log_to_monitors true 2024-09-15T21:13:08.194 INFO:tasks.workunit.client.0.smithi136.stderr:251097: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:13:08.195 INFO:tasks.workunit.client.0.smithi136.stderr:251097: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:13:08.195 INFO:tasks.workunit.client.0.smithi136.stderr:251097: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-15T21:13:08.195 INFO:tasks.workunit.client.0.smithi136.stderr:251097: 4 2024-09-15T21:13:08.195 INFO:tasks.workunit.client.0.smithi136.stderr:251097: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:13:08.195 INFO:tasks.workunit.client.0.smithi136.stderr:251097: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-15T21:13:08.195 INFO:tasks.workunit.client.0.smithi136.stderr:251097: osd.5 up in weight 1 up_from 97 up_thru 84 down_at 93 last_clean_interval [84,92) [v2:127.0.0.1:6842/1531117251,v1:127.0.0.1:6843/1531117251] [v2:127.0.0.1:6844/1531117251,v1:127.0.0.1:6845/1531117251] exists,up 9918af07-0bbe-4711-bea1-402a4b5b6247 2024-09-15T21:13:08.195 INFO:tasks.workunit.client.0.smithi136.stderr:251097: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-15T21:13:08.195 INFO:tasks.workunit.client.0.smithi136.stderr:251097: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-15T21:13:08.195 INFO:tasks.workunit.client.0.smithi136.stderr:251097: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-15T21:13:08.195 INFO:tasks.workunit.client.0.smithi136.stderr:251097: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2024-09-15T21:13:08.195 INFO:tasks.workunit.client.0.smithi136.stderr:251097: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-15T21:13:08.196 INFO:tasks.workunit.client.0.smithi136.stderr:251097: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-15T21:13:08.196 INFO:tasks.workunit.client.0.smithi136.stderr:251097: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-15T21:13:08.196 INFO:tasks.workunit.client.0.smithi136.stderr:251097: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-15T21:13:08.196 INFO:tasks.workunit.client.0.smithi136.stderr:251097: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-15T21:13:08.196 INFO:tasks.workunit.client.0.smithi136.stderr:251097: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-15T21:13:08.196 INFO:tasks.workunit.client.0.smithi136.stderr:251097: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-15T21:13:08.196 INFO:tasks.workunit.client.0.smithi136.stderr:251097: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-15T21:13:08.196 INFO:tasks.workunit.client.0.smithi136.stderr:251097: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-15T21:13:08.196 INFO:tasks.workunit.client.0.smithi136.stderr:251097: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-15T21:13:08.196 INFO:tasks.workunit.client.0.smithi136.stderr:251097: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-15T21:13:08.196 INFO:tasks.workunit.client.0.smithi136.stderr:251097: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-15T21:13:08.196 INFO:tasks.workunit.client.0.smithi136.stderr:251097: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-15T21:13:08.196 INFO:tasks.workunit.client.0.smithi136.stderr:251097: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-15T21:13:08.196 INFO:tasks.workunit.client.0.smithi136.stderr:251097: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-15T21:13:08.197 INFO:tasks.workunit.client.0.smithi136.stderr:251097: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-15T21:13:08.197 INFO:tasks.workunit.client.0.smithi136.stderr:251097: 1 2024-09-15T21:13:08.197 INFO:tasks.workunit.client.0.smithi136.stderr:251097: 2 2024-09-15T21:13:08.197 INFO:tasks.workunit.client.0.smithi136.stderr:251097: 3 2024-09-15T21:13:08.197 INFO:tasks.workunit.client.0.smithi136.stderr:251097: 4 2024-09-15T21:13:08.197 INFO:tasks.workunit.client.0.smithi136.stderr:251097: 5 2024-09-15T21:13:08.197 INFO:tasks.workunit.client.0.smithi136.stderr:251097: 6 2024-09-15T21:13:08.197 INFO:tasks.workunit.client.0.smithi136.stderr:251097: 7 2024-09-15T21:13:08.197 INFO:tasks.workunit.client.0.smithi136.stderr:251097: 8 2024-09-15T21:13:08.197 INFO:tasks.workunit.client.0.smithi136.stderr:251097: 9' 2024-09-15T21:13:08.197 INFO:tasks.workunit.client.0.smithi136.stderr:251097: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-15T21:13:08.197 INFO:tasks.workunit.client.0.smithi136.stderr:251097: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T21:13:08.197 INFO:tasks.workunit.client.0.smithi136.stderr:251097: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-15T21:13:08.197 INFO:tasks.workunit.client.0.smithi136.stderr:251097: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836572 2024-09-15T21:13:08.198 INFO:tasks.workunit.client.0.smithi136.stderr:251097: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836572 2024-09-15T21:13:08.198 INFO:tasks.workunit.client.0.smithi136.stderr:251097: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836572' 2024-09-15T21:13:08.198 INFO:tasks.workunit.client.0.smithi136.stderr:251097: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T21:13:08.564 INFO:tasks.workunit.client.0.smithi136.stderr:251097: /1 osd.9 92 log_to_monitors true 2024-09-15T21:13:08.564 INFO:tasks.workunit.client.0.smithi136.stderr:251099: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:13:08.564 INFO:tasks.workunit.client.0.smithi136.stderr:251099: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:13:08.564 INFO:tasks.workunit.client.0.smithi136.stderr:251099: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-15T21:13:08.564 INFO:tasks.workunit.client.0.smithi136.stderr:251099: 4 2024-09-15T21:13:08.564 INFO:tasks.workunit.client.0.smithi136.stderr:251099: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:13:08.564 INFO:tasks.workunit.client.0.smithi136.stderr:251099: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.9 up' 2024-09-15T21:13:08.564 INFO:tasks.workunit.client.0.smithi136.stderr:251099: osd.9 up in weight 1 up_from 97 up_thru 0 down_at 93 last_clean_interval [90,92) [v2:127.0.0.1:6874/11506240,v1:127.0.0.1:6875/11506240] [v2:127.0.0.1:6876/11506240,v1:127.0.0.1:6877/11506240] exists,up 78ecdb67-c9df-44e1-b77a-4e850e67a453 2024-09-15T21:13:08.564 INFO:tasks.workunit.client.0.smithi136.stderr:251099: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-15T21:13:08.564 INFO:tasks.workunit.client.0.smithi136.stderr:251099: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-15T21:13:08.565 INFO:tasks.workunit.client.0.smithi136.stderr:251099: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-15T21:13:08.565 INFO:tasks.workunit.client.0.smithi136.stderr:251099: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2024-09-15T21:13:08.565 INFO:tasks.workunit.client.0.smithi136.stderr:251099: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-15T21:13:08.565 INFO:tasks.workunit.client.0.smithi136.stderr:251099: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-15T21:13:08.565 INFO:tasks.workunit.client.0.smithi136.stderr:251099: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-15T21:13:08.565 INFO:tasks.workunit.client.0.smithi136.stderr:251099: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-15T21:13:08.565 INFO:tasks.workunit.client.0.smithi136.stderr:251099: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-15T21:13:08.565 INFO:tasks.workunit.client.0.smithi136.stderr:251099: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-15T21:13:08.565 INFO:tasks.workunit.client.0.smithi136.stderr:251099: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-15T21:13:08.565 INFO:tasks.workunit.client.0.smithi136.stderr:251099: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-15T21:13:08.565 INFO:tasks.workunit.client.0.smithi136.stderr:251099: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-15T21:13:08.565 INFO:tasks.workunit.client.0.smithi136.stderr:251099: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-15T21:13:08.566 INFO:tasks.workunit.client.0.smithi136.stderr:251099: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-15T21:13:08.566 INFO:tasks.workunit.client.0.smithi136.stderr:251099: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-15T21:13:08.566 INFO:tasks.workunit.client.0.smithi136.stderr:251099: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-15T21:13:08.566 INFO:tasks.workunit.client.0.smithi136.stderr:251099: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-15T21:13:08.566 INFO:tasks.workunit.client.0.smithi136.stderr:251099: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-15T21:13:08.566 INFO:tasks.workunit.client.0.smithi136.stderr:251099: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-15T21:13:08.566 INFO:tasks.workunit.client.0.smithi136.stderr:251099: 1 2024-09-15T21:13:08.566 INFO:tasks.workunit.client.0.smithi136.stderr:251099: 2 2024-09-15T21:13:08.566 INFO:tasks.workunit.client.0.smithi136.stderr:251099: 3 2024-09-15T21:13:08.566 INFO:tasks.workunit.client.0.smithi136.stderr:251099: 4 2024-09-15T21:13:08.566 INFO:tasks.workunit.client.0.smithi136.stderr:251099: 5 2024-09-15T21:13:08.566 INFO:tasks.workunit.client.0.smithi136.stderr:251099: 6 2024-09-15T21:13:08.566 INFO:tasks.workunit.client.0.smithi136.stderr:251099: 7 2024-09-15T21:13:08.566 INFO:tasks.workunit.client.0.smithi136.stderr:251099: 8 2024-09-15T21:13:08.566 INFO:tasks.workunit.client.0.smithi136.stderr:251099: 9' 2024-09-15T21:13:08.567 INFO:tasks.workunit.client.0.smithi136.stderr:251099: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-15T21:13:08.567 INFO:tasks.workunit.client.0.smithi136.stderr:251099: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T21:13:08.567 INFO:tasks.workunit.client.0.smithi136.stderr:251099: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-15T21:13:08.567 INFO:tasks.workunit.client.0.smithi136.stderr:251099: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836573 2024-09-15T21:13:08.567 INFO:tasks.workunit.client.0.smithi136.stderr:251099: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836573 2024-09-15T21:13:08.567 INFO:tasks.workunit.client.0.smithi136.stderr:251099: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836573' 2024-09-15T21:13:08.567 INFO:tasks.workunit.client.0.smithi136.stderr:251099: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T21:13:09.478 INFO:tasks.workunit.client.0.smithi136.stderr:251099: //home/ubu/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-15T21:13:09.479 INFO:tasks.workunit.client.0.smithi136.stderr:251097: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=42949673049 2024-09-15T21:13:09.479 INFO:tasks.workunit.client.0.smithi136.stderr:251097: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 42949673049 2024-09-15T21:13:09.479 INFO:tasks.workunit.client.0.smithi136.stderr:251097: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836572 1-42949673049' 2024-09-15T21:13:09.479 INFO:tasks.workunit.client.0.smithi136.stderr:251097: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T21:13:09.479 INFO:tasks.workunit.client.0.smithi136.stderr:251097: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-15T21:13:09.479 INFO:tasks.workunit.client.0.smithi136.stderr:251097: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=64424509519 2024-09-15T21:13:09.479 INFO:tasks.workunit.client.0.smithi136.stderr:251097: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 64424509519 2024-09-15T21:13:09.479 INFO:tasks.workunit.client.0.smithi136.stderr:251097: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836572 1-42949673049 2-64424509519' 2024-09-15T21:13:09.479 INFO:tasks.workunit.client.0.smithi136.stderr:251097: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T21:13:09.479 INFO:tasks.workunit.client.0.smithi136.stderr:251097: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-15T21:13:09.479 INFO:tasks.workunit.client.0.smithi136.stderr:251097: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=313532612628 2024-09-15T21:13:09.480 INFO:tasks.workunit.client.0.smithi136.stderr:251097: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 313532612628 2024-09-15T21:13:09.480 INFO:tasks.workunit.client.0.smithi136.stderr:251097: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836572 1-42949673049 2-64424509519 3-313532612628' 2024-09-15T21:13:09.480 INFO:tasks.workunit.client.0.smithi136.stderr:251097: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T21:13:09.480 INFO:tasks.workunit.client.0.smithi136.stderr:251097: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-09-15T21:13:09.480 INFO:tasks.workunit.client.0.smithi136.stderr:251097: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=107374182467 2024-09-15T21:13:09.480 INFO:tasks.workunit.client.0.smithi136.stderr:251097: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 107374182467 2024-09-15T21:13:09.480 INFO:tasks.workunit.client.0.smithi136.stderr:251097: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836572 1-42949673049 2-64424509519 3-313532612628 4-107374182467' 2024-09-15T21:13:09.480 INFO:tasks.workunit.client.0.smithi136.stderr:251097: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T21:13:09.480 INFO:tasks.workunit.client.0.smithi136.stderr:251097: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-09-15T21:13:09.480 INFO:tasks.workunit.client.0.smithi136.stderr:251097: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=416611827714 2024-09-15T21:13:09.480 INFO:tasks.workunit.client.0.smithi136.stderr:251097: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 416611827714 2024-09-15T21:13:09.480 INFO:tasks.workunit.client.0.smithi136.stderr:251097: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836572 1-42949673049 2-64424509519 3-313532612628 4-107374182467 5-416611827714' 2024-09-15T21:13:09.481 INFO:tasks.workunit.client.0.smithi136.stderr:251097: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T21:13:09.481 INFO:tasks.workunit.client.0.smithi136.stderr:251097: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-09-15T21:13:09.481 INFO:tasks.workunit.client.0.smithi136.stderr:251097: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=150323855415 2024-09-15T21:13:09.481 INFO:tasks.workunit.client.0.smithi136.stderr:251097: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 150323855415 2024-09-15T21:13:09.481 INFO:tasks.workunit.client.0.smithi136.stderr:251097: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836572 1-42949673049 2-64424509519 3-313532612628 4-107374182467 5-416611827714 6-150323855415' 2024-09-15T21:13:09.481 INFO:tasks.workunit.client.0.smithi136.stderr:251097: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T21:13:09.481 INFO:tasks.workunit.client.0.smithi136.stderr:251097: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2024-09-15T21:13:09.604 INFO:tasks.workunit.client.0.smithi136.stderr:251097: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_ntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-15T21:13:09.604 INFO:tasks.workunit.client.0.smithi136.stderr:251099: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=42949673050 2024-09-15T21:13:09.604 INFO:tasks.workunit.client.0.smithi136.stderr:251099: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 42949673050 2024-09-15T21:13:09.604 INFO:tasks.workunit.client.0.smithi136.stderr:251099: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836573 1-42949673050' 2024-09-15T21:13:09.604 INFO:tasks.workunit.client.0.smithi136.stderr:251099: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T21:13:09.604 INFO:tasks.workunit.client.0.smithi136.stderr:251099: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-15T21:13:09.604 INFO:tasks.workunit.client.0.smithi136.stderr:251099: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=64424509520 2024-09-15T21:13:09.604 INFO:tasks.workunit.client.0.smithi136.stderr:251099: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 64424509520 2024-09-15T21:13:09.604 INFO:tasks.workunit.client.0.smithi136.stderr:251099: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836573 1-42949673050 2-64424509520' 2024-09-15T21:13:09.604 INFO:tasks.workunit.client.0.smithi136.stderr:251099: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T21:13:09.605 INFO:tasks.workunit.client.0.smithi136.stderr:251099: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-15T21:13:09.605 INFO:tasks.workunit.client.0.smithi136.stderr:251099: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=313532612629 2024-09-15T21:13:09.605 INFO:tasks.workunit.client.0.smithi136.stderr:251099: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 313532612629 2024-09-15T21:13:09.605 INFO:tasks.workunit.client.0.smithi136.stderr:251099: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836573 1-42949673050 2-64424509520 3-313532612629' 2024-09-15T21:13:09.605 INFO:tasks.workunit.client.0.smithi136.stderr:251099: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T21:13:09.605 INFO:tasks.workunit.client.0.smithi136.stderr:251099: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-09-15T21:13:09.605 INFO:tasks.workunit.client.0.smithi136.stderr:251099: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=107374182468 2024-09-15T21:13:09.605 INFO:tasks.workunit.client.0.smithi136.stderr:251099: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 107374182468 2024-09-15T21:13:09.605 INFO:tasks.workunit.client.0.smithi136.stderr:251099: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836573 1-42949673050 2-64424509520 3-313532612629 4-107374182468' 2024-09-15T21:13:09.605 INFO:tasks.workunit.client.0.smithi136.stderr:251099: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T21:13:09.605 INFO:tasks.workunit.client.0.smithi136.stderr:251099: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-09-15T21:13:09.605 INFO:tasks.workunit.client.0.smithi136.stderr:251099: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=416611827715 2024-09-15T21:13:09.605 INFO:tasks.workunit.client.0.smithi136.stderr:251099: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 416611827715 2024-09-15T21:13:09.606 INFO:tasks.workunit.client.0.smithi136.stderr:251099: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836573 1-42949673050 2-64424509520 3-313532612629 4-107374182468 5-416611827715' 2024-09-15T21:13:09.606 INFO:tasks.workunit.client.0.smithi136.stderr:251099: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T21:13:09.606 INFO:tasks.workunit.client.0.smithi136.stderr:251099: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-09-15T21:13:09.606 INFO:tasks.workunit.client.0.smithi136.stderr:251099: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=150323855416 2024-09-15T21:13:09.606 INFO:tasks.workunit.client.0.smithi136.stderr:251099: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 150323855416 2024-09-15T21:13:09.606 INFO:tasks.workunit.client.0.smithi136.stderr:251099: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836573 1-42949673050 2-64424509520 3-313532612629 4-107374182468 5-416611827715 6-150323855416' 2024-09-15T21:13:09.606 INFO:tasks.workunit.client.0.smithi136.stderr:251099: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T21:13:09.606 INFO:tasks.workunit.client.0.smithi136.stderr:251099: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2024-09-15T21:13:10.032 INFO:tasks.workunit.client.0.smithi136.stderr:251099: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats:pg_stats: seq=171798691887 2024-09-15T21:13:10.033 INFO:tasks.workunit.client.0.smithi136.stderr:251097: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 171798691887 2024-09-15T21:13:10.033 INFO:tasks.workunit.client.0.smithi136.stderr:251097: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836572 1-42949673049 2-64424509519 3-313532612628 4-107374182467 5-416611827714 6-150323855415 7-171798691887' 2024-09-15T21:13:10.033 INFO:tasks.workunit.client.0.smithi136.stderr:251097: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T21:13:10.033 INFO:tasks.workunit.client.0.smithi136.stderr:251097: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2024-09-15T21:13:10.033 INFO:tasks.workunit.client.0.smithi136.stderr:251097: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=193273528361 2024-09-15T21:13:10.033 INFO:tasks.workunit.client.0.smithi136.stderr:251097: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 193273528361 2024-09-15T21:13:10.033 INFO:tasks.workunit.client.0.smithi136.stderr:251097: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836572 1-42949673049 2-64424509519 3-313532612628 4-107374182467 5-416611827714 6-150323855415 7-171798691887 8-193273528361' 2024-09-15T21:13:10.033 INFO:tasks.workunit.client.0.smithi136.stderr:251097: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T21:13:10.033 INFO:tasks.workunit.client.0.smithi136.stderr:251097: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2024-09-15T21:13:10.033 INFO:tasks.workunit.client.0.smithi136.stderr:251097: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=416611827714 2024-09-15T21:13:10.033 INFO:tasks.workunit.client.0.smithi136.stderr:251097: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 416611827714 2024-09-15T21:13:10.033 INFO:tasks.workunit.client.0.smithi136.stderr:251097: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836572 1-42949673049 2-64424509519 3-313532612628 4-107374182467 5-416611827714 6-150323855415 7-171798691887 8-193273528361 9-416611827714' 2024-09-15T21:13:10.034 INFO:tasks.workunit.client.0.smithi136.stderr:251097: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T21:13:10.034 INFO:tasks.workunit.client.0.smithi136.stderr:251097: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836572 2024-09-15T21:13:10.034 INFO:tasks.workunit.client.0.smithi136.stderr:251097: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T21:13:10.034 INFO:tasks.workunit.client.0.smithi136.stderr:251097: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-15T21:13:10.034 INFO:tasks.workunit.client.0.smithi136.stderr:251097: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836572 2024-09-15T21:13:10.034 INFO:tasks.workunit.client.0.smithi136.stderr:251097: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T21:13:10.034 INFO:tasks.workunit.client.0.smithi136.stderr:251097: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836572 2024-09-15T21:13:10.034 INFO:tasks.workunit.client.0.smithi136.stderr:251097: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836572' 2024-09-15T21:13:10.034 INFO:tasks.workunit.client.0.smithi136.stderr:251097: waiting osd.0 seq 21474836572 2024-09-15T21:13:10.034 INFO:tasks.workunit.client.0.smithi136.stderr:251097: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T21:13:10.034 INFO:tasks.workunit.client.0.smithi136.stderr:251097: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836574 -lt 21474836572 2024-09-15T21:13:10.034 INFO:tasks.workunit.client.0.smithi136.stderr:251097: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T21:13:10.034 INFO:tasks.workunit.client.0.smithi136.stderr:251097: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-42949673049 2024-09-15T21:13:10.034 INFO:tasks.workunit.client.0.smithi136.stderr:251097: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T21:13:10.035 INFO:tasks.workunit.client.0.smithi136.stderr:251097: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-15T21:13:10.035 INFO:tasks.workunit.client.0.smithi136.stderr:251097: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-42949673049 2024-09-15T21:13:10.035 INFO:tasks.workunit.client.0.smithi136.stderr:251097: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T21:13:10.035 INFO:tasks.workunit.client.0.smithi136.stderr:251097: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=42949673049 2024-09-15T21:13:10.035 INFO:tasks.workunit.client.0.smithi136.stderr:251097: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 42949673049' 2024-09-15T21:13:10.035 INFO:tasks.workunit.client.0.smithi136.stderr:251097: waiting osd.1 seq 42949673049 2024-09-15T21:13:10.109 INFO:tasks.workunit.client.0.smithi136.stderr:251097: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_s seq=171798691888 2024-09-15T21:13:10.109 INFO:tasks.workunit.client.0.smithi136.stderr:251099: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 171798691888 2024-09-15T21:13:10.109 INFO:tasks.workunit.client.0.smithi136.stderr:251099: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836573 1-42949673050 2-64424509520 3-313532612629 4-107374182468 5-416611827715 6-150323855416 7-171798691888' 2024-09-15T21:13:10.109 INFO:tasks.workunit.client.0.smithi136.stderr:251099: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T21:13:10.109 INFO:tasks.workunit.client.0.smithi136.stderr:251099: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2024-09-15T21:13:10.110 INFO:tasks.workunit.client.0.smithi136.stderr:251099: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=193273528362 2024-09-15T21:13:10.110 INFO:tasks.workunit.client.0.smithi136.stderr:251099: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 193273528362 2024-09-15T21:13:10.110 INFO:tasks.workunit.client.0.smithi136.stderr:251099: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836573 1-42949673050 2-64424509520 3-313532612629 4-107374182468 5-416611827715 6-150323855416 7-171798691888 8-193273528362' 2024-09-15T21:13:10.110 INFO:tasks.workunit.client.0.smithi136.stderr:251099: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T21:13:10.110 INFO:tasks.workunit.client.0.smithi136.stderr:251099: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2024-09-15T21:13:10.110 INFO:tasks.workunit.client.0.smithi136.stderr:251099: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=416611827715 2024-09-15T21:13:10.110 INFO:tasks.workunit.client.0.smithi136.stderr:251099: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 416611827715 2024-09-15T21:13:10.110 INFO:tasks.workunit.client.0.smithi136.stderr:251099: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836573 1-42949673050 2-64424509520 3-313532612629 4-107374182468 5-416611827715 6-150323855416 7-171798691888 8-193273528362 9-416611827715' 2024-09-15T21:13:10.110 INFO:tasks.workunit.client.0.smithi136.stderr:251099: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T21:13:10.110 INFO:tasks.workunit.client.0.smithi136.stderr:251099: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836573 2024-09-15T21:13:10.110 INFO:tasks.workunit.client.0.smithi136.stderr:251099: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T21:13:10.110 INFO:tasks.workunit.client.0.smithi136.stderr:251099: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-15T21:13:10.110 INFO:tasks.workunit.client.0.smithi136.stderr:251099: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836573 2024-09-15T21:13:10.110 INFO:tasks.workunit.client.0.smithi136.stderr:251099: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T21:13:10.110 INFO:tasks.workunit.client.0.smithi136.stderr:251099: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836573 2024-09-15T21:13:10.111 INFO:tasks.workunit.client.0.smithi136.stderr:251099: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836573' 2024-09-15T21:13:10.111 INFO:tasks.workunit.client.0.smithi136.stderr:251099: waiting osd.0 seq 21474836573 2024-09-15T21:13:10.111 INFO:tasks.workunit.client.0.smithi136.stderr:251099: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T21:13:10.111 INFO:tasks.workunit.client.0.smithi136.stderr:251099: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836574 -lt 21474836573 2024-09-15T21:13:10.111 INFO:tasks.workunit.client.0.smithi136.stderr:251099: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T21:13:10.111 INFO:tasks.workunit.client.0.smithi136.stderr:251099: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-42949673050 2024-09-15T21:13:10.111 INFO:tasks.workunit.client.0.smithi136.stderr:251099: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T21:13:10.111 INFO:tasks.workunit.client.0.smithi136.stderr:251099: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-15T21:13:10.111 INFO:tasks.workunit.client.0.smithi136.stderr:251099: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-42949673050 2024-09-15T21:13:10.111 INFO:tasks.workunit.client.0.smithi136.stderr:251099: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T21:13:10.111 INFO:tasks.workunit.client.0.smithi136.stderr:251099: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=42949673050 2024-09-15T21:13:10.111 INFO:tasks.workunit.client.0.smithi136.stderr:251099: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 42949673050' 2024-09-15T21:13:10.111 INFO:tasks.workunit.client.0.smithi136.stderr:251099: waiting osd.1 seq 42949673050 2024-09-15T21:13:11.249 INFO:tasks.workunit.client.0.smithi136.stderr:251099: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: cetats: ceph osd last-stat-seq 1 2024-09-15T21:13:11.250 INFO:tasks.workunit.client.0.smithi136.stderr:251097: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 42949673050 -lt 42949673049 2024-09-15T21:13:11.250 INFO:tasks.workunit.client.0.smithi136.stderr:251097: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T21:13:11.250 INFO:tasks.workunit.client.0.smithi136.stderr:251097: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-64424509519 2024-09-15T21:13:11.250 INFO:tasks.workunit.client.0.smithi136.stderr:251097: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T21:13:11.250 INFO:tasks.workunit.client.0.smithi136.stderr:251097: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-15T21:13:11.250 INFO:tasks.workunit.client.0.smithi136.stderr:251097: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-64424509519 2024-09-15T21:13:11.250 INFO:tasks.workunit.client.0.smithi136.stderr:251097: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T21:13:11.250 INFO:tasks.workunit.client.0.smithi136.stderr:251097: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=64424509519 2024-09-15T21:13:11.250 INFO:tasks.workunit.client.0.smithi136.stderr:251097: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 64424509519' 2024-09-15T21:13:11.250 INFO:tasks.workunit.client.0.smithi136.stderr:251097: waiting osd.2 seq 64424509519 2024-09-15T21:13:11.250 INFO:tasks.workunit.client.0.smithi136.stderr:251097: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-15T21:13:11.250 INFO:tasks.workunit.client.0.smithi136.stderr:251097: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 64424509520 -lt 64424509519 2024-09-15T21:13:11.251 INFO:tasks.workunit.client.0.smithi136.stderr:251097: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T21:13:11.251 INFO:tasks.workunit.client.0.smithi136.stderr:251097: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 3-313532612628 2024-09-15T21:13:11.251 INFO:tasks.workunit.client.0.smithi136.stderr:251097: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T21:13:11.251 INFO:tasks.workunit.client.0.smithi136.stderr:251097: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=3 2024-09-15T21:13:11.251 INFO:tasks.workunit.client.0.smithi136.stderr:251097: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 3-313532612628 2024-09-15T21:13:11.251 INFO:tasks.workunit.client.0.smithi136.stderr:251097: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T21:13:11.251 INFO:tasks.workunit.client.0.smithi136.stderr:251097: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=313532612628 2024-09-15T21:13:11.251 INFO:tasks.workunit.client.0.smithi136.stderr:251097: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.3 seq 313532612628' 2024-09-15T21:13:11.251 INFO:tasks.workunit.client.0.smithi136.stderr:251097: waiting osd.3 seq 313532612628 2024-09-15T21:13:11.251 INFO:tasks.workunit.client.0.smithi136.stderr:251097: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-15T21:13:11.251 INFO:tasks.workunit.client.0.smithi136.stderr:251097: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 313532612629 -lt 313532612628 2024-09-15T21:13:11.251 INFO:tasks.workunit.client.0.smithi136.stderr:251097: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T21:13:11.251 INFO:tasks.workunit.client.0.smithi136.stderr:251097: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 4-107374182467 2024-09-15T21:13:11.251 INFO:tasks.workunit.client.0.smithi136.stderr:251097: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T21:13:11.251 INFO:tasks.workunit.client.0.smithi136.stderr:251097: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=4 2024-09-15T21:13:11.252 INFO:tasks.workunit.client.0.smithi136.stderr:251097: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 4-107374182467 2024-09-15T21:13:11.252 INFO:tasks.workunit.client.0.smithi136.stderr:251097: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T21:13:11.252 INFO:tasks.workunit.client.0.smithi136.stderr:251097: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=107374182467 2024-09-15T21:13:11.252 INFO:tasks.workunit.client.0.smithi136.stderr:251097: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.4 seq 107374182467' 2024-09-15T21:13:11.252 INFO:tasks.workunit.client.0.smithi136.stderr:251097: waiting osd.4 seq 107374182467 2024-09-15T21:13:11.252 INFO:tasks.workunit.client.0.smithi136.stderr:251097: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 4 2024-09-15T21:13:11.252 INFO:tasks.workunit.client.0.smithi136.stderr:251097: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 107374182468 -lt 107374182467 2024-09-15T21:13:11.252 INFO:tasks.workunit.client.0.smithi136.stderr:251097: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T21:13:11.252 INFO:tasks.workunit.client.0.smithi136.stderr:251097: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 5-416611827714 2024-09-15T21:13:11.320 INFO:tasks.workunit.client.0.smithi136.stderr:251097: //home/ubuntu/ceph osd last-stat-seq 1 2024-09-15T21:13:11.320 INFO:tasks.workunit.client.0.smithi136.stderr:251099: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 42949673050 -lt 42949673050 2024-09-15T21:13:11.320 INFO:tasks.workunit.client.0.smithi136.stderr:251099: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T21:13:11.320 INFO:tasks.workunit.client.0.smithi136.stderr:251099: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-64424509520 2024-09-15T21:13:11.321 INFO:tasks.workunit.client.0.smithi136.stderr:251099: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T21:13:11.321 INFO:tasks.workunit.client.0.smithi136.stderr:251099: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-15T21:13:11.321 INFO:tasks.workunit.client.0.smithi136.stderr:251099: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-64424509520 2024-09-15T21:13:11.321 INFO:tasks.workunit.client.0.smithi136.stderr:251099: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T21:13:11.321 INFO:tasks.workunit.client.0.smithi136.stderr:251099: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=64424509520 2024-09-15T21:13:11.321 INFO:tasks.workunit.client.0.smithi136.stderr:251099: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 64424509520' 2024-09-15T21:13:11.321 INFO:tasks.workunit.client.0.smithi136.stderr:251099: waiting osd.2 seq 64424509520 2024-09-15T21:13:11.321 INFO:tasks.workunit.client.0.smithi136.stderr:251099: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-15T21:13:11.321 INFO:tasks.workunit.client.0.smithi136.stderr:251099: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 64424509520 -lt 64424509520 2024-09-15T21:13:11.321 INFO:tasks.workunit.client.0.smithi136.stderr:251099: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T21:13:11.321 INFO:tasks.workunit.client.0.smithi136.stderr:251099: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 3-313532612629 2024-09-15T21:13:11.321 INFO:tasks.workunit.client.0.smithi136.stderr:251099: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T21:13:11.321 INFO:tasks.workunit.client.0.smithi136.stderr:251099: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=3 2024-09-15T21:13:11.321 INFO:tasks.workunit.client.0.smithi136.stderr:251099: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 3-313532612629 2024-09-15T21:13:11.322 INFO:tasks.workunit.client.0.smithi136.stderr:251099: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T21:13:11.322 INFO:tasks.workunit.client.0.smithi136.stderr:251099: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=313532612629 2024-09-15T21:13:11.322 INFO:tasks.workunit.client.0.smithi136.stderr:251099: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.3 seq 313532612629' 2024-09-15T21:13:11.322 INFO:tasks.workunit.client.0.smithi136.stderr:251099: waiting osd.3 seq 313532612629 2024-09-15T21:13:11.322 INFO:tasks.workunit.client.0.smithi136.stderr:251099: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-15T21:13:11.322 INFO:tasks.workunit.client.0.smithi136.stderr:251099: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 313532612629 -lt 313532612629 2024-09-15T21:13:11.322 INFO:tasks.workunit.client.0.smithi136.stderr:251099: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T21:13:11.322 INFO:tasks.workunit.client.0.smithi136.stderr:251099: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 4-107374182468 2024-09-15T21:13:11.322 INFO:tasks.workunit.client.0.smithi136.stderr:251099: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T21:13:11.322 INFO:tasks.workunit.client.0.smithi136.stderr:251099: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=4 2024-09-15T21:13:11.322 INFO:tasks.workunit.client.0.smithi136.stderr:251099: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 4-107374182468 2024-09-15T21:13:11.322 INFO:tasks.workunit.client.0.smithi136.stderr:251099: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T21:13:11.322 INFO:tasks.workunit.client.0.smithi136.stderr:251099: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=107374182468 2024-09-15T21:13:11.322 INFO:tasks.workunit.client.0.smithi136.stderr:251099: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.4 seq 107374182468' 2024-09-15T21:13:11.323 INFO:tasks.workunit.client.0.smithi136.stderr:251099: waiting osd.4 seq 107374182468 2024-09-15T21:13:11.323 INFO:tasks.workunit.client.0.smithi136.stderr:251099: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 4 2024-09-15T21:13:11.323 INFO:tasks.workunit.client.0.smithi136.stderr:251099: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 107374182468 -lt 107374182468 2024-09-15T21:13:11.323 INFO:tasks.workunit.client.0.smithi136.stderr:251099: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T21:13:11.323 INFO:tasks.workunit.client.0.smithi136.stderr:251099: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 5-416611827715 2024-09-15T21:13:12.191 INFO:tasks.workunit.client.0.smithi136.stderr:251099: //home/ubuntu/cephtest/clphtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T21:13:12.192 INFO:tasks.workunit.client.0.smithi136.stderr:251097: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=5 2024-09-15T21:13:12.192 INFO:tasks.workunit.client.0.smithi136.stderr:251097: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 5-416611827714 2024-09-15T21:13:12.192 INFO:tasks.workunit.client.0.smithi136.stderr:251097: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T21:13:12.192 INFO:tasks.workunit.client.0.smithi136.stderr:251097: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=416611827714 2024-09-15T21:13:12.192 INFO:tasks.workunit.client.0.smithi136.stderr:251097: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.5 seq 416611827714' 2024-09-15T21:13:12.192 INFO:tasks.workunit.client.0.smithi136.stderr:251097: waiting osd.5 seq 416611827714 2024-09-15T21:13:12.192 INFO:tasks.workunit.client.0.smithi136.stderr:251097: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 5 2024-09-15T21:13:12.192 INFO:tasks.workunit.client.0.smithi136.stderr:251097: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 416611827715 -lt 416611827714 2024-09-15T21:13:12.192 INFO:tasks.workunit.client.0.smithi136.stderr:251097: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T21:13:12.192 INFO:tasks.workunit.client.0.smithi136.stderr:251097: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 6-150323855415 2024-09-15T21:13:12.192 INFO:tasks.workunit.client.0.smithi136.stderr:251097: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T21:13:12.193 INFO:tasks.workunit.client.0.smithi136.stderr:251097: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=6 2024-09-15T21:13:12.193 INFO:tasks.workunit.client.0.smithi136.stderr:251097: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 6-150323855415 2024-09-15T21:13:12.193 INFO:tasks.workunit.client.0.smithi136.stderr:251097: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T21:13:12.193 INFO:tasks.workunit.client.0.smithi136.stderr:251097: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=150323855415 2024-09-15T21:13:12.193 INFO:tasks.workunit.client.0.smithi136.stderr:251097: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.6 seq 150323855415' 2024-09-15T21:13:12.193 INFO:tasks.workunit.client.0.smithi136.stderr:251097: waiting osd.6 seq 150323855415 2024-09-15T21:13:12.193 INFO:tasks.workunit.client.0.smithi136.stderr:251097: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 6 2024-09-15T21:13:12.193 INFO:tasks.workunit.client.0.smithi136.stderr:251097: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 150323855416 -lt 150323855415 2024-09-15T21:13:12.193 INFO:tasks.workunit.client.0.smithi136.stderr:251097: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T21:13:12.193 INFO:tasks.workunit.client.0.smithi136.stderr:251097: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 7-171798691887 2024-09-15T21:13:12.193 INFO:tasks.workunit.client.0.smithi136.stderr:251097: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T21:13:12.193 INFO:tasks.workunit.client.0.smithi136.stderr:251097: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=7 2024-09-15T21:13:12.193 INFO:tasks.workunit.client.0.smithi136.stderr:251097: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 7-171798691887 2024-09-15T21:13:12.193 INFO:tasks.workunit.client.0.smithi136.stderr:251097: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T21:13:12.194 INFO:tasks.workunit.client.0.smithi136.stderr:251097: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=171798691887 2024-09-15T21:13:12.194 INFO:tasks.workunit.client.0.smithi136.stderr:251097: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.7 seq 171798691887' 2024-09-15T21:13:12.194 INFO:tasks.workunit.client.0.smithi136.stderr:251097: waiting osd.7 seq 171798691887 2024-09-15T21:13:12.194 INFO:tasks.workunit.client.0.smithi136.stderr:251097: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 7 2024-09-15T21:13:12.194 INFO:tasks.workunit.client.0.smithi136.stderr:251097: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 171798691889 -lt 171798691887 2024-09-15T21:13:12.194 INFO:tasks.workunit.client.0.smithi136.stderr:251097: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T21:13:12.194 INFO:tasks.workunit.client.0.smithi136.stderr:251097: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 8-193273528361 2024-09-15T21:13:12.194 INFO:tasks.workunit.client.0.smithi136.stderr:251097: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T21:13:12.194 INFO:tasks.workunit.client.0.smithi136.stderr:251097: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=8 2024-09-15T21:13:12.194 INFO:tasks.workunit.client.0.smithi136.stderr:251097: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 8-193273528361 2024-09-15T21:13:12.209 INFO:tasks.workunit.client.0.smithi136.stderr:251097: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T21:13:12.209 INFO:tasks.workunit.client.0.smithi136.stderr:251099: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=5 2024-09-15T21:13:12.209 INFO:tasks.workunit.client.0.smithi136.stderr:251099: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 5-416611827715 2024-09-15T21:13:12.209 INFO:tasks.workunit.client.0.smithi136.stderr:251099: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T21:13:12.209 INFO:tasks.workunit.client.0.smithi136.stderr:251099: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=416611827715 2024-09-15T21:13:12.209 INFO:tasks.workunit.client.0.smithi136.stderr:251099: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.5 seq 416611827715' 2024-09-15T21:13:12.209 INFO:tasks.workunit.client.0.smithi136.stderr:251099: waiting osd.5 seq 416611827715 2024-09-15T21:13:12.209 INFO:tasks.workunit.client.0.smithi136.stderr:251099: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 5 2024-09-15T21:13:12.209 INFO:tasks.workunit.client.0.smithi136.stderr:251099: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 416611827715 -lt 416611827715 2024-09-15T21:13:12.209 INFO:tasks.workunit.client.0.smithi136.stderr:251099: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T21:13:12.209 INFO:tasks.workunit.client.0.smithi136.stderr:251099: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 6-150323855416 2024-09-15T21:13:12.209 INFO:tasks.workunit.client.0.smithi136.stderr:251099: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T21:13:12.210 INFO:tasks.workunit.client.0.smithi136.stderr:251099: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=6 2024-09-15T21:13:12.210 INFO:tasks.workunit.client.0.smithi136.stderr:251099: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 6-150323855416 2024-09-15T21:13:12.210 INFO:tasks.workunit.client.0.smithi136.stderr:251099: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T21:13:12.210 INFO:tasks.workunit.client.0.smithi136.stderr:251099: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=150323855416 2024-09-15T21:13:12.210 INFO:tasks.workunit.client.0.smithi136.stderr:251099: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.6 seq 150323855416' 2024-09-15T21:13:12.210 INFO:tasks.workunit.client.0.smithi136.stderr:251099: waiting osd.6 seq 150323855416 2024-09-15T21:13:12.210 INFO:tasks.workunit.client.0.smithi136.stderr:251099: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 6 2024-09-15T21:13:12.210 INFO:tasks.workunit.client.0.smithi136.stderr:251099: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 150323855416 -lt 150323855416 2024-09-15T21:13:12.210 INFO:tasks.workunit.client.0.smithi136.stderr:251099: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T21:13:12.210 INFO:tasks.workunit.client.0.smithi136.stderr:251099: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 7-171798691888 2024-09-15T21:13:12.210 INFO:tasks.workunit.client.0.smithi136.stderr:251099: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T21:13:12.210 INFO:tasks.workunit.client.0.smithi136.stderr:251099: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=7 2024-09-15T21:13:12.210 INFO:tasks.workunit.client.0.smithi136.stderr:251099: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 7-171798691888 2024-09-15T21:13:12.210 INFO:tasks.workunit.client.0.smithi136.stderr:251099: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T21:13:12.211 INFO:tasks.workunit.client.0.smithi136.stderr:251099: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=171798691888 2024-09-15T21:13:12.211 INFO:tasks.workunit.client.0.smithi136.stderr:251099: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.7 seq 171798691888' 2024-09-15T21:13:12.211 INFO:tasks.workunit.client.0.smithi136.stderr:251099: waiting osd.7 seq 171798691888 2024-09-15T21:13:12.211 INFO:tasks.workunit.client.0.smithi136.stderr:251099: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 7 2024-09-15T21:13:12.211 INFO:tasks.workunit.client.0.smithi136.stderr:251099: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 171798691889 -lt 171798691888 2024-09-15T21:13:12.211 INFO:tasks.workunit.client.0.smithi136.stderr:251099: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T21:13:12.211 INFO:tasks.workunit.client.0.smithi136.stderr:251099: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 8-193273528362 2024-09-15T21:13:12.211 INFO:tasks.workunit.client.0.smithi136.stderr:251099: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T21:13:12.211 INFO:tasks.workunit.client.0.smithi136.stderr:251099: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=8 2024-09-15T21:13:12.211 INFO:tasks.workunit.client.0.smithi136.stderr:251099: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 8-193273528362 2024-09-15T21:13:13.852 INFO:tasks.workunit.client.0.smithi136.stderr:251099: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -ut -d - -f 2 2024-09-15T21:13:13.852 INFO:tasks.workunit.client.0.smithi136.stderr:251097: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=193273528361 2024-09-15T21:13:13.852 INFO:tasks.workunit.client.0.smithi136.stderr:251097: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.8 seq 193273528361' 2024-09-15T21:13:13.852 INFO:tasks.workunit.client.0.smithi136.stderr:251097: waiting osd.8 seq 193273528361 2024-09-15T21:13:13.852 INFO:tasks.workunit.client.0.smithi136.stderr:251097: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 8 2024-09-15T21:13:13.852 INFO:tasks.workunit.client.0.smithi136.stderr:251097: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 193273528363 -lt 193273528361 2024-09-15T21:13:13.852 INFO:tasks.workunit.client.0.smithi136.stderr:251097: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T21:13:13.852 INFO:tasks.workunit.client.0.smithi136.stderr:251097: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 9-416611827714 2024-09-15T21:13:13.852 INFO:tasks.workunit.client.0.smithi136.stderr:251097: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T21:13:13.852 INFO:tasks.workunit.client.0.smithi136.stderr:251097: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=9 2024-09-15T21:13:13.852 INFO:tasks.workunit.client.0.smithi136.stderr:251097: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 9-416611827714 2024-09-15T21:13:13.853 INFO:tasks.workunit.client.0.smithi136.stderr:251097: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T21:13:13.853 INFO:tasks.workunit.client.0.smithi136.stderr:251097: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=416611827714 2024-09-15T21:13:13.853 INFO:tasks.workunit.client.0.smithi136.stderr:251097: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.9 seq 416611827714' 2024-09-15T21:13:13.853 INFO:tasks.workunit.client.0.smithi136.stderr:251097: waiting osd.9 seq 416611827714 2024-09-15T21:13:13.853 INFO:tasks.workunit.client.0.smithi136.stderr:251097: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 9 2024-09-15T21:13:13.853 INFO:tasks.workunit.client.0.smithi136.stderr:251097: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 416611827716 -lt 416611827714 2024-09-15T21:13:13.853 INFO:tasks.workunit.client.0.smithi136.stderr:251097: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-15T21:13:13.853 INFO:tasks.workunit.client.0.smithi136.stderr:251097: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T21:13:13.853 INFO:tasks.workunit.client.0.smithi136.stderr:251097: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T21:13:13.853 INFO:tasks.workunit.client.0.smithi136.stderr:251097: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 5 == 0 2024-09-15T21:13:13.853 INFO:tasks.workunit.client.0.smithi136.stderr:251097: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-15T21:13:13.853 INFO:tasks.workunit.client.0.smithi136.stderr:251097: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-15T21:13:13.853 INFO:tasks.workunit.client.0.smithi136.stderr:251097: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-15T21:13:13.854 INFO:tasks.workunit.client.0.smithi136.stderr:251097: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T21:13:13.854 INFO:tasks.workunit.client.0.smithi136.stderr:251097: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T21:13:13.854 INFO:tasks.workunit.client.0.smithi136.stderr:251097: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T21:13:13.854 INFO:tasks.workunit.client.0.smithi136.stderr:251097: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T21:13:13.854 INFO:tasks.workunit.client.0.smithi136.stderr:251097: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=5 2024-09-15T21:13:13.854 INFO:tasks.workunit.client.0.smithi136.stderr:251097: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-15T21:13:13.854 INFO:tasks.workunit.client.0.smithi136.stderr:251097: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T21:13:13.854 INFO:tasks.workunit.client.0.smithi136.stderr:251097: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T21:13:13.854 INFO:tasks.workunit.client.0.smithi136.stderr:251097: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 = 5 2024-09-15T21:13:13.854 INFO:tasks.workunit.client.0.smithi136.stderr:251097: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-15T21:13:13.854 INFO:tasks.workunit.client.0.smithi136.stderr:251097: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-15T21:13:13.854 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: run_in_background: return 0 2024-09-15T21:13:13.854 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: wait_background: for pid in $pids 2024-09-15T21:13:13.854 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2175: wait_background: wait 251095 2024-09-15T21:13:13.931 INFO:tasks.workunit.client.0.smithi136.stderr:f 2 2024-09-15T21:13:13.931 INFO:tasks.workunit.client.0.smithi136.stderr:251099: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=193273528362 2024-09-15T21:13:13.931 INFO:tasks.workunit.client.0.smithi136.stderr:251099: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.8 seq 193273528362' 2024-09-15T21:13:13.931 INFO:tasks.workunit.client.0.smithi136.stderr:251099: waiting osd.8 seq 193273528362 2024-09-15T21:13:13.931 INFO:tasks.workunit.client.0.smithi136.stderr:251099: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 8 2024-09-15T21:13:13.931 INFO:tasks.workunit.client.0.smithi136.stderr:251099: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 193273528363 -lt 193273528362 2024-09-15T21:13:13.931 INFO:tasks.workunit.client.0.smithi136.stderr:251099: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T21:13:13.931 INFO:tasks.workunit.client.0.smithi136.stderr:251099: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 9-416611827715 2024-09-15T21:13:13.931 INFO:tasks.workunit.client.0.smithi136.stderr:251099: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T21:13:13.931 INFO:tasks.workunit.client.0.smithi136.stderr:251099: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=9 2024-09-15T21:13:13.931 INFO:tasks.workunit.client.0.smithi136.stderr:251099: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 9-416611827715 2024-09-15T21:13:13.932 INFO:tasks.workunit.client.0.smithi136.stderr:251099: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T21:13:13.932 INFO:tasks.workunit.client.0.smithi136.stderr:251099: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=416611827715 2024-09-15T21:13:13.932 INFO:tasks.workunit.client.0.smithi136.stderr:251099: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.9 seq 416611827715' 2024-09-15T21:13:13.932 INFO:tasks.workunit.client.0.smithi136.stderr:251099: waiting osd.9 seq 416611827715 2024-09-15T21:13:13.932 INFO:tasks.workunit.client.0.smithi136.stderr:251099: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 9 2024-09-15T21:13:13.932 INFO:tasks.workunit.client.0.smithi136.stderr:251099: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 416611827716 -lt 416611827715 2024-09-15T21:13:13.932 INFO:tasks.workunit.client.0.smithi136.stderr:251099: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-15T21:13:13.932 INFO:tasks.workunit.client.0.smithi136.stderr:251099: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T21:13:13.932 INFO:tasks.workunit.client.0.smithi136.stderr:251099: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T21:13:13.932 INFO:tasks.workunit.client.0.smithi136.stderr:251099: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 5 == 0 2024-09-15T21:13:13.932 INFO:tasks.workunit.client.0.smithi136.stderr:251099: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-15T21:13:13.932 INFO:tasks.workunit.client.0.smithi136.stderr:251099: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-15T21:13:13.932 INFO:tasks.workunit.client.0.smithi136.stderr:251099: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-15T21:13:13.933 INFO:tasks.workunit.client.0.smithi136.stderr:251099: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T21:13:13.933 INFO:tasks.workunit.client.0.smithi136.stderr:251099: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T21:13:13.933 INFO:tasks.workunit.client.0.smithi136.stderr:251099: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T21:13:13.933 INFO:tasks.workunit.client.0.smithi136.stderr:251099: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T21:13:13.933 INFO:tasks.workunit.client.0.smithi136.stderr:251099: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=5 2024-09-15T21:13:13.933 INFO:tasks.workunit.client.0.smithi136.stderr:251099: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-15T21:13:13.933 INFO:tasks.workunit.client.0.smithi136.stderr:251099: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T21:13:13.933 INFO:tasks.workunit.client.0.smithi136.stderr:251099: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T21:13:13.933 INFO:tasks.workunit.client.0.smithi136.stderr:251099: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 = 5 2024-09-15T21:13:13.933 INFO:tasks.workunit.client.0.smithi136.stderr:251099: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-15T21:13:13.933 INFO:tasks.workunit.client.0.smithi136.stderr:251099: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-15T21:13:13.933 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: run_in_background: return 0 2024-09-15T21:13:13.933 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: wait_background: eval 'pids='\'''\''' 2024-09-15T21:13:13.934 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: wait_background: pids= 2024-09-15T21:13:13.934 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: wait_background: return 0 2024-09-15T21:13:13.934 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:197: corrupt_and_repair_two: return_code=0 2024-09-15T21:13:13.934 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:198: corrupt_and_repair_two: '[' 0 -ne 0 ']' 2024-09-15T21:13:13.934 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:203: corrupt_and_repair_two: get_pg ecpool SOMETHING 2024-09-15T21:13:13.934 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1084: get_pg: local poolname=ecpool 2024-09-15T21:13:13.934 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1085: get_pg: local objectname=SOMETHING 2024-09-15T21:13:13.934 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1087: get_pg: ceph --format json osd map ecpool SOMETHING 2024-09-15T21:13:13.934 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1087: get_pg: jq -r .pgid 2024-09-15T21:13:14.212 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:203: corrupt_and_repair_two: local pg=2.0 2024-09-15T21:13:14.213 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:204: corrupt_and_repair_two: repair 2.0 2024-09-15T21:13:14.213 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1846: repair: local pgid=2.0 2024-09-15T21:13:14.213 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1847: repair: get_last_scrub_stamp 2.0 2024-09-15T21:13:14.213 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2024-09-15T21:13:14.213 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-15T21:13:14.213 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-15T21:13:14.213 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-09-15T21:13:14.486 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1847: repair: local last_scrub=2024-09-15T21:12:38.602968+0000 2024-09-15T21:13:14.486 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1848: repair: ceph pg repair 2.0 2024-09-15T21:13:14.747 INFO:tasks.workunit.client.0.smithi136.stderr:instructing pg 2.0s0 on osd.3 to repair 2024-09-15T21:13:14.759 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1849: repair: wait_for_scrub 2.0 2024-09-15T21:12:38.602968+0000 2024-09-15T21:13:14.759 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: local pgid=2.0 2024-09-15T21:13:14.759 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2018: wait_for_scrub: local last_scrub=2024-09-15T21:12:38.602968+0000 2024-09-15T21:13:14.759 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2019: wait_for_scrub: local sname=last_scrub_stamp 2024-09-15T21:13:14.759 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i=0 )) 2024-09-15T21:13:14.759 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-15T21:13:14.759 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-09-15T21:13:14.759 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2024-09-15T21:13:14.759 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-15T21:13:14.759 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-15T21:13:14.760 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-09-15T21:13:15.036 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-15T21:12:38.602968+0000 '>' 2024-09-15T21:12:38.602968+0000 2024-09-15T21:13:15.036 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-15T21:13:16.037 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-15T21:13:16.037 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-15T21:13:16.038 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-09-15T21:13:16.038 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2024-09-15T21:13:16.038 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-15T21:13:16.038 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-15T21:13:16.038 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-09-15T21:13:16.322 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-15T21:12:38.602968+0000 '>' 2024-09-15T21:12:38.602968+0000 2024-09-15T21:13:16.322 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-15T21:13:17.323 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-15T21:13:17.324 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-15T21:13:17.324 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-09-15T21:13:17.324 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2024-09-15T21:13:17.325 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-15T21:13:17.325 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-15T21:13:17.325 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-09-15T21:13:17.607 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-15T21:12:38.602968+0000 '>' 2024-09-15T21:12:38.602968+0000 2024-09-15T21:13:17.607 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-15T21:13:18.608 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-15T21:13:18.608 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-15T21:13:18.609 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-09-15T21:13:18.609 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2024-09-15T21:13:18.609 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-15T21:13:18.609 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-15T21:13:18.609 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-09-15T21:13:18.897 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-15T21:12:38.602968+0000 '>' 2024-09-15T21:12:38.602968+0000 2024-09-15T21:13:18.897 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-15T21:13:19.898 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-15T21:13:19.898 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-15T21:13:19.899 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-09-15T21:13:19.899 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2024-09-15T21:13:19.899 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-15T21:13:19.899 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-15T21:13:19.899 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-09-15T21:13:20.234 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-15T21:12:38.602968+0000 '>' 2024-09-15T21:12:38.602968+0000 2024-09-15T21:13:20.234 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-15T21:13:21.235 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-15T21:13:21.235 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-15T21:13:21.236 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-09-15T21:13:21.236 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2024-09-15T21:13:21.236 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-15T21:13:21.236 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-15T21:13:21.236 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-09-15T21:13:21.514 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-15T21:12:38.602968+0000 '>' 2024-09-15T21:12:38.602968+0000 2024-09-15T21:13:21.515 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-15T21:13:22.516 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-15T21:13:22.516 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-15T21:13:22.516 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-09-15T21:13:22.517 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2024-09-15T21:13:22.517 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-15T21:13:22.517 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-15T21:13:22.517 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-09-15T21:13:22.801 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-15T21:13:15.713253+0000 '>' 2024-09-15T21:12:38.602968+0000 2024-09-15T21:13:22.801 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2023: wait_for_scrub: return 0 2024-09-15T21:13:22.801 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1850: repair: sleep 2 2024-09-15T21:13:24.807 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:208: corrupt_and_repair_two: pids= 2024-09-15T21:13:24.807 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:209: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 5 SOMETHING list-attrs 2024-09-15T21:13:24.807 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2135: run_in_background: local pid_variable=pids 2024-09-15T21:13:24.807 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2136: run_in_background: shift 2024-09-15T21:13:24.807 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2140: run_in_background: eval 'pids+=" 254619"' 2024-09-15T21:13:24.807 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2140: run_in_background: pids+=' 254619' 2024-09-15T21:13:24.807 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:210: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 9 SOMETHING list-attrs 2024-09-15T21:13:24.807 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2135: run_in_background: local pid_variable=pids 2024-09-15T21:13:24.807 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2136: run_in_background: shift 2024-09-15T21:13:24.807 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2140: run_in_background: eval 'pids+=" 254620"' 2024-09-15T21:13:24.808 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2140: run_in_background: pids+=' 254620' 2024-09-15T21:13:24.808 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:211: corrupt_and_repair_two: wait_background pids 2024-09-15T21:13:24.808 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: wait_background: pids=' 254619 254620' 2024-09-15T21:13:24.808 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: wait_background: return_code=0 2024-09-15T21:13:24.808 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: wait_background: for pid in $pids 2024-09-15T21:13:24.808 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2175: wait_background: wait 254619 2024-09-15T21:13:24.808 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: run_in_background: objectstore_tool td/osd-scrub-repair 5 SOMETHING list-attrs 2024-09-15T21:13:24.808 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: run_in_background: sed 's/^/254622: /' 2024-09-15T21:13:24.808 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: run_in_background: objectstore_tool td/osd-scrub-repair 9 SOMETHING list-attrs 2024-09-15T21:13:24.808 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: run_in_background: sed 's/^/254624: /' 2024-09-15T21:13:25.902 INFO:tasks.workunit.client.0.smithi136.stderr:254624: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2024-09-15T21:13:25.902 INFO:tasks.workunit.client.0.smithi136.stderr:254624: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2024-09-15T21:13:25.902 INFO:tasks.workunit.client.0.smithi136.stderr:254624: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=9 2024-09-15T21:13:25.902 INFO:tasks.workunit.client.0.smithi136.stderr:254624: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2024-09-15T21:13:25.902 INFO:tasks.workunit.client.0.smithi136.stderr:254624: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 9 SOMETHING list-attrs 2024-09-15T21:13:25.902 INFO:tasks.workunit.client.0.smithi136.stderr:254624: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-09-15T21:13:25.902 INFO:tasks.workunit.client.0.smithi136.stderr:254624: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2024-09-15T21:13:25.902 INFO:tasks.workunit.client.0.smithi136.stderr:254624: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=9 2024-09-15T21:13:25.902 INFO:tasks.workunit.client.0.smithi136.stderr:254624: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2024-09-15T21:13:25.902 INFO:tasks.workunit.client.0.smithi136.stderr:254624: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.9 2024-09-15T21:13:25.903 INFO:tasks.workunit.client.0.smithi136.stderr:254624: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-15T21:13:25.903 INFO:tasks.workunit.client.0.smithi136.stderr:254624: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-15T21:13:25.903 INFO:tasks.workunit.client.0.smithi136.stderr:254624: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-15T21:13:25.903 INFO:tasks.workunit.client.0.smithi136.stderr:254624: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-15T21:13:25.903 INFO:tasks.workunit.client.0.smithi136.stderr:254624: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-15T21:13:25.903 INFO:tasks.workunit.client.0.smithi136.stderr:254624: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-15T21:13:25.903 INFO:tasks.workunit.client.0.smithi136.stderr:254624: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 9 SOMETHING list-attrs 2024-09-15T21:13:25.903 INFO:tasks.workunit.client.0.smithi136.stderr:254624: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-15T21:13:25.903 INFO:tasks.workunit.client.0.smithi136.stderr:254624: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-15T21:13:25.903 INFO:tasks.workunit.client.0.smithi136.stderr:254624: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=9 2024-09-15T21:13:25.903 INFO:tasks.workunit.client.0.smithi136.stderr:254624: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-15T21:13:25.903 INFO:tasks.workunit.client.0.smithi136.stderr:254624: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/9 2024-09-15T21:13:25.903 INFO:tasks.workunit.client.0.smithi136.stderr:254624: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/9 SOMETHING list-attrs 2024-09-15T21:13:25.903 INFO:tasks.workunit.client.0.smithi136.stderr:254624: _ 2024-09-15T21:13:25.904 INFO:tasks.workunit.client.0.smithi136.stderr:254624: hinfo_key 2024-09-15T21:13:25.904 INFO:tasks.workunit.client.0.smithi136.stderr:254624: snapset 2024-09-15T21:13:25.904 INFO:tasks.workunit.client.0.smithi136.stderr:254624: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 9 2024-09-15T21:13:25.904 INFO:tasks.workunit.client.0.smithi136.stderr:254624: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2024-09-15T21:13:25.904 INFO:tasks.workunit.client.0.smithi136.stderr:254624: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-15T21:13:25.904 INFO:tasks.workunit.client.0.smithi136.stderr:254624: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=9 2024-09-15T21:13:25.904 INFO:tasks.workunit.client.0.smithi136.stderr:254624: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-15T21:13:25.904 INFO:tasks.workunit.client.0.smithi136.stderr:254624: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/9 2024-09-15T21:13:25.904 INFO:tasks.workunit.client.0.smithi136.stderr:254624: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=ce4892c7-6c6d-4b4f-8e6f-07a3e11133e5 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-15T21:13:25.904 INFO:tasks.workunit.client.0.smithi136.stderr:254624: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T21:13:25.905 INFO:tasks.workunit.client.0.smithi136.stderr:254624: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T21:13:25.905 INFO:tasks.workunit.client.0.smithi136.stderr:254624: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T21:13:25.905 INFO:tasks.workunit.client.0.smithi136.stderr:254624: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/9' 2024-09-15T21:13:25.905 INFO:tasks.workunit.client.0.smithi136.stderr:254624: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/9/journal' 2024-09-15T21:13:25.905 INFO:tasks.workunit.client.0.smithi136.stderr:254624: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-15T21:13:25.905 INFO:tasks.workunit.client.0.smithi136.stderr:254624: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-15T21:13:25.905 INFO:tasks.workunit.client.0.smithi136.stderr:254624: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-15T21:13:25.905 INFO:tasks.workunit.client.0.smithi136.stderr:254624: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-15T21:13:25.905 INFO:tasks.workunit.client.0.smithi136.stderr:254624: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-15T21:13:25.906 INFO:tasks.workunit.client.0.smithi136.stderr:254624: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-15T21:13:25.906 INFO:tasks.workunit.client.0.smithi136.stderr:254624: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-15T21:13:25.906 INFO:tasks.workunit.client.0.smithi136.stderr:254624: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T21:13:25.906 INFO:tasks.workunit.client.0.smithi136.stderr:254624: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52602 2024-09-15T21:13:25.906 INFO:tasks.workunit.client.0.smithi136.stderr:254624: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.52602/$cluster-$name.asok' 2024-09-15T21:13:25.906 INFO:tasks.workunit.client.0.smithi136.stderr:254624: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.52602/$cluster-$name.asok' 2024-09-15T21:13:25.906 INFO:tasks.workunit.client.0.smithi136.stderr:254624: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-15T21:13:25.906 INFO:tasks.workunit.client.0.smithi136.stderr:254624: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-15T21:13:25.906 INFO:tasks.workunit.client.0.smithi136.stderr:254624: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-15T21:13:25.906 INFO:tasks.workunit.client.0.smithi136.stderr:254624: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T21:13:25.906 INFO:tasks.workunit.client.0.smithi136.stderr:254624: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T21:13:25.906 INFO:tasks.workunit.client.0.smithi136.stderr:254624: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T21:13:25.906 INFO:tasks.workunit.client.0.smithi136.stderr:254624: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T21:13:25.906 INFO:tasks.workunit.client.0.smithi136.stderr:254624: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-15T21:13:25.907 INFO:tasks.workunit.client.0.smithi136.stderr:254624: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-15T21:13:25.907 INFO:tasks.workunit.client.0.smithi136.stderr:254624: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/9 2024-09-15T21:13:25.907 INFO:tasks.workunit.client.0.smithi136.stderr:254624: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.9 2024-09-15T21:13:25.907 INFO:tasks.workunit.client.0.smithi136.stderr:254624: start osd.9 2024-09-15T21:13:25.951 INFO:tasks.workunit.client.0.smithi136.stderr:254624: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 9 --fsid=ce4892c7-6c6d-4b4f-8e6f-07a3e11133e5 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/9 --osd-journal=td/osd-scrub-repair/9/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.52602/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_reco254622: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2024-09-15T21:13:25.951 INFO:tasks.workunit.client.0.smithi136.stderr:254622: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2024-09-15T21:13:25.951 INFO:tasks.workunit.client.0.smithi136.stderr:254622: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=5 2024-09-15T21:13:25.951 INFO:tasks.workunit.client.0.smithi136.stderr:254622: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2024-09-15T21:13:25.951 INFO:tasks.workunit.client.0.smithi136.stderr:254622: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 5 SOMETHING list-attrs 2024-09-15T21:13:25.952 INFO:tasks.workunit.client.0.smithi136.stderr:254622: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-09-15T21:13:25.952 INFO:tasks.workunit.client.0.smithi136.stderr:254622: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2024-09-15T21:13:25.952 INFO:tasks.workunit.client.0.smithi136.stderr:254622: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=5 2024-09-15T21:13:25.952 INFO:tasks.workunit.client.0.smithi136.stderr:254622: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2024-09-15T21:13:25.952 INFO:tasks.workunit.client.0.smithi136.stderr:254622: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.5 2024-09-15T21:13:25.952 INFO:tasks.workunit.client.0.smithi136.stderr:254622: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-15T21:13:25.952 INFO:tasks.workunit.client.0.smithi136.stderr:254622: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-15T21:13:25.952 INFO:tasks.workunit.client.0.smithi136.stderr:254622: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-15T21:13:25.952 INFO:tasks.workunit.client.0.smithi136.stderr:254622: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-15T21:13:25.952 INFO:tasks.workunit.client.0.smithi136.stderr:254622: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-15T21:13:25.952 INFO:tasks.workunit.client.0.smithi136.stderr:254622: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-15T21:13:25.952 INFO:tasks.workunit.client.0.smithi136.stderr:254622: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 5 SOMETHING list-attrs 2024-09-15T21:13:25.953 INFO:tasks.workunit.client.0.smithi136.stderr:254622: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-15T21:13:25.953 INFO:tasks.workunit.client.0.smithi136.stderr:254622: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-15T21:13:25.953 INFO:tasks.workunit.client.0.smithi136.stderr:254622: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=5 2024-09-15T21:13:25.953 INFO:tasks.workunit.client.0.smithi136.stderr:254622: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-15T21:13:25.953 INFO:tasks.workunit.client.0.smithi136.stderr:254622: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/5 2024-09-15T21:13:25.953 INFO:tasks.workunit.client.0.smithi136.stderr:254622: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/5 SOMETHING list-attrs 2024-09-15T21:13:25.953 INFO:tasks.workunit.client.0.smithi136.stderr:254622: _ 2024-09-15T21:13:25.953 INFO:tasks.workunit.client.0.smithi136.stderr:254622: hinfo_key 2024-09-15T21:13:25.953 INFO:tasks.workunit.client.0.smithi136.stderr:254622: snapset 2024-09-15T21:13:25.953 INFO:tasks.workunit.client.0.smithi136.stderr:254622: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 5 2024-09-15T21:13:25.953 INFO:tasks.workunit.client.0.smithi136.stderr:254622: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2024-09-15T21:13:25.953 INFO:tasks.workunit.client.0.smithi136.stderr:254622: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-15T21:13:25.953 INFO:tasks.workunit.client.0.smithi136.stderr:254622: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=5 2024-09-15T21:13:25.954 INFO:tasks.workunit.client.0.smithi136.stderr:254622: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-15T21:13:25.954 INFO:tasks.workunit.client.0.smithi136.stderr:254622: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/5 2024-09-15T21:13:25.954 INFO:tasks.workunit.client.0.smithi136.stderr:254622: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=ce4892c7-6c6d-4b4f-8e6f-07a3e11133e5 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-15T21:13:25.954 INFO:tasks.workunit.client.0.smithi136.stderr:254622: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T21:13:25.954 INFO:tasks.workunit.client.0.smithi136.stderr:254622: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T21:13:25.954 INFO:tasks.workunit.client.0.smithi136.stderr:254622: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T21:13:25.954 INFO:tasks.workunit.client.0.smithi136.stderr:254622: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/5' 2024-09-15T21:13:25.954 INFO:tasks.workunit.client.0.smithi136.stderr:254622: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/5/journal' 2024-09-15T21:13:25.955 INFO:tasks.workunit.client.0.smithi136.stderr:254622: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-15T21:13:25.955 INFO:tasks.workunit.client.0.smithi136.stderr:254622: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-15T21:13:25.955 INFO:tasks.workunit.client.0.smithi136.stderr:254622: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-15T21:13:25.955 INFO:tasks.workunit.client.0.smithi136.stderr:254622: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-15T21:13:25.955 INFO:tasks.workunit.client.0.smithi136.stderr:254622: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-15T21:13:25.955 INFO:tasks.workunit.client.0.smithi136.stderr:254622: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-15T21:13:25.955 INFO:tasks.workunit.client.0.smithi136.stderr:254622: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-15T21:13:25.955 INFO:tasks.workunit.client.0.smithi136.stderr:254622: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T21:13:25.955 INFO:tasks.workunit.client.0.smithi136.stderr:254622: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52602 2024-09-15T21:13:25.955 INFO:tasks.workunit.client.0.smithi136.stderr:254622: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.52602/$cluster-$name.asok' 2024-09-15T21:13:25.955 INFO:tasks.workunit.client.0.smithi136.stderr:254622: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.52602/$cluster-$name.asok' 2024-09-15T21:13:25.956 INFO:tasks.workunit.client.0.smithi136.stderr:254622: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-15T21:13:25.956 INFO:tasks.workunit.client.0.smithi136.stderr:254622: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-15T21:13:25.956 INFO:tasks.workunit.client.0.smithi136.stderr:254622: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-15T21:13:25.956 INFO:tasks.workunit.client.0.smithi136.stderr:254622: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T21:13:25.956 INFO:tasks.workunit.client.0.smithi136.stderr:254622: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T21:13:25.956 INFO:tasks.workunit.client.0.smithi136.stderr:254622: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T21:13:25.956 INFO:tasks.workunit.client.0.smithi136.stderr:254622: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T21:13:25.956 INFO:tasks.workunit.client.0.smithi136.stderr:254622: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-15T21:13:25.956 INFO:tasks.workunit.client.0.smithi136.stderr:254622: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-15T21:13:25.956 INFO:tasks.workunit.client.0.smithi136.stderr:254622: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/5 2024-09-15T21:13:25.957 INFO:tasks.workunit.client.0.smithi136.stderr:254622: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.5 2024-09-15T21:13:25.957 INFO:tasks.workunit.client.0.smithi136.stderr:254622: start osd.5 2024-09-15T21:13:31.010 INFO:tasks.workunit.client.0.smithi136.stderr:254622: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 5 --fsid=ce4892c7-6c6d-4b4f-8e6f-07a3e11133e5 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/5 --osd-journal=td/osd-scrub-repair/5/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.52602/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T21:13:31.010 INFO:tasks.workunit.client.0.smithi136.stderr:254624: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/9/whoami 2024-09-15T21:13:31.010 INFO:tasks.workunit.client.0.smithi136.stderr:254624: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 9 = 9 ']' 2024-09-15T21:13:31.010 INFO:tasks.workunit.client.0.smithi136.stderr:254624: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-15T21:13:31.010 INFO:tasks.workunit.client.0.smithi136.stderr:254624: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-15T21:13:31.011 INFO:tasks.workunit.client.0.smithi136.stderr:254624: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-15T21:13:31.011 INFO:tasks.workunit.client.0.smithi136.stderr:254624: 2024-09-15T21:13:25.932+0000 7f5b9393e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:13:31.011 INFO:tasks.workunit.client.0.smithi136.stderr:254624: 2024-09-15T21:13:25.935+0000 7f5b9393e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:13:31.011 INFO:tasks.workunit.client.0.smithi136.stderr:254624: 2024-09-15T21:13:25.936+0000 7f5b9393e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:13:31.011 INFO:tasks.workunit.client.0.smithi136.stderr:254624: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 9 2024-09-15T21:13:31.011 INFO:tasks.workunit.client.0.smithi136.stderr:254624: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-15T21:13:31.011 INFO:tasks.workunit.client.0.smithi136.stderr:254624: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=9 2024-09-15T21:13:31.011 INFO:tasks.workunit.client.0.smithi136.stderr:254624: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-15T21:13:31.011 INFO:tasks.workunit.client.0.smithi136.stderr:254624: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-15T21:13:31.011 INFO:tasks.workunit.client.0.smithi136.stderr:254624: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:13:31.011 INFO:tasks.workunit.client.0.smithi136.stderr:254624: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-15T21:13:31.011 INFO:tasks.workunit.client.0.smithi136.stderr:254624: 0 2024-09-15T21:13:31.011 INFO:tasks.workunit.client.0.smithi136.stderr:254624: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:13:31.011 INFO:tasks.workunit.client.0.smithi136.stderr:254624: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.9 up' 2024-09-15T21:13:31.011 INFO:tasks.workunit.client.0.smithi136.stderr:254624: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:13:31.012 INFO:tasks.workunit.client.0.smithi136.stderr:254624: 2024-09-15T21:13:27.545+0000 7f5b9393e640 -1 Falling back to public interface 2024-09-15T21:13:31.012 INFO:tasks.workunit.client.0.smithi136.stderr:254624: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:13:31.012 INFO:tasks.workunit.client.0.smithi136.stderr:254624: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:13:31.012 INFO:tasks.workunit.client.0.smithi136.stderr:254624: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-15T21:13:31.012 INFO:tasks.workunit.client.0.smithi136.stderr:254624: 1 2024-09-15T21:13:31.012 INFO:tasks.workunit.client.0.smithi136.stderr:254624: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:13:31.012 INFO:tasks.workunit.client.0.smithi136.stderr:254624: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.9 up' 2024-09-15T21:13:31.013 INFO:tasks.workunit.client.0.smithi136.stderr:254624: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:13:31.013 INFO:tasks.workunit.client.0.smithi136.stderr:254624: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:13:31.013 INFO:tasks.workunit.client.0.smithi136.stderr:254624: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:13:31.013 INFO:tasks.workunit.client.0.smithi136.stderr:254624: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-15T21:13:31.013 INFO:tasks.workunit.client.0.smithi136.stderr:254624: 2 2024-09-15T21:13:31.013 INFO:tasks.workunit.client.0.smithi136.stderr:254624: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:13:31.013 INFO:tasks.workunit.client.0.smithi136.stderr:254624: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.9 up' 2024-09-15T21:13:31.013 INFO:tasks.workunit.client.0.smithi136.stderr:254624: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:13:31.013 INFO:tasks.workunit.client.0.smithi136.stderr:254624: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:13:31.013 INFO:tasks.workunit.client.0.smithi136.stderr:254624: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:13:31.013 INFO:tasks.workunit.client.0.smithi136.stderr:254624: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-15T21:13:31.013 INFO:tasks.workunit.client.0.smithi136.stderr:254624: 3 2024-09-15T21:13:31.014 INFO:tasks.workunit.client.0.smithi136.stderr:254624: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:13:31.014 INFO:tasks.workunit.client.0.smithi136.stderr:254624: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.9 up' 2024-09-15T21:13:31.014 INFO:tasks.workunit.client.0.smithi136.stderr:254624: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:13:31.586 INFO:tasks.workunit.client.0.smithi136.stderr:254624: 2024-09-15T21:13:31.007+0000 7f5bvery_ops 2024-09-15T21:13:31.586 INFO:tasks.workunit.client.0.smithi136.stderr:254622: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/5/whoami 2024-09-15T21:13:31.586 INFO:tasks.workunit.client.0.smithi136.stderr:254622: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 5 = 5 ']' 2024-09-15T21:13:31.587 INFO:tasks.workunit.client.0.smithi136.stderr:254622: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-15T21:13:31.587 INFO:tasks.workunit.client.0.smithi136.stderr:254622: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-15T21:13:31.587 INFO:tasks.workunit.client.0.smithi136.stderr:254622: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-15T21:13:31.587 INFO:tasks.workunit.client.0.smithi136.stderr:254622: 2024-09-15T21:13:25.979+0000 7f086022c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:13:31.587 INFO:tasks.workunit.client.0.smithi136.stderr:254622: 2024-09-15T21:13:25.982+0000 7f086022c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:13:31.587 INFO:tasks.workunit.client.0.smithi136.stderr:254622: 2024-09-15T21:13:25.983+0000 7f086022c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:13:31.588 INFO:tasks.workunit.client.0.smithi136.stderr:254622: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 5 2024-09-15T21:13:31.588 INFO:tasks.workunit.client.0.smithi136.stderr:254622: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-15T21:13:31.588 INFO:tasks.workunit.client.0.smithi136.stderr:254622: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=5 2024-09-15T21:13:31.588 INFO:tasks.workunit.client.0.smithi136.stderr:254622: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-15T21:13:31.588 INFO:tasks.workunit.client.0.smithi136.stderr:254622: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-15T21:13:31.589 INFO:tasks.workunit.client.0.smithi136.stderr:254622: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:13:31.589 INFO:tasks.workunit.client.0.smithi136.stderr:254622: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-15T21:13:31.589 INFO:tasks.workunit.client.0.smithi136.stderr:254622: 0 2024-09-15T21:13:31.589 INFO:tasks.workunit.client.0.smithi136.stderr:254622: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:13:31.589 INFO:tasks.workunit.client.0.smithi136.stderr:254622: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-15T21:13:31.589 INFO:tasks.workunit.client.0.smithi136.stderr:254622: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:13:31.589 INFO:tasks.workunit.client.0.smithi136.stderr:254622: 2024-09-15T21:13:27.604+0000 7f086022c640 -1 Falling back to public interface 2024-09-15T21:13:31.589 INFO:tasks.workunit.client.0.smithi136.stderr:254622: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:13:31.589 INFO:tasks.workunit.client.0.smithi136.stderr:254622: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:13:31.589 INFO:tasks.workunit.client.0.smithi136.stderr:254622: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-15T21:13:31.589 INFO:tasks.workunit.client.0.smithi136.stderr:254622: 1 2024-09-15T21:13:31.589 INFO:tasks.workunit.client.0.smithi136.stderr:254622: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:13:31.590 INFO:tasks.workunit.client.0.smithi136.stderr:254622: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-15T21:13:31.590 INFO:tasks.workunit.client.0.smithi136.stderr:254622: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:13:31.590 INFO:tasks.workunit.client.0.smithi136.stderr:254622: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:13:31.590 INFO:tasks.workunit.client.0.smithi136.stderr:254622: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:13:31.590 INFO:tasks.workunit.client.0.smithi136.stderr:254622: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-15T21:13:31.590 INFO:tasks.workunit.client.0.smithi136.stderr:254622: 2 2024-09-15T21:13:31.591 INFO:tasks.workunit.client.0.smithi136.stderr:254622: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:13:31.591 INFO:tasks.workunit.client.0.smithi136.stderr:254622: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-15T21:13:31.591 INFO:tasks.workunit.client.0.smithi136.stderr:254622: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:13:31.591 INFO:tasks.workunit.client.0.smithi136.stderr:254622: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:13:31.591 INFO:tasks.workunit.client.0.smithi136.stderr:254622: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:13:31.591 INFO:tasks.workunit.client.0.smithi136.stderr:254622: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-15T21:13:31.591 INFO:tasks.workunit.client.0.smithi136.stderr:254622: 3 2024-09-15T21:13:31.591 INFO:tasks.workunit.client.0.smithi136.stderr:254622: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:13:31.591 INFO:tasks.workunit.client.0.smithi136.stderr:254622: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-15T21:13:31.591 INFO:tasks.workunit.client.0.smithi136.stderr:254622: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:13:32.653 INFO:tasks.workunit.client.0.smithi136.stderr:254622: 2024-09-15T21:13:31.583+0000 7f089393e640 -1 osd.9 99 log_to_monitors true 2024-09-15T21:13:32.653 INFO:tasks.workunit.client.0.smithi136.stderr:254624: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:13:32.654 INFO:tasks.workunit.client.0.smithi136.stderr:254624: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:13:32.654 INFO:tasks.workunit.client.0.smithi136.stderr:254624: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-15T21:13:32.654 INFO:tasks.workunit.client.0.smithi136.stderr:254624: 4 2024-09-15T21:13:32.654 INFO:tasks.workunit.client.0.smithi136.stderr:254624: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:13:32.654 INFO:tasks.workunit.client.0.smithi136.stderr:254624: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.9 up' 2024-09-15T21:13:32.655 INFO:tasks.workunit.client.0.smithi136.stderr:254624: osd.9 up in weight 1 up_from 104 up_thru 0 down_at 100 last_clean_interval [97,99) [v2:127.0.0.1:6842/1835174157,v1:127.0.0.1:6843/1835174157] [v2:127.0.0.1:6844/1835174157,v1:127.0.0.1:6845/1835174157] exists,up 78ecdb67-c9df-44e1-b77a-4e850e67a453 2024-09-15T21:13:32.655 INFO:tasks.workunit.client.0.smithi136.stderr:254624: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-15T21:13:32.655 INFO:tasks.workunit.client.0.smithi136.stderr:254624: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-15T21:13:32.655 INFO:tasks.workunit.client.0.smithi136.stderr:254624: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-15T21:13:32.655 INFO:tasks.workunit.client.0.smithi136.stderr:254624: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2024-09-15T21:13:32.655 INFO:tasks.workunit.client.0.smithi136.stderr:254624: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-15T21:13:32.655 INFO:tasks.workunit.client.0.smithi136.stderr:254624: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-15T21:13:32.655 INFO:tasks.workunit.client.0.smithi136.stderr:254624: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-15T21:13:32.656 INFO:tasks.workunit.client.0.smithi136.stderr:254624: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-15T21:13:32.656 INFO:tasks.workunit.client.0.smithi136.stderr:254624: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-15T21:13:32.656 INFO:tasks.workunit.client.0.smithi136.stderr:254624: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-15T21:13:32.656 INFO:tasks.workunit.client.0.smithi136.stderr:254624: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-15T21:13:32.656 INFO:tasks.workunit.client.0.smithi136.stderr:254624: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-15T21:13:32.656 INFO:tasks.workunit.client.0.smithi136.stderr:254624: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-15T21:13:32.657 INFO:tasks.workunit.client.0.smithi136.stderr:254624: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-15T21:13:32.657 INFO:tasks.workunit.client.0.smithi136.stderr:254624: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-15T21:13:32.657 INFO:tasks.workunit.client.0.smithi136.stderr:254624: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-15T21:13:32.657 INFO:tasks.workunit.client.0.smithi136.stderr:254624: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-15T21:13:32.657 INFO:tasks.workunit.client.0.smithi136.stderr:254624: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-15T21:13:32.657 INFO:tasks.workunit.client.0.smithi136.stderr:254624: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-15T21:13:32.658 INFO:tasks.workunit.client.0.smithi136.stderr:254624: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-15T21:13:32.658 INFO:tasks.workunit.client.0.smithi136.stderr:254624: 1 2024-09-15T21:13:32.658 INFO:tasks.workunit.client.0.smithi136.stderr:254624: 2 2024-09-15T21:13:32.658 INFO:tasks.workunit.client.0.smithi136.stderr:254624: 3 2024-09-15T21:13:32.658 INFO:tasks.workunit.client.0.smithi136.stderr:254624: 4 2024-09-15T21:13:32.658 INFO:tasks.workunit.client.0.smithi136.stderr:254624: 5 2024-09-15T21:13:32.659 INFO:tasks.workunit.client.0.smithi136.stderr:254624: 6 2024-09-15T21:13:32.659 INFO:tasks.workunit.client.0.smithi136.stderr:254624: 7 2024-09-15T21:13:32.659 INFO:tasks.workunit.client.0.smithi136.stderr:254624: 8 2024-09-15T21:13:32.659 INFO:tasks.workunit.client.0.smithi136.stderr:254624: 9' 2024-09-15T21:13:32.659 INFO:tasks.workunit.client.0.smithi136.stderr:254624: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-15T21:13:32.659 INFO:tasks.workunit.client.0.smithi136.stderr:254624: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T21:13:32.659 INFO:tasks.workunit.client.0.smithi136.stderr:254624: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-15T21:13:32.660 INFO:tasks.workunit.client.0.smithi136.stderr:254624: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836579 2024-09-15T21:13:32.660 INFO:tasks.workunit.client.0.smithi136.stderr:254624: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836579 2024-09-15T21:13:32.660 INFO:tasks.workunit.client.0.smithi136.stderr:254624: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836579' 2024-09-15T21:13:33.993 INFO:tasks.workunit.client.0.smithi136.stderr:254624: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $id6022c640 -1 osd.5 99 log_to_monitors true 2024-09-15T21:13:33.993 INFO:tasks.workunit.client.0.smithi136.stderr:254622: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:13:33.993 INFO:tasks.workunit.client.0.smithi136.stderr:254622: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:13:33.993 INFO:tasks.workunit.client.0.smithi136.stderr:254622: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-15T21:13:33.993 INFO:tasks.workunit.client.0.smithi136.stderr:254622: 4 2024-09-15T21:13:33.994 INFO:tasks.workunit.client.0.smithi136.stderr:254622: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:13:33.994 INFO:tasks.workunit.client.0.smithi136.stderr:254622: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-15T21:13:33.994 INFO:tasks.workunit.client.0.smithi136.stderr:254622: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:13:33.994 INFO:tasks.workunit.client.0.smithi136.stderr:254622: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:13:33.994 INFO:tasks.workunit.client.0.smithi136.stderr:254622: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:13:33.994 INFO:tasks.workunit.client.0.smithi136.stderr:254622: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-15T21:13:33.994 INFO:tasks.workunit.client.0.smithi136.stderr:254622: 5 2024-09-15T21:13:33.994 INFO:tasks.workunit.client.0.smithi136.stderr:254622: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:13:33.994 INFO:tasks.workunit.client.0.smithi136.stderr:254622: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-15T21:13:33.994 INFO:tasks.workunit.client.0.smithi136.stderr:254622: osd.5 up in weight 1 up_from 106 up_thru 106 down_at 100 last_clean_interval [97,99) [v2:127.0.0.1:6874/1502190426,v1:127.0.0.1:6875/1502190426] [v2:127.0.0.1:6876/1502190426,v1:127.0.0.1:6877/1502190426] exists,up 9918af07-0bbe-4711-bea1-402a4b5b6247 2024-09-15T21:13:33.994 INFO:tasks.workunit.client.0.smithi136.stderr:254622: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-15T21:13:33.994 INFO:tasks.workunit.client.0.smithi136.stderr:254622: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-15T21:13:33.995 INFO:tasks.workunit.client.0.smithi136.stderr:254622: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-15T21:13:33.995 INFO:tasks.workunit.client.0.smithi136.stderr:254622: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2024-09-15T21:13:33.995 INFO:tasks.workunit.client.0.smithi136.stderr:254622: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-15T21:13:33.995 INFO:tasks.workunit.client.0.smithi136.stderr:254622: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-15T21:13:33.995 INFO:tasks.workunit.client.0.smithi136.stderr:254622: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-15T21:13:33.995 INFO:tasks.workunit.client.0.smithi136.stderr:254622: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-15T21:13:33.995 INFO:tasks.workunit.client.0.smithi136.stderr:254622: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-15T21:13:33.995 INFO:tasks.workunit.client.0.smithi136.stderr:254622: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-15T21:13:33.995 INFO:tasks.workunit.client.0.smithi136.stderr:254622: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-15T21:13:33.995 INFO:tasks.workunit.client.0.smithi136.stderr:254622: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-15T21:13:33.995 INFO:tasks.workunit.client.0.smithi136.stderr:254622: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-15T21:13:33.995 INFO:tasks.workunit.client.0.smithi136.stderr:254622: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-15T21:13:33.995 INFO:tasks.workunit.client.0.smithi136.stderr:254622: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-15T21:13:33.995 INFO:tasks.workunit.client.0.smithi136.stderr:254622: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-15T21:13:33.996 INFO:tasks.workunit.client.0.smithi136.stderr:254622: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-15T21:13:33.996 INFO:tasks.workunit.client.0.smithi136.stderr:254622: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-15T21:13:33.996 INFO:tasks.workunit.client.0.smithi136.stderr:254622: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-15T21:13:33.996 INFO:tasks.workunit.client.0.smithi136.stderr:254622: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-15T21:13:33.996 INFO:tasks.workunit.client.0.smithi136.stderr:254622: 1 2024-09-15T21:13:33.996 INFO:tasks.workunit.client.0.smithi136.stderr:254622: 2 2024-09-15T21:13:33.996 INFO:tasks.workunit.client.0.smithi136.stderr:254622: 3 2024-09-15T21:13:33.996 INFO:tasks.workunit.client.0.smithi136.stderr:254622: 4 2024-09-15T21:13:33.996 INFO:tasks.workunit.client.0.smithi136.stderr:254622: 5 2024-09-15T21:13:33.996 INFO:tasks.workunit.client.0.smithi136.stderr:254622: 6 2024-09-15T21:13:33.996 INFO:tasks.workunit.client.0.smithi136.stderr:254622: 7 2024-09-15T21:13:33.996 INFO:tasks.workunit.client.0.smithi136.stderr:254622: 8 2024-09-15T21:13:33.996 INFO:tasks.workunit.client.0.smithi136.stderr:254622: 9' 2024-09-15T21:13:33.996 INFO:tasks.workunit.client.0.smithi136.stderr:254622: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-15T21:13:34.594 INFO:tasks.workunit.client.0.smithi136.stderr:254622: /home/ubuntu/cephtest/clone.client.0/qa/standalons 2024-09-15T21:13:34.594 INFO:tasks.workunit.client.0.smithi136.stderr:254624: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-15T21:13:34.594 INFO:tasks.workunit.client.0.smithi136.stderr:254624: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=42949673055 2024-09-15T21:13:34.594 INFO:tasks.workunit.client.0.smithi136.stderr:254624: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 42949673055 2024-09-15T21:13:34.594 INFO:tasks.workunit.client.0.smithi136.stderr:254624: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836579 1-42949673055' 2024-09-15T21:13:34.594 INFO:tasks.workunit.client.0.smithi136.stderr:254624: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T21:13:34.594 INFO:tasks.workunit.client.0.smithi136.stderr:254624: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-15T21:13:34.594 INFO:tasks.workunit.client.0.smithi136.stderr:254624: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=64424509525 2024-09-15T21:13:34.595 INFO:tasks.workunit.client.0.smithi136.stderr:254624: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 64424509525 2024-09-15T21:13:34.595 INFO:tasks.workunit.client.0.smithi136.stderr:254624: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836579 1-42949673055 2-64424509525' 2024-09-15T21:13:34.595 INFO:tasks.workunit.client.0.smithi136.stderr:254624: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T21:13:34.595 INFO:tasks.workunit.client.0.smithi136.stderr:254624: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-15T21:13:34.595 INFO:tasks.workunit.client.0.smithi136.stderr:254624: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=313532612635 2024-09-15T21:13:34.595 INFO:tasks.workunit.client.0.smithi136.stderr:254624: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 313532612635 2024-09-15T21:13:34.595 INFO:tasks.workunit.client.0.smithi136.stderr:254624: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836579 1-42949673055 2-64424509525 3-313532612635' 2024-09-15T21:13:34.595 INFO:tasks.workunit.client.0.smithi136.stderr:254624: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T21:13:34.595 INFO:tasks.workunit.client.0.smithi136.stderr:254624: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-09-15T21:13:34.595 INFO:tasks.workunit.client.0.smithi136.stderr:254624: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=107374182474 2024-09-15T21:13:34.595 INFO:tasks.workunit.client.0.smithi136.stderr:254624: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 107374182474 2024-09-15T21:13:34.595 INFO:tasks.workunit.client.0.smithi136.stderr:254624: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836579 1-42949673055 2-64424509525 3-313532612635 4-107374182474' 2024-09-15T21:13:34.595 INFO:tasks.workunit.client.0.smithi136.stderr:254624: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T21:13:34.596 INFO:tasks.workunit.client.0.smithi136.stderr:254624: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-09-15T21:13:34.596 INFO:tasks.workunit.client.0.smithi136.stderr:254624: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=455266533378 2024-09-15T21:13:34.596 INFO:tasks.workunit.client.0.smithi136.stderr:254624: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 455266533378 2024-09-15T21:13:34.596 INFO:tasks.workunit.client.0.smithi136.stderr:254624: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836579 1-42949673055 2-64424509525 3-313532612635 4-107374182474 5-455266533378' 2024-09-15T21:13:34.596 INFO:tasks.workunit.client.0.smithi136.stderr:254624: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T21:13:34.596 INFO:tasks.workunit.client.0.smithi136.stderr:254624: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-09-15T21:13:34.596 INFO:tasks.workunit.client.0.smithi136.stderr:254624: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=150323855422 2024-09-15T21:13:34.596 INFO:tasks.workunit.client.0.smithi136.stderr:254624: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 150323855422 2024-09-15T21:13:34.596 INFO:tasks.workunit.client.0.smithi136.stderr:254624: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836579 1-42949673055 2-64424509525 3-313532612635 4-107374182474 5-455266533378 6-150323855422' 2024-09-15T21:13:34.596 INFO:tasks.workunit.client.0.smithi136.stderr:254624: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T21:13:34.596 INFO:tasks.workunit.client.0.smithi136.stderr:254624: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2024-09-15T21:13:35.051 INFO:tasks.workunit.client.0.smithi136.stderr:254624: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2e/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T21:13:35.051 INFO:tasks.workunit.client.0.smithi136.stderr:254622: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-15T21:13:35.051 INFO:tasks.workunit.client.0.smithi136.stderr:254622: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836581 2024-09-15T21:13:35.051 INFO:tasks.workunit.client.0.smithi136.stderr:254622: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836581 2024-09-15T21:13:35.052 INFO:tasks.workunit.client.0.smithi136.stderr:254622: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836581' 2024-09-15T21:13:35.052 INFO:tasks.workunit.client.0.smithi136.stderr:254622: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T21:13:35.052 INFO:tasks.workunit.client.0.smithi136.stderr:254622: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-15T21:13:35.052 INFO:tasks.workunit.client.0.smithi136.stderr:254622: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=42949673057 2024-09-15T21:13:35.052 INFO:tasks.workunit.client.0.smithi136.stderr:254622: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 42949673057 2024-09-15T21:13:35.052 INFO:tasks.workunit.client.0.smithi136.stderr:254622: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836581 1-42949673057' 2024-09-15T21:13:35.052 INFO:tasks.workunit.client.0.smithi136.stderr:254622: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T21:13:35.053 INFO:tasks.workunit.client.0.smithi136.stderr:254622: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-15T21:13:35.053 INFO:tasks.workunit.client.0.smithi136.stderr:254622: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=64424509527 2024-09-15T21:13:35.053 INFO:tasks.workunit.client.0.smithi136.stderr:254622: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 64424509527 2024-09-15T21:13:35.053 INFO:tasks.workunit.client.0.smithi136.stderr:254622: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836581 1-42949673057 2-64424509527' 2024-09-15T21:13:35.053 INFO:tasks.workunit.client.0.smithi136.stderr:254622: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T21:13:35.053 INFO:tasks.workunit.client.0.smithi136.stderr:254622: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-15T21:13:35.053 INFO:tasks.workunit.client.0.smithi136.stderr:254622: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=313532612636 2024-09-15T21:13:35.053 INFO:tasks.workunit.client.0.smithi136.stderr:254622: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 313532612636 2024-09-15T21:13:35.054 INFO:tasks.workunit.client.0.smithi136.stderr:254622: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836581 1-42949673057 2-64424509527 3-313532612636' 2024-09-15T21:13:35.054 INFO:tasks.workunit.client.0.smithi136.stderr:254622: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T21:13:35.054 INFO:tasks.workunit.client.0.smithi136.stderr:254622: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-09-15T21:13:35.054 INFO:tasks.workunit.client.0.smithi136.stderr:254622: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=107374182475 2024-09-15T21:13:35.054 INFO:tasks.workunit.client.0.smithi136.stderr:254622: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 107374182475 2024-09-15T21:13:35.054 INFO:tasks.workunit.client.0.smithi136.stderr:254622: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836581 1-42949673057 2-64424509527 3-313532612636 4-107374182475' 2024-09-15T21:13:35.055 INFO:tasks.workunit.client.0.smithi136.stderr:254622: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T21:13:35.055 INFO:tasks.workunit.client.0.smithi136.stderr:254622: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-09-15T21:13:35.055 INFO:tasks.workunit.client.0.smithi136.stderr:254622: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=455266533379 2024-09-15T21:13:35.055 INFO:tasks.workunit.client.0.smithi136.stderr:254622: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 455266533379 2024-09-15T21:13:35.055 INFO:tasks.workunit.client.0.smithi136.stderr:254622: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836581 1-42949673057 2-64424509527 3-313532612636 4-107374182475 5-455266533379' 2024-09-15T21:13:35.055 INFO:tasks.workunit.client.0.smithi136.stderr:254622: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T21:13:35.055 INFO:tasks.workunit.client.0.smithi136.stderr:254622: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-09-15T21:13:35.123 INFO:tasks.workunit.client.0.smithi136.stderr:254622: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=150323855210: flush_pg_stats: seq=171798691894 2024-09-15T21:13:35.123 INFO:tasks.workunit.client.0.smithi136.stderr:254624: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 171798691894 2024-09-15T21:13:35.124 INFO:tasks.workunit.client.0.smithi136.stderr:254624: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836579 1-42949673055 2-64424509525 3-313532612635 4-107374182474 5-455266533378 6-150323855422 7-171798691894' 2024-09-15T21:13:35.124 INFO:tasks.workunit.client.0.smithi136.stderr:254624: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T21:13:35.124 INFO:tasks.workunit.client.0.smithi136.stderr:254624: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2024-09-15T21:13:35.124 INFO:tasks.workunit.client.0.smithi136.stderr:254624: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=193273528368 2024-09-15T21:13:35.124 INFO:tasks.workunit.client.0.smithi136.stderr:254624: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 193273528368 2024-09-15T21:13:35.124 INFO:tasks.workunit.client.0.smithi136.stderr:254624: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836579 1-42949673055 2-64424509525 3-313532612635 4-107374182474 5-455266533378 6-150323855422 7-171798691894 8-193273528368' 2024-09-15T21:13:35.124 INFO:tasks.workunit.client.0.smithi136.stderr:254624: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T21:13:35.124 INFO:tasks.workunit.client.0.smithi136.stderr:254624: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2024-09-15T21:13:35.124 INFO:tasks.workunit.client.0.smithi136.stderr:254624: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=446676598786 2024-09-15T21:13:35.124 INFO:tasks.workunit.client.0.smithi136.stderr:254624: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 446676598786 2024-09-15T21:13:35.124 INFO:tasks.workunit.client.0.smithi136.stderr:254624: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836579 1-42949673055 2-64424509525 3-313532612635 4-107374182474 5-455266533378 6-150323855422 7-171798691894 8-193273528368 9-446676598786' 2024-09-15T21:13:35.124 INFO:tasks.workunit.client.0.smithi136.stderr:254624: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T21:13:35.125 INFO:tasks.workunit.client.0.smithi136.stderr:254624: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836579 2024-09-15T21:13:35.125 INFO:tasks.workunit.client.0.smithi136.stderr:254624: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T21:13:35.125 INFO:tasks.workunit.client.0.smithi136.stderr:254624: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-15T21:13:35.125 INFO:tasks.workunit.client.0.smithi136.stderr:254624: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836579 2024-09-15T21:13:35.125 INFO:tasks.workunit.client.0.smithi136.stderr:254624: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T21:13:35.125 INFO:tasks.workunit.client.0.smithi136.stderr:254624: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836579 2024-09-15T21:13:35.125 INFO:tasks.workunit.client.0.smithi136.stderr:254624: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836579' 2024-09-15T21:13:35.125 INFO:tasks.workunit.client.0.smithi136.stderr:254624: waiting osd.0 seq 21474836579 2024-09-15T21:13:35.125 INFO:tasks.workunit.client.0.smithi136.stderr:254624: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T21:13:35.125 INFO:tasks.workunit.client.0.smithi136.stderr:254624: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836580 -lt 21474836579 2024-09-15T21:13:35.125 INFO:tasks.workunit.client.0.smithi136.stderr:254624: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T21:13:35.125 INFO:tasks.workunit.client.0.smithi136.stderr:254624: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-42949673055 2024-09-15T21:13:35.125 INFO:tasks.workunit.client.0.smithi136.stderr:254624: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T21:13:35.126 INFO:tasks.workunit.client.0.smithi136.stderr:254624: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-15T21:13:35.126 INFO:tasks.workunit.client.0.smithi136.stderr:254624: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-42949673055 2024-09-15T21:13:35.126 INFO:tasks.workunit.client.0.smithi136.stderr:254624: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T21:13:35.126 INFO:tasks.workunit.client.0.smithi136.stderr:254624: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=42949673055 2024-09-15T21:13:35.126 INFO:tasks.workunit.client.0.smithi136.stderr:254624: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 42949673055' 2024-09-15T21:13:35.126 INFO:tasks.workunit.client.0.smithi136.stderr:254624: waiting osd.1 seq 42949673055 2024-09-15T21:13:36.357 INFO:tasks.workunit.client.0.smithi136.stderr:254624: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-15T21:13:36.357 INFO:tasks.workunit.client.0.smithi136.stderr:254624: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 42949673056 -lt 42949673055 2024-09-15T21:13:36.358 INFO:tasks.workunit.client.0.smithi136.stderr:254624: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T21:13:36.358 INFO:tasks.workunit.client.0.smithi136.stderr:254624: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-64424509525 2024-09-15T21:13:36.358 INFO:tasks.workunit.client.0.smithi136.stderr:254624: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T21:13:36.358 INFO:tasks.workunit.client.0.smithi136.stderr:254624: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-15T21:13:36.358 INFO:tasks.workunit.client.0.smithi136.stderr:254624: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-64424509525 2024-09-15T21:13:36.358 INFO:tasks.workunit.client.0.smithi136.stderr:254624: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T21:13:36.358 INFO:tasks.workunit.client.0.smithi136.stderr:254624: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=64424509525 2024-09-15T21:13:36.358 INFO:tasks.workunit.client.0.smithi136.stderr:254624: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 64424509525' 2024-09-15T21:13:36.358 INFO:tasks.workunit.client.0.smithi136.stderr:254624: waiting osd.2 seq 64424509525 2024-09-15T21:13:36.359 INFO:tasks.workunit.client.0.smithi136.stderr:254624: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-15T21:13:36.359 INFO:tasks.workunit.client.0.smithi136.stderr:254624: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 64424509526 -lt 64424509525 2024-09-15T21:13:36.359 INFO:tasks.workunit.client.0.smithi136.stderr:254624: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T21:13:36.359 INFO:tasks.workunit.client.0.smithi136.stderr:254624: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 3-313532612635 2024-09-15T21:13:36.359 INFO:tasks.workunit.client.0.smithi136.stderr:254624: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T21:13:36.359 INFO:tasks.workunit.client.0.smithi136.stderr:254624: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=3 2024-09-15T21:13:36.359 INFO:tasks.workunit.client.0.smithi136.stderr:254624: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 3-313532612635 2024-09-15T21:13:36.359 INFO:tasks.workunit.client.0.smithi136.stderr:254624: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T21:13:36.360 INFO:tasks.workunit.client.0.smithi136.stderr:254624: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=313532612635 2024-09-15T21:13:36.360 INFO:tasks.workunit.client.0.smithi136.stderr:254624: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.3 seq 313532612635' 2024-09-15T21:13:36.360 INFO:tasks.workunit.client.0.smithi136.stderr:254624: waiting osd.3 seq 313532612635 2024-09-15T21:13:36.360 INFO:tasks.workunit.client.0.smithi136.stderr:254624: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-15T21:13:36.360 INFO:tasks.workunit.client.0.smithi136.stderr:254624: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 313532612637 -lt 313532612635 2024-09-15T21:13:36.360 INFO:tasks.workunit.client.0.smithi136.stderr:254624: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T21:13:36.360 INFO:tasks.workunit.client.0.smithi136.stderr:254624: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 4-107374182474 2024-09-15T21:13:36.360 INFO:tasks.workunit.client.0.smithi136.stderr:254624: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T21:13:36.361 INFO:tasks.workunit.client.0.smithi136.stderr:254624: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=4 2024-09-15T21:13:36.361 INFO:tasks.workunit.client.0.smithi136.stderr:254624: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 4-107374182474 2024-09-15T21:13:36.361 INFO:tasks.workunit.client.0.smithi136.stderr:254624: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T21:13:36.361 INFO:tasks.workunit.client.0.smithi136.stderr:254624: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=107374182474 2024-09-15T21:13:36.361 INFO:tasks.workunit.client.0.smithi136.stderr:254624: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.4 seq 107374182474' 2024-09-15T21:13:36.361 INFO:tasks.workunit.client.0.smithi136.stderr:254624: waiting osd.4 seq 107374182474 2024-09-15T21:13:36.361 INFO:tasks.workunit.client.0.smithi136.stderr:254624: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 4 2024-09-15T21:13:36.361 INFO:tasks.workunit.client.0.smithi136.stderr:254624: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 107374182475 -lt 107374182474 2024-09-15T21:13:36.362 INFO:tasks.workunit.client.0.smithi136.stderr:254624: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T21:13:36.362 INFO:tasks.workunit.client.0.smithi136.stderr:254624: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 5-455266533378 2024-09-15T21:13:37.011 INFO:tasks.workunit.client.0.smithi136.stderr:254624: //hom423 2024-09-15T21:13:37.011 INFO:tasks.workunit.client.0.smithi136.stderr:254622: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 150323855423 2024-09-15T21:13:37.011 INFO:tasks.workunit.client.0.smithi136.stderr:254622: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836581 1-42949673057 2-64424509527 3-313532612636 4-107374182475 5-455266533379 6-150323855423' 2024-09-15T21:13:37.011 INFO:tasks.workunit.client.0.smithi136.stderr:254622: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T21:13:37.011 INFO:tasks.workunit.client.0.smithi136.stderr:254622: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2024-09-15T21:13:37.011 INFO:tasks.workunit.client.0.smithi136.stderr:254622: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=171798691895 2024-09-15T21:13:37.011 INFO:tasks.workunit.client.0.smithi136.stderr:254622: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 171798691895 2024-09-15T21:13:37.011 INFO:tasks.workunit.client.0.smithi136.stderr:254622: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836581 1-42949673057 2-64424509527 3-313532612636 4-107374182475 5-455266533379 6-150323855423 7-171798691895' 2024-09-15T21:13:37.011 INFO:tasks.workunit.client.0.smithi136.stderr:254622: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T21:13:37.011 INFO:tasks.workunit.client.0.smithi136.stderr:254622: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2024-09-15T21:13:37.011 INFO:tasks.workunit.client.0.smithi136.stderr:254622: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=193273528369 2024-09-15T21:13:37.012 INFO:tasks.workunit.client.0.smithi136.stderr:254622: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 193273528369 2024-09-15T21:13:37.012 INFO:tasks.workunit.client.0.smithi136.stderr:254622: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836581 1-42949673057 2-64424509527 3-313532612636 4-107374182475 5-455266533379 6-150323855423 7-171798691895 8-193273528369' 2024-09-15T21:13:37.012 INFO:tasks.workunit.client.0.smithi136.stderr:254622: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T21:13:37.012 INFO:tasks.workunit.client.0.smithi136.stderr:254622: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2024-09-15T21:13:37.012 INFO:tasks.workunit.client.0.smithi136.stderr:254622: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=446676598787 2024-09-15T21:13:37.012 INFO:tasks.workunit.client.0.smithi136.stderr:254622: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 446676598787 2024-09-15T21:13:37.012 INFO:tasks.workunit.client.0.smithi136.stderr:254622: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836581 1-42949673057 2-64424509527 3-313532612636 4-107374182475 5-455266533379 6-150323855423 7-171798691895 8-193273528369 9-446676598787' 2024-09-15T21:13:37.012 INFO:tasks.workunit.client.0.smithi136.stderr:254622: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T21:13:37.012 INFO:tasks.workunit.client.0.smithi136.stderr:254622: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836581 2024-09-15T21:13:37.012 INFO:tasks.workunit.client.0.smithi136.stderr:254622: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T21:13:37.012 INFO:tasks.workunit.client.0.smithi136.stderr:254622: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-15T21:13:37.012 INFO:tasks.workunit.client.0.smithi136.stderr:254622: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836581 2024-09-15T21:13:37.012 INFO:tasks.workunit.client.0.smithi136.stderr:254622: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T21:13:37.013 INFO:tasks.workunit.client.0.smithi136.stderr:254622: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836581 2024-09-15T21:13:37.013 INFO:tasks.workunit.client.0.smithi136.stderr:254622: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836581' 2024-09-15T21:13:37.013 INFO:tasks.workunit.client.0.smithi136.stderr:254622: waiting osd.0 seq 21474836581 2024-09-15T21:13:37.013 INFO:tasks.workunit.client.0.smithi136.stderr:254622: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T21:13:37.013 INFO:tasks.workunit.client.0.smithi136.stderr:254622: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836580 -lt 21474836581 2024-09-15T21:13:37.013 INFO:tasks.workunit.client.0.smithi136.stderr:254622: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-15T21:13:37.013 INFO:tasks.workunit.client.0.smithi136.stderr:254622: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-15T21:13:37.013 INFO:tasks.workunit.client.0.smithi136.stderr:254622: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T21:13:37.247 INFO:tasks.workunit.client.0.smithi136.stderr:254622: /home/ubuntu/cephtest/clone.client.0/qa/ste/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T21:13:37.247 INFO:tasks.workunit.client.0.smithi136.stderr:254624: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=5 2024-09-15T21:13:37.247 INFO:tasks.workunit.client.0.smithi136.stderr:254624: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 5-455266533378 2024-09-15T21:13:37.247 INFO:tasks.workunit.client.0.smithi136.stderr:254624: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T21:13:37.248 INFO:tasks.workunit.client.0.smithi136.stderr:254624: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=455266533378 2024-09-15T21:13:37.248 INFO:tasks.workunit.client.0.smithi136.stderr:254624: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.5 seq 455266533378' 2024-09-15T21:13:37.248 INFO:tasks.workunit.client.0.smithi136.stderr:254624: waiting osd.5 seq 455266533378 2024-09-15T21:13:37.248 INFO:tasks.workunit.client.0.smithi136.stderr:254624: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 5 2024-09-15T21:13:37.248 INFO:tasks.workunit.client.0.smithi136.stderr:254624: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 455266533379 -lt 455266533378 2024-09-15T21:13:37.248 INFO:tasks.workunit.client.0.smithi136.stderr:254624: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T21:13:37.248 INFO:tasks.workunit.client.0.smithi136.stderr:254624: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 6-150323855422 2024-09-15T21:13:37.248 INFO:tasks.workunit.client.0.smithi136.stderr:254624: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T21:13:37.248 INFO:tasks.workunit.client.0.smithi136.stderr:254624: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=6 2024-09-15T21:13:37.248 INFO:tasks.workunit.client.0.smithi136.stderr:254624: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 6-150323855422 2024-09-15T21:13:37.248 INFO:tasks.workunit.client.0.smithi136.stderr:254624: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T21:13:37.248 INFO:tasks.workunit.client.0.smithi136.stderr:254624: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=150323855422 2024-09-15T21:13:37.249 INFO:tasks.workunit.client.0.smithi136.stderr:254624: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.6 seq 150323855422' 2024-09-15T21:13:37.249 INFO:tasks.workunit.client.0.smithi136.stderr:254624: waiting osd.6 seq 150323855422 2024-09-15T21:13:37.249 INFO:tasks.workunit.client.0.smithi136.stderr:254624: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 6 2024-09-15T21:13:37.249 INFO:tasks.workunit.client.0.smithi136.stderr:254624: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 150323855424 -lt 150323855422 2024-09-15T21:13:37.249 INFO:tasks.workunit.client.0.smithi136.stderr:254624: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T21:13:37.249 INFO:tasks.workunit.client.0.smithi136.stderr:254624: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 7-171798691894 2024-09-15T21:13:37.249 INFO:tasks.workunit.client.0.smithi136.stderr:254624: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T21:13:37.249 INFO:tasks.workunit.client.0.smithi136.stderr:254624: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=7 2024-09-15T21:13:37.249 INFO:tasks.workunit.client.0.smithi136.stderr:254624: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 7-171798691894 2024-09-15T21:13:37.249 INFO:tasks.workunit.client.0.smithi136.stderr:254624: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T21:13:37.249 INFO:tasks.workunit.client.0.smithi136.stderr:254624: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=171798691894 2024-09-15T21:13:37.249 INFO:tasks.workunit.client.0.smithi136.stderr:254624: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.7 seq 171798691894' 2024-09-15T21:13:37.249 INFO:tasks.workunit.client.0.smithi136.stderr:254624: waiting osd.7 seq 171798691894 2024-09-15T21:13:37.250 INFO:tasks.workunit.client.0.smithi136.stderr:254624: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 7 2024-09-15T21:13:37.250 INFO:tasks.workunit.client.0.smithi136.stderr:254624: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 171798691895 -lt 171798691894 2024-09-15T21:13:37.250 INFO:tasks.workunit.client.0.smithi136.stderr:254624: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T21:13:37.250 INFO:tasks.workunit.client.0.smithi136.stderr:254624: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 8-193273528368 2024-09-15T21:13:37.250 INFO:tasks.workunit.client.0.smithi136.stderr:254624: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T21:13:37.250 INFO:tasks.workunit.client.0.smithi136.stderr:254624: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=8 2024-09-15T21:13:37.250 INFO:tasks.workunit.client.0.smithi136.stderr:254624: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 8-193273528368 2024-09-15T21:13:37.954 INFO:tasks.workunit.client.0.smithi136.stderr:254624: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pandalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836581 -lt 21474836581 2024-09-15T21:13:37.955 INFO:tasks.workunit.client.0.smithi136.stderr:254622: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T21:13:37.955 INFO:tasks.workunit.client.0.smithi136.stderr:254622: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-42949673057 2024-09-15T21:13:37.955 INFO:tasks.workunit.client.0.smithi136.stderr:254622: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T21:13:37.955 INFO:tasks.workunit.client.0.smithi136.stderr:254622: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-15T21:13:37.955 INFO:tasks.workunit.client.0.smithi136.stderr:254622: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-42949673057 2024-09-15T21:13:37.955 INFO:tasks.workunit.client.0.smithi136.stderr:254622: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T21:13:37.955 INFO:tasks.workunit.client.0.smithi136.stderr:254622: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=42949673057 2024-09-15T21:13:37.955 INFO:tasks.workunit.client.0.smithi136.stderr:254622: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 42949673057' 2024-09-15T21:13:37.955 INFO:tasks.workunit.client.0.smithi136.stderr:254622: waiting osd.1 seq 42949673057 2024-09-15T21:13:37.956 INFO:tasks.workunit.client.0.smithi136.stderr:254622: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-15T21:13:37.956 INFO:tasks.workunit.client.0.smithi136.stderr:254622: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 42949673057 -lt 42949673057 2024-09-15T21:13:37.956 INFO:tasks.workunit.client.0.smithi136.stderr:254622: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T21:13:37.956 INFO:tasks.workunit.client.0.smithi136.stderr:254622: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-64424509527 2024-09-15T21:13:37.956 INFO:tasks.workunit.client.0.smithi136.stderr:254622: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T21:13:37.956 INFO:tasks.workunit.client.0.smithi136.stderr:254622: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-15T21:13:37.956 INFO:tasks.workunit.client.0.smithi136.stderr:254622: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-64424509527 2024-09-15T21:13:37.956 INFO:tasks.workunit.client.0.smithi136.stderr:254622: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T21:13:37.956 INFO:tasks.workunit.client.0.smithi136.stderr:254622: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=64424509527 2024-09-15T21:13:37.957 INFO:tasks.workunit.client.0.smithi136.stderr:254622: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 64424509527' 2024-09-15T21:13:37.957 INFO:tasks.workunit.client.0.smithi136.stderr:254622: waiting osd.2 seq 64424509527 2024-09-15T21:13:37.957 INFO:tasks.workunit.client.0.smithi136.stderr:254622: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-15T21:13:37.957 INFO:tasks.workunit.client.0.smithi136.stderr:254622: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 64424509527 -lt 64424509527 2024-09-15T21:13:37.957 INFO:tasks.workunit.client.0.smithi136.stderr:254622: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T21:13:37.957 INFO:tasks.workunit.client.0.smithi136.stderr:254622: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 3-313532612636 2024-09-15T21:13:37.957 INFO:tasks.workunit.client.0.smithi136.stderr:254622: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T21:13:37.957 INFO:tasks.workunit.client.0.smithi136.stderr:254622: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=3 2024-09-15T21:13:37.957 INFO:tasks.workunit.client.0.smithi136.stderr:254622: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 3-313532612636 2024-09-15T21:13:37.957 INFO:tasks.workunit.client.0.smithi136.stderr:254622: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T21:13:37.957 INFO:tasks.workunit.client.0.smithi136.stderr:254622: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=313532612636 2024-09-15T21:13:37.957 INFO:tasks.workunit.client.0.smithi136.stderr:254622: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.3 seq 313532612636' 2024-09-15T21:13:37.957 INFO:tasks.workunit.client.0.smithi136.stderr:254622: waiting osd.3 seq 313532612636 2024-09-15T21:13:37.958 INFO:tasks.workunit.client.0.smithi136.stderr:254622: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-15T21:13:37.958 INFO:tasks.workunit.client.0.smithi136.stderr:254622: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 313532612637 -lt 313532612636 2024-09-15T21:13:37.958 INFO:tasks.workunit.client.0.smithi136.stderr:254622: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T21:13:37.958 INFO:tasks.workunit.client.0.smithi136.stderr:254622: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 4-107374182475 2024-09-15T21:13:37.958 INFO:tasks.workunit.client.0.smithi136.stderr:254622: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T21:13:38.866 INFO:tasks.workunit.client.0.smithi136.stderr:254622: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=4 2024-09-15T21:13:38.866 INFO:tasks.workunit.client.0.smithi136.stderr:254622: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 4-107374182475 2024-09-15T21:13:38.866 INFO:tasks.workunit.client.0.smithi136.stderr:254622: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T21:13:38.866 INFO:tasks.workunit.client.0.smithi136.stderr:254622: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=107374182475 2024-09-15T21:13:38.866 INFO:tasks.workunit.client.0.smithi136.stderr:254622: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.4 seq 107374182475' 2024-09-15T21:13:38.866 INFO:tasks.workunit.client.0.smithi136.stderr:254622: waiting osd.4 seq 107374182475 2024-09-15T21:13:38.866 INFO:tasks.workunit.client.0.smithi136.stderr:254622: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 4 2024-09-15T21:13:38.866 INFO:tasks.workunit.client.0.smithi136.stderr:254622: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 107374182476 -lt 107374182475 2024-09-15T21:13:38.866 INFO:tasks.workunit.client.0.smithi136.stderr:254622: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T21:13:38.866 INFO:tasks.workunit.client.0.smithi136.stderr:254622: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 5-455266533379 2024-09-15T21:13:38.867 INFO:tasks.workunit.client.0.smithi136.stderr:254622: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T21:13:38.867 INFO:tasks.workunit.client.0.smithi136.stderr:254622: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=5 2024-09-15T21:13:38.867 INFO:tasks.workunit.client.0.smithi136.stderr:254622: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 5-455266533379 2024-09-15T21:13:38.867 INFO:tasks.workunit.client.0.smithi136.stderr:254622: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T21:13:38.867 INFO:tasks.workunit.client.0.smithi136.stderr:254622: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=455266533379 2024-09-15T21:13:38.867 INFO:tasks.workunit.client.0.smithi136.stderr:254622: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.5 seq 455266533379' 2024-09-15T21:13:38.867 INFO:tasks.workunit.client.0.smithi136.stderr:254622: waiting osd.5 seq 455266533379 2024-09-15T21:13:38.867 INFO:tasks.workunit.client.0.smithi136.stderr:254622: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 5 2024-09-15T21:13:38.867 INFO:tasks.workunit.client.0.smithi136.stderr:254622: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 455266533380 -lt 455266533379 2024-09-15T21:13:38.867 INFO:tasks.workunit.client.0.smithi136.stderr:254622: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T21:13:38.867 INFO:tasks.workunit.client.0.smithi136.stderr:254622: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 6-150323855423 2024-09-15T21:13:38.868 INFO:tasks.workunit.client.0.smithi136.stderr:254622: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T21:13:38.868 INFO:tasks.workunit.client.0.smithi136.stderr:254622: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=6 2024-09-15T21:13:38.868 INFO:tasks.workunit.client.0.smithi136.stderr:254622: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 6-150323855423 2024-09-15T21:13:38.868 INFO:tasks.workunit.client.0.smithi136.stderr:254622: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T21:13:38.868 INFO:tasks.workunit.client.0.smithi136.stderr:254622: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=150323855423 2024-09-15T21:13:38.868 INFO:tasks.workunit.client.0.smithi136.stderr:254622: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.6 seq 150323855423' 2024-09-15T21:13:38.868 INFO:tasks.workunit.client.0.smithi136.stderr:254622: waiting osd.6 seq 150323855423 2024-09-15T21:13:38.868 INFO:tasks.workunit.client.0.smithi136.stderr:254622: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 6 2024-09-15T21:13:38.868 INFO:tasks.workunit.client.0.smithi136.stderr:254622: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 150323855424 -lt 150323855423 2024-09-15T21:13:38.868 INFO:tasks.workunit.client.0.smithi136.stderr:254622: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T21:13:38.868 INFO:tasks.workunit.client.0.smithi136.stderr:254622: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 7-171798691895 2024-09-15T21:13:38.868 INFO:tasks.workunit.client.0.smithi136.stderr:254622: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T21:13:38.869 INFO:tasks.workunit.client.0.smithi136.stderr:254622: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=7 2024-09-15T21:13:38.869 INFO:tasks.workunit.client.0.smithi136.stderr:254622: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 7-171798691895 2024-09-15T21:13:38.869 INFO:tasks.workunit.client.0.smithi136.stderr:254622: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T21:13:38.872 INFO:tasks.workunit.client.0.smithi136.stderr:254622: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2g_stats: cut -d - -f 2 2024-09-15T21:13:38.872 INFO:tasks.workunit.client.0.smithi136.stderr:254624: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=193273528368 2024-09-15T21:13:38.872 INFO:tasks.workunit.client.0.smithi136.stderr:254624: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.8 seq 193273528368' 2024-09-15T21:13:38.872 INFO:tasks.workunit.client.0.smithi136.stderr:254624: waiting osd.8 seq 193273528368 2024-09-15T21:13:38.872 INFO:tasks.workunit.client.0.smithi136.stderr:254624: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 8 2024-09-15T21:13:38.872 INFO:tasks.workunit.client.0.smithi136.stderr:254624: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 193273528369 -lt 193273528368 2024-09-15T21:13:38.872 INFO:tasks.workunit.client.0.smithi136.stderr:254624: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T21:13:38.872 INFO:tasks.workunit.client.0.smithi136.stderr:254624: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 9-446676598786 2024-09-15T21:13:38.872 INFO:tasks.workunit.client.0.smithi136.stderr:254624: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T21:13:38.872 INFO:tasks.workunit.client.0.smithi136.stderr:254624: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=9 2024-09-15T21:13:38.872 INFO:tasks.workunit.client.0.smithi136.stderr:254624: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 9-446676598786 2024-09-15T21:13:38.872 INFO:tasks.workunit.client.0.smithi136.stderr:254624: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T21:13:38.872 INFO:tasks.workunit.client.0.smithi136.stderr:254624: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=446676598786 2024-09-15T21:13:38.873 INFO:tasks.workunit.client.0.smithi136.stderr:254624: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.9 seq 446676598786' 2024-09-15T21:13:38.873 INFO:tasks.workunit.client.0.smithi136.stderr:254624: waiting osd.9 seq 446676598786 2024-09-15T21:13:38.873 INFO:tasks.workunit.client.0.smithi136.stderr:254624: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 9 2024-09-15T21:13:38.873 INFO:tasks.workunit.client.0.smithi136.stderr:254624: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 446676598787 -lt 446676598786 2024-09-15T21:13:38.873 INFO:tasks.workunit.client.0.smithi136.stderr:254624: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-15T21:13:38.873 INFO:tasks.workunit.client.0.smithi136.stderr:254624: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T21:13:38.873 INFO:tasks.workunit.client.0.smithi136.stderr:254624: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T21:13:38.873 INFO:tasks.workunit.client.0.smithi136.stderr:254624: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 5 == 0 2024-09-15T21:13:38.873 INFO:tasks.workunit.client.0.smithi136.stderr:254624: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-15T21:13:38.873 INFO:tasks.workunit.client.0.smithi136.stderr:254624: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-15T21:13:38.873 INFO:tasks.workunit.client.0.smithi136.stderr:254624: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-15T21:13:38.873 INFO:tasks.workunit.client.0.smithi136.stderr:254624: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T21:13:38.873 INFO:tasks.workunit.client.0.smithi136.stderr:254624: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T21:13:38.874 INFO:tasks.workunit.client.0.smithi136.stderr:254624: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T21:13:38.874 INFO:tasks.workunit.client.0.smithi136.stderr:254624: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T21:13:38.874 INFO:tasks.workunit.client.0.smithi136.stderr:254624: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=5 2024-09-15T21:13:38.874 INFO:tasks.workunit.client.0.smithi136.stderr:254624: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-15T21:13:38.874 INFO:tasks.workunit.client.0.smithi136.stderr:254624: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T21:13:38.874 INFO:tasks.workunit.client.0.smithi136.stderr:254624: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T21:13:38.874 INFO:tasks.workunit.client.0.smithi136.stderr:254624: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 = 5 2024-09-15T21:13:38.874 INFO:tasks.workunit.client.0.smithi136.stderr:254624: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-15T21:13:38.874 INFO:tasks.workunit.client.0.smithi136.stderr:254624: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-15T21:13:38.874 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: run_in_background: return 0 2024-09-15T21:13:40.101 INFO:tasks.workunit.client.0.smithi136.stderr:220: flush_pg_stats: seq=171798691895 2024-09-15T21:13:40.101 INFO:tasks.workunit.client.0.smithi136.stderr:254622: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.7 seq 171798691895' 2024-09-15T21:13:40.101 INFO:tasks.workunit.client.0.smithi136.stderr:254622: waiting osd.7 seq 171798691895 2024-09-15T21:13:40.101 INFO:tasks.workunit.client.0.smithi136.stderr:254622: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 7 2024-09-15T21:13:40.101 INFO:tasks.workunit.client.0.smithi136.stderr:254622: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 171798691896 -lt 171798691895 2024-09-15T21:13:40.101 INFO:tasks.workunit.client.0.smithi136.stderr:254622: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T21:13:40.102 INFO:tasks.workunit.client.0.smithi136.stderr:254622: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 8-193273528369 2024-09-15T21:13:40.102 INFO:tasks.workunit.client.0.smithi136.stderr:254622: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T21:13:40.102 INFO:tasks.workunit.client.0.smithi136.stderr:254622: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=8 2024-09-15T21:13:40.102 INFO:tasks.workunit.client.0.smithi136.stderr:254622: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 8-193273528369 2024-09-15T21:13:40.102 INFO:tasks.workunit.client.0.smithi136.stderr:254622: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T21:13:40.102 INFO:tasks.workunit.client.0.smithi136.stderr:254622: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=193273528369 2024-09-15T21:13:40.102 INFO:tasks.workunit.client.0.smithi136.stderr:254622: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.8 seq 193273528369' 2024-09-15T21:13:40.102 INFO:tasks.workunit.client.0.smithi136.stderr:254622: waiting osd.8 seq 193273528369 2024-09-15T21:13:40.102 INFO:tasks.workunit.client.0.smithi136.stderr:254622: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 8 2024-09-15T21:13:40.102 INFO:tasks.workunit.client.0.smithi136.stderr:254622: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 193273528370 -lt 193273528369 2024-09-15T21:13:40.102 INFO:tasks.workunit.client.0.smithi136.stderr:254622: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T21:13:40.102 INFO:tasks.workunit.client.0.smithi136.stderr:254622: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 9-446676598787 2024-09-15T21:13:40.102 INFO:tasks.workunit.client.0.smithi136.stderr:254622: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T21:13:40.103 INFO:tasks.workunit.client.0.smithi136.stderr:254622: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=9 2024-09-15T21:13:40.103 INFO:tasks.workunit.client.0.smithi136.stderr:254622: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 9-446676598787 2024-09-15T21:13:40.103 INFO:tasks.workunit.client.0.smithi136.stderr:254622: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T21:13:40.103 INFO:tasks.workunit.client.0.smithi136.stderr:254622: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=446676598787 2024-09-15T21:13:40.103 INFO:tasks.workunit.client.0.smithi136.stderr:254622: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.9 seq 446676598787' 2024-09-15T21:13:40.103 INFO:tasks.workunit.client.0.smithi136.stderr:254622: waiting osd.9 seq 446676598787 2024-09-15T21:13:40.103 INFO:tasks.workunit.client.0.smithi136.stderr:254622: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 9 2024-09-15T21:13:40.103 INFO:tasks.workunit.client.0.smithi136.stderr:254622: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 446676598788 -lt 446676598787 2024-09-15T21:13:40.103 INFO:tasks.workunit.client.0.smithi136.stderr:254622: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-15T21:13:40.103 INFO:tasks.workunit.client.0.smithi136.stderr:254622: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T21:13:40.103 INFO:tasks.workunit.client.0.smithi136.stderr:254622: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T21:13:40.103 INFO:tasks.workunit.client.0.smithi136.stderr:254622: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 5 == 0 2024-09-15T21:13:40.103 INFO:tasks.workunit.client.0.smithi136.stderr:254622: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-15T21:13:40.104 INFO:tasks.workunit.client.0.smithi136.stderr:254622: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-15T21:13:40.104 INFO:tasks.workunit.client.0.smithi136.stderr:254622: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-15T21:13:40.104 INFO:tasks.workunit.client.0.smithi136.stderr:254622: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T21:13:40.104 INFO:tasks.workunit.client.0.smithi136.stderr:254622: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T21:13:40.723 INFO:tasks.workunit.client.0.smithi136.stderr:254622: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T21:13:40.724 INFO:tasks.workunit.client.0.smithi136.stderr:254622: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T21:13:40.724 INFO:tasks.workunit.client.0.smithi136.stderr:254622: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=5 2024-09-15T21:13:40.724 INFO:tasks.workunit.client.0.smithi136.stderr:254622: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-15T21:13:40.724 INFO:tasks.workunit.client.0.smithi136.stderr:254622: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T21:13:40.724 INFO:tasks.workunit.client.0.smithi136.stderr:254622: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T21:13:40.724 INFO:tasks.workunit.client.0.smithi136.stderr:254622: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 = 5 2024-09-15T21:13:40.724 INFO:tasks.workunit.client.0.smithi136.stderr:254622: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-15T21:13:40.724 INFO:tasks.workunit.client.0.smithi136.stderr:254622: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-15T21:13:40.725 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: run_in_background: return 0 2024-09-15T21:13:40.725 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: wait_background: for pid in $pids 2024-09-15T21:13:40.725 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2175: wait_background: wait 254620 2024-09-15T21:13:40.725 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: wait_background: eval 'pids='\'''\''' 2024-09-15T21:13:40.726 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: wait_background: pids= 2024-09-15T21:13:40.726 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: wait_background: return 0 2024-09-15T21:13:40.726 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:212: corrupt_and_repair_two: return_code=0 2024-09-15T21:13:40.726 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:213: corrupt_and_repair_two: '[' 0 -ne 0 ']' 2024-09-15T21:13:40.726 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:215: corrupt_and_repair_two: rados --pool ecpool get SOMETHING td/osd-scrub-repair/COPY 2024-09-15T21:13:40.755 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:216: corrupt_and_repair_two: diff td/osd-scrub-repair/ORIGINAL td/osd-scrub-repair/COPY 2024-09-15T21:13:40.757 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:263: corrupt_and_repair_erasure_coded: corrupt_and_repair_two td/osd-scrub-repair ecpool 3 5 2024-09-15T21:13:40.757 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:185: corrupt_and_repair_two: local dir=td/osd-scrub-repair 2024-09-15T21:13:40.758 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:186: corrupt_and_repair_two: local poolname=ecpool 2024-09-15T21:13:40.758 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:187: corrupt_and_repair_two: local first=3 2024-09-15T21:13:40.758 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:188: corrupt_and_repair_two: local second=5 2024-09-15T21:13:40.758 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:193: corrupt_and_repair_two: pids= 2024-09-15T21:13:40.758 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:194: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 3 SOMETHING remove 2024-09-15T21:13:40.758 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2135: run_in_background: local pid_variable=pids 2024-09-15T21:13:40.758 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2136: run_in_background: shift 2024-09-15T21:13:40.758 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2140: run_in_background: eval 'pids+=" 257601"' 2024-09-15T21:13:40.758 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2140: run_in_background: pids+=' 257601' 2024-09-15T21:13:40.758 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:195: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 5 SOMETHING remove 2024-09-15T21:13:40.758 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2135: run_in_background: local pid_variable=pids 2024-09-15T21:13:40.758 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2136: run_in_background: shift 2024-09-15T21:13:40.758 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2140: run_in_background: eval 'pids+=" 257603"' 2024-09-15T21:13:40.759 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2140: run_in_background: pids+=' 257603' 2024-09-15T21:13:40.759 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: run_in_background: objectstore_tool td/osd-scrub-repair 3 SOMETHING remove 2024-09-15T21:13:40.759 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:196: corrupt_and_repair_two: wait_background pids 2024-09-15T21:13:40.759 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: wait_background: pids=' 257601 257603' 2024-09-15T21:13:40.759 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: wait_background: return_code=0 2024-09-15T21:13:40.759 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: wait_background: for pid in $pids 2024-09-15T21:13:40.759 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: run_in_background: sed 's/^/257604: /' 2024-09-15T21:13:40.759 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2175: wait_background: wait 257601 2024-09-15T21:13:40.759 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: run_in_background: objectstore_tool td/osd-scrub-repair 5 SOMETHING remove 2024-09-15T21:13:40.759 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: run_in_background: sed 's/^/257606: /' 2024-09-15T21:13:44.152 INFO:tasks.workunit.client.0.smithi136.stderr:257604: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2024-09-15T21:13:44.152 INFO:tasks.workunit.client.0.smithi136.stderr:257604: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2024-09-15T21:13:44.152 INFO:tasks.workunit.client.0.smithi136.stderr:257604: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=3 2024-09-15T21:13:44.152 INFO:tasks.workunit.client.0.smithi136.stderr:257604: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2024-09-15T21:13:44.152 INFO:tasks.workunit.client.0.smithi136.stderr:257604: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 3 SOMETHING remove 2024-09-15T21:13:44.152 INFO:tasks.workunit.client.0.smithi136.stderr:257604: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-09-15T21:13:44.152 INFO:tasks.workunit.client.0.smithi136.stderr:257604: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2024-09-15T21:13:44.152 INFO:tasks.workunit.client.0.smithi136.stderr:257604: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=3 2024-09-15T21:13:44.152 INFO:tasks.workunit.client.0.smithi136.stderr:257604: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2024-09-15T21:13:44.152 INFO:tasks.workunit.client.0.smithi136.stderr:257604: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.3 2024-09-15T21:13:44.152 INFO:tasks.workunit.client.0.smithi136.stderr:257604: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-15T21:13:44.152 INFO:tasks.workunit.client.0.smithi136.stderr:257604: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-15T21:13:44.153 INFO:tasks.workunit.client.0.smithi136.stderr:257604: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-15T21:13:44.153 INFO:tasks.workunit.client.0.smithi136.stderr:257604: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-15T21:13:44.153 INFO:tasks.workunit.client.0.smithi136.stderr:257604: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-15T21:13:44.153 INFO:tasks.workunit.client.0.smithi136.stderr:257604: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-15T21:13:44.153 INFO:tasks.workunit.client.0.smithi136.stderr:257604: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 3 SOMETHING remove 2024-09-15T21:13:44.153 INFO:tasks.workunit.client.0.smithi136.stderr:257604: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-15T21:13:44.153 INFO:tasks.workunit.client.0.smithi136.stderr:257604: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-15T21:13:44.153 INFO:tasks.workunit.client.0.smithi136.stderr:257604: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=3 2024-09-15T21:13:44.153 INFO:tasks.workunit.client.0.smithi136.stderr:257604: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-15T21:13:44.153 INFO:tasks.workunit.client.0.smithi136.stderr:257604: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/3 2024-09-15T21:13:44.153 INFO:tasks.workunit.client.0.smithi136.stderr:257604: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/3 SOMETHING remove 2024-09-15T21:13:44.153 INFO:tasks.workunit.client.0.smithi136.stderr:257604: remove 0#2:eb822e21:::SOMETHING:head# 2024-09-15T21:13:44.153 INFO:tasks.workunit.client.0.smithi136.stderr:257604: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 3 2024-09-15T21:13:44.153 INFO:tasks.workunit.client.0.smithi136.stderr:257604: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2024-09-15T21:13:44.154 INFO:tasks.workunit.client.0.smithi136.stderr:257604: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-15T21:13:44.154 INFO:tasks.workunit.client.0.smithi136.stderr:257604: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=3 2024-09-15T21:13:44.154 INFO:tasks.workunit.client.0.smithi136.stderr:257604: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-15T21:13:44.154 INFO:tasks.workunit.client.0.smithi136.stderr:257604: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/3 2024-09-15T21:13:44.154 INFO:tasks.workunit.client.0.smithi136.stderr:257604: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=ce4892c7-6c6d-4b4f-8e6f-07a3e11133e5 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-15T21:13:44.154 INFO:tasks.workunit.client.0.smithi136.stderr:257604: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T21:13:44.155 INFO:tasks.workunit.client.0.smithi136.stderr:257604: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T21:13:44.155 INFO:tasks.workunit.client.0.smithi136.stderr:257604: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T21:13:44.155 INFO:tasks.workunit.client.0.smithi136.stderr:257604: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/3' 2024-09-15T21:13:44.155 INFO:tasks.workunit.client.0.smithi136.stderr:257604: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/3/journal' 2024-09-15T21:13:44.155 INFO:tasks.workunit.client.0.smithi136.stderr:257604: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-15T21:13:44.155 INFO:tasks.workunit.client.0.smithi136.stderr:257604: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-15T21:13:44.155 INFO:tasks.workunit.client.0.smithi136.stderr:257604: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-15T21:13:44.156 INFO:tasks.workunit.client.0.smithi136.stderr:257604: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-15T21:13:44.156 INFO:tasks.workunit.client.0.smithi136.stderr:257604: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-15T21:13:44.156 INFO:tasks.workunit.client.0.smithi136.stderr:257604: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-15T21:13:44.156 INFO:tasks.workunit.client.0.smithi136.stderr:257604: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-15T21:13:44.156 INFO:tasks.workunit.client.0.smithi136.stderr:257604: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T21:13:44.156 INFO:tasks.workunit.client.0.smithi136.stderr:257604: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52602 2024-09-15T21:13:44.156 INFO:tasks.workunit.client.0.smithi136.stderr:257604: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.52602/$cluster-$name.asok' 2024-09-15T21:13:44.156 INFO:tasks.workunit.client.0.smithi136.stderr:257604: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.52602/$cluster-$name.asok' 2024-09-15T21:13:44.156 INFO:tasks.workunit.client.0.smithi136.stderr:257604: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-15T21:13:44.156 INFO:tasks.workunit.client.0.smithi136.stderr:257604: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-15T21:13:44.156 INFO:tasks.workunit.client.0.smithi136.stderr:257604: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-15T21:13:44.156 INFO:tasks.workunit.client.0.smithi136.stderr:257604: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T21:13:44.156 INFO:tasks.workunit.client.0.smithi136.stderr:257604: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T21:13:44.157 INFO:tasks.workunit.client.0.smithi136.stderr:257604: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T21:13:44.157 INFO:tasks.workunit.client.0.smithi136.stderr:257604: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T21:13:44.157 INFO:tasks.workunit.client.0.smithi136.stderr:257604: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-15T21:13:44.157 INFO:tasks.workunit.client.0.smithi136.stderr:257604: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-15T21:13:44.157 INFO:tasks.workunit.client.0.smithi136.stderr:257604: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/3 2024-09-15T21:13:44.157 INFO:tasks.workunit.client.0.smithi136.stderr:257604: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.3 2024-09-15T21:13:44.157 INFO:tasks.workunit.client.0.smithi136.stderr:257604: start osd.3 2024-09-15T21:13:44.157 INFO:tasks.workunit.client.0.smithi136.stderr:257604: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 3 --fsid=ce4892c7-6c6d-4b4f-8e6f-07a3e11133e5 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/3 --osd-journal=td/osd-scrub-repair/3/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.52602/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T21:13:44.518 INFO:tasks.workunit.client.0.smithi136.stderr:2257606: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2024-09-15T21:13:44.518 INFO:tasks.workunit.client.0.smithi136.stderr:257606: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2024-09-15T21:13:44.518 INFO:tasks.workunit.client.0.smithi136.stderr:257606: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=5 2024-09-15T21:13:44.518 INFO:tasks.workunit.client.0.smithi136.stderr:257606: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2024-09-15T21:13:44.518 INFO:tasks.workunit.client.0.smithi136.stderr:257606: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 5 SOMETHING remove 2024-09-15T21:13:44.518 INFO:tasks.workunit.client.0.smithi136.stderr:257606: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-09-15T21:13:44.518 INFO:tasks.workunit.client.0.smithi136.stderr:257606: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2024-09-15T21:13:44.518 INFO:tasks.workunit.client.0.smithi136.stderr:257606: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=5 2024-09-15T21:13:44.518 INFO:tasks.workunit.client.0.smithi136.stderr:257606: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2024-09-15T21:13:44.518 INFO:tasks.workunit.client.0.smithi136.stderr:257606: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.5 2024-09-15T21:13:44.518 INFO:tasks.workunit.client.0.smithi136.stderr:257606: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-15T21:13:44.519 INFO:tasks.workunit.client.0.smithi136.stderr:257606: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-15T21:13:44.519 INFO:tasks.workunit.client.0.smithi136.stderr:257606: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-15T21:13:44.519 INFO:tasks.workunit.client.0.smithi136.stderr:257606: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-15T21:13:44.519 INFO:tasks.workunit.client.0.smithi136.stderr:257606: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-15T21:13:44.519 INFO:tasks.workunit.client.0.smithi136.stderr:257606: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-15T21:13:44.519 INFO:tasks.workunit.client.0.smithi136.stderr:257606: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 5 SOMETHING remove 2024-09-15T21:13:44.519 INFO:tasks.workunit.client.0.smithi136.stderr:257606: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-15T21:13:44.519 INFO:tasks.workunit.client.0.smithi136.stderr:257606: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-15T21:13:44.519 INFO:tasks.workunit.client.0.smithi136.stderr:257606: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=5 2024-09-15T21:13:44.519 INFO:tasks.workunit.client.0.smithi136.stderr:257606: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-15T21:13:44.519 INFO:tasks.workunit.client.0.smithi136.stderr:257606: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/5 2024-09-15T21:13:44.519 INFO:tasks.workunit.client.0.smithi136.stderr:257606: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/5 SOMETHING remove 2024-09-15T21:13:44.519 INFO:tasks.workunit.client.0.smithi136.stderr:257606: remove 1#2:eb822e21:::SOMETHING:head# 2024-09-15T21:13:44.519 INFO:tasks.workunit.client.0.smithi136.stderr:257606: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 5 2024-09-15T21:13:44.520 INFO:tasks.workunit.client.0.smithi136.stderr:257606: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2024-09-15T21:13:44.520 INFO:tasks.workunit.client.0.smithi136.stderr:257606: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-15T21:13:44.520 INFO:tasks.workunit.client.0.smithi136.stderr:257606: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=5 2024-09-15T21:13:44.520 INFO:tasks.workunit.client.0.smithi136.stderr:257606: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-15T21:13:44.520 INFO:tasks.workunit.client.0.smithi136.stderr:257606: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/5 2024-09-15T21:13:44.520 INFO:tasks.workunit.client.0.smithi136.stderr:257606: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=ce4892c7-6c6d-4b4f-8e6f-07a3e11133e5 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-15T21:13:44.520 INFO:tasks.workunit.client.0.smithi136.stderr:257606: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T21:13:44.520 INFO:tasks.workunit.client.0.smithi136.stderr:257606: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T21:13:44.520 INFO:tasks.workunit.client.0.smithi136.stderr:257606: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T21:13:44.521 INFO:tasks.workunit.client.0.smithi136.stderr:257606: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/5' 2024-09-15T21:13:44.521 INFO:tasks.workunit.client.0.smithi136.stderr:257606: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/5/journal' 2024-09-15T21:13:44.521 INFO:tasks.workunit.client.0.smithi136.stderr:257606: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-15T21:13:44.521 INFO:tasks.workunit.client.0.smithi136.stderr:257606: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-15T21:13:44.521 INFO:tasks.workunit.client.0.smithi136.stderr:257606: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-15T21:13:44.521 INFO:tasks.workunit.client.0.smithi136.stderr:257606: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-15T21:13:44.521 INFO:tasks.workunit.client.0.smithi136.stderr:257606: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-15T21:13:44.521 INFO:tasks.workunit.client.0.smithi136.stderr:257606: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-15T21:13:44.521 INFO:tasks.workunit.client.0.smithi136.stderr:257606: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-15T21:13:44.521 INFO:tasks.workunit.client.0.smithi136.stderr:257606: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T21:13:44.521 INFO:tasks.workunit.client.0.smithi136.stderr:257606: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52602 2024-09-15T21:13:44.521 INFO:tasks.workunit.client.0.smithi136.stderr:257606: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.52602/$cluster-$name.asok' 2024-09-15T21:13:44.521 INFO:tasks.workunit.client.0.smithi136.stderr:257606: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.52602/$cluster-$name.asok' 2024-09-15T21:13:44.521 INFO:tasks.workunit.client.0.smithi136.stderr:257606: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-15T21:13:44.522 INFO:tasks.workunit.client.0.smithi136.stderr:257606: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-15T21:13:44.522 INFO:tasks.workunit.client.0.smithi136.stderr:257606: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-15T21:13:44.522 INFO:tasks.workunit.client.0.smithi136.stderr:257606: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T21:13:44.522 INFO:tasks.workunit.client.0.smithi136.stderr:257606: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T21:13:44.522 INFO:tasks.workunit.client.0.smithi136.stderr:257606: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T21:13:44.522 INFO:tasks.workunit.client.0.smithi136.stderr:257606: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T21:13:44.522 INFO:tasks.workunit.client.0.smithi136.stderr:257606: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-15T21:13:44.522 INFO:tasks.workunit.client.0.smithi136.stderr:257606: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-15T21:13:44.522 INFO:tasks.workunit.client.0.smithi136.stderr:257606: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/5 2024-09-15T21:13:44.522 INFO:tasks.workunit.client.0.smithi136.stderr:257606: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.5 2024-09-15T21:13:44.522 INFO:tasks.workunit.client.0.smithi136.stderr:257606: start osd.5 2024-09-15T21:13:44.522 INFO:tasks.workunit.client.0.smithi136.stderr:257606: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 5 --fsid=ce4892c7-6c6d-4b4f-8e6f-07a3e11133e5 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/5 --osd-journal=td/osd-scrub-repair/5/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.52602/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T21:13:48.621 INFO:tasks.workunit.client.0.smithi136.stderr:257604: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/3/whoami 2024-09-15T21:13:48.621 INFO:tasks.workunit.client.0.smithi136.stderr:257604: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 3 = 3 ']' 2024-09-15T21:13:48.621 INFO:tasks.workunit.client.0.smithi136.stderr:257604: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-15T21:13:48.621 INFO:tasks.workunit.client.0.smithi136.stderr:257604: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-15T21:13:48.621 INFO:tasks.workunit.client.0.smithi136.stderr:257604: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-15T21:13:48.622 INFO:tasks.workunit.client.0.smithi136.stderr:257604: 2024-09-15T21:13:44.182+0000 7ff487844640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:13:48.622 INFO:tasks.workunit.client.0.smithi136.stderr:257604: 2024-09-15T21:13:44.185+0000 7ff487844640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:13:48.622 INFO:tasks.workunit.client.0.smithi136.stderr:257604: 2024-09-15T21:13:44.186+0000 7ff487844640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:13:48.622 INFO:tasks.workunit.client.0.smithi136.stderr:257604: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 3 2024-09-15T21:13:48.622 INFO:tasks.workunit.client.0.smithi136.stderr:257604: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-15T21:13:48.622 INFO:tasks.workunit.client.0.smithi136.stderr:257604: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=3 2024-09-15T21:13:48.622 INFO:tasks.workunit.client.0.smithi136.stderr:257604: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-15T21:13:48.622 INFO:tasks.workunit.client.0.smithi136.stderr:257604: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-15T21:13:48.622 INFO:tasks.workunit.client.0.smithi136.stderr:257604: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:13:48.622 INFO:tasks.workunit.client.0.smithi136.stderr:257604: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-15T21:13:48.622 INFO:tasks.workunit.client.0.smithi136.stderr:257604: 0 2024-09-15T21:13:48.622 INFO:tasks.workunit.client.0.smithi136.stderr:257604: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:13:48.622 INFO:tasks.workunit.client.0.smithi136.stderr:257604: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-15T21:13:48.623 INFO:tasks.workunit.client.0.smithi136.stderr:257604: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:13:48.623 INFO:tasks.workunit.client.0.smithi136.stderr:257604: 2024-09-15T21:13:45.311+0000 7ff487844640 -1 Falling back to public interface 2024-09-15T21:13:48.623 INFO:tasks.workunit.client.0.smithi136.stderr:257604: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:13:48.623 INFO:tasks.workunit.client.0.smithi136.stderr:257604: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:13:48.623 INFO:tasks.workunit.client.0.smithi136.stderr:257604: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-15T21:13:48.623 INFO:tasks.workunit.client.0.smithi136.stderr:257604: 1 2024-09-15T21:13:48.623 INFO:tasks.workunit.client.0.smithi136.stderr:257604: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:13:48.623 INFO:tasks.workunit.client.0.smithi136.stderr:257604: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-15T21:13:48.623 INFO:tasks.workunit.client.0.smithi136.stderr:257604: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:13:48.623 INFO:tasks.workunit.client.0.smithi136.stderr:257604: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:13:48.623 INFO:tasks.workunit.client.0.smithi136.stderr:257604: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:13:48.623 INFO:tasks.workunit.client.0.smithi136.stderr:257604: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-15T21:13:48.623 INFO:tasks.workunit.client.0.smithi136.stderr:257604: 2 2024-09-15T21:13:48.623 INFO:tasks.workunit.client.0.smithi136.stderr:257604: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:13:48.623 INFO:tasks.workunit.client.0.smithi136.stderr:257604: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-15T21:13:48.624 INFO:tasks.workunit.client.0.smithi136.stderr:257604: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:13:48.624 INFO:tasks.workunit.client.0.smithi136.stderr:257604: 2024-09-15T21:13:48.083+0000 7ff487844640 -1 osd.3 107 log_to_monitors true 2024-09-15T21:13:48.624 INFO:tasks.workunit.client.0.smithi136.stderr:257604: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:13:48.624 INFO:tasks.workunit.client.0.smithi136.stderr:257604: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:13:48.624 INFO:tasks.workunit.client.0.smithi136.stderr:257604: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-15T21:13:48.624 INFO:tasks.workunit.client.0.smithi136.stderr:257604: 3 2024-09-15T21:13:48.624 INFO:tasks.workunit.client.0.smithi136.stderr:257604: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:13:48.624 INFO:tasks.workunit.client.0.smithi136.stderr:257604: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-15T21:13:49.300 INFO:tasks.workunit.client.0.smithi136.stderr:257604: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helper57606: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/5/whoami 2024-09-15T21:13:49.300 INFO:tasks.workunit.client.0.smithi136.stderr:257606: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 5 = 5 ']' 2024-09-15T21:13:49.300 INFO:tasks.workunit.client.0.smithi136.stderr:257606: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-15T21:13:49.300 INFO:tasks.workunit.client.0.smithi136.stderr:257606: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-15T21:13:49.301 INFO:tasks.workunit.client.0.smithi136.stderr:257606: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-15T21:13:49.301 INFO:tasks.workunit.client.0.smithi136.stderr:257606: 2024-09-15T21:13:44.548+0000 7f06039c5640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:13:49.301 INFO:tasks.workunit.client.0.smithi136.stderr:257606: 2024-09-15T21:13:44.577+0000 7f06039c5640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:13:49.301 INFO:tasks.workunit.client.0.smithi136.stderr:257606: 2024-09-15T21:13:44.579+0000 7f06039c5640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:13:49.301 INFO:tasks.workunit.client.0.smithi136.stderr:257606: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 5 2024-09-15T21:13:49.301 INFO:tasks.workunit.client.0.smithi136.stderr:257606: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-15T21:13:49.301 INFO:tasks.workunit.client.0.smithi136.stderr:257606: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=5 2024-09-15T21:13:49.301 INFO:tasks.workunit.client.0.smithi136.stderr:257606: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-15T21:13:49.301 INFO:tasks.workunit.client.0.smithi136.stderr:257606: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-15T21:13:49.301 INFO:tasks.workunit.client.0.smithi136.stderr:257606: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:13:49.301 INFO:tasks.workunit.client.0.smithi136.stderr:257606: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-15T21:13:49.301 INFO:tasks.workunit.client.0.smithi136.stderr:257606: 0 2024-09-15T21:13:49.301 INFO:tasks.workunit.client.0.smithi136.stderr:257606: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:13:49.301 INFO:tasks.workunit.client.0.smithi136.stderr:257606: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-15T21:13:49.302 INFO:tasks.workunit.client.0.smithi136.stderr:257606: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:13:49.302 INFO:tasks.workunit.client.0.smithi136.stderr:257606: 2024-09-15T21:13:45.680+0000 7f06039c5640 -1 Falling back to public interface 2024-09-15T21:13:49.302 INFO:tasks.workunit.client.0.smithi136.stderr:257606: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:13:49.302 INFO:tasks.workunit.client.0.smithi136.stderr:257606: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:13:49.302 INFO:tasks.workunit.client.0.smithi136.stderr:257606: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-15T21:13:49.302 INFO:tasks.workunit.client.0.smithi136.stderr:257606: 1 2024-09-15T21:13:49.302 INFO:tasks.workunit.client.0.smithi136.stderr:257606: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:13:49.302 INFO:tasks.workunit.client.0.smithi136.stderr:257606: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-15T21:13:49.302 INFO:tasks.workunit.client.0.smithi136.stderr:257606: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:13:49.302 INFO:tasks.workunit.client.0.smithi136.stderr:257606: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:13:49.302 INFO:tasks.workunit.client.0.smithi136.stderr:257606: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:13:49.302 INFO:tasks.workunit.client.0.smithi136.stderr:257606: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-15T21:13:49.303 INFO:tasks.workunit.client.0.smithi136.stderr:257606: 2 2024-09-15T21:13:49.303 INFO:tasks.workunit.client.0.smithi136.stderr:257606: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:13:49.303 INFO:tasks.workunit.client.0.smithi136.stderr:257606: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-15T21:13:49.303 INFO:tasks.workunit.client.0.smithi136.stderr:257606: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:13:49.303 INFO:tasks.workunit.client.0.smithi136.stderr:257606: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:13:49.303 INFO:tasks.workunit.client.0.smithi136.stderr:257606: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:13:49.303 INFO:tasks.workunit.client.0.smithi136.stderr:257606: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-15T21:13:49.303 INFO:tasks.workunit.client.0.smithi136.stderr:257606: 3 2024-09-15T21:13:49.303 INFO:tasks.workunit.client.0.smithi136.stderr:257606: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:13:49.303 INFO:tasks.workunit.client.0.smithi136.stderr:257606: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-15T21:13:49.303 INFO:tasks.workunit.client.0.smithi136.stderr:257606: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:13:51.001 INFO:tasks.workunit.client.0.smithi136.stderr:257606: 2024-09-15T21:13:49.297+0000 7f06039c5640 -s.sh:976: wait_for_osd: sleep 1 2024-09-15T21:13:51.001 INFO:tasks.workunit.client.0.smithi136.stderr:257604: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:13:51.001 INFO:tasks.workunit.client.0.smithi136.stderr:257604: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:13:51.001 INFO:tasks.workunit.client.0.smithi136.stderr:257604: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-15T21:13:51.001 INFO:tasks.workunit.client.0.smithi136.stderr:257604: 4 2024-09-15T21:13:51.001 INFO:tasks.workunit.client.0.smithi136.stderr:257604: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:13:51.001 INFO:tasks.workunit.client.0.smithi136.stderr:257604: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-15T21:13:51.001 INFO:tasks.workunit.client.0.smithi136.stderr:257604: osd.3 up in weight 1 up_from 112 up_thru 112 down_at 108 last_clean_interval [73,107) [v2:127.0.0.1:6826/2799415284,v1:127.0.0.1:6827/2799415284] [v2:127.0.0.1:6828/2799415284,v1:127.0.0.1:6829/2799415284] exists,up 0e7c66ad-8619-49df-8643-12288ef1d7ba 2024-09-15T21:13:51.001 INFO:tasks.workunit.client.0.smithi136.stderr:257604: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-15T21:13:51.001 INFO:tasks.workunit.client.0.smithi136.stderr:257604: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-15T21:13:51.002 INFO:tasks.workunit.client.0.smithi136.stderr:257604: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-15T21:13:51.002 INFO:tasks.workunit.client.0.smithi136.stderr:257604: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2024-09-15T21:13:51.002 INFO:tasks.workunit.client.0.smithi136.stderr:257604: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-15T21:13:51.002 INFO:tasks.workunit.client.0.smithi136.stderr:257604: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-15T21:13:51.002 INFO:tasks.workunit.client.0.smithi136.stderr:257604: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-15T21:13:51.002 INFO:tasks.workunit.client.0.smithi136.stderr:257604: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-15T21:13:51.002 INFO:tasks.workunit.client.0.smithi136.stderr:257604: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-15T21:13:51.002 INFO:tasks.workunit.client.0.smithi136.stderr:257604: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-15T21:13:51.002 INFO:tasks.workunit.client.0.smithi136.stderr:257604: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-15T21:13:51.002 INFO:tasks.workunit.client.0.smithi136.stderr:257604: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-15T21:13:51.002 INFO:tasks.workunit.client.0.smithi136.stderr:257604: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-15T21:13:51.003 INFO:tasks.workunit.client.0.smithi136.stderr:257604: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-15T21:13:51.003 INFO:tasks.workunit.client.0.smithi136.stderr:257604: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-15T21:13:51.003 INFO:tasks.workunit.client.0.smithi136.stderr:257604: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-15T21:13:51.003 INFO:tasks.workunit.client.0.smithi136.stderr:257604: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-15T21:13:51.003 INFO:tasks.workunit.client.0.smithi136.stderr:257604: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-15T21:13:51.003 INFO:tasks.workunit.client.0.smithi136.stderr:257604: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-15T21:13:51.003 INFO:tasks.workunit.client.0.smithi136.stderr:257604: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-15T21:13:51.003 INFO:tasks.workunit.client.0.smithi136.stderr:257604: 1 2024-09-15T21:13:51.003 INFO:tasks.workunit.client.0.smithi136.stderr:257604: 2 2024-09-15T21:13:51.003 INFO:tasks.workunit.client.0.smithi136.stderr:257604: 3 2024-09-15T21:13:51.003 INFO:tasks.workunit.client.0.smithi136.stderr:257604: 4 2024-09-15T21:13:51.003 INFO:tasks.workunit.client.0.smithi136.stderr:257604: 5 2024-09-15T21:13:51.004 INFO:tasks.workunit.client.0.smithi136.stderr:257604: 6 2024-09-15T21:13:51.004 INFO:tasks.workunit.client.0.smithi136.stderr:257604: 7 2024-09-15T21:13:51.004 INFO:tasks.workunit.client.0.smithi136.stderr:257604: 8 2024-09-15T21:13:51.004 INFO:tasks.workunit.client.0.smithi136.stderr:257604: 9' 2024-09-15T21:13:51.004 INFO:tasks.workunit.client.0.smithi136.stderr:257604: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-15T21:13:51.004 INFO:tasks.workunit.client.0.smithi136.stderr:257604: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T21:13:51.004 INFO:tasks.workunit.client.0.smithi136.stderr:257604: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-15T21:13:51.004 INFO:tasks.workunit.client.0.smithi136.stderr:257604: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836585 2024-09-15T21:13:51.004 INFO:tasks.workunit.client.0.smithi136.stderr:257604: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836585 2024-09-15T21:13:51.005 INFO:tasks.workunit.client.0.smithi136.stderr:257604: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836585' 2024-09-15T21:13:51.005 INFO:tasks.workunit.client.0.smithi136.stderr:257604: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T21:13:51.497 INFO:tasks.workunit.client.0.smithi136.stderr:25761 osd.5 108 log_to_monitors true 2024-09-15T21:13:51.497 INFO:tasks.workunit.client.0.smithi136.stderr:257606: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:13:51.497 INFO:tasks.workunit.client.0.smithi136.stderr:257606: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:13:51.497 INFO:tasks.workunit.client.0.smithi136.stderr:257606: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-15T21:13:51.497 INFO:tasks.workunit.client.0.smithi136.stderr:257606: 4 2024-09-15T21:13:51.497 INFO:tasks.workunit.client.0.smithi136.stderr:257606: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:13:51.498 INFO:tasks.workunit.client.0.smithi136.stderr:257606: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-15T21:13:51.498 INFO:tasks.workunit.client.0.smithi136.stderr:257606: osd.5 up in weight 1 up_from 115 up_thru 106 down_at 108 last_clean_interval [106,108) [v2:127.0.0.1:6874/1972904372,v1:127.0.0.1:6875/1972904372] [v2:127.0.0.1:6876/1972904372,v1:127.0.0.1:6877/1972904372] exists,up 9918af07-0bbe-4711-bea1-402a4b5b6247 2024-09-15T21:13:51.498 INFO:tasks.workunit.client.0.smithi136.stderr:257606: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-15T21:13:51.498 INFO:tasks.workunit.client.0.smithi136.stderr:257606: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-15T21:13:51.498 INFO:tasks.workunit.client.0.smithi136.stderr:257606: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-15T21:13:51.498 INFO:tasks.workunit.client.0.smithi136.stderr:257606: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2024-09-15T21:13:51.498 INFO:tasks.workunit.client.0.smithi136.stderr:257606: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-15T21:13:51.498 INFO:tasks.workunit.client.0.smithi136.stderr:257606: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-15T21:13:51.498 INFO:tasks.workunit.client.0.smithi136.stderr:257606: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-15T21:13:51.498 INFO:tasks.workunit.client.0.smithi136.stderr:257606: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-15T21:13:51.498 INFO:tasks.workunit.client.0.smithi136.stderr:257606: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-15T21:13:51.498 INFO:tasks.workunit.client.0.smithi136.stderr:257606: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-15T21:13:51.498 INFO:tasks.workunit.client.0.smithi136.stderr:257606: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-15T21:13:51.498 INFO:tasks.workunit.client.0.smithi136.stderr:257606: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-15T21:13:51.498 INFO:tasks.workunit.client.0.smithi136.stderr:257606: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-15T21:13:51.499 INFO:tasks.workunit.client.0.smithi136.stderr:257606: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-15T21:13:51.499 INFO:tasks.workunit.client.0.smithi136.stderr:257606: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-15T21:13:51.499 INFO:tasks.workunit.client.0.smithi136.stderr:257606: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-15T21:13:51.499 INFO:tasks.workunit.client.0.smithi136.stderr:257606: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-15T21:13:51.499 INFO:tasks.workunit.client.0.smithi136.stderr:257606: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-15T21:13:51.499 INFO:tasks.workunit.client.0.smithi136.stderr:257606: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-15T21:13:51.499 INFO:tasks.workunit.client.0.smithi136.stderr:257606: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-15T21:13:51.499 INFO:tasks.workunit.client.0.smithi136.stderr:257606: 1 2024-09-15T21:13:51.499 INFO:tasks.workunit.client.0.smithi136.stderr:257606: 2 2024-09-15T21:13:51.499 INFO:tasks.workunit.client.0.smithi136.stderr:257606: 3 2024-09-15T21:13:51.499 INFO:tasks.workunit.client.0.smithi136.stderr:257606: 4 2024-09-15T21:13:51.499 INFO:tasks.workunit.client.0.smithi136.stderr:257606: 5 2024-09-15T21:13:51.499 INFO:tasks.workunit.client.0.smithi136.stderr:257606: 6 2024-09-15T21:13:51.499 INFO:tasks.workunit.client.0.smithi136.stderr:257606: 7 2024-09-15T21:13:51.500 INFO:tasks.workunit.client.0.smithi136.stderr:257606: 8 2024-09-15T21:13:51.500 INFO:tasks.workunit.client.0.smithi136.stderr:257606: 9' 2024-09-15T21:13:51.500 INFO:tasks.workunit.client.0.smithi136.stderr:257606: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-15T21:13:51.500 INFO:tasks.workunit.client.0.smithi136.stderr:257606: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T21:13:51.500 INFO:tasks.workunit.client.0.smithi136.stderr:257606: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-15T21:13:51.500 INFO:tasks.workunit.client.0.smithi136.stderr:257606: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836586 2024-09-15T21:13:51.500 INFO:tasks.workunit.client.0.smithi136.stderr:257606: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836586 2024-09-15T21:13:51.500 INFO:tasks.workunit.client.0.smithi136.stderr:257606: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836586' 2024-09-15T21:13:51.500 INFO:tasks.workunit.client.0.smithi136.stderr:257606: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T21:13:52.518 INFO:tasks.workunit.client.0.smithi136.stderr:25704: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-15T21:13:52.519 INFO:tasks.workunit.client.0.smithi136.stderr:257604: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=42949673061 2024-09-15T21:13:52.519 INFO:tasks.workunit.client.0.smithi136.stderr:257604: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 42949673061 2024-09-15T21:13:52.519 INFO:tasks.workunit.client.0.smithi136.stderr:257604: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836585 1-42949673061' 2024-09-15T21:13:52.519 INFO:tasks.workunit.client.0.smithi136.stderr:257604: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T21:13:52.519 INFO:tasks.workunit.client.0.smithi136.stderr:257604: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-15T21:13:52.519 INFO:tasks.workunit.client.0.smithi136.stderr:257604: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=64424509531 2024-09-15T21:13:52.519 INFO:tasks.workunit.client.0.smithi136.stderr:257604: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 64424509531 2024-09-15T21:13:52.519 INFO:tasks.workunit.client.0.smithi136.stderr:257604: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836585 1-42949673061 2-64424509531' 2024-09-15T21:13:52.519 INFO:tasks.workunit.client.0.smithi136.stderr:257604: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T21:13:52.519 INFO:tasks.workunit.client.0.smithi136.stderr:257604: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-15T21:13:52.519 INFO:tasks.workunit.client.0.smithi136.stderr:257604: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=481036337154 2024-09-15T21:13:52.520 INFO:tasks.workunit.client.0.smithi136.stderr:257604: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 481036337154 2024-09-15T21:13:52.520 INFO:tasks.workunit.client.0.smithi136.stderr:257604: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836585 1-42949673061 2-64424509531 3-481036337154' 2024-09-15T21:13:52.520 INFO:tasks.workunit.client.0.smithi136.stderr:257604: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T21:13:52.520 INFO:tasks.workunit.client.0.smithi136.stderr:257604: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-09-15T21:13:52.520 INFO:tasks.workunit.client.0.smithi136.stderr:257604: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=107374182480 2024-09-15T21:13:52.520 INFO:tasks.workunit.client.0.smithi136.stderr:257604: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 107374182480 2024-09-15T21:13:52.520 INFO:tasks.workunit.client.0.smithi136.stderr:257604: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836585 1-42949673061 2-64424509531 3-481036337154 4-107374182480' 2024-09-15T21:13:52.520 INFO:tasks.workunit.client.0.smithi136.stderr:257604: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T21:13:52.520 INFO:tasks.workunit.client.0.smithi136.stderr:257604: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-09-15T21:13:52.520 INFO:tasks.workunit.client.0.smithi136.stderr:257604: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=493921239042 2024-09-15T21:13:52.520 INFO:tasks.workunit.client.0.smithi136.stderr:257604: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 493921239042 2024-09-15T21:13:52.520 INFO:tasks.workunit.client.0.smithi136.stderr:257604: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836585 1-42949673061 2-64424509531 3-481036337154 4-107374182480 5-493921239042' 2024-09-15T21:13:52.520 INFO:tasks.workunit.client.0.smithi136.stderr:257604: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T21:13:52.520 INFO:tasks.workunit.client.0.smithi136.stderr:257604: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-09-15T21:13:52.521 INFO:tasks.workunit.client.0.smithi136.stderr:257604: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=150323855428 2024-09-15T21:13:52.521 INFO:tasks.workunit.client.0.smithi136.stderr:257604: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 150323855428 2024-09-15T21:13:52.521 INFO:tasks.workunit.client.0.smithi136.stderr:257604: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836585 1-42949673061 2-64424509531 3-481036337154 4-107374182480 5-493921239042 6-150323855428' 2024-09-15T21:13:52.521 INFO:tasks.workunit.client.0.smithi136.stderr:257604: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T21:13:52.521 INFO:tasks.workunit.client.0.smithi136.stderr:257604: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2024-09-15T21:13:52.738 INFO:tasks.workunit.client.0.smithi136.stderr:257604: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: f606: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-15T21:13:52.738 INFO:tasks.workunit.client.0.smithi136.stderr:257606: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=42949673062 2024-09-15T21:13:52.738 INFO:tasks.workunit.client.0.smithi136.stderr:257606: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 42949673062 2024-09-15T21:13:52.738 INFO:tasks.workunit.client.0.smithi136.stderr:257606: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836586 1-42949673062' 2024-09-15T21:13:52.739 INFO:tasks.workunit.client.0.smithi136.stderr:257606: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T21:13:52.739 INFO:tasks.workunit.client.0.smithi136.stderr:257606: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-15T21:13:52.739 INFO:tasks.workunit.client.0.smithi136.stderr:257606: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=64424509532 2024-09-15T21:13:52.739 INFO:tasks.workunit.client.0.smithi136.stderr:257606: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 64424509532 2024-09-15T21:13:52.739 INFO:tasks.workunit.client.0.smithi136.stderr:257606: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836586 1-42949673062 2-64424509532' 2024-09-15T21:13:52.739 INFO:tasks.workunit.client.0.smithi136.stderr:257606: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T21:13:52.739 INFO:tasks.workunit.client.0.smithi136.stderr:257606: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-15T21:13:52.739 INFO:tasks.workunit.client.0.smithi136.stderr:257606: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=481036337155 2024-09-15T21:13:52.739 INFO:tasks.workunit.client.0.smithi136.stderr:257606: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 481036337155 2024-09-15T21:13:52.739 INFO:tasks.workunit.client.0.smithi136.stderr:257606: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836586 1-42949673062 2-64424509532 3-481036337155' 2024-09-15T21:13:52.739 INFO:tasks.workunit.client.0.smithi136.stderr:257606: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T21:13:52.739 INFO:tasks.workunit.client.0.smithi136.stderr:257606: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-09-15T21:13:52.739 INFO:tasks.workunit.client.0.smithi136.stderr:257606: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=107374182481 2024-09-15T21:13:52.739 INFO:tasks.workunit.client.0.smithi136.stderr:257606: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 107374182481 2024-09-15T21:13:52.740 INFO:tasks.workunit.client.0.smithi136.stderr:257606: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836586 1-42949673062 2-64424509532 3-481036337155 4-107374182481' 2024-09-15T21:13:52.740 INFO:tasks.workunit.client.0.smithi136.stderr:257606: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T21:13:52.740 INFO:tasks.workunit.client.0.smithi136.stderr:257606: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-09-15T21:13:52.740 INFO:tasks.workunit.client.0.smithi136.stderr:257606: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=493921239043 2024-09-15T21:13:52.740 INFO:tasks.workunit.client.0.smithi136.stderr:257606: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 493921239043 2024-09-15T21:13:52.740 INFO:tasks.workunit.client.0.smithi136.stderr:257606: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836586 1-42949673062 2-64424509532 3-481036337155 4-107374182481 5-493921239043' 2024-09-15T21:13:52.740 INFO:tasks.workunit.client.0.smithi136.stderr:257606: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T21:13:52.740 INFO:tasks.workunit.client.0.smithi136.stderr:257606: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-09-15T21:13:52.740 INFO:tasks.workunit.client.0.smithi136.stderr:257606: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=150323855429 2024-09-15T21:13:52.740 INFO:tasks.workunit.client.0.smithi136.stderr:257606: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 150323855429 2024-09-15T21:13:52.740 INFO:tasks.workunit.client.0.smithi136.stderr:257606: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836586 1-42949673062 2-64424509532 3-481036337155 4-107374182481 5-493921239043 6-150323855429' 2024-09-15T21:13:52.740 INFO:tasks.workunit.client.0.smithi136.stderr:257606: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T21:13:52.740 INFO:tasks.workunit.client.0.smithi136.stderr:257606: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2024-09-15T21:13:53.031 INFO:tasks.workunit.client.0.smithi136.stderr:257606: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: lush_pg_stats: seq=171798691900 2024-09-15T21:13:53.031 INFO:tasks.workunit.client.0.smithi136.stderr:257604: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 171798691900 2024-09-15T21:13:53.031 INFO:tasks.workunit.client.0.smithi136.stderr:257604: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836585 1-42949673061 2-64424509531 3-481036337154 4-107374182480 5-493921239042 6-150323855428 7-171798691900' 2024-09-15T21:13:53.031 INFO:tasks.workunit.client.0.smithi136.stderr:257604: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T21:13:53.031 INFO:tasks.workunit.client.0.smithi136.stderr:257604: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2024-09-15T21:13:53.031 INFO:tasks.workunit.client.0.smithi136.stderr:257604: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=193273528374 2024-09-15T21:13:53.031 INFO:tasks.workunit.client.0.smithi136.stderr:257604: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 193273528374 2024-09-15T21:13:53.031 INFO:tasks.workunit.client.0.smithi136.stderr:257604: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836585 1-42949673061 2-64424509531 3-481036337154 4-107374182480 5-493921239042 6-150323855428 7-171798691900 8-193273528374' 2024-09-15T21:13:53.031 INFO:tasks.workunit.client.0.smithi136.stderr:257604: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T21:13:53.031 INFO:tasks.workunit.client.0.smithi136.stderr:257604: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2024-09-15T21:13:53.032 INFO:tasks.workunit.client.0.smithi136.stderr:257604: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=446676598792 2024-09-15T21:13:53.032 INFO:tasks.workunit.client.0.smithi136.stderr:257604: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 446676598792 2024-09-15T21:13:53.032 INFO:tasks.workunit.client.0.smithi136.stderr:257604: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836585 1-42949673061 2-64424509531 3-481036337154 4-107374182480 5-493921239042 6-150323855428 7-171798691900 8-193273528374 9-446676598792' 2024-09-15T21:13:53.032 INFO:tasks.workunit.client.0.smithi136.stderr:257604: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T21:13:53.032 INFO:tasks.workunit.client.0.smithi136.stderr:257604: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836585 2024-09-15T21:13:53.032 INFO:tasks.workunit.client.0.smithi136.stderr:257604: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T21:13:53.032 INFO:tasks.workunit.client.0.smithi136.stderr:257604: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-15T21:13:53.032 INFO:tasks.workunit.client.0.smithi136.stderr:257604: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836585 2024-09-15T21:13:53.032 INFO:tasks.workunit.client.0.smithi136.stderr:257604: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T21:13:53.032 INFO:tasks.workunit.client.0.smithi136.stderr:257604: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836585 2024-09-15T21:13:53.032 INFO:tasks.workunit.client.0.smithi136.stderr:257604: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836585' 2024-09-15T21:13:53.032 INFO:tasks.workunit.client.0.smithi136.stderr:257604: waiting osd.0 seq 21474836585 2024-09-15T21:13:53.032 INFO:tasks.workunit.client.0.smithi136.stderr:257604: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T21:13:53.033 INFO:tasks.workunit.client.0.smithi136.stderr:257604: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836586 -lt 21474836585 2024-09-15T21:13:53.033 INFO:tasks.workunit.client.0.smithi136.stderr:257604: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T21:13:53.033 INFO:tasks.workunit.client.0.smithi136.stderr:257604: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-42949673061 2024-09-15T21:13:53.033 INFO:tasks.workunit.client.0.smithi136.stderr:257604: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T21:13:53.033 INFO:tasks.workunit.client.0.smithi136.stderr:257604: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-15T21:13:53.033 INFO:tasks.workunit.client.0.smithi136.stderr:257604: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-42949673061 2024-09-15T21:13:53.033 INFO:tasks.workunit.client.0.smithi136.stderr:257604: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T21:13:53.033 INFO:tasks.workunit.client.0.smithi136.stderr:257604: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=42949673061 2024-09-15T21:13:53.033 INFO:tasks.workunit.client.0.smithi136.stderr:257604: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 42949673061' 2024-09-15T21:13:53.033 INFO:tasks.workunit.client.0.smithi136.stderr:257604: waiting osd.1 seq 42949673061 2024-09-15T21:13:53.258 INFO:tasks.workunit.client.0.smithi136.stderr:257604: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flushflush_pg_stats: seq=171798691901 2024-09-15T21:13:53.259 INFO:tasks.workunit.client.0.smithi136.stderr:257606: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 171798691901 2024-09-15T21:13:53.259 INFO:tasks.workunit.client.0.smithi136.stderr:257606: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836586 1-42949673062 2-64424509532 3-481036337155 4-107374182481 5-493921239043 6-150323855429 7-171798691901' 2024-09-15T21:13:53.259 INFO:tasks.workunit.client.0.smithi136.stderr:257606: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T21:13:53.259 INFO:tasks.workunit.client.0.smithi136.stderr:257606: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2024-09-15T21:13:53.259 INFO:tasks.workunit.client.0.smithi136.stderr:257606: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=193273528375 2024-09-15T21:13:53.260 INFO:tasks.workunit.client.0.smithi136.stderr:257606: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 193273528375 2024-09-15T21:13:53.260 INFO:tasks.workunit.client.0.smithi136.stderr:257606: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836586 1-42949673062 2-64424509532 3-481036337155 4-107374182481 5-493921239043 6-150323855429 7-171798691901 8-193273528375' 2024-09-15T21:13:53.260 INFO:tasks.workunit.client.0.smithi136.stderr:257606: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T21:13:53.260 INFO:tasks.workunit.client.0.smithi136.stderr:257606: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2024-09-15T21:13:53.260 INFO:tasks.workunit.client.0.smithi136.stderr:257606: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=446676598793 2024-09-15T21:13:53.260 INFO:tasks.workunit.client.0.smithi136.stderr:257606: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 446676598793 2024-09-15T21:13:53.261 INFO:tasks.workunit.client.0.smithi136.stderr:257606: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836586 1-42949673062 2-64424509532 3-481036337155 4-107374182481 5-493921239043 6-150323855429 7-171798691901 8-193273528375 9-446676598793' 2024-09-15T21:13:53.261 INFO:tasks.workunit.client.0.smithi136.stderr:257606: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T21:13:53.261 INFO:tasks.workunit.client.0.smithi136.stderr:257606: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836586 2024-09-15T21:13:53.261 INFO:tasks.workunit.client.0.smithi136.stderr:257606: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T21:13:53.261 INFO:tasks.workunit.client.0.smithi136.stderr:257606: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-15T21:13:53.262 INFO:tasks.workunit.client.0.smithi136.stderr:257606: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836586 2024-09-15T21:13:53.262 INFO:tasks.workunit.client.0.smithi136.stderr:257606: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T21:13:53.262 INFO:tasks.workunit.client.0.smithi136.stderr:257606: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836586 2024-09-15T21:13:53.262 INFO:tasks.workunit.client.0.smithi136.stderr:257606: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836586' 2024-09-15T21:13:53.263 INFO:tasks.workunit.client.0.smithi136.stderr:257606: waiting osd.0 seq 21474836586 2024-09-15T21:13:53.263 INFO:tasks.workunit.client.0.smithi136.stderr:257606: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T21:13:53.263 INFO:tasks.workunit.client.0.smithi136.stderr:257606: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836586 -lt 21474836586 2024-09-15T21:13:53.264 INFO:tasks.workunit.client.0.smithi136.stderr:257606: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T21:13:53.264 INFO:tasks.workunit.client.0.smithi136.stderr:257606: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-42949673062 2024-09-15T21:13:53.264 INFO:tasks.workunit.client.0.smithi136.stderr:257606: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T21:13:53.264 INFO:tasks.workunit.client.0.smithi136.stderr:257606: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-15T21:13:53.265 INFO:tasks.workunit.client.0.smithi136.stderr:257606: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-42949673062 2024-09-15T21:13:53.265 INFO:tasks.workunit.client.0.smithi136.stderr:257606: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T21:13:53.265 INFO:tasks.workunit.client.0.smithi136.stderr:257606: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=42949673062 2024-09-15T21:13:53.265 INFO:tasks.workunit.client.0.smithi136.stderr:257606: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 42949673062' 2024-09-15T21:13:53.265 INFO:tasks.workunit.client.0.smithi136.stderr:257606: waiting osd.1 seq 42949673062 2024-09-15T21:13:54.427 INFO:tasks.workunit.client.0.smithi136.stderr:257606: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flus_pg_stats: ceph osd last-stat-seq 1 2024-09-15T21:13:54.427 INFO:tasks.workunit.client.0.smithi136.stderr:257604: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 42949673062 -lt 42949673061 2024-09-15T21:13:54.427 INFO:tasks.workunit.client.0.smithi136.stderr:257604: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T21:13:54.427 INFO:tasks.workunit.client.0.smithi136.stderr:257604: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-64424509531 2024-09-15T21:13:54.428 INFO:tasks.workunit.client.0.smithi136.stderr:257604: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T21:13:54.428 INFO:tasks.workunit.client.0.smithi136.stderr:257604: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-15T21:13:54.428 INFO:tasks.workunit.client.0.smithi136.stderr:257604: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-64424509531 2024-09-15T21:13:54.428 INFO:tasks.workunit.client.0.smithi136.stderr:257604: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T21:13:54.428 INFO:tasks.workunit.client.0.smithi136.stderr:257604: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=64424509531 2024-09-15T21:13:54.428 INFO:tasks.workunit.client.0.smithi136.stderr:257604: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 64424509531' 2024-09-15T21:13:54.428 INFO:tasks.workunit.client.0.smithi136.stderr:257604: waiting osd.2 seq 64424509531 2024-09-15T21:13:54.429 INFO:tasks.workunit.client.0.smithi136.stderr:257604: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-15T21:13:54.429 INFO:tasks.workunit.client.0.smithi136.stderr:257604: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 64424509532 -lt 64424509531 2024-09-15T21:13:54.429 INFO:tasks.workunit.client.0.smithi136.stderr:257604: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T21:13:54.429 INFO:tasks.workunit.client.0.smithi136.stderr:257604: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 3-481036337154 2024-09-15T21:13:54.429 INFO:tasks.workunit.client.0.smithi136.stderr:257604: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T21:13:54.429 INFO:tasks.workunit.client.0.smithi136.stderr:257604: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=3 2024-09-15T21:13:54.429 INFO:tasks.workunit.client.0.smithi136.stderr:257604: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 3-481036337154 2024-09-15T21:13:54.430 INFO:tasks.workunit.client.0.smithi136.stderr:257604: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T21:13:54.430 INFO:tasks.workunit.client.0.smithi136.stderr:257604: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=481036337154 2024-09-15T21:13:54.430 INFO:tasks.workunit.client.0.smithi136.stderr:257604: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.3 seq 481036337154' 2024-09-15T21:13:54.430 INFO:tasks.workunit.client.0.smithi136.stderr:257604: waiting osd.3 seq 481036337154 2024-09-15T21:13:54.430 INFO:tasks.workunit.client.0.smithi136.stderr:257604: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-15T21:13:54.431 INFO:tasks.workunit.client.0.smithi136.stderr:257604: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 481036337156 -lt 481036337154 2024-09-15T21:13:54.431 INFO:tasks.workunit.client.0.smithi136.stderr:257604: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T21:13:54.431 INFO:tasks.workunit.client.0.smithi136.stderr:257604: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 4-107374182480 2024-09-15T21:13:54.431 INFO:tasks.workunit.client.0.smithi136.stderr:257604: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T21:13:54.431 INFO:tasks.workunit.client.0.smithi136.stderr:257604: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=4 2024-09-15T21:13:54.431 INFO:tasks.workunit.client.0.smithi136.stderr:257604: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 4-107374182480 2024-09-15T21:13:54.431 INFO:tasks.workunit.client.0.smithi136.stderr:257604: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T21:13:54.431 INFO:tasks.workunit.client.0.smithi136.stderr:257604: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=107374182480 2024-09-15T21:13:54.431 INFO:tasks.workunit.client.0.smithi136.stderr:257604: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.4 seq 107374182480' 2024-09-15T21:13:54.431 INFO:tasks.workunit.client.0.smithi136.stderr:257604: waiting osd.4 seq 107374182480 2024-09-15T21:13:54.431 INFO:tasks.workunit.client.0.smithi136.stderr:257604: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 4 2024-09-15T21:13:54.431 INFO:tasks.workunit.client.0.smithi136.stderr:257604: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 107374182481 -lt 107374182480 2024-09-15T21:13:54.431 INFO:tasks.workunit.client.0.smithi136.stderr:257604: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T21:13:54.432 INFO:tasks.workunit.client.0.smithi136.stderr:257604: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 5-493921239042 2024-09-15T21:13:54.565 INFO:tasks.workunit.client.0.smithi136.stderr:257604: //home/ubunh_pg_stats: ceph osd last-stat-seq 1 2024-09-15T21:13:54.565 INFO:tasks.workunit.client.0.smithi136.stderr:257606: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 42949673062 -lt 42949673062 2024-09-15T21:13:54.565 INFO:tasks.workunit.client.0.smithi136.stderr:257606: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T21:13:54.565 INFO:tasks.workunit.client.0.smithi136.stderr:257606: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-64424509532 2024-09-15T21:13:54.566 INFO:tasks.workunit.client.0.smithi136.stderr:257606: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T21:13:54.566 INFO:tasks.workunit.client.0.smithi136.stderr:257606: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-15T21:13:54.566 INFO:tasks.workunit.client.0.smithi136.stderr:257606: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-64424509532 2024-09-15T21:13:54.566 INFO:tasks.workunit.client.0.smithi136.stderr:257606: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T21:13:54.566 INFO:tasks.workunit.client.0.smithi136.stderr:257606: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=64424509532 2024-09-15T21:13:54.566 INFO:tasks.workunit.client.0.smithi136.stderr:257606: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 64424509532' 2024-09-15T21:13:54.566 INFO:tasks.workunit.client.0.smithi136.stderr:257606: waiting osd.2 seq 64424509532 2024-09-15T21:13:54.566 INFO:tasks.workunit.client.0.smithi136.stderr:257606: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-15T21:13:54.567 INFO:tasks.workunit.client.0.smithi136.stderr:257606: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 64424509532 -lt 64424509532 2024-09-15T21:13:54.567 INFO:tasks.workunit.client.0.smithi136.stderr:257606: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T21:13:54.567 INFO:tasks.workunit.client.0.smithi136.stderr:257606: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 3-481036337155 2024-09-15T21:13:54.567 INFO:tasks.workunit.client.0.smithi136.stderr:257606: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T21:13:54.567 INFO:tasks.workunit.client.0.smithi136.stderr:257606: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=3 2024-09-15T21:13:54.567 INFO:tasks.workunit.client.0.smithi136.stderr:257606: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 3-481036337155 2024-09-15T21:13:54.567 INFO:tasks.workunit.client.0.smithi136.stderr:257606: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T21:13:54.567 INFO:tasks.workunit.client.0.smithi136.stderr:257606: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=481036337155 2024-09-15T21:13:54.568 INFO:tasks.workunit.client.0.smithi136.stderr:257606: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.3 seq 481036337155' 2024-09-15T21:13:54.568 INFO:tasks.workunit.client.0.smithi136.stderr:257606: waiting osd.3 seq 481036337155 2024-09-15T21:13:54.568 INFO:tasks.workunit.client.0.smithi136.stderr:257606: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-15T21:13:54.568 INFO:tasks.workunit.client.0.smithi136.stderr:257606: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 481036337156 -lt 481036337155 2024-09-15T21:13:54.568 INFO:tasks.workunit.client.0.smithi136.stderr:257606: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T21:13:54.568 INFO:tasks.workunit.client.0.smithi136.stderr:257606: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 4-107374182481 2024-09-15T21:13:54.568 INFO:tasks.workunit.client.0.smithi136.stderr:257606: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T21:13:54.568 INFO:tasks.workunit.client.0.smithi136.stderr:257606: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=4 2024-09-15T21:13:54.569 INFO:tasks.workunit.client.0.smithi136.stderr:257606: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 4-107374182481 2024-09-15T21:13:54.569 INFO:tasks.workunit.client.0.smithi136.stderr:257606: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T21:13:54.569 INFO:tasks.workunit.client.0.smithi136.stderr:257606: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=107374182481 2024-09-15T21:13:54.569 INFO:tasks.workunit.client.0.smithi136.stderr:257606: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.4 seq 107374182481' 2024-09-15T21:13:54.569 INFO:tasks.workunit.client.0.smithi136.stderr:257606: waiting osd.4 seq 107374182481 2024-09-15T21:13:54.569 INFO:tasks.workunit.client.0.smithi136.stderr:257606: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 4 2024-09-15T21:13:54.569 INFO:tasks.workunit.client.0.smithi136.stderr:257606: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 107374182481 -lt 107374182481 2024-09-15T21:13:54.569 INFO:tasks.workunit.client.0.smithi136.stderr:257606: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T21:13:54.570 INFO:tasks.workunit.client.0.smithi136.stderr:257606: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 5-493921239043 2024-09-15T21:13:55.425 INFO:tasks.workunit.client.0.smithi136.stderr:257606: //home/ubutu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T21:13:55.426 INFO:tasks.workunit.client.0.smithi136.stderr:257604: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=5 2024-09-15T21:13:55.426 INFO:tasks.workunit.client.0.smithi136.stderr:257604: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 5-493921239042 2024-09-15T21:13:55.426 INFO:tasks.workunit.client.0.smithi136.stderr:257604: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T21:13:55.426 INFO:tasks.workunit.client.0.smithi136.stderr:257604: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=493921239042 2024-09-15T21:13:55.426 INFO:tasks.workunit.client.0.smithi136.stderr:257604: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.5 seq 493921239042' 2024-09-15T21:13:55.426 INFO:tasks.workunit.client.0.smithi136.stderr:257604: waiting osd.5 seq 493921239042 2024-09-15T21:13:55.426 INFO:tasks.workunit.client.0.smithi136.stderr:257604: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 5 2024-09-15T21:13:55.426 INFO:tasks.workunit.client.0.smithi136.stderr:257604: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 493921239043 -lt 493921239042 2024-09-15T21:13:55.426 INFO:tasks.workunit.client.0.smithi136.stderr:257604: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T21:13:55.427 INFO:tasks.workunit.client.0.smithi136.stderr:257604: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 6-150323855428 2024-09-15T21:13:55.427 INFO:tasks.workunit.client.0.smithi136.stderr:257604: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T21:13:55.427 INFO:tasks.workunit.client.0.smithi136.stderr:257604: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=6 2024-09-15T21:13:55.427 INFO:tasks.workunit.client.0.smithi136.stderr:257604: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 6-150323855428 2024-09-15T21:13:55.427 INFO:tasks.workunit.client.0.smithi136.stderr:257604: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T21:13:55.427 INFO:tasks.workunit.client.0.smithi136.stderr:257604: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=150323855428 2024-09-15T21:13:55.427 INFO:tasks.workunit.client.0.smithi136.stderr:257604: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.6 seq 150323855428' 2024-09-15T21:13:55.427 INFO:tasks.workunit.client.0.smithi136.stderr:257604: waiting osd.6 seq 150323855428 2024-09-15T21:13:55.427 INFO:tasks.workunit.client.0.smithi136.stderr:257604: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 6 2024-09-15T21:13:55.427 INFO:tasks.workunit.client.0.smithi136.stderr:257604: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 150323855429 -lt 150323855428 2024-09-15T21:13:55.427 INFO:tasks.workunit.client.0.smithi136.stderr:257604: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T21:13:55.427 INFO:tasks.workunit.client.0.smithi136.stderr:257604: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 7-171798691900 2024-09-15T21:13:55.427 INFO:tasks.workunit.client.0.smithi136.stderr:257604: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T21:13:55.427 INFO:tasks.workunit.client.0.smithi136.stderr:257604: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=7 2024-09-15T21:13:55.428 INFO:tasks.workunit.client.0.smithi136.stderr:257604: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 7-171798691900 2024-09-15T21:13:55.428 INFO:tasks.workunit.client.0.smithi136.stderr:257604: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T21:13:55.428 INFO:tasks.workunit.client.0.smithi136.stderr:257604: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=171798691900 2024-09-15T21:13:55.428 INFO:tasks.workunit.client.0.smithi136.stderr:257604: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.7 seq 171798691900' 2024-09-15T21:13:55.428 INFO:tasks.workunit.client.0.smithi136.stderr:257604: waiting osd.7 seq 171798691900 2024-09-15T21:13:55.428 INFO:tasks.workunit.client.0.smithi136.stderr:257604: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 7 2024-09-15T21:13:55.428 INFO:tasks.workunit.client.0.smithi136.stderr:257604: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 171798691901 -lt 171798691900 2024-09-15T21:13:55.428 INFO:tasks.workunit.client.0.smithi136.stderr:257604: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T21:13:55.428 INFO:tasks.workunit.client.0.smithi136.stderr:257604: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 8-193273528374 2024-09-15T21:13:55.428 INFO:tasks.workunit.client.0.smithi136.stderr:257604: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T21:13:55.429 INFO:tasks.workunit.client.0.smithi136.stderr:257604: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=8 2024-09-15T21:13:55.429 INFO:tasks.workunit.client.0.smithi136.stderr:257604: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 8-193273528374 2024-09-15T21:13:55.549 INFO:tasks.workunit.client.0.smithi136.stderr:257604: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_statntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T21:13:55.549 INFO:tasks.workunit.client.0.smithi136.stderr:257606: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=5 2024-09-15T21:13:55.549 INFO:tasks.workunit.client.0.smithi136.stderr:257606: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 5-493921239043 2024-09-15T21:13:55.549 INFO:tasks.workunit.client.0.smithi136.stderr:257606: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T21:13:55.549 INFO:tasks.workunit.client.0.smithi136.stderr:257606: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=493921239043 2024-09-15T21:13:55.549 INFO:tasks.workunit.client.0.smithi136.stderr:257606: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.5 seq 493921239043' 2024-09-15T21:13:55.549 INFO:tasks.workunit.client.0.smithi136.stderr:257606: waiting osd.5 seq 493921239043 2024-09-15T21:13:55.549 INFO:tasks.workunit.client.0.smithi136.stderr:257606: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 5 2024-09-15T21:13:55.549 INFO:tasks.workunit.client.0.smithi136.stderr:257606: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 493921239043 -lt 493921239043 2024-09-15T21:13:55.549 INFO:tasks.workunit.client.0.smithi136.stderr:257606: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T21:13:55.550 INFO:tasks.workunit.client.0.smithi136.stderr:257606: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 6-150323855429 2024-09-15T21:13:55.550 INFO:tasks.workunit.client.0.smithi136.stderr:257606: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T21:13:55.550 INFO:tasks.workunit.client.0.smithi136.stderr:257606: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=6 2024-09-15T21:13:55.550 INFO:tasks.workunit.client.0.smithi136.stderr:257606: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 6-150323855429 2024-09-15T21:13:55.550 INFO:tasks.workunit.client.0.smithi136.stderr:257606: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T21:13:55.550 INFO:tasks.workunit.client.0.smithi136.stderr:257606: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=150323855429 2024-09-15T21:13:55.550 INFO:tasks.workunit.client.0.smithi136.stderr:257606: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.6 seq 150323855429' 2024-09-15T21:13:55.550 INFO:tasks.workunit.client.0.smithi136.stderr:257606: waiting osd.6 seq 150323855429 2024-09-15T21:13:55.550 INFO:tasks.workunit.client.0.smithi136.stderr:257606: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 6 2024-09-15T21:13:55.550 INFO:tasks.workunit.client.0.smithi136.stderr:257606: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 150323855429 -lt 150323855429 2024-09-15T21:13:55.550 INFO:tasks.workunit.client.0.smithi136.stderr:257606: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T21:13:55.550 INFO:tasks.workunit.client.0.smithi136.stderr:257606: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 7-171798691901 2024-09-15T21:13:55.550 INFO:tasks.workunit.client.0.smithi136.stderr:257606: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T21:13:55.551 INFO:tasks.workunit.client.0.smithi136.stderr:257606: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=7 2024-09-15T21:13:55.551 INFO:tasks.workunit.client.0.smithi136.stderr:257606: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 7-171798691901 2024-09-15T21:13:55.551 INFO:tasks.workunit.client.0.smithi136.stderr:257606: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T21:13:55.551 INFO:tasks.workunit.client.0.smithi136.stderr:257606: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=171798691901 2024-09-15T21:13:55.551 INFO:tasks.workunit.client.0.smithi136.stderr:257606: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.7 seq 171798691901' 2024-09-15T21:13:55.551 INFO:tasks.workunit.client.0.smithi136.stderr:257606: waiting osd.7 seq 171798691901 2024-09-15T21:13:55.551 INFO:tasks.workunit.client.0.smithi136.stderr:257606: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 7 2024-09-15T21:13:55.551 INFO:tasks.workunit.client.0.smithi136.stderr:257606: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 171798691901 -lt 171798691901 2024-09-15T21:13:55.551 INFO:tasks.workunit.client.0.smithi136.stderr:257606: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T21:13:55.551 INFO:tasks.workunit.client.0.smithi136.stderr:257606: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 8-193273528375 2024-09-15T21:13:55.551 INFO:tasks.workunit.client.0.smithi136.stderr:257606: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T21:13:55.551 INFO:tasks.workunit.client.0.smithi136.stderr:257606: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=8 2024-09-15T21:13:55.551 INFO:tasks.workunit.client.0.smithi136.stderr:257606: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 8-193273528375 2024-09-15T21:13:57.122 INFO:tasks.workunit.client.0.smithi136.stderr:257606: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stas: cut -d - -f 2 2024-09-15T21:13:57.122 INFO:tasks.workunit.client.0.smithi136.stderr:257604: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=193273528374 2024-09-15T21:13:57.122 INFO:tasks.workunit.client.0.smithi136.stderr:257604: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.8 seq 193273528374' 2024-09-15T21:13:57.122 INFO:tasks.workunit.client.0.smithi136.stderr:257604: waiting osd.8 seq 193273528374 2024-09-15T21:13:57.122 INFO:tasks.workunit.client.0.smithi136.stderr:257604: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 8 2024-09-15T21:13:57.122 INFO:tasks.workunit.client.0.smithi136.stderr:257604: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 193273528375 -lt 193273528374 2024-09-15T21:13:57.122 INFO:tasks.workunit.client.0.smithi136.stderr:257604: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T21:13:57.122 INFO:tasks.workunit.client.0.smithi136.stderr:257604: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 9-446676598792 2024-09-15T21:13:57.122 INFO:tasks.workunit.client.0.smithi136.stderr:257604: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T21:13:57.122 INFO:tasks.workunit.client.0.smithi136.stderr:257604: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=9 2024-09-15T21:13:57.122 INFO:tasks.workunit.client.0.smithi136.stderr:257604: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 9-446676598792 2024-09-15T21:13:57.123 INFO:tasks.workunit.client.0.smithi136.stderr:257604: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T21:13:57.123 INFO:tasks.workunit.client.0.smithi136.stderr:257604: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=446676598792 2024-09-15T21:13:57.123 INFO:tasks.workunit.client.0.smithi136.stderr:257604: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.9 seq 446676598792' 2024-09-15T21:13:57.123 INFO:tasks.workunit.client.0.smithi136.stderr:257604: waiting osd.9 seq 446676598792 2024-09-15T21:13:57.123 INFO:tasks.workunit.client.0.smithi136.stderr:257604: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 9 2024-09-15T21:13:57.123 INFO:tasks.workunit.client.0.smithi136.stderr:257604: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 446676598793 -lt 446676598792 2024-09-15T21:13:57.123 INFO:tasks.workunit.client.0.smithi136.stderr:257604: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-15T21:13:57.123 INFO:tasks.workunit.client.0.smithi136.stderr:257604: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T21:13:57.123 INFO:tasks.workunit.client.0.smithi136.stderr:257604: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T21:13:57.123 INFO:tasks.workunit.client.0.smithi136.stderr:257604: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 5 == 0 2024-09-15T21:13:57.123 INFO:tasks.workunit.client.0.smithi136.stderr:257604: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-15T21:13:57.123 INFO:tasks.workunit.client.0.smithi136.stderr:257604: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-15T21:13:57.123 INFO:tasks.workunit.client.0.smithi136.stderr:257604: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-15T21:13:57.123 INFO:tasks.workunit.client.0.smithi136.stderr:257604: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T21:13:57.123 INFO:tasks.workunit.client.0.smithi136.stderr:257604: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T21:13:57.124 INFO:tasks.workunit.client.0.smithi136.stderr:257604: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T21:13:57.124 INFO:tasks.workunit.client.0.smithi136.stderr:257604: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T21:13:57.124 INFO:tasks.workunit.client.0.smithi136.stderr:257604: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=5 2024-09-15T21:13:57.124 INFO:tasks.workunit.client.0.smithi136.stderr:257604: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-15T21:13:57.124 INFO:tasks.workunit.client.0.smithi136.stderr:257604: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T21:13:57.124 INFO:tasks.workunit.client.0.smithi136.stderr:257604: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T21:13:57.124 INFO:tasks.workunit.client.0.smithi136.stderr:257604: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 = 5 2024-09-15T21:13:57.124 INFO:tasks.workunit.client.0.smithi136.stderr:257604: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-15T21:13:57.124 INFO:tasks.workunit.client.0.smithi136.stderr:257604: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-15T21:13:57.124 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: run_in_background: return 0 2024-09-15T21:13:57.124 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: wait_background: for pid in $pids 2024-09-15T21:13:57.124 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2175: wait_background: wait 257603 2024-09-15T21:13:57.267 INFO:tasks.workunit.client.0.smithi136.stderr:ts: cut -d - -f 2 2024-09-15T21:13:57.267 INFO:tasks.workunit.client.0.smithi136.stderr:257606: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=193273528375 2024-09-15T21:13:57.267 INFO:tasks.workunit.client.0.smithi136.stderr:257606: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.8 seq 193273528375' 2024-09-15T21:13:57.268 INFO:tasks.workunit.client.0.smithi136.stderr:257606: waiting osd.8 seq 193273528375 2024-09-15T21:13:57.268 INFO:tasks.workunit.client.0.smithi136.stderr:257606: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 8 2024-09-15T21:13:57.268 INFO:tasks.workunit.client.0.smithi136.stderr:257606: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 193273528375 -lt 193273528375 2024-09-15T21:13:57.268 INFO:tasks.workunit.client.0.smithi136.stderr:257606: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T21:13:57.268 INFO:tasks.workunit.client.0.smithi136.stderr:257606: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 9-446676598793 2024-09-15T21:13:57.268 INFO:tasks.workunit.client.0.smithi136.stderr:257606: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T21:13:57.268 INFO:tasks.workunit.client.0.smithi136.stderr:257606: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=9 2024-09-15T21:13:57.268 INFO:tasks.workunit.client.0.smithi136.stderr:257606: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 9-446676598793 2024-09-15T21:13:57.268 INFO:tasks.workunit.client.0.smithi136.stderr:257606: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T21:13:57.268 INFO:tasks.workunit.client.0.smithi136.stderr:257606: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=446676598793 2024-09-15T21:13:57.268 INFO:tasks.workunit.client.0.smithi136.stderr:257606: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.9 seq 446676598793' 2024-09-15T21:13:57.268 INFO:tasks.workunit.client.0.smithi136.stderr:257606: waiting osd.9 seq 446676598793 2024-09-15T21:13:57.268 INFO:tasks.workunit.client.0.smithi136.stderr:257606: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 9 2024-09-15T21:13:57.269 INFO:tasks.workunit.client.0.smithi136.stderr:257606: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 446676598793 -lt 446676598793 2024-09-15T21:13:57.269 INFO:tasks.workunit.client.0.smithi136.stderr:257606: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-15T21:13:57.269 INFO:tasks.workunit.client.0.smithi136.stderr:257606: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T21:13:57.269 INFO:tasks.workunit.client.0.smithi136.stderr:257606: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T21:13:57.269 INFO:tasks.workunit.client.0.smithi136.stderr:257606: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 5 == 0 2024-09-15T21:13:57.269 INFO:tasks.workunit.client.0.smithi136.stderr:257606: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-15T21:13:57.269 INFO:tasks.workunit.client.0.smithi136.stderr:257606: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-15T21:13:57.269 INFO:tasks.workunit.client.0.smithi136.stderr:257606: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-15T21:13:57.269 INFO:tasks.workunit.client.0.smithi136.stderr:257606: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T21:13:57.269 INFO:tasks.workunit.client.0.smithi136.stderr:257606: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T21:13:57.269 INFO:tasks.workunit.client.0.smithi136.stderr:257606: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T21:13:57.269 INFO:tasks.workunit.client.0.smithi136.stderr:257606: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T21:13:57.269 INFO:tasks.workunit.client.0.smithi136.stderr:257606: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=5 2024-09-15T21:13:57.270 INFO:tasks.workunit.client.0.smithi136.stderr:257606: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-15T21:13:57.270 INFO:tasks.workunit.client.0.smithi136.stderr:257606: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T21:13:57.270 INFO:tasks.workunit.client.0.smithi136.stderr:257606: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T21:13:57.270 INFO:tasks.workunit.client.0.smithi136.stderr:257606: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 = 5 2024-09-15T21:13:57.270 INFO:tasks.workunit.client.0.smithi136.stderr:257606: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-15T21:13:57.270 INFO:tasks.workunit.client.0.smithi136.stderr:257606: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-15T21:13:57.270 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: run_in_background: return 0 2024-09-15T21:13:57.270 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: wait_background: eval 'pids='\'''\''' 2024-09-15T21:13:57.270 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: wait_background: pids= 2024-09-15T21:13:57.270 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: wait_background: return 0 2024-09-15T21:13:57.270 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:197: corrupt_and_repair_two: return_code=0 2024-09-15T21:13:57.270 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:198: corrupt_and_repair_two: '[' 0 -ne 0 ']' 2024-09-15T21:13:57.270 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:203: corrupt_and_repair_two: get_pg ecpool SOMETHING 2024-09-15T21:13:57.270 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1084: get_pg: local poolname=ecpool 2024-09-15T21:13:57.271 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1085: get_pg: local objectname=SOMETHING 2024-09-15T21:13:57.271 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1087: get_pg: ceph --format json osd map ecpool SOMETHING 2024-09-15T21:13:57.271 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1087: get_pg: jq -r .pgid 2024-09-15T21:13:57.553 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:203: corrupt_and_repair_two: local pg=2.0 2024-09-15T21:13:57.553 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:204: corrupt_and_repair_two: repair 2.0 2024-09-15T21:13:57.553 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1846: repair: local pgid=2.0 2024-09-15T21:13:57.553 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1847: repair: get_last_scrub_stamp 2.0 2024-09-15T21:13:57.553 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2024-09-15T21:13:57.553 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-15T21:13:57.553 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-15T21:13:57.553 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-09-15T21:13:57.826 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1847: repair: local last_scrub=2024-09-15T21:13:15.713253+0000 2024-09-15T21:13:57.826 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1848: repair: ceph pg repair 2.0 2024-09-15T21:13:58.093 INFO:tasks.workunit.client.0.smithi136.stderr:instructing pg 2.0s0 on osd.3 to repair 2024-09-15T21:13:58.106 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1849: repair: wait_for_scrub 2.0 2024-09-15T21:13:15.713253+0000 2024-09-15T21:13:58.106 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: local pgid=2.0 2024-09-15T21:13:58.106 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2018: wait_for_scrub: local last_scrub=2024-09-15T21:13:15.713253+0000 2024-09-15T21:13:58.106 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2019: wait_for_scrub: local sname=last_scrub_stamp 2024-09-15T21:13:58.106 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i=0 )) 2024-09-15T21:13:58.106 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-15T21:13:58.106 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-09-15T21:13:58.106 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2024-09-15T21:13:58.106 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-15T21:13:58.107 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-15T21:13:58.107 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-09-15T21:13:58.383 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-15T21:13:15.713253+0000 '>' 2024-09-15T21:13:15.713253+0000 2024-09-15T21:13:58.384 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-15T21:13:59.385 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-15T21:13:59.385 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-15T21:13:59.385 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-09-15T21:13:59.386 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2024-09-15T21:13:59.386 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-15T21:13:59.386 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-15T21:13:59.386 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-09-15T21:13:59.668 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-15T21:13:15.713253+0000 '>' 2024-09-15T21:13:15.713253+0000 2024-09-15T21:13:59.669 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-15T21:14:00.670 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-15T21:14:00.670 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-15T21:14:00.671 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-09-15T21:14:00.671 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2024-09-15T21:14:00.671 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-15T21:14:00.671 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-15T21:14:00.671 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-09-15T21:14:00.957 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-15T21:13:58.251531+0000 '>' 2024-09-15T21:13:15.713253+0000 2024-09-15T21:14:00.957 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2023: wait_for_scrub: return 0 2024-09-15T21:14:00.957 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1850: repair: sleep 2 2024-09-15T21:14:02.960 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:208: corrupt_and_repair_two: pids= 2024-09-15T21:14:02.960 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:209: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 3 SOMETHING list-attrs 2024-09-15T21:14:02.960 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2135: run_in_background: local pid_variable=pids 2024-09-15T21:14:02.960 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2136: run_in_background: shift 2024-09-15T21:14:02.960 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2140: run_in_background: eval 'pids+=" 261026"' 2024-09-15T21:14:02.960 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2140: run_in_background: pids+=' 261026' 2024-09-15T21:14:02.960 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:210: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 5 SOMETHING list-attrs 2024-09-15T21:14:02.960 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2135: run_in_background: local pid_variable=pids 2024-09-15T21:14:02.960 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2136: run_in_background: shift 2024-09-15T21:14:02.961 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2140: run_in_background: eval 'pids+=" 261027"' 2024-09-15T21:14:02.961 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2140: run_in_background: pids+=' 261027' 2024-09-15T21:14:02.961 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:211: corrupt_and_repair_two: wait_background pids 2024-09-15T21:14:02.961 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: wait_background: pids=' 261026 261027' 2024-09-15T21:14:02.961 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: wait_background: return_code=0 2024-09-15T21:14:02.961 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: wait_background: for pid in $pids 2024-09-15T21:14:02.961 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2175: wait_background: wait 261026 2024-09-15T21:14:02.961 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: run_in_background: objectstore_tool td/osd-scrub-repair 3 SOMETHING list-attrs 2024-09-15T21:14:02.961 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: run_in_background: sed 's/^/261029: /' 2024-09-15T21:14:02.961 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: run_in_background: objectstore_tool td/osd-scrub-repair 5 SOMETHING list-attrs 2024-09-15T21:14:02.961 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: run_in_background: sed 's/^/261031: /' 2024-09-15T21:14:04.151 INFO:tasks.workunit.client.0.smithi136.stderr:261029: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2024-09-15T21:14:04.151 INFO:tasks.workunit.client.0.smithi136.stderr:261029: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2024-09-15T21:14:04.151 INFO:tasks.workunit.client.0.smithi136.stderr:261029: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=3 2024-09-15T21:14:04.151 INFO:tasks.workunit.client.0.smithi136.stderr:261029: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2024-09-15T21:14:04.151 INFO:tasks.workunit.client.0.smithi136.stderr:261029: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 3 SOMETHING list-attrs 2024-09-15T21:14:04.151 INFO:tasks.workunit.client.0.smithi136.stderr:261029: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-09-15T21:14:04.152 INFO:tasks.workunit.client.0.smithi136.stderr:261029: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2024-09-15T21:14:04.152 INFO:tasks.workunit.client.0.smithi136.stderr:261029: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=3 2024-09-15T21:14:04.152 INFO:tasks.workunit.client.0.smithi136.stderr:261029: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2024-09-15T21:14:04.152 INFO:tasks.workunit.client.0.smithi136.stderr:261029: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.3 2024-09-15T21:14:04.152 INFO:tasks.workunit.client.0.smithi136.stderr:261029: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-15T21:14:04.152 INFO:tasks.workunit.client.0.smithi136.stderr:261029: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-15T21:14:04.152 INFO:tasks.workunit.client.0.smithi136.stderr:261029: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-15T21:14:04.152 INFO:tasks.workunit.client.0.smithi136.stderr:261029: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-15T21:14:04.152 INFO:tasks.workunit.client.0.smithi136.stderr:261029: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-15T21:14:04.152 INFO:tasks.workunit.client.0.smithi136.stderr:261029: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-15T21:14:04.152 INFO:tasks.workunit.client.0.smithi136.stderr:261029: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 3 SOMETHING list-attrs 2024-09-15T21:14:04.152 INFO:tasks.workunit.client.0.smithi136.stderr:261029: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-15T21:14:04.152 INFO:tasks.workunit.client.0.smithi136.stderr:261029: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-15T21:14:04.152 INFO:tasks.workunit.client.0.smithi136.stderr:261029: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=3 2024-09-15T21:14:04.153 INFO:tasks.workunit.client.0.smithi136.stderr:261029: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-15T21:14:04.153 INFO:tasks.workunit.client.0.smithi136.stderr:261029: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/3 2024-09-15T21:14:04.153 INFO:tasks.workunit.client.0.smithi136.stderr:261029: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/3 SOMETHING list-attrs 2024-09-15T21:14:04.153 INFO:tasks.workunit.client.0.smithi136.stderr:261029: Error getting attr on : 2.0s0_head,0#-4:00000000:::scrub_2.0s0:head#, (61) No data available 2024-09-15T21:14:04.153 INFO:tasks.workunit.client.0.smithi136.stderr:261029: _ 2024-09-15T21:14:04.153 INFO:tasks.workunit.client.0.smithi136.stderr:261029: hinfo_key 2024-09-15T21:14:04.153 INFO:tasks.workunit.client.0.smithi136.stderr:261029: snapset 2024-09-15T21:14:04.153 INFO:tasks.workunit.client.0.smithi136.stderr:261029: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 3 2024-09-15T21:14:04.153 INFO:tasks.workunit.client.0.smithi136.stderr:261029: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2024-09-15T21:14:04.153 INFO:tasks.workunit.client.0.smithi136.stderr:261029: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-15T21:14:04.153 INFO:tasks.workunit.client.0.smithi136.stderr:261029: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=3 2024-09-15T21:14:04.153 INFO:tasks.workunit.client.0.smithi136.stderr:261029: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-15T21:14:04.153 INFO:tasks.workunit.client.0.smithi136.stderr:261029: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/3 2024-09-15T21:14:04.154 INFO:tasks.workunit.client.0.smithi136.stderr:261029: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=ce4892c7-6c6d-4b4f-8e6f-07a3e11133e5 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-15T21:14:04.154 INFO:tasks.workunit.client.0.smithi136.stderr:261029: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T21:14:04.154 INFO:tasks.workunit.client.0.smithi136.stderr:261029: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T21:14:04.154 INFO:tasks.workunit.client.0.smithi136.stderr:261029: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T21:14:04.154 INFO:tasks.workunit.client.0.smithi136.stderr:261029: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/3' 2024-09-15T21:14:04.154 INFO:tasks.workunit.client.0.smithi136.stderr:261029: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/3/journal' 2024-09-15T21:14:04.154 INFO:tasks.workunit.client.0.smithi136.stderr:261029: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-15T21:14:04.154 INFO:tasks.workunit.client.0.smithi136.stderr:261029: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-15T21:14:04.154 INFO:tasks.workunit.client.0.smithi136.stderr:261029: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-15T21:14:04.154 INFO:tasks.workunit.client.0.smithi136.stderr:261029: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-15T21:14:04.155 INFO:tasks.workunit.client.0.smithi136.stderr:261029: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-15T21:14:04.155 INFO:tasks.workunit.client.0.smithi136.stderr:261029: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-15T21:14:04.155 INFO:tasks.workunit.client.0.smithi136.stderr:261029: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-15T21:14:04.155 INFO:tasks.workunit.client.0.smithi136.stderr:261029: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T21:14:04.155 INFO:tasks.workunit.client.0.smithi136.stderr:261029: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52602 2024-09-15T21:14:04.155 INFO:tasks.workunit.client.0.smithi136.stderr:261029: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.52602/$cluster-$name.asok' 2024-09-15T21:14:04.155 INFO:tasks.workunit.client.0.smithi136.stderr:261029: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.52602/$cluster-$name.asok' 2024-09-15T21:14:04.155 INFO:tasks.workunit.client.0.smithi136.stderr:261029: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-15T21:14:04.155 INFO:tasks.workunit.client.0.smithi136.stderr:261029: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-15T21:14:04.155 INFO:tasks.workunit.client.0.smithi136.stderr:261029: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-15T21:14:04.155 INFO:tasks.workunit.client.0.smithi136.stderr:261029: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T21:14:04.155 INFO:tasks.workunit.client.0.smithi136.stderr:261029: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T21:14:04.155 INFO:tasks.workunit.client.0.smithi136.stderr:261029: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T21:14:04.155 INFO:tasks.workunit.client.0.smithi136.stderr:261029: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T21:14:04.155 INFO:tasks.workunit.client.0.smithi136.stderr:261029: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-15T21:14:04.156 INFO:tasks.workunit.client.0.smithi136.stderr:261029: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-15T21:14:04.156 INFO:tasks.workunit.client.0.smithi136.stderr:261029: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/3 2024-09-15T21:14:04.156 INFO:tasks.workunit.client.0.smithi136.stderr:261029: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.3 2024-09-15T21:14:04.156 INFO:tasks.workunit.client.0.smithi136.stderr:261029: start osd.3 2024-09-15T21:14:04.192 INFO:tasks.workunit.client.0.smithi136.stderr:261029: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 3 --fsid=ce4892c7-6c6d-4b4f-8e6f-07a3e11133e5 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/3 --osd-journal=td/osd-scrub-repair/3/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.52602/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-l261031: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2024-09-15T21:14:04.192 INFO:tasks.workunit.client.0.smithi136.stderr:261031: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2024-09-15T21:14:04.192 INFO:tasks.workunit.client.0.smithi136.stderr:261031: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=5 2024-09-15T21:14:04.192 INFO:tasks.workunit.client.0.smithi136.stderr:261031: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2024-09-15T21:14:04.193 INFO:tasks.workunit.client.0.smithi136.stderr:261031: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 5 SOMETHING list-attrs 2024-09-15T21:14:04.193 INFO:tasks.workunit.client.0.smithi136.stderr:261031: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-09-15T21:14:04.193 INFO:tasks.workunit.client.0.smithi136.stderr:261031: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2024-09-15T21:14:04.193 INFO:tasks.workunit.client.0.smithi136.stderr:261031: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=5 2024-09-15T21:14:04.193 INFO:tasks.workunit.client.0.smithi136.stderr:261031: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2024-09-15T21:14:04.193 INFO:tasks.workunit.client.0.smithi136.stderr:261031: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.5 2024-09-15T21:14:04.193 INFO:tasks.workunit.client.0.smithi136.stderr:261031: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-15T21:14:04.193 INFO:tasks.workunit.client.0.smithi136.stderr:261031: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-15T21:14:04.193 INFO:tasks.workunit.client.0.smithi136.stderr:261031: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-15T21:14:04.193 INFO:tasks.workunit.client.0.smithi136.stderr:261031: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-15T21:14:04.193 INFO:tasks.workunit.client.0.smithi136.stderr:261031: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-15T21:14:04.193 INFO:tasks.workunit.client.0.smithi136.stderr:261031: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-15T21:14:04.193 INFO:tasks.workunit.client.0.smithi136.stderr:261031: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 5 SOMETHING list-attrs 2024-09-15T21:14:04.193 INFO:tasks.workunit.client.0.smithi136.stderr:261031: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-15T21:14:04.194 INFO:tasks.workunit.client.0.smithi136.stderr:261031: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-15T21:14:04.194 INFO:tasks.workunit.client.0.smithi136.stderr:261031: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=5 2024-09-15T21:14:04.194 INFO:tasks.workunit.client.0.smithi136.stderr:261031: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-15T21:14:04.194 INFO:tasks.workunit.client.0.smithi136.stderr:261031: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/5 2024-09-15T21:14:04.194 INFO:tasks.workunit.client.0.smithi136.stderr:261031: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/5 SOMETHING list-attrs 2024-09-15T21:14:04.194 INFO:tasks.workunit.client.0.smithi136.stderr:261031: _ 2024-09-15T21:14:04.194 INFO:tasks.workunit.client.0.smithi136.stderr:261031: hinfo_key 2024-09-15T21:14:04.194 INFO:tasks.workunit.client.0.smithi136.stderr:261031: snapset 2024-09-15T21:14:04.194 INFO:tasks.workunit.client.0.smithi136.stderr:261031: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 5 2024-09-15T21:14:04.194 INFO:tasks.workunit.client.0.smithi136.stderr:261031: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2024-09-15T21:14:04.194 INFO:tasks.workunit.client.0.smithi136.stderr:261031: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-15T21:14:04.194 INFO:tasks.workunit.client.0.smithi136.stderr:261031: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=5 2024-09-15T21:14:04.194 INFO:tasks.workunit.client.0.smithi136.stderr:261031: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-15T21:14:04.194 INFO:tasks.workunit.client.0.smithi136.stderr:261031: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/5 2024-09-15T21:14:04.195 INFO:tasks.workunit.client.0.smithi136.stderr:261031: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=ce4892c7-6c6d-4b4f-8e6f-07a3e11133e5 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-15T21:14:04.195 INFO:tasks.workunit.client.0.smithi136.stderr:261031: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T21:14:04.195 INFO:tasks.workunit.client.0.smithi136.stderr:261031: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T21:14:04.195 INFO:tasks.workunit.client.0.smithi136.stderr:261031: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T21:14:04.196 INFO:tasks.workunit.client.0.smithi136.stderr:261031: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/5' 2024-09-15T21:14:04.196 INFO:tasks.workunit.client.0.smithi136.stderr:261031: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/5/journal' 2024-09-15T21:14:04.196 INFO:tasks.workunit.client.0.smithi136.stderr:261031: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-15T21:14:04.196 INFO:tasks.workunit.client.0.smithi136.stderr:261031: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-15T21:14:04.196 INFO:tasks.workunit.client.0.smithi136.stderr:261031: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-15T21:14:04.196 INFO:tasks.workunit.client.0.smithi136.stderr:261031: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-15T21:14:04.196 INFO:tasks.workunit.client.0.smithi136.stderr:261031: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-15T21:14:04.196 INFO:tasks.workunit.client.0.smithi136.stderr:261031: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-15T21:14:04.196 INFO:tasks.workunit.client.0.smithi136.stderr:261031: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-15T21:14:04.196 INFO:tasks.workunit.client.0.smithi136.stderr:261031: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T21:14:04.196 INFO:tasks.workunit.client.0.smithi136.stderr:261031: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52602 2024-09-15T21:14:04.196 INFO:tasks.workunit.client.0.smithi136.stderr:261031: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.52602/$cluster-$name.asok' 2024-09-15T21:14:04.196 INFO:tasks.workunit.client.0.smithi136.stderr:261031: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.52602/$cluster-$name.asok' 2024-09-15T21:14:04.197 INFO:tasks.workunit.client.0.smithi136.stderr:261031: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-15T21:14:04.197 INFO:tasks.workunit.client.0.smithi136.stderr:261031: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-15T21:14:04.197 INFO:tasks.workunit.client.0.smithi136.stderr:261031: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-15T21:14:04.197 INFO:tasks.workunit.client.0.smithi136.stderr:261031: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T21:14:04.197 INFO:tasks.workunit.client.0.smithi136.stderr:261031: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T21:14:04.197 INFO:tasks.workunit.client.0.smithi136.stderr:261031: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T21:14:04.197 INFO:tasks.workunit.client.0.smithi136.stderr:261031: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T21:14:04.197 INFO:tasks.workunit.client.0.smithi136.stderr:261031: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-15T21:14:04.197 INFO:tasks.workunit.client.0.smithi136.stderr:261031: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-15T21:14:04.197 INFO:tasks.workunit.client.0.smithi136.stderr:261031: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/5 2024-09-15T21:14:04.197 INFO:tasks.workunit.client.0.smithi136.stderr:261031: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.5 2024-09-15T21:14:04.197 INFO:tasks.workunit.client.0.smithi136.stderr:261031: start osd.5 2024-09-15T21:14:08.633 INFO:tasks.workunit.client.0.smithi136.stderr:261031: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 5 --fsid=ce4892c7-6c6d-4b4f-8e6f-07a3e11133e5 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/5 --osd-journal=td/osd-scrub-repair/5/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.52602/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recoen=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T21:14:08.633 INFO:tasks.workunit.client.0.smithi136.stderr:261029: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/3/whoami 2024-09-15T21:14:08.634 INFO:tasks.workunit.client.0.smithi136.stderr:261029: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 3 = 3 ']' 2024-09-15T21:14:08.634 INFO:tasks.workunit.client.0.smithi136.stderr:261029: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-15T21:14:08.634 INFO:tasks.workunit.client.0.smithi136.stderr:261029: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-15T21:14:08.634 INFO:tasks.workunit.client.0.smithi136.stderr:261029: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-15T21:14:08.634 INFO:tasks.workunit.client.0.smithi136.stderr:261029: 2024-09-15T21:14:04.181+0000 7f98d1e0f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:14:08.634 INFO:tasks.workunit.client.0.smithi136.stderr:261029: 2024-09-15T21:14:04.243+0000 7f98d1e0f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:14:08.634 INFO:tasks.workunit.client.0.smithi136.stderr:261029: 2024-09-15T21:14:04.244+0000 7f98d1e0f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:14:08.634 INFO:tasks.workunit.client.0.smithi136.stderr:261029: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 3 2024-09-15T21:14:08.634 INFO:tasks.workunit.client.0.smithi136.stderr:261029: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-15T21:14:08.634 INFO:tasks.workunit.client.0.smithi136.stderr:261029: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=3 2024-09-15T21:14:08.634 INFO:tasks.workunit.client.0.smithi136.stderr:261029: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-15T21:14:08.635 INFO:tasks.workunit.client.0.smithi136.stderr:261029: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-15T21:14:08.635 INFO:tasks.workunit.client.0.smithi136.stderr:261029: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:14:08.635 INFO:tasks.workunit.client.0.smithi136.stderr:261029: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-15T21:14:08.635 INFO:tasks.workunit.client.0.smithi136.stderr:261029: 0 2024-09-15T21:14:08.635 INFO:tasks.workunit.client.0.smithi136.stderr:261029: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:14:08.635 INFO:tasks.workunit.client.0.smithi136.stderr:261029: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-15T21:14:08.635 INFO:tasks.workunit.client.0.smithi136.stderr:261029: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:14:08.635 INFO:tasks.workunit.client.0.smithi136.stderr:261029: 2024-09-15T21:14:05.588+0000 7f98d1e0f640 -1 Falling back to public interface 2024-09-15T21:14:08.635 INFO:tasks.workunit.client.0.smithi136.stderr:261029: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:14:08.635 INFO:tasks.workunit.client.0.smithi136.stderr:261029: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:14:08.635 INFO:tasks.workunit.client.0.smithi136.stderr:261029: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-15T21:14:08.635 INFO:tasks.workunit.client.0.smithi136.stderr:261029: 1 2024-09-15T21:14:08.635 INFO:tasks.workunit.client.0.smithi136.stderr:261029: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:14:08.636 INFO:tasks.workunit.client.0.smithi136.stderr:261029: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-15T21:14:08.636 INFO:tasks.workunit.client.0.smithi136.stderr:261029: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:14:08.636 INFO:tasks.workunit.client.0.smithi136.stderr:261029: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:14:08.636 INFO:tasks.workunit.client.0.smithi136.stderr:261029: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:14:08.636 INFO:tasks.workunit.client.0.smithi136.stderr:261029: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-15T21:14:08.636 INFO:tasks.workunit.client.0.smithi136.stderr:261029: 2 2024-09-15T21:14:08.636 INFO:tasks.workunit.client.0.smithi136.stderr:261029: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:14:08.636 INFO:tasks.workunit.client.0.smithi136.stderr:261029: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-15T21:14:08.636 INFO:tasks.workunit.client.0.smithi136.stderr:261029: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:14:08.636 INFO:tasks.workunit.client.0.smithi136.stderr:261029: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:14:08.636 INFO:tasks.workunit.client.0.smithi136.stderr:261029: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:14:08.636 INFO:tasks.workunit.client.0.smithi136.stderr:261029: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-15T21:14:08.636 INFO:tasks.workunit.client.0.smithi136.stderr:261029: 3 2024-09-15T21:14:08.636 INFO:tasks.workunit.client.0.smithi136.stderr:261029: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:14:08.637 INFO:tasks.workunit.client.0.smithi136.stderr:261029: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-15T21:14:09.659 INFO:tasks.workunit.client.0.smithi136.stderr:261029: /home/ubuntu/cephtest/clone.client.very_ops 2024-09-15T21:14:09.659 INFO:tasks.workunit.client.0.smithi136.stderr:261031: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/5/whoami 2024-09-15T21:14:09.659 INFO:tasks.workunit.client.0.smithi136.stderr:261031: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 5 = 5 ']' 2024-09-15T21:14:09.659 INFO:tasks.workunit.client.0.smithi136.stderr:261031: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-15T21:14:09.659 INFO:tasks.workunit.client.0.smithi136.stderr:261031: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-15T21:14:09.659 INFO:tasks.workunit.client.0.smithi136.stderr:261031: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-15T21:14:09.659 INFO:tasks.workunit.client.0.smithi136.stderr:261031: 2024-09-15T21:14:04.222+0000 7f915d8d6640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:14:09.659 INFO:tasks.workunit.client.0.smithi136.stderr:261031: 2024-09-15T21:14:04.243+0000 7f915d8d6640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:14:09.660 INFO:tasks.workunit.client.0.smithi136.stderr:261031: 2024-09-15T21:14:04.244+0000 7f915d8d6640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:14:09.660 INFO:tasks.workunit.client.0.smithi136.stderr:261031: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 5 2024-09-15T21:14:09.660 INFO:tasks.workunit.client.0.smithi136.stderr:261031: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-15T21:14:09.660 INFO:tasks.workunit.client.0.smithi136.stderr:261031: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=5 2024-09-15T21:14:09.660 INFO:tasks.workunit.client.0.smithi136.stderr:261031: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-15T21:14:09.660 INFO:tasks.workunit.client.0.smithi136.stderr:261031: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-15T21:14:09.660 INFO:tasks.workunit.client.0.smithi136.stderr:261031: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:14:09.660 INFO:tasks.workunit.client.0.smithi136.stderr:261031: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-15T21:14:09.660 INFO:tasks.workunit.client.0.smithi136.stderr:261031: 0 2024-09-15T21:14:09.660 INFO:tasks.workunit.client.0.smithi136.stderr:261031: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:14:09.660 INFO:tasks.workunit.client.0.smithi136.stderr:261031: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-15T21:14:09.660 INFO:tasks.workunit.client.0.smithi136.stderr:261031: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:14:09.660 INFO:tasks.workunit.client.0.smithi136.stderr:261031: 2024-09-15T21:14:05.620+0000 7f915d8d6640 -1 Falling back to public interface 2024-09-15T21:14:09.660 INFO:tasks.workunit.client.0.smithi136.stderr:261031: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:14:09.661 INFO:tasks.workunit.client.0.smithi136.stderr:261031: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:14:09.661 INFO:tasks.workunit.client.0.smithi136.stderr:261031: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-15T21:14:09.661 INFO:tasks.workunit.client.0.smithi136.stderr:261031: 1 2024-09-15T21:14:09.661 INFO:tasks.workunit.client.0.smithi136.stderr:261031: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:14:09.661 INFO:tasks.workunit.client.0.smithi136.stderr:261031: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-15T21:14:09.661 INFO:tasks.workunit.client.0.smithi136.stderr:261031: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:14:09.661 INFO:tasks.workunit.client.0.smithi136.stderr:261031: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:14:09.661 INFO:tasks.workunit.client.0.smithi136.stderr:261031: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:14:09.661 INFO:tasks.workunit.client.0.smithi136.stderr:261031: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-15T21:14:09.661 INFO:tasks.workunit.client.0.smithi136.stderr:261031: 2 2024-09-15T21:14:09.661 INFO:tasks.workunit.client.0.smithi136.stderr:261031: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:14:09.661 INFO:tasks.workunit.client.0.smithi136.stderr:261031: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-15T21:14:09.661 INFO:tasks.workunit.client.0.smithi136.stderr:261031: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:14:09.662 INFO:tasks.workunit.client.0.smithi136.stderr:261031: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:14:09.662 INFO:tasks.workunit.client.0.smithi136.stderr:261031: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:14:09.662 INFO:tasks.workunit.client.0.smithi136.stderr:261031: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-15T21:14:09.662 INFO:tasks.workunit.client.0.smithi136.stderr:261031: 3 2024-09-15T21:14:09.662 INFO:tasks.workunit.client.0.smithi136.stderr:261031: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:14:09.662 INFO:tasks.workunit.client.0.smithi136.stderr:261031: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-15T21:14:09.662 INFO:tasks.workunit.client.0.smithi136.stderr:261031: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:14:10.924 INFO:tasks.workunit.client.0.smithi136.stderr:261031: /home/ubuntu/cephtest/clone.clien0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:14:10.924 INFO:tasks.workunit.client.0.smithi136.stderr:261029: 2024-09-15T21:14:08.918+0000 7f98d1e0f640 -1 osd.3 117 log_to_monitors true 2024-09-15T21:14:10.924 INFO:tasks.workunit.client.0.smithi136.stderr:261029: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:14:10.924 INFO:tasks.workunit.client.0.smithi136.stderr:261029: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:14:10.925 INFO:tasks.workunit.client.0.smithi136.stderr:261029: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-15T21:14:10.925 INFO:tasks.workunit.client.0.smithi136.stderr:261029: 4 2024-09-15T21:14:10.925 INFO:tasks.workunit.client.0.smithi136.stderr:261029: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:14:10.925 INFO:tasks.workunit.client.0.smithi136.stderr:261029: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-15T21:14:10.925 INFO:tasks.workunit.client.0.smithi136.stderr:261029: osd.3 up in weight 1 up_from 121 up_thru 115 down_at 118 last_clean_interval [112,117) [v2:127.0.0.1:6826/3382191673,v1:127.0.0.1:6827/3382191673] [v2:127.0.0.1:6828/3382191673,v1:127.0.0.1:6829/3382191673] exists,up 0e7c66ad-8619-49df-8643-12288ef1d7ba 2024-09-15T21:14:10.925 INFO:tasks.workunit.client.0.smithi136.stderr:261029: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-15T21:14:10.925 INFO:tasks.workunit.client.0.smithi136.stderr:261029: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-15T21:14:10.925 INFO:tasks.workunit.client.0.smithi136.stderr:261029: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-15T21:14:10.925 INFO:tasks.workunit.client.0.smithi136.stderr:261029: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2024-09-15T21:14:10.925 INFO:tasks.workunit.client.0.smithi136.stderr:261029: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-15T21:14:10.925 INFO:tasks.workunit.client.0.smithi136.stderr:261029: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-15T21:14:10.925 INFO:tasks.workunit.client.0.smithi136.stderr:261029: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-15T21:14:10.925 INFO:tasks.workunit.client.0.smithi136.stderr:261029: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-15T21:14:10.926 INFO:tasks.workunit.client.0.smithi136.stderr:261029: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-15T21:14:10.926 INFO:tasks.workunit.client.0.smithi136.stderr:261029: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-15T21:14:10.926 INFO:tasks.workunit.client.0.smithi136.stderr:261029: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-15T21:14:10.926 INFO:tasks.workunit.client.0.smithi136.stderr:261029: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-15T21:14:10.926 INFO:tasks.workunit.client.0.smithi136.stderr:261029: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-15T21:14:10.926 INFO:tasks.workunit.client.0.smithi136.stderr:261029: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-15T21:14:10.926 INFO:tasks.workunit.client.0.smithi136.stderr:261029: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-15T21:14:10.926 INFO:tasks.workunit.client.0.smithi136.stderr:261029: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-15T21:14:10.926 INFO:tasks.workunit.client.0.smithi136.stderr:261029: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-15T21:14:10.926 INFO:tasks.workunit.client.0.smithi136.stderr:261029: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-15T21:14:10.926 INFO:tasks.workunit.client.0.smithi136.stderr:261029: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-15T21:14:10.926 INFO:tasks.workunit.client.0.smithi136.stderr:261029: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-15T21:14:10.926 INFO:tasks.workunit.client.0.smithi136.stderr:261029: 1 2024-09-15T21:14:10.926 INFO:tasks.workunit.client.0.smithi136.stderr:261029: 2 2024-09-15T21:14:10.927 INFO:tasks.workunit.client.0.smithi136.stderr:261029: 3 2024-09-15T21:14:10.927 INFO:tasks.workunit.client.0.smithi136.stderr:261029: 4 2024-09-15T21:14:10.927 INFO:tasks.workunit.client.0.smithi136.stderr:261029: 5 2024-09-15T21:14:10.927 INFO:tasks.workunit.client.0.smithi136.stderr:261029: 6 2024-09-15T21:14:10.927 INFO:tasks.workunit.client.0.smithi136.stderr:261029: 7 2024-09-15T21:14:10.927 INFO:tasks.workunit.client.0.smithi136.stderr:261029: 8 2024-09-15T21:14:10.927 INFO:tasks.workunit.client.0.smithi136.stderr:261029: 9' 2024-09-15T21:14:10.927 INFO:tasks.workunit.client.0.smithi136.stderr:261029: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-15T21:14:10.927 INFO:tasks.workunit.client.0.smithi136.stderr:261029: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T21:14:10.927 INFO:tasks.workunit.client.0.smithi136.stderr:261029: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-15T21:14:10.927 INFO:tasks.workunit.client.0.smithi136.stderr:261029: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836591 2024-09-15T21:14:10.927 INFO:tasks.workunit.client.0.smithi136.stderr:261029: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836591 2024-09-15T21:14:10.927 INFO:tasks.workunit.client.0.smithi136.stderr:261029: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836591' 2024-09-15T21:14:12.195 INFO:tasks.workunit.client.0.smithi136.stderr:261029t.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:14:12.195 INFO:tasks.workunit.client.0.smithi136.stderr:261031: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:14:12.195 INFO:tasks.workunit.client.0.smithi136.stderr:261031: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-15T21:14:12.195 INFO:tasks.workunit.client.0.smithi136.stderr:261031: 4 2024-09-15T21:14:12.195 INFO:tasks.workunit.client.0.smithi136.stderr:261031: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:14:12.195 INFO:tasks.workunit.client.0.smithi136.stderr:261031: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-15T21:14:12.195 INFO:tasks.workunit.client.0.smithi136.stderr:261031: 2024-09-15T21:14:10.004+0000 7f915d8d6640 -1 osd.5 117 log_to_monitors true 2024-09-15T21:14:12.195 INFO:tasks.workunit.client.0.smithi136.stderr:261031: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:14:12.195 INFO:tasks.workunit.client.0.smithi136.stderr:261031: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:14:12.195 INFO:tasks.workunit.client.0.smithi136.stderr:261031: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:14:12.196 INFO:tasks.workunit.client.0.smithi136.stderr:261031: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-15T21:14:12.196 INFO:tasks.workunit.client.0.smithi136.stderr:261031: 5 2024-09-15T21:14:12.196 INFO:tasks.workunit.client.0.smithi136.stderr:261031: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:14:12.196 INFO:tasks.workunit.client.0.smithi136.stderr:261031: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-15T21:14:12.196 INFO:tasks.workunit.client.0.smithi136.stderr:261031: osd.5 up in weight 1 up_from 124 up_thru 115 down_at 118 last_clean_interval [115,117) [v2:127.0.0.1:6874/414460184,v1:127.0.0.1:6875/414460184] [v2:127.0.0.1:6876/414460184,v1:127.0.0.1:6877/414460184] exists,up 9918af07-0bbe-4711-bea1-402a4b5b6247 2024-09-15T21:14:12.196 INFO:tasks.workunit.client.0.smithi136.stderr:261031: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-15T21:14:12.196 INFO:tasks.workunit.client.0.smithi136.stderr:261031: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-15T21:14:12.196 INFO:tasks.workunit.client.0.smithi136.stderr:261031: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-15T21:14:12.196 INFO:tasks.workunit.client.0.smithi136.stderr:261031: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2024-09-15T21:14:12.196 INFO:tasks.workunit.client.0.smithi136.stderr:261031: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-15T21:14:12.196 INFO:tasks.workunit.client.0.smithi136.stderr:261031: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-15T21:14:12.197 INFO:tasks.workunit.client.0.smithi136.stderr:261031: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-15T21:14:12.197 INFO:tasks.workunit.client.0.smithi136.stderr:261031: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-15T21:14:12.197 INFO:tasks.workunit.client.0.smithi136.stderr:261031: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-15T21:14:12.197 INFO:tasks.workunit.client.0.smithi136.stderr:261031: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-15T21:14:12.197 INFO:tasks.workunit.client.0.smithi136.stderr:261031: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-15T21:14:12.197 INFO:tasks.workunit.client.0.smithi136.stderr:261031: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-15T21:14:12.197 INFO:tasks.workunit.client.0.smithi136.stderr:261031: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-15T21:14:12.197 INFO:tasks.workunit.client.0.smithi136.stderr:261031: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-15T21:14:12.197 INFO:tasks.workunit.client.0.smithi136.stderr:261031: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-15T21:14:12.197 INFO:tasks.workunit.client.0.smithi136.stderr:261031: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-15T21:14:12.197 INFO:tasks.workunit.client.0.smithi136.stderr:261031: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-15T21:14:12.197 INFO:tasks.workunit.client.0.smithi136.stderr:261031: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-15T21:14:12.197 INFO:tasks.workunit.client.0.smithi136.stderr:261031: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-15T21:14:12.198 INFO:tasks.workunit.client.0.smithi136.stderr:261031: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-15T21:14:12.198 INFO:tasks.workunit.client.0.smithi136.stderr:261031: 1 2024-09-15T21:14:12.198 INFO:tasks.workunit.client.0.smithi136.stderr:261031: 2 2024-09-15T21:14:12.198 INFO:tasks.workunit.client.0.smithi136.stderr:261031: 3 2024-09-15T21:14:12.198 INFO:tasks.workunit.client.0.smithi136.stderr:261031: 4 2024-09-15T21:14:12.198 INFO:tasks.workunit.client.0.smithi136.stderr:261031: 5 2024-09-15T21:14:12.198 INFO:tasks.workunit.client.0.smithi136.stderr:261031: 6 2024-09-15T21:14:12.198 INFO:tasks.workunit.client.0.smithi136.stderr:261031: 7 2024-09-15T21:14:12.198 INFO:tasks.workunit.client.0.smithi136.stderr:261031: 8 2024-09-15T21:14:12.198 INFO:tasks.workunit.client.0.smithi136.stderr:261031: 9' 2024-09-15T21:14:12.198 INFO:tasks.workunit.client.0.smithi136.stderr:261031: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-15T21:14:12.446 INFO:tasks.workunit.client.0.smithi136.stderr:261031: /home/ubuntu/cephtest/clone.client.0/qa/standalone: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T21:14:12.446 INFO:tasks.workunit.client.0.smithi136.stderr:261029: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-15T21:14:12.446 INFO:tasks.workunit.client.0.smithi136.stderr:261029: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=42949673067 2024-09-15T21:14:12.446 INFO:tasks.workunit.client.0.smithi136.stderr:261029: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 42949673067 2024-09-15T21:14:12.446 INFO:tasks.workunit.client.0.smithi136.stderr:261029: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836591 1-42949673067' 2024-09-15T21:14:12.446 INFO:tasks.workunit.client.0.smithi136.stderr:261029: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T21:14:12.446 INFO:tasks.workunit.client.0.smithi136.stderr:261029: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-15T21:14:12.446 INFO:tasks.workunit.client.0.smithi136.stderr:261029: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=64424509537 2024-09-15T21:14:12.446 INFO:tasks.workunit.client.0.smithi136.stderr:261029: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 64424509537 2024-09-15T21:14:12.446 INFO:tasks.workunit.client.0.smithi136.stderr:261029: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836591 1-42949673067 2-64424509537' 2024-09-15T21:14:12.446 INFO:tasks.workunit.client.0.smithi136.stderr:261029: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T21:14:12.447 INFO:tasks.workunit.client.0.smithi136.stderr:261029: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-15T21:14:12.447 INFO:tasks.workunit.client.0.smithi136.stderr:261029: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=519691042818 2024-09-15T21:14:12.447 INFO:tasks.workunit.client.0.smithi136.stderr:261029: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 519691042818 2024-09-15T21:14:12.447 INFO:tasks.workunit.client.0.smithi136.stderr:261029: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836591 1-42949673067 2-64424509537 3-519691042818' 2024-09-15T21:14:12.447 INFO:tasks.workunit.client.0.smithi136.stderr:261029: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T21:14:12.447 INFO:tasks.workunit.client.0.smithi136.stderr:261029: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-09-15T21:14:12.447 INFO:tasks.workunit.client.0.smithi136.stderr:261029: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=107374182486 2024-09-15T21:14:12.447 INFO:tasks.workunit.client.0.smithi136.stderr:261029: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 107374182486 2024-09-15T21:14:12.447 INFO:tasks.workunit.client.0.smithi136.stderr:261029: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836591 1-42949673067 2-64424509537 3-519691042818 4-107374182486' 2024-09-15T21:14:12.447 INFO:tasks.workunit.client.0.smithi136.stderr:261029: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T21:14:12.447 INFO:tasks.workunit.client.0.smithi136.stderr:261029: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-09-15T21:14:12.447 INFO:tasks.workunit.client.0.smithi136.stderr:261029: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=532575944706 2024-09-15T21:14:12.447 INFO:tasks.workunit.client.0.smithi136.stderr:261029: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 532575944706 2024-09-15T21:14:12.448 INFO:tasks.workunit.client.0.smithi136.stderr:261029: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836591 1-42949673067 2-64424509537 3-519691042818 4-107374182486 5-532575944706' 2024-09-15T21:14:12.448 INFO:tasks.workunit.client.0.smithi136.stderr:261029: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T21:14:12.448 INFO:tasks.workunit.client.0.smithi136.stderr:261029: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-09-15T21:14:12.448 INFO:tasks.workunit.client.0.smithi136.stderr:261029: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=150323855434 2024-09-15T21:14:12.448 INFO:tasks.workunit.client.0.smithi136.stderr:261029: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 150323855434 2024-09-15T21:14:12.448 INFO:tasks.workunit.client.0.smithi136.stderr:261029: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836591 1-42949673067 2-64424509537 3-519691042818 4-107374182486 5-532575944706 6-150323855434' 2024-09-15T21:14:12.448 INFO:tasks.workunit.client.0.smithi136.stderr:261029: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T21:14:13.416 INFO:tasks.workunit.client.0.smithi136.stderr:261029: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2024-09-15T21:14:13.416 INFO:tasks.workunit.client.0.smithi136.stderr:261029: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=171798691906 2024-09-15T21:14:13.416 INFO:tasks.workunit.client.0.smithi136.stderr:261029: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 171798691906 2024-09-15T21:14:13.416 INFO:tasks.workunit.client.0.smithi136.stderr:261029: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836591 1-42949673067 2-64424509537 3-519691042818 4-107374182486 5-532575944706 6-150323855434 7-171798691906' 2024-09-15T21:14:13.416 INFO:tasks.workunit.client.0.smithi136.stderr:261029: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T21:14:13.416 INFO:tasks.workunit.client.0.smithi136.stderr:261029: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2024-09-15T21:14:13.416 INFO:tasks.workunit.client.0.smithi136.stderr:261029: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=193273528380 2024-09-15T21:14:13.416 INFO:tasks.workunit.client.0.smithi136.stderr:261029: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 193273528380 2024-09-15T21:14:13.416 INFO:tasks.workunit.client.0.smithi136.stderr:261029: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836591 1-42949673067 2-64424509537 3-519691042818 4-107374182486 5-532575944706 6-150323855434 7-171798691906 8-193273528380' 2024-09-15T21:14:13.416 INFO:tasks.workunit.client.0.smithi136.stderr:261029: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T21:14:13.417 INFO:tasks.workunit.client.0.smithi136.stderr:261029: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2024-09-15T21:14:13.417 INFO:tasks.workunit.client.0.smithi136.stderr:261029: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=446676598798 2024-09-15T21:14:13.417 INFO:tasks.workunit.client.0.smithi136.stderr:261029: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 446676598798 2024-09-15T21:14:13.417 INFO:tasks.workunit.client.0.smithi136.stderr:261029: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836591 1-42949673067 2-64424509537 3-519691042818 4-107374182486 5-532575944706 6-150323855434 7-171798691906 8-193273528380 9-446676598798' 2024-09-15T21:14:13.417 INFO:tasks.workunit.client.0.smithi136.stderr:261029: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T21:14:13.417 INFO:tasks.workunit.client.0.smithi136.stderr:261029: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836591 2024-09-15T21:14:13.417 INFO:tasks.workunit.client.0.smithi136.stderr:261029: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T21:14:13.417 INFO:tasks.workunit.client.0.smithi136.stderr:261029: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-15T21:14:13.417 INFO:tasks.workunit.client.0.smithi136.stderr:261029: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836591 2024-09-15T21:14:13.417 INFO:tasks.workunit.client.0.smithi136.stderr:261029: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T21:14:13.417 INFO:tasks.workunit.client.0.smithi136.stderr:261029: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836591 2024-09-15T21:14:13.417 INFO:tasks.workunit.client.0.smithi136.stderr:261029: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836591' 2024-09-15T21:14:13.417 INFO:tasks.workunit.client.0.smithi136.stderr:261029: waiting osd.0 seq 21474836591 2024-09-15T21:14:13.418 INFO:tasks.workunit.client.0.smithi136.stderr:261029: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T21:14:13.418 INFO:tasks.workunit.client.0.smithi136.stderr:261029: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836591 -lt 21474836591 2024-09-15T21:14:13.418 INFO:tasks.workunit.client.0.smithi136.stderr:261029: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T21:14:13.418 INFO:tasks.workunit.client.0.smithi136.stderr:261029: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-42949673067 2024-09-15T21:14:13.418 INFO:tasks.workunit.client.0.smithi136.stderr:261029: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T21:14:13.418 INFO:tasks.workunit.client.0.smithi136.stderr:261029: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-15T21:14:13.418 INFO:tasks.workunit.client.0.smithi136.stderr:261029: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-42949673067 2024-09-15T21:14:13.418 INFO:tasks.workunit.client.0.smithi136.stderr:261029: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T21:14:13.418 INFO:tasks.workunit.client.0.smithi136.stderr:261029: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=42949673067 2024-09-15T21:14:13.418 INFO:tasks.workunit.client.0.smithi136.stderr:261029: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 42949673067' 2024-09-15T21:14:13.428 INFO:tasks.workunit.client.0.smithi136.stderr:261029: waiti/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T21:14:13.428 INFO:tasks.workunit.client.0.smithi136.stderr:261031: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-15T21:14:13.429 INFO:tasks.workunit.client.0.smithi136.stderr:261031: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836592 2024-09-15T21:14:13.429 INFO:tasks.workunit.client.0.smithi136.stderr:261031: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836592 2024-09-15T21:14:13.429 INFO:tasks.workunit.client.0.smithi136.stderr:261031: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836592' 2024-09-15T21:14:13.429 INFO:tasks.workunit.client.0.smithi136.stderr:261031: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T21:14:13.429 INFO:tasks.workunit.client.0.smithi136.stderr:261031: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-15T21:14:13.429 INFO:tasks.workunit.client.0.smithi136.stderr:261031: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=42949673068 2024-09-15T21:14:13.429 INFO:tasks.workunit.client.0.smithi136.stderr:261031: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 42949673068 2024-09-15T21:14:13.429 INFO:tasks.workunit.client.0.smithi136.stderr:261031: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836592 1-42949673068' 2024-09-15T21:14:13.429 INFO:tasks.workunit.client.0.smithi136.stderr:261031: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T21:14:13.429 INFO:tasks.workunit.client.0.smithi136.stderr:261031: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-15T21:14:13.429 INFO:tasks.workunit.client.0.smithi136.stderr:261031: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=64424509538 2024-09-15T21:14:13.429 INFO:tasks.workunit.client.0.smithi136.stderr:261031: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 64424509538 2024-09-15T21:14:13.429 INFO:tasks.workunit.client.0.smithi136.stderr:261031: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836592 1-42949673068 2-64424509538' 2024-09-15T21:14:13.430 INFO:tasks.workunit.client.0.smithi136.stderr:261031: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T21:14:13.430 INFO:tasks.workunit.client.0.smithi136.stderr:261031: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-15T21:14:13.430 INFO:tasks.workunit.client.0.smithi136.stderr:261031: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=519691042819 2024-09-15T21:14:13.430 INFO:tasks.workunit.client.0.smithi136.stderr:261031: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 519691042819 2024-09-15T21:14:13.430 INFO:tasks.workunit.client.0.smithi136.stderr:261031: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836592 1-42949673068 2-64424509538 3-519691042819' 2024-09-15T21:14:13.430 INFO:tasks.workunit.client.0.smithi136.stderr:261031: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T21:14:13.430 INFO:tasks.workunit.client.0.smithi136.stderr:261031: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-09-15T21:14:13.430 INFO:tasks.workunit.client.0.smithi136.stderr:261031: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=107374182487 2024-09-15T21:14:13.430 INFO:tasks.workunit.client.0.smithi136.stderr:261031: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 107374182487 2024-09-15T21:14:13.430 INFO:tasks.workunit.client.0.smithi136.stderr:261031: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836592 1-42949673068 2-64424509538 3-519691042819 4-107374182487' 2024-09-15T21:14:13.430 INFO:tasks.workunit.client.0.smithi136.stderr:261031: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T21:14:13.430 INFO:tasks.workunit.client.0.smithi136.stderr:261031: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-09-15T21:14:13.430 INFO:tasks.workunit.client.0.smithi136.stderr:261031: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=532575944707 2024-09-15T21:14:13.430 INFO:tasks.workunit.client.0.smithi136.stderr:261031: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 532575944707 2024-09-15T21:14:13.431 INFO:tasks.workunit.client.0.smithi136.stderr:261031: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836592 1-42949673068 2-64424509538 3-519691042819 4-107374182487 5-532575944707' 2024-09-15T21:14:13.431 INFO:tasks.workunit.client.0.smithi136.stderr:261031: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T21:14:13.431 INFO:tasks.workunit.client.0.smithi136.stderr:261031: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-09-15T21:14:14.059 INFO:tasks.workunit.client.0.smithi136.stderr:261031: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=150323855435 2024-09-15T21:14:14.059 INFO:tasks.workunit.client.0.smithi136.stderr:261031: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 150323855435 2024-09-15T21:14:14.059 INFO:tasks.workunit.client.0.smithi136.stderr:261031: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836592 1-42949673068 2-64424509538 3-519691042819 4-107374182487 5-532575944707 6-150323855435' 2024-09-15T21:14:14.059 INFO:tasks.workunit.client.0.smithi136.stderr:261031: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T21:14:14.059 INFO:tasks.workunit.client.0.smithi136.stderr:261031: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2024-09-15T21:14:14.059 INFO:tasks.workunit.client.0.smithi136.stderr:261031: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=171798691907 2024-09-15T21:14:14.059 INFO:tasks.workunit.client.0.smithi136.stderr:261031: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 171798691907 2024-09-15T21:14:14.059 INFO:tasks.workunit.client.0.smithi136.stderr:261031: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836592 1-42949673068 2-64424509538 3-519691042819 4-107374182487 5-532575944707 6-150323855435 7-171798691907' 2024-09-15T21:14:14.060 INFO:tasks.workunit.client.0.smithi136.stderr:261031: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T21:14:14.060 INFO:tasks.workunit.client.0.smithi136.stderr:261031: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2024-09-15T21:14:14.060 INFO:tasks.workunit.client.0.smithi136.stderr:261031: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=193273528381 2024-09-15T21:14:14.060 INFO:tasks.workunit.client.0.smithi136.stderr:261031: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 193273528381 2024-09-15T21:14:14.060 INFO:tasks.workunit.client.0.smithi136.stderr:261031: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836592 1-42949673068 2-64424509538 3-519691042819 4-107374182487 5-532575944707 6-150323855435 7-171798691907 8-193273528381' 2024-09-15T21:14:14.060 INFO:tasks.workunit.client.0.smithi136.stderr:261031: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T21:14:14.060 INFO:tasks.workunit.client.0.smithi136.stderr:261031: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2024-09-15T21:14:14.060 INFO:tasks.workunit.client.0.smithi136.stderr:261031: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=446676598799 2024-09-15T21:14:14.060 INFO:tasks.workunit.client.0.smithi136.stderr:261031: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 446676598799 2024-09-15T21:14:14.060 INFO:tasks.workunit.client.0.smithi136.stderr:261031: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836592 1-42949673068 2-64424509538 3-519691042819 4-107374182487 5-532575944707 6-150323855435 7-171798691907 8-193273528381 9-446676598799' 2024-09-15T21:14:14.060 INFO:tasks.workunit.client.0.smithi136.stderr:261031: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T21:14:14.061 INFO:tasks.workunit.client.0.smithi136.stderr:261031: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836592 2024-09-15T21:14:14.061 INFO:tasks.workunit.client.0.smithi136.stderr:261031: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T21:14:14.061 INFO:tasks.workunit.client.0.smithi136.stderr:261031: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-15T21:14:14.061 INFO:tasks.workunit.client.0.smithi136.stderr:261031: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836592 2024-09-15T21:14:14.061 INFO:tasks.workunit.client.0.smithi136.stderr:261031: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T21:14:14.061 INFO:tasks.workunit.client.0.smithi136.stderr:261031: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836592 2024-09-15T21:14:14.061 INFO:tasks.workunit.client.0.smithi136.stderr:261031: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836592' 2024-09-15T21:14:14.061 INFO:tasks.workunit.client.0.smithi136.stderr:261031: waiting osd.0 seq 21474836592 2024-09-15T21:14:14.061 INFO:tasks.workunit.client.0.smithi136.stderr:261031: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T21:14:14.061 INFO:tasks.workunit.client.0.smithi136.stderr:261031: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836593 -lt 21474836592 2024-09-15T21:14:14.061 INFO:tasks.workunit.client.0.smithi136.stderr:261031: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T21:14:14.061 INFO:tasks.workunit.client.0.smithi136.stderr:261031: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-42949673068 2024-09-15T21:14:14.061 INFO:tasks.workunit.client.0.smithi136.stderr:261031: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T21:14:14.592 INFO:tasks.workunit.client.0.smithi136.stderr:261031: /home/ubuntu/cephtest/clone.client.0/qa/standng osd.1 seq 42949673067 2024-09-15T21:14:14.592 INFO:tasks.workunit.client.0.smithi136.stderr:261029: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-15T21:14:14.592 INFO:tasks.workunit.client.0.smithi136.stderr:261029: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 42949673067 -lt 42949673067 2024-09-15T21:14:14.592 INFO:tasks.workunit.client.0.smithi136.stderr:261029: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T21:14:14.592 INFO:tasks.workunit.client.0.smithi136.stderr:261029: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-64424509537 2024-09-15T21:14:14.592 INFO:tasks.workunit.client.0.smithi136.stderr:261029: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T21:14:14.592 INFO:tasks.workunit.client.0.smithi136.stderr:261029: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-15T21:14:14.592 INFO:tasks.workunit.client.0.smithi136.stderr:261029: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-64424509537 2024-09-15T21:14:14.592 INFO:tasks.workunit.client.0.smithi136.stderr:261029: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T21:14:14.592 INFO:tasks.workunit.client.0.smithi136.stderr:261029: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=64424509537 2024-09-15T21:14:14.593 INFO:tasks.workunit.client.0.smithi136.stderr:261029: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 64424509537' 2024-09-15T21:14:14.593 INFO:tasks.workunit.client.0.smithi136.stderr:261029: waiting osd.2 seq 64424509537 2024-09-15T21:14:14.593 INFO:tasks.workunit.client.0.smithi136.stderr:261029: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-15T21:14:14.593 INFO:tasks.workunit.client.0.smithi136.stderr:261029: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 64424509539 -lt 64424509537 2024-09-15T21:14:14.593 INFO:tasks.workunit.client.0.smithi136.stderr:261029: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T21:14:14.593 INFO:tasks.workunit.client.0.smithi136.stderr:261029: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 3-519691042818 2024-09-15T21:14:14.593 INFO:tasks.workunit.client.0.smithi136.stderr:261029: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T21:14:14.593 INFO:tasks.workunit.client.0.smithi136.stderr:261029: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=3 2024-09-15T21:14:14.593 INFO:tasks.workunit.client.0.smithi136.stderr:261029: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 3-519691042818 2024-09-15T21:14:14.593 INFO:tasks.workunit.client.0.smithi136.stderr:261029: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T21:14:14.593 INFO:tasks.workunit.client.0.smithi136.stderr:261029: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=519691042818 2024-09-15T21:14:14.593 INFO:tasks.workunit.client.0.smithi136.stderr:261029: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.3 seq 519691042818' 2024-09-15T21:14:14.593 INFO:tasks.workunit.client.0.smithi136.stderr:261029: waiting osd.3 seq 519691042818 2024-09-15T21:14:14.594 INFO:tasks.workunit.client.0.smithi136.stderr:261029: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-15T21:14:14.594 INFO:tasks.workunit.client.0.smithi136.stderr:261029: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 519691042819 -lt 519691042818 2024-09-15T21:14:14.594 INFO:tasks.workunit.client.0.smithi136.stderr:261029: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T21:14:14.594 INFO:tasks.workunit.client.0.smithi136.stderr:261029: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 4-107374182486 2024-09-15T21:14:14.594 INFO:tasks.workunit.client.0.smithi136.stderr:261029: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T21:14:14.594 INFO:tasks.workunit.client.0.smithi136.stderr:261029: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=4 2024-09-15T21:14:14.594 INFO:tasks.workunit.client.0.smithi136.stderr:261029: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 4-107374182486 2024-09-15T21:14:14.594 INFO:tasks.workunit.client.0.smithi136.stderr:261029: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T21:14:14.594 INFO:tasks.workunit.client.0.smithi136.stderr:261029: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=107374182486 2024-09-15T21:14:14.594 INFO:tasks.workunit.client.0.smithi136.stderr:261029: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.4 seq 107374182486' 2024-09-15T21:14:14.594 INFO:tasks.workunit.client.0.smithi136.stderr:261029: waiting osd.4 seq 107374182486 2024-09-15T21:14:14.594 INFO:tasks.workunit.client.0.smithi136.stderr:261029: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 4 2024-09-15T21:14:14.594 INFO:tasks.workunit.client.0.smithi136.stderr:261029: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 107374182487 -lt 107374182486 2024-09-15T21:14:14.594 INFO:tasks.workunit.client.0.smithi136.stderr:261029: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T21:14:14.978 INFO:tasks.workunit.client.0.smithi136.stderr:261029: //home/ubuntu/cephalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-15T21:14:14.978 INFO:tasks.workunit.client.0.smithi136.stderr:261031: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-42949673068 2024-09-15T21:14:14.978 INFO:tasks.workunit.client.0.smithi136.stderr:261031: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T21:14:14.978 INFO:tasks.workunit.client.0.smithi136.stderr:261031: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=42949673068 2024-09-15T21:14:14.978 INFO:tasks.workunit.client.0.smithi136.stderr:261031: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 42949673068' 2024-09-15T21:14:14.978 INFO:tasks.workunit.client.0.smithi136.stderr:261031: waiting osd.1 seq 42949673068 2024-09-15T21:14:14.978 INFO:tasks.workunit.client.0.smithi136.stderr:261031: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-15T21:14:14.978 INFO:tasks.workunit.client.0.smithi136.stderr:261031: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 42949673069 -lt 42949673068 2024-09-15T21:14:14.978 INFO:tasks.workunit.client.0.smithi136.stderr:261031: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T21:14:14.978 INFO:tasks.workunit.client.0.smithi136.stderr:261031: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-64424509538 2024-09-15T21:14:14.978 INFO:tasks.workunit.client.0.smithi136.stderr:261031: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T21:14:14.979 INFO:tasks.workunit.client.0.smithi136.stderr:261031: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-15T21:14:14.979 INFO:tasks.workunit.client.0.smithi136.stderr:261031: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-64424509538 2024-09-15T21:14:14.979 INFO:tasks.workunit.client.0.smithi136.stderr:261031: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T21:14:14.979 INFO:tasks.workunit.client.0.smithi136.stderr:261031: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=64424509538 2024-09-15T21:14:14.979 INFO:tasks.workunit.client.0.smithi136.stderr:261031: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 64424509538' 2024-09-15T21:14:14.979 INFO:tasks.workunit.client.0.smithi136.stderr:261031: waiting osd.2 seq 64424509538 2024-09-15T21:14:14.979 INFO:tasks.workunit.client.0.smithi136.stderr:261031: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-15T21:14:14.979 INFO:tasks.workunit.client.0.smithi136.stderr:261031: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 64424509539 -lt 64424509538 2024-09-15T21:14:14.979 INFO:tasks.workunit.client.0.smithi136.stderr:261031: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T21:14:14.979 INFO:tasks.workunit.client.0.smithi136.stderr:261031: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 3-519691042819 2024-09-15T21:14:14.979 INFO:tasks.workunit.client.0.smithi136.stderr:261031: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T21:14:14.979 INFO:tasks.workunit.client.0.smithi136.stderr:261031: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=3 2024-09-15T21:14:14.979 INFO:tasks.workunit.client.0.smithi136.stderr:261031: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 3-519691042819 2024-09-15T21:14:14.980 INFO:tasks.workunit.client.0.smithi136.stderr:261031: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T21:14:14.980 INFO:tasks.workunit.client.0.smithi136.stderr:261031: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=519691042819 2024-09-15T21:14:14.980 INFO:tasks.workunit.client.0.smithi136.stderr:261031: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.3 seq 519691042819' 2024-09-15T21:14:14.980 INFO:tasks.workunit.client.0.smithi136.stderr:261031: waiting osd.3 seq 519691042819 2024-09-15T21:14:14.980 INFO:tasks.workunit.client.0.smithi136.stderr:261031: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-15T21:14:14.980 INFO:tasks.workunit.client.0.smithi136.stderr:261031: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 519691042819 -lt 519691042819 2024-09-15T21:14:14.980 INFO:tasks.workunit.client.0.smithi136.stderr:261031: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T21:14:14.980 INFO:tasks.workunit.client.0.smithi136.stderr:261031: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 4-107374182487 2024-09-15T21:14:14.980 INFO:tasks.workunit.client.0.smithi136.stderr:261031: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T21:14:14.980 INFO:tasks.workunit.client.0.smithi136.stderr:261031: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=4 2024-09-15T21:14:14.980 INFO:tasks.workunit.client.0.smithi136.stderr:261031: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 4-107374182487 2024-09-15T21:14:14.980 INFO:tasks.workunit.client.0.smithi136.stderr:261031: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T21:14:14.980 INFO:tasks.workunit.client.0.smithi136.stderr:261031: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=107374182487 2024-09-15T21:14:15.542 INFO:tasks.workunit.client.0.smithi136.stderr:261031: /home/ubuntu/cephtetest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 5-532575944706 2024-09-15T21:14:15.542 INFO:tasks.workunit.client.0.smithi136.stderr:261029: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T21:14:15.542 INFO:tasks.workunit.client.0.smithi136.stderr:261029: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=5 2024-09-15T21:14:15.542 INFO:tasks.workunit.client.0.smithi136.stderr:261029: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 5-532575944706 2024-09-15T21:14:15.543 INFO:tasks.workunit.client.0.smithi136.stderr:261029: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T21:14:15.543 INFO:tasks.workunit.client.0.smithi136.stderr:261029: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=532575944706 2024-09-15T21:14:15.543 INFO:tasks.workunit.client.0.smithi136.stderr:261029: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.5 seq 532575944706' 2024-09-15T21:14:15.543 INFO:tasks.workunit.client.0.smithi136.stderr:261029: waiting osd.5 seq 532575944706 2024-09-15T21:14:15.543 INFO:tasks.workunit.client.0.smithi136.stderr:261029: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 5 2024-09-15T21:14:15.543 INFO:tasks.workunit.client.0.smithi136.stderr:261029: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 532575944707 -lt 532575944706 2024-09-15T21:14:15.543 INFO:tasks.workunit.client.0.smithi136.stderr:261029: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T21:14:15.543 INFO:tasks.workunit.client.0.smithi136.stderr:261029: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 6-150323855434 2024-09-15T21:14:15.543 INFO:tasks.workunit.client.0.smithi136.stderr:261029: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T21:14:15.543 INFO:tasks.workunit.client.0.smithi136.stderr:261029: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=6 2024-09-15T21:14:15.543 INFO:tasks.workunit.client.0.smithi136.stderr:261029: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 6-150323855434 2024-09-15T21:14:15.543 INFO:tasks.workunit.client.0.smithi136.stderr:261029: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T21:14:15.544 INFO:tasks.workunit.client.0.smithi136.stderr:261029: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=150323855434 2024-09-15T21:14:15.544 INFO:tasks.workunit.client.0.smithi136.stderr:261029: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.6 seq 150323855434' 2024-09-15T21:14:15.544 INFO:tasks.workunit.client.0.smithi136.stderr:261029: waiting osd.6 seq 150323855434 2024-09-15T21:14:15.544 INFO:tasks.workunit.client.0.smithi136.stderr:261029: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 6 2024-09-15T21:14:15.544 INFO:tasks.workunit.client.0.smithi136.stderr:261029: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 150323855435 -lt 150323855434 2024-09-15T21:14:15.544 INFO:tasks.workunit.client.0.smithi136.stderr:261029: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T21:14:15.544 INFO:tasks.workunit.client.0.smithi136.stderr:261029: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 7-171798691906 2024-09-15T21:14:15.544 INFO:tasks.workunit.client.0.smithi136.stderr:261029: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T21:14:15.544 INFO:tasks.workunit.client.0.smithi136.stderr:261029: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=7 2024-09-15T21:14:15.544 INFO:tasks.workunit.client.0.smithi136.stderr:261029: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 7-171798691906 2024-09-15T21:14:15.544 INFO:tasks.workunit.client.0.smithi136.stderr:261029: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T21:14:15.544 INFO:tasks.workunit.client.0.smithi136.stderr:261029: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=171798691906 2024-09-15T21:14:15.545 INFO:tasks.workunit.client.0.smithi136.stderr:261029: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.7 seq 171798691906' 2024-09-15T21:14:15.545 INFO:tasks.workunit.client.0.smithi136.stderr:261029: waiting osd.7 seq 171798691906 2024-09-15T21:14:15.545 INFO:tasks.workunit.client.0.smithi136.stderr:261029: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 7 2024-09-15T21:14:15.545 INFO:tasks.workunit.client.0.smithi136.stderr:261029: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 171798691907 -lt 171798691906 2024-09-15T21:14:15.545 INFO:tasks.workunit.client.0.smithi136.stderr:261029: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T21:14:15.545 INFO:tasks.workunit.client.0.smithi136.stderr:261029: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 8-193273528380 2024-09-15T21:14:15.545 INFO:tasks.workunit.client.0.smithi136.stderr:261029: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T21:14:15.545 INFO:tasks.workunit.client.0.smithi136.stderr:261029: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=8 2024-09-15T21:14:16.257 INFO:tasks.workunit.client.0.smithi136.stderr:261029: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echst/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.4 seq 107374182487' 2024-09-15T21:14:16.257 INFO:tasks.workunit.client.0.smithi136.stderr:261031: waiting osd.4 seq 107374182487 2024-09-15T21:14:16.257 INFO:tasks.workunit.client.0.smithi136.stderr:261031: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 4 2024-09-15T21:14:16.257 INFO:tasks.workunit.client.0.smithi136.stderr:261031: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 107374182487 -lt 107374182487 2024-09-15T21:14:16.257 INFO:tasks.workunit.client.0.smithi136.stderr:261031: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T21:14:16.257 INFO:tasks.workunit.client.0.smithi136.stderr:261031: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 5-532575944707 2024-09-15T21:14:16.257 INFO:tasks.workunit.client.0.smithi136.stderr:261031: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T21:14:16.257 INFO:tasks.workunit.client.0.smithi136.stderr:261031: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=5 2024-09-15T21:14:16.257 INFO:tasks.workunit.client.0.smithi136.stderr:261031: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 5-532575944707 2024-09-15T21:14:16.258 INFO:tasks.workunit.client.0.smithi136.stderr:261031: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T21:14:16.258 INFO:tasks.workunit.client.0.smithi136.stderr:261031: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=532575944707 2024-09-15T21:14:16.258 INFO:tasks.workunit.client.0.smithi136.stderr:261031: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.5 seq 532575944707' 2024-09-15T21:14:16.258 INFO:tasks.workunit.client.0.smithi136.stderr:261031: waiting osd.5 seq 532575944707 2024-09-15T21:14:16.258 INFO:tasks.workunit.client.0.smithi136.stderr:261031: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 5 2024-09-15T21:14:16.258 INFO:tasks.workunit.client.0.smithi136.stderr:261031: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 532575944707 -lt 532575944707 2024-09-15T21:14:16.258 INFO:tasks.workunit.client.0.smithi136.stderr:261031: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T21:14:16.258 INFO:tasks.workunit.client.0.smithi136.stderr:261031: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 6-150323855435 2024-09-15T21:14:16.258 INFO:tasks.workunit.client.0.smithi136.stderr:261031: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T21:14:16.258 INFO:tasks.workunit.client.0.smithi136.stderr:261031: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=6 2024-09-15T21:14:16.258 INFO:tasks.workunit.client.0.smithi136.stderr:261031: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 6-150323855435 2024-09-15T21:14:16.258 INFO:tasks.workunit.client.0.smithi136.stderr:261031: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T21:14:16.258 INFO:tasks.workunit.client.0.smithi136.stderr:261031: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=150323855435 2024-09-15T21:14:16.258 INFO:tasks.workunit.client.0.smithi136.stderr:261031: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.6 seq 150323855435' 2024-09-15T21:14:16.259 INFO:tasks.workunit.client.0.smithi136.stderr:261031: waiting osd.6 seq 150323855435 2024-09-15T21:14:16.259 INFO:tasks.workunit.client.0.smithi136.stderr:261031: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 6 2024-09-15T21:14:16.259 INFO:tasks.workunit.client.0.smithi136.stderr:261031: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 150323855435 -lt 150323855435 2024-09-15T21:14:16.259 INFO:tasks.workunit.client.0.smithi136.stderr:261031: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T21:14:16.259 INFO:tasks.workunit.client.0.smithi136.stderr:261031: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 7-171798691907 2024-09-15T21:14:16.259 INFO:tasks.workunit.client.0.smithi136.stderr:261031: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T21:14:16.259 INFO:tasks.workunit.client.0.smithi136.stderr:261031: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=7 2024-09-15T21:14:16.259 INFO:tasks.workunit.client.0.smithi136.stderr:261031: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 7-171798691907 2024-09-15T21:14:16.259 INFO:tasks.workunit.client.0.smithi136.stderr:261031: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T21:14:16.259 INFO:tasks.workunit.client.0.smithi136.stderr:261031: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=171798691907 2024-09-15T21:14:16.259 INFO:tasks.workunit.client.0.smithi136.stderr:261031: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.7 seq 171798691907' 2024-09-15T21:14:16.259 INFO:tasks.workunit.client.0.smithi136.stderr:261031: waiting osd.7 seq 171798691907 2024-09-15T21:14:16.260 INFO:tasks.workunit.client.0.smithi136.stderr:261031: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 7 2024-09-15T21:14:16.260 INFO:tasks.workunit.client.0.smithi136.stderr:261031: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 171798691907 -lt 171798691907 2024-09-15T21:14:17.175 INFO:tasks.workunit.client.0.smithi136.stderr:261031:o 8-193273528380 2024-09-15T21:14:17.175 INFO:tasks.workunit.client.0.smithi136.stderr:261029: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T21:14:17.175 INFO:tasks.workunit.client.0.smithi136.stderr:261029: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=193273528380 2024-09-15T21:14:17.175 INFO:tasks.workunit.client.0.smithi136.stderr:261029: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.8 seq 193273528380' 2024-09-15T21:14:17.175 INFO:tasks.workunit.client.0.smithi136.stderr:261029: waiting osd.8 seq 193273528380 2024-09-15T21:14:17.175 INFO:tasks.workunit.client.0.smithi136.stderr:261029: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 8 2024-09-15T21:14:17.175 INFO:tasks.workunit.client.0.smithi136.stderr:261029: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 193273528381 -lt 193273528380 2024-09-15T21:14:17.175 INFO:tasks.workunit.client.0.smithi136.stderr:261029: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T21:14:17.175 INFO:tasks.workunit.client.0.smithi136.stderr:261029: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 9-446676598798 2024-09-15T21:14:17.175 INFO:tasks.workunit.client.0.smithi136.stderr:261029: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T21:14:17.175 INFO:tasks.workunit.client.0.smithi136.stderr:261029: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=9 2024-09-15T21:14:17.175 INFO:tasks.workunit.client.0.smithi136.stderr:261029: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 9-446676598798 2024-09-15T21:14:17.176 INFO:tasks.workunit.client.0.smithi136.stderr:261029: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T21:14:17.176 INFO:tasks.workunit.client.0.smithi136.stderr:261029: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=446676598798 2024-09-15T21:14:17.176 INFO:tasks.workunit.client.0.smithi136.stderr:261029: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.9 seq 446676598798' 2024-09-15T21:14:17.176 INFO:tasks.workunit.client.0.smithi136.stderr:261029: waiting osd.9 seq 446676598798 2024-09-15T21:14:17.176 INFO:tasks.workunit.client.0.smithi136.stderr:261029: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 9 2024-09-15T21:14:17.176 INFO:tasks.workunit.client.0.smithi136.stderr:261029: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 446676598799 -lt 446676598798 2024-09-15T21:14:17.176 INFO:tasks.workunit.client.0.smithi136.stderr:261029: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-15T21:14:17.176 INFO:tasks.workunit.client.0.smithi136.stderr:261029: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T21:14:17.176 INFO:tasks.workunit.client.0.smithi136.stderr:261029: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T21:14:17.176 INFO:tasks.workunit.client.0.smithi136.stderr:261029: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 5 == 0 2024-09-15T21:14:17.176 INFO:tasks.workunit.client.0.smithi136.stderr:261029: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-15T21:14:17.176 INFO:tasks.workunit.client.0.smithi136.stderr:261029: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-15T21:14:17.176 INFO:tasks.workunit.client.0.smithi136.stderr:261029: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-15T21:14:17.176 INFO:tasks.workunit.client.0.smithi136.stderr:261029: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T21:14:17.177 INFO:tasks.workunit.client.0.smithi136.stderr:261029: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T21:14:17.177 INFO:tasks.workunit.client.0.smithi136.stderr:261029: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T21:14:17.177 INFO:tasks.workunit.client.0.smithi136.stderr:261029: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T21:14:17.177 INFO:tasks.workunit.client.0.smithi136.stderr:261029: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=5 2024-09-15T21:14:17.177 INFO:tasks.workunit.client.0.smithi136.stderr:261029: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-15T21:14:17.177 INFO:tasks.workunit.client.0.smithi136.stderr:261029: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T21:14:17.177 INFO:tasks.workunit.client.0.smithi136.stderr:261029: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T21:14:17.177 INFO:tasks.workunit.client.0.smithi136.stderr:261029: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 = 5 2024-09-15T21:14:17.177 INFO:tasks.workunit.client.0.smithi136.stderr:261029: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-15T21:14:17.177 INFO:tasks.workunit.client.0.smithi136.stderr:261029: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-15T21:14:17.177 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: run_in_background: return 0 2024-09-15T21:14:17.177 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: wait_background: for pid in $pids 2024-09-15T21:14:17.177 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2175: wait_background: wait 261027 2024-09-15T21:14:17.504 INFO:tasks.workunit.client.0.smithi136.stderr: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T21:14:17.504 INFO:tasks.workunit.client.0.smithi136.stderr:261031: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 8-193273528381 2024-09-15T21:14:17.504 INFO:tasks.workunit.client.0.smithi136.stderr:261031: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T21:14:17.504 INFO:tasks.workunit.client.0.smithi136.stderr:261031: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=8 2024-09-15T21:14:17.504 INFO:tasks.workunit.client.0.smithi136.stderr:261031: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 8-193273528381 2024-09-15T21:14:17.504 INFO:tasks.workunit.client.0.smithi136.stderr:261031: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T21:14:17.504 INFO:tasks.workunit.client.0.smithi136.stderr:261031: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=193273528381 2024-09-15T21:14:17.504 INFO:tasks.workunit.client.0.smithi136.stderr:261031: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.8 seq 193273528381' 2024-09-15T21:14:17.504 INFO:tasks.workunit.client.0.smithi136.stderr:261031: waiting osd.8 seq 193273528381 2024-09-15T21:14:17.504 INFO:tasks.workunit.client.0.smithi136.stderr:261031: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 8 2024-09-15T21:14:17.504 INFO:tasks.workunit.client.0.smithi136.stderr:261031: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 193273528381 -lt 193273528381 2024-09-15T21:14:17.504 INFO:tasks.workunit.client.0.smithi136.stderr:261031: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T21:14:17.505 INFO:tasks.workunit.client.0.smithi136.stderr:261031: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 9-446676598799 2024-09-15T21:14:17.505 INFO:tasks.workunit.client.0.smithi136.stderr:261031: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T21:14:17.505 INFO:tasks.workunit.client.0.smithi136.stderr:261031: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=9 2024-09-15T21:14:17.505 INFO:tasks.workunit.client.0.smithi136.stderr:261031: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 9-446676598799 2024-09-15T21:14:17.505 INFO:tasks.workunit.client.0.smithi136.stderr:261031: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T21:14:17.505 INFO:tasks.workunit.client.0.smithi136.stderr:261031: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=446676598799 2024-09-15T21:14:17.505 INFO:tasks.workunit.client.0.smithi136.stderr:261031: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.9 seq 446676598799' 2024-09-15T21:14:17.505 INFO:tasks.workunit.client.0.smithi136.stderr:261031: waiting osd.9 seq 446676598799 2024-09-15T21:14:17.505 INFO:tasks.workunit.client.0.smithi136.stderr:261031: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 9 2024-09-15T21:14:17.505 INFO:tasks.workunit.client.0.smithi136.stderr:261031: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 446676598799 -lt 446676598799 2024-09-15T21:14:17.505 INFO:tasks.workunit.client.0.smithi136.stderr:261031: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-15T21:14:17.505 INFO:tasks.workunit.client.0.smithi136.stderr:261031: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T21:14:17.506 INFO:tasks.workunit.client.0.smithi136.stderr:261031: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T21:14:17.506 INFO:tasks.workunit.client.0.smithi136.stderr:261031: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 5 == 0 2024-09-15T21:14:17.506 INFO:tasks.workunit.client.0.smithi136.stderr:261031: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-15T21:14:17.506 INFO:tasks.workunit.client.0.smithi136.stderr:261031: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-15T21:14:17.506 INFO:tasks.workunit.client.0.smithi136.stderr:261031: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-15T21:14:17.506 INFO:tasks.workunit.client.0.smithi136.stderr:261031: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T21:14:17.506 INFO:tasks.workunit.client.0.smithi136.stderr:261031: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T21:14:17.506 INFO:tasks.workunit.client.0.smithi136.stderr:261031: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T21:14:17.506 INFO:tasks.workunit.client.0.smithi136.stderr:261031: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T21:14:17.506 INFO:tasks.workunit.client.0.smithi136.stderr:261031: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=5 2024-09-15T21:14:17.848 INFO:tasks.workunit.client.0.smithi136.stderr:261031: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-15T21:14:17.849 INFO:tasks.workunit.client.0.smithi136.stderr:261031: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T21:14:17.849 INFO:tasks.workunit.client.0.smithi136.stderr:261031: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T21:14:17.849 INFO:tasks.workunit.client.0.smithi136.stderr:261031: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 = 5 2024-09-15T21:14:17.849 INFO:tasks.workunit.client.0.smithi136.stderr:261031: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-15T21:14:17.849 INFO:tasks.workunit.client.0.smithi136.stderr:261031: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-15T21:14:17.849 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: run_in_background: return 0 2024-09-15T21:14:17.849 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: wait_background: eval 'pids='\'''\''' 2024-09-15T21:14:17.849 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: wait_background: pids= 2024-09-15T21:14:17.849 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: wait_background: return 0 2024-09-15T21:14:17.849 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:212: corrupt_and_repair_two: return_code=0 2024-09-15T21:14:17.849 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:213: corrupt_and_repair_two: '[' 0 -ne 0 ']' 2024-09-15T21:14:17.850 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:215: corrupt_and_repair_two: rados --pool ecpool get SOMETHING td/osd-scrub-repair/COPY 2024-09-15T21:14:17.888 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:216: corrupt_and_repair_two: diff td/osd-scrub-repair/ORIGINAL td/osd-scrub-repair/COPY 2024-09-15T21:14:17.890 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:66: run: teardown td/osd-scrub-repair 2024-09-15T21:14:17.890 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/osd-scrub-repair 2024-09-15T21:14:17.890 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-15T21:14:17.890 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/osd-scrub-repair KILL 2024-09-15T21:14:17.890 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-15T21:14:17.890 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-15T21:14:17.890 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-15T21:14:17.890 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-15T21:14:17.890 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-15T21:14:18.003 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-15T21:14:18.004 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-15T21:14:18.004 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-15T21:14:18.005 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-15T21:14:18.006 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-15T21:14:18.006 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-15T21:14:18.006 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-15T21:14:18.007 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T21:14:18.007 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-15T21:14:18.008 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-15T21:14:18.008 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T21:14:18.008 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-15T21:14:18.010 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-15T21:14:18.010 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/osd-scrub-repair 2024-09-15T21:14:18.075 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-15T21:14:18.075 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T21:14:18.075 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52602 2024-09-15T21:14:18.075 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.52602 2024-09-15T21:14:18.077 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-15T21:14:18.077 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-15T21:14:18.077 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:63: run: for func in $funcs 2024-09-15T21:14:18.077 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:64: run: setup td/osd-scrub-repair 2024-09-15T21:14:18.077 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/osd-scrub-repair 2024-09-15T21:14:18.077 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/osd-scrub-repair 2024-09-15T21:14:18.077 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/osd-scrub-repair 2024-09-15T21:14:18.077 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-15T21:14:18.077 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/osd-scrub-repair KILL 2024-09-15T21:14:18.077 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-15T21:14:18.077 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-15T21:14:18.078 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-15T21:14:18.078 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-15T21:14:18.078 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-15T21:14:18.078 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-15T21:14:18.079 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-15T21:14:18.079 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-15T21:14:18.080 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-15T21:14:18.081 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-15T21:14:18.081 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-15T21:14:18.081 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-15T21:14:18.082 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T21:14:18.082 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-15T21:14:18.083 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-15T21:14:18.083 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T21:14:18.083 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-15T21:14:18.084 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-15T21:14:18.084 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/osd-scrub-repair 2024-09-15T21:14:18.086 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-15T21:14:18.086 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T21:14:18.086 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52602 2024-09-15T21:14:18.086 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.52602 2024-09-15T21:14:18.087 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-15T21:14:18.087 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-15T21:14:18.087 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/osd-scrub-repair 2024-09-15T21:14:18.088 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2024-09-15T21:14:18.088 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T21:14:18.088 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52602 2024-09-15T21:14:18.088 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.52602 2024-09-15T21:14:18.090 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2024-09-15T21:14:18.090 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 4096 -le 1024 ']' 2024-09-15T21:14:18.091 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2024-09-15T21:14:18.091 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/osd-scrub-repair 1' TERM HUP INT 2024-09-15T21:14:18.091 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:65: run: TEST_corrupt_and_repair_replicated td/osd-scrub-repair 2024-09-15T21:14:18.091 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:94: TEST_corrupt_and_repair_replicated: local dir=td/osd-scrub-repair 2024-09-15T21:14:18.091 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:95: TEST_corrupt_and_repair_replicated: local poolname=rbd 2024-09-15T21:14:18.091 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:97: TEST_corrupt_and_repair_replicated: run_mon td/osd-scrub-repair a --osd_pool_default_size=2 2024-09-15T21:14:18.091 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/osd-scrub-repair 2024-09-15T21:14:18.091 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2024-09-15T21:14:18.091 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2024-09-15T21:14:18.091 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2024-09-15T21:14:18.091 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/osd-scrub-repair/a 2024-09-15T21:14:18.091 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-scrub-repair/a --run-dir=td/osd-scrub-repair --osd_pool_default_size=2 2024-09-15T21:14:18.557 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2024-09-15T21:14:18.557 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-15T21:14:18.557 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-15T21:14:18.558 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-15T21:14:18.558 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T21:14:18.558 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52602 2024-09-15T21:14:18.558 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.52602/$cluster-$name.asok' 2024-09-15T21:14:18.558 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-scrub-repair/a '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.52602/$cluster-$name.asok' --mon-cluster-log-file=td/osd-scrub-repair/log --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=2 2024-09-15T21:14:18.749 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2024-09-15T21:14:18.750 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2024-09-15T21:14:18.750 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-15T21:14:18.750 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-15T21:14:18.751 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2024-09-15T21:14:18.752 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2024-09-15T21:14:18.752 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-15T21:14:18.752 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-15T21:14:18.752 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-15T21:14:18.752 INFO:tasks.workunit.client.0.smithi136.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-15T21:14:18.752 INFO:tasks.workunit.client.0.smithi136.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T21:14:18.752 INFO:tasks.workunit.client.0.smithi136.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52602 2024-09-15T21:14:18.752 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.52602/ceph-mon.a.asok 2024-09-15T21:14:18.752 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-15T21:14:18.752 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.52602/ceph-mon.a.asok config get fsid 2024-09-15T21:14:18.832 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2024-09-15T21:14:18.832 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-15T21:14:18.832 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-15T21:14:18.832 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2024-09-15T21:14:18.832 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2024-09-15T21:14:18.832 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-15T21:14:18.832 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-15T21:14:18.832 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-15T21:14:18.833 INFO:tasks.workunit.client.0.smithi136.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-15T21:14:18.833 INFO:tasks.workunit.client.0.smithi136.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T21:14:18.833 INFO:tasks.workunit.client.0.smithi136.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52602 2024-09-15T21:14:18.833 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.52602/ceph-mon.a.asok 2024-09-15T21:14:18.834 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-15T21:14:18.834 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.52602/ceph-mon.a.asok config get mon_host 2024-09-15T21:14:18.912 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:98: TEST_corrupt_and_repair_replicated: run_mgr td/osd-scrub-repair x 2024-09-15T21:14:18.912 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: run_mgr: local dir=td/osd-scrub-repair 2024-09-15T21:14:18.912 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: run_mgr: shift 2024-09-15T21:14:18.912 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: run_mgr: local id=x 2024-09-15T21:14:18.912 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: run_mgr: shift 2024-09-15T21:14:18.912 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:548: run_mgr: local data=td/osd-scrub-repair/x 2024-09-15T21:14:18.912 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:550: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-15T21:14:19.726 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: get_asok_path 2024-09-15T21:14:19.726 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-15T21:14:19.726 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-15T21:14:19.727 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-15T21:14:19.727 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T21:14:19.727 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52602 2024-09-15T21:14:19.727 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.52602/$cluster-$name.asok' 2024-09-15T21:14:19.727 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-15T21:14:19.728 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-scrub-repair/x '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.52602/$cluster-$name.asok' --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-15T21:14:19.759 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:99: TEST_corrupt_and_repair_replicated: run_osd td/osd-scrub-repair 0 2024-09-15T21:14:19.759 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-scrub-repair 2024-09-15T21:14:19.759 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-15T21:14:19.759 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=0 2024-09-15T21:14:19.759 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-15T21:14:19.759 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-scrub-repair/0 2024-09-15T21:14:19.759 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=ce4892c7-6c6d-4b4f-8e6f-07a3e11133e5 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-15T21:14:19.759 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T21:14:19.759 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T21:14:19.759 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T21:14:19.759 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-09-15T21:14:19.760 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-09-15T21:14:19.760 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-15T21:14:19.760 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-15T21:14:19.760 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-15T21:14:19.760 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-15T21:14:19.760 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-15T21:14:19.760 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-15T21:14:19.760 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-15T21:14:19.760 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T21:14:19.760 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52602 2024-09-15T21:14:19.760 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.52602/$cluster-$name.asok' 2024-09-15T21:14:19.760 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.52602/$cluster-$name.asok' 2024-09-15T21:14:19.760 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-15T21:14:19.760 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-15T21:14:19.761 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-15T21:14:19.761 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-15T21:14:19.761 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-15T21:14:19.761 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T21:14:19.761 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T21:14:19.761 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T21:14:19.761 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T21:14:19.761 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-15T21:14:19.761 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-15T21:14:19.762 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-scrub-repair/0 2024-09-15T21:14:19.762 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-15T21:14:19.762 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=1648d064-efe0-4075-8bd5-eb97534ec982 2024-09-15T21:14:19.762 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd0 1648d064-efe0-4075-8bd5-eb97534ec982' 2024-09-15T21:14:19.763 INFO:tasks.workunit.client.0.smithi136.stdout:add osd0 1648d064-efe0-4075-8bd5-eb97534ec982 2024-09-15T21:14:19.763 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-15T21:14:19.778 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQArTudmxgI4LhAAQwamLvqAiH9VpYSZfsqv+w== 2024-09-15T21:14:19.779 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQArTudmxgI4LhAAQwamLvqAiH9VpYSZfsqv+w=="}' 2024-09-15T21:14:19.779 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 1648d064-efe0-4075-8bd5-eb97534ec982 -i td/osd-scrub-repair/0/new.json 2024-09-15T21:14:19.982 INFO:tasks.workunit.client.0.smithi136.stdout:0 2024-09-15T21:14:19.993 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-scrub-repair/0/new.json 2024-09-15T21:14:19.994 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 0 --fsid=ce4892c7-6c6d-4b4f-8e6f-07a3e11133e5 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.52602/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQArTudmxgI4LhAAQwamLvqAiH9VpYSZfsqv+w== --osd-uuid 1648d064-efe0-4075-8bd5-eb97534ec982 2024-09-15T21:14:20.023 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:14:20.020+0000 7fbf7a81d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:14:20.033 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:14:20.030+0000 7fbf7a81d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:14:20.035 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:14:20.031+0000 7fbf7a81d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:14:20.036 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:14:20.032+0000 7fbf7a81d640 -1 bdev(0x55f0e8227c00 td/osd-scrub-repair/0/block) open stat got: (1) Operation not permitted 2024-09-15T21:14:20.036 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:14:20.032+0000 7fbf7a81d640 -1 bluestore(td/osd-scrub-repair/0) _read_fsid unparsable uuid 2024-09-15T21:14:25.796 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-scrub-repair/0/keyring 2024-09-15T21:14:25.797 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-15T21:14:25.797 INFO:tasks.workunit.client.0.smithi136.stdout:adding osd0 key to auth repository 2024-09-15T21:14:25.798 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd0 key to auth repository 2024-09-15T21:14:25.798 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-scrub-repair/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-15T21:14:26.163 INFO:tasks.workunit.client.0.smithi136.stdout:start osd.0 2024-09-15T21:14:26.164 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.0 2024-09-15T21:14:26.164 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 0 --fsid=ce4892c7-6c6d-4b4f-8e6f-07a3e11133e5 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.52602/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T21:14:26.164 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-15T21:14:26.164 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-15T21:14:26.164 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-15T21:14:26.191 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:14:26.188+0000 7fdd3bdbb640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:14:26.193 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:14:26.190+0000 7fdd3bdbb640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:14:26.194 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:14:26.191+0000 7fdd3bdbb640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:14:26.442 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 0 2024-09-15T21:14:26.442 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-15T21:14:26.442 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2024-09-15T21:14:26.442 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-15T21:14:26.442 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-15T21:14:26.442 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:14:26.442 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-15T21:14:26.443 INFO:tasks.workunit.client.0.smithi136.stdout:0 2024-09-15T21:14:26.443 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:14:26.443 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:14:26.715 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:14:27.717 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:14:27.717 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:14:27.717 INFO:tasks.workunit.client.0.smithi136.stdout:1 2024-09-15T21:14:27.718 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-15T21:14:27.718 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:14:27.718 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:14:28.001 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:14:28.036 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:14:28.033+0000 7fdd3bdbb640 -1 Falling back to public interface 2024-09-15T21:14:29.003 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:14:29.003 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:14:29.003 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-15T21:14:29.003 INFO:tasks.workunit.client.0.smithi136.stdout:2 2024-09-15T21:14:29.004 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:14:29.004 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:14:29.288 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:14:30.289 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:14:30.289 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:14:30.289 INFO:tasks.workunit.client.0.smithi136.stdout:3 2024-09-15T21:14:30.290 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-15T21:14:30.290 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:14:30.290 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:14:30.575 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:14:30.998 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:14:30.994+0000 7fdd3bdbb640 -1 osd.0 0 log_to_monitors true 2024-09-15T21:14:31.577 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:14:31.577 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:14:31.577 INFO:tasks.workunit.client.0.smithi136.stdout:4 2024-09-15T21:14:31.578 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-15T21:14:31.578 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:14:31.578 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:14:31.973 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:14:32.975 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:14:32.975 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:14:32.975 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-15T21:14:32.975 INFO:tasks.workunit.client.0.smithi136.stdout:5 2024-09-15T21:14:32.976 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:14:32.976 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:14:33.263 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:14:34.265 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:14:34.265 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:14:34.265 INFO:tasks.workunit.client.0.smithi136.stdout:6 2024-09-15T21:14:34.266 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-15T21:14:34.266 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:14:34.266 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:14:34.602 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:14:35.603 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:14:35.604 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:14:35.604 INFO:tasks.workunit.client.0.smithi136.stdout:7 2024-09-15T21:14:35.605 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-15T21:14:35.605 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:14:35.605 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:14:36.015 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:14:37.017 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:14:37.017 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:14:37.017 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-15T21:14:37.017 INFO:tasks.workunit.client.0.smithi136.stdout:8 2024-09-15T21:14:37.018 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:14:37.018 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:14:37.308 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:14:38.309 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:14:38.309 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:14:38.309 INFO:tasks.workunit.client.0.smithi136.stdout:9 2024-09-15T21:14:38.310 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-15T21:14:38.310 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:14:38.310 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:14:38.602 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:14:39.603 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:14:39.603 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:14:39.604 INFO:tasks.workunit.client.0.smithi136.stdout:10 2024-09-15T21:14:39.604 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-15T21:14:39.604 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:14:39.604 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:14:40.020 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:14:41.022 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:14:41.022 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:14:41.022 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-15T21:14:41.023 INFO:tasks.workunit.client.0.smithi136.stdout:11 2024-09-15T21:14:41.023 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:14:41.023 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:14:41.307 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:14:42.308 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:14:42.308 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:14:42.308 INFO:tasks.workunit.client.0.smithi136.stdout:12 2024-09-15T21:14:42.309 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-15T21:14:42.309 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:14:42.309 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:14:42.725 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:14:43.726 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:14:43.726 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:14:43.726 INFO:tasks.workunit.client.0.smithi136.stdout:13 2024-09-15T21:14:43.727 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-15T21:14:43.727 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:14:43.727 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:14:44.427 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:14:45.429 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:14:45.429 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:14:45.429 INFO:tasks.workunit.client.0.smithi136.stdout:14 2024-09-15T21:14:45.430 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-15T21:14:45.430 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:14:45.430 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:14:45.842 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:14:46.844 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:14:46.844 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:14:46.844 INFO:tasks.workunit.client.0.smithi136.stdout:15 2024-09-15T21:14:46.845 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-15T21:14:46.845 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:14:46.845 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:14:47.321 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:14:48.322 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:14:48.322 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:14:48.322 INFO:tasks.workunit.client.0.smithi136.stdout:16 2024-09-15T21:14:48.322 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-15T21:14:48.323 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:14:48.323 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:14:48.977 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:14:49.978 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:14:49.978 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:14:49.979 INFO:tasks.workunit.client.0.smithi136.stdout:17 2024-09-15T21:14:49.979 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-15T21:14:49.979 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:14:49.979 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:14:50.824 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:14:51.822 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:14:51.822 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:14:51.823 INFO:tasks.workunit.client.0.smithi136.stdout:18 2024-09-15T21:14:51.823 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 18 2024-09-15T21:14:51.823 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:14:51.823 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:14:52.515 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:14:53.517 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:14:53.517 INFO:tasks.workunit.client.0.smithi136.stdout:19 2024-09-15T21:14:53.517 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:14:53.518 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 19 2024-09-15T21:14:53.518 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:14:53.518 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:14:53.803 INFO:tasks.workunit.client.0.smithi136.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/1410482665,v1:127.0.0.1:6803/1410482665] [v2:127.0.0.1:6804/1410482665,v1:127.0.0.1:6805/1410482665] exists,up 1648d064-efe0-4075-8bd5-eb97534ec982 2024-09-15T21:14:53.807 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-15T21:14:53.807 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-15T21:14:53.807 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-15T21:14:53.807 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:100: TEST_corrupt_and_repair_replicated: run_osd td/osd-scrub-repair 1 2024-09-15T21:14:53.807 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-scrub-repair 2024-09-15T21:14:53.807 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-15T21:14:53.807 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=1 2024-09-15T21:14:53.807 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-15T21:14:53.807 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-scrub-repair/1 2024-09-15T21:14:53.807 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=ce4892c7-6c6d-4b4f-8e6f-07a3e11133e5 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-15T21:14:53.807 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T21:14:53.807 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T21:14:53.808 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T21:14:53.808 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-09-15T21:14:53.808 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-09-15T21:14:53.808 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-15T21:14:53.808 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-15T21:14:53.808 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-15T21:14:53.808 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-15T21:14:53.808 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-15T21:14:53.808 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-15T21:14:53.808 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-15T21:14:53.808 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T21:14:53.808 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52602 2024-09-15T21:14:53.808 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.52602/$cluster-$name.asok' 2024-09-15T21:14:53.809 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.52602/$cluster-$name.asok' 2024-09-15T21:14:53.809 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-15T21:14:53.809 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-15T21:14:53.809 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-15T21:14:53.809 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-15T21:14:53.809 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-15T21:14:53.809 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T21:14:53.809 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T21:14:53.809 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T21:14:53.809 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T21:14:53.809 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-15T21:14:53.809 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-15T21:14:53.809 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-scrub-repair/1 2024-09-15T21:14:53.809 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-15T21:14:53.810 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=175ec0d6-2e48-426a-b80f-e422eb3e2951 2024-09-15T21:14:53.810 INFO:tasks.workunit.client.0.smithi136.stdout:add osd1 175ec0d6-2e48-426a-b80f-e422eb3e2951 2024-09-15T21:14:53.810 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd1 175ec0d6-2e48-426a-b80f-e422eb3e2951' 2024-09-15T21:14:53.810 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-15T21:14:53.826 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQBNTudmYJkQMRAAimKlsOSivo6yptwmzmUd/A== 2024-09-15T21:14:53.826 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQBNTudmYJkQMRAAimKlsOSivo6yptwmzmUd/A=="}' 2024-09-15T21:14:53.826 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 175ec0d6-2e48-426a-b80f-e422eb3e2951 -i td/osd-scrub-repair/1/new.json 2024-09-15T21:14:54.158 INFO:tasks.workunit.client.0.smithi136.stdout:1 2024-09-15T21:14:54.170 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-scrub-repair/1/new.json 2024-09-15T21:14:54.171 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 1 --fsid=ce4892c7-6c6d-4b4f-8e6f-07a3e11133e5 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.52602/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBNTudmYJkQMRAAimKlsOSivo6yptwmzmUd/A== --osd-uuid 175ec0d6-2e48-426a-b80f-e422eb3e2951 2024-09-15T21:14:54.200 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:14:54.196+0000 7f29085b2640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:14:54.235 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:14:54.231+0000 7f29085b2640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:14:54.236 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:14:54.232+0000 7f29085b2640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:14:54.236 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:14:54.232+0000 7f29085b2640 -1 bdev(0x5582e7adfc00 td/osd-scrub-repair/1/block) open stat got: (1) Operation not permitted 2024-09-15T21:14:54.236 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:14:54.232+0000 7f29085b2640 -1 bluestore(td/osd-scrub-repair/1) _read_fsid unparsable uuid 2024-09-15T21:15:00.140 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-scrub-repair/1/keyring 2024-09-15T21:15:00.141 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-15T21:15:00.141 INFO:tasks.workunit.client.0.smithi136.stdout:adding osd1 key to auth repository 2024-09-15T21:15:00.142 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd1 key to auth repository 2024-09-15T21:15:00.142 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-scrub-repair/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-15T21:15:00.501 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.1 2024-09-15T21:15:00.501 INFO:tasks.workunit.client.0.smithi136.stdout:start osd.1 2024-09-15T21:15:00.502 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 1 --fsid=ce4892c7-6c6d-4b4f-8e6f-07a3e11133e5 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.52602/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T21:15:00.502 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-15T21:15:00.502 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-15T21:15:00.502 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-15T21:15:00.529 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:15:00.526+0000 7f53c200c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:15:00.531 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:15:00.528+0000 7f53c200c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:15:00.532 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:15:00.529+0000 7f53c200c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:15:00.780 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 1 2024-09-15T21:15:00.780 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-15T21:15:00.780 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2024-09-15T21:15:00.780 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-15T21:15:00.781 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-15T21:15:00.781 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:15:00.781 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-15T21:15:00.781 INFO:tasks.workunit.client.0.smithi136.stdout:0 2024-09-15T21:15:00.781 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:15:00.781 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T21:15:01.057 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:15:01.895 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:15:01.892+0000 7f53c200c640 -1 Falling back to public interface 2024-09-15T21:15:02.058 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:15:02.059 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:15:02.059 INFO:tasks.workunit.client.0.smithi136.stdout:1 2024-09-15T21:15:02.059 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-15T21:15:02.059 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:15:02.059 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T21:15:02.344 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:15:03.345 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:15:03.345 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:15:03.345 INFO:tasks.workunit.client.0.smithi136.stdout:2 2024-09-15T21:15:03.346 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-15T21:15:03.346 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:15:03.346 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T21:15:03.633 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:15:04.575 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:15:04.572+0000 7f53c200c640 -1 osd.1 0 log_to_monitors true 2024-09-15T21:15:04.635 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:15:04.635 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:15:04.635 INFO:tasks.workunit.client.0.smithi136.stdout:3 2024-09-15T21:15:04.635 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-15T21:15:04.636 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:15:04.636 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T21:15:05.067 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:15:06.069 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:15:06.069 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:15:06.069 INFO:tasks.workunit.client.0.smithi136.stdout:4 2024-09-15T21:15:06.069 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-15T21:15:06.069 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:15:06.070 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T21:15:06.378 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:15:07.379 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:15:07.379 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:15:07.380 INFO:tasks.workunit.client.0.smithi136.stdout:5 2024-09-15T21:15:07.380 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-15T21:15:07.380 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:15:07.380 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T21:15:07.667 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:15:08.668 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:15:08.668 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:15:08.669 INFO:tasks.workunit.client.0.smithi136.stdout:6 2024-09-15T21:15:08.669 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-15T21:15:08.670 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:15:08.670 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T21:15:08.959 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:15:09.961 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:15:09.961 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:15:09.961 INFO:tasks.workunit.client.0.smithi136.stdout:7 2024-09-15T21:15:09.962 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-15T21:15:09.962 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:15:09.962 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T21:15:10.380 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:15:11.382 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:15:11.382 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:15:11.382 INFO:tasks.workunit.client.0.smithi136.stdout:8 2024-09-15T21:15:11.383 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-15T21:15:11.383 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:15:11.383 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T21:15:11.670 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:15:12.671 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:15:12.672 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:15:12.672 INFO:tasks.workunit.client.0.smithi136.stdout:9 2024-09-15T21:15:12.672 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-15T21:15:12.672 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:15:12.673 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T21:15:12.959 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:15:13.961 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:15:13.961 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:15:13.961 INFO:tasks.workunit.client.0.smithi136.stdout:10 2024-09-15T21:15:13.962 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-15T21:15:13.962 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:15:13.962 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T21:15:14.345 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:15:15.347 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:15:15.347 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:15:15.347 INFO:tasks.workunit.client.0.smithi136.stdout:11 2024-09-15T21:15:15.348 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-15T21:15:15.348 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:15:15.348 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T21:15:15.637 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:15:16.638 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:15:16.638 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:15:16.639 INFO:tasks.workunit.client.0.smithi136.stdout:12 2024-09-15T21:15:16.639 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-15T21:15:16.639 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:15:16.639 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T21:15:16.912 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:15:17.914 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:15:17.914 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:15:17.914 INFO:tasks.workunit.client.0.smithi136.stdout:13 2024-09-15T21:15:17.915 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-15T21:15:17.915 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:15:17.915 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T21:15:18.200 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:15:19.201 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:15:19.201 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:15:19.202 INFO:tasks.workunit.client.0.smithi136.stdout:14 2024-09-15T21:15:19.202 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-15T21:15:19.202 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:15:19.202 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T21:15:19.837 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:15:20.838 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:15:20.838 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:15:20.839 INFO:tasks.workunit.client.0.smithi136.stdout:15 2024-09-15T21:15:20.839 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-15T21:15:20.839 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:15:20.839 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T21:15:21.277 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:15:22.278 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:15:22.278 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:15:22.278 INFO:tasks.workunit.client.0.smithi136.stdout:16 2024-09-15T21:15:22.279 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-15T21:15:22.279 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:15:22.279 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T21:15:23.083 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:15:24.085 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:15:24.085 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:15:24.085 INFO:tasks.workunit.client.0.smithi136.stdout:17 2024-09-15T21:15:24.086 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-15T21:15:24.086 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:15:24.086 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T21:15:24.894 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:15:25.896 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:15:25.896 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:15:25.896 INFO:tasks.workunit.client.0.smithi136.stdout:18 2024-09-15T21:15:25.897 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 18 2024-09-15T21:15:25.897 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:15:25.897 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T21:15:26.247 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:15:27.248 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:15:27.248 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:15:27.248 INFO:tasks.workunit.client.0.smithi136.stdout:19 2024-09-15T21:15:27.249 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 19 2024-09-15T21:15:27.249 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:15:27.249 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T21:15:27.538 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:15:28.539 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:15:28.539 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:15:28.540 INFO:tasks.workunit.client.0.smithi136.stdout:20 2024-09-15T21:15:28.540 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 20 2024-09-15T21:15:28.540 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:15:28.540 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T21:15:28.845 INFO:tasks.workunit.client.0.smithi136.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/3580207667,v1:127.0.0.1:6811/3580207667] [v2:127.0.0.1:6812/3580207667,v1:127.0.0.1:6813/3580207667] exists,up 175ec0d6-2e48-426a-b80f-e422eb3e2951 2024-09-15T21:15:28.846 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-15T21:15:28.846 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-15T21:15:28.846 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-15T21:15:28.846 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:101: TEST_corrupt_and_repair_replicated: create_rbd_pool 2024-09-15T21:15:28.846 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:526: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2024-09-15T21:15:29.121 INFO:tasks.workunit.client.0.smithi136.stderr:pool 'rbd' does not exist 2024-09-15T21:15:29.132 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:527: create_rbd_pool: create_pool rbd 4 2024-09-15T21:15:29.133 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create rbd 4 2024-09-15T21:15:29.509 INFO:tasks.workunit.client.0.smithi136.stderr:pool 'rbd' created 2024-09-15T21:15:29.521 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2024-09-15T21:15:30.523 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:528: create_rbd_pool: rbd pool init rbd 2024-09-15T21:15:32.381 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:102: TEST_corrupt_and_repair_replicated: wait_for_clean 2024-09-15T21:15:32.382 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-15T21:15:32.382 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-15T21:15:32.382 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-15T21:15:32.382 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-15T21:15:32.383 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-15T21:15:32.383 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-15T21:15:32.383 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-15T21:15:32.383 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-15T21:15:32.383 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-15T21:15:32.483 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-15T21:15:32.483 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-15T21:15:32.483 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-15T21:15:32.483 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-15T21:15:32.483 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-15T21:15:32.484 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-15T21:15:32.764 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-15T21:15:32.764 INFO:tasks.workunit.client.0.smithi136.stderr:1' 2024-09-15T21:15:32.764 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-15T21:15:32.764 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T21:15:32.764 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-15T21:15:32.868 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836490 2024-09-15T21:15:32.868 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836490 2024-09-15T21:15:32.868 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836490' 2024-09-15T21:15:32.868 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T21:15:32.869 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-15T21:15:32.973 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=42949672969 2024-09-15T21:15:32.973 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 42949672969 2024-09-15T21:15:32.973 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836490 1-42949672969' 2024-09-15T21:15:32.973 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T21:15:32.973 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836490 2024-09-15T21:15:32.973 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T21:15:32.973 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-15T21:15:32.974 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836490 2024-09-15T21:15:32.974 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T21:15:32.975 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836490 2024-09-15T21:15:32.975 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836490' 2024-09-15T21:15:32.975 INFO:tasks.workunit.client.0.smithi136.stdout:waiting osd.0 seq 21474836490 2024-09-15T21:15:32.976 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T21:15:33.252 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836489 -lt 21474836490 2024-09-15T21:15:33.252 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-15T21:15:34.252 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-15T21:15:34.252 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T21:15:34.536 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836491 -lt 21474836490 2024-09-15T21:15:34.536 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T21:15:34.537 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-42949672969 2024-09-15T21:15:34.537 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T21:15:34.537 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-15T21:15:34.538 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-42949672969 2024-09-15T21:15:34.538 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T21:15:34.539 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=42949672969 2024-09-15T21:15:34.540 INFO:tasks.workunit.client.0.smithi136.stdout:waiting osd.1 seq 42949672969 2024-09-15T21:15:34.540 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 42949672969' 2024-09-15T21:15:34.540 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-15T21:15:34.818 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 42949672970 -lt 42949672969 2024-09-15T21:15:34.818 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-15T21:15:34.819 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T21:15:34.819 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T21:15:35.222 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 4 == 0 2024-09-15T21:15:35.222 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-15T21:15:35.222 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-15T21:15:35.223 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-15T21:15:35.223 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T21:15:35.223 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T21:15:35.223 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T21:15:35.223 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T21:15:35.498 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=4 2024-09-15T21:15:35.499 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-15T21:15:35.499 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T21:15:35.499 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T21:15:35.861 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 = 4 2024-09-15T21:15:35.862 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-15T21:15:35.862 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-15T21:15:35.862 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:104: TEST_corrupt_and_repair_replicated: add_something td/osd-scrub-repair rbd 2024-09-15T21:15:35.862 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-09-15T21:15:35.862 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=rbd 2024-09-15T21:15:35.862 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=SOMETHING 2024-09-15T21:15:35.862 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-09-15T21:15:35.862 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-09-15T21:15:35.862 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-09-15T21:15:36.241 INFO:tasks.workunit.client.0.smithi136.stderr:noscrub is set 2024-09-15T21:15:36.253 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-09-15T21:15:36.811 INFO:tasks.workunit.client.0.smithi136.stderr:nodeep-scrub is set 2024-09-15T21:15:36.823 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-09-15T21:15:36.823 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-09-15T21:15:36.823 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool rbd put SOMETHING td/osd-scrub-repair/ORIGINAL 2024-09-15T21:15:37.048 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:105: TEST_corrupt_and_repair_replicated: get_not_primary rbd SOMETHING 2024-09-15T21:15:37.049 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1220: get_not_primary: local poolname=rbd 2024-09-15T21:15:37.049 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1221: get_not_primary: local objectname=SOMETHING 2024-09-15T21:15:37.049 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1223: get_not_primary: get_primary rbd SOMETHING 2024-09-15T21:15:37.049 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1187: get_primary: local poolname=rbd 2024-09-15T21:15:37.049 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1188: get_primary: local objectname=SOMETHING 2024-09-15T21:15:37.049 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1190: get_primary: ceph --format json osd map rbd SOMETHING 2024-09-15T21:15:37.049 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1191: get_primary: jq .acting_primary 2024-09-15T21:15:37.327 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1223: get_not_primary: local primary=1 2024-09-15T21:15:37.327 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1224: get_not_primary: ceph --format json osd map rbd SOMETHING 2024-09-15T21:15:37.327 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1225: get_not_primary: jq '.acting | map(select (. != 1)) | .[0]' 2024-09-15T21:15:37.606 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:105: TEST_corrupt_and_repair_replicated: corrupt_and_repair_one td/osd-scrub-repair rbd 0 2024-09-15T21:15:37.606 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:226: corrupt_and_repair_one: local dir=td/osd-scrub-repair 2024-09-15T21:15:37.607 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:227: corrupt_and_repair_one: local poolname=rbd 2024-09-15T21:15:37.607 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:228: corrupt_and_repair_one: local osd=0 2024-09-15T21:15:37.607 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:233: corrupt_and_repair_one: objectstore_tool td/osd-scrub-repair 0 SOMETHING remove 2024-09-15T21:15:37.607 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2024-09-15T21:15:37.607 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2024-09-15T21:15:37.607 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=0 2024-09-15T21:15:37.607 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2024-09-15T21:15:37.608 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 SOMETHING remove 2024-09-15T21:15:37.608 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-09-15T21:15:37.608 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2024-09-15T21:15:37.608 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=0 2024-09-15T21:15:37.608 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2024-09-15T21:15:37.608 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2024-09-15T21:15:37.609 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-15T21:15:37.609 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-15T21:15:37.609 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-15T21:15:37.609 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-15T21:15:37.609 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-15T21:15:37.910 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-15T21:15:37.910 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 SOMETHING remove 2024-09-15T21:15:37.911 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-15T21:15:37.911 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-15T21:15:37.911 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=0 2024-09-15T21:15:37.911 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-15T21:15:37.911 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-09-15T21:15:37.911 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 SOMETHING remove 2024-09-15T21:15:39.363 INFO:tasks.workunit.client.0.smithi136.stdout:remove #1:eb822e21:::SOMETHING:head# 2024-09-15T21:15:39.881 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 2024-09-15T21:15:39.881 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2024-09-15T21:15:39.881 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-15T21:15:39.881 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=0 2024-09-15T21:15:39.881 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-15T21:15:39.881 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-09-15T21:15:39.881 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=ce4892c7-6c6d-4b4f-8e6f-07a3e11133e5 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-15T21:15:39.881 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T21:15:39.881 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T21:15:39.882 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T21:15:39.882 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-09-15T21:15:39.882 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-09-15T21:15:39.882 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-15T21:15:39.882 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-15T21:15:39.882 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-15T21:15:39.882 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-15T21:15:39.882 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-15T21:15:39.882 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-15T21:15:39.882 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-15T21:15:39.882 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T21:15:39.882 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52602 2024-09-15T21:15:39.882 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.52602/$cluster-$name.asok' 2024-09-15T21:15:39.882 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.52602/$cluster-$name.asok' 2024-09-15T21:15:39.883 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-15T21:15:39.883 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-15T21:15:39.883 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-15T21:15:39.883 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T21:15:39.883 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T21:15:39.883 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T21:15:39.883 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T21:15:39.883 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-15T21:15:39.883 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-15T21:15:39.883 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-09-15T21:15:39.884 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.0 2024-09-15T21:15:39.884 INFO:tasks.workunit.client.0.smithi136.stderr:start osd.0 2024-09-15T21:15:39.884 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 0 --fsid=ce4892c7-6c6d-4b4f-8e6f-07a3e11133e5 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.52602/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T21:15:39.884 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-09-15T21:15:39.884 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 0 = 0 ']' 2024-09-15T21:15:39.884 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-15T21:15:39.885 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-15T21:15:39.885 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-15T21:15:39.912 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:15:39.907+0000 7f8ca780d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:15:39.925 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:15:39.919+0000 7f8ca780d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:15:39.925 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:15:39.921+0000 7f8ca780d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:15:40.168 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 0 2024-09-15T21:15:40.168 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-15T21:15:40.168 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2024-09-15T21:15:40.168 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-15T21:15:40.168 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-15T21:15:40.169 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:15:40.169 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-15T21:15:40.169 INFO:tasks.workunit.client.0.smithi136.stderr:0 2024-09-15T21:15:40.169 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:15:40.169 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:15:40.443 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:15:41.445 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:15:41.445 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:15:41.445 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-15T21:15:41.445 INFO:tasks.workunit.client.0.smithi136.stderr:1 2024-09-15T21:15:41.445 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:15:41.445 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:15:41.543 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:15:41.539+0000 7f8ca780d640 -1 Falling back to public interface 2024-09-15T21:15:41.736 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:15:42.737 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:15:42.737 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:15:42.737 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-15T21:15:42.738 INFO:tasks.workunit.client.0.smithi136.stderr:2 2024-09-15T21:15:42.738 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:15:42.738 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:15:43.025 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:15:44.027 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:15:44.027 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:15:44.027 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-15T21:15:44.027 INFO:tasks.workunit.client.0.smithi136.stderr:3 2024-09-15T21:15:44.027 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:15:44.027 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:15:44.307 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:15:44.321 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:15:44.317+0000 7f8ca780d640 -1 osd.0 21 log_to_monitors true 2024-09-15T21:15:45.309 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:15:45.309 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:15:45.309 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-15T21:15:45.309 INFO:tasks.workunit.client.0.smithi136.stderr:4 2024-09-15T21:15:45.309 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:15:45.310 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:15:45.591 INFO:tasks.workunit.client.0.smithi136.stderr:osd.0 up in weight 1 up_from 25 up_thru 25 down_at 22 last_clean_interval [5,21) [v2:127.0.0.1:6802/3496783797,v1:127.0.0.1:6803/3496783797] [v2:127.0.0.1:6804/3496783797,v1:127.0.0.1:6805/3496783797] exists,up 1648d064-efe0-4075-8bd5-eb97534ec982 2024-09-15T21:15:45.591 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-15T21:15:45.591 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-15T21:15:45.591 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-15T21:15:45.591 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2024-09-15T21:15:45.591 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-15T21:15:45.592 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-15T21:15:45.592 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-15T21:15:45.592 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-15T21:15:45.592 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-15T21:15:45.592 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-15T21:15:45.592 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-15T21:15:45.592 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-15T21:15:45.592 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-15T21:15:45.681 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-15T21:15:45.681 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-15T21:15:45.681 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-15T21:15:45.681 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-15T21:15:45.681 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-15T21:15:45.681 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-15T21:15:45.959 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-15T21:15:45.959 INFO:tasks.workunit.client.0.smithi136.stderr:1' 2024-09-15T21:15:45.959 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-15T21:15:45.959 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T21:15:45.959 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-15T21:15:46.062 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=107374182402 2024-09-15T21:15:46.062 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 107374182402 2024-09-15T21:15:46.062 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-107374182402' 2024-09-15T21:15:46.062 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T21:15:46.062 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-15T21:15:46.168 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=42949672973 2024-09-15T21:15:46.168 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 42949672973 2024-09-15T21:15:46.168 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-107374182402 1-42949672973' 2024-09-15T21:15:46.169 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T21:15:46.170 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-107374182402 2024-09-15T21:15:46.170 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T21:15:46.170 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-15T21:15:46.170 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-107374182402 2024-09-15T21:15:46.170 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T21:15:46.170 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=107374182402 2024-09-15T21:15:46.170 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 107374182402' 2024-09-15T21:15:46.170 INFO:tasks.workunit.client.0.smithi136.stderr:waiting osd.0 seq 107374182402 2024-09-15T21:15:46.170 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T21:15:46.445 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 1 -lt 107374182402 2024-09-15T21:15:46.445 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-15T21:15:47.446 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-15T21:15:47.447 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T21:15:47.728 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 1 -lt 107374182402 2024-09-15T21:15:47.728 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-15T21:15:48.729 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 299 -eq 0 ']' 2024-09-15T21:15:48.730 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T21:15:49.008 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 107374182402 -lt 107374182402 2024-09-15T21:15:49.008 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T21:15:49.009 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-42949672973 2024-09-15T21:15:49.009 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T21:15:49.009 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-15T21:15:49.010 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-42949672973 2024-09-15T21:15:49.010 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T21:15:49.011 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=42949672973 2024-09-15T21:15:49.011 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 42949672973' 2024-09-15T21:15:49.011 INFO:tasks.workunit.client.0.smithi136.stderr:waiting osd.1 seq 42949672973 2024-09-15T21:15:49.012 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-15T21:15:49.294 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 42949672973 -lt 42949672973 2024-09-15T21:15:49.295 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-15T21:15:49.295 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T21:15:49.295 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T21:15:49.639 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 4 == 0 2024-09-15T21:15:49.639 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-15T21:15:49.639 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-15T21:15:49.640 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-15T21:15:49.640 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T21:15:49.640 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T21:15:49.640 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T21:15:49.640 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T21:15:49.913 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=4 2024-09-15T21:15:49.913 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-15T21:15:49.913 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T21:15:49.913 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T21:15:50.256 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 = 4 2024-09-15T21:15:50.257 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-15T21:15:50.257 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-15T21:15:50.257 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:237: corrupt_and_repair_one: get_pg rbd SOMETHING 2024-09-15T21:15:50.257 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1084: get_pg: local poolname=rbd 2024-09-15T21:15:50.257 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1085: get_pg: local objectname=SOMETHING 2024-09-15T21:15:50.257 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1087: get_pg: ceph --format json osd map rbd SOMETHING 2024-09-15T21:15:50.257 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1087: get_pg: jq -r .pgid 2024-09-15T21:15:50.535 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:237: corrupt_and_repair_one: local pg=1.3 2024-09-15T21:15:50.535 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:238: corrupt_and_repair_one: repair 1.3 2024-09-15T21:15:50.535 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1846: repair: local pgid=1.3 2024-09-15T21:15:50.535 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1847: repair: get_last_scrub_stamp 1.3 2024-09-15T21:15:50.536 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.3 2024-09-15T21:15:50.536 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-15T21:15:50.536 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-15T21:15:50.536 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.3") | .last_scrub_stamp' 2024-09-15T21:15:50.809 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1847: repair: local last_scrub=2024-09-15T21:15:29.450930+0000 2024-09-15T21:15:50.810 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1848: repair: ceph pg repair 1.3 2024-09-15T21:15:51.092 INFO:tasks.workunit.client.0.smithi136.stderr:instructing pg 1.3 on osd.1 to repair 2024-09-15T21:15:51.104 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1849: repair: wait_for_scrub 1.3 2024-09-15T21:15:29.450930+0000 2024-09-15T21:15:51.104 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: local pgid=1.3 2024-09-15T21:15:51.104 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2018: wait_for_scrub: local last_scrub=2024-09-15T21:15:29.450930+0000 2024-09-15T21:15:51.104 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2019: wait_for_scrub: local sname=last_scrub_stamp 2024-09-15T21:15:51.104 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i=0 )) 2024-09-15T21:15:51.104 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-15T21:15:51.104 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.3 last_scrub_stamp 2024-09-15T21:15:51.104 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.3 2024-09-15T21:15:51.104 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-15T21:15:51.104 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-15T21:15:51.105 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.3") | .last_scrub_stamp' 2024-09-15T21:15:51.371 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-15T21:15:29.450930+0000 '>' 2024-09-15T21:15:29.450930+0000 2024-09-15T21:15:51.371 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-15T21:15:52.373 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-15T21:15:52.373 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-15T21:15:52.373 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.3 last_scrub_stamp 2024-09-15T21:15:52.374 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.3 2024-09-15T21:15:52.374 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-15T21:15:52.374 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-15T21:15:52.374 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.3") | .last_scrub_stamp' 2024-09-15T21:15:52.654 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-15T21:15:29.450930+0000 '>' 2024-09-15T21:15:29.450930+0000 2024-09-15T21:15:52.655 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-15T21:15:53.656 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-15T21:15:53.656 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-15T21:15:53.657 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.3 last_scrub_stamp 2024-09-15T21:15:53.657 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.3 2024-09-15T21:15:53.657 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-15T21:15:53.657 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-15T21:15:53.657 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.3") | .last_scrub_stamp' 2024-09-15T21:15:53.936 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-15T21:15:51.993374+0000 '>' 2024-09-15T21:15:29.450930+0000 2024-09-15T21:15:53.936 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2023: wait_for_scrub: return 0 2024-09-15T21:15:53.936 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1850: repair: sleep 2 2024-09-15T21:15:55.938 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:242: corrupt_and_repair_one: objectstore_tool td/osd-scrub-repair 1 SOMETHING list-attrs 2024-09-15T21:15:55.938 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2024-09-15T21:15:55.939 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2024-09-15T21:15:55.939 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=1 2024-09-15T21:15:55.939 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2024-09-15T21:15:55.939 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 SOMETHING list-attrs 2024-09-15T21:15:55.939 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-09-15T21:15:55.939 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2024-09-15T21:15:55.939 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=1 2024-09-15T21:15:55.939 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2024-09-15T21:15:55.939 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-09-15T21:15:55.939 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-15T21:15:55.939 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-15T21:15:55.939 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-15T21:15:55.939 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-15T21:15:55.940 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-15T21:15:56.247 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-15T21:15:56.247 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 SOMETHING list-attrs 2024-09-15T21:15:56.247 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-15T21:15:56.247 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-15T21:15:56.247 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=1 2024-09-15T21:15:56.247 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-15T21:15:56.247 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-09-15T21:15:56.247 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 SOMETHING list-attrs 2024-09-15T21:15:56.752 INFO:tasks.workunit.client.0.smithi136.stderr:Error getting attr on : 1.3_head,#-3:c0000000:::scrub_1.3:head#, (61) No data available 2024-09-15T21:15:56.752 INFO:tasks.workunit.client.0.smithi136.stdout:_ 2024-09-15T21:15:56.753 INFO:tasks.workunit.client.0.smithi136.stdout:snapset 2024-09-15T21:15:57.022 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2024-09-15T21:15:57.022 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2024-09-15T21:15:57.022 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-15T21:15:57.023 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=1 2024-09-15T21:15:57.023 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-15T21:15:57.023 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-09-15T21:15:57.023 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=ce4892c7-6c6d-4b4f-8e6f-07a3e11133e5 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-15T21:15:57.023 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T21:15:57.023 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T21:15:57.023 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T21:15:57.023 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-09-15T21:15:57.023 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-09-15T21:15:57.023 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-15T21:15:57.023 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-15T21:15:57.023 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-15T21:15:57.024 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-15T21:15:57.024 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-15T21:15:57.024 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-15T21:15:57.024 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-15T21:15:57.024 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T21:15:57.024 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52602 2024-09-15T21:15:57.024 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.52602/$cluster-$name.asok' 2024-09-15T21:15:57.024 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.52602/$cluster-$name.asok' 2024-09-15T21:15:57.024 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-15T21:15:57.024 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-15T21:15:57.024 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-15T21:15:57.024 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T21:15:57.024 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T21:15:57.024 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T21:15:57.025 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T21:15:57.025 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-15T21:15:57.025 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-15T21:15:57.025 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-09-15T21:15:57.025 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.1 2024-09-15T21:15:57.025 INFO:tasks.workunit.client.0.smithi136.stderr:start osd.1 2024-09-15T21:15:57.025 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 1 --fsid=ce4892c7-6c6d-4b4f-8e6f-07a3e11133e5 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.52602/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T21:15:57.025 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-09-15T21:15:57.026 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 1 = 1 ']' 2024-09-15T21:15:57.026 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-15T21:15:57.026 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-15T21:15:57.026 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-15T21:15:57.053 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:15:57.049+0000 7f1fbb1d9640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:15:57.055 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:15:57.052+0000 7f1fbb1d9640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:15:57.057 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:15:57.054+0000 7f1fbb1d9640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:15:57.309 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 1 2024-09-15T21:15:57.309 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-15T21:15:57.309 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2024-09-15T21:15:57.309 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-15T21:15:57.310 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-15T21:15:57.310 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:15:57.310 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-15T21:15:57.310 INFO:tasks.workunit.client.0.smithi136.stderr:0 2024-09-15T21:15:57.310 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:15:57.310 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T21:15:57.584 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:15:58.585 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:15:58.585 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:15:58.585 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-15T21:15:58.586 INFO:tasks.workunit.client.0.smithi136.stderr:1 2024-09-15T21:15:58.586 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:15:58.586 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T21:15:58.655 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:15:58.652+0000 7f1fbb1d9640 -1 Falling back to public interface 2024-09-15T21:15:58.866 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:15:59.868 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:15:59.868 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:15:59.868 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-15T21:15:59.868 INFO:tasks.workunit.client.0.smithi136.stderr:2 2024-09-15T21:15:59.868 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:15:59.868 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T21:16:00.146 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:16:01.147 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:16:01.147 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:16:01.147 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-15T21:16:01.147 INFO:tasks.workunit.client.0.smithi136.stderr:3 2024-09-15T21:16:01.148 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:16:01.148 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T21:16:01.423 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:16:01.854 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:16:01.851+0000 7f1fbb1d9640 -1 osd.1 26 log_to_monitors true 2024-09-15T21:16:02.425 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:16:02.425 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:16:02.425 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-15T21:16:02.425 INFO:tasks.workunit.client.0.smithi136.stderr:4 2024-09-15T21:16:02.425 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:16:02.425 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T21:16:02.780 INFO:tasks.workunit.client.0.smithi136.stderr:osd.1 up in weight 1 up_from 30 up_thru 30 down_at 27 last_clean_interval [10,26) [v2:127.0.0.1:6810/112916819,v1:127.0.0.1:6811/112916819] [v2:127.0.0.1:6812/112916819,v1:127.0.0.1:6813/112916819] exists,up 175ec0d6-2e48-426a-b80f-e422eb3e2951 2024-09-15T21:16:02.780 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-15T21:16:02.780 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-15T21:16:02.781 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-15T21:16:02.781 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2024-09-15T21:16:02.781 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-15T21:16:02.781 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-15T21:16:02.781 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-15T21:16:02.781 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-15T21:16:02.781 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-15T21:16:02.781 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-15T21:16:02.781 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-15T21:16:02.781 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-15T21:16:02.781 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-15T21:16:02.870 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-15T21:16:02.870 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-15T21:16:02.870 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-15T21:16:02.870 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-15T21:16:02.870 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-15T21:16:02.870 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-15T21:16:03.216 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-15T21:16:03.216 INFO:tasks.workunit.client.0.smithi136.stderr:1' 2024-09-15T21:16:03.216 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-15T21:16:03.217 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T21:16:03.217 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-15T21:16:03.319 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=107374182406 2024-09-15T21:16:03.319 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 107374182406 2024-09-15T21:16:03.319 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-107374182406' 2024-09-15T21:16:03.319 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T21:16:03.320 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-15T21:16:03.423 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=128849018882 2024-09-15T21:16:03.423 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 128849018882 2024-09-15T21:16:03.424 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-107374182406 1-128849018882' 2024-09-15T21:16:03.424 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T21:16:03.424 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-107374182406 2024-09-15T21:16:03.424 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T21:16:03.424 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-15T21:16:03.425 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-107374182406 2024-09-15T21:16:03.426 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T21:16:03.426 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=107374182406 2024-09-15T21:16:03.426 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 107374182406' 2024-09-15T21:16:03.426 INFO:tasks.workunit.client.0.smithi136.stderr:waiting osd.0 seq 107374182406 2024-09-15T21:16:03.426 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T21:16:03.703 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 107374182405 -lt 107374182406 2024-09-15T21:16:03.703 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-15T21:16:04.704 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-15T21:16:04.705 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T21:16:04.991 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 107374182406 -lt 107374182406 2024-09-15T21:16:04.991 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T21:16:04.991 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-128849018882 2024-09-15T21:16:04.991 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T21:16:04.992 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-15T21:16:04.992 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-128849018882 2024-09-15T21:16:04.993 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T21:16:04.994 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=128849018882 2024-09-15T21:16:04.994 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 128849018882' 2024-09-15T21:16:04.994 INFO:tasks.workunit.client.0.smithi136.stderr:waiting osd.1 seq 128849018882 2024-09-15T21:16:04.994 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-15T21:16:05.268 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 128849018882 -lt 128849018882 2024-09-15T21:16:05.269 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-15T21:16:05.269 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T21:16:05.269 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T21:16:05.614 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 4 == 0 2024-09-15T21:16:05.616 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-15T21:16:05.616 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-15T21:16:05.616 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-15T21:16:05.616 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T21:16:05.616 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T21:16:05.616 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T21:16:05.616 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T21:16:05.889 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=4 2024-09-15T21:16:05.890 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-15T21:16:05.890 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T21:16:05.890 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T21:16:06.233 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 = 4 2024-09-15T21:16:06.233 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-15T21:16:06.233 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-15T21:16:06.233 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:243: corrupt_and_repair_one: rados --pool rbd get SOMETHING td/osd-scrub-repair/COPY 2024-09-15T21:16:06.263 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:244: corrupt_and_repair_one: diff td/osd-scrub-repair/ORIGINAL td/osd-scrub-repair/COPY 2024-09-15T21:16:06.265 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:107: TEST_corrupt_and_repair_replicated: get_primary rbd SOMETHING 2024-09-15T21:16:06.265 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1187: get_primary: local poolname=rbd 2024-09-15T21:16:06.265 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1188: get_primary: local objectname=SOMETHING 2024-09-15T21:16:06.265 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1190: get_primary: ceph --format json osd map rbd SOMETHING 2024-09-15T21:16:06.265 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1191: get_primary: jq .acting_primary 2024-09-15T21:16:06.565 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:107: TEST_corrupt_and_repair_replicated: corrupt_and_repair_one td/osd-scrub-repair rbd 1 2024-09-15T21:16:06.565 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:226: corrupt_and_repair_one: local dir=td/osd-scrub-repair 2024-09-15T21:16:06.565 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:227: corrupt_and_repair_one: local poolname=rbd 2024-09-15T21:16:06.565 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:228: corrupt_and_repair_one: local osd=1 2024-09-15T21:16:06.566 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:233: corrupt_and_repair_one: objectstore_tool td/osd-scrub-repair 1 SOMETHING remove 2024-09-15T21:16:06.566 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2024-09-15T21:16:06.566 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2024-09-15T21:16:06.566 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=1 2024-09-15T21:16:06.566 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2024-09-15T21:16:06.566 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 SOMETHING remove 2024-09-15T21:16:06.566 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-09-15T21:16:06.566 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2024-09-15T21:16:06.566 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=1 2024-09-15T21:16:06.566 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2024-09-15T21:16:06.566 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-09-15T21:16:06.566 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-15T21:16:06.567 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-15T21:16:06.567 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-15T21:16:06.567 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-15T21:16:06.567 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-15T21:16:06.871 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-15T21:16:06.871 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 SOMETHING remove 2024-09-15T21:16:06.871 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-15T21:16:06.871 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-15T21:16:06.871 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=1 2024-09-15T21:16:06.871 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-15T21:16:06.871 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-09-15T21:16:06.872 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 SOMETHING remove 2024-09-15T21:16:08.380 INFO:tasks.workunit.client.0.smithi136.stdout:remove #1:eb822e21:::SOMETHING:head# 2024-09-15T21:16:08.899 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2024-09-15T21:16:08.899 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2024-09-15T21:16:08.900 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-15T21:16:08.900 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=1 2024-09-15T21:16:08.900 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-15T21:16:08.900 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-09-15T21:16:08.900 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=ce4892c7-6c6d-4b4f-8e6f-07a3e11133e5 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-15T21:16:08.900 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T21:16:08.900 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T21:16:08.900 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T21:16:08.900 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-09-15T21:16:08.900 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-09-15T21:16:08.900 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-15T21:16:08.900 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-15T21:16:08.900 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-15T21:16:08.900 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-15T21:16:08.901 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-15T21:16:08.901 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-15T21:16:08.901 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-15T21:16:08.901 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T21:16:08.901 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52602 2024-09-15T21:16:08.901 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.52602/$cluster-$name.asok' 2024-09-15T21:16:08.902 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.52602/$cluster-$name.asok' 2024-09-15T21:16:08.902 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-15T21:16:08.903 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-15T21:16:08.903 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-15T21:16:08.903 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T21:16:08.903 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T21:16:08.903 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T21:16:08.903 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T21:16:08.903 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-15T21:16:08.903 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-15T21:16:08.903 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-09-15T21:16:08.903 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.1 2024-09-15T21:16:08.904 INFO:tasks.workunit.client.0.smithi136.stderr:start osd.1 2024-09-15T21:16:08.904 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 1 --fsid=ce4892c7-6c6d-4b4f-8e6f-07a3e11133e5 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.52602/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T21:16:08.904 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-09-15T21:16:08.904 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 1 = 1 ']' 2024-09-15T21:16:08.904 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-15T21:16:08.905 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-15T21:16:08.905 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-15T21:16:08.933 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:16:08.929+0000 7f9fd791b640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:16:08.935 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:16:08.931+0000 7f9fd791b640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:16:08.936 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:16:08.933+0000 7f9fd791b640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:16:09.192 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 1 2024-09-15T21:16:09.192 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-15T21:16:09.192 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2024-09-15T21:16:09.192 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-15T21:16:09.192 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-15T21:16:09.192 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:16:09.192 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-15T21:16:09.193 INFO:tasks.workunit.client.0.smithi136.stderr:0 2024-09-15T21:16:09.193 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:16:09.193 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T21:16:09.468 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:16:10.292 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:16:10.289+0000 7f9fd791b640 -1 Falling back to public interface 2024-09-15T21:16:10.469 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:16:10.470 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:16:10.470 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-15T21:16:10.470 INFO:tasks.workunit.client.0.smithi136.stderr:1 2024-09-15T21:16:10.470 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:16:10.470 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T21:16:10.756 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:16:11.758 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:16:11.758 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:16:11.758 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-15T21:16:11.758 INFO:tasks.workunit.client.0.smithi136.stderr:2 2024-09-15T21:16:11.758 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:16:11.758 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T21:16:12.066 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:16:12.672 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:16:12.669+0000 7f9fd791b640 -1 osd.1 31 log_to_monitors true 2024-09-15T21:16:13.068 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:16:13.068 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:16:13.068 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-15T21:16:13.068 INFO:tasks.workunit.client.0.smithi136.stderr:3 2024-09-15T21:16:13.068 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:16:13.069 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T21:16:13.418 INFO:tasks.workunit.client.0.smithi136.stderr:osd.1 up in weight 1 up_from 35 up_thru 30 down_at 32 last_clean_interval [30,31) [v2:127.0.0.1:6810/2687731019,v1:127.0.0.1:6811/2687731019] [v2:127.0.0.1:6812/2687731019,v1:127.0.0.1:6813/2687731019] exists,up 175ec0d6-2e48-426a-b80f-e422eb3e2951 2024-09-15T21:16:13.418 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-15T21:16:13.418 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-15T21:16:13.418 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-15T21:16:13.419 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2024-09-15T21:16:13.419 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-15T21:16:13.419 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-15T21:16:13.419 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-15T21:16:13.419 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-15T21:16:13.420 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-15T21:16:13.420 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-15T21:16:13.420 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-15T21:16:13.420 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-15T21:16:13.420 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-15T21:16:13.510 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-15T21:16:13.510 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-15T21:16:13.510 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-15T21:16:13.511 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-15T21:16:13.511 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-15T21:16:13.511 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-15T21:16:13.913 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-15T21:16:13.913 INFO:tasks.workunit.client.0.smithi136.stderr:1' 2024-09-15T21:16:13.914 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-15T21:16:13.914 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T21:16:13.914 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-15T21:16:14.109 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=107374182409 2024-09-15T21:16:14.109 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 107374182409 2024-09-15T21:16:14.109 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-107374182409' 2024-09-15T21:16:14.109 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T21:16:14.109 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-15T21:16:14.234 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=150323855362 2024-09-15T21:16:14.234 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 150323855362 2024-09-15T21:16:14.234 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-107374182409 1-150323855362' 2024-09-15T21:16:14.234 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T21:16:14.234 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-107374182409 2024-09-15T21:16:14.234 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T21:16:14.235 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-15T21:16:14.236 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-107374182409 2024-09-15T21:16:14.236 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T21:16:14.237 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=107374182409 2024-09-15T21:16:14.237 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 107374182409' 2024-09-15T21:16:14.237 INFO:tasks.workunit.client.0.smithi136.stderr:waiting osd.0 seq 107374182409 2024-09-15T21:16:14.237 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T21:16:14.514 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 107374182408 -lt 107374182409 2024-09-15T21:16:14.515 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-15T21:16:15.515 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-15T21:16:15.516 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T21:16:15.811 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 107374182408 -lt 107374182409 2024-09-15T21:16:15.811 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-15T21:16:16.813 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 299 -eq 0 ']' 2024-09-15T21:16:16.813 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T21:16:17.101 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 107374182410 -lt 107374182409 2024-09-15T21:16:17.101 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T21:16:17.101 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-150323855362 2024-09-15T21:16:17.102 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T21:16:17.102 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-15T21:16:17.103 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-150323855362 2024-09-15T21:16:17.103 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T21:16:17.104 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=150323855362 2024-09-15T21:16:17.104 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 150323855362' 2024-09-15T21:16:17.104 INFO:tasks.workunit.client.0.smithi136.stderr:waiting osd.1 seq 150323855362 2024-09-15T21:16:17.105 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-15T21:16:17.377 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 150323855362 -lt 150323855362 2024-09-15T21:16:17.377 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-15T21:16:17.378 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T21:16:17.378 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T21:16:17.725 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 4 == 0 2024-09-15T21:16:17.725 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-15T21:16:17.726 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-15T21:16:17.726 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-15T21:16:17.726 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T21:16:17.726 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T21:16:17.726 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T21:16:17.726 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T21:16:18.001 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=4 2024-09-15T21:16:18.002 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-15T21:16:18.002 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T21:16:18.002 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T21:16:18.347 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 = 4 2024-09-15T21:16:18.347 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-15T21:16:18.347 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-15T21:16:18.348 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:237: corrupt_and_repair_one: get_pg rbd SOMETHING 2024-09-15T21:16:18.348 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1084: get_pg: local poolname=rbd 2024-09-15T21:16:18.348 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1085: get_pg: local objectname=SOMETHING 2024-09-15T21:16:18.348 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1087: get_pg: ceph --format json osd map rbd SOMETHING 2024-09-15T21:16:18.348 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1087: get_pg: jq -r .pgid 2024-09-15T21:16:18.625 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:237: corrupt_and_repair_one: local pg=1.3 2024-09-15T21:16:18.625 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:238: corrupt_and_repair_one: repair 1.3 2024-09-15T21:16:18.625 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1846: repair: local pgid=1.3 2024-09-15T21:16:18.625 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1847: repair: get_last_scrub_stamp 1.3 2024-09-15T21:16:18.625 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.3 2024-09-15T21:16:18.625 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-15T21:16:18.625 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-15T21:16:18.625 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.3") | .last_scrub_stamp' 2024-09-15T21:16:18.899 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1847: repair: local last_scrub=2024-09-15T21:15:51.993374+0000 2024-09-15T21:16:18.899 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1848: repair: ceph pg repair 1.3 2024-09-15T21:16:19.161 INFO:tasks.workunit.client.0.smithi136.stderr:instructing pg 1.3 on osd.1 to repair 2024-09-15T21:16:19.173 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1849: repair: wait_for_scrub 1.3 2024-09-15T21:15:51.993374+0000 2024-09-15T21:16:19.174 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: local pgid=1.3 2024-09-15T21:16:19.174 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2018: wait_for_scrub: local last_scrub=2024-09-15T21:15:51.993374+0000 2024-09-15T21:16:19.174 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2019: wait_for_scrub: local sname=last_scrub_stamp 2024-09-15T21:16:19.174 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i=0 )) 2024-09-15T21:16:19.174 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-15T21:16:19.174 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.3 last_scrub_stamp 2024-09-15T21:16:19.174 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.3 2024-09-15T21:16:19.174 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-15T21:16:19.174 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-15T21:16:19.174 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.3") | .last_scrub_stamp' 2024-09-15T21:16:19.446 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-15T21:15:51.993374+0000 '>' 2024-09-15T21:15:51.993374+0000 2024-09-15T21:16:19.446 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-15T21:16:20.447 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-15T21:16:20.447 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-15T21:16:20.448 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.3 last_scrub_stamp 2024-09-15T21:16:20.448 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.3 2024-09-15T21:16:20.448 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-15T21:16:20.448 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-15T21:16:20.448 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.3") | .last_scrub_stamp' 2024-09-15T21:16:20.730 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-15T21:15:51.993374+0000 '>' 2024-09-15T21:15:51.993374+0000 2024-09-15T21:16:20.731 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-15T21:16:21.731 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-15T21:16:21.732 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-15T21:16:21.732 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.3 last_scrub_stamp 2024-09-15T21:16:21.732 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.3 2024-09-15T21:16:21.733 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-15T21:16:21.733 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-15T21:16:21.733 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.3") | .last_scrub_stamp' 2024-09-15T21:16:22.017 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-15T21:15:51.993374+0000 '>' 2024-09-15T21:15:51.993374+0000 2024-09-15T21:16:22.017 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-15T21:16:23.017 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-15T21:16:23.018 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-15T21:16:23.018 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.3 last_scrub_stamp 2024-09-15T21:16:23.018 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.3 2024-09-15T21:16:23.019 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-15T21:16:23.019 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-15T21:16:23.019 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.3") | .last_scrub_stamp' 2024-09-15T21:16:23.301 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-15T21:15:51.993374+0000 '>' 2024-09-15T21:15:51.993374+0000 2024-09-15T21:16:23.301 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-15T21:16:24.302 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-15T21:16:24.302 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-15T21:16:24.303 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.3 last_scrub_stamp 2024-09-15T21:16:24.303 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.3 2024-09-15T21:16:24.303 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-15T21:16:24.303 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-15T21:16:24.303 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.3") | .last_scrub_stamp' 2024-09-15T21:16:24.587 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-15T21:16:19.732196+0000 '>' 2024-09-15T21:15:51.993374+0000 2024-09-15T21:16:24.588 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2023: wait_for_scrub: return 0 2024-09-15T21:16:24.588 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1850: repair: sleep 2 2024-09-15T21:16:26.591 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:242: corrupt_and_repair_one: objectstore_tool td/osd-scrub-repair 1 SOMETHING list-attrs 2024-09-15T21:16:26.591 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2024-09-15T21:16:26.591 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2024-09-15T21:16:26.592 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=1 2024-09-15T21:16:26.592 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2024-09-15T21:16:26.592 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 SOMETHING list-attrs 2024-09-15T21:16:26.592 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-09-15T21:16:26.592 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2024-09-15T21:16:26.592 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=1 2024-09-15T21:16:26.593 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2024-09-15T21:16:26.593 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-09-15T21:16:26.593 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-15T21:16:26.594 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-15T21:16:26.594 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-15T21:16:26.594 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-15T21:16:26.594 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-15T21:16:26.898 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-15T21:16:26.898 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 SOMETHING list-attrs 2024-09-15T21:16:26.898 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-15T21:16:26.898 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-15T21:16:26.898 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=1 2024-09-15T21:16:26.898 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-15T21:16:26.898 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-09-15T21:16:26.898 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 SOMETHING list-attrs 2024-09-15T21:16:27.348 INFO:tasks.workunit.client.0.smithi136.stderr:Error getting attr on : 1.3_head,#-3:c0000000:::scrub_1.3:head#, (61) No data available 2024-09-15T21:16:27.349 INFO:tasks.workunit.client.0.smithi136.stdout:_ 2024-09-15T21:16:27.349 INFO:tasks.workunit.client.0.smithi136.stdout:snapset 2024-09-15T21:16:27.617 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2024-09-15T21:16:27.617 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2024-09-15T21:16:27.617 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-15T21:16:27.617 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=1 2024-09-15T21:16:27.617 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-15T21:16:27.617 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-09-15T21:16:27.618 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=ce4892c7-6c6d-4b4f-8e6f-07a3e11133e5 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-15T21:16:27.618 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T21:16:27.618 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T21:16:27.618 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T21:16:27.618 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-09-15T21:16:27.618 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-09-15T21:16:27.618 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-15T21:16:27.618 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-15T21:16:27.618 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-15T21:16:27.618 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-15T21:16:27.618 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-15T21:16:27.619 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-15T21:16:27.619 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-15T21:16:27.619 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T21:16:27.619 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52602 2024-09-15T21:16:27.619 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.52602/$cluster-$name.asok' 2024-09-15T21:16:27.619 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.52602/$cluster-$name.asok' 2024-09-15T21:16:27.619 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-15T21:16:27.619 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-15T21:16:27.619 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-15T21:16:27.619 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T21:16:27.619 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T21:16:27.619 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T21:16:27.619 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T21:16:27.619 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-15T21:16:27.620 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-15T21:16:27.620 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-09-15T21:16:27.620 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.1 2024-09-15T21:16:27.621 INFO:tasks.workunit.client.0.smithi136.stderr:start osd.1 2024-09-15T21:16:27.621 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 1 --fsid=ce4892c7-6c6d-4b4f-8e6f-07a3e11133e5 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.52602/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T21:16:27.621 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-09-15T21:16:27.621 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 1 = 1 ']' 2024-09-15T21:16:27.621 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-15T21:16:27.621 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-15T21:16:27.621 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-15T21:16:27.648 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:16:27.644+0000 7fdbe6424640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:16:27.650 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:16:27.646+0000 7fdbe6424640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:16:27.651 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:16:27.647+0000 7fdbe6424640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:16:27.934 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 1 2024-09-15T21:16:27.934 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-15T21:16:27.934 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2024-09-15T21:16:27.934 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-15T21:16:27.934 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-15T21:16:27.934 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:16:27.934 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-15T21:16:27.934 INFO:tasks.workunit.client.0.smithi136.stderr:0 2024-09-15T21:16:27.934 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:16:27.934 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T21:16:28.210 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:16:28.771 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:16:28.767+0000 7fdbe6424640 -1 Falling back to public interface 2024-09-15T21:16:29.212 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:16:29.212 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:16:29.212 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-15T21:16:29.212 INFO:tasks.workunit.client.0.smithi136.stderr:1 2024-09-15T21:16:29.212 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:16:29.212 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T21:16:29.496 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:16:30.497 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:16:30.497 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:16:30.498 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-15T21:16:30.498 INFO:tasks.workunit.client.0.smithi136.stderr:2 2024-09-15T21:16:30.498 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:16:30.498 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T21:16:30.791 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:16:31.657 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:16:31.654+0000 7fdbe6424640 -1 osd.1 37 log_to_monitors true 2024-09-15T21:16:31.792 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:16:31.792 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:16:31.792 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-15T21:16:31.792 INFO:tasks.workunit.client.0.smithi136.stderr:3 2024-09-15T21:16:31.792 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:16:31.793 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T21:16:32.143 INFO:tasks.workunit.client.0.smithi136.stderr:osd.1 up in weight 1 up_from 41 up_thru 35 down_at 38 last_clean_interval [35,37) [v2:127.0.0.1:6810/1457266851,v1:127.0.0.1:6811/1457266851] [v2:127.0.0.1:6812/1457266851,v1:127.0.0.1:6813/1457266851] exists,up 175ec0d6-2e48-426a-b80f-e422eb3e2951 2024-09-15T21:16:32.143 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-15T21:16:32.143 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-15T21:16:32.143 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-15T21:16:32.143 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2024-09-15T21:16:32.143 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-15T21:16:32.143 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-15T21:16:32.143 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-15T21:16:32.143 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-15T21:16:32.144 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-15T21:16:32.144 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-15T21:16:32.144 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-15T21:16:32.144 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-15T21:16:32.144 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-15T21:16:32.233 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-15T21:16:32.234 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-15T21:16:32.234 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-15T21:16:32.234 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-15T21:16:32.234 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-15T21:16:32.234 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-15T21:16:32.575 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-15T21:16:32.575 INFO:tasks.workunit.client.0.smithi136.stderr:1' 2024-09-15T21:16:32.575 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-15T21:16:32.575 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T21:16:32.575 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-15T21:16:32.746 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=107374182414 2024-09-15T21:16:32.746 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 107374182414 2024-09-15T21:16:32.746 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-107374182414' 2024-09-15T21:16:32.746 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T21:16:32.747 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-15T21:16:32.911 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=176093659138 2024-09-15T21:16:32.912 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 176093659138 2024-09-15T21:16:32.912 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-107374182414 1-176093659138' 2024-09-15T21:16:32.912 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T21:16:32.912 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-107374182414 2024-09-15T21:16:32.912 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T21:16:32.913 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-15T21:16:32.914 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-107374182414 2024-09-15T21:16:32.914 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T21:16:32.914 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=107374182414 2024-09-15T21:16:32.915 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 107374182414' 2024-09-15T21:16:32.915 INFO:tasks.workunit.client.0.smithi136.stderr:waiting osd.0 seq 107374182414 2024-09-15T21:16:32.915 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T21:16:33.193 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 107374182413 -lt 107374182414 2024-09-15T21:16:33.194 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-15T21:16:34.195 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-15T21:16:34.195 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T21:16:34.482 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 107374182414 -lt 107374182414 2024-09-15T21:16:34.482 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T21:16:34.483 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-176093659138 2024-09-15T21:16:34.483 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T21:16:34.484 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-15T21:16:34.484 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-176093659138 2024-09-15T21:16:34.484 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T21:16:34.485 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=176093659138 2024-09-15T21:16:34.485 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 176093659138' 2024-09-15T21:16:34.485 INFO:tasks.workunit.client.0.smithi136.stderr:waiting osd.1 seq 176093659138 2024-09-15T21:16:34.486 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-15T21:16:34.762 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 176093659138 -lt 176093659138 2024-09-15T21:16:34.763 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-15T21:16:34.763 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T21:16:34.763 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T21:16:35.111 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 4 == 0 2024-09-15T21:16:35.112 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-15T21:16:35.112 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-15T21:16:35.112 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-15T21:16:35.112 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T21:16:35.112 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T21:16:35.112 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T21:16:35.112 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T21:16:35.386 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=2 2024-09-15T21:16:35.386 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-15T21:16:35.386 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T21:16:35.387 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T21:16:35.733 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 2 = 4 2024-09-15T21:16:35.733 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 2 '!=' -1 2024-09-15T21:16:35.733 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: loop=0 2024-09-15T21:16:35.733 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1652: wait_for_clean: num_active_clean=2 2024-09-15T21:16:35.733 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-15T21:16:35.734 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 0.1 2024-09-15T21:16:35.835 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-15T21:16:35.835 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-15T21:16:35.835 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-15T21:16:35.835 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-15T21:16:35.835 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T21:16:35.835 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T21:16:35.835 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T21:16:35.835 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T21:16:36.111 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=4 2024-09-15T21:16:36.111 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-15T21:16:36.112 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T21:16:36.112 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T21:16:36.457 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 = 4 2024-09-15T21:16:36.457 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-15T21:16:36.457 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-15T21:16:36.457 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:243: corrupt_and_repair_one: rados --pool rbd get SOMETHING td/osd-scrub-repair/COPY 2024-09-15T21:16:36.488 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:244: corrupt_and_repair_one: diff td/osd-scrub-repair/ORIGINAL td/osd-scrub-repair/COPY 2024-09-15T21:16:36.489 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:66: run: teardown td/osd-scrub-repair 2024-09-15T21:16:36.489 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/osd-scrub-repair 2024-09-15T21:16:36.489 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-15T21:16:36.489 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/osd-scrub-repair KILL 2024-09-15T21:16:36.489 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-15T21:16:36.489 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-15T21:16:36.490 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-15T21:16:36.490 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-15T21:16:36.490 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-15T21:16:36.597 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-15T21:16:36.597 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-15T21:16:36.597 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-15T21:16:36.598 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-15T21:16:36.599 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-15T21:16:36.599 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-15T21:16:36.599 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-15T21:16:36.600 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T21:16:36.600 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-15T21:16:36.600 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-15T21:16:36.600 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T21:16:36.601 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-15T21:16:36.602 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-15T21:16:36.602 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/osd-scrub-repair 2024-09-15T21:16:36.616 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-15T21:16:36.616 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T21:16:36.616 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52602 2024-09-15T21:16:36.616 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.52602 2024-09-15T21:16:36.618 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-15T21:16:36.618 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-15T21:16:36.618 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:63: run: for func in $funcs 2024-09-15T21:16:36.618 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:64: run: setup td/osd-scrub-repair 2024-09-15T21:16:36.618 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/osd-scrub-repair 2024-09-15T21:16:36.618 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/osd-scrub-repair 2024-09-15T21:16:36.618 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/osd-scrub-repair 2024-09-15T21:16:36.619 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-15T21:16:36.619 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/osd-scrub-repair KILL 2024-09-15T21:16:36.619 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-15T21:16:36.619 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-15T21:16:36.619 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-15T21:16:36.619 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-15T21:16:36.619 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-15T21:16:36.620 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-15T21:16:36.620 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-15T21:16:36.620 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-15T21:16:36.621 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-15T21:16:36.622 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-15T21:16:36.622 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-15T21:16:36.622 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-15T21:16:36.623 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T21:16:36.623 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-15T21:16:36.623 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-15T21:16:36.623 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T21:16:36.624 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-15T21:16:36.626 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-15T21:16:36.626 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/osd-scrub-repair 2024-09-15T21:16:36.627 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-15T21:16:36.628 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T21:16:36.628 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52602 2024-09-15T21:16:36.628 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.52602 2024-09-15T21:16:36.628 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-15T21:16:36.628 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-15T21:16:36.628 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/osd-scrub-repair 2024-09-15T21:16:36.629 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2024-09-15T21:16:36.631 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T21:16:36.631 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52602 2024-09-15T21:16:36.631 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.52602 2024-09-15T21:16:36.632 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2024-09-15T21:16:36.633 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 4096 -le 1024 ']' 2024-09-15T21:16:36.633 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2024-09-15T21:16:36.633 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/osd-scrub-repair 1' TERM HUP INT 2024-09-15T21:16:36.633 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:65: run: TEST_corrupt_scrub_erasure_appends td/osd-scrub-repair 2024-09-15T21:16:36.633 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5746: TEST_corrupt_scrub_erasure_appends: corrupt_scrub_erasure td/osd-scrub-repair false 2024-09-15T21:16:36.634 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3596: corrupt_scrub_erasure: local dir=td/osd-scrub-repair 2024-09-15T21:16:36.634 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3597: corrupt_scrub_erasure: local allow_overwrites=false 2024-09-15T21:16:36.634 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3598: corrupt_scrub_erasure: local poolname=ecpool 2024-09-15T21:16:36.634 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3599: corrupt_scrub_erasure: local total_objs=7 2024-09-15T21:16:36.634 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3601: corrupt_scrub_erasure: run_mon td/osd-scrub-repair a 2024-09-15T21:16:36.635 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/osd-scrub-repair 2024-09-15T21:16:36.635 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2024-09-15T21:16:36.635 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2024-09-15T21:16:36.635 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2024-09-15T21:16:36.635 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/osd-scrub-repair/a 2024-09-15T21:16:36.635 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-scrub-repair/a --run-dir=td/osd-scrub-repair 2024-09-15T21:16:37.069 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2024-09-15T21:16:37.069 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-15T21:16:37.069 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-15T21:16:37.069 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-15T21:16:37.069 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T21:16:37.069 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52602 2024-09-15T21:16:37.070 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.52602/$cluster-$name.asok' 2024-09-15T21:16:37.070 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-scrub-repair/a '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.52602/$cluster-$name.asok' --mon-cluster-log-file=td/osd-scrub-repair/log --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-09-15T21:16:37.261 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2024-09-15T21:16:37.262 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2024-09-15T21:16:37.262 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-15T21:16:37.262 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-15T21:16:37.262 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2024-09-15T21:16:37.263 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2024-09-15T21:16:37.263 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-15T21:16:37.263 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-15T21:16:37.263 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-15T21:16:37.263 INFO:tasks.workunit.client.0.smithi136.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-15T21:16:37.264 INFO:tasks.workunit.client.0.smithi136.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T21:16:37.264 INFO:tasks.workunit.client.0.smithi136.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52602 2024-09-15T21:16:37.264 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.52602/ceph-mon.a.asok 2024-09-15T21:16:37.264 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-15T21:16:37.264 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.52602/ceph-mon.a.asok config get fsid 2024-09-15T21:16:37.342 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2024-09-15T21:16:37.342 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-15T21:16:37.342 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-15T21:16:37.342 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2024-09-15T21:16:37.342 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2024-09-15T21:16:37.342 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-15T21:16:37.343 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-15T21:16:37.343 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-15T21:16:37.343 INFO:tasks.workunit.client.0.smithi136.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-15T21:16:37.343 INFO:tasks.workunit.client.0.smithi136.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T21:16:37.343 INFO:tasks.workunit.client.0.smithi136.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52602 2024-09-15T21:16:37.343 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.52602/ceph-mon.a.asok 2024-09-15T21:16:37.343 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-15T21:16:37.343 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.52602/ceph-mon.a.asok config get mon_host 2024-09-15T21:16:37.421 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3602: corrupt_scrub_erasure: run_mgr td/osd-scrub-repair x 2024-09-15T21:16:37.422 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: run_mgr: local dir=td/osd-scrub-repair 2024-09-15T21:16:37.422 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: run_mgr: shift 2024-09-15T21:16:37.422 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: run_mgr: local id=x 2024-09-15T21:16:37.422 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: run_mgr: shift 2024-09-15T21:16:37.422 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:548: run_mgr: local data=td/osd-scrub-repair/x 2024-09-15T21:16:37.422 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:550: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-15T21:16:38.230 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: get_asok_path 2024-09-15T21:16:38.230 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-15T21:16:38.230 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-15T21:16:38.231 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-15T21:16:38.231 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T21:16:38.231 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52602 2024-09-15T21:16:38.231 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.52602/$cluster-$name.asok' 2024-09-15T21:16:38.231 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-15T21:16:38.231 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-scrub-repair/x '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.52602/$cluster-$name.asok' --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-15T21:16:38.260 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3603: corrupt_scrub_erasure: seq 0 2 2024-09-15T21:16:38.263 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3603: corrupt_scrub_erasure: for id in $(seq 0 2) 2024-09-15T21:16:38.264 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3604: corrupt_scrub_erasure: run_osd td/osd-scrub-repair 0 2024-09-15T21:16:38.264 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-scrub-repair 2024-09-15T21:16:38.264 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-15T21:16:38.264 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=0 2024-09-15T21:16:38.264 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-15T21:16:38.264 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-scrub-repair/0 2024-09-15T21:16:38.264 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=ce4892c7-6c6d-4b4f-8e6f-07a3e11133e5 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-15T21:16:38.264 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T21:16:38.264 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T21:16:38.264 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T21:16:38.264 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-09-15T21:16:38.264 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-09-15T21:16:38.265 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-15T21:16:38.265 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-15T21:16:38.265 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-15T21:16:38.265 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-15T21:16:38.265 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-15T21:16:38.265 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-15T21:16:38.265 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-15T21:16:38.265 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T21:16:38.265 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52602 2024-09-15T21:16:38.265 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.52602/$cluster-$name.asok' 2024-09-15T21:16:38.265 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.52602/$cluster-$name.asok' 2024-09-15T21:16:38.265 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-15T21:16:38.265 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-15T21:16:38.265 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-15T21:16:38.265 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-15T21:16:38.266 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-15T21:16:38.266 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T21:16:38.266 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T21:16:38.266 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T21:16:38.266 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T21:16:38.266 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-15T21:16:38.266 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-15T21:16:38.266 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-scrub-repair/0 2024-09-15T21:16:38.266 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-15T21:16:38.267 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=9087e8dc-fe45-44af-af39-0644acbc5cf9 2024-09-15T21:16:38.267 INFO:tasks.workunit.client.0.smithi136.stdout:add osd0 9087e8dc-fe45-44af-af39-0644acbc5cf9 2024-09-15T21:16:38.267 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd0 9087e8dc-fe45-44af-af39-0644acbc5cf9' 2024-09-15T21:16:38.267 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-15T21:16:38.284 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQC2Tudm95a+EBAA3mT8jeZnVhCkzQOmyndnQA== 2024-09-15T21:16:38.284 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQC2Tudm95a+EBAA3mT8jeZnVhCkzQOmyndnQA=="}' 2024-09-15T21:16:38.284 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 9087e8dc-fe45-44af-af39-0644acbc5cf9 -i td/osd-scrub-repair/0/new.json 2024-09-15T21:16:38.485 INFO:tasks.workunit.client.0.smithi136.stdout:0 2024-09-15T21:16:38.495 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-scrub-repair/0/new.json 2024-09-15T21:16:38.496 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 0 --fsid=ce4892c7-6c6d-4b4f-8e6f-07a3e11133e5 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.52602/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQC2Tudm95a+EBAA3mT8jeZnVhCkzQOmyndnQA== --osd-uuid 9087e8dc-fe45-44af-af39-0644acbc5cf9 2024-09-15T21:16:38.525 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:16:38.522+0000 7f3be9618640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:16:38.536 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:16:38.533+0000 7f3be9618640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:16:38.537 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:16:38.534+0000 7f3be9618640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:16:38.537 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:16:38.534+0000 7f3be9618640 -1 bdev(0x55dd5e175c00 td/osd-scrub-repair/0/block) open stat got: (1) Operation not permitted 2024-09-15T21:16:38.537 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:16:38.534+0000 7f3be9618640 -1 bluestore(td/osd-scrub-repair/0) _read_fsid unparsable uuid 2024-09-15T21:16:44.325 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-scrub-repair/0/keyring 2024-09-15T21:16:44.325 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-15T21:16:44.326 INFO:tasks.workunit.client.0.smithi136.stdout:adding osd0 key to auth repository 2024-09-15T21:16:44.326 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd0 key to auth repository 2024-09-15T21:16:44.326 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-scrub-repair/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-15T21:16:44.704 INFO:tasks.workunit.client.0.smithi136.stdout:start osd.0 2024-09-15T21:16:44.705 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.0 2024-09-15T21:16:44.705 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 0 --fsid=ce4892c7-6c6d-4b4f-8e6f-07a3e11133e5 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.52602/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T21:16:44.705 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-15T21:16:44.705 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-15T21:16:44.705 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-15T21:16:44.732 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:16:44.729+0000 7f09b8810640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:16:44.734 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:16:44.731+0000 7f09b8810640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:16:44.735 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:16:44.732+0000 7f09b8810640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:16:44.985 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 0 2024-09-15T21:16:44.985 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-15T21:16:44.985 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2024-09-15T21:16:44.986 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-15T21:16:44.986 INFO:tasks.workunit.client.0.smithi136.stdout:0 2024-09-15T21:16:44.986 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-15T21:16:44.986 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:16:44.986 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-15T21:16:44.986 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:16:44.987 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:16:45.260 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:16:46.108 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:16:46.105+0000 7f09b8810640 -1 Falling back to public interface 2024-09-15T21:16:46.261 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:16:46.262 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:16:46.262 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-15T21:16:46.262 INFO:tasks.workunit.client.0.smithi136.stdout:1 2024-09-15T21:16:46.262 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:16:46.262 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:16:46.545 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:16:47.547 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:16:47.547 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:16:47.547 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-15T21:16:47.548 INFO:tasks.workunit.client.0.smithi136.stdout:2 2024-09-15T21:16:47.548 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:16:47.548 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:16:47.824 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:16:48.771 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:16:48.768+0000 7f09b8810640 -1 osd.0 0 log_to_monitors true 2024-09-15T21:16:48.826 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:16:48.826 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:16:48.826 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-15T21:16:48.826 INFO:tasks.workunit.client.0.smithi136.stdout:3 2024-09-15T21:16:48.827 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:16:48.827 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:16:49.215 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:16:50.215 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:16:50.216 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:16:50.216 INFO:tasks.workunit.client.0.smithi136.stdout:4 2024-09-15T21:16:50.216 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-15T21:16:50.217 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:16:50.217 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:16:50.525 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:16:51.527 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:16:51.527 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:16:51.527 INFO:tasks.workunit.client.0.smithi136.stdout:5 2024-09-15T21:16:51.528 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-15T21:16:51.528 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:16:51.528 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:16:51.814 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:16:52.816 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:16:52.816 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:16:52.817 INFO:tasks.workunit.client.0.smithi136.stdout:6 2024-09-15T21:16:52.818 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-15T21:16:52.818 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:16:52.818 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:16:53.106 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:16:54.107 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:16:54.107 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:16:54.107 INFO:tasks.workunit.client.0.smithi136.stdout:7 2024-09-15T21:16:54.107 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-15T21:16:54.108 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:16:54.108 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:16:54.393 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:16:55.394 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:16:55.394 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:16:55.394 INFO:tasks.workunit.client.0.smithi136.stdout:8 2024-09-15T21:16:55.395 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-15T21:16:55.395 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:16:55.395 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:16:55.729 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:16:56.730 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:16:56.730 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:16:56.730 INFO:tasks.workunit.client.0.smithi136.stdout:9 2024-09-15T21:16:56.731 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-15T21:16:56.731 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:16:56.731 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:16:57.015 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:16:58.017 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:16:58.017 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:16:58.017 INFO:tasks.workunit.client.0.smithi136.stdout:10 2024-09-15T21:16:58.018 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-15T21:16:58.018 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:16:58.018 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:16:58.307 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:16:59.308 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:16:59.309 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:16:59.309 INFO:tasks.workunit.client.0.smithi136.stdout:11 2024-09-15T21:16:59.309 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-15T21:16:59.309 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:16:59.309 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:16:59.604 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:17:00.605 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:17:00.605 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:17:00.605 INFO:tasks.workunit.client.0.smithi136.stdout:12 2024-09-15T21:17:00.606 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-15T21:17:00.606 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:17:00.606 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:17:00.996 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:17:01.998 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:17:01.998 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:17:01.999 INFO:tasks.workunit.client.0.smithi136.stdout:13 2024-09-15T21:17:01.999 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-15T21:17:01.999 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:17:02.000 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:17:02.845 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:17:03.847 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:17:03.847 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:17:03.847 INFO:tasks.workunit.client.0.smithi136.stdout:14 2024-09-15T21:17:03.848 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-15T21:17:03.848 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:17:03.848 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:17:04.347 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:17:05.349 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:17:05.349 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:17:05.349 INFO:tasks.workunit.client.0.smithi136.stdout:15 2024-09-15T21:17:05.350 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-15T21:17:05.350 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:17:05.351 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:17:05.936 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:17:06.937 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:17:06.937 INFO:tasks.workunit.client.0.smithi136.stdout:16 2024-09-15T21:17:06.938 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:17:06.938 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-15T21:17:06.938 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:17:06.938 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:17:07.604 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:17:08.606 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:17:08.606 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:17:08.607 INFO:tasks.workunit.client.0.smithi136.stdout:17 2024-09-15T21:17:08.607 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-15T21:17:08.608 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:17:08.608 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:17:08.945 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:17:09.947 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:17:09.947 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:17:09.947 INFO:tasks.workunit.client.0.smithi136.stdout:18 2024-09-15T21:17:09.948 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 18 2024-09-15T21:17:09.948 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:17:09.948 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:17:10.465 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:17:11.466 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:17:11.466 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:17:11.467 INFO:tasks.workunit.client.0.smithi136.stdout:19 2024-09-15T21:17:11.467 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 19 2024-09-15T21:17:11.467 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:17:11.467 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:17:11.756 INFO:tasks.workunit.client.0.smithi136.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/3000386302,v1:127.0.0.1:6803/3000386302] [v2:127.0.0.1:6804/3000386302,v1:127.0.0.1:6805/3000386302] exists,up 9087e8dc-fe45-44af-af39-0644acbc5cf9 2024-09-15T21:17:11.760 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-15T21:17:11.760 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-15T21:17:11.760 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-15T21:17:11.760 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3603: corrupt_scrub_erasure: for id in $(seq 0 2) 2024-09-15T21:17:11.760 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3604: corrupt_scrub_erasure: run_osd td/osd-scrub-repair 1 2024-09-15T21:17:11.761 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-scrub-repair 2024-09-15T21:17:11.761 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-15T21:17:11.761 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=1 2024-09-15T21:17:11.761 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-15T21:17:11.761 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-scrub-repair/1 2024-09-15T21:17:11.761 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=ce4892c7-6c6d-4b4f-8e6f-07a3e11133e5 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-15T21:17:11.761 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T21:17:11.761 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T21:17:11.761 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T21:17:11.761 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-09-15T21:17:11.761 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-09-15T21:17:11.761 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-15T21:17:11.761 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-15T21:17:11.761 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-15T21:17:11.762 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-15T21:17:11.762 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-15T21:17:11.762 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-15T21:17:11.762 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-15T21:17:11.762 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T21:17:11.762 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52602 2024-09-15T21:17:11.762 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.52602/$cluster-$name.asok' 2024-09-15T21:17:11.762 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.52602/$cluster-$name.asok' 2024-09-15T21:17:11.762 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-15T21:17:11.762 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-15T21:17:11.762 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-15T21:17:11.762 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-15T21:17:11.762 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-15T21:17:11.763 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T21:17:11.763 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T21:17:11.763 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T21:17:11.763 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T21:17:11.763 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-15T21:17:11.763 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-15T21:17:11.763 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-scrub-repair/1 2024-09-15T21:17:11.763 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-15T21:17:11.763 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=87e38427-d391-4909-9e45-e7b2d413fd8f 2024-09-15T21:17:11.763 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd1 87e38427-d391-4909-9e45-e7b2d413fd8f' 2024-09-15T21:17:11.763 INFO:tasks.workunit.client.0.smithi136.stdout:add osd1 87e38427-d391-4909-9e45-e7b2d413fd8f 2024-09-15T21:17:11.764 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-15T21:17:11.779 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQDXTudmLRY7LhAATI6zaNqBXUPUdH4TtxCGLQ== 2024-09-15T21:17:11.779 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQDXTudmLRY7LhAATI6zaNqBXUPUdH4TtxCGLQ=="}' 2024-09-15T21:17:11.779 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 87e38427-d391-4909-9e45-e7b2d413fd8f -i td/osd-scrub-repair/1/new.json 2024-09-15T21:17:12.185 INFO:tasks.workunit.client.0.smithi136.stdout:1 2024-09-15T21:17:12.196 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-scrub-repair/1/new.json 2024-09-15T21:17:12.197 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 1 --fsid=ce4892c7-6c6d-4b4f-8e6f-07a3e11133e5 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.52602/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDXTudmLRY7LhAATI6zaNqBXUPUdH4TtxCGLQ== --osd-uuid 87e38427-d391-4909-9e45-e7b2d413fd8f 2024-09-15T21:17:12.227 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:17:12.223+0000 7f6188ea5640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:17:12.270 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:17:12.265+0000 7f6188ea5640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:17:12.271 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:17:12.266+0000 7f6188ea5640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:17:12.271 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:17:12.267+0000 7f6188ea5640 -1 bdev(0x5648aff2bc00 td/osd-scrub-repair/1/block) open stat got: (1) Operation not permitted 2024-09-15T21:17:12.271 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:17:12.267+0000 7f6188ea5640 -1 bluestore(td/osd-scrub-repair/1) _read_fsid unparsable uuid 2024-09-15T21:17:18.503 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-scrub-repair/1/keyring 2024-09-15T21:17:18.503 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-15T21:17:18.504 INFO:tasks.workunit.client.0.smithi136.stdout:adding osd1 key to auth repository 2024-09-15T21:17:18.505 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd1 key to auth repository 2024-09-15T21:17:18.505 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-scrub-repair/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-15T21:17:18.867 INFO:tasks.workunit.client.0.smithi136.stdout:start osd.1 2024-09-15T21:17:18.868 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.1 2024-09-15T21:17:18.868 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 1 --fsid=ce4892c7-6c6d-4b4f-8e6f-07a3e11133e5 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.52602/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T21:17:18.868 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-15T21:17:18.868 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-15T21:17:18.868 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-15T21:17:18.894 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:17:18.891+0000 7f42dac0b640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:17:18.896 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:17:18.893+0000 7f42dac0b640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:17:18.897 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:17:18.894+0000 7f42dac0b640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:17:19.147 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 1 2024-09-15T21:17:19.147 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-15T21:17:19.147 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2024-09-15T21:17:19.147 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-15T21:17:19.147 INFO:tasks.workunit.client.0.smithi136.stdout:0 2024-09-15T21:17:19.148 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-15T21:17:19.148 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:17:19.148 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-15T21:17:19.148 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:17:19.148 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T21:17:19.423 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:17:20.259 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:17:20.256+0000 7f42dac0b640 -1 Falling back to public interface 2024-09-15T21:17:20.425 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:17:20.425 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:17:20.425 INFO:tasks.workunit.client.0.smithi136.stdout:1 2024-09-15T21:17:20.426 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-15T21:17:20.426 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:17:20.426 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T21:17:20.710 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:17:21.712 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:17:21.712 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:17:21.712 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-15T21:17:21.712 INFO:tasks.workunit.client.0.smithi136.stdout:2 2024-09-15T21:17:21.713 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:17:21.713 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T21:17:21.994 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:17:22.922 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:17:22.919+0000 7f42dac0b640 -1 osd.1 0 log_to_monitors true 2024-09-15T21:17:22.996 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:17:22.996 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:17:22.996 INFO:tasks.workunit.client.0.smithi136.stdout:3 2024-09-15T21:17:22.996 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-15T21:17:22.997 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:17:22.997 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T21:17:23.442 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:17:24.443 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:17:24.443 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:17:24.443 INFO:tasks.workunit.client.0.smithi136.stdout:4 2024-09-15T21:17:24.444 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-15T21:17:24.444 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:17:24.444 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T21:17:24.730 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:17:25.732 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:17:25.732 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:17:25.732 INFO:tasks.workunit.client.0.smithi136.stdout:5 2024-09-15T21:17:25.733 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-15T21:17:25.733 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:17:25.733 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T21:17:26.018 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:17:27.020 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:17:27.020 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:17:27.020 INFO:tasks.workunit.client.0.smithi136.stdout:6 2024-09-15T21:17:27.020 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-15T21:17:27.021 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:17:27.021 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T21:17:27.330 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:17:28.331 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:17:28.331 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:17:28.331 INFO:tasks.workunit.client.0.smithi136.stdout:7 2024-09-15T21:17:28.332 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-15T21:17:28.332 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:17:28.332 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T21:17:28.686 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:17:29.688 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:17:29.688 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:17:29.688 INFO:tasks.workunit.client.0.smithi136.stdout:8 2024-09-15T21:17:29.689 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-15T21:17:29.689 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:17:29.689 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T21:17:29.989 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:17:30.990 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:17:30.990 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:17:30.990 INFO:tasks.workunit.client.0.smithi136.stdout:9 2024-09-15T21:17:30.991 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-15T21:17:30.991 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:17:30.991 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T21:17:31.280 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:17:32.281 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:17:32.281 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:17:32.282 INFO:tasks.workunit.client.0.smithi136.stdout:10 2024-09-15T21:17:32.282 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-15T21:17:32.283 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:17:32.283 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T21:17:32.711 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:17:33.713 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:17:33.713 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:17:33.713 INFO:tasks.workunit.client.0.smithi136.stdout:11 2024-09-15T21:17:33.714 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-15T21:17:33.714 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:17:33.714 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T21:17:34.002 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:17:35.003 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:17:35.003 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:17:35.003 INFO:tasks.workunit.client.0.smithi136.stdout:12 2024-09-15T21:17:35.004 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-15T21:17:35.004 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:17:35.004 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T21:17:35.290 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:17:36.291 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:17:36.292 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:17:36.292 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-15T21:17:36.292 INFO:tasks.workunit.client.0.smithi136.stdout:13 2024-09-15T21:17:36.292 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:17:36.292 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T21:17:37.199 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:17:38.201 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:17:38.201 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:17:38.201 INFO:tasks.workunit.client.0.smithi136.stdout:14 2024-09-15T21:17:38.202 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-15T21:17:38.202 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:17:38.202 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T21:17:38.644 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:17:39.645 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:17:39.645 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:17:39.646 INFO:tasks.workunit.client.0.smithi136.stdout:15 2024-09-15T21:17:39.646 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-15T21:17:39.646 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:17:39.646 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T21:17:40.202 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:17:41.204 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:17:41.204 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:17:41.204 INFO:tasks.workunit.client.0.smithi136.stdout:16 2024-09-15T21:17:41.205 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-15T21:17:41.205 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:17:41.205 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T21:17:41.506 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:17:42.507 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:17:42.508 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:17:42.508 INFO:tasks.workunit.client.0.smithi136.stdout:17 2024-09-15T21:17:42.508 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-15T21:17:42.508 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:17:42.509 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T21:17:43.176 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:17:44.177 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:17:44.178 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:17:44.178 INFO:tasks.workunit.client.0.smithi136.stdout:18 2024-09-15T21:17:44.179 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 18 2024-09-15T21:17:44.179 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:17:44.179 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T21:17:44.695 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:17:45.696 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:17:45.696 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:17:45.697 INFO:tasks.workunit.client.0.smithi136.stdout:19 2024-09-15T21:17:45.697 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 19 2024-09-15T21:17:45.697 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:17:45.697 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T21:17:46.118 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:17:47.119 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:17:47.119 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:17:47.119 INFO:tasks.workunit.client.0.smithi136.stdout:20 2024-09-15T21:17:47.120 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 20 2024-09-15T21:17:47.120 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:17:47.120 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T21:17:47.406 INFO:tasks.workunit.client.0.smithi136.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/758479205,v1:127.0.0.1:6811/758479205] [v2:127.0.0.1:6812/758479205,v1:127.0.0.1:6813/758479205] exists,up 87e38427-d391-4909-9e45-e7b2d413fd8f 2024-09-15T21:17:47.410 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-15T21:17:47.410 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-15T21:17:47.410 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-15T21:17:47.410 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3603: corrupt_scrub_erasure: for id in $(seq 0 2) 2024-09-15T21:17:47.411 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3604: corrupt_scrub_erasure: run_osd td/osd-scrub-repair 2 2024-09-15T21:17:47.411 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-scrub-repair 2024-09-15T21:17:47.411 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-15T21:17:47.411 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=2 2024-09-15T21:17:47.411 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-15T21:17:47.411 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-scrub-repair/2 2024-09-15T21:17:47.411 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=ce4892c7-6c6d-4b4f-8e6f-07a3e11133e5 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-15T21:17:47.411 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T21:17:47.411 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T21:17:47.411 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T21:17:47.411 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/2' 2024-09-15T21:17:47.411 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/2/journal' 2024-09-15T21:17:47.411 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-15T21:17:47.411 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-15T21:17:47.412 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-15T21:17:47.412 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-15T21:17:47.412 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-15T21:17:47.412 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-15T21:17:47.412 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-15T21:17:47.412 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T21:17:47.412 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52602 2024-09-15T21:17:47.412 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.52602/$cluster-$name.asok' 2024-09-15T21:17:47.412 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.52602/$cluster-$name.asok' 2024-09-15T21:17:47.412 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-15T21:17:47.412 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-15T21:17:47.412 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-15T21:17:47.412 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-15T21:17:47.413 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-15T21:17:47.413 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T21:17:47.413 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T21:17:47.413 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T21:17:47.413 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T21:17:47.413 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-15T21:17:47.413 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-15T21:17:47.413 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-scrub-repair/2 2024-09-15T21:17:47.413 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-15T21:17:47.413 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=12467c91-a626-4bc5-ba50-e5426a3bf5b2 2024-09-15T21:17:47.413 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd2 12467c91-a626-4bc5-ba50-e5426a3bf5b2' 2024-09-15T21:17:47.414 INFO:tasks.workunit.client.0.smithi136.stdout:add osd2 12467c91-a626-4bc5-ba50-e5426a3bf5b2 2024-09-15T21:17:47.414 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-15T21:17:47.428 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQD7TudmkUZaGRAAFRgcey1LNtWq3+/c94EJvQ== 2024-09-15T21:17:47.429 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQD7TudmkUZaGRAAFRgcey1LNtWq3+/c94EJvQ=="}' 2024-09-15T21:17:47.429 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 12467c91-a626-4bc5-ba50-e5426a3bf5b2 -i td/osd-scrub-repair/2/new.json 2024-09-15T21:17:47.756 INFO:tasks.workunit.client.0.smithi136.stdout:2 2024-09-15T21:17:47.768 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-scrub-repair/2/new.json 2024-09-15T21:17:47.769 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 2 --fsid=ce4892c7-6c6d-4b4f-8e6f-07a3e11133e5 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.52602/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQD7TudmkUZaGRAAFRgcey1LNtWq3+/c94EJvQ== --osd-uuid 12467c91-a626-4bc5-ba50-e5426a3bf5b2 2024-09-15T21:17:47.798 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:17:47.794+0000 7f08144a5640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:17:47.845 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:17:47.841+0000 7f08144a5640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:17:47.846 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:17:47.843+0000 7f08144a5640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:17:47.846 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:17:47.843+0000 7f08144a5640 -1 bdev(0x556e19b9bc00 td/osd-scrub-repair/2/block) open stat got: (1) Operation not permitted 2024-09-15T21:17:47.846 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:17:47.843+0000 7f08144a5640 -1 bluestore(td/osd-scrub-repair/2) _read_fsid unparsable uuid 2024-09-15T21:17:54.089 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-scrub-repair/2/keyring 2024-09-15T21:17:54.089 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-15T21:17:54.089 INFO:tasks.workunit.client.0.smithi136.stdout:adding osd2 key to auth repository 2024-09-15T21:17:54.090 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd2 key to auth repository 2024-09-15T21:17:54.090 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-scrub-repair/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-15T21:17:54.451 INFO:tasks.workunit.client.0.smithi136.stdout:start osd.2 2024-09-15T21:17:54.451 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.2 2024-09-15T21:17:54.452 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 2 --fsid=ce4892c7-6c6d-4b4f-8e6f-07a3e11133e5 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.52602/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T21:17:54.452 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-15T21:17:54.452 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-15T21:17:54.452 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-15T21:17:54.478 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:17:54.475+0000 7ff6eed45640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:17:54.481 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:17:54.477+0000 7ff6eed45640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:17:54.482 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:17:54.478+0000 7ff6eed45640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:17:54.728 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 2 2024-09-15T21:17:54.728 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-15T21:17:54.728 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=2 2024-09-15T21:17:54.728 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-15T21:17:54.728 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-15T21:17:54.728 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:17:54.729 INFO:tasks.workunit.client.0.smithi136.stdout:0 2024-09-15T21:17:54.729 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-15T21:17:54.729 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:17:54.729 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T21:17:55.006 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:17:55.609 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:17:55.606+0000 7ff6eed45640 -1 Falling back to public interface 2024-09-15T21:17:56.007 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:17:56.007 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:17:56.007 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-15T21:17:56.008 INFO:tasks.workunit.client.0.smithi136.stdout:1 2024-09-15T21:17:56.008 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:17:56.008 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T21:17:56.301 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:17:57.302 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:17:57.302 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:17:57.302 INFO:tasks.workunit.client.0.smithi136.stdout:2 2024-09-15T21:17:57.303 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-15T21:17:57.303 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:17:57.303 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T21:17:57.592 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:17:58.306 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:17:58.302+0000 7ff6eed45640 -1 osd.2 0 log_to_monitors true 2024-09-15T21:17:58.594 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:17:58.594 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:17:58.594 INFO:tasks.workunit.client.0.smithi136.stdout:3 2024-09-15T21:17:58.594 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-15T21:17:58.595 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:17:58.595 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T21:17:59.026 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:18:00.027 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:18:00.028 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:18:00.028 INFO:tasks.workunit.client.0.smithi136.stdout:4 2024-09-15T21:18:00.028 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-15T21:18:00.029 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:18:00.029 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T21:18:00.374 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:18:01.375 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:18:01.376 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:18:01.376 INFO:tasks.workunit.client.0.smithi136.stdout:5 2024-09-15T21:18:01.376 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-15T21:18:01.376 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:18:01.377 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T21:18:01.667 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:18:02.669 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:18:02.669 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:18:02.669 INFO:tasks.workunit.client.0.smithi136.stdout:6 2024-09-15T21:18:02.669 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-15T21:18:02.670 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:18:02.670 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T21:18:02.991 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:18:03.992 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:18:03.993 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:18:03.993 INFO:tasks.workunit.client.0.smithi136.stdout:7 2024-09-15T21:18:03.993 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-15T21:18:03.993 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:18:03.994 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T21:18:04.381 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:18:05.382 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:18:05.382 INFO:tasks.workunit.client.0.smithi136.stdout:8 2024-09-15T21:18:05.383 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:18:05.383 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-15T21:18:05.383 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:18:05.383 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T21:18:05.670 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:18:06.671 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:18:06.671 INFO:tasks.workunit.client.0.smithi136.stdout:9 2024-09-15T21:18:06.672 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:18:06.672 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-15T21:18:06.672 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:18:06.672 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T21:18:06.962 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:18:07.963 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:18:07.963 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:18:07.964 INFO:tasks.workunit.client.0.smithi136.stdout:10 2024-09-15T21:18:07.964 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-15T21:18:07.964 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:18:07.964 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T21:18:08.383 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:18:09.385 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:18:09.385 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:18:09.385 INFO:tasks.workunit.client.0.smithi136.stdout:11 2024-09-15T21:18:09.386 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-15T21:18:09.386 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:18:09.386 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T21:18:09.671 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:18:10.672 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:18:10.673 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:18:10.673 INFO:tasks.workunit.client.0.smithi136.stdout:12 2024-09-15T21:18:10.673 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-15T21:18:10.674 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:18:10.674 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T21:18:11.040 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:18:12.042 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:18:12.042 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:18:12.042 INFO:tasks.workunit.client.0.smithi136.stdout:13 2024-09-15T21:18:12.043 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-15T21:18:12.043 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:18:12.043 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T21:18:12.543 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:18:13.545 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:18:13.545 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:18:13.545 INFO:tasks.workunit.client.0.smithi136.stdout:14 2024-09-15T21:18:13.546 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-15T21:18:13.546 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:18:13.546 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T21:18:14.238 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:18:15.239 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:18:15.239 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:18:15.239 INFO:tasks.workunit.client.0.smithi136.stdout:15 2024-09-15T21:18:15.240 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-15T21:18:15.240 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:18:15.240 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T21:18:15.949 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:18:16.950 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:18:16.950 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:18:16.951 INFO:tasks.workunit.client.0.smithi136.stdout:16 2024-09-15T21:18:16.951 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-15T21:18:16.951 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:18:16.951 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T21:18:17.684 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:18:18.686 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:18:18.686 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:18:18.686 INFO:tasks.workunit.client.0.smithi136.stdout:17 2024-09-15T21:18:18.687 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-15T21:18:18.687 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:18:18.687 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T21:18:19.396 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:18:20.397 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:18:20.398 INFO:tasks.workunit.client.0.smithi136.stdout:18 2024-09-15T21:18:20.398 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:18:20.398 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 18 2024-09-15T21:18:20.398 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:18:20.398 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T21:18:20.790 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:18:21.792 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:18:21.792 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:18:21.792 INFO:tasks.workunit.client.0.smithi136.stdout:19 2024-09-15T21:18:21.793 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 19 2024-09-15T21:18:21.793 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:18:21.793 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T21:18:22.069 INFO:tasks.workunit.client.0.smithi136.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/3516125687,v1:127.0.0.1:6819/3516125687] [v2:127.0.0.1:6820/3516125687,v1:127.0.0.1:6821/3516125687] exists,up 12467c91-a626-4bc5-ba50-e5426a3bf5b2 2024-09-15T21:18:22.070 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-15T21:18:22.070 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-15T21:18:22.070 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-15T21:18:22.070 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3606: corrupt_scrub_erasure: create_rbd_pool 2024-09-15T21:18:22.070 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:526: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2024-09-15T21:18:22.358 INFO:tasks.workunit.client.0.smithi136.stderr:pool 'rbd' does not exist 2024-09-15T21:18:22.369 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:527: create_rbd_pool: create_pool rbd 4 2024-09-15T21:18:22.369 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create rbd 4 2024-09-15T21:18:22.751 INFO:tasks.workunit.client.0.smithi136.stderr:pool 'rbd' created 2024-09-15T21:18:22.763 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2024-09-15T21:18:23.764 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:528: create_rbd_pool: rbd pool init rbd 2024-09-15T21:18:25.873 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3607: corrupt_scrub_erasure: create_pool foo 1 2024-09-15T21:18:25.874 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create foo 1 2024-09-15T21:18:26.263 INFO:tasks.workunit.client.0.smithi136.stderr:pool 'foo' created 2024-09-15T21:18:26.274 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2024-09-15T21:18:27.276 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3609: corrupt_scrub_erasure: create_ec_pool ecpool false k=2 m=1 stripe_unit=2K --force 2024-09-15T21:18:27.276 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2443: create_ec_pool: local pool_name=ecpool 2024-09-15T21:18:27.276 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2444: create_ec_pool: shift 2024-09-15T21:18:27.276 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2445: create_ec_pool: local allow_overwrites=false 2024-09-15T21:18:27.276 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2446: create_ec_pool: shift 2024-09-15T21:18:27.276 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2448: create_ec_pool: ceph osd erasure-code-profile set myprofile crush-failure-domain=osd k=2 m=1 stripe_unit=2K --force 2024-09-15T21:18:27.698 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2450: create_ec_pool: create_pool ecpool 1 1 erasure myprofile 2024-09-15T21:18:27.699 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create ecpool 1 1 erasure myprofile 2024-09-15T21:18:28.482 INFO:tasks.workunit.client.0.smithi136.stderr:pool 'ecpool' created 2024-09-15T21:18:28.494 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2024-09-15T21:18:29.498 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2452: create_ec_pool: '[' false = true ']' 2024-09-15T21:18:29.498 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2456: create_ec_pool: wait_for_clean 2024-09-15T21:18:29.498 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-15T21:18:29.498 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-15T21:18:29.498 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-15T21:18:29.499 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-15T21:18:29.499 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-15T21:18:29.499 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-15T21:18:29.499 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-15T21:18:29.499 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-15T21:18:29.499 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-15T21:18:29.600 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-15T21:18:29.600 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-15T21:18:29.600 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-15T21:18:29.601 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-15T21:18:29.601 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-15T21:18:29.601 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-15T21:18:29.918 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-15T21:18:29.918 INFO:tasks.workunit.client.0.smithi136.stderr:1 2024-09-15T21:18:29.918 INFO:tasks.workunit.client.0.smithi136.stderr:2' 2024-09-15T21:18:29.918 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-15T21:18:29.918 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T21:18:29.918 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-15T21:18:30.022 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836499 2024-09-15T21:18:30.022 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836499 2024-09-15T21:18:30.022 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836499' 2024-09-15T21:18:30.022 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T21:18:30.022 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-15T21:18:30.126 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=42949672977 2024-09-15T21:18:30.126 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 42949672977 2024-09-15T21:18:30.126 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836499 1-42949672977' 2024-09-15T21:18:30.126 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T21:18:30.126 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-15T21:18:30.230 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=64424509446 2024-09-15T21:18:30.230 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 64424509446 2024-09-15T21:18:30.230 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836499 1-42949672977 2-64424509446' 2024-09-15T21:18:30.230 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T21:18:30.231 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836499 2024-09-15T21:18:30.231 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T21:18:30.231 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-15T21:18:30.232 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836499 2024-09-15T21:18:30.232 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T21:18:30.233 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836499 2024-09-15T21:18:30.233 INFO:tasks.workunit.client.0.smithi136.stdout:waiting osd.0 seq 21474836499 2024-09-15T21:18:30.234 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836499' 2024-09-15T21:18:30.234 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T21:18:30.662 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836499 -lt 21474836499 2024-09-15T21:18:30.662 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T21:18:30.663 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-42949672977 2024-09-15T21:18:30.663 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T21:18:30.664 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-15T21:18:30.664 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-42949672977 2024-09-15T21:18:30.664 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T21:18:30.665 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=42949672977 2024-09-15T21:18:30.665 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 42949672977' 2024-09-15T21:18:30.666 INFO:tasks.workunit.client.0.smithi136.stdout:waiting osd.1 seq 42949672977 2024-09-15T21:18:30.666 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-15T21:18:30.961 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 42949672977 -lt 42949672977 2024-09-15T21:18:30.961 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T21:18:30.962 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-64424509446 2024-09-15T21:18:30.962 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T21:18:30.962 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-15T21:18:30.963 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-64424509446 2024-09-15T21:18:30.963 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T21:18:30.964 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=64424509446 2024-09-15T21:18:30.964 INFO:tasks.workunit.client.0.smithi136.stdout:waiting osd.2 seq 64424509446 2024-09-15T21:18:30.965 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 64424509446' 2024-09-15T21:18:30.965 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-15T21:18:31.246 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 64424509445 -lt 64424509446 2024-09-15T21:18:31.246 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-15T21:18:32.247 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-15T21:18:32.247 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-15T21:18:32.537 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 64424509447 -lt 64424509446 2024-09-15T21:18:32.538 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-15T21:18:32.538 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T21:18:32.538 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T21:18:32.885 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 6 == 0 2024-09-15T21:18:32.885 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-15T21:18:32.885 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-15T21:18:32.885 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-15T21:18:32.885 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T21:18:32.885 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T21:18:32.885 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T21:18:32.885 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T21:18:33.162 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=5 2024-09-15T21:18:33.162 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-15T21:18:33.163 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T21:18:33.163 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T21:18:33.509 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 = 6 2024-09-15T21:18:33.509 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 5 '!=' -1 2024-09-15T21:18:33.509 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: loop=0 2024-09-15T21:18:33.509 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1652: wait_for_clean: num_active_clean=5 2024-09-15T21:18:33.509 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-15T21:18:33.509 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 0.1 2024-09-15T21:18:33.610 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-15T21:18:33.610 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-15T21:18:33.611 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-15T21:18:33.611 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-15T21:18:33.611 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T21:18:33.611 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T21:18:33.611 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T21:18:33.611 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T21:18:33.887 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=5 2024-09-15T21:18:33.888 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-15T21:18:33.888 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T21:18:33.888 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T21:18:34.232 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 = 6 2024-09-15T21:18:34.232 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 5 '!=' 5 2024-09-15T21:18:34.232 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: get_is_making_recovery_progress 2024-09-15T21:18:34.232 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2024-09-15T21:18:34.232 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-15T21:18:34.232 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-15T21:18:34.232 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-15T21:18:34.232 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2024-09-15T21:18:34.232 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-15T21:18:34.575 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=null 2024-09-15T21:18:34.575 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test null '!=' null 2024-09-15T21:18:34.575 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1655: wait_for_clean: (( 1 >= 13 )) 2024-09-15T21:18:34.575 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-15T21:18:34.575 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 0.2 2024-09-15T21:18:34.776 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-15T21:18:34.777 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-15T21:18:34.777 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-15T21:18:34.777 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-15T21:18:34.777 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T21:18:34.777 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T21:18:34.777 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T21:18:34.777 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T21:18:35.049 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=6 2024-09-15T21:18:35.049 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-15T21:18:35.049 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T21:18:35.049 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T21:18:35.392 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 = 6 2024-09-15T21:18:35.392 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-15T21:18:35.392 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-15T21:18:35.392 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2457: create_ec_pool: return 0 2024-09-15T21:18:35.392 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3610: corrupt_scrub_erasure: wait_for_clean 2024-09-15T21:18:35.392 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-15T21:18:35.392 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-15T21:18:35.393 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-15T21:18:35.393 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-15T21:18:35.393 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-15T21:18:35.393 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-15T21:18:35.393 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-15T21:18:35.393 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-15T21:18:35.393 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-15T21:18:35.484 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-15T21:18:35.484 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-15T21:18:35.484 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-15T21:18:35.484 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-15T21:18:35.484 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-15T21:18:35.485 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-15T21:18:35.778 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-15T21:18:35.778 INFO:tasks.workunit.client.0.smithi136.stderr:1 2024-09-15T21:18:35.778 INFO:tasks.workunit.client.0.smithi136.stderr:2' 2024-09-15T21:18:35.778 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-15T21:18:35.778 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T21:18:35.778 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-15T21:18:35.881 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836501 2024-09-15T21:18:35.881 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836501 2024-09-15T21:18:35.882 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836501' 2024-09-15T21:18:35.882 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T21:18:35.882 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-15T21:18:35.985 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=42949672979 2024-09-15T21:18:35.985 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 42949672979 2024-09-15T21:18:35.986 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836501 1-42949672979' 2024-09-15T21:18:35.986 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T21:18:35.986 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-15T21:18:36.089 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=64424509448 2024-09-15T21:18:36.089 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 64424509448 2024-09-15T21:18:36.089 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836501 1-42949672979 2-64424509448' 2024-09-15T21:18:36.089 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T21:18:36.090 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836501 2024-09-15T21:18:36.090 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T21:18:36.090 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-15T21:18:36.091 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836501 2024-09-15T21:18:36.091 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T21:18:36.092 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836501 2024-09-15T21:18:36.092 INFO:tasks.workunit.client.0.smithi136.stdout:waiting osd.0 seq 21474836501 2024-09-15T21:18:36.092 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836501' 2024-09-15T21:18:36.093 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T21:18:36.381 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836501 -lt 21474836501 2024-09-15T21:18:36.381 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T21:18:36.381 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-42949672979 2024-09-15T21:18:36.381 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T21:18:36.382 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-15T21:18:36.382 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-42949672979 2024-09-15T21:18:36.383 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T21:18:36.384 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=42949672979 2024-09-15T21:18:36.384 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 42949672979' 2024-09-15T21:18:36.384 INFO:tasks.workunit.client.0.smithi136.stdout:waiting osd.1 seq 42949672979 2024-09-15T21:18:36.384 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-15T21:18:36.661 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 42949672979 -lt 42949672979 2024-09-15T21:18:36.661 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T21:18:36.661 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-64424509448 2024-09-15T21:18:36.661 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T21:18:36.661 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-15T21:18:36.662 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-64424509448 2024-09-15T21:18:36.662 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T21:18:36.663 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=64424509448 2024-09-15T21:18:36.663 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 64424509448' 2024-09-15T21:18:36.664 INFO:tasks.workunit.client.0.smithi136.stdout:waiting osd.2 seq 64424509448 2024-09-15T21:18:36.664 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-15T21:18:36.972 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 64424509448 -lt 64424509448 2024-09-15T21:18:36.972 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-15T21:18:36.972 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T21:18:36.972 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T21:18:37.316 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 6 == 0 2024-09-15T21:18:37.316 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-15T21:18:37.316 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-15T21:18:37.316 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-15T21:18:37.316 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T21:18:37.316 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T21:18:37.317 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T21:18:37.317 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T21:18:37.591 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=6 2024-09-15T21:18:37.591 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-15T21:18:37.591 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T21:18:37.591 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T21:18:37.934 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 = 6 2024-09-15T21:18:37.934 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-15T21:18:37.934 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-15T21:18:37.934 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3612: corrupt_scrub_erasure: seq 1 7 2024-09-15T21:18:37.935 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3612: corrupt_scrub_erasure: for i in $(seq 1 $total_objs) 2024-09-15T21:18:37.935 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3613: corrupt_scrub_erasure: objname=EOBJ1 2024-09-15T21:18:37.935 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3614: corrupt_scrub_erasure: add_something td/osd-scrub-repair ecpool EOBJ1 2024-09-15T21:18:37.935 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-09-15T21:18:37.935 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=ecpool 2024-09-15T21:18:37.936 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=EOBJ1 2024-09-15T21:18:37.936 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-09-15T21:18:37.936 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-09-15T21:18:37.936 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-09-15T21:18:38.306 INFO:tasks.workunit.client.0.smithi136.stderr:noscrub is set 2024-09-15T21:18:38.318 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-09-15T21:18:38.825 INFO:tasks.workunit.client.0.smithi136.stderr:nodeep-scrub is set 2024-09-15T21:18:38.837 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-09-15T21:18:38.837 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-09-15T21:18:38.837 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool ecpool put EOBJ1 td/osd-scrub-repair/ORIGINAL 2024-09-15T21:18:39.146 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3616: corrupt_scrub_erasure: expr 1 % 2 2024-09-15T21:18:39.149 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3616: corrupt_scrub_erasure: local osd=1 2024-09-15T21:18:39.149 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3618: corrupt_scrub_erasure: case $i in 2024-09-15T21:18:39.149 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3621: corrupt_scrub_erasure: local payload=UVWXYZZZ 2024-09-15T21:18:39.149 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3622: corrupt_scrub_erasure: echo UVWXYZZZ 2024-09-15T21:18:39.149 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3623: corrupt_scrub_erasure: objectstore_tool td/osd-scrub-repair 1 EOBJ1 set-bytes td/osd-scrub-repair/CORRUPT 2024-09-15T21:18:39.149 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2024-09-15T21:18:39.150 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2024-09-15T21:18:39.150 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=1 2024-09-15T21:18:39.150 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2024-09-15T21:18:39.150 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 EOBJ1 set-bytes td/osd-scrub-repair/CORRUPT 2024-09-15T21:18:39.150 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-09-15T21:18:39.150 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2024-09-15T21:18:39.150 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=1 2024-09-15T21:18:39.150 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2024-09-15T21:18:39.150 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-09-15T21:18:39.150 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-15T21:18:39.150 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-15T21:18:39.150 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-15T21:18:39.150 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-15T21:18:39.150 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-15T21:18:39.456 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-15T21:18:39.456 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 EOBJ1 set-bytes td/osd-scrub-repair/CORRUPT 2024-09-15T21:18:39.456 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-15T21:18:39.456 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-15T21:18:39.456 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=1 2024-09-15T21:18:39.456 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-15T21:18:39.456 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-09-15T21:18:39.456 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 EOBJ1 set-bytes td/osd-scrub-repair/CORRUPT 2024-09-15T21:18:41.975 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2024-09-15T21:18:41.975 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2024-09-15T21:18:41.975 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-15T21:18:41.975 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=1 2024-09-15T21:18:41.975 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-15T21:18:41.975 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-09-15T21:18:41.975 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=ce4892c7-6c6d-4b4f-8e6f-07a3e11133e5 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-15T21:18:41.975 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T21:18:41.976 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T21:18:41.976 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T21:18:41.976 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-09-15T21:18:41.976 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-09-15T21:18:41.976 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-15T21:18:41.976 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-15T21:18:41.976 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-15T21:18:41.976 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-15T21:18:41.976 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-15T21:18:41.976 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-15T21:18:41.976 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-15T21:18:41.976 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T21:18:41.976 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52602 2024-09-15T21:18:41.976 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.52602/$cluster-$name.asok' 2024-09-15T21:18:41.977 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.52602/$cluster-$name.asok' 2024-09-15T21:18:41.977 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-15T21:18:41.977 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-15T21:18:41.977 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-15T21:18:41.977 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T21:18:41.977 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T21:18:41.977 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T21:18:41.977 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T21:18:41.977 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-15T21:18:41.977 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-15T21:18:41.977 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-09-15T21:18:41.978 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.1 2024-09-15T21:18:41.978 INFO:tasks.workunit.client.0.smithi136.stderr:start osd.1 2024-09-15T21:18:41.978 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 1 --fsid=ce4892c7-6c6d-4b4f-8e6f-07a3e11133e5 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.52602/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T21:18:41.978 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-09-15T21:18:41.978 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 1 = 1 ']' 2024-09-15T21:18:41.978 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-15T21:18:41.979 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-15T21:18:41.979 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-15T21:18:42.006 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:18:42.002+0000 7fb444219640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:18:42.008 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:18:42.005+0000 7fb444219640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:18:42.009 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:18:42.006+0000 7fb444219640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:18:42.267 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 1 2024-09-15T21:18:42.267 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-15T21:18:42.267 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2024-09-15T21:18:42.267 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-15T21:18:42.267 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-15T21:18:42.267 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:18:42.267 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-15T21:18:42.267 INFO:tasks.workunit.client.0.smithi136.stderr:0 2024-09-15T21:18:42.267 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:18:42.267 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T21:18:42.597 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:18:43.148 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:18:43.145+0000 7fb444219640 -1 Falling back to public interface 2024-09-15T21:18:43.599 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:18:43.599 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:18:43.599 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-15T21:18:43.599 INFO:tasks.workunit.client.0.smithi136.stderr:1 2024-09-15T21:18:43.599 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:18:43.599 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T21:18:43.879 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:18:44.880 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:18:44.880 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:18:44.881 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-15T21:18:44.881 INFO:tasks.workunit.client.0.smithi136.stderr:2 2024-09-15T21:18:44.881 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:18:44.881 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T21:18:45.170 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:18:45.812 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:18:45.808+0000 7fb444219640 -1 osd.1 35 log_to_monitors true 2024-09-15T21:18:46.171 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:18:46.171 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:18:46.171 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-15T21:18:46.172 INFO:tasks.workunit.client.0.smithi136.stderr:3 2024-09-15T21:18:46.172 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:18:46.172 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T21:18:46.628 INFO:tasks.workunit.client.0.smithi136.stderr:osd.1 up in weight 1 up_from 40 up_thru 31 down_at 36 last_clean_interval [10,35) [v2:127.0.0.1:6810/407381366,v1:127.0.0.1:6811/407381366] [v2:127.0.0.1:6812/407381366,v1:127.0.0.1:6813/407381366] exists,up 87e38427-d391-4909-9e45-e7b2d413fd8f 2024-09-15T21:18:46.628 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-15T21:18:46.628 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-15T21:18:46.628 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-15T21:18:46.628 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2024-09-15T21:18:46.628 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-15T21:18:46.628 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-15T21:18:46.628 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-15T21:18:46.628 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-15T21:18:46.629 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-15T21:18:46.629 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-15T21:18:46.629 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-15T21:18:46.629 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-15T21:18:46.629 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-15T21:18:46.719 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-15T21:18:46.719 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-15T21:18:46.720 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-15T21:18:46.720 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-15T21:18:46.720 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-15T21:18:46.720 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-15T21:18:47.069 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-15T21:18:47.070 INFO:tasks.workunit.client.0.smithi136.stderr:1 2024-09-15T21:18:47.070 INFO:tasks.workunit.client.0.smithi136.stderr:2' 2024-09-15T21:18:47.070 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-15T21:18:47.070 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T21:18:47.070 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-15T21:18:47.417 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836505 2024-09-15T21:18:47.417 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836505 2024-09-15T21:18:47.417 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836505' 2024-09-15T21:18:47.417 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T21:18:47.417 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-15T21:18:47.614 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=171798691842 2024-09-15T21:18:47.614 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 171798691842 2024-09-15T21:18:47.614 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836505 1-171798691842' 2024-09-15T21:18:47.614 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T21:18:47.614 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-15T21:18:47.718 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=64424509452 2024-09-15T21:18:47.718 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 64424509452 2024-09-15T21:18:47.718 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836505 1-171798691842 2-64424509452' 2024-09-15T21:18:47.718 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T21:18:47.718 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836505 2024-09-15T21:18:47.718 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T21:18:47.719 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-15T21:18:47.719 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836505 2024-09-15T21:18:47.720 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T21:18:47.721 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836505 2024-09-15T21:18:47.721 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836505' 2024-09-15T21:18:47.721 INFO:tasks.workunit.client.0.smithi136.stderr:waiting osd.0 seq 21474836505 2024-09-15T21:18:47.721 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T21:18:48.002 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836503 -lt 21474836505 2024-09-15T21:18:48.003 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-15T21:18:49.004 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-15T21:18:49.004 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T21:18:49.291 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836505 -lt 21474836505 2024-09-15T21:18:49.291 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T21:18:49.292 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-171798691842 2024-09-15T21:18:49.292 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T21:18:49.292 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-15T21:18:49.293 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-171798691842 2024-09-15T21:18:49.293 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T21:18:49.294 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=171798691842 2024-09-15T21:18:49.294 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 171798691842' 2024-09-15T21:18:49.294 INFO:tasks.workunit.client.0.smithi136.stderr:waiting osd.1 seq 171798691842 2024-09-15T21:18:49.294 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-15T21:18:49.570 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 171798691842 -lt 171798691842 2024-09-15T21:18:49.570 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T21:18:49.571 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-64424509452 2024-09-15T21:18:49.571 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T21:18:49.572 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-15T21:18:49.572 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-64424509452 2024-09-15T21:18:49.573 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T21:18:49.573 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=64424509452 2024-09-15T21:18:49.573 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 64424509452' 2024-09-15T21:18:49.574 INFO:tasks.workunit.client.0.smithi136.stderr:waiting osd.2 seq 64424509452 2024-09-15T21:18:49.574 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-15T21:18:49.850 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 64424509452 -lt 64424509452 2024-09-15T21:18:49.851 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-15T21:18:49.851 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T21:18:49.851 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T21:18:50.198 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 6 == 0 2024-09-15T21:18:50.199 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-15T21:18:50.199 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-15T21:18:50.199 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-15T21:18:50.199 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T21:18:50.199 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T21:18:50.199 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T21:18:50.199 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T21:18:50.473 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=3 2024-09-15T21:18:50.473 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-15T21:18:50.473 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T21:18:50.473 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T21:18:50.818 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 3 = 6 2024-09-15T21:18:50.818 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 3 '!=' -1 2024-09-15T21:18:50.818 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: loop=0 2024-09-15T21:18:50.818 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1652: wait_for_clean: num_active_clean=3 2024-09-15T21:18:50.818 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-15T21:18:50.818 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 0.1 2024-09-15T21:18:50.919 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-15T21:18:50.919 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-15T21:18:50.919 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-15T21:18:50.920 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-15T21:18:50.920 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T21:18:50.920 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T21:18:50.920 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T21:18:50.920 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T21:18:51.194 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=3 2024-09-15T21:18:51.194 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-15T21:18:51.195 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T21:18:51.195 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T21:18:51.541 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 3 = 6 2024-09-15T21:18:51.541 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 3 '!=' 3 2024-09-15T21:18:51.542 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: get_is_making_recovery_progress 2024-09-15T21:18:51.542 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2024-09-15T21:18:51.542 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-15T21:18:51.542 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-15T21:18:51.542 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-15T21:18:51.542 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2024-09-15T21:18:51.542 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-15T21:18:51.884 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=null 2024-09-15T21:18:51.884 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test null '!=' null 2024-09-15T21:18:51.885 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1655: wait_for_clean: (( 1 >= 13 )) 2024-09-15T21:18:51.885 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-15T21:18:51.885 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 0.2 2024-09-15T21:18:52.085 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-15T21:18:52.085 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-15T21:18:52.086 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-15T21:18:52.086 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-15T21:18:52.086 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T21:18:52.086 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T21:18:52.086 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T21:18:52.086 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T21:18:52.359 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=6 2024-09-15T21:18:52.360 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-15T21:18:52.360 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T21:18:52.360 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T21:18:52.704 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 = 6 2024-09-15T21:18:52.704 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-15T21:18:52.704 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-15T21:18:52.704 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3612: corrupt_scrub_erasure: for i in $(seq 1 $total_objs) 2024-09-15T21:18:52.704 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3613: corrupt_scrub_erasure: objname=EOBJ2 2024-09-15T21:18:52.704 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3614: corrupt_scrub_erasure: add_something td/osd-scrub-repair ecpool EOBJ2 2024-09-15T21:18:52.704 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-09-15T21:18:52.704 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=ecpool 2024-09-15T21:18:52.704 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=EOBJ2 2024-09-15T21:18:52.704 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-09-15T21:18:52.705 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-09-15T21:18:52.705 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-09-15T21:18:53.077 INFO:tasks.workunit.client.0.smithi136.stderr:noscrub is set 2024-09-15T21:18:53.088 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-09-15T21:18:53.587 INFO:tasks.workunit.client.0.smithi136.stderr:nodeep-scrub is set 2024-09-15T21:18:53.599 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-09-15T21:18:53.599 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-09-15T21:18:53.599 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool ecpool put EOBJ2 td/osd-scrub-repair/ORIGINAL 2024-09-15T21:18:53.807 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3616: corrupt_scrub_erasure: expr 2 % 2 2024-09-15T21:18:53.808 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3616: corrupt_scrub_erasure: local osd=0 2024-09-15T21:18:53.809 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3618: corrupt_scrub_erasure: case $i in 2024-09-15T21:18:53.809 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3628: corrupt_scrub_erasure: dd if=/dev/urandom of=td/osd-scrub-repair/CORRUPT bs=2048 count=1 2024-09-15T21:18:53.810 INFO:tasks.workunit.client.0.smithi136.stderr:1+0 records in 2024-09-15T21:18:53.811 INFO:tasks.workunit.client.0.smithi136.stderr:1+0 records out 2024-09-15T21:18:53.811 INFO:tasks.workunit.client.0.smithi136.stderr:2048 bytes (2.0 kB, 2.0 KiB) copied, 6.4595e-05 s, 31.7 MB/s 2024-09-15T21:18:53.811 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3629: corrupt_scrub_erasure: objectstore_tool td/osd-scrub-repair 0 EOBJ2 set-bytes td/osd-scrub-repair/CORRUPT 2024-09-15T21:18:53.811 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2024-09-15T21:18:53.811 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2024-09-15T21:18:53.811 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=0 2024-09-15T21:18:53.811 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2024-09-15T21:18:53.811 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 EOBJ2 set-bytes td/osd-scrub-repair/CORRUPT 2024-09-15T21:18:53.811 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-09-15T21:18:53.811 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2024-09-15T21:18:53.811 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=0 2024-09-15T21:18:53.811 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2024-09-15T21:18:53.812 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2024-09-15T21:18:53.812 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-15T21:18:53.812 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-15T21:18:53.812 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-15T21:18:53.812 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-15T21:18:53.812 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-15T21:18:54.117 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-15T21:18:54.117 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 EOBJ2 set-bytes td/osd-scrub-repair/CORRUPT 2024-09-15T21:18:54.117 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-15T21:18:54.117 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-15T21:18:54.118 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=0 2024-09-15T21:18:54.118 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-15T21:18:54.118 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-09-15T21:18:54.118 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 EOBJ2 set-bytes td/osd-scrub-repair/CORRUPT 2024-09-15T21:18:56.427 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 2024-09-15T21:18:56.428 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2024-09-15T21:18:56.428 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-15T21:18:56.428 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=0 2024-09-15T21:18:56.428 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-15T21:18:56.428 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-09-15T21:18:56.428 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=ce4892c7-6c6d-4b4f-8e6f-07a3e11133e5 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-15T21:18:56.428 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T21:18:56.428 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T21:18:56.428 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T21:18:56.428 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-09-15T21:18:56.429 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-09-15T21:18:56.429 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-15T21:18:56.429 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-15T21:18:56.429 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-15T21:18:56.429 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-15T21:18:56.429 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-15T21:18:56.429 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-15T21:18:56.429 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-15T21:18:56.429 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T21:18:56.429 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52602 2024-09-15T21:18:56.429 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.52602/$cluster-$name.asok' 2024-09-15T21:18:56.429 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.52602/$cluster-$name.asok' 2024-09-15T21:18:56.430 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-15T21:18:56.430 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-15T21:18:56.430 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-15T21:18:56.430 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T21:18:56.430 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T21:18:56.430 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T21:18:56.430 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T21:18:56.430 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-15T21:18:56.430 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-15T21:18:56.430 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-09-15T21:18:56.431 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.0 2024-09-15T21:18:56.431 INFO:tasks.workunit.client.0.smithi136.stderr:start osd.0 2024-09-15T21:18:56.431 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 0 --fsid=ce4892c7-6c6d-4b4f-8e6f-07a3e11133e5 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.52602/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T21:18:56.431 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-09-15T21:18:56.431 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 0 = 0 ']' 2024-09-15T21:18:56.432 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-15T21:18:56.432 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-15T21:18:56.432 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-15T21:18:56.459 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:18:56.456+0000 7f9608119640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:18:56.462 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:18:56.458+0000 7f9608119640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:18:56.463 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:18:56.460+0000 7f9608119640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:18:56.717 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 0 2024-09-15T21:18:56.718 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-15T21:18:56.718 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2024-09-15T21:18:56.718 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-15T21:18:56.718 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-15T21:18:56.718 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:18:56.718 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-15T21:18:56.718 INFO:tasks.workunit.client.0.smithi136.stderr:0 2024-09-15T21:18:56.718 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:18:56.718 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:18:56.993 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:18:57.577 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:18:57.574+0000 7f9608119640 -1 Falling back to public interface 2024-09-15T21:18:57.995 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:18:57.995 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:18:57.995 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-15T21:18:57.995 INFO:tasks.workunit.client.0.smithi136.stderr:1 2024-09-15T21:18:57.995 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:18:57.995 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:18:58.307 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:18:59.308 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:18:59.308 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:18:59.308 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-15T21:18:59.308 INFO:tasks.workunit.client.0.smithi136.stderr:2 2024-09-15T21:18:59.308 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:18:59.309 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:18:59.606 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:19:00.058 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:19:00.055+0000 7f9608119640 -1 osd.0 44 log_to_monitors true 2024-09-15T21:19:00.608 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:19:00.608 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:19:00.608 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-15T21:19:00.608 INFO:tasks.workunit.client.0.smithi136.stderr:3 2024-09-15T21:19:00.608 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:19:00.608 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:19:00.982 INFO:tasks.workunit.client.0.smithi136.stderr:osd.0 up in weight 1 up_from 49 up_thru 40 down_at 45 last_clean_interval [5,44) [v2:127.0.0.1:6802/2720334194,v1:127.0.0.1:6803/2720334194] [v2:127.0.0.1:6804/2720334194,v1:127.0.0.1:6805/2720334194] exists,up 9087e8dc-fe45-44af-af39-0644acbc5cf9 2024-09-15T21:19:00.982 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-15T21:19:00.982 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-15T21:19:00.982 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-15T21:19:00.982 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2024-09-15T21:19:00.983 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-15T21:19:00.983 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-15T21:19:00.983 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-15T21:19:00.983 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-15T21:19:00.983 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-15T21:19:00.984 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-15T21:19:00.984 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-15T21:19:00.984 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-15T21:19:00.984 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-15T21:19:01.075 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-15T21:19:01.075 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-15T21:19:01.075 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-15T21:19:01.076 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-15T21:19:01.076 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-15T21:19:01.076 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-15T21:19:01.396 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-15T21:19:01.396 INFO:tasks.workunit.client.0.smithi136.stderr:1 2024-09-15T21:19:01.396 INFO:tasks.workunit.client.0.smithi136.stderr:2' 2024-09-15T21:19:01.396 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-15T21:19:01.396 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T21:19:01.396 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-15T21:19:01.567 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=210453397506 2024-09-15T21:19:01.567 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 210453397506 2024-09-15T21:19:01.567 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-210453397506' 2024-09-15T21:19:01.567 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T21:19:01.567 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-15T21:19:01.728 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=171798691846 2024-09-15T21:19:01.728 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 171798691846 2024-09-15T21:19:01.728 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-210453397506 1-171798691846' 2024-09-15T21:19:01.728 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T21:19:01.728 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-15T21:19:01.833 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=64424509455 2024-09-15T21:19:01.833 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 64424509455 2024-09-15T21:19:01.833 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-210453397506 1-171798691846 2-64424509455' 2024-09-15T21:19:01.833 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T21:19:01.833 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-210453397506 2024-09-15T21:19:01.833 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T21:19:01.834 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-15T21:19:01.834 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-210453397506 2024-09-15T21:19:01.834 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T21:19:01.835 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=210453397506 2024-09-15T21:19:01.835 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 210453397506' 2024-09-15T21:19:01.835 INFO:tasks.workunit.client.0.smithi136.stderr:waiting osd.0 seq 210453397506 2024-09-15T21:19:01.835 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T21:19:02.178 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836506 -lt 210453397506 2024-09-15T21:19:02.179 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-15T21:19:03.180 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-15T21:19:03.181 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T21:19:03.464 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 210453397506 -lt 210453397506 2024-09-15T21:19:03.465 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T21:19:03.465 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-171798691846 2024-09-15T21:19:03.465 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T21:19:03.466 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-15T21:19:03.467 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-171798691846 2024-09-15T21:19:03.467 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T21:19:03.467 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=171798691846 2024-09-15T21:19:03.468 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 171798691846' 2024-09-15T21:19:03.468 INFO:tasks.workunit.client.0.smithi136.stderr:waiting osd.1 seq 171798691846 2024-09-15T21:19:03.468 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-15T21:19:03.743 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 171798691846 -lt 171798691846 2024-09-15T21:19:03.743 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T21:19:03.744 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-64424509455 2024-09-15T21:19:03.744 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T21:19:03.744 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-15T21:19:03.745 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-64424509455 2024-09-15T21:19:03.745 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T21:19:03.746 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=64424509455 2024-09-15T21:19:03.746 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 64424509455' 2024-09-15T21:19:03.746 INFO:tasks.workunit.client.0.smithi136.stderr:waiting osd.2 seq 64424509455 2024-09-15T21:19:03.747 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-15T21:19:04.025 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 64424509456 -lt 64424509455 2024-09-15T21:19:04.026 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-15T21:19:04.026 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T21:19:04.026 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T21:19:04.386 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 6 == 0 2024-09-15T21:19:04.386 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-15T21:19:04.387 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-15T21:19:04.387 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-15T21:19:04.387 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T21:19:04.387 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T21:19:04.387 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T21:19:04.387 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T21:19:04.663 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=2 2024-09-15T21:19:04.664 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-15T21:19:04.664 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T21:19:04.664 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T21:19:05.007 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 2 = 6 2024-09-15T21:19:05.007 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 2 '!=' -1 2024-09-15T21:19:05.007 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: loop=0 2024-09-15T21:19:05.008 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1652: wait_for_clean: num_active_clean=2 2024-09-15T21:19:05.008 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-15T21:19:05.008 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 0.1 2024-09-15T21:19:05.109 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-15T21:19:05.109 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-15T21:19:05.109 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-15T21:19:05.109 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-15T21:19:05.109 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T21:19:05.109 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T21:19:05.109 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T21:19:05.109 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T21:19:05.383 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=2 2024-09-15T21:19:05.384 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-15T21:19:05.384 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T21:19:05.384 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T21:19:05.729 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 2 = 6 2024-09-15T21:19:05.729 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 2 '!=' 2 2024-09-15T21:19:05.729 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: get_is_making_recovery_progress 2024-09-15T21:19:05.729 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2024-09-15T21:19:05.729 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-15T21:19:05.730 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-15T21:19:05.730 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-15T21:19:05.730 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2024-09-15T21:19:05.730 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-15T21:19:06.077 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=null 2024-09-15T21:19:06.077 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test null '!=' null 2024-09-15T21:19:06.077 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1655: wait_for_clean: (( 1 >= 13 )) 2024-09-15T21:19:06.077 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-15T21:19:06.078 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 0.2 2024-09-15T21:19:06.278 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-15T21:19:06.278 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-15T21:19:06.279 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-15T21:19:06.279 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-15T21:19:06.279 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T21:19:06.279 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T21:19:06.279 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T21:19:06.279 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T21:19:06.570 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=6 2024-09-15T21:19:06.571 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-15T21:19:06.571 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T21:19:06.571 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T21:19:06.915 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 = 6 2024-09-15T21:19:06.916 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-15T21:19:06.916 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-15T21:19:06.916 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3612: corrupt_scrub_erasure: for i in $(seq 1 $total_objs) 2024-09-15T21:19:06.916 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3613: corrupt_scrub_erasure: objname=EOBJ3 2024-09-15T21:19:06.916 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3614: corrupt_scrub_erasure: add_something td/osd-scrub-repair ecpool EOBJ3 2024-09-15T21:19:06.916 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-09-15T21:19:06.916 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=ecpool 2024-09-15T21:19:06.916 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=EOBJ3 2024-09-15T21:19:06.916 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-09-15T21:19:06.916 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-09-15T21:19:06.916 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-09-15T21:19:07.295 INFO:tasks.workunit.client.0.smithi136.stderr:noscrub is set 2024-09-15T21:19:07.307 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-09-15T21:19:07.805 INFO:tasks.workunit.client.0.smithi136.stderr:nodeep-scrub is set 2024-09-15T21:19:07.816 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-09-15T21:19:07.817 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-09-15T21:19:07.817 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool ecpool put EOBJ3 td/osd-scrub-repair/ORIGINAL 2024-09-15T21:19:08.071 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3616: corrupt_scrub_erasure: expr 3 % 2 2024-09-15T21:19:08.074 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3616: corrupt_scrub_erasure: local osd=1 2024-09-15T21:19:08.074 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3618: corrupt_scrub_erasure: case $i in 2024-09-15T21:19:08.074 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3634: corrupt_scrub_erasure: objectstore_tool td/osd-scrub-repair 1 EOBJ3 remove 2024-09-15T21:19:08.074 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2024-09-15T21:19:08.074 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2024-09-15T21:19:08.074 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=1 2024-09-15T21:19:08.075 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2024-09-15T21:19:08.075 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 EOBJ3 remove 2024-09-15T21:19:08.075 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-09-15T21:19:08.075 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2024-09-15T21:19:08.075 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=1 2024-09-15T21:19:08.075 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2024-09-15T21:19:08.075 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-09-15T21:19:08.075 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-15T21:19:08.075 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-15T21:19:08.075 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-15T21:19:08.075 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-15T21:19:08.075 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-15T21:19:08.380 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-15T21:19:08.400 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 EOBJ3 remove 2024-09-15T21:19:08.400 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-15T21:19:08.400 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-15T21:19:08.400 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=1 2024-09-15T21:19:08.400 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-15T21:19:08.400 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-09-15T21:19:08.401 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 EOBJ3 remove 2024-09-15T21:19:10.193 INFO:tasks.workunit.client.0.smithi136.stdout:remove 0#3:b197b25d:::EOBJ3:head# 2024-09-15T21:19:10.717 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2024-09-15T21:19:10.717 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2024-09-15T21:19:10.717 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-15T21:19:10.718 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=1 2024-09-15T21:19:10.718 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-15T21:19:10.718 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-09-15T21:19:10.718 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=ce4892c7-6c6d-4b4f-8e6f-07a3e11133e5 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-15T21:19:10.718 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T21:19:10.718 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T21:19:10.718 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T21:19:10.718 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-09-15T21:19:10.718 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-09-15T21:19:10.718 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-15T21:19:10.718 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-15T21:19:10.718 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-15T21:19:10.718 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-15T21:19:10.718 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-15T21:19:10.719 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-15T21:19:10.719 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-15T21:19:10.719 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T21:19:10.719 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52602 2024-09-15T21:19:10.719 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.52602/$cluster-$name.asok' 2024-09-15T21:19:10.719 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.52602/$cluster-$name.asok' 2024-09-15T21:19:10.719 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-15T21:19:10.719 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-15T21:19:10.719 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-15T21:19:10.719 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T21:19:10.720 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T21:19:10.720 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T21:19:10.720 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T21:19:10.720 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-15T21:19:10.720 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-15T21:19:10.720 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-09-15T21:19:10.720 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.1 2024-09-15T21:19:10.720 INFO:tasks.workunit.client.0.smithi136.stderr:start osd.1 2024-09-15T21:19:10.720 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 1 --fsid=ce4892c7-6c6d-4b4f-8e6f-07a3e11133e5 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.52602/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T21:19:10.721 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-09-15T21:19:10.721 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 1 = 1 ']' 2024-09-15T21:19:10.721 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-15T21:19:10.721 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-15T21:19:10.721 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-15T21:19:10.748 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:19:10.744+0000 7f0557260640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:19:10.750 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:19:10.747+0000 7f0557260640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:19:10.751 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:19:10.748+0000 7f0557260640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:19:11.005 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 1 2024-09-15T21:19:11.005 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-15T21:19:11.005 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2024-09-15T21:19:11.005 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-15T21:19:11.005 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-15T21:19:11.005 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:19:11.005 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-15T21:19:11.006 INFO:tasks.workunit.client.0.smithi136.stderr:0 2024-09-15T21:19:11.006 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:19:11.006 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T21:19:11.281 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:19:12.123 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:19:12.120+0000 7f0557260640 -1 Falling back to public interface 2024-09-15T21:19:12.283 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:19:12.283 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:19:12.283 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-15T21:19:12.283 INFO:tasks.workunit.client.0.smithi136.stderr:1 2024-09-15T21:19:12.283 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:19:12.283 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T21:19:12.583 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:19:13.585 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:19:13.585 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:19:13.585 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-15T21:19:13.585 INFO:tasks.workunit.client.0.smithi136.stderr:2 2024-09-15T21:19:13.585 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:19:13.585 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T21:19:13.872 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:19:14.845 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:19:14.842+0000 7f0557260640 -1 osd.1 53 log_to_monitors true 2024-09-15T21:19:14.874 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:19:14.874 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:19:14.874 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-15T21:19:14.874 INFO:tasks.workunit.client.0.smithi136.stderr:3 2024-09-15T21:19:14.874 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:19:14.874 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T21:19:15.221 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:19:16.223 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:19:16.223 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:19:16.223 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-15T21:19:16.223 INFO:tasks.workunit.client.0.smithi136.stderr:4 2024-09-15T21:19:16.223 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:19:16.223 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T21:19:16.562 INFO:tasks.workunit.client.0.smithi136.stderr:osd.1 up in weight 1 up_from 58 up_thru 58 down_at 54 last_clean_interval [40,53) [v2:127.0.0.1:6810/2841197666,v1:127.0.0.1:6811/2841197666] [v2:127.0.0.1:6812/2841197666,v1:127.0.0.1:6813/2841197666] exists,up 87e38427-d391-4909-9e45-e7b2d413fd8f 2024-09-15T21:19:16.563 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-15T21:19:16.563 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-15T21:19:16.563 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-15T21:19:16.563 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2024-09-15T21:19:16.563 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-15T21:19:16.563 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-15T21:19:16.563 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-15T21:19:16.563 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-15T21:19:16.563 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-15T21:19:16.563 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-15T21:19:16.563 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-15T21:19:16.564 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-15T21:19:16.564 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-15T21:19:16.654 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-15T21:19:16.654 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-15T21:19:16.654 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-15T21:19:16.654 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-15T21:19:16.655 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-15T21:19:16.655 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-15T21:19:17.069 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-15T21:19:17.070 INFO:tasks.workunit.client.0.smithi136.stderr:1 2024-09-15T21:19:17.070 INFO:tasks.workunit.client.0.smithi136.stderr:2' 2024-09-15T21:19:17.070 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-15T21:19:17.070 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T21:19:17.070 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-15T21:19:17.204 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=210453397510 2024-09-15T21:19:17.205 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 210453397510 2024-09-15T21:19:17.205 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-210453397510' 2024-09-15T21:19:17.205 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T21:19:17.205 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-15T21:19:17.307 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=249108103170 2024-09-15T21:19:17.307 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 249108103170 2024-09-15T21:19:17.308 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-210453397510 1-249108103170' 2024-09-15T21:19:17.308 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T21:19:17.308 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-15T21:19:17.410 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=64424509460 2024-09-15T21:19:17.411 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 64424509460 2024-09-15T21:19:17.411 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-210453397510 1-249108103170 2-64424509460' 2024-09-15T21:19:17.411 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T21:19:17.411 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-210453397510 2024-09-15T21:19:17.411 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T21:19:17.412 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-15T21:19:17.412 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-210453397510 2024-09-15T21:19:17.413 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T21:19:17.414 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=210453397510 2024-09-15T21:19:17.414 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 210453397510' 2024-09-15T21:19:17.414 INFO:tasks.workunit.client.0.smithi136.stderr:waiting osd.0 seq 210453397510 2024-09-15T21:19:17.414 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T21:19:17.691 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 210453397509 -lt 210453397510 2024-09-15T21:19:17.691 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-15T21:19:18.692 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-15T21:19:18.692 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T21:19:18.977 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 210453397510 -lt 210453397510 2024-09-15T21:19:18.977 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T21:19:18.978 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-249108103170 2024-09-15T21:19:18.978 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T21:19:18.978 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-15T21:19:18.979 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-249108103170 2024-09-15T21:19:18.979 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T21:19:18.980 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=249108103170 2024-09-15T21:19:18.980 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 249108103170' 2024-09-15T21:19:18.980 INFO:tasks.workunit.client.0.smithi136.stderr:waiting osd.1 seq 249108103170 2024-09-15T21:19:18.981 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-15T21:19:19.255 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 249108103170 -lt 249108103170 2024-09-15T21:19:19.255 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T21:19:19.256 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-64424509460 2024-09-15T21:19:19.256 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T21:19:19.256 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-15T21:19:19.257 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-64424509460 2024-09-15T21:19:19.257 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T21:19:19.258 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=64424509460 2024-09-15T21:19:19.258 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 64424509460' 2024-09-15T21:19:19.258 INFO:tasks.workunit.client.0.smithi136.stderr:waiting osd.2 seq 64424509460 2024-09-15T21:19:19.259 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-15T21:19:19.537 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 64424509460 -lt 64424509460 2024-09-15T21:19:19.537 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-15T21:19:19.538 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T21:19:19.538 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T21:19:19.882 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 6 == 0 2024-09-15T21:19:19.883 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-15T21:19:19.883 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-15T21:19:19.883 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-15T21:19:19.883 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T21:19:19.883 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T21:19:19.884 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T21:19:19.884 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T21:19:20.156 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=6 2024-09-15T21:19:20.157 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-15T21:19:20.157 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T21:19:20.157 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T21:19:20.506 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 = 6 2024-09-15T21:19:20.506 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-15T21:19:20.506 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-15T21:19:20.506 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3612: corrupt_scrub_erasure: for i in $(seq 1 $total_objs) 2024-09-15T21:19:20.506 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3613: corrupt_scrub_erasure: objname=EOBJ4 2024-09-15T21:19:20.506 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3614: corrupt_scrub_erasure: add_something td/osd-scrub-repair ecpool EOBJ4 2024-09-15T21:19:20.506 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-09-15T21:19:20.506 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=ecpool 2024-09-15T21:19:20.507 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=EOBJ4 2024-09-15T21:19:20.507 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-09-15T21:19:20.507 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-09-15T21:19:20.507 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-09-15T21:19:20.914 INFO:tasks.workunit.client.0.smithi136.stderr:noscrub is set 2024-09-15T21:19:20.926 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-09-15T21:19:21.441 INFO:tasks.workunit.client.0.smithi136.stderr:nodeep-scrub is set 2024-09-15T21:19:21.453 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-09-15T21:19:21.453 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-09-15T21:19:21.454 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool ecpool put EOBJ4 td/osd-scrub-repair/ORIGINAL 2024-09-15T21:19:21.695 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3616: corrupt_scrub_erasure: expr 4 % 2 2024-09-15T21:19:21.696 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3616: corrupt_scrub_erasure: local osd=0 2024-09-15T21:19:21.696 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3618: corrupt_scrub_erasure: case $i in 2024-09-15T21:19:21.696 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3638: corrupt_scrub_erasure: rados --pool ecpool setxattr EOBJ4 key1-EOBJ4 val1-EOBJ4 2024-09-15T21:19:21.896 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3639: corrupt_scrub_erasure: rados --pool ecpool setxattr EOBJ4 key2-EOBJ4 val2-EOBJ4 2024-09-15T21:19:22.064 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3642: corrupt_scrub_erasure: echo -n bad-val 2024-09-15T21:19:22.065 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3643: corrupt_scrub_erasure: objectstore_tool td/osd-scrub-repair 0 EOBJ4 set-attr _key1-EOBJ4 td/osd-scrub-repair/bad-val 2024-09-15T21:19:22.065 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2024-09-15T21:19:22.065 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2024-09-15T21:19:22.065 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=0 2024-09-15T21:19:22.065 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2024-09-15T21:19:22.065 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 EOBJ4 set-attr _key1-EOBJ4 td/osd-scrub-repair/bad-val 2024-09-15T21:19:22.065 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-09-15T21:19:22.065 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2024-09-15T21:19:22.065 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=0 2024-09-15T21:19:22.065 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2024-09-15T21:19:22.065 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2024-09-15T21:19:22.065 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-15T21:19:22.066 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-15T21:19:22.066 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-15T21:19:22.066 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-15T21:19:22.066 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-15T21:19:22.371 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-15T21:19:22.372 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 EOBJ4 set-attr _key1-EOBJ4 td/osd-scrub-repair/bad-val 2024-09-15T21:19:22.372 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-15T21:19:22.372 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-15T21:19:22.372 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=0 2024-09-15T21:19:22.372 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-15T21:19:22.372 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-09-15T21:19:22.372 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 EOBJ4 set-attr _key1-EOBJ4 td/osd-scrub-repair/bad-val 2024-09-15T21:19:24.699 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 2024-09-15T21:19:24.700 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2024-09-15T21:19:24.700 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-15T21:19:24.700 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=0 2024-09-15T21:19:24.700 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-15T21:19:24.700 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-09-15T21:19:24.700 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=ce4892c7-6c6d-4b4f-8e6f-07a3e11133e5 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-15T21:19:24.700 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T21:19:24.700 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T21:19:24.700 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T21:19:24.700 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-09-15T21:19:24.700 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-09-15T21:19:24.700 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-15T21:19:24.700 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-15T21:19:24.701 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-15T21:19:24.701 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-15T21:19:24.701 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-15T21:19:24.701 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-15T21:19:24.701 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-15T21:19:24.701 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T21:19:24.701 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52602 2024-09-15T21:19:24.701 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.52602/$cluster-$name.asok' 2024-09-15T21:19:24.701 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.52602/$cluster-$name.asok' 2024-09-15T21:19:24.701 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-15T21:19:24.701 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-15T21:19:24.701 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-15T21:19:24.701 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T21:19:24.701 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T21:19:24.702 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T21:19:24.702 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T21:19:24.702 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-15T21:19:24.702 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-15T21:19:24.702 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-09-15T21:19:24.703 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.0 2024-09-15T21:19:24.703 INFO:tasks.workunit.client.0.smithi136.stderr:start osd.0 2024-09-15T21:19:24.703 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 0 --fsid=ce4892c7-6c6d-4b4f-8e6f-07a3e11133e5 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.52602/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T21:19:24.703 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-09-15T21:19:24.703 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 0 = 0 ']' 2024-09-15T21:19:24.703 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-15T21:19:24.703 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-15T21:19:24.703 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-15T21:19:24.733 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:19:24.730+0000 7fe789a2c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:19:24.735 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:19:24.732+0000 7fe789a2c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:19:24.737 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:19:24.734+0000 7fe789a2c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:19:24.989 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 0 2024-09-15T21:19:24.989 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-15T21:19:24.989 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2024-09-15T21:19:24.989 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-15T21:19:24.989 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-15T21:19:24.989 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:19:24.989 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-15T21:19:24.989 INFO:tasks.workunit.client.0.smithi136.stderr:0 2024-09-15T21:19:24.989 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:19:24.989 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:19:25.265 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:19:25.859 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:19:25.856+0000 7fe789a2c640 -1 Falling back to public interface 2024-09-15T21:19:26.266 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:19:26.267 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:19:26.267 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-15T21:19:26.267 INFO:tasks.workunit.client.0.smithi136.stderr:1 2024-09-15T21:19:26.267 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:19:26.267 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:19:26.556 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:19:27.558 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:19:27.558 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:19:27.558 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-15T21:19:27.558 INFO:tasks.workunit.client.0.smithi136.stderr:2 2024-09-15T21:19:27.558 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:19:27.558 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:19:27.845 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:19:28.847 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:19:28.847 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:19:28.847 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-15T21:19:28.847 INFO:tasks.workunit.client.0.smithi136.stderr:3 2024-09-15T21:19:28.847 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:19:28.847 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:19:28.852 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:19:28.849+0000 7fe789a2c640 -1 osd.0 62 log_to_monitors true 2024-09-15T21:19:29.240 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:19:30.241 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:19:30.242 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:19:30.242 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-15T21:19:30.242 INFO:tasks.workunit.client.0.smithi136.stderr:4 2024-09-15T21:19:30.242 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:19:30.242 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:19:30.669 INFO:tasks.workunit.client.0.smithi136.stderr:osd.0 up in weight 1 up_from 67 up_thru 67 down_at 63 last_clean_interval [49,62) [v2:127.0.0.1:6802/1831941761,v1:127.0.0.1:6803/1831941761] [v2:127.0.0.1:6804/1831941761,v1:127.0.0.1:6805/1831941761] exists,up 9087e8dc-fe45-44af-af39-0644acbc5cf9 2024-09-15T21:19:30.669 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-15T21:19:30.669 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-15T21:19:30.669 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-15T21:19:30.670 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2024-09-15T21:19:30.670 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-15T21:19:30.670 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-15T21:19:30.670 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-15T21:19:30.670 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-15T21:19:30.670 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-15T21:19:30.670 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-15T21:19:30.670 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-15T21:19:30.670 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-15T21:19:30.670 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-15T21:19:30.761 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-15T21:19:30.761 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-15T21:19:30.761 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-15T21:19:30.761 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-15T21:19:30.761 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-15T21:19:30.762 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-15T21:19:31.153 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-15T21:19:31.153 INFO:tasks.workunit.client.0.smithi136.stderr:1 2024-09-15T21:19:31.153 INFO:tasks.workunit.client.0.smithi136.stderr:2' 2024-09-15T21:19:31.153 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-15T21:19:31.153 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T21:19:31.153 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-15T21:19:31.257 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=287762808834 2024-09-15T21:19:31.257 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 287762808834 2024-09-15T21:19:31.257 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-287762808834' 2024-09-15T21:19:31.257 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T21:19:31.257 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-15T21:19:31.360 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=249108103174 2024-09-15T21:19:31.360 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 249108103174 2024-09-15T21:19:31.360 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-287762808834 1-249108103174' 2024-09-15T21:19:31.360 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T21:19:31.361 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-15T21:19:31.463 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=64424509463 2024-09-15T21:19:31.463 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 64424509463 2024-09-15T21:19:31.463 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-287762808834 1-249108103174 2-64424509463' 2024-09-15T21:19:31.463 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T21:19:31.464 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-287762808834 2024-09-15T21:19:31.464 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T21:19:31.464 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-15T21:19:31.465 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-287762808834 2024-09-15T21:19:31.465 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T21:19:31.466 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=287762808834 2024-09-15T21:19:31.466 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 287762808834' 2024-09-15T21:19:31.466 INFO:tasks.workunit.client.0.smithi136.stderr:waiting osd.0 seq 287762808834 2024-09-15T21:19:31.466 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T21:19:31.740 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 1 -lt 287762808834 2024-09-15T21:19:31.740 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-15T21:19:32.741 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-15T21:19:32.742 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T21:19:33.024 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 287762808834 -lt 287762808834 2024-09-15T21:19:33.024 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T21:19:33.025 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-249108103174 2024-09-15T21:19:33.025 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T21:19:33.026 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-15T21:19:33.026 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-249108103174 2024-09-15T21:19:33.026 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T21:19:33.028 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=249108103174 2024-09-15T21:19:33.028 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 249108103174' 2024-09-15T21:19:33.028 INFO:tasks.workunit.client.0.smithi136.stderr:waiting osd.1 seq 249108103174 2024-09-15T21:19:33.028 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-15T21:19:33.306 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 249108103174 -lt 249108103174 2024-09-15T21:19:33.307 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T21:19:33.307 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-64424509463 2024-09-15T21:19:33.307 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T21:19:33.308 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-15T21:19:33.309 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-64424509463 2024-09-15T21:19:33.309 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T21:19:33.310 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=64424509463 2024-09-15T21:19:33.310 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 64424509463' 2024-09-15T21:19:33.310 INFO:tasks.workunit.client.0.smithi136.stderr:waiting osd.2 seq 64424509463 2024-09-15T21:19:33.310 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-15T21:19:33.589 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 64424509464 -lt 64424509463 2024-09-15T21:19:33.590 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-15T21:19:33.590 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T21:19:33.590 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T21:19:33.934 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 6 == 0 2024-09-15T21:19:33.934 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-15T21:19:33.934 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-15T21:19:33.934 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-15T21:19:33.934 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T21:19:33.934 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T21:19:33.934 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T21:19:33.934 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T21:19:34.205 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=6 2024-09-15T21:19:34.205 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-15T21:19:34.206 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T21:19:34.206 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T21:19:34.574 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 = 6 2024-09-15T21:19:34.575 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-15T21:19:34.575 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-15T21:19:34.575 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3644: corrupt_scrub_erasure: objectstore_tool td/osd-scrub-repair 2 EOBJ4 rm-attr _key2-EOBJ4 2024-09-15T21:19:34.575 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2024-09-15T21:19:34.575 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2024-09-15T21:19:34.575 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=2 2024-09-15T21:19:34.575 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2024-09-15T21:19:34.575 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 2 EOBJ4 rm-attr _key2-EOBJ4 2024-09-15T21:19:34.575 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-09-15T21:19:34.575 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2024-09-15T21:19:34.575 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=2 2024-09-15T21:19:34.575 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2024-09-15T21:19:34.576 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.2 2024-09-15T21:19:34.576 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-15T21:19:34.576 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-15T21:19:34.576 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-15T21:19:34.576 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-15T21:19:34.576 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-15T21:19:34.881 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-15T21:19:34.881 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 2 EOBJ4 rm-attr _key2-EOBJ4 2024-09-15T21:19:34.881 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-15T21:19:34.881 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-15T21:19:34.881 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=2 2024-09-15T21:19:34.881 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-15T21:19:34.882 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/2 2024-09-15T21:19:34.882 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/2 EOBJ4 rm-attr _key2-EOBJ4 2024-09-15T21:19:37.294 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 2 2024-09-15T21:19:37.295 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2024-09-15T21:19:37.295 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-15T21:19:37.295 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=2 2024-09-15T21:19:37.295 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-15T21:19:37.295 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/2 2024-09-15T21:19:37.295 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=ce4892c7-6c6d-4b4f-8e6f-07a3e11133e5 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-15T21:19:37.295 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T21:19:37.295 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T21:19:37.295 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T21:19:37.295 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/2' 2024-09-15T21:19:37.295 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/2/journal' 2024-09-15T21:19:37.295 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-15T21:19:37.295 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-15T21:19:37.296 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-15T21:19:37.296 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-15T21:19:37.296 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-15T21:19:37.296 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-15T21:19:37.296 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-15T21:19:37.296 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T21:19:37.296 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52602 2024-09-15T21:19:37.296 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.52602/$cluster-$name.asok' 2024-09-15T21:19:37.296 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.52602/$cluster-$name.asok' 2024-09-15T21:19:37.296 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-15T21:19:37.297 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-15T21:19:37.297 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-15T21:19:37.297 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T21:19:37.297 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T21:19:37.297 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T21:19:37.297 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T21:19:37.297 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-15T21:19:37.297 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-15T21:19:37.297 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/2 2024-09-15T21:19:37.297 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.2 2024-09-15T21:19:37.297 INFO:tasks.workunit.client.0.smithi136.stderr:start osd.2 2024-09-15T21:19:37.297 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 2 --fsid=ce4892c7-6c6d-4b4f-8e6f-07a3e11133e5 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.52602/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T21:19:37.297 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/2/whoami 2024-09-15T21:19:37.297 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 2 = 2 ']' 2024-09-15T21:19:37.298 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-15T21:19:37.298 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-15T21:19:37.298 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-15T21:19:37.322 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:19:37.319+0000 7fa06cc9c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:19:37.325 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:19:37.321+0000 7fa06cc9c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:19:37.327 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:19:37.323+0000 7fa06cc9c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:19:37.579 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 2 2024-09-15T21:19:37.580 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-15T21:19:37.580 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=2 2024-09-15T21:19:37.580 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-15T21:19:37.580 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-15T21:19:37.580 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:19:37.580 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-15T21:19:37.580 INFO:tasks.workunit.client.0.smithi136.stderr:0 2024-09-15T21:19:37.580 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:19:37.580 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T21:19:37.858 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:19:38.444 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:19:38.441+0000 7fa06cc9c640 -1 Falling back to public interface 2024-09-15T21:19:38.859 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:19:38.859 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:19:38.859 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-15T21:19:38.859 INFO:tasks.workunit.client.0.smithi136.stderr:1 2024-09-15T21:19:38.859 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:19:38.859 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T21:19:39.143 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:19:40.145 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:19:40.145 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:19:40.145 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-15T21:19:40.145 INFO:tasks.workunit.client.0.smithi136.stderr:2 2024-09-15T21:19:40.145 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:19:40.145 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T21:19:40.453 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:19:41.132 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:19:41.128+0000 7fa06cc9c640 -1 osd.2 70 log_to_monitors true 2024-09-15T21:19:41.454 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:19:41.454 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:19:41.454 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-15T21:19:41.454 INFO:tasks.workunit.client.0.smithi136.stderr:3 2024-09-15T21:19:41.455 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:19:41.455 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T21:19:41.833 INFO:tasks.workunit.client.0.smithi136.stderr:osd.2 up in weight 1 up_from 75 up_thru 67 down_at 71 last_clean_interval [15,70) [v2:127.0.0.1:6818/1046239958,v1:127.0.0.1:6819/1046239958] [v2:127.0.0.1:6820/1046239958,v1:127.0.0.1:6821/1046239958] exists,up 12467c91-a626-4bc5-ba50-e5426a3bf5b2 2024-09-15T21:19:41.833 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-15T21:19:41.833 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-15T21:19:41.833 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-15T21:19:41.833 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2024-09-15T21:19:41.834 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-15T21:19:41.834 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-15T21:19:41.834 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-15T21:19:41.834 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-15T21:19:41.834 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-15T21:19:41.834 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-15T21:19:41.834 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-15T21:19:41.834 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-15T21:19:41.834 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-15T21:19:41.926 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-15T21:19:41.926 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-15T21:19:41.926 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-15T21:19:41.926 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-15T21:19:41.926 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-15T21:19:41.926 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-15T21:19:42.235 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-15T21:19:42.235 INFO:tasks.workunit.client.0.smithi136.stderr:1 2024-09-15T21:19:42.235 INFO:tasks.workunit.client.0.smithi136.stderr:2' 2024-09-15T21:19:42.235 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-15T21:19:42.235 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T21:19:42.236 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-15T21:19:42.338 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=287762808837 2024-09-15T21:19:42.338 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 287762808837 2024-09-15T21:19:42.339 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-287762808837' 2024-09-15T21:19:42.339 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T21:19:42.339 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-15T21:19:42.478 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=249108103177 2024-09-15T21:19:42.479 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 249108103177 2024-09-15T21:19:42.479 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-287762808837 1-249108103177' 2024-09-15T21:19:42.479 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T21:19:42.479 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-15T21:19:42.649 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=322122547202 2024-09-15T21:19:42.649 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 322122547202 2024-09-15T21:19:42.650 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-287762808837 1-249108103177 2-322122547202' 2024-09-15T21:19:42.650 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T21:19:42.650 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-287762808837 2024-09-15T21:19:42.650 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T21:19:42.650 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-15T21:19:42.651 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-287762808837 2024-09-15T21:19:42.651 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T21:19:42.652 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=287762808837 2024-09-15T21:19:42.652 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 287762808837' 2024-09-15T21:19:42.652 INFO:tasks.workunit.client.0.smithi136.stderr:waiting osd.0 seq 287762808837 2024-09-15T21:19:42.652 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T21:19:42.929 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 287762808836 -lt 287762808837 2024-09-15T21:19:42.929 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-15T21:19:43.930 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-15T21:19:43.930 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T21:19:44.220 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 287762808836 -lt 287762808837 2024-09-15T21:19:44.220 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-15T21:19:45.221 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 299 -eq 0 ']' 2024-09-15T21:19:45.222 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T21:19:45.508 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 287762808838 -lt 287762808837 2024-09-15T21:19:45.508 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T21:19:45.508 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-249108103177 2024-09-15T21:19:45.508 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T21:19:45.509 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-15T21:19:45.510 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-249108103177 2024-09-15T21:19:45.510 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T21:19:45.511 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=249108103177 2024-09-15T21:19:45.511 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 249108103177' 2024-09-15T21:19:45.511 INFO:tasks.workunit.client.0.smithi136.stderr:waiting osd.1 seq 249108103177 2024-09-15T21:19:45.511 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-15T21:19:45.785 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 249108103177 -lt 249108103177 2024-09-15T21:19:45.786 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T21:19:45.786 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-322122547202 2024-09-15T21:19:45.786 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T21:19:45.787 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-15T21:19:45.787 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-322122547202 2024-09-15T21:19:45.788 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T21:19:45.789 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=322122547202 2024-09-15T21:19:45.789 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 322122547202' 2024-09-15T21:19:45.789 INFO:tasks.workunit.client.0.smithi136.stderr:waiting osd.2 seq 322122547202 2024-09-15T21:19:45.789 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-15T21:19:46.068 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 322122547202 -lt 322122547202 2024-09-15T21:19:46.069 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-15T21:19:46.069 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T21:19:46.069 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T21:19:46.435 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 6 == 0 2024-09-15T21:19:46.435 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-15T21:19:46.436 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-15T21:19:46.436 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-15T21:19:46.436 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T21:19:46.436 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T21:19:46.436 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T21:19:46.436 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T21:19:46.710 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=6 2024-09-15T21:19:46.710 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-15T21:19:46.711 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T21:19:46.711 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T21:19:47.056 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 = 6 2024-09-15T21:19:47.056 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-15T21:19:47.056 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-15T21:19:47.056 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3645: corrupt_scrub_erasure: echo -n val3-EOBJ4 2024-09-15T21:19:47.056 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3646: corrupt_scrub_erasure: objectstore_tool td/osd-scrub-repair 2 EOBJ4 set-attr _key3-EOBJ4 td/osd-scrub-repair/newval 2024-09-15T21:19:47.056 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2024-09-15T21:19:47.057 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2024-09-15T21:19:47.057 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=2 2024-09-15T21:19:47.057 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2024-09-15T21:19:47.057 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 2 EOBJ4 set-attr _key3-EOBJ4 td/osd-scrub-repair/newval 2024-09-15T21:19:47.057 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-09-15T21:19:47.057 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2024-09-15T21:19:47.057 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=2 2024-09-15T21:19:47.057 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2024-09-15T21:19:47.057 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.2 2024-09-15T21:19:47.057 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-15T21:19:47.058 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-15T21:19:47.058 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-15T21:19:47.058 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-15T21:19:47.058 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-15T21:19:47.362 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-15T21:19:47.362 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 2 EOBJ4 set-attr _key3-EOBJ4 td/osd-scrub-repair/newval 2024-09-15T21:19:47.362 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-15T21:19:47.362 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-15T21:19:47.363 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=2 2024-09-15T21:19:47.363 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-15T21:19:47.363 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/2 2024-09-15T21:19:47.363 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/2 EOBJ4 set-attr _key3-EOBJ4 td/osd-scrub-repair/newval 2024-09-15T21:19:49.846 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 2 2024-09-15T21:19:49.846 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2024-09-15T21:19:49.846 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-15T21:19:49.846 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=2 2024-09-15T21:19:49.846 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-15T21:19:49.846 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/2 2024-09-15T21:19:49.846 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=ce4892c7-6c6d-4b4f-8e6f-07a3e11133e5 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-15T21:19:49.846 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T21:19:49.847 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T21:19:49.847 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T21:19:49.847 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/2' 2024-09-15T21:19:49.847 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/2/journal' 2024-09-15T21:19:49.847 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-15T21:19:49.847 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-15T21:19:49.847 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-15T21:19:49.847 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-15T21:19:49.848 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-15T21:19:49.848 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-15T21:19:49.848 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-15T21:19:49.848 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T21:19:49.848 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52602 2024-09-15T21:19:49.848 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.52602/$cluster-$name.asok' 2024-09-15T21:19:49.848 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.52602/$cluster-$name.asok' 2024-09-15T21:19:49.848 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-15T21:19:49.848 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-15T21:19:49.848 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-15T21:19:49.848 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T21:19:49.848 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T21:19:49.849 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T21:19:49.849 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T21:19:49.849 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-15T21:19:49.849 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-15T21:19:49.849 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/2 2024-09-15T21:19:49.849 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.2 2024-09-15T21:19:49.849 INFO:tasks.workunit.client.0.smithi136.stderr:start osd.2 2024-09-15T21:19:49.849 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 2 --fsid=ce4892c7-6c6d-4b4f-8e6f-07a3e11133e5 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.52602/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T21:19:49.850 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/2/whoami 2024-09-15T21:19:49.850 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 2 = 2 ']' 2024-09-15T21:19:49.850 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-15T21:19:49.850 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-15T21:19:49.850 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-15T21:19:49.876 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:19:49.873+0000 7f8b77b17640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:19:49.878 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:19:49.875+0000 7f8b77b17640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:19:49.879 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:19:49.876+0000 7f8b77b17640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:19:50.134 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 2 2024-09-15T21:19:50.134 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-15T21:19:50.134 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=2 2024-09-15T21:19:50.134 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-15T21:19:50.134 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-15T21:19:50.134 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:19:50.134 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-15T21:19:50.134 INFO:tasks.workunit.client.0.smithi136.stderr:0 2024-09-15T21:19:50.134 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:19:50.135 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T21:19:50.413 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:19:51.414 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:19:51.414 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:19:51.415 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-15T21:19:51.415 INFO:tasks.workunit.client.0.smithi136.stderr:1 2024-09-15T21:19:51.415 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:19:51.415 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T21:19:51.507 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:19:51.504+0000 7f8b77b17640 -1 Falling back to public interface 2024-09-15T21:19:51.700 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:19:52.702 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:19:52.702 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:19:52.702 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-15T21:19:52.702 INFO:tasks.workunit.client.0.smithi136.stderr:2 2024-09-15T21:19:52.702 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:19:52.702 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T21:19:52.987 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:19:53.988 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:19:53.989 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:19:53.989 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-15T21:19:53.989 INFO:tasks.workunit.client.0.smithi136.stderr:3 2024-09-15T21:19:53.989 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:19:53.989 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T21:19:54.277 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:19:54.444 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:19:54.432+0000 7f8b77b17640 -1 osd.2 77 log_to_monitors true 2024-09-15T21:19:55.278 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:19:55.278 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:19:55.278 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-15T21:19:55.279 INFO:tasks.workunit.client.0.smithi136.stderr:4 2024-09-15T21:19:55.279 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:19:55.279 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T21:19:55.733 INFO:tasks.workunit.client.0.smithi136.stderr:osd.2 up in weight 1 up_from 82 up_thru 82 down_at 78 last_clean_interval [75,77) [v2:127.0.0.1:6818/1484351765,v1:127.0.0.1:6819/1484351765] [v2:127.0.0.1:6820/1484351765,v1:127.0.0.1:6821/1484351765] exists,up 12467c91-a626-4bc5-ba50-e5426a3bf5b2 2024-09-15T21:19:55.733 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-15T21:19:55.733 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-15T21:19:55.733 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-15T21:19:55.733 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2024-09-15T21:19:55.733 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-15T21:19:55.734 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-15T21:19:55.734 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-15T21:19:55.734 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-15T21:19:55.734 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-15T21:19:55.734 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-15T21:19:55.734 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-15T21:19:55.734 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-15T21:19:55.734 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-15T21:19:55.825 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-15T21:19:55.825 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-15T21:19:55.825 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-15T21:19:55.825 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-15T21:19:55.825 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-15T21:19:55.825 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-15T21:19:56.103 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-15T21:19:56.103 INFO:tasks.workunit.client.0.smithi136.stderr:1 2024-09-15T21:19:56.103 INFO:tasks.workunit.client.0.smithi136.stderr:2' 2024-09-15T21:19:56.103 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-15T21:19:56.103 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T21:19:56.103 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-15T21:19:56.206 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=287762808841 2024-09-15T21:19:56.206 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 287762808841 2024-09-15T21:19:56.206 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-287762808841' 2024-09-15T21:19:56.207 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T21:19:56.207 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-15T21:19:56.311 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=249108103181 2024-09-15T21:19:56.312 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 249108103181 2024-09-15T21:19:56.312 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-287762808841 1-249108103181' 2024-09-15T21:19:56.312 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T21:19:56.312 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-15T21:19:56.425 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=352187318274 2024-09-15T21:19:56.425 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 352187318274 2024-09-15T21:19:56.425 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-287762808841 1-249108103181 2-352187318274' 2024-09-15T21:19:56.425 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T21:19:56.426 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-287762808841 2024-09-15T21:19:56.426 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T21:19:56.426 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-15T21:19:56.427 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-287762808841 2024-09-15T21:19:56.427 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T21:19:56.428 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=287762808841 2024-09-15T21:19:56.428 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 287762808841' 2024-09-15T21:19:56.428 INFO:tasks.workunit.client.0.smithi136.stderr:waiting osd.0 seq 287762808841 2024-09-15T21:19:56.428 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T21:19:56.705 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 287762808841 -lt 287762808841 2024-09-15T21:19:56.705 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T21:19:56.706 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-249108103181 2024-09-15T21:19:56.706 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T21:19:56.706 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-15T21:19:56.707 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-249108103181 2024-09-15T21:19:56.707 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T21:19:56.708 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=249108103181 2024-09-15T21:19:56.708 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 249108103181' 2024-09-15T21:19:56.708 INFO:tasks.workunit.client.0.smithi136.stderr:waiting osd.1 seq 249108103181 2024-09-15T21:19:56.709 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-15T21:19:56.983 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 249108103180 -lt 249108103181 2024-09-15T21:19:56.983 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-15T21:19:57.984 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-15T21:19:57.984 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-15T21:19:58.265 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 249108103180 -lt 249108103181 2024-09-15T21:19:58.265 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-15T21:19:59.267 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 299 -eq 0 ']' 2024-09-15T21:19:59.267 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-15T21:19:59.553 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 249108103181 -lt 249108103181 2024-09-15T21:19:59.553 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T21:19:59.553 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-352187318274 2024-09-15T21:19:59.554 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T21:19:59.555 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-15T21:19:59.555 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-352187318274 2024-09-15T21:19:59.555 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T21:19:59.556 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=352187318274 2024-09-15T21:19:59.556 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 352187318274' 2024-09-15T21:19:59.556 INFO:tasks.workunit.client.0.smithi136.stderr:waiting osd.2 seq 352187318274 2024-09-15T21:19:59.556 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-15T21:19:59.829 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 352187318274 -lt 352187318274 2024-09-15T21:19:59.830 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-15T21:19:59.830 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T21:19:59.830 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T21:20:00.174 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 6 == 0 2024-09-15T21:20:00.174 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-15T21:20:00.174 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-15T21:20:00.174 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-15T21:20:00.174 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T21:20:00.175 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T21:20:00.175 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T21:20:00.175 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T21:20:00.448 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=6 2024-09-15T21:20:00.448 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-15T21:20:00.449 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T21:20:00.449 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T21:20:00.795 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 = 6 2024-09-15T21:20:00.795 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-15T21:20:00.795 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-15T21:20:00.795 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3647: corrupt_scrub_erasure: rm td/osd-scrub-repair/bad-val td/osd-scrub-repair/newval 2024-09-15T21:20:00.796 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3612: corrupt_scrub_erasure: for i in $(seq 1 $total_objs) 2024-09-15T21:20:00.796 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3613: corrupt_scrub_erasure: objname=EOBJ5 2024-09-15T21:20:00.797 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3614: corrupt_scrub_erasure: add_something td/osd-scrub-repair ecpool EOBJ5 2024-09-15T21:20:00.797 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-09-15T21:20:00.797 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=ecpool 2024-09-15T21:20:00.797 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=EOBJ5 2024-09-15T21:20:00.797 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-09-15T21:20:00.797 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-09-15T21:20:00.797 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-09-15T21:20:01.197 INFO:tasks.workunit.client.0.smithi136.stderr:noscrub is set 2024-09-15T21:20:01.209 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-09-15T21:20:01.689 INFO:tasks.workunit.client.0.smithi136.stderr:nodeep-scrub is set 2024-09-15T21:20:01.701 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-09-15T21:20:01.701 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-09-15T21:20:01.701 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool ecpool put EOBJ5 td/osd-scrub-repair/ORIGINAL 2024-09-15T21:20:01.834 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3616: corrupt_scrub_erasure: expr 5 % 2 2024-09-15T21:20:01.835 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3616: corrupt_scrub_erasure: local osd=1 2024-09-15T21:20:01.835 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3618: corrupt_scrub_erasure: case $i in 2024-09-15T21:20:01.836 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3652: corrupt_scrub_erasure: dd if=/dev/urandom of=td/osd-scrub-repair/CORRUPT bs=2048 count=2 2024-09-15T21:20:01.837 INFO:tasks.workunit.client.0.smithi136.stderr:2+0 records in 2024-09-15T21:20:01.837 INFO:tasks.workunit.client.0.smithi136.stderr:2+0 records out 2024-09-15T21:20:01.838 INFO:tasks.workunit.client.0.smithi136.stderr:4096 bytes (4.1 kB, 4.0 KiB) copied, 7.0772e-05 s, 57.9 MB/s 2024-09-15T21:20:01.838 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3653: corrupt_scrub_erasure: objectstore_tool td/osd-scrub-repair 1 EOBJ5 set-bytes td/osd-scrub-repair/CORRUPT 2024-09-15T21:20:01.838 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2024-09-15T21:20:01.838 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2024-09-15T21:20:01.838 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=1 2024-09-15T21:20:01.838 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2024-09-15T21:20:01.838 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 EOBJ5 set-bytes td/osd-scrub-repair/CORRUPT 2024-09-15T21:20:01.838 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-09-15T21:20:01.838 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2024-09-15T21:20:01.838 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=1 2024-09-15T21:20:01.838 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2024-09-15T21:20:01.838 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-09-15T21:20:01.838 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-15T21:20:01.839 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-15T21:20:01.839 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-15T21:20:01.839 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-15T21:20:01.839 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-15T21:20:02.144 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-15T21:20:02.144 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 EOBJ5 set-bytes td/osd-scrub-repair/CORRUPT 2024-09-15T21:20:02.144 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-15T21:20:02.144 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-15T21:20:02.145 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=1 2024-09-15T21:20:02.145 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-15T21:20:02.145 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-09-15T21:20:02.145 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 EOBJ5 set-bytes td/osd-scrub-repair/CORRUPT 2024-09-15T21:20:04.478 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2024-09-15T21:20:04.478 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2024-09-15T21:20:04.478 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-15T21:20:04.478 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=1 2024-09-15T21:20:04.478 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-15T21:20:04.478 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-09-15T21:20:04.478 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=ce4892c7-6c6d-4b4f-8e6f-07a3e11133e5 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-15T21:20:04.478 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T21:20:04.478 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T21:20:04.478 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T21:20:04.479 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-09-15T21:20:04.479 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-09-15T21:20:04.479 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-15T21:20:04.479 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-15T21:20:04.479 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-15T21:20:04.479 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-15T21:20:04.479 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-15T21:20:04.479 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-15T21:20:04.479 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-15T21:20:04.479 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T21:20:04.479 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52602 2024-09-15T21:20:04.479 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.52602/$cluster-$name.asok' 2024-09-15T21:20:04.480 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.52602/$cluster-$name.asok' 2024-09-15T21:20:04.480 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-15T21:20:04.480 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-15T21:20:04.480 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-15T21:20:04.480 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T21:20:04.480 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T21:20:04.480 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T21:20:04.480 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T21:20:04.480 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-15T21:20:04.481 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-15T21:20:04.481 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-09-15T21:20:04.481 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.1 2024-09-15T21:20:04.481 INFO:tasks.workunit.client.0.smithi136.stderr:start osd.1 2024-09-15T21:20:04.482 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 1 --fsid=ce4892c7-6c6d-4b4f-8e6f-07a3e11133e5 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.52602/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T21:20:04.482 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-09-15T21:20:04.482 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 1 = 1 ']' 2024-09-15T21:20:04.482 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-15T21:20:04.482 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-15T21:20:04.482 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-15T21:20:04.508 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:20:04.505+0000 7fda02620640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:20:04.511 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:20:04.508+0000 7fda02620640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:20:04.512 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:20:04.509+0000 7fda02620640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:20:04.764 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 1 2024-09-15T21:20:04.764 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-15T21:20:04.764 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2024-09-15T21:20:04.764 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-15T21:20:04.764 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-15T21:20:04.764 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:20:04.764 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-15T21:20:04.764 INFO:tasks.workunit.client.0.smithi136.stderr:0 2024-09-15T21:20:04.764 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:20:04.765 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T21:20:05.041 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:20:05.642 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:20:05.639+0000 7fda02620640 -1 Falling back to public interface 2024-09-15T21:20:06.043 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:20:06.043 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:20:06.043 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-15T21:20:06.043 INFO:tasks.workunit.client.0.smithi136.stderr:1 2024-09-15T21:20:06.043 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:20:06.043 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T21:20:06.332 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:20:07.333 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:20:07.333 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:20:07.333 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-15T21:20:07.334 INFO:tasks.workunit.client.0.smithi136.stderr:2 2024-09-15T21:20:07.334 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:20:07.334 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T21:20:07.623 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:20:08.625 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:20:08.625 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:20:08.625 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-15T21:20:08.625 INFO:tasks.workunit.client.0.smithi136.stderr:3 2024-09-15T21:20:08.625 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:20:08.625 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T21:20:08.692 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:20:08.688+0000 7fda02620640 -1 osd.1 86 log_to_monitors true 2024-09-15T21:20:08.971 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:20:09.972 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:20:10.010 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:20:10.010 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-15T21:20:10.010 INFO:tasks.workunit.client.0.smithi136.stderr:4 2024-09-15T21:20:10.010 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:20:10.011 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T21:20:10.286 INFO:tasks.workunit.client.0.smithi136.stderr:osd.1 up in weight 1 up_from 91 up_thru 91 down_at 87 last_clean_interval [58,86) [v2:127.0.0.1:6810/3804305835,v1:127.0.0.1:6811/3804305835] [v2:127.0.0.1:6812/3804305835,v1:127.0.0.1:6813/3804305835] exists,up 87e38427-d391-4909-9e45-e7b2d413fd8f 2024-09-15T21:20:10.286 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-15T21:20:10.286 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-15T21:20:10.286 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-15T21:20:10.286 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2024-09-15T21:20:10.286 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-15T21:20:10.286 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-15T21:20:10.286 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-15T21:20:10.286 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-15T21:20:10.287 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-15T21:20:10.287 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-15T21:20:10.287 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-15T21:20:10.287 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-15T21:20:10.287 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-15T21:20:10.378 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-15T21:20:10.378 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-15T21:20:10.378 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-15T21:20:10.378 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-15T21:20:10.378 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-15T21:20:10.378 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-15T21:20:10.658 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-15T21:20:10.658 INFO:tasks.workunit.client.0.smithi136.stderr:1 2024-09-15T21:20:10.658 INFO:tasks.workunit.client.0.smithi136.stderr:2' 2024-09-15T21:20:10.658 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-15T21:20:10.658 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T21:20:10.658 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-15T21:20:10.762 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=287762808845 2024-09-15T21:20:10.762 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 287762808845 2024-09-15T21:20:10.762 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-287762808845' 2024-09-15T21:20:10.762 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T21:20:10.762 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-15T21:20:10.888 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=390842023938 2024-09-15T21:20:10.888 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 390842023938 2024-09-15T21:20:10.888 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-287762808845 1-390842023938' 2024-09-15T21:20:10.888 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T21:20:10.888 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-15T21:20:10.992 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=352187318278 2024-09-15T21:20:10.992 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 352187318278 2024-09-15T21:20:10.992 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-287762808845 1-390842023938 2-352187318278' 2024-09-15T21:20:10.992 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T21:20:10.992 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-287762808845 2024-09-15T21:20:10.992 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T21:20:10.993 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-15T21:20:10.994 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-287762808845 2024-09-15T21:20:10.994 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T21:20:10.995 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=287762808845 2024-09-15T21:20:10.995 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 287762808845' 2024-09-15T21:20:10.995 INFO:tasks.workunit.client.0.smithi136.stderr:waiting osd.0 seq 287762808845 2024-09-15T21:20:10.995 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T21:20:11.270 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 287762808844 -lt 287762808845 2024-09-15T21:20:11.270 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-15T21:20:12.272 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-15T21:20:12.272 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T21:20:12.588 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 287762808845 -lt 287762808845 2024-09-15T21:20:12.588 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T21:20:12.589 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-390842023938 2024-09-15T21:20:12.589 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T21:20:12.590 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-15T21:20:12.590 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-390842023938 2024-09-15T21:20:12.590 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T21:20:12.591 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=390842023938 2024-09-15T21:20:12.591 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 390842023938' 2024-09-15T21:20:12.591 INFO:tasks.workunit.client.0.smithi136.stderr:waiting osd.1 seq 390842023938 2024-09-15T21:20:12.592 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-15T21:20:12.868 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 390842023938 -lt 390842023938 2024-09-15T21:20:12.868 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T21:20:12.868 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-352187318278 2024-09-15T21:20:12.868 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T21:20:12.869 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-15T21:20:12.870 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-352187318278 2024-09-15T21:20:12.870 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T21:20:12.871 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=352187318278 2024-09-15T21:20:12.871 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 352187318278' 2024-09-15T21:20:12.871 INFO:tasks.workunit.client.0.smithi136.stderr:waiting osd.2 seq 352187318278 2024-09-15T21:20:12.871 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-15T21:20:13.150 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 352187318278 -lt 352187318278 2024-09-15T21:20:13.150 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-15T21:20:13.151 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T21:20:13.151 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T21:20:13.494 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 6 == 0 2024-09-15T21:20:13.494 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-15T21:20:13.494 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-15T21:20:13.494 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-15T21:20:13.494 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T21:20:13.494 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T21:20:13.494 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T21:20:13.494 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T21:20:13.768 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=4 2024-09-15T21:20:13.769 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-15T21:20:13.769 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T21:20:13.769 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T21:20:14.113 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 = 6 2024-09-15T21:20:14.113 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 4 '!=' -1 2024-09-15T21:20:14.113 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: loop=0 2024-09-15T21:20:14.113 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1652: wait_for_clean: num_active_clean=4 2024-09-15T21:20:14.113 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-15T21:20:14.113 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 0.1 2024-09-15T21:20:14.214 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-15T21:20:14.214 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-15T21:20:14.215 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-15T21:20:14.215 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-15T21:20:14.215 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T21:20:14.215 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T21:20:14.215 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T21:20:14.215 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T21:20:14.490 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=4 2024-09-15T21:20:14.491 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-15T21:20:14.491 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T21:20:14.491 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T21:20:14.836 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 = 6 2024-09-15T21:20:14.837 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 4 '!=' 4 2024-09-15T21:20:14.837 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: get_is_making_recovery_progress 2024-09-15T21:20:14.837 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2024-09-15T21:20:14.837 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-15T21:20:14.837 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-15T21:20:14.837 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-15T21:20:14.837 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2024-09-15T21:20:14.837 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-15T21:20:15.183 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=null 2024-09-15T21:20:15.183 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test null '!=' null 2024-09-15T21:20:15.184 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1655: wait_for_clean: (( 1 >= 13 )) 2024-09-15T21:20:15.184 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-15T21:20:15.184 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 0.2 2024-09-15T21:20:15.384 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-15T21:20:15.384 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-15T21:20:15.385 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-15T21:20:15.385 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-15T21:20:15.385 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T21:20:15.386 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T21:20:15.386 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T21:20:15.386 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T21:20:15.679 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=4 2024-09-15T21:20:15.679 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-15T21:20:15.679 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T21:20:15.679 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T21:20:16.021 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 = 6 2024-09-15T21:20:16.021 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 4 '!=' 4 2024-09-15T21:20:16.021 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: get_is_making_recovery_progress 2024-09-15T21:20:16.021 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2024-09-15T21:20:16.022 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-15T21:20:16.022 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-15T21:20:16.022 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-15T21:20:16.022 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2024-09-15T21:20:16.022 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-15T21:20:16.370 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=null 2024-09-15T21:20:16.370 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test null '!=' null 2024-09-15T21:20:16.371 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1655: wait_for_clean: (( 2 >= 13 )) 2024-09-15T21:20:16.371 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-15T21:20:16.371 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 0.4 2024-09-15T21:20:16.771 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-15T21:20:16.771 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-15T21:20:16.772 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-15T21:20:16.772 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-15T21:20:16.772 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T21:20:16.772 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T21:20:16.772 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T21:20:16.772 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T21:20:17.042 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=6 2024-09-15T21:20:17.042 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-15T21:20:17.042 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T21:20:17.042 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T21:20:17.391 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 = 6 2024-09-15T21:20:17.391 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-15T21:20:17.391 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-15T21:20:17.391 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3612: corrupt_scrub_erasure: for i in $(seq 1 $total_objs) 2024-09-15T21:20:17.391 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3613: corrupt_scrub_erasure: objname=EOBJ6 2024-09-15T21:20:17.391 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3614: corrupt_scrub_erasure: add_something td/osd-scrub-repair ecpool EOBJ6 2024-09-15T21:20:17.391 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-09-15T21:20:17.391 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=ecpool 2024-09-15T21:20:17.391 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=EOBJ6 2024-09-15T21:20:17.392 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-09-15T21:20:17.392 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-09-15T21:20:17.392 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-09-15T21:20:17.769 INFO:tasks.workunit.client.0.smithi136.stderr:noscrub is set 2024-09-15T21:20:17.780 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-09-15T21:20:18.271 INFO:tasks.workunit.client.0.smithi136.stderr:nodeep-scrub is set 2024-09-15T21:20:18.283 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-09-15T21:20:18.283 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-09-15T21:20:18.283 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool ecpool put EOBJ6 td/osd-scrub-repair/ORIGINAL 2024-09-15T21:20:18.562 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3616: corrupt_scrub_erasure: expr 6 % 2 2024-09-15T21:20:18.564 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3616: corrupt_scrub_erasure: local osd=0 2024-09-15T21:20:18.564 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3618: corrupt_scrub_erasure: case $i in 2024-09-15T21:20:18.565 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3657: corrupt_scrub_erasure: objectstore_tool td/osd-scrub-repair 0 EOBJ6 rm-attr hinfo_key 2024-09-15T21:20:18.565 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2024-09-15T21:20:18.565 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2024-09-15T21:20:18.565 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=0 2024-09-15T21:20:18.565 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2024-09-15T21:20:18.565 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 EOBJ6 rm-attr hinfo_key 2024-09-15T21:20:18.566 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-09-15T21:20:18.566 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2024-09-15T21:20:18.566 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=0 2024-09-15T21:20:18.566 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2024-09-15T21:20:18.566 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2024-09-15T21:20:18.566 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-15T21:20:18.566 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-15T21:20:18.566 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-15T21:20:18.566 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-15T21:20:18.567 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-15T21:20:18.871 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-15T21:20:18.872 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 EOBJ6 rm-attr hinfo_key 2024-09-15T21:20:18.872 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-15T21:20:18.872 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-15T21:20:18.872 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=0 2024-09-15T21:20:18.872 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-15T21:20:18.872 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-09-15T21:20:18.873 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 EOBJ6 rm-attr hinfo_key 2024-09-15T21:20:21.407 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 2024-09-15T21:20:21.407 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2024-09-15T21:20:21.407 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-15T21:20:21.407 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=0 2024-09-15T21:20:21.407 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-15T21:20:21.407 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-09-15T21:20:21.407 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=ce4892c7-6c6d-4b4f-8e6f-07a3e11133e5 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-15T21:20:21.407 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T21:20:21.408 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T21:20:21.408 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T21:20:21.408 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-09-15T21:20:21.408 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-09-15T21:20:21.408 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-15T21:20:21.408 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-15T21:20:21.408 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-15T21:20:21.408 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-15T21:20:21.408 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-15T21:20:21.408 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-15T21:20:21.408 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-15T21:20:21.408 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T21:20:21.408 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52602 2024-09-15T21:20:21.409 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.52602/$cluster-$name.asok' 2024-09-15T21:20:21.409 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.52602/$cluster-$name.asok' 2024-09-15T21:20:21.409 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-15T21:20:21.409 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-15T21:20:21.409 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-15T21:20:21.409 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T21:20:21.409 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T21:20:21.409 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T21:20:21.409 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T21:20:21.409 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-15T21:20:21.409 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-15T21:20:21.409 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-09-15T21:20:21.410 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.0 2024-09-15T21:20:21.410 INFO:tasks.workunit.client.0.smithi136.stderr:start osd.0 2024-09-15T21:20:21.410 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 0 --fsid=ce4892c7-6c6d-4b4f-8e6f-07a3e11133e5 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.52602/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T21:20:21.411 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-09-15T21:20:21.411 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 0 = 0 ']' 2024-09-15T21:20:21.411 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-15T21:20:21.411 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-15T21:20:21.411 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-15T21:20:21.439 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:20:21.435+0000 7fb152810640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:20:21.441 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:20:21.437+0000 7fb152810640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:20:21.443 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:20:21.439+0000 7fb152810640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:20:21.696 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 0 2024-09-15T21:20:21.696 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-15T21:20:21.696 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2024-09-15T21:20:21.696 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-15T21:20:21.696 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-15T21:20:21.696 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:20:21.696 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-15T21:20:21.696 INFO:tasks.workunit.client.0.smithi136.stderr:0 2024-09-15T21:20:21.697 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:20:21.697 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:20:21.972 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:20:22.544 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:20:22.540+0000 7fb152810640 -1 Falling back to public interface 2024-09-15T21:20:22.973 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:20:22.973 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:20:22.974 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-15T21:20:22.974 INFO:tasks.workunit.client.0.smithi136.stderr:1 2024-09-15T21:20:22.974 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:20:22.974 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:20:23.261 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:20:24.262 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:20:24.262 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:20:24.262 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-15T21:20:24.263 INFO:tasks.workunit.client.0.smithi136.stderr:2 2024-09-15T21:20:24.263 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:20:24.263 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:20:24.547 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:20:25.490 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:20:25.487+0000 7fb152810640 -1 osd.0 95 log_to_monitors true 2024-09-15T21:20:25.549 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:20:25.549 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:20:25.549 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-15T21:20:25.549 INFO:tasks.workunit.client.0.smithi136.stderr:3 2024-09-15T21:20:25.549 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:20:25.549 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:20:25.970 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:20:26.971 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:20:26.971 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:20:26.972 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-15T21:20:26.972 INFO:tasks.workunit.client.0.smithi136.stderr:4 2024-09-15T21:20:26.972 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:20:26.972 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:20:27.261 INFO:tasks.workunit.client.0.smithi136.stderr:osd.0 up in weight 1 up_from 100 up_thru 100 down_at 96 last_clean_interval [67,95) [v2:127.0.0.1:6802/858820717,v1:127.0.0.1:6803/858820717] [v2:127.0.0.1:6804/858820717,v1:127.0.0.1:6805/858820717] exists,up 9087e8dc-fe45-44af-af39-0644acbc5cf9 2024-09-15T21:20:27.261 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-15T21:20:27.261 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-15T21:20:27.261 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-15T21:20:27.261 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2024-09-15T21:20:27.261 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-15T21:20:27.261 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-15T21:20:27.261 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-15T21:20:27.262 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-15T21:20:27.262 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-15T21:20:27.262 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-15T21:20:27.262 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-15T21:20:27.262 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-15T21:20:27.262 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-15T21:20:27.353 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-15T21:20:27.353 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-15T21:20:27.353 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-15T21:20:27.354 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-15T21:20:27.354 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-15T21:20:27.354 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-15T21:20:27.725 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-15T21:20:27.725 INFO:tasks.workunit.client.0.smithi136.stderr:1 2024-09-15T21:20:27.725 INFO:tasks.workunit.client.0.smithi136.stderr:2' 2024-09-15T21:20:27.725 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-15T21:20:27.725 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T21:20:27.726 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-15T21:20:27.827 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=429496729602 2024-09-15T21:20:27.827 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 429496729602 2024-09-15T21:20:27.828 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-429496729602' 2024-09-15T21:20:27.828 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T21:20:27.828 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-15T21:20:27.931 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=390842023942 2024-09-15T21:20:27.931 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 390842023942 2024-09-15T21:20:27.931 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-429496729602 1-390842023942' 2024-09-15T21:20:27.931 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T21:20:27.931 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-15T21:20:28.034 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=352187318282 2024-09-15T21:20:28.034 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 352187318282 2024-09-15T21:20:28.034 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-429496729602 1-390842023942 2-352187318282' 2024-09-15T21:20:28.034 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T21:20:28.035 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-429496729602 2024-09-15T21:20:28.035 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T21:20:28.035 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-15T21:20:28.036 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-429496729602 2024-09-15T21:20:28.036 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T21:20:28.037 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=429496729602 2024-09-15T21:20:28.037 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 429496729602' 2024-09-15T21:20:28.037 INFO:tasks.workunit.client.0.smithi136.stderr:waiting osd.0 seq 429496729602 2024-09-15T21:20:28.037 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T21:20:28.328 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 1 -lt 429496729602 2024-09-15T21:20:28.328 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-15T21:20:29.329 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-15T21:20:29.330 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T21:20:29.612 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 429496729602 -lt 429496729602 2024-09-15T21:20:29.612 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T21:20:29.612 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-390842023942 2024-09-15T21:20:29.612 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T21:20:29.613 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-15T21:20:29.614 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-390842023942 2024-09-15T21:20:29.614 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T21:20:29.615 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=390842023942 2024-09-15T21:20:29.615 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 390842023942' 2024-09-15T21:20:29.615 INFO:tasks.workunit.client.0.smithi136.stderr:waiting osd.1 seq 390842023942 2024-09-15T21:20:29.615 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-15T21:20:29.893 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 390842023942 -lt 390842023942 2024-09-15T21:20:29.893 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T21:20:29.894 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-352187318282 2024-09-15T21:20:29.894 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T21:20:29.894 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-15T21:20:29.895 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-352187318282 2024-09-15T21:20:29.895 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T21:20:29.896 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=352187318282 2024-09-15T21:20:29.896 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 352187318282' 2024-09-15T21:20:29.897 INFO:tasks.workunit.client.0.smithi136.stderr:waiting osd.2 seq 352187318282 2024-09-15T21:20:29.897 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-15T21:20:30.172 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 352187318282 -lt 352187318282 2024-09-15T21:20:30.172 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-15T21:20:30.173 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T21:20:30.173 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T21:20:30.520 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 6 == 0 2024-09-15T21:20:30.520 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-15T21:20:30.520 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-15T21:20:30.520 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-15T21:20:30.521 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T21:20:30.521 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T21:20:30.521 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T21:20:30.521 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T21:20:30.802 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=6 2024-09-15T21:20:30.803 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-15T21:20:30.803 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T21:20:30.803 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T21:20:31.148 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 = 6 2024-09-15T21:20:31.148 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-15T21:20:31.148 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-15T21:20:31.148 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3658: corrupt_scrub_erasure: echo -n bad-val 2024-09-15T21:20:31.148 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3659: corrupt_scrub_erasure: objectstore_tool td/osd-scrub-repair 1 EOBJ6 set-attr hinfo_key td/osd-scrub-repair/bad-val 2024-09-15T21:20:31.149 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2024-09-15T21:20:31.149 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2024-09-15T21:20:31.149 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=1 2024-09-15T21:20:31.149 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2024-09-15T21:20:31.149 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 EOBJ6 set-attr hinfo_key td/osd-scrub-repair/bad-val 2024-09-15T21:20:31.149 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-09-15T21:20:31.149 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2024-09-15T21:20:31.149 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=1 2024-09-15T21:20:31.149 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2024-09-15T21:20:31.149 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-09-15T21:20:31.149 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-15T21:20:31.149 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-15T21:20:31.149 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-15T21:20:31.150 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-15T21:20:31.150 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-15T21:20:31.454 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-15T21:20:31.454 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 EOBJ6 set-attr hinfo_key td/osd-scrub-repair/bad-val 2024-09-15T21:20:31.455 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-15T21:20:31.455 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-15T21:20:31.455 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=1 2024-09-15T21:20:31.455 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-15T21:20:31.455 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-09-15T21:20:31.455 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 EOBJ6 set-attr hinfo_key td/osd-scrub-repair/bad-val 2024-09-15T21:20:33.656 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2024-09-15T21:20:33.656 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2024-09-15T21:20:33.656 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-15T21:20:33.656 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=1 2024-09-15T21:20:33.656 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-15T21:20:33.657 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-09-15T21:20:33.657 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=ce4892c7-6c6d-4b4f-8e6f-07a3e11133e5 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-15T21:20:33.657 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T21:20:33.657 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T21:20:33.657 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T21:20:33.657 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-09-15T21:20:33.657 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-09-15T21:20:33.657 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-15T21:20:33.657 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-15T21:20:33.657 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-15T21:20:33.657 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-15T21:20:33.657 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-15T21:20:33.657 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-15T21:20:33.658 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-15T21:20:33.658 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T21:20:33.658 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52602 2024-09-15T21:20:33.658 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.52602/$cluster-$name.asok' 2024-09-15T21:20:33.658 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.52602/$cluster-$name.asok' 2024-09-15T21:20:33.658 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-15T21:20:33.658 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-15T21:20:33.658 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-15T21:20:33.658 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T21:20:33.658 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T21:20:33.658 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T21:20:33.658 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T21:20:33.658 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-15T21:20:33.658 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-15T21:20:33.659 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-09-15T21:20:33.659 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.1 2024-09-15T21:20:33.659 INFO:tasks.workunit.client.0.smithi136.stderr:start osd.1 2024-09-15T21:20:33.659 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 1 --fsid=ce4892c7-6c6d-4b4f-8e6f-07a3e11133e5 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.52602/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T21:20:33.659 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-09-15T21:20:33.660 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 1 = 1 ']' 2024-09-15T21:20:33.660 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-15T21:20:33.660 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-15T21:20:33.660 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-15T21:20:33.686 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:20:33.683+0000 7f1239a0f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:20:33.689 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:20:33.686+0000 7f1239a0f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:20:33.690 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:20:33.687+0000 7f1239a0f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:20:33.943 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 1 2024-09-15T21:20:33.943 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-15T21:20:33.943 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2024-09-15T21:20:33.943 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-15T21:20:33.943 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-15T21:20:33.943 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:20:33.943 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-15T21:20:33.943 INFO:tasks.workunit.client.0.smithi136.stderr:0 2024-09-15T21:20:33.944 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:20:33.944 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T21:20:34.217 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:20:35.054 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:20:35.051+0000 7f1239a0f640 -1 Falling back to public interface 2024-09-15T21:20:35.219 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:20:35.219 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:20:35.219 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-15T21:20:35.219 INFO:tasks.workunit.client.0.smithi136.stderr:1 2024-09-15T21:20:35.219 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:20:35.219 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T21:20:35.505 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:20:36.506 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:20:36.507 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:20:36.507 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-15T21:20:36.507 INFO:tasks.workunit.client.0.smithi136.stderr:2 2024-09-15T21:20:36.507 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:20:36.507 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T21:20:36.784 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:20:37.743 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:20:37.740+0000 7f1239a0f640 -1 osd.1 102 log_to_monitors true 2024-09-15T21:20:37.785 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:20:37.785 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:20:37.786 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-15T21:20:37.786 INFO:tasks.workunit.client.0.smithi136.stderr:3 2024-09-15T21:20:37.786 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:20:37.786 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T21:20:38.148 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:20:39.149 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:20:39.149 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:20:39.150 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-15T21:20:39.150 INFO:tasks.workunit.client.0.smithi136.stderr:4 2024-09-15T21:20:39.150 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:20:39.150 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T21:20:39.510 INFO:tasks.workunit.client.0.smithi136.stderr:osd.1 up in weight 1 up_from 107 up_thru 107 down_at 103 last_clean_interval [91,102) [v2:127.0.0.1:6810/1051736006,v1:127.0.0.1:6811/1051736006] [v2:127.0.0.1:6812/1051736006,v1:127.0.0.1:6813/1051736006] exists,up 87e38427-d391-4909-9e45-e7b2d413fd8f 2024-09-15T21:20:39.510 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-15T21:20:39.510 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-15T21:20:39.510 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-15T21:20:39.510 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2024-09-15T21:20:39.510 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-15T21:20:39.510 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-15T21:20:39.510 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-15T21:20:39.510 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-15T21:20:39.511 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-15T21:20:39.511 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-15T21:20:39.511 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-15T21:20:39.511 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-15T21:20:39.511 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-15T21:20:39.601 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-15T21:20:39.602 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-15T21:20:39.602 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-15T21:20:39.602 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-15T21:20:39.602 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-15T21:20:39.602 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-15T21:20:39.905 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-15T21:20:39.905 INFO:tasks.workunit.client.0.smithi136.stderr:1 2024-09-15T21:20:39.905 INFO:tasks.workunit.client.0.smithi136.stderr:2' 2024-09-15T21:20:39.905 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-15T21:20:39.905 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T21:20:39.905 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-15T21:20:40.008 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=429496729605 2024-09-15T21:20:40.008 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 429496729605 2024-09-15T21:20:40.009 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-429496729605' 2024-09-15T21:20:40.009 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T21:20:40.009 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-15T21:20:40.113 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=459561500674 2024-09-15T21:20:40.113 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 459561500674 2024-09-15T21:20:40.113 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-429496729605 1-459561500674' 2024-09-15T21:20:40.113 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T21:20:40.113 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-15T21:20:40.214 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=352187318286 2024-09-15T21:20:40.214 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 352187318286 2024-09-15T21:20:40.214 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-429496729605 1-459561500674 2-352187318286' 2024-09-15T21:20:40.214 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T21:20:40.215 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-429496729605 2024-09-15T21:20:40.215 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T21:20:40.215 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-15T21:20:40.216 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-429496729605 2024-09-15T21:20:40.216 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T21:20:40.217 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=429496729605 2024-09-15T21:20:40.217 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 429496729605' 2024-09-15T21:20:40.217 INFO:tasks.workunit.client.0.smithi136.stderr:waiting osd.0 seq 429496729605 2024-09-15T21:20:40.218 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T21:20:40.523 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 429496729605 -lt 429496729605 2024-09-15T21:20:40.523 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T21:20:40.524 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-459561500674 2024-09-15T21:20:40.524 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T21:20:40.524 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-15T21:20:40.525 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-459561500674 2024-09-15T21:20:40.525 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T21:20:40.526 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=459561500674 2024-09-15T21:20:40.526 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 459561500674' 2024-09-15T21:20:40.526 INFO:tasks.workunit.client.0.smithi136.stderr:waiting osd.1 seq 459561500674 2024-09-15T21:20:40.527 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-15T21:20:40.816 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 459561500674 -lt 459561500674 2024-09-15T21:20:40.870 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T21:20:40.871 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-352187318286 2024-09-15T21:20:40.871 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T21:20:40.872 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-15T21:20:40.872 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-352187318286 2024-09-15T21:20:40.872 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T21:20:40.872 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=352187318286 2024-09-15T21:20:40.872 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 352187318286' 2024-09-15T21:20:40.872 INFO:tasks.workunit.client.0.smithi136.stderr:waiting osd.2 seq 352187318286 2024-09-15T21:20:40.873 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-15T21:20:41.096 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 352187318286 -lt 352187318286 2024-09-15T21:20:41.096 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-15T21:20:41.096 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T21:20:41.097 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T21:20:41.445 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 6 == 0 2024-09-15T21:20:41.445 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-15T21:20:41.445 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-15T21:20:41.445 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-15T21:20:41.445 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T21:20:41.445 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T21:20:41.445 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T21:20:41.445 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T21:20:41.726 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=6 2024-09-15T21:20:41.727 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-15T21:20:41.727 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T21:20:41.727 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T21:20:42.069 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 = 6 2024-09-15T21:20:42.069 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-15T21:20:42.069 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-15T21:20:42.070 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3612: corrupt_scrub_erasure: for i in $(seq 1 $total_objs) 2024-09-15T21:20:42.070 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3613: corrupt_scrub_erasure: objname=EOBJ7 2024-09-15T21:20:42.070 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3614: corrupt_scrub_erasure: add_something td/osd-scrub-repair ecpool EOBJ7 2024-09-15T21:20:42.070 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-09-15T21:20:42.070 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=ecpool 2024-09-15T21:20:42.070 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=EOBJ7 2024-09-15T21:20:42.070 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-09-15T21:20:42.070 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-09-15T21:20:42.070 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-09-15T21:20:42.486 INFO:tasks.workunit.client.0.smithi136.stderr:noscrub is set 2024-09-15T21:20:42.497 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-09-15T21:20:42.988 INFO:tasks.workunit.client.0.smithi136.stderr:nodeep-scrub is set 2024-09-15T21:20:43.000 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-09-15T21:20:43.000 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-09-15T21:20:43.000 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool ecpool put EOBJ7 td/osd-scrub-repair/ORIGINAL 2024-09-15T21:20:43.168 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3616: corrupt_scrub_erasure: expr 7 % 2 2024-09-15T21:20:43.169 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3616: corrupt_scrub_erasure: local osd=1 2024-09-15T21:20:43.169 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3618: corrupt_scrub_erasure: case $i in 2024-09-15T21:20:43.169 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3663: corrupt_scrub_erasure: local payload=MAKETHISDIFFERENTFROMOTHEROBJECTS 2024-09-15T21:20:43.169 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3664: corrupt_scrub_erasure: echo MAKETHISDIFFERENTFROMOTHEROBJECTS 2024-09-15T21:20:43.169 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3665: corrupt_scrub_erasure: rados --pool ecpool put EOBJ7 td/osd-scrub-repair/DIFFERENT 2024-09-15T21:20:43.372 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3668: corrupt_scrub_erasure: objectstore_tool td/osd-scrub-repair 0 EOBJ1 get-attr hinfo_key 2024-09-15T21:20:43.372 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2024-09-15T21:20:43.372 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2024-09-15T21:20:43.372 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=0 2024-09-15T21:20:43.372 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2024-09-15T21:20:43.372 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 EOBJ1 get-attr hinfo_key 2024-09-15T21:20:43.372 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-09-15T21:20:43.372 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2024-09-15T21:20:43.372 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=0 2024-09-15T21:20:43.372 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2024-09-15T21:20:43.372 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2024-09-15T21:20:43.372 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-15T21:20:43.373 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-15T21:20:43.373 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-15T21:20:43.373 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-15T21:20:43.373 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-15T21:20:43.676 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-15T21:20:43.676 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 EOBJ1 get-attr hinfo_key 2024-09-15T21:20:43.676 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-15T21:20:43.676 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-15T21:20:43.676 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=0 2024-09-15T21:20:43.676 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-15T21:20:43.677 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-09-15T21:20:43.677 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 EOBJ1 get-attr hinfo_key 2024-09-15T21:20:45.672 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 2024-09-15T21:20:45.672 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2024-09-15T21:20:45.673 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-15T21:20:45.673 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=0 2024-09-15T21:20:45.673 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-15T21:20:45.673 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-09-15T21:20:45.673 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=ce4892c7-6c6d-4b4f-8e6f-07a3e11133e5 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-15T21:20:45.673 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T21:20:45.674 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T21:20:45.674 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T21:20:45.674 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-09-15T21:20:45.674 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-09-15T21:20:45.674 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-15T21:20:45.674 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-15T21:20:45.674 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-15T21:20:45.674 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-15T21:20:45.674 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-15T21:20:45.675 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-15T21:20:45.675 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-15T21:20:45.675 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T21:20:45.675 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52602 2024-09-15T21:20:45.675 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.52602/$cluster-$name.asok' 2024-09-15T21:20:45.675 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.52602/$cluster-$name.asok' 2024-09-15T21:20:45.675 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-15T21:20:45.675 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-15T21:20:45.675 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-15T21:20:45.675 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T21:20:45.676 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T21:20:45.676 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T21:20:45.676 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T21:20:45.676 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-15T21:20:45.676 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-15T21:20:45.676 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-09-15T21:20:45.676 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.0 2024-09-15T21:20:45.676 INFO:tasks.workunit.client.0.smithi136.stderr:start osd.0 2024-09-15T21:20:45.676 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 0 --fsid=ce4892c7-6c6d-4b4f-8e6f-07a3e11133e5 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.52602/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T21:20:45.676 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-09-15T21:20:45.677 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 0 = 0 ']' 2024-09-15T21:20:45.677 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-15T21:20:45.677 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-15T21:20:45.677 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-15T21:20:45.703 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:20:45.699+0000 7fd1e8a32640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:20:45.705 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:20:45.702+0000 7fd1e8a32640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:20:45.706 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:20:45.703+0000 7fd1e8a32640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:20:45.963 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 0 2024-09-15T21:20:45.963 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-15T21:20:45.963 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2024-09-15T21:20:45.963 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-15T21:20:45.963 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-15T21:20:45.963 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:20:45.963 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-15T21:20:45.963 INFO:tasks.workunit.client.0.smithi136.stderr:0 2024-09-15T21:20:45.963 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:20:45.964 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:20:46.243 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:20:46.835 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:20:46.832+0000 7fd1e8a32640 -1 Falling back to public interface 2024-09-15T21:20:47.245 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:20:47.245 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:20:47.245 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-15T21:20:47.246 INFO:tasks.workunit.client.0.smithi136.stderr:1 2024-09-15T21:20:47.246 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:20:47.246 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:20:47.533 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:20:48.534 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:20:48.535 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:20:48.535 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-15T21:20:48.535 INFO:tasks.workunit.client.0.smithi136.stderr:2 2024-09-15T21:20:48.535 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:20:48.535 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:20:48.834 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:20:49.321 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:20:49.318+0000 7fd1e8a32640 -1 osd.0 111 log_to_monitors true 2024-09-15T21:20:49.835 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:20:49.835 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:20:49.836 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-15T21:20:49.836 INFO:tasks.workunit.client.0.smithi136.stderr:3 2024-09-15T21:20:49.836 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:20:49.836 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:20:50.274 INFO:tasks.workunit.client.0.smithi136.stderr:osd.0 up in weight 1 up_from 116 up_thru 107 down_at 112 last_clean_interval [100,111) [v2:127.0.0.1:6802/921017123,v1:127.0.0.1:6803/921017123] [v2:127.0.0.1:6804/921017123,v1:127.0.0.1:6805/921017123] exists,up 9087e8dc-fe45-44af-af39-0644acbc5cf9 2024-09-15T21:20:50.274 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-15T21:20:50.274 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-15T21:20:50.274 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-15T21:20:50.274 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2024-09-15T21:20:50.274 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-15T21:20:50.274 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-15T21:20:50.274 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-15T21:20:50.274 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-15T21:20:50.275 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-15T21:20:50.275 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-15T21:20:50.275 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-15T21:20:50.275 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-15T21:20:50.275 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-15T21:20:50.366 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-15T21:20:50.366 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-15T21:20:50.367 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-15T21:20:50.367 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-15T21:20:50.367 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-15T21:20:50.367 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-15T21:20:50.658 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-15T21:20:50.658 INFO:tasks.workunit.client.0.smithi136.stderr:1 2024-09-15T21:20:50.658 INFO:tasks.workunit.client.0.smithi136.stderr:2' 2024-09-15T21:20:50.658 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-15T21:20:50.658 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T21:20:50.658 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-15T21:20:50.851 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=498216206338 2024-09-15T21:20:50.851 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 498216206338 2024-09-15T21:20:50.851 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-498216206338' 2024-09-15T21:20:50.851 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T21:20:50.851 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-15T21:20:51.045 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=459561500677 2024-09-15T21:20:51.045 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 459561500677 2024-09-15T21:20:51.045 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-498216206338 1-459561500677' 2024-09-15T21:20:51.045 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T21:20:51.045 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-15T21:20:51.151 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=352187318289 2024-09-15T21:20:51.151 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 352187318289 2024-09-15T21:20:51.151 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-498216206338 1-459561500677 2-352187318289' 2024-09-15T21:20:51.151 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T21:20:51.152 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-498216206338 2024-09-15T21:20:51.152 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T21:20:51.152 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-15T21:20:51.153 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-498216206338 2024-09-15T21:20:51.153 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T21:20:51.154 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=498216206338 2024-09-15T21:20:51.155 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 498216206338' 2024-09-15T21:20:51.155 INFO:tasks.workunit.client.0.smithi136.stderr:waiting osd.0 seq 498216206338 2024-09-15T21:20:51.155 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T21:20:51.460 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 1 -lt 498216206338 2024-09-15T21:20:51.460 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-15T21:20:52.461 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-15T21:20:52.462 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T21:20:52.749 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 498216206338 -lt 498216206338 2024-09-15T21:20:52.750 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T21:20:52.750 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-459561500677 2024-09-15T21:20:52.750 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T21:20:52.751 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-15T21:20:52.751 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-459561500677 2024-09-15T21:20:52.752 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T21:20:52.753 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=459561500677 2024-09-15T21:20:52.753 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 459561500677' 2024-09-15T21:20:52.753 INFO:tasks.workunit.client.0.smithi136.stderr:waiting osd.1 seq 459561500677 2024-09-15T21:20:52.753 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-15T21:20:53.027 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 459561500677 -lt 459561500677 2024-09-15T21:20:53.028 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T21:20:53.028 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-352187318289 2024-09-15T21:20:53.028 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T21:20:53.029 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-15T21:20:53.029 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-352187318289 2024-09-15T21:20:53.030 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T21:20:53.030 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=352187318289 2024-09-15T21:20:53.031 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 352187318289' 2024-09-15T21:20:53.031 INFO:tasks.workunit.client.0.smithi136.stderr:waiting osd.2 seq 352187318289 2024-09-15T21:20:53.031 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-15T21:20:53.305 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 352187318289 -lt 352187318289 2024-09-15T21:20:53.306 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-15T21:20:53.306 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T21:20:53.306 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T21:20:53.648 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 6 == 0 2024-09-15T21:20:53.648 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-15T21:20:53.649 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-15T21:20:53.649 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-15T21:20:53.649 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T21:20:53.649 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T21:20:53.649 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T21:20:53.649 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T21:20:53.922 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=2 2024-09-15T21:20:53.923 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-15T21:20:53.923 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T21:20:53.923 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T21:20:54.265 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 2 = 6 2024-09-15T21:20:54.265 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 2 '!=' -1 2024-09-15T21:20:54.265 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: loop=0 2024-09-15T21:20:54.265 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1652: wait_for_clean: num_active_clean=2 2024-09-15T21:20:54.265 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-15T21:20:54.265 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 0.1 2024-09-15T21:20:54.366 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-15T21:20:54.366 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-15T21:20:54.367 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-15T21:20:54.367 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-15T21:20:54.367 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T21:20:54.367 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T21:20:54.367 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T21:20:54.367 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T21:20:54.654 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=5 2024-09-15T21:20:54.655 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-15T21:20:54.655 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T21:20:54.655 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T21:20:55.001 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 = 6 2024-09-15T21:20:55.001 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 5 '!=' 2 2024-09-15T21:20:55.001 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: loop=0 2024-09-15T21:20:55.001 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1652: wait_for_clean: num_active_clean=5 2024-09-15T21:20:55.001 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-15T21:20:55.001 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 0.1 2024-09-15T21:20:55.101 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-15T21:20:55.102 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-15T21:20:55.102 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-15T21:20:55.103 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-15T21:20:55.103 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T21:20:55.103 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T21:20:55.103 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T21:20:55.103 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T21:20:55.375 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=5 2024-09-15T21:20:55.375 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-15T21:20:55.375 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T21:20:55.375 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T21:20:55.719 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 = 6 2024-09-15T21:20:55.719 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 5 '!=' 5 2024-09-15T21:20:55.719 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: get_is_making_recovery_progress 2024-09-15T21:20:55.719 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2024-09-15T21:20:55.719 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-15T21:20:55.719 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-15T21:20:55.720 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-15T21:20:55.720 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2024-09-15T21:20:55.720 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-15T21:20:56.065 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=null 2024-09-15T21:20:56.065 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test null '!=' null 2024-09-15T21:20:56.065 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1655: wait_for_clean: (( 1 >= 13 )) 2024-09-15T21:20:56.065 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-15T21:20:56.065 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 0.2 2024-09-15T21:20:56.266 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-15T21:20:56.266 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-15T21:20:56.267 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-15T21:20:56.267 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-15T21:20:56.267 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T21:20:56.267 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T21:20:56.267 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T21:20:56.267 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T21:20:56.559 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=6 2024-09-15T21:20:56.560 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-15T21:20:56.560 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T21:20:56.560 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T21:20:56.908 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 = 6 2024-09-15T21:20:56.908 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-15T21:20:56.908 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-15T21:20:56.909 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3669: corrupt_scrub_erasure: objectstore_tool td/osd-scrub-repair 0 EOBJ7 set-attr hinfo_key td/osd-scrub-repair/hinfo 2024-09-15T21:20:56.909 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2024-09-15T21:20:56.909 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2024-09-15T21:20:56.909 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=0 2024-09-15T21:20:56.909 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2024-09-15T21:20:56.909 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 EOBJ7 set-attr hinfo_key td/osd-scrub-repair/hinfo 2024-09-15T21:20:56.909 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-09-15T21:20:56.909 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2024-09-15T21:20:56.909 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=0 2024-09-15T21:20:56.909 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2024-09-15T21:20:56.909 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2024-09-15T21:20:56.909 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-15T21:20:56.909 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-15T21:20:56.910 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-15T21:20:56.910 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-15T21:20:56.910 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-15T21:20:57.215 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-15T21:20:57.215 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 EOBJ7 set-attr hinfo_key td/osd-scrub-repair/hinfo 2024-09-15T21:20:57.215 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-15T21:20:57.215 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-15T21:20:57.215 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=0 2024-09-15T21:20:57.215 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-15T21:20:57.215 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-09-15T21:20:57.216 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 EOBJ7 set-attr hinfo_key td/osd-scrub-repair/hinfo 2024-09-15T21:20:59.563 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 2024-09-15T21:20:59.563 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2024-09-15T21:20:59.563 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-15T21:20:59.564 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=0 2024-09-15T21:20:59.564 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-15T21:20:59.564 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-09-15T21:20:59.564 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=ce4892c7-6c6d-4b4f-8e6f-07a3e11133e5 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-15T21:20:59.564 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T21:20:59.564 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T21:20:59.564 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T21:20:59.564 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-09-15T21:20:59.564 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-09-15T21:20:59.564 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-15T21:20:59.564 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-15T21:20:59.564 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-15T21:20:59.565 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-15T21:20:59.565 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-15T21:20:59.565 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-15T21:20:59.565 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-15T21:20:59.565 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T21:20:59.565 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52602 2024-09-15T21:20:59.565 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.52602/$cluster-$name.asok' 2024-09-15T21:20:59.565 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.52602/$cluster-$name.asok' 2024-09-15T21:20:59.565 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-15T21:20:59.565 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-15T21:20:59.565 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-15T21:20:59.565 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T21:20:59.565 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T21:20:59.565 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T21:20:59.566 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T21:20:59.566 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-15T21:20:59.566 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-15T21:20:59.566 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-09-15T21:20:59.566 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.0 2024-09-15T21:20:59.566 INFO:tasks.workunit.client.0.smithi136.stderr:start osd.0 2024-09-15T21:20:59.566 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 0 --fsid=ce4892c7-6c6d-4b4f-8e6f-07a3e11133e5 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.52602/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T21:20:59.566 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-09-15T21:20:59.567 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 0 = 0 ']' 2024-09-15T21:20:59.567 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-15T21:20:59.567 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-15T21:20:59.567 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-15T21:20:59.592 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:20:59.589+0000 7f9c4b1db640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:20:59.595 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:20:59.591+0000 7f9c4b1db640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:20:59.596 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:20:59.593+0000 7f9c4b1db640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:20:59.853 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 0 2024-09-15T21:20:59.853 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-15T21:20:59.853 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2024-09-15T21:20:59.853 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-15T21:20:59.853 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-15T21:20:59.853 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:20:59.853 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-15T21:20:59.853 INFO:tasks.workunit.client.0.smithi136.stderr:0 2024-09-15T21:20:59.853 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:20:59.853 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:21:00.131 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:21:01.132 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:21:01.133 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:21:01.133 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-15T21:21:01.133 INFO:tasks.workunit.client.0.smithi136.stderr:1 2024-09-15T21:21:01.133 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:21:01.133 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:21:01.207 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:21:01.204+0000 7f9c4b1db640 -1 Falling back to public interface 2024-09-15T21:21:01.418 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:21:02.420 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:21:02.420 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:21:02.420 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-15T21:21:02.420 INFO:tasks.workunit.client.0.smithi136.stderr:2 2024-09-15T21:21:02.420 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:21:02.420 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:21:02.705 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:21:03.706 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:21:03.707 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:21:03.707 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-15T21:21:03.707 INFO:tasks.workunit.client.0.smithi136.stderr:3 2024-09-15T21:21:03.707 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:21:03.707 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:21:03.991 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:21:04.111 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:21:04.107+0000 7f9c4b1db640 -1 osd.0 118 log_to_monitors true 2024-09-15T21:21:04.993 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:21:04.993 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:21:04.993 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-15T21:21:04.993 INFO:tasks.workunit.client.0.smithi136.stderr:4 2024-09-15T21:21:04.993 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:21:04.993 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:21:05.297 INFO:tasks.workunit.client.0.smithi136.stderr:osd.0 up in weight 1 up_from 123 up_thru 123 down_at 119 last_clean_interval [116,118) [v2:127.0.0.1:6802/1028549135,v1:127.0.0.1:6803/1028549135] [v2:127.0.0.1:6804/1028549135,v1:127.0.0.1:6805/1028549135] exists,up 9087e8dc-fe45-44af-af39-0644acbc5cf9 2024-09-15T21:21:05.297 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-15T21:21:05.298 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-15T21:21:05.298 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-15T21:21:05.298 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2024-09-15T21:21:05.298 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-15T21:21:05.298 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-15T21:21:05.298 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-15T21:21:05.298 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-15T21:21:05.298 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-15T21:21:05.298 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-15T21:21:05.298 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-15T21:21:05.298 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-15T21:21:05.298 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-15T21:21:05.388 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-15T21:21:05.388 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-15T21:21:05.388 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-15T21:21:05.388 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-15T21:21:05.389 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-15T21:21:05.389 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-15T21:21:05.775 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-15T21:21:05.775 INFO:tasks.workunit.client.0.smithi136.stderr:1 2024-09-15T21:21:05.776 INFO:tasks.workunit.client.0.smithi136.stderr:2' 2024-09-15T21:21:05.776 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-15T21:21:05.776 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T21:21:05.776 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-15T21:21:05.968 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=528280977410 2024-09-15T21:21:05.969 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 528280977410 2024-09-15T21:21:05.969 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-528280977410' 2024-09-15T21:21:05.969 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T21:21:05.969 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-15T21:21:06.110 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=459561500681 2024-09-15T21:21:06.111 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 459561500681 2024-09-15T21:21:06.111 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-528280977410 1-459561500681' 2024-09-15T21:21:06.111 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T21:21:06.111 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-15T21:21:06.215 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=352187318293 2024-09-15T21:21:06.215 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 352187318293 2024-09-15T21:21:06.215 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-528280977410 1-459561500681 2-352187318293' 2024-09-15T21:21:06.215 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T21:21:06.216 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-528280977410 2024-09-15T21:21:06.216 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T21:21:06.216 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-15T21:21:06.217 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-528280977410 2024-09-15T21:21:06.218 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T21:21:06.218 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=528280977410 2024-09-15T21:21:06.218 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 528280977410' 2024-09-15T21:21:06.218 INFO:tasks.workunit.client.0.smithi136.stderr:waiting osd.0 seq 528280977410 2024-09-15T21:21:06.219 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T21:21:06.550 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 528280977410 -lt 528280977410 2024-09-15T21:21:06.550 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T21:21:06.550 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-459561500681 2024-09-15T21:21:06.550 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T21:21:06.551 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-15T21:21:06.552 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-459561500681 2024-09-15T21:21:06.552 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T21:21:06.553 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=459561500681 2024-09-15T21:21:06.553 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 459561500681' 2024-09-15T21:21:06.553 INFO:tasks.workunit.client.0.smithi136.stderr:waiting osd.1 seq 459561500681 2024-09-15T21:21:06.553 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-15T21:21:06.871 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 459561500681 -lt 459561500681 2024-09-15T21:21:06.872 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T21:21:06.872 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-352187318293 2024-09-15T21:21:06.872 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T21:21:06.873 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-15T21:21:06.874 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-352187318293 2024-09-15T21:21:06.874 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T21:21:06.875 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=352187318293 2024-09-15T21:21:06.875 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 352187318293' 2024-09-15T21:21:06.875 INFO:tasks.workunit.client.0.smithi136.stderr:waiting osd.2 seq 352187318293 2024-09-15T21:21:06.875 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-15T21:21:07.153 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 352187318293 -lt 352187318293 2024-09-15T21:21:07.154 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-15T21:21:07.154 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T21:21:07.154 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T21:21:07.500 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 6 == 0 2024-09-15T21:21:07.500 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-15T21:21:07.500 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-15T21:21:07.500 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-15T21:21:07.500 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T21:21:07.500 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T21:21:07.500 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T21:21:07.501 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T21:21:07.775 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=4 2024-09-15T21:21:07.776 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-15T21:21:07.776 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T21:21:07.776 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T21:21:08.119 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 = 6 2024-09-15T21:21:08.119 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 4 '!=' -1 2024-09-15T21:21:08.120 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: loop=0 2024-09-15T21:21:08.120 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1652: wait_for_clean: num_active_clean=4 2024-09-15T21:21:08.120 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-15T21:21:08.120 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 0.1 2024-09-15T21:21:08.220 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-15T21:21:08.220 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-15T21:21:08.221 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-15T21:21:08.221 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-15T21:21:08.221 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T21:21:08.221 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T21:21:08.221 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T21:21:08.221 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T21:21:08.496 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=4 2024-09-15T21:21:08.496 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-15T21:21:08.497 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T21:21:08.497 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T21:21:08.839 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 = 6 2024-09-15T21:21:08.839 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 4 '!=' 4 2024-09-15T21:21:08.839 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: get_is_making_recovery_progress 2024-09-15T21:21:08.839 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2024-09-15T21:21:08.839 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-15T21:21:08.840 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-15T21:21:08.840 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-15T21:21:08.840 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2024-09-15T21:21:08.840 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-15T21:21:09.181 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=null 2024-09-15T21:21:09.181 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test null '!=' null 2024-09-15T21:21:09.181 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1655: wait_for_clean: (( 1 >= 13 )) 2024-09-15T21:21:09.181 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-15T21:21:09.181 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 0.2 2024-09-15T21:21:09.382 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-15T21:21:09.382 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-15T21:21:09.383 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-15T21:21:09.383 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-15T21:21:09.383 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T21:21:09.383 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T21:21:09.383 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T21:21:09.383 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T21:21:09.662 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=4 2024-09-15T21:21:09.662 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-15T21:21:09.662 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T21:21:09.663 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T21:21:10.005 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 = 6 2024-09-15T21:21:10.005 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 4 '!=' 4 2024-09-15T21:21:10.005 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: get_is_making_recovery_progress 2024-09-15T21:21:10.005 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2024-09-15T21:21:10.005 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-15T21:21:10.005 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-15T21:21:10.005 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-15T21:21:10.005 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2024-09-15T21:21:10.006 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-15T21:21:10.355 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=null 2024-09-15T21:21:10.355 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test null '!=' null 2024-09-15T21:21:10.356 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1655: wait_for_clean: (( 2 >= 13 )) 2024-09-15T21:21:10.356 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-15T21:21:10.356 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 0.4 2024-09-15T21:21:10.756 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-15T21:21:10.757 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-15T21:21:10.758 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-15T21:21:10.758 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-15T21:21:10.758 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T21:21:10.758 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T21:21:10.758 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T21:21:10.758 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T21:21:11.034 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=6 2024-09-15T21:21:11.035 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-15T21:21:11.035 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T21:21:11.035 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T21:21:11.387 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 = 6 2024-09-15T21:21:11.387 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-15T21:21:11.387 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-15T21:21:11.387 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3670: corrupt_scrub_erasure: rm -f td/osd-scrub-repair/hinfo 2024-09-15T21:21:11.388 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3676: corrupt_scrub_erasure: get_pg ecpool EOBJ0 2024-09-15T21:21:11.389 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1084: get_pg: local poolname=ecpool 2024-09-15T21:21:11.389 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1085: get_pg: local objectname=EOBJ0 2024-09-15T21:21:11.389 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1087: get_pg: ceph --format json osd map ecpool EOBJ0 2024-09-15T21:21:11.389 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1087: get_pg: jq -r .pgid 2024-09-15T21:21:11.668 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3676: corrupt_scrub_erasure: local pg=3.0 2024-09-15T21:21:11.668 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3678: corrupt_scrub_erasure: pg_scrub 3.0 2024-09-15T21:21:11.668 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1883: pg_scrub: local pgid=3.0 2024-09-15T21:21:11.668 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1884: pg_scrub: get_last_scrub_stamp 3.0 2024-09-15T21:21:11.668 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=3.0 2024-09-15T21:21:11.668 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-15T21:21:11.668 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-15T21:21:11.668 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_scrub_stamp' 2024-09-15T21:21:11.940 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1884: pg_scrub: local last_scrub=2024-09-15T21:18:28.337213+0000 2024-09-15T21:21:11.941 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1885: pg_scrub: ceph pg scrub 3.0 2024-09-15T21:21:12.203 INFO:tasks.workunit.client.0.smithi136.stderr:instructing pg 3.0s0 on osd.1 to scrub 2024-09-15T21:21:12.216 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1886: pg_scrub: wait_for_scrub 3.0 2024-09-15T21:18:28.337213+0000 2024-09-15T21:21:12.216 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: local pgid=3.0 2024-09-15T21:21:12.216 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2018: wait_for_scrub: local last_scrub=2024-09-15T21:18:28.337213+0000 2024-09-15T21:21:12.216 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2019: wait_for_scrub: local sname=last_scrub_stamp 2024-09-15T21:21:12.216 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i=0 )) 2024-09-15T21:21:12.216 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-15T21:21:12.216 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 3.0 last_scrub_stamp 2024-09-15T21:21:12.216 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=3.0 2024-09-15T21:21:12.216 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-15T21:21:12.216 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-15T21:21:12.217 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_scrub_stamp' 2024-09-15T21:21:12.488 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-15T21:18:28.337213+0000 '>' 2024-09-15T21:18:28.337213+0000 2024-09-15T21:21:12.488 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-15T21:21:13.489 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-15T21:21:13.489 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-15T21:21:13.490 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 3.0 last_scrub_stamp 2024-09-15T21:21:13.490 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=3.0 2024-09-15T21:21:13.490 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-15T21:21:13.490 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-15T21:21:13.490 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_scrub_stamp' 2024-09-15T21:21:13.769 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-15T21:18:28.337213+0000 '>' 2024-09-15T21:18:28.337213+0000 2024-09-15T21:21:13.769 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-15T21:21:14.770 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-15T21:21:14.770 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-15T21:21:14.771 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 3.0 last_scrub_stamp 2024-09-15T21:21:14.771 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=3.0 2024-09-15T21:21:14.771 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-15T21:21:14.771 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-15T21:21:14.771 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_scrub_stamp' 2024-09-15T21:21:15.054 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-15T21:21:12.692759+0000 '>' 2024-09-15T21:18:28.337213+0000 2024-09-15T21:21:15.054 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2023: wait_for_scrub: return 0 2024-09-15T21:21:15.054 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3680: corrupt_scrub_erasure: rados list-inconsistent-pg ecpool 2024-09-15T21:21:15.083 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3682: corrupt_scrub_erasure: jq '. | length' td/osd-scrub-repair/json 2024-09-15T21:21:15.085 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3682: corrupt_scrub_erasure: test 1 = 1 2024-09-15T21:21:15.085 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3684: corrupt_scrub_erasure: jq -r '.[0]' td/osd-scrub-repair/json 2024-09-15T21:21:15.087 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3684: corrupt_scrub_erasure: test 3.0 = 3.0 2024-09-15T21:21:15.088 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3686: corrupt_scrub_erasure: rados list-inconsistent-obj 3.0 2024-09-15T21:21:15.115 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3688: corrupt_scrub_erasure: jq .epoch td/osd-scrub-repair/json 2024-09-15T21:21:15.118 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3688: corrupt_scrub_erasure: epoch=123 2024-09-15T21:21:15.118 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3690: corrupt_scrub_erasure: jq 'def walk(f): 2024-09-15T21:21:15.118 INFO:tasks.workunit.client.0.smithi136.stderr: . as $in 2024-09-15T21:21:15.118 INFO:tasks.workunit.client.0.smithi136.stderr: | if type == "object" then 2024-09-15T21:21:15.118 INFO:tasks.workunit.client.0.smithi136.stderr: reduce keys[] as $key 2024-09-15T21:21:15.119 INFO:tasks.workunit.client.0.smithi136.stderr: ( {}; . + { ($key): ($in[$key] | walk(f)) } ) | f 2024-09-15T21:21:15.119 INFO:tasks.workunit.client.0.smithi136.stderr: elif type == "array" then map( walk(f) ) | f 2024-09-15T21:21:15.119 INFO:tasks.workunit.client.0.smithi136.stderr: else f 2024-09-15T21:21:15.119 INFO:tasks.workunit.client.0.smithi136.stderr: end; 2024-09-15T21:21:15.119 INFO:tasks.workunit.client.0.smithi136.stderr:walk(if type == "object" then del(.mtime) else . end) 2024-09-15T21:21:15.119 INFO:tasks.workunit.client.0.smithi136.stderr:| walk(if type == "object" then del(.local_mtime) else . end) 2024-09-15T21:21:15.119 INFO:tasks.workunit.client.0.smithi136.stderr:| walk(if type == "object" then del(.last_reqid) else . end) 2024-09-15T21:21:15.119 INFO:tasks.workunit.client.0.smithi136.stderr:| walk(if type == "object" then del(.version) else . end) 2024-09-15T21:21:15.119 INFO:tasks.workunit.client.0.smithi136.stderr:| walk(if type == "object" then del(.prior_version) else . end)' 2024-09-15T21:21:15.119 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3690: corrupt_scrub_erasure: jq .inconsistents 2024-09-15T21:21:15.120 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3690: corrupt_scrub_erasure: python3 -c 'import json; import sys ; JSON=sys.stdin.read() ; ud = json.loads(JSON) ; print(json.dumps(ud, sort_keys=True, indent=2))' 2024-09-15T21:21:15.142 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4310: corrupt_scrub_erasure: jq 'def walk(f): 2024-09-15T21:21:15.142 INFO:tasks.workunit.client.0.smithi136.stderr: . as $in 2024-09-15T21:21:15.142 INFO:tasks.workunit.client.0.smithi136.stderr: | if type == "object" then 2024-09-15T21:21:15.142 INFO:tasks.workunit.client.0.smithi136.stderr: reduce keys[] as $key 2024-09-15T21:21:15.142 INFO:tasks.workunit.client.0.smithi136.stderr: ( {}; . + { ($key): ($in[$key] | walk(f)) } ) | f 2024-09-15T21:21:15.142 INFO:tasks.workunit.client.0.smithi136.stderr: elif type == "array" then map( walk(f) ) | f 2024-09-15T21:21:15.142 INFO:tasks.workunit.client.0.smithi136.stderr: else f 2024-09-15T21:21:15.142 INFO:tasks.workunit.client.0.smithi136.stderr: end; 2024-09-15T21:21:15.142 INFO:tasks.workunit.client.0.smithi136.stderr:walk(if type == "object" then del(.mtime) else . end) 2024-09-15T21:21:15.142 INFO:tasks.workunit.client.0.smithi136.stderr:| walk(if type == "object" then del(.local_mtime) else . end) 2024-09-15T21:21:15.142 INFO:tasks.workunit.client.0.smithi136.stderr:| walk(if type == "object" then del(.last_reqid) else . end) 2024-09-15T21:21:15.142 INFO:tasks.workunit.client.0.smithi136.stderr:| walk(if type == "object" then del(.version) else . end) 2024-09-15T21:21:15.142 INFO:tasks.workunit.client.0.smithi136.stderr:| walk(if type == "object" then del(.prior_version) else . end)' td/osd-scrub-repair/json 2024-09-15T21:21:15.143 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4310: corrupt_scrub_erasure: jq .inconsistents 2024-09-15T21:21:15.143 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4310: corrupt_scrub_erasure: python3 -c 'import json; import sys ; JSON=sys.stdin.read() ; ud = json.loads(JSON) ; print(json.dumps(ud, sort_keys=True, indent=2))' 2024-09-15T21:21:15.164 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4311: corrupt_scrub_erasure: multidiff td/osd-scrub-repair/checkcsjson td/osd-scrub-repair/csjson 2024-09-15T21:21:15.164 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2434: multidiff: diff td/osd-scrub-repair/checkcsjson td/osd-scrub-repair/csjson 2024-09-15T21:21:15.165 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4312: corrupt_scrub_erasure: test no = yes 2024-09-15T21:21:15.165 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4317: corrupt_scrub_erasure: test '' = yes 2024-09-15T21:21:15.165 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4322: corrupt_scrub_erasure: pg_deep_scrub 3.0 2024-09-15T21:21:15.165 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1890: pg_deep_scrub: local pgid=3.0 2024-09-15T21:21:15.166 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1891: pg_deep_scrub: get_last_scrub_stamp 3.0 last_deep_scrub_stamp 2024-09-15T21:21:15.166 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=3.0 2024-09-15T21:21:15.166 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_deep_scrub_stamp 2024-09-15T21:21:15.166 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-15T21:21:15.166 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_deep_scrub_stamp' 2024-09-15T21:21:15.438 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1891: pg_deep_scrub: local last_scrub=2024-09-15T21:18:28.337213+0000 2024-09-15T21:21:15.438 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1892: pg_deep_scrub: ceph pg deep-scrub 3.0 2024-09-15T21:21:15.701 INFO:tasks.workunit.client.0.smithi136.stderr:instructing pg 3.0s0 on osd.1 to deep-scrub 2024-09-15T21:21:15.713 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1893: pg_deep_scrub: wait_for_scrub 3.0 2024-09-15T21:18:28.337213+0000 last_deep_scrub_stamp 2024-09-15T21:21:15.713 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: local pgid=3.0 2024-09-15T21:21:15.714 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2018: wait_for_scrub: local last_scrub=2024-09-15T21:18:28.337213+0000 2024-09-15T21:21:15.714 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2019: wait_for_scrub: local sname=last_deep_scrub_stamp 2024-09-15T21:21:15.714 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i=0 )) 2024-09-15T21:21:15.714 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-15T21:21:15.714 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 3.0 last_deep_scrub_stamp 2024-09-15T21:21:15.714 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=3.0 2024-09-15T21:21:15.714 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_deep_scrub_stamp 2024-09-15T21:21:15.714 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-15T21:21:15.714 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_deep_scrub_stamp' 2024-09-15T21:21:15.987 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-15T21:18:28.337213+0000 '>' 2024-09-15T21:18:28.337213+0000 2024-09-15T21:21:15.987 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-15T21:21:16.988 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-15T21:21:16.988 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-15T21:21:16.989 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 3.0 last_deep_scrub_stamp 2024-09-15T21:21:16.989 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=3.0 2024-09-15T21:21:16.989 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_deep_scrub_stamp 2024-09-15T21:21:16.989 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-15T21:21:16.989 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_deep_scrub_stamp' 2024-09-15T21:21:17.264 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-15T21:18:28.337213+0000 '>' 2024-09-15T21:18:28.337213+0000 2024-09-15T21:21:17.264 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-15T21:21:18.265 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-15T21:21:18.265 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-15T21:21:18.266 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 3.0 last_deep_scrub_stamp 2024-09-15T21:21:18.266 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=3.0 2024-09-15T21:21:18.266 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_deep_scrub_stamp 2024-09-15T21:21:18.266 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-15T21:21:18.266 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_deep_scrub_stamp' 2024-09-15T21:21:18.552 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-15T21:21:15.855764+0000 '>' 2024-09-15T21:18:28.337213+0000 2024-09-15T21:21:18.553 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2023: wait_for_scrub: return 0 2024-09-15T21:21:18.553 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4324: corrupt_scrub_erasure: rados list-inconsistent-pg ecpool 2024-09-15T21:21:18.582 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4326: corrupt_scrub_erasure: jq '. | length' td/osd-scrub-repair/json 2024-09-15T21:21:18.584 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4326: corrupt_scrub_erasure: test 1 = 1 2024-09-15T21:21:18.584 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4328: corrupt_scrub_erasure: jq -r '.[0]' td/osd-scrub-repair/json 2024-09-15T21:21:18.586 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4328: corrupt_scrub_erasure: test 3.0 = 3.0 2024-09-15T21:21:18.587 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4330: corrupt_scrub_erasure: rados list-inconsistent-obj 3.0 2024-09-15T21:21:18.614 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4332: corrupt_scrub_erasure: jq .epoch td/osd-scrub-repair/json 2024-09-15T21:21:18.617 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4332: corrupt_scrub_erasure: epoch=123 2024-09-15T21:21:18.617 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4334: corrupt_scrub_erasure: '[' false = true ']' 2024-09-15T21:21:18.617 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4991: corrupt_scrub_erasure: jq 'def walk(f): 2024-09-15T21:21:18.617 INFO:tasks.workunit.client.0.smithi136.stderr: . as $in 2024-09-15T21:21:18.617 INFO:tasks.workunit.client.0.smithi136.stderr: | if type == "object" then 2024-09-15T21:21:18.617 INFO:tasks.workunit.client.0.smithi136.stderr: reduce keys[] as $key 2024-09-15T21:21:18.617 INFO:tasks.workunit.client.0.smithi136.stderr: ( {}; . + { ($key): ($in[$key] | walk(f)) } ) | f 2024-09-15T21:21:18.617 INFO:tasks.workunit.client.0.smithi136.stderr: elif type == "array" then map( walk(f) ) | f 2024-09-15T21:21:18.617 INFO:tasks.workunit.client.0.smithi136.stderr: else f 2024-09-15T21:21:18.618 INFO:tasks.workunit.client.0.smithi136.stderr: end; 2024-09-15T21:21:18.618 INFO:tasks.workunit.client.0.smithi136.stderr:walk(if type == "object" then del(.mtime) else . end) 2024-09-15T21:21:18.618 INFO:tasks.workunit.client.0.smithi136.stderr:| walk(if type == "object" then del(.local_mtime) else . end) 2024-09-15T21:21:18.618 INFO:tasks.workunit.client.0.smithi136.stderr:| walk(if type == "object" then del(.last_reqid) else . end) 2024-09-15T21:21:18.618 INFO:tasks.workunit.client.0.smithi136.stderr:| walk(if type == "object" then del(.version) else . end) 2024-09-15T21:21:18.618 INFO:tasks.workunit.client.0.smithi136.stderr:| walk(if type == "object" then del(.prior_version) else . end)' 2024-09-15T21:21:18.618 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4991: corrupt_scrub_erasure: jq .inconsistents 2024-09-15T21:21:18.618 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4991: corrupt_scrub_erasure: python3 -c 'import json; import sys ; JSON=sys.stdin.read() ; ud = json.loads(JSON) ; print(json.dumps(ud, sort_keys=True, indent=2))' 2024-09-15T21:21:18.641 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5724: corrupt_scrub_erasure: jq 'def walk(f): 2024-09-15T21:21:18.641 INFO:tasks.workunit.client.0.smithi136.stderr: . as $in 2024-09-15T21:21:18.641 INFO:tasks.workunit.client.0.smithi136.stderr: | if type == "object" then 2024-09-15T21:21:18.641 INFO:tasks.workunit.client.0.smithi136.stderr: reduce keys[] as $key 2024-09-15T21:21:18.641 INFO:tasks.workunit.client.0.smithi136.stderr: ( {}; . + { ($key): ($in[$key] | walk(f)) } ) | f 2024-09-15T21:21:18.641 INFO:tasks.workunit.client.0.smithi136.stderr: elif type == "array" then map( walk(f) ) | f 2024-09-15T21:21:18.641 INFO:tasks.workunit.client.0.smithi136.stderr: else f 2024-09-15T21:21:18.641 INFO:tasks.workunit.client.0.smithi136.stderr: end; 2024-09-15T21:21:18.641 INFO:tasks.workunit.client.0.smithi136.stderr:walk(if type == "object" then del(.mtime) else . end) 2024-09-15T21:21:18.641 INFO:tasks.workunit.client.0.smithi136.stderr:| walk(if type == "object" then del(.local_mtime) else . end) 2024-09-15T21:21:18.641 INFO:tasks.workunit.client.0.smithi136.stderr:| walk(if type == "object" then del(.last_reqid) else . end) 2024-09-15T21:21:18.641 INFO:tasks.workunit.client.0.smithi136.stderr:| walk(if type == "object" then del(.version) else . end) 2024-09-15T21:21:18.642 INFO:tasks.workunit.client.0.smithi136.stderr:| walk(if type == "object" then del(.prior_version) else . end)' td/osd-scrub-repair/json 2024-09-15T21:21:18.642 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5724: corrupt_scrub_erasure: jq .inconsistents 2024-09-15T21:21:18.642 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5724: corrupt_scrub_erasure: python3 -c 'import json; import sys ; JSON=sys.stdin.read() ; ud = json.loads(JSON) ; print(json.dumps(ud, sort_keys=True, indent=2))' 2024-09-15T21:21:18.663 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5725: corrupt_scrub_erasure: multidiff td/osd-scrub-repair/checkcsjson td/osd-scrub-repair/csjson 2024-09-15T21:21:18.663 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2434: multidiff: diff td/osd-scrub-repair/checkcsjson td/osd-scrub-repair/csjson 2024-09-15T21:21:18.665 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5726: corrupt_scrub_erasure: test no = yes 2024-09-15T21:21:18.665 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5737: corrupt_scrub_erasure: test '' = yes 2024-09-15T21:21:18.665 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5742: corrupt_scrub_erasure: ceph osd pool rm ecpool ecpool --yes-i-really-really-mean-it 2024-09-15T21:21:19.027 INFO:tasks.workunit.client.0.smithi136.stderr:pool 'ecpool' removed 2024-09-15T21:21:19.039 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:66: run: teardown td/osd-scrub-repair 2024-09-15T21:21:19.040 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/osd-scrub-repair 2024-09-15T21:21:19.040 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-15T21:21:19.040 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/osd-scrub-repair KILL 2024-09-15T21:21:19.040 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-15T21:21:19.040 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-15T21:21:19.041 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-15T21:21:19.041 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-15T21:21:19.041 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-15T21:21:19.148 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-15T21:21:19.149 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-15T21:21:19.149 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-15T21:21:19.150 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-15T21:21:19.151 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-15T21:21:19.151 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-15T21:21:19.151 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-15T21:21:19.152 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T21:21:19.152 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-15T21:21:19.153 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-15T21:21:19.153 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T21:21:19.153 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-15T21:21:19.154 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-15T21:21:19.154 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/osd-scrub-repair 2024-09-15T21:21:19.183 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-15T21:21:19.183 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T21:21:19.183 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52602 2024-09-15T21:21:19.183 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.52602 2024-09-15T21:21:19.185 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-15T21:21:19.185 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-15T21:21:19.185 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:63: run: for func in $funcs 2024-09-15T21:21:19.185 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:64: run: setup td/osd-scrub-repair 2024-09-15T21:21:19.185 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/osd-scrub-repair 2024-09-15T21:21:19.185 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/osd-scrub-repair 2024-09-15T21:21:19.185 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/osd-scrub-repair 2024-09-15T21:21:19.185 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-15T21:21:19.186 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/osd-scrub-repair KILL 2024-09-15T21:21:19.186 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-15T21:21:19.186 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-15T21:21:19.186 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-15T21:21:19.186 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-15T21:21:19.186 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-15T21:21:19.187 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-15T21:21:19.187 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-15T21:21:19.188 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-15T21:21:19.188 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-15T21:21:19.189 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-15T21:21:19.189 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-15T21:21:19.190 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-15T21:21:19.190 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T21:21:19.191 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-15T21:21:19.191 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-15T21:21:19.191 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T21:21:19.192 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-15T21:21:19.193 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-15T21:21:19.193 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/osd-scrub-repair 2024-09-15T21:21:19.194 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-15T21:21:19.194 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T21:21:19.194 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52602 2024-09-15T21:21:19.195 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.52602 2024-09-15T21:21:19.195 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-15T21:21:19.195 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-15T21:21:19.195 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/osd-scrub-repair 2024-09-15T21:21:19.197 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2024-09-15T21:21:19.197 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T21:21:19.197 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52602 2024-09-15T21:21:19.197 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.52602 2024-09-15T21:21:19.200 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2024-09-15T21:21:19.200 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 4096 -le 1024 ']' 2024-09-15T21:21:19.200 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2024-09-15T21:21:19.200 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/osd-scrub-repair 1' TERM HUP INT 2024-09-15T21:21:19.200 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:65: run: TEST_corrupt_scrub_erasure_overwrites td/osd-scrub-repair 2024-09-15T21:21:19.200 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5750: TEST_corrupt_scrub_erasure_overwrites: '[' true = true ']' 2024-09-15T21:21:19.200 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5751: TEST_corrupt_scrub_erasure_overwrites: corrupt_scrub_erasure td/osd-scrub-repair true 2024-09-15T21:21:19.200 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3596: corrupt_scrub_erasure: local dir=td/osd-scrub-repair 2024-09-15T21:21:19.200 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3597: corrupt_scrub_erasure: local allow_overwrites=true 2024-09-15T21:21:19.200 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3598: corrupt_scrub_erasure: local poolname=ecpool 2024-09-15T21:21:19.200 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3599: corrupt_scrub_erasure: local total_objs=7 2024-09-15T21:21:19.200 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3601: corrupt_scrub_erasure: run_mon td/osd-scrub-repair a 2024-09-15T21:21:19.201 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/osd-scrub-repair 2024-09-15T21:21:19.201 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2024-09-15T21:21:19.201 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2024-09-15T21:21:19.201 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2024-09-15T21:21:19.201 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/osd-scrub-repair/a 2024-09-15T21:21:19.201 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-scrub-repair/a --run-dir=td/osd-scrub-repair 2024-09-15T21:21:19.644 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2024-09-15T21:21:19.645 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-15T21:21:19.645 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-15T21:21:19.645 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-15T21:21:19.645 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T21:21:19.645 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52602 2024-09-15T21:21:19.645 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.52602/$cluster-$name.asok' 2024-09-15T21:21:19.645 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-scrub-repair/a '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.52602/$cluster-$name.asok' --mon-cluster-log-file=td/osd-scrub-repair/log --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-09-15T21:21:19.844 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2024-09-15T21:21:19.845 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2024-09-15T21:21:19.845 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-15T21:21:19.845 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-15T21:21:19.845 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2024-09-15T21:21:19.845 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2024-09-15T21:21:19.847 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-15T21:21:19.847 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-15T21:21:19.848 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-15T21:21:19.848 INFO:tasks.workunit.client.0.smithi136.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-15T21:21:19.848 INFO:tasks.workunit.client.0.smithi136.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T21:21:19.848 INFO:tasks.workunit.client.0.smithi136.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52602 2024-09-15T21:21:19.848 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.52602/ceph-mon.a.asok 2024-09-15T21:21:19.848 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-15T21:21:19.848 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.52602/ceph-mon.a.asok config get fsid 2024-09-15T21:21:19.927 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2024-09-15T21:21:19.927 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-15T21:21:19.927 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-15T21:21:19.927 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2024-09-15T21:21:19.928 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2024-09-15T21:21:19.928 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-15T21:21:19.928 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-15T21:21:19.928 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-15T21:21:19.928 INFO:tasks.workunit.client.0.smithi136.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-15T21:21:19.928 INFO:tasks.workunit.client.0.smithi136.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T21:21:19.928 INFO:tasks.workunit.client.0.smithi136.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52602 2024-09-15T21:21:19.928 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.52602/ceph-mon.a.asok 2024-09-15T21:21:19.929 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-15T21:21:19.929 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.52602/ceph-mon.a.asok config get mon_host 2024-09-15T21:21:20.005 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3602: corrupt_scrub_erasure: run_mgr td/osd-scrub-repair x 2024-09-15T21:21:20.005 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: run_mgr: local dir=td/osd-scrub-repair 2024-09-15T21:21:20.005 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: run_mgr: shift 2024-09-15T21:21:20.006 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: run_mgr: local id=x 2024-09-15T21:21:20.006 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: run_mgr: shift 2024-09-15T21:21:20.006 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:548: run_mgr: local data=td/osd-scrub-repair/x 2024-09-15T21:21:20.006 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:550: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-15T21:21:20.813 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: get_asok_path 2024-09-15T21:21:20.813 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-15T21:21:20.813 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-15T21:21:20.813 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-15T21:21:20.813 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T21:21:20.813 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52602 2024-09-15T21:21:20.813 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.52602/$cluster-$name.asok' 2024-09-15T21:21:20.814 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-15T21:21:20.814 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-scrub-repair/x '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.52602/$cluster-$name.asok' --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-15T21:21:20.842 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3603: corrupt_scrub_erasure: seq 0 2 2024-09-15T21:21:20.846 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3603: corrupt_scrub_erasure: for id in $(seq 0 2) 2024-09-15T21:21:20.847 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3604: corrupt_scrub_erasure: run_osd td/osd-scrub-repair 0 2024-09-15T21:21:20.847 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-scrub-repair 2024-09-15T21:21:20.847 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-15T21:21:20.847 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=0 2024-09-15T21:21:20.847 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-15T21:21:20.847 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-scrub-repair/0 2024-09-15T21:21:20.847 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=ce4892c7-6c6d-4b4f-8e6f-07a3e11133e5 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-15T21:21:20.847 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T21:21:20.847 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T21:21:20.847 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T21:21:20.847 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-09-15T21:21:20.847 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-09-15T21:21:20.847 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-15T21:21:20.848 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-15T21:21:20.848 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-15T21:21:20.848 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-15T21:21:20.848 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-15T21:21:20.848 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-15T21:21:20.848 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-15T21:21:20.848 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T21:21:20.848 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52602 2024-09-15T21:21:20.848 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.52602/$cluster-$name.asok' 2024-09-15T21:21:20.848 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.52602/$cluster-$name.asok' 2024-09-15T21:21:20.848 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-15T21:21:20.848 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-15T21:21:20.848 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-15T21:21:20.849 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-15T21:21:20.849 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-15T21:21:20.849 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T21:21:20.849 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T21:21:20.849 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T21:21:20.849 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T21:21:20.849 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-15T21:21:20.849 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-15T21:21:20.849 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-scrub-repair/0 2024-09-15T21:21:20.849 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-15T21:21:20.850 INFO:tasks.workunit.client.0.smithi136.stdout:add osd0 980ffc84-a9d7-4f90-928b-2a70e4c90857 2024-09-15T21:21:20.850 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=980ffc84-a9d7-4f90-928b-2a70e4c90857 2024-09-15T21:21:20.850 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd0 980ffc84-a9d7-4f90-928b-2a70e4c90857' 2024-09-15T21:21:20.850 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-15T21:21:20.868 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQDQT+dmM3JxMxAATGbe2LOg9UEtme+EAzoR5w== 2024-09-15T21:21:20.868 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQDQT+dmM3JxMxAATGbe2LOg9UEtme+EAzoR5w=="}' 2024-09-15T21:21:20.868 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 980ffc84-a9d7-4f90-928b-2a70e4c90857 -i td/osd-scrub-repair/0/new.json 2024-09-15T21:21:21.069 INFO:tasks.workunit.client.0.smithi136.stdout:0 2024-09-15T21:21:21.079 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-scrub-repair/0/new.json 2024-09-15T21:21:21.080 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 0 --fsid=ce4892c7-6c6d-4b4f-8e6f-07a3e11133e5 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.52602/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDQT+dmM3JxMxAATGbe2LOg9UEtme+EAzoR5w== --osd-uuid 980ffc84-a9d7-4f90-928b-2a70e4c90857 2024-09-15T21:21:21.109 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:21:21.106+0000 7f6af180c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:21:21.120 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:21:21.117+0000 7f6af180c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:21:21.121 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:21:21.118+0000 7f6af180c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:21:21.121 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:21:21.118+0000 7f6af180c640 -1 bdev(0x562245af7c00 td/osd-scrub-repair/0/block) open stat got: (1) Operation not permitted 2024-09-15T21:21:21.121 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:21:21.118+0000 7f6af180c640 -1 bluestore(td/osd-scrub-repair/0) _read_fsid unparsable uuid 2024-09-15T21:21:26.867 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-scrub-repair/0/keyring 2024-09-15T21:21:26.867 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-15T21:21:26.868 INFO:tasks.workunit.client.0.smithi136.stdout:adding osd0 key to auth repository 2024-09-15T21:21:26.868 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd0 key to auth repository 2024-09-15T21:21:26.868 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-scrub-repair/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-15T21:21:27.261 INFO:tasks.workunit.client.0.smithi136.stdout:start osd.0 2024-09-15T21:21:27.262 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.0 2024-09-15T21:21:27.263 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 0 --fsid=ce4892c7-6c6d-4b4f-8e6f-07a3e11133e5 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.52602/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T21:21:27.263 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-15T21:21:27.263 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-15T21:21:27.263 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-15T21:21:27.288 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:21:27.285+0000 7f62eaa14640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:21:27.290 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:21:27.287+0000 7f62eaa14640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:21:27.291 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:21:27.288+0000 7f62eaa14640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:21:27.539 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 0 2024-09-15T21:21:27.539 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-15T21:21:27.539 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2024-09-15T21:21:27.539 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-15T21:21:27.539 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-15T21:21:27.539 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:21:27.540 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-15T21:21:27.540 INFO:tasks.workunit.client.0.smithi136.stdout:0 2024-09-15T21:21:27.541 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:21:27.541 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:21:27.816 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:21:28.396 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:21:28.393+0000 7f62eaa14640 -1 Falling back to public interface 2024-09-15T21:21:28.818 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:21:28.818 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:21:28.818 INFO:tasks.workunit.client.0.smithi136.stdout:1 2024-09-15T21:21:28.819 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-15T21:21:28.819 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:21:28.819 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:21:29.103 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:21:30.105 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:21:30.105 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:21:30.105 INFO:tasks.workunit.client.0.smithi136.stdout:2 2024-09-15T21:21:30.105 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-15T21:21:30.106 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:21:30.106 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:21:30.388 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:21:30.869 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:21:30.865+0000 7f62eaa14640 -1 osd.0 0 log_to_monitors true 2024-09-15T21:21:31.389 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:21:31.389 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:21:31.390 INFO:tasks.workunit.client.0.smithi136.stdout:3 2024-09-15T21:21:31.390 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-15T21:21:31.390 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:21:31.390 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:21:31.683 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:21:32.684 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:21:32.685 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:21:32.685 INFO:tasks.workunit.client.0.smithi136.stdout:4 2024-09-15T21:21:32.685 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-15T21:21:32.685 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:21:32.686 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:21:32.972 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:21:33.974 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:21:33.974 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:21:33.974 INFO:tasks.workunit.client.0.smithi136.stdout:5 2024-09-15T21:21:33.975 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-15T21:21:33.975 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:21:33.975 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:21:34.265 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:21:35.267 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:21:35.267 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:21:35.267 INFO:tasks.workunit.client.0.smithi136.stdout:6 2024-09-15T21:21:35.268 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-15T21:21:35.268 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:21:35.268 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:21:35.653 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:21:36.654 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:21:36.654 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:21:36.654 INFO:tasks.workunit.client.0.smithi136.stdout:7 2024-09-15T21:21:36.655 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-15T21:21:36.655 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:21:36.655 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:21:36.944 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:21:37.946 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:21:37.946 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:21:37.946 INFO:tasks.workunit.client.0.smithi136.stdout:8 2024-09-15T21:21:37.947 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-15T21:21:37.947 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:21:37.947 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:21:38.236 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:21:39.237 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:21:39.237 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:21:39.237 INFO:tasks.workunit.client.0.smithi136.stdout:9 2024-09-15T21:21:39.238 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-15T21:21:39.238 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:21:39.238 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:21:39.529 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:21:40.531 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:21:40.531 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:21:40.531 INFO:tasks.workunit.client.0.smithi136.stdout:10 2024-09-15T21:21:40.532 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-15T21:21:40.532 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:21:40.532 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:21:40.852 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:21:41.853 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:21:41.853 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:21:41.854 INFO:tasks.workunit.client.0.smithi136.stdout:11 2024-09-15T21:21:41.854 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-15T21:21:41.854 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:21:41.854 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:21:42.140 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:21:43.141 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:21:43.141 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:21:43.142 INFO:tasks.workunit.client.0.smithi136.stdout:12 2024-09-15T21:21:43.142 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-15T21:21:43.142 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:21:43.142 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:21:43.602 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:21:44.604 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:21:44.604 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:21:44.604 INFO:tasks.workunit.client.0.smithi136.stdout:13 2024-09-15T21:21:44.604 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-15T21:21:44.605 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:21:44.605 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:21:45.302 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:21:46.304 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:21:46.304 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:21:46.304 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-15T21:21:46.305 INFO:tasks.workunit.client.0.smithi136.stdout:14 2024-09-15T21:21:46.305 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:21:46.305 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:21:46.957 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:21:47.958 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:21:47.959 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:21:47.959 INFO:tasks.workunit.client.0.smithi136.stdout:15 2024-09-15T21:21:47.959 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-15T21:21:47.959 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:21:47.959 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:21:48.482 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:21:49.484 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:21:49.484 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:21:49.485 INFO:tasks.workunit.client.0.smithi136.stdout:16 2024-09-15T21:21:49.485 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-15T21:21:49.485 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:21:49.485 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:21:50.164 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:21:51.168 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:21:51.168 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:21:51.169 INFO:tasks.workunit.client.0.smithi136.stdout:17 2024-09-15T21:21:51.169 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-15T21:21:51.169 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:21:51.169 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:21:51.903 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:21:52.905 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:21:52.905 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:21:52.905 INFO:tasks.workunit.client.0.smithi136.stdout:18 2024-09-15T21:21:52.906 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 18 2024-09-15T21:21:52.906 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:21:52.906 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:21:53.299 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:21:54.300 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:21:54.301 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:21:54.301 INFO:tasks.workunit.client.0.smithi136.stdout:19 2024-09-15T21:21:54.301 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 19 2024-09-15T21:21:54.301 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:21:54.301 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:21:54.584 INFO:tasks.workunit.client.0.smithi136.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/1071038501,v1:127.0.0.1:6803/1071038501] [v2:127.0.0.1:6804/1071038501,v1:127.0.0.1:6805/1071038501] exists,up 980ffc84-a9d7-4f90-928b-2a70e4c90857 2024-09-15T21:21:54.587 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-15T21:21:54.587 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-15T21:21:54.587 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-15T21:21:54.587 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3603: corrupt_scrub_erasure: for id in $(seq 0 2) 2024-09-15T21:21:54.588 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3604: corrupt_scrub_erasure: run_osd td/osd-scrub-repair 1 2024-09-15T21:21:54.588 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-scrub-repair 2024-09-15T21:21:54.588 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-15T21:21:54.588 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=1 2024-09-15T21:21:54.588 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-15T21:21:54.588 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-scrub-repair/1 2024-09-15T21:21:54.588 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=ce4892c7-6c6d-4b4f-8e6f-07a3e11133e5 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-15T21:21:54.588 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T21:21:54.588 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T21:21:54.588 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T21:21:54.588 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-09-15T21:21:54.588 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-09-15T21:21:54.588 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-15T21:21:54.589 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-15T21:21:54.589 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-15T21:21:54.589 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-15T21:21:54.589 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-15T21:21:54.589 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-15T21:21:54.589 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-15T21:21:54.589 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T21:21:54.589 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52602 2024-09-15T21:21:54.589 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.52602/$cluster-$name.asok' 2024-09-15T21:21:54.589 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.52602/$cluster-$name.asok' 2024-09-15T21:21:54.589 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-15T21:21:54.589 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-15T21:21:54.590 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-15T21:21:54.590 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-15T21:21:54.590 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-15T21:21:54.590 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T21:21:54.590 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T21:21:54.590 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T21:21:54.590 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T21:21:54.590 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-15T21:21:54.590 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-15T21:21:54.590 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-scrub-repair/1 2024-09-15T21:21:54.590 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-15T21:21:54.591 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=ea0d46ed-c598-4c0c-a177-f572f3259311 2024-09-15T21:21:54.591 INFO:tasks.workunit.client.0.smithi136.stdout:add osd1 ea0d46ed-c598-4c0c-a177-f572f3259311 2024-09-15T21:21:54.591 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd1 ea0d46ed-c598-4c0c-a177-f572f3259311' 2024-09-15T21:21:54.591 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-15T21:21:54.606 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQDyT+dm+mX1IxAA2uqUt72DgkYBIToiTBrS7Q== 2024-09-15T21:21:54.607 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQDyT+dm+mX1IxAA2uqUt72DgkYBIToiTBrS7Q=="}' 2024-09-15T21:21:54.607 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new ea0d46ed-c598-4c0c-a177-f572f3259311 -i td/osd-scrub-repair/1/new.json 2024-09-15T21:21:54.965 INFO:tasks.workunit.client.0.smithi136.stdout:1 2024-09-15T21:21:54.977 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-scrub-repair/1/new.json 2024-09-15T21:21:54.978 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 1 --fsid=ce4892c7-6c6d-4b4f-8e6f-07a3e11133e5 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.52602/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDyT+dm+mX1IxAA2uqUt72DgkYBIToiTBrS7Q== --osd-uuid ea0d46ed-c598-4c0c-a177-f572f3259311 2024-09-15T21:21:55.007 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:21:55.003+0000 7f143bb34640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:21:55.042 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:21:55.037+0000 7f143bb34640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:21:55.042 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:21:55.038+0000 7f143bb34640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:21:55.043 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:21:55.039+0000 7f143bb34640 -1 bdev(0x55f6bcfe7c00 td/osd-scrub-repair/1/block) open stat got: (1) Operation not permitted 2024-09-15T21:21:55.043 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:21:55.039+0000 7f143bb34640 -1 bluestore(td/osd-scrub-repair/1) _read_fsid unparsable uuid 2024-09-15T21:22:00.937 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-scrub-repair/1/keyring 2024-09-15T21:22:00.937 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-15T21:22:00.938 INFO:tasks.workunit.client.0.smithi136.stdout:adding osd1 key to auth repository 2024-09-15T21:22:00.939 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd1 key to auth repository 2024-09-15T21:22:00.939 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-scrub-repair/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-15T21:22:01.304 INFO:tasks.workunit.client.0.smithi136.stdout:start osd.1 2024-09-15T21:22:01.305 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.1 2024-09-15T21:22:01.305 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 1 --fsid=ce4892c7-6c6d-4b4f-8e6f-07a3e11133e5 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.52602/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T21:22:01.305 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-15T21:22:01.305 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-15T21:22:01.305 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-15T21:22:01.332 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:22:01.328+0000 7f4bf34e6640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:22:01.333 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:22:01.330+0000 7f4bf34e6640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:22:01.334 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:22:01.331+0000 7f4bf34e6640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:22:01.582 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 1 2024-09-15T21:22:01.582 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-15T21:22:01.582 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2024-09-15T21:22:01.582 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-15T21:22:01.582 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-15T21:22:01.582 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:22:01.582 INFO:tasks.workunit.client.0.smithi136.stdout:0 2024-09-15T21:22:01.583 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-15T21:22:01.583 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:22:01.583 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T21:22:01.858 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:22:02.440 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:22:02.437+0000 7f4bf34e6640 -1 Falling back to public interface 2024-09-15T21:22:02.859 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:22:02.859 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:22:02.859 INFO:tasks.workunit.client.0.smithi136.stdout:1 2024-09-15T21:22:02.860 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-15T21:22:02.860 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:22:02.860 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T21:22:03.162 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:22:04.164 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:22:04.164 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:22:04.164 INFO:tasks.workunit.client.0.smithi136.stdout:2 2024-09-15T21:22:04.165 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-15T21:22:04.165 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:22:04.165 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T21:22:04.446 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:22:05.206 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:22:05.203+0000 7f4bf34e6640 -1 osd.1 0 log_to_monitors true 2024-09-15T21:22:05.447 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:22:05.447 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:22:05.448 INFO:tasks.workunit.client.0.smithi136.stdout:3 2024-09-15T21:22:05.448 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-15T21:22:05.448 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:22:05.448 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T21:22:05.901 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:22:06.903 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:22:06.903 INFO:tasks.workunit.client.0.smithi136.stdout:4 2024-09-15T21:22:06.903 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:22:06.904 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-15T21:22:06.904 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:22:06.904 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T21:22:07.415 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:22:08.416 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:22:08.417 INFO:tasks.workunit.client.0.smithi136.stdout:5 2024-09-15T21:22:08.417 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:22:08.417 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-15T21:22:08.417 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:22:08.417 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T21:22:08.705 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:22:09.706 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:22:09.707 INFO:tasks.workunit.client.0.smithi136.stdout:6 2024-09-15T21:22:09.707 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:22:09.707 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-15T21:22:09.707 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:22:09.707 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T21:22:10.026 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:22:11.027 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:22:11.027 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:22:11.027 INFO:tasks.workunit.client.0.smithi136.stdout:7 2024-09-15T21:22:11.028 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-15T21:22:11.028 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:22:11.028 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T21:22:11.317 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:22:12.318 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:22:12.318 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:22:12.318 INFO:tasks.workunit.client.0.smithi136.stdout:8 2024-09-15T21:22:12.319 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-15T21:22:12.319 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:22:12.319 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T21:22:12.609 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:22:13.610 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:22:13.610 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:22:13.610 INFO:tasks.workunit.client.0.smithi136.stdout:9 2024-09-15T21:22:13.611 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-15T21:22:13.611 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:22:13.611 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T21:22:14.036 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:22:15.038 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:22:15.038 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:22:15.038 INFO:tasks.workunit.client.0.smithi136.stdout:10 2024-09-15T21:22:15.039 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-15T21:22:15.039 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:22:15.039 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T21:22:15.439 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:22:16.441 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:22:16.441 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:22:16.441 INFO:tasks.workunit.client.0.smithi136.stdout:11 2024-09-15T21:22:16.442 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-15T21:22:16.442 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:22:16.442 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T21:22:16.732 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:22:17.734 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:22:17.734 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:22:17.735 INFO:tasks.workunit.client.0.smithi136.stdout:12 2024-09-15T21:22:17.735 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-15T21:22:17.735 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:22:17.735 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T21:22:18.553 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:22:19.555 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:22:19.555 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:22:19.556 INFO:tasks.workunit.client.0.smithi136.stdout:13 2024-09-15T21:22:19.556 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-15T21:22:19.556 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:22:19.557 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T21:22:20.319 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:22:21.321 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:22:21.321 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:22:21.321 INFO:tasks.workunit.client.0.smithi136.stdout:14 2024-09-15T21:22:21.321 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-15T21:22:21.322 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:22:21.322 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T21:22:21.636 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:22:22.637 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:22:22.637 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:22:22.637 INFO:tasks.workunit.client.0.smithi136.stdout:15 2024-09-15T21:22:22.638 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-15T21:22:22.638 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:22:22.638 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T21:22:22.979 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:22:23.981 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:22:23.981 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:22:23.981 INFO:tasks.workunit.client.0.smithi136.stdout:16 2024-09-15T21:22:23.981 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-15T21:22:23.982 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:22:23.982 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T21:22:24.534 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:22:25.536 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:22:25.536 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:22:25.536 INFO:tasks.workunit.client.0.smithi136.stdout:17 2024-09-15T21:22:25.537 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-15T21:22:25.537 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:22:25.537 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T21:22:26.348 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:22:27.349 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:22:27.349 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:22:27.350 INFO:tasks.workunit.client.0.smithi136.stdout:18 2024-09-15T21:22:27.350 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 18 2024-09-15T21:22:27.350 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:22:27.350 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T21:22:27.718 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:22:28.719 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:22:28.719 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:22:28.719 INFO:tasks.workunit.client.0.smithi136.stdout:19 2024-09-15T21:22:28.720 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 19 2024-09-15T21:22:28.720 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:22:28.720 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T21:22:29.010 INFO:tasks.workunit.client.0.smithi136.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/1848192594,v1:127.0.0.1:6811/1848192594] [v2:127.0.0.1:6812/1848192594,v1:127.0.0.1:6813/1848192594] exists,up ea0d46ed-c598-4c0c-a177-f572f3259311 2024-09-15T21:22:29.014 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-15T21:22:29.014 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-15T21:22:29.014 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-15T21:22:29.014 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3603: corrupt_scrub_erasure: for id in $(seq 0 2) 2024-09-15T21:22:29.014 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3604: corrupt_scrub_erasure: run_osd td/osd-scrub-repair 2 2024-09-15T21:22:29.014 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-scrub-repair 2024-09-15T21:22:29.014 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-15T21:22:29.014 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=2 2024-09-15T21:22:29.014 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-15T21:22:29.014 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-scrub-repair/2 2024-09-15T21:22:29.014 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=ce4892c7-6c6d-4b4f-8e6f-07a3e11133e5 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-15T21:22:29.015 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T21:22:29.015 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T21:22:29.015 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T21:22:29.015 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/2' 2024-09-15T21:22:29.015 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/2/journal' 2024-09-15T21:22:29.015 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-15T21:22:29.015 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-15T21:22:29.015 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-15T21:22:29.015 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-15T21:22:29.015 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-15T21:22:29.015 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-15T21:22:29.015 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-15T21:22:29.015 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T21:22:29.016 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52602 2024-09-15T21:22:29.016 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.52602/$cluster-$name.asok' 2024-09-15T21:22:29.016 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.52602/$cluster-$name.asok' 2024-09-15T21:22:29.016 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-15T21:22:29.016 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-15T21:22:29.016 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-15T21:22:29.016 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-15T21:22:29.016 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-15T21:22:29.016 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T21:22:29.016 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T21:22:29.016 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T21:22:29.016 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T21:22:29.016 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-15T21:22:29.016 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-15T21:22:29.017 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-scrub-repair/2 2024-09-15T21:22:29.017 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-15T21:22:29.017 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=c090ac95-7b74-4c87-8395-5ca55c2b79b2 2024-09-15T21:22:29.017 INFO:tasks.workunit.client.0.smithi136.stdout:add osd2 c090ac95-7b74-4c87-8395-5ca55c2b79b2 2024-09-15T21:22:29.017 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd2 c090ac95-7b74-4c87-8395-5ca55c2b79b2' 2024-09-15T21:22:29.017 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-15T21:22:29.033 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQAVUOdmKD/EARAAYpRB0DLWqbEYT2ce6uLW/Q== 2024-09-15T21:22:29.033 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQAVUOdmKD/EARAAYpRB0DLWqbEYT2ce6uLW/Q=="}' 2024-09-15T21:22:29.033 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new c090ac95-7b74-4c87-8395-5ca55c2b79b2 -i td/osd-scrub-repair/2/new.json 2024-09-15T21:22:29.399 INFO:tasks.workunit.client.0.smithi136.stdout:2 2024-09-15T21:22:29.410 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-scrub-repair/2/new.json 2024-09-15T21:22:29.411 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 2 --fsid=ce4892c7-6c6d-4b4f-8e6f-07a3e11133e5 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.52602/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAVUOdmKD/EARAAYpRB0DLWqbEYT2ce6uLW/Q== --osd-uuid c090ac95-7b74-4c87-8395-5ca55c2b79b2 2024-09-15T21:22:29.441 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:22:29.437+0000 7f4df180b640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:22:29.483 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:22:29.480+0000 7f4df180b640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:22:29.484 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:22:29.481+0000 7f4df180b640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:22:29.484 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:22:29.481+0000 7f4df180b640 -1 bdev(0x5579541efc00 td/osd-scrub-repair/2/block) open stat got: (1) Operation not permitted 2024-09-15T21:22:29.484 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:22:29.481+0000 7f4df180b640 -1 bluestore(td/osd-scrub-repair/2) _read_fsid unparsable uuid 2024-09-15T21:22:35.574 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-scrub-repair/2/keyring 2024-09-15T21:22:35.575 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-15T21:22:35.576 INFO:tasks.workunit.client.0.smithi136.stdout:adding osd2 key to auth repository 2024-09-15T21:22:35.576 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd2 key to auth repository 2024-09-15T21:22:35.576 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-scrub-repair/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-15T21:22:35.940 INFO:tasks.workunit.client.0.smithi136.stdout:start osd.2 2024-09-15T21:22:35.940 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.2 2024-09-15T21:22:35.940 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 2 --fsid=ce4892c7-6c6d-4b4f-8e6f-07a3e11133e5 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.52602/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T21:22:35.941 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-15T21:22:35.941 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-15T21:22:35.941 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-15T21:22:35.967 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:22:35.964+0000 7fad02f2e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:22:35.969 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:22:35.966+0000 7fad02f2e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:22:35.970 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:22:35.967+0000 7fad02f2e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:22:36.219 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 2 2024-09-15T21:22:36.219 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-15T21:22:36.219 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=2 2024-09-15T21:22:36.219 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-15T21:22:36.219 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-15T21:22:36.219 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:22:36.220 INFO:tasks.workunit.client.0.smithi136.stdout:0 2024-09-15T21:22:36.220 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-15T21:22:36.220 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:22:36.220 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T21:22:36.499 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:22:37.080 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:22:37.077+0000 7fad02f2e640 -1 Falling back to public interface 2024-09-15T21:22:37.500 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:22:37.500 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:22:37.500 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-15T21:22:37.501 INFO:tasks.workunit.client.0.smithi136.stdout:1 2024-09-15T21:22:37.501 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:22:37.501 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T21:22:37.785 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:22:38.787 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:22:38.787 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:22:38.787 INFO:tasks.workunit.client.0.smithi136.stdout:2 2024-09-15T21:22:38.787 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-15T21:22:38.788 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:22:38.788 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T21:22:39.084 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:22:39.716 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:22:39.711+0000 7fad02f2e640 -1 osd.2 0 log_to_monitors true 2024-09-15T21:22:40.085 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:22:40.086 INFO:tasks.workunit.client.0.smithi136.stdout:3 2024-09-15T21:22:40.086 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:22:40.086 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-15T21:22:40.086 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:22:40.086 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T21:22:40.411 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:22:41.413 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:22:41.413 INFO:tasks.workunit.client.0.smithi136.stdout:4 2024-09-15T21:22:41.413 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:22:41.414 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-15T21:22:41.414 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:22:41.414 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T21:22:41.731 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:22:42.733 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:22:42.733 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:22:42.733 INFO:tasks.workunit.client.0.smithi136.stdout:5 2024-09-15T21:22:42.734 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-15T21:22:42.734 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:22:42.734 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T21:22:43.083 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:22:44.085 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:22:44.085 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:22:44.085 INFO:tasks.workunit.client.0.smithi136.stdout:6 2024-09-15T21:22:44.086 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-15T21:22:44.086 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:22:44.086 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T21:22:44.503 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:22:45.505 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:22:45.505 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:22:45.505 INFO:tasks.workunit.client.0.smithi136.stdout:7 2024-09-15T21:22:45.506 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-15T21:22:45.506 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:22:45.506 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T21:22:45.842 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:22:46.843 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:22:46.843 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:22:46.844 INFO:tasks.workunit.client.0.smithi136.stdout:8 2024-09-15T21:22:46.844 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-15T21:22:46.844 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:22:46.844 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T21:22:47.216 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:22:48.218 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:22:48.218 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:22:48.218 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-15T21:22:48.219 INFO:tasks.workunit.client.0.smithi136.stdout:9 2024-09-15T21:22:48.219 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:22:48.219 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T21:22:48.548 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:22:49.546 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:22:49.547 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:22:49.547 INFO:tasks.workunit.client.0.smithi136.stdout:10 2024-09-15T21:22:49.547 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-15T21:22:49.547 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:22:49.547 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T21:22:49.836 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:22:50.837 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:22:50.837 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:22:50.838 INFO:tasks.workunit.client.0.smithi136.stdout:11 2024-09-15T21:22:50.838 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-15T21:22:50.838 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:22:50.838 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T21:22:51.216 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:22:52.218 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:22:52.218 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:22:52.218 INFO:tasks.workunit.client.0.smithi136.stdout:12 2024-09-15T21:22:52.219 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-15T21:22:52.219 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:22:52.219 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T21:22:52.676 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:22:53.677 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:22:53.677 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:22:53.677 INFO:tasks.workunit.client.0.smithi136.stdout:13 2024-09-15T21:22:53.678 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-15T21:22:53.678 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:22:53.678 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T21:22:54.110 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:22:55.112 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:22:55.112 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:22:55.112 INFO:tasks.workunit.client.0.smithi136.stdout:14 2024-09-15T21:22:55.113 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-15T21:22:55.113 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:22:55.113 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T21:22:55.889 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:22:56.890 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:22:56.891 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:22:56.891 INFO:tasks.workunit.client.0.smithi136.stdout:15 2024-09-15T21:22:56.891 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-15T21:22:56.891 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:22:56.891 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T21:22:57.658 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:22:58.659 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:22:58.659 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:22:58.659 INFO:tasks.workunit.client.0.smithi136.stdout:16 2024-09-15T21:22:58.660 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-15T21:22:58.660 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:22:58.660 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T21:22:59.369 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:23:00.370 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:23:00.371 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:23:00.371 INFO:tasks.workunit.client.0.smithi136.stdout:17 2024-09-15T21:23:00.371 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-15T21:23:00.371 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:23:00.371 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T21:23:00.978 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:23:01.979 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:23:01.979 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:23:01.979 INFO:tasks.workunit.client.0.smithi136.stdout:18 2024-09-15T21:23:01.980 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 18 2024-09-15T21:23:01.980 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:23:01.980 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T21:23:02.385 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:23:03.387 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:23:03.387 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:23:03.387 INFO:tasks.workunit.client.0.smithi136.stdout:19 2024-09-15T21:23:03.387 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 19 2024-09-15T21:23:03.388 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:23:03.388 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T21:23:03.669 INFO:tasks.workunit.client.0.smithi136.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/2949339179,v1:127.0.0.1:6819/2949339179] [v2:127.0.0.1:6820/2949339179,v1:127.0.0.1:6821/2949339179] exists,up c090ac95-7b74-4c87-8395-5ca55c2b79b2 2024-09-15T21:23:03.669 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-15T21:23:03.670 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-15T21:23:03.670 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-15T21:23:03.670 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3606: corrupt_scrub_erasure: create_rbd_pool 2024-09-15T21:23:03.670 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:526: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2024-09-15T21:23:03.939 INFO:tasks.workunit.client.0.smithi136.stderr:pool 'rbd' does not exist 2024-09-15T21:23:03.951 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:527: create_rbd_pool: create_pool rbd 4 2024-09-15T21:23:03.951 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create rbd 4 2024-09-15T21:23:04.342 INFO:tasks.workunit.client.0.smithi136.stderr:pool 'rbd' created 2024-09-15T21:23:04.354 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2024-09-15T21:23:05.355 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:528: create_rbd_pool: rbd pool init rbd 2024-09-15T21:23:07.349 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3607: corrupt_scrub_erasure: create_pool foo 1 2024-09-15T21:23:07.349 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create foo 1 2024-09-15T21:23:07.728 INFO:tasks.workunit.client.0.smithi136.stderr:pool 'foo' created 2024-09-15T21:23:07.740 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2024-09-15T21:23:08.742 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3609: corrupt_scrub_erasure: create_ec_pool ecpool true k=2 m=1 stripe_unit=2K --force 2024-09-15T21:23:08.742 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2443: create_ec_pool: local pool_name=ecpool 2024-09-15T21:23:08.742 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2444: create_ec_pool: shift 2024-09-15T21:23:08.742 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2445: create_ec_pool: local allow_overwrites=true 2024-09-15T21:23:08.742 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2446: create_ec_pool: shift 2024-09-15T21:23:08.742 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2448: create_ec_pool: ceph osd erasure-code-profile set myprofile crush-failure-domain=osd k=2 m=1 stripe_unit=2K --force 2024-09-15T21:23:09.332 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2450: create_ec_pool: create_pool ecpool 1 1 erasure myprofile 2024-09-15T21:23:09.332 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create ecpool 1 1 erasure myprofile 2024-09-15T21:23:09.980 INFO:tasks.workunit.client.0.smithi136.stderr:pool 'ecpool' created 2024-09-15T21:23:09.994 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2024-09-15T21:23:10.996 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2452: create_ec_pool: '[' true = true ']' 2024-09-15T21:23:10.996 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2453: create_ec_pool: ceph osd pool set ecpool allow_ec_overwrites true 2024-09-15T21:23:11.883 INFO:tasks.workunit.client.0.smithi136.stderr:set pool 3 allow_ec_overwrites to true 2024-09-15T21:23:11.896 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2456: create_ec_pool: wait_for_clean 2024-09-15T21:23:11.896 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-15T21:23:11.896 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-15T21:23:11.897 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-15T21:23:11.897 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-15T21:23:11.897 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-15T21:23:11.897 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-15T21:23:11.897 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-15T21:23:11.898 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-15T21:23:11.898 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-15T21:23:11.990 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-15T21:23:11.990 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-15T21:23:11.991 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-15T21:23:11.991 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-15T21:23:11.991 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-15T21:23:11.991 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-15T21:23:12.324 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-15T21:23:12.324 INFO:tasks.workunit.client.0.smithi136.stderr:1 2024-09-15T21:23:12.324 INFO:tasks.workunit.client.0.smithi136.stderr:2' 2024-09-15T21:23:12.327 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-15T21:23:12.327 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T21:23:12.327 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-15T21:23:12.456 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836507 2024-09-15T21:23:12.456 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836507 2024-09-15T21:23:12.456 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836507' 2024-09-15T21:23:12.456 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T21:23:12.456 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-15T21:23:12.558 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=42949672977 2024-09-15T21:23:12.558 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 42949672977 2024-09-15T21:23:12.558 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836507 1-42949672977' 2024-09-15T21:23:12.558 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T21:23:12.559 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-15T21:23:12.661 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=64424509448 2024-09-15T21:23:12.661 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 64424509448 2024-09-15T21:23:12.661 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836507 1-42949672977 2-64424509448' 2024-09-15T21:23:12.661 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T21:23:12.661 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836507 2024-09-15T21:23:12.662 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T21:23:12.662 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-15T21:23:12.663 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836507 2024-09-15T21:23:12.663 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T21:23:12.664 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836507 2024-09-15T21:23:12.664 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836507' 2024-09-15T21:23:12.664 INFO:tasks.workunit.client.0.smithi136.stdout:waiting osd.0 seq 21474836507 2024-09-15T21:23:12.664 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T21:23:12.948 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836507 -lt 21474836507 2024-09-15T21:23:12.948 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T21:23:12.949 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-42949672977 2024-09-15T21:23:12.949 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T21:23:12.949 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-15T21:23:12.950 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-42949672977 2024-09-15T21:23:12.950 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T21:23:12.951 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=42949672977 2024-09-15T21:23:12.951 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 42949672977' 2024-09-15T21:23:12.951 INFO:tasks.workunit.client.0.smithi136.stdout:waiting osd.1 seq 42949672977 2024-09-15T21:23:12.952 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-15T21:23:13.228 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 42949672977 -lt 42949672977 2024-09-15T21:23:13.228 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T21:23:13.229 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-64424509448 2024-09-15T21:23:13.229 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T21:23:13.230 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-15T21:23:13.230 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-64424509448 2024-09-15T21:23:13.230 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T21:23:13.231 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=64424509448 2024-09-15T21:23:13.232 INFO:tasks.workunit.client.0.smithi136.stdout:waiting osd.2 seq 64424509448 2024-09-15T21:23:13.232 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 64424509448' 2024-09-15T21:23:13.232 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-15T21:23:13.508 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 64424509448 -lt 64424509448 2024-09-15T21:23:13.508 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-15T21:23:13.508 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T21:23:13.508 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T21:23:13.852 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 6 == 0 2024-09-15T21:23:13.853 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-15T21:23:13.853 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-15T21:23:13.853 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-15T21:23:13.853 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T21:23:13.853 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T21:23:13.853 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T21:23:13.853 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T21:23:14.126 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=6 2024-09-15T21:23:14.126 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-15T21:23:14.126 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T21:23:14.127 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T21:23:14.473 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 = 6 2024-09-15T21:23:14.474 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-15T21:23:14.474 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-15T21:23:14.474 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2457: create_ec_pool: return 0 2024-09-15T21:23:14.474 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3610: corrupt_scrub_erasure: wait_for_clean 2024-09-15T21:23:14.474 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-15T21:23:14.474 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-15T21:23:14.474 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-15T21:23:14.474 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-15T21:23:14.474 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-15T21:23:14.474 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-15T21:23:14.474 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-15T21:23:14.474 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-15T21:23:14.475 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-15T21:23:14.566 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-15T21:23:14.566 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-15T21:23:14.566 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-15T21:23:14.566 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-15T21:23:14.566 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-15T21:23:14.567 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-15T21:23:14.843 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-15T21:23:14.843 INFO:tasks.workunit.client.0.smithi136.stderr:1 2024-09-15T21:23:14.843 INFO:tasks.workunit.client.0.smithi136.stderr:2' 2024-09-15T21:23:14.843 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-15T21:23:14.843 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T21:23:14.843 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-15T21:23:14.946 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836509 2024-09-15T21:23:14.947 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836509 2024-09-15T21:23:14.947 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836509' 2024-09-15T21:23:14.947 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T21:23:14.947 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-15T21:23:15.050 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=42949672979 2024-09-15T21:23:15.050 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 42949672979 2024-09-15T21:23:15.050 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836509 1-42949672979' 2024-09-15T21:23:15.050 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T21:23:15.050 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-15T21:23:15.153 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=64424509450 2024-09-15T21:23:15.153 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 64424509450 2024-09-15T21:23:15.153 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836509 1-42949672979 2-64424509450' 2024-09-15T21:23:15.153 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T21:23:15.154 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836509 2024-09-15T21:23:15.154 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T21:23:15.154 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-15T21:23:15.155 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836509 2024-09-15T21:23:15.155 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T21:23:15.156 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836509 2024-09-15T21:23:15.156 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836509' 2024-09-15T21:23:15.156 INFO:tasks.workunit.client.0.smithi136.stdout:waiting osd.0 seq 21474836509 2024-09-15T21:23:15.157 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T21:23:15.430 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836508 -lt 21474836509 2024-09-15T21:23:15.431 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-15T21:23:16.432 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-15T21:23:16.432 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T21:23:16.723 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836508 -lt 21474836509 2024-09-15T21:23:16.723 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-15T21:23:17.724 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 299 -eq 0 ']' 2024-09-15T21:23:17.725 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T21:23:18.009 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836509 -lt 21474836509 2024-09-15T21:23:18.009 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T21:23:18.010 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-42949672979 2024-09-15T21:23:18.010 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T21:23:18.011 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-15T21:23:18.011 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-42949672979 2024-09-15T21:23:18.011 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T21:23:18.013 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=42949672979 2024-09-15T21:23:18.013 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 42949672979' 2024-09-15T21:23:18.013 INFO:tasks.workunit.client.0.smithi136.stdout:waiting osd.1 seq 42949672979 2024-09-15T21:23:18.013 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-15T21:23:18.290 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 42949672979 -lt 42949672979 2024-09-15T21:23:18.290 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T21:23:18.290 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-64424509450 2024-09-15T21:23:18.291 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T21:23:18.291 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-15T21:23:18.292 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-64424509450 2024-09-15T21:23:18.292 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T21:23:18.293 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=64424509450 2024-09-15T21:23:18.293 INFO:tasks.workunit.client.0.smithi136.stdout:waiting osd.2 seq 64424509450 2024-09-15T21:23:18.294 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 64424509450' 2024-09-15T21:23:18.294 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-15T21:23:18.570 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 64424509450 -lt 64424509450 2024-09-15T21:23:18.570 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-15T21:23:18.570 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T21:23:18.571 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T21:23:18.919 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 6 == 0 2024-09-15T21:23:18.919 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-15T21:23:18.919 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-15T21:23:18.919 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-15T21:23:18.920 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T21:23:18.920 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T21:23:18.920 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T21:23:18.920 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T21:23:19.209 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=6 2024-09-15T21:23:19.210 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-15T21:23:19.210 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T21:23:19.210 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T21:23:19.552 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 = 6 2024-09-15T21:23:19.552 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-15T21:23:19.552 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-15T21:23:19.552 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3612: corrupt_scrub_erasure: seq 1 7 2024-09-15T21:23:19.553 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3612: corrupt_scrub_erasure: for i in $(seq 1 $total_objs) 2024-09-15T21:23:19.553 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3613: corrupt_scrub_erasure: objname=EOBJ1 2024-09-15T21:23:19.554 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3614: corrupt_scrub_erasure: add_something td/osd-scrub-repair ecpool EOBJ1 2024-09-15T21:23:19.554 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-09-15T21:23:19.554 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=ecpool 2024-09-15T21:23:19.554 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=EOBJ1 2024-09-15T21:23:19.554 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-09-15T21:23:19.554 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-09-15T21:23:19.554 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-09-15T21:23:19.938 INFO:tasks.workunit.client.0.smithi136.stderr:noscrub is set 2024-09-15T21:23:19.950 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-09-15T21:23:20.457 INFO:tasks.workunit.client.0.smithi136.stderr:nodeep-scrub is set 2024-09-15T21:23:20.469 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-09-15T21:23:20.469 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-09-15T21:23:20.469 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool ecpool put EOBJ1 td/osd-scrub-repair/ORIGINAL 2024-09-15T21:23:20.720 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3616: corrupt_scrub_erasure: expr 1 % 2 2024-09-15T21:23:20.723 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3616: corrupt_scrub_erasure: local osd=1 2024-09-15T21:23:20.723 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3618: corrupt_scrub_erasure: case $i in 2024-09-15T21:23:20.723 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3621: corrupt_scrub_erasure: local payload=UVWXYZZZ 2024-09-15T21:23:20.723 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3622: corrupt_scrub_erasure: echo UVWXYZZZ 2024-09-15T21:23:20.723 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3623: corrupt_scrub_erasure: objectstore_tool td/osd-scrub-repair 1 EOBJ1 set-bytes td/osd-scrub-repair/CORRUPT 2024-09-15T21:23:20.723 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2024-09-15T21:23:20.723 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2024-09-15T21:23:20.723 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=1 2024-09-15T21:23:20.723 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2024-09-15T21:23:20.724 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 EOBJ1 set-bytes td/osd-scrub-repair/CORRUPT 2024-09-15T21:23:20.724 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-09-15T21:23:20.724 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2024-09-15T21:23:20.724 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=1 2024-09-15T21:23:20.724 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2024-09-15T21:23:20.724 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-09-15T21:23:20.724 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-15T21:23:20.724 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-15T21:23:20.724 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-15T21:23:20.724 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-15T21:23:20.724 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-15T21:23:21.029 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-15T21:23:21.029 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 EOBJ1 set-bytes td/osd-scrub-repair/CORRUPT 2024-09-15T21:23:21.029 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-15T21:23:21.029 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-15T21:23:21.029 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=1 2024-09-15T21:23:21.029 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-15T21:23:21.029 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-09-15T21:23:21.029 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 EOBJ1 set-bytes td/osd-scrub-repair/CORRUPT 2024-09-15T21:23:23.088 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2024-09-15T21:23:23.089 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2024-09-15T21:23:23.089 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-15T21:23:23.089 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=1 2024-09-15T21:23:23.089 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-15T21:23:23.089 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-09-15T21:23:23.089 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=ce4892c7-6c6d-4b4f-8e6f-07a3e11133e5 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-15T21:23:23.089 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T21:23:23.089 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T21:23:23.089 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T21:23:23.089 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-09-15T21:23:23.089 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-09-15T21:23:23.089 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-15T21:23:23.090 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-15T21:23:23.090 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-15T21:23:23.090 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-15T21:23:23.090 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-15T21:23:23.090 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-15T21:23:23.090 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-15T21:23:23.090 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T21:23:23.090 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52602 2024-09-15T21:23:23.090 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.52602/$cluster-$name.asok' 2024-09-15T21:23:23.090 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.52602/$cluster-$name.asok' 2024-09-15T21:23:23.090 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-15T21:23:23.090 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-15T21:23:23.090 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-15T21:23:23.090 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T21:23:23.090 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T21:23:23.091 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T21:23:23.091 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T21:23:23.091 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-15T21:23:23.091 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-15T21:23:23.091 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-09-15T21:23:23.092 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.1 2024-09-15T21:23:23.092 INFO:tasks.workunit.client.0.smithi136.stderr:start osd.1 2024-09-15T21:23:23.092 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 1 --fsid=ce4892c7-6c6d-4b4f-8e6f-07a3e11133e5 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.52602/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T21:23:23.092 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-09-15T21:23:23.092 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 1 = 1 ']' 2024-09-15T21:23:23.092 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-15T21:23:23.092 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-15T21:23:23.092 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-15T21:23:23.120 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:23:23.117+0000 7fc492cd8640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:23:23.122 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:23:23.119+0000 7fc492cd8640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:23:23.124 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:23:23.121+0000 7fc492cd8640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:23:23.378 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 1 2024-09-15T21:23:23.378 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-15T21:23:23.378 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2024-09-15T21:23:23.378 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-15T21:23:23.378 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-15T21:23:23.378 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:23:23.378 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-15T21:23:23.379 INFO:tasks.workunit.client.0.smithi136.stderr:0 2024-09-15T21:23:23.379 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:23:23.379 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T21:23:23.655 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:23:24.220 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:23:24.216+0000 7fc492cd8640 -1 Falling back to public interface 2024-09-15T21:23:24.657 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:23:24.657 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:23:24.658 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-15T21:23:24.658 INFO:tasks.workunit.client.0.smithi136.stderr:1 2024-09-15T21:23:24.658 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:23:24.658 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T21:23:24.986 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:23:25.988 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:23:25.988 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:23:25.988 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-15T21:23:25.988 INFO:tasks.workunit.client.0.smithi136.stderr:2 2024-09-15T21:23:25.988 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:23:25.988 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T21:23:26.301 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:23:27.296 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:23:27.292+0000 7fc492cd8640 -1 osd.1 34 log_to_monitors true 2024-09-15T21:23:27.302 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:23:27.302 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:23:27.302 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-15T21:23:27.302 INFO:tasks.workunit.client.0.smithi136.stderr:3 2024-09-15T21:23:27.303 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:23:27.303 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T21:23:27.635 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:23:28.636 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:23:28.637 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:23:28.637 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-15T21:23:28.637 INFO:tasks.workunit.client.0.smithi136.stderr:4 2024-09-15T21:23:28.637 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:23:28.637 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T21:23:29.010 INFO:tasks.workunit.client.0.smithi136.stderr:osd.1 up in weight 1 up_from 38 up_thru 38 down_at 35 last_clean_interval [10,34) [v2:127.0.0.1:6810/1261580742,v1:127.0.0.1:6811/1261580742] [v2:127.0.0.1:6812/1261580742,v1:127.0.0.1:6813/1261580742] exists,up ea0d46ed-c598-4c0c-a177-f572f3259311 2024-09-15T21:23:29.010 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-15T21:23:29.010 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-15T21:23:29.010 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-15T21:23:29.010 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2024-09-15T21:23:29.011 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-15T21:23:29.011 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-15T21:23:29.011 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-15T21:23:29.011 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-15T21:23:29.011 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-15T21:23:29.011 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-15T21:23:29.012 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-15T21:23:29.012 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-15T21:23:29.012 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-15T21:23:29.104 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-15T21:23:29.104 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-15T21:23:29.104 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-15T21:23:29.104 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-15T21:23:29.104 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-15T21:23:29.104 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-15T21:23:29.383 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-15T21:23:29.383 INFO:tasks.workunit.client.0.smithi136.stderr:1 2024-09-15T21:23:29.383 INFO:tasks.workunit.client.0.smithi136.stderr:2' 2024-09-15T21:23:29.383 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-15T21:23:29.383 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T21:23:29.383 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-15T21:23:29.492 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836513 2024-09-15T21:23:29.492 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836513 2024-09-15T21:23:29.492 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836513' 2024-09-15T21:23:29.492 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T21:23:29.492 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-15T21:23:29.703 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=163208757250 2024-09-15T21:23:29.703 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 163208757250 2024-09-15T21:23:29.703 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836513 1-163208757250' 2024-09-15T21:23:29.704 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T21:23:29.704 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-15T21:23:29.807 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=64424509454 2024-09-15T21:23:29.807 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 64424509454 2024-09-15T21:23:29.807 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836513 1-163208757250 2-64424509454' 2024-09-15T21:23:29.807 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T21:23:29.808 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836513 2024-09-15T21:23:29.808 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T21:23:29.808 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-15T21:23:29.809 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836513 2024-09-15T21:23:29.809 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T21:23:29.810 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836513 2024-09-15T21:23:29.810 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836513' 2024-09-15T21:23:29.810 INFO:tasks.workunit.client.0.smithi136.stderr:waiting osd.0 seq 21474836513 2024-09-15T21:23:29.810 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T21:23:30.085 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836511 -lt 21474836513 2024-09-15T21:23:30.085 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-15T21:23:31.086 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-15T21:23:31.087 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T21:23:31.374 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836513 -lt 21474836513 2024-09-15T21:23:31.375 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T21:23:31.375 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-163208757250 2024-09-15T21:23:31.375 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T21:23:31.376 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-15T21:23:31.377 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-163208757250 2024-09-15T21:23:31.377 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T21:23:31.378 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=163208757250 2024-09-15T21:23:31.378 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 163208757250' 2024-09-15T21:23:31.378 INFO:tasks.workunit.client.0.smithi136.stderr:waiting osd.1 seq 163208757250 2024-09-15T21:23:31.378 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-15T21:23:31.658 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 163208757250 -lt 163208757250 2024-09-15T21:23:31.658 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T21:23:31.658 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-64424509454 2024-09-15T21:23:31.658 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T21:23:31.659 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-15T21:23:31.659 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-64424509454 2024-09-15T21:23:31.660 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T21:23:31.661 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=64424509454 2024-09-15T21:23:31.661 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 64424509454' 2024-09-15T21:23:31.661 INFO:tasks.workunit.client.0.smithi136.stderr:waiting osd.2 seq 64424509454 2024-09-15T21:23:31.661 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-15T21:23:31.937 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 64424509454 -lt 64424509454 2024-09-15T21:23:31.938 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-15T21:23:31.938 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T21:23:31.938 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T21:23:32.283 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 6 == 0 2024-09-15T21:23:32.283 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-15T21:23:32.284 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-15T21:23:32.284 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-15T21:23:32.284 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T21:23:32.284 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T21:23:32.284 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T21:23:32.284 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T21:23:32.558 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=6 2024-09-15T21:23:32.559 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-15T21:23:32.559 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T21:23:32.559 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T21:23:32.902 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 = 6 2024-09-15T21:23:32.902 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-15T21:23:32.903 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-15T21:23:32.903 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3612: corrupt_scrub_erasure: for i in $(seq 1 $total_objs) 2024-09-15T21:23:32.903 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3613: corrupt_scrub_erasure: objname=EOBJ2 2024-09-15T21:23:32.903 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3614: corrupt_scrub_erasure: add_something td/osd-scrub-repair ecpool EOBJ2 2024-09-15T21:23:32.903 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-09-15T21:23:32.903 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=ecpool 2024-09-15T21:23:32.903 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=EOBJ2 2024-09-15T21:23:32.903 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-09-15T21:23:32.903 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-09-15T21:23:32.903 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-09-15T21:23:33.280 INFO:tasks.workunit.client.0.smithi136.stderr:noscrub is set 2024-09-15T21:23:33.291 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-09-15T21:23:33.773 INFO:tasks.workunit.client.0.smithi136.stderr:nodeep-scrub is set 2024-09-15T21:23:33.785 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-09-15T21:23:33.785 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-09-15T21:23:33.785 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool ecpool put EOBJ2 td/osd-scrub-repair/ORIGINAL 2024-09-15T21:23:33.961 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3616: corrupt_scrub_erasure: expr 2 % 2 2024-09-15T21:23:33.962 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3616: corrupt_scrub_erasure: local osd=0 2024-09-15T21:23:33.962 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3618: corrupt_scrub_erasure: case $i in 2024-09-15T21:23:33.962 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3628: corrupt_scrub_erasure: dd if=/dev/urandom of=td/osd-scrub-repair/CORRUPT bs=2048 count=1 2024-09-15T21:23:33.964 INFO:tasks.workunit.client.0.smithi136.stderr:1+0 records in 2024-09-15T21:23:33.964 INFO:tasks.workunit.client.0.smithi136.stderr:1+0 records out 2024-09-15T21:23:33.964 INFO:tasks.workunit.client.0.smithi136.stderr:2048 bytes (2.0 kB, 2.0 KiB) copied, 7.668e-05 s, 26.7 MB/s 2024-09-15T21:23:33.964 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3629: corrupt_scrub_erasure: objectstore_tool td/osd-scrub-repair 0 EOBJ2 set-bytes td/osd-scrub-repair/CORRUPT 2024-09-15T21:23:33.964 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2024-09-15T21:23:33.964 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2024-09-15T21:23:33.964 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=0 2024-09-15T21:23:33.964 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2024-09-15T21:23:33.964 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 EOBJ2 set-bytes td/osd-scrub-repair/CORRUPT 2024-09-15T21:23:33.964 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-09-15T21:23:33.965 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2024-09-15T21:23:33.965 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=0 2024-09-15T21:23:33.965 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2024-09-15T21:23:33.965 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2024-09-15T21:23:33.965 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-15T21:23:33.965 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-15T21:23:33.965 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-15T21:23:33.965 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-15T21:23:33.965 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-15T21:23:34.270 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-15T21:23:34.270 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 EOBJ2 set-bytes td/osd-scrub-repair/CORRUPT 2024-09-15T21:23:34.270 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-15T21:23:34.270 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-15T21:23:34.270 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=0 2024-09-15T21:23:34.271 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-15T21:23:34.271 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-09-15T21:23:34.271 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 EOBJ2 set-bytes td/osd-scrub-repair/CORRUPT 2024-09-15T21:23:36.696 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 2024-09-15T21:23:36.696 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2024-09-15T21:23:36.696 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-15T21:23:36.696 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=0 2024-09-15T21:23:36.696 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-15T21:23:36.696 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-09-15T21:23:36.696 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=ce4892c7-6c6d-4b4f-8e6f-07a3e11133e5 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-15T21:23:36.696 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T21:23:36.697 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T21:23:36.697 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T21:23:36.697 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-09-15T21:23:36.697 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-09-15T21:23:36.697 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-15T21:23:36.697 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-15T21:23:36.697 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-15T21:23:36.697 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-15T21:23:36.697 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-15T21:23:36.697 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-15T21:23:36.697 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-15T21:23:36.697 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T21:23:36.697 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52602 2024-09-15T21:23:36.697 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.52602/$cluster-$name.asok' 2024-09-15T21:23:36.698 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.52602/$cluster-$name.asok' 2024-09-15T21:23:36.698 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-15T21:23:36.698 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-15T21:23:36.698 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-15T21:23:36.698 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T21:23:36.698 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T21:23:36.698 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T21:23:36.698 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T21:23:36.698 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-15T21:23:36.700 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-15T21:23:36.700 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-09-15T21:23:36.701 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.0 2024-09-15T21:23:36.701 INFO:tasks.workunit.client.0.smithi136.stderr:start osd.0 2024-09-15T21:23:36.701 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 0 --fsid=ce4892c7-6c6d-4b4f-8e6f-07a3e11133e5 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.52602/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T21:23:36.701 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-09-15T21:23:36.701 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 0 = 0 ']' 2024-09-15T21:23:36.701 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-15T21:23:36.701 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-15T21:23:36.701 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-15T21:23:36.728 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:23:36.724+0000 7f6a267d9640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:23:36.730 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:23:36.727+0000 7f6a267d9640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:23:36.731 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:23:36.728+0000 7f6a267d9640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:23:36.992 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 0 2024-09-15T21:23:36.992 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-15T21:23:36.992 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2024-09-15T21:23:36.992 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-15T21:23:36.992 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-15T21:23:36.992 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:23:36.992 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-15T21:23:36.992 INFO:tasks.workunit.client.0.smithi136.stderr:0 2024-09-15T21:23:36.993 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:23:36.993 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:23:37.269 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:23:37.859 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:23:37.856+0000 7f6a267d9640 -1 Falling back to public interface 2024-09-15T21:23:38.270 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:23:38.270 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:23:38.270 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-15T21:23:38.271 INFO:tasks.workunit.client.0.smithi136.stderr:1 2024-09-15T21:23:38.271 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:23:38.271 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:23:38.557 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:23:39.558 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:23:39.558 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:23:39.559 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-15T21:23:39.559 INFO:tasks.workunit.client.0.smithi136.stderr:2 2024-09-15T21:23:39.559 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:23:39.559 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:23:39.841 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:23:40.749 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:23:40.746+0000 7f6a267d9640 -1 osd.0 42 log_to_monitors true 2024-09-15T21:23:40.843 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:23:40.843 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:23:40.843 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-15T21:23:40.843 INFO:tasks.workunit.client.0.smithi136.stderr:3 2024-09-15T21:23:40.843 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:23:40.843 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:23:41.178 INFO:tasks.workunit.client.0.smithi136.stderr:osd.0 up in weight 1 up_from 47 up_thru 38 down_at 43 last_clean_interval [5,42) [v2:127.0.0.1:6802/823180816,v1:127.0.0.1:6803/823180816] [v2:127.0.0.1:6804/823180816,v1:127.0.0.1:6805/823180816] exists,up 980ffc84-a9d7-4f90-928b-2a70e4c90857 2024-09-15T21:23:41.178 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-15T21:23:41.179 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-15T21:23:41.179 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-15T21:23:41.179 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2024-09-15T21:23:41.179 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-15T21:23:41.179 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-15T21:23:41.179 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-15T21:23:41.179 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-15T21:23:41.179 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-15T21:23:41.179 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-15T21:23:41.179 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-15T21:23:41.179 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-15T21:23:41.179 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-15T21:23:41.271 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-15T21:23:41.271 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-15T21:23:41.272 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-15T21:23:41.272 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-15T21:23:41.272 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-15T21:23:41.272 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-15T21:23:41.569 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-15T21:23:41.569 INFO:tasks.workunit.client.0.smithi136.stderr:1 2024-09-15T21:23:41.569 INFO:tasks.workunit.client.0.smithi136.stderr:2' 2024-09-15T21:23:41.569 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-15T21:23:41.569 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T21:23:41.569 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-15T21:23:41.742 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=201863462914 2024-09-15T21:23:41.742 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 201863462914 2024-09-15T21:23:41.742 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-201863462914' 2024-09-15T21:23:41.742 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T21:23:41.742 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-15T21:23:42.002 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=163208757253 2024-09-15T21:23:42.002 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 163208757253 2024-09-15T21:23:42.003 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-201863462914 1-163208757253' 2024-09-15T21:23:42.003 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T21:23:42.003 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-15T21:23:42.170 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=64424509457 2024-09-15T21:23:42.170 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 64424509457 2024-09-15T21:23:42.170 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-201863462914 1-163208757253 2-64424509457' 2024-09-15T21:23:42.170 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T21:23:42.170 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-201863462914 2024-09-15T21:23:42.170 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T21:23:42.171 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-15T21:23:42.171 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-201863462914 2024-09-15T21:23:42.171 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T21:23:42.172 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=201863462914 2024-09-15T21:23:42.172 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 201863462914' 2024-09-15T21:23:42.173 INFO:tasks.workunit.client.0.smithi136.stderr:waiting osd.0 seq 201863462914 2024-09-15T21:23:42.173 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T21:23:42.683 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836513 -lt 201863462914 2024-09-15T21:23:42.684 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-15T21:23:43.685 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-15T21:23:43.685 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T21:23:43.973 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 201863462914 -lt 201863462914 2024-09-15T21:23:43.974 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T21:23:43.974 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-163208757253 2024-09-15T21:23:43.974 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T21:23:43.974 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-15T21:23:43.975 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-163208757253 2024-09-15T21:23:43.975 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T21:23:43.976 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=163208757253 2024-09-15T21:23:43.976 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 163208757253' 2024-09-15T21:23:43.976 INFO:tasks.workunit.client.0.smithi136.stderr:waiting osd.1 seq 163208757253 2024-09-15T21:23:43.977 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-15T21:23:44.256 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 163208757254 -lt 163208757253 2024-09-15T21:23:44.256 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T21:23:44.256 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-64424509457 2024-09-15T21:23:44.257 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T21:23:44.257 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-15T21:23:44.258 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-64424509457 2024-09-15T21:23:44.258 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T21:23:44.259 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=64424509457 2024-09-15T21:23:44.259 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 64424509457' 2024-09-15T21:23:44.259 INFO:tasks.workunit.client.0.smithi136.stderr:waiting osd.2 seq 64424509457 2024-09-15T21:23:44.260 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-15T21:23:44.534 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 64424509457 -lt 64424509457 2024-09-15T21:23:44.534 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-15T21:23:44.535 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T21:23:44.535 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T21:23:44.880 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 6 == 0 2024-09-15T21:23:44.880 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-15T21:23:44.880 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-15T21:23:44.880 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-15T21:23:44.880 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T21:23:44.880 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T21:23:44.880 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T21:23:44.880 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T21:23:45.170 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=5 2024-09-15T21:23:45.171 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-15T21:23:45.171 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T21:23:45.171 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T21:23:45.518 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 = 6 2024-09-15T21:23:45.518 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 5 '!=' -1 2024-09-15T21:23:45.518 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: loop=0 2024-09-15T21:23:45.518 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1652: wait_for_clean: num_active_clean=5 2024-09-15T21:23:45.518 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-15T21:23:45.518 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 0.1 2024-09-15T21:23:45.619 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-15T21:23:45.619 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-15T21:23:45.619 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-15T21:23:45.619 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-15T21:23:45.619 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T21:23:45.619 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T21:23:45.620 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T21:23:45.620 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T21:23:45.890 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=5 2024-09-15T21:23:45.891 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-15T21:23:45.891 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T21:23:45.891 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T21:23:46.277 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 = 6 2024-09-15T21:23:46.277 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 5 '!=' 5 2024-09-15T21:23:46.278 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: get_is_making_recovery_progress 2024-09-15T21:23:46.278 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2024-09-15T21:23:46.278 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-15T21:23:46.278 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-15T21:23:46.278 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-15T21:23:46.278 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2024-09-15T21:23:46.279 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-15T21:23:46.624 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=null 2024-09-15T21:23:46.624 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test null '!=' null 2024-09-15T21:23:46.624 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1655: wait_for_clean: (( 1 >= 13 )) 2024-09-15T21:23:46.624 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-15T21:23:46.624 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 0.2 2024-09-15T21:23:46.825 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-15T21:23:46.825 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-15T21:23:46.826 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-15T21:23:46.826 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-15T21:23:46.826 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T21:23:46.826 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T21:23:46.826 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T21:23:46.826 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T21:23:47.097 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=6 2024-09-15T21:23:47.098 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-15T21:23:47.098 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T21:23:47.098 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T21:23:47.449 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 = 6 2024-09-15T21:23:47.449 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-15T21:23:47.449 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-15T21:23:47.449 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3612: corrupt_scrub_erasure: for i in $(seq 1 $total_objs) 2024-09-15T21:23:47.449 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3613: corrupt_scrub_erasure: objname=EOBJ3 2024-09-15T21:23:47.450 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3614: corrupt_scrub_erasure: add_something td/osd-scrub-repair ecpool EOBJ3 2024-09-15T21:23:47.450 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-09-15T21:23:47.450 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=ecpool 2024-09-15T21:23:47.450 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=EOBJ3 2024-09-15T21:23:47.450 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-09-15T21:23:47.450 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-09-15T21:23:47.450 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-09-15T21:23:47.819 INFO:tasks.workunit.client.0.smithi136.stderr:noscrub is set 2024-09-15T21:23:47.831 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-09-15T21:23:48.330 INFO:tasks.workunit.client.0.smithi136.stderr:nodeep-scrub is set 2024-09-15T21:23:48.342 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-09-15T21:23:48.343 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-09-15T21:23:48.343 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool ecpool put EOBJ3 td/osd-scrub-repair/ORIGINAL 2024-09-15T21:23:48.617 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3616: corrupt_scrub_erasure: expr 3 % 2 2024-09-15T21:23:48.620 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3616: corrupt_scrub_erasure: local osd=1 2024-09-15T21:23:48.620 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3618: corrupt_scrub_erasure: case $i in 2024-09-15T21:23:48.620 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3634: corrupt_scrub_erasure: objectstore_tool td/osd-scrub-repair 1 EOBJ3 remove 2024-09-15T21:23:48.620 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2024-09-15T21:23:48.620 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2024-09-15T21:23:48.620 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=1 2024-09-15T21:23:48.621 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2024-09-15T21:23:48.621 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 EOBJ3 remove 2024-09-15T21:23:48.621 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-09-15T21:23:48.621 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2024-09-15T21:23:48.621 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=1 2024-09-15T21:23:48.621 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2024-09-15T21:23:48.621 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-09-15T21:23:48.621 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-15T21:23:48.621 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-15T21:23:48.621 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-15T21:23:48.621 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-15T21:23:48.621 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-15T21:23:48.926 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-15T21:23:48.927 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 EOBJ3 remove 2024-09-15T21:23:48.927 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-15T21:23:48.927 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-15T21:23:48.927 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=1 2024-09-15T21:23:48.927 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-15T21:23:48.927 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-09-15T21:23:48.927 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 EOBJ3 remove 2024-09-15T21:23:50.709 INFO:tasks.workunit.client.0.smithi136.stdout:remove 0#3:b197b25d:::EOBJ3:head# 2024-09-15T21:23:51.227 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2024-09-15T21:23:51.227 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2024-09-15T21:23:51.227 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-15T21:23:51.227 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=1 2024-09-15T21:23:51.227 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-15T21:23:51.228 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-09-15T21:23:51.228 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=ce4892c7-6c6d-4b4f-8e6f-07a3e11133e5 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-15T21:23:51.228 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T21:23:51.228 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T21:23:51.228 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T21:23:51.228 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-09-15T21:23:51.228 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-09-15T21:23:51.228 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-15T21:23:51.228 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-15T21:23:51.228 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-15T21:23:51.228 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-15T21:23:51.228 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-15T21:23:51.228 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-15T21:23:51.228 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-15T21:23:51.229 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T21:23:51.229 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52602 2024-09-15T21:23:51.229 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.52602/$cluster-$name.asok' 2024-09-15T21:23:51.229 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.52602/$cluster-$name.asok' 2024-09-15T21:23:51.229 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-15T21:23:51.229 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-15T21:23:51.229 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-15T21:23:51.229 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T21:23:51.229 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T21:23:51.229 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T21:23:51.229 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T21:23:51.229 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-15T21:23:51.229 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-15T21:23:51.229 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-09-15T21:23:51.230 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.1 2024-09-15T21:23:51.230 INFO:tasks.workunit.client.0.smithi136.stderr:start osd.1 2024-09-15T21:23:51.231 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 1 --fsid=ce4892c7-6c6d-4b4f-8e6f-07a3e11133e5 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.52602/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T21:23:51.231 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-09-15T21:23:51.231 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 1 = 1 ']' 2024-09-15T21:23:51.231 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-15T21:23:51.231 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-15T21:23:51.231 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-15T21:23:51.259 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:23:51.256+0000 7f6ab9e1c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:23:51.262 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:23:51.259+0000 7f6ab9e1c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:23:51.263 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:23:51.260+0000 7f6ab9e1c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:23:51.514 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 1 2024-09-15T21:23:51.514 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-15T21:23:51.514 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2024-09-15T21:23:51.514 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-15T21:23:51.514 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-15T21:23:51.514 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:23:51.514 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-15T21:23:51.514 INFO:tasks.workunit.client.0.smithi136.stderr:0 2024-09-15T21:23:51.514 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:23:51.514 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T21:23:51.791 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:23:52.615 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:23:52.612+0000 7f6ab9e1c640 -1 Falling back to public interface 2024-09-15T21:23:52.793 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:23:52.793 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:23:52.793 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-15T21:23:52.793 INFO:tasks.workunit.client.0.smithi136.stderr:1 2024-09-15T21:23:52.793 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:23:52.793 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T21:23:53.074 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:23:54.075 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:23:54.075 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:23:54.075 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-15T21:23:54.075 INFO:tasks.workunit.client.0.smithi136.stderr:2 2024-09-15T21:23:54.076 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:23:54.076 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T21:23:54.367 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:23:55.322 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:23:55.318+0000 7f6ab9e1c640 -1 osd.1 51 log_to_monitors true 2024-09-15T21:23:55.369 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:23:55.369 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:23:55.369 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-15T21:23:55.370 INFO:tasks.workunit.client.0.smithi136.stderr:3 2024-09-15T21:23:55.370 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:23:55.370 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T21:23:55.719 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:23:56.721 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:23:56.721 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:23:56.721 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-15T21:23:56.722 INFO:tasks.workunit.client.0.smithi136.stderr:4 2024-09-15T21:23:56.722 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:23:56.722 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T21:23:57.115 INFO:tasks.workunit.client.0.smithi136.stderr:osd.1 up in weight 1 up_from 56 up_thru 56 down_at 52 last_clean_interval [38,51) [v2:127.0.0.1:6810/1701807245,v1:127.0.0.1:6811/1701807245] [v2:127.0.0.1:6812/1701807245,v1:127.0.0.1:6813/1701807245] exists,up ea0d46ed-c598-4c0c-a177-f572f3259311 2024-09-15T21:23:57.116 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-15T21:23:57.116 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-15T21:23:57.116 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-15T21:23:57.116 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2024-09-15T21:23:57.116 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-15T21:23:57.116 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-15T21:23:57.116 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-15T21:23:57.116 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-15T21:23:57.116 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-15T21:23:57.116 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-15T21:23:57.117 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-15T21:23:57.117 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-15T21:23:57.117 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-15T21:23:57.208 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-15T21:23:57.208 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-15T21:23:57.209 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-15T21:23:57.209 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-15T21:23:57.209 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-15T21:23:57.209 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-15T21:23:57.622 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-15T21:23:57.622 INFO:tasks.workunit.client.0.smithi136.stderr:1 2024-09-15T21:23:57.622 INFO:tasks.workunit.client.0.smithi136.stderr:2' 2024-09-15T21:23:57.622 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-15T21:23:57.622 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T21:23:57.623 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-15T21:23:57.737 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=201863462918 2024-09-15T21:23:57.738 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 201863462918 2024-09-15T21:23:57.738 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-201863462918' 2024-09-15T21:23:57.738 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T21:23:57.738 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-15T21:23:57.840 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=240518168578 2024-09-15T21:23:57.841 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 240518168578 2024-09-15T21:23:57.841 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-201863462918 1-240518168578' 2024-09-15T21:23:57.841 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T21:23:57.841 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-15T21:23:57.944 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=64424509461 2024-09-15T21:23:57.945 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 64424509461 2024-09-15T21:23:57.945 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-201863462918 1-240518168578 2-64424509461' 2024-09-15T21:23:57.945 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T21:23:57.945 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-201863462918 2024-09-15T21:23:57.945 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T21:23:57.946 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-15T21:23:57.947 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-201863462918 2024-09-15T21:23:57.947 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T21:23:57.948 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=201863462918 2024-09-15T21:23:57.948 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 201863462918' 2024-09-15T21:23:57.948 INFO:tasks.workunit.client.0.smithi136.stderr:waiting osd.0 seq 201863462918 2024-09-15T21:23:57.948 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T21:23:58.225 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 201863462917 -lt 201863462918 2024-09-15T21:23:58.225 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-15T21:23:59.226 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-15T21:23:59.227 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T21:23:59.512 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 201863462918 -lt 201863462918 2024-09-15T21:23:59.512 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T21:23:59.513 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-240518168578 2024-09-15T21:23:59.513 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T21:23:59.513 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-15T21:23:59.514 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-240518168578 2024-09-15T21:23:59.514 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T21:23:59.515 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=240518168578 2024-09-15T21:23:59.516 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 240518168578' 2024-09-15T21:23:59.516 INFO:tasks.workunit.client.0.smithi136.stderr:waiting osd.1 seq 240518168578 2024-09-15T21:23:59.516 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-15T21:23:59.792 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 240518168578 -lt 240518168578 2024-09-15T21:23:59.792 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T21:23:59.793 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-64424509461 2024-09-15T21:23:59.793 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T21:23:59.794 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-15T21:23:59.794 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-64424509461 2024-09-15T21:23:59.794 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T21:23:59.795 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=64424509461 2024-09-15T21:23:59.795 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 64424509461' 2024-09-15T21:23:59.795 INFO:tasks.workunit.client.0.smithi136.stderr:waiting osd.2 seq 64424509461 2024-09-15T21:23:59.796 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-15T21:24:00.070 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 64424509462 -lt 64424509461 2024-09-15T21:24:00.071 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-15T21:24:00.071 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T21:24:00.071 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T21:24:00.417 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 6 == 0 2024-09-15T21:24:00.417 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-15T21:24:00.418 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-15T21:24:00.418 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-15T21:24:00.418 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T21:24:00.418 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T21:24:00.418 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T21:24:00.418 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T21:24:00.690 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=6 2024-09-15T21:24:00.690 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-15T21:24:00.690 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T21:24:00.690 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T21:24:01.071 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 = 6 2024-09-15T21:24:01.071 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-15T21:24:01.071 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-15T21:24:01.071 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3612: corrupt_scrub_erasure: for i in $(seq 1 $total_objs) 2024-09-15T21:24:01.072 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3613: corrupt_scrub_erasure: objname=EOBJ4 2024-09-15T21:24:01.072 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3614: corrupt_scrub_erasure: add_something td/osd-scrub-repair ecpool EOBJ4 2024-09-15T21:24:01.072 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-09-15T21:24:01.072 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=ecpool 2024-09-15T21:24:01.072 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=EOBJ4 2024-09-15T21:24:01.072 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-09-15T21:24:01.073 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-09-15T21:24:01.073 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-09-15T21:24:01.439 INFO:tasks.workunit.client.0.smithi136.stderr:noscrub is set 2024-09-15T21:24:01.451 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-09-15T21:24:01.950 INFO:tasks.workunit.client.0.smithi136.stderr:nodeep-scrub is set 2024-09-15T21:24:01.962 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-09-15T21:24:01.962 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-09-15T21:24:01.962 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool ecpool put EOBJ4 td/osd-scrub-repair/ORIGINAL 2024-09-15T21:24:02.287 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3616: corrupt_scrub_erasure: expr 4 % 2 2024-09-15T21:24:02.288 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3616: corrupt_scrub_erasure: local osd=0 2024-09-15T21:24:02.289 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3618: corrupt_scrub_erasure: case $i in 2024-09-15T21:24:02.289 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3638: corrupt_scrub_erasure: rados --pool ecpool setxattr EOBJ4 key1-EOBJ4 val1-EOBJ4 2024-09-15T21:24:02.455 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3639: corrupt_scrub_erasure: rados --pool ecpool setxattr EOBJ4 key2-EOBJ4 val2-EOBJ4 2024-09-15T21:24:02.623 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3642: corrupt_scrub_erasure: echo -n bad-val 2024-09-15T21:24:02.623 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3643: corrupt_scrub_erasure: objectstore_tool td/osd-scrub-repair 0 EOBJ4 set-attr _key1-EOBJ4 td/osd-scrub-repair/bad-val 2024-09-15T21:24:02.624 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2024-09-15T21:24:02.624 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2024-09-15T21:24:02.624 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=0 2024-09-15T21:24:02.624 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2024-09-15T21:24:02.624 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 EOBJ4 set-attr _key1-EOBJ4 td/osd-scrub-repair/bad-val 2024-09-15T21:24:02.624 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-09-15T21:24:02.624 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2024-09-15T21:24:02.624 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=0 2024-09-15T21:24:02.624 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2024-09-15T21:24:02.624 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2024-09-15T21:24:02.624 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-15T21:24:02.624 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-15T21:24:02.624 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-15T21:24:02.625 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-15T21:24:02.625 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-15T21:24:02.930 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-15T21:24:02.930 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 EOBJ4 set-attr _key1-EOBJ4 td/osd-scrub-repair/bad-val 2024-09-15T21:24:02.930 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-15T21:24:02.930 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-15T21:24:02.930 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=0 2024-09-15T21:24:02.930 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-15T21:24:02.930 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-09-15T21:24:02.930 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 EOBJ4 set-attr _key1-EOBJ4 td/osd-scrub-repair/bad-val 2024-09-15T21:24:05.174 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 2024-09-15T21:24:05.174 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2024-09-15T21:24:05.174 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-15T21:24:05.174 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=0 2024-09-15T21:24:05.174 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-15T21:24:05.174 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-09-15T21:24:05.174 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=ce4892c7-6c6d-4b4f-8e6f-07a3e11133e5 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-15T21:24:05.174 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T21:24:05.174 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T21:24:05.175 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T21:24:05.175 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-09-15T21:24:05.175 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-09-15T21:24:05.175 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-15T21:24:05.175 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-15T21:24:05.175 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-15T21:24:05.175 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-15T21:24:05.175 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-15T21:24:05.175 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-15T21:24:05.175 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-15T21:24:05.175 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T21:24:05.175 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52602 2024-09-15T21:24:05.175 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.52602/$cluster-$name.asok' 2024-09-15T21:24:05.175 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.52602/$cluster-$name.asok' 2024-09-15T21:24:05.176 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-15T21:24:05.176 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-15T21:24:05.176 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-15T21:24:05.176 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T21:24:05.176 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T21:24:05.176 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T21:24:05.176 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T21:24:05.176 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-15T21:24:05.176 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-15T21:24:05.176 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-09-15T21:24:05.177 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.0 2024-09-15T21:24:05.177 INFO:tasks.workunit.client.0.smithi136.stderr:start osd.0 2024-09-15T21:24:05.177 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 0 --fsid=ce4892c7-6c6d-4b4f-8e6f-07a3e11133e5 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.52602/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T21:24:05.177 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-09-15T21:24:05.177 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 0 = 0 ']' 2024-09-15T21:24:05.178 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-15T21:24:05.178 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-15T21:24:05.178 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-15T21:24:05.206 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:24:05.203+0000 7f7739216640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:24:05.209 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:24:05.206+0000 7f7739216640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:24:05.210 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:24:05.208+0000 7f7739216640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:24:05.461 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 0 2024-09-15T21:24:05.461 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-15T21:24:05.461 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2024-09-15T21:24:05.461 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-15T21:24:05.461 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-15T21:24:05.462 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:24:05.462 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-15T21:24:05.462 INFO:tasks.workunit.client.0.smithi136.stderr:0 2024-09-15T21:24:05.462 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:24:05.462 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:24:05.736 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:24:06.321 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:24:06.318+0000 7f7739216640 -1 Falling back to public interface 2024-09-15T21:24:06.738 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:24:06.738 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:24:06.738 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-15T21:24:06.738 INFO:tasks.workunit.client.0.smithi136.stderr:1 2024-09-15T21:24:06.738 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:24:06.738 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:24:07.043 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:24:08.048 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:24:08.048 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:24:08.048 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-15T21:24:08.048 INFO:tasks.workunit.client.0.smithi136.stderr:2 2024-09-15T21:24:08.048 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:24:08.049 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:24:08.340 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:24:09.053 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:24:09.050+0000 7f7739216640 -1 osd.0 60 log_to_monitors true 2024-09-15T21:24:09.342 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:24:09.342 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:24:09.342 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-15T21:24:09.342 INFO:tasks.workunit.client.0.smithi136.stderr:3 2024-09-15T21:24:09.342 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:24:09.342 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:24:09.842 INFO:tasks.workunit.client.0.smithi136.stderr:osd.0 up in weight 1 up_from 64 up_thru 57 down_at 61 last_clean_interval [47,60) [v2:127.0.0.1:6802/2376832832,v1:127.0.0.1:6803/2376832832] [v2:127.0.0.1:6804/2376832832,v1:127.0.0.1:6805/2376832832] exists,up 980ffc84-a9d7-4f90-928b-2a70e4c90857 2024-09-15T21:24:09.842 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-15T21:24:09.842 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-15T21:24:09.842 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-15T21:24:09.843 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2024-09-15T21:24:09.843 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-15T21:24:09.843 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-15T21:24:09.843 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-15T21:24:09.843 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-15T21:24:09.844 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-15T21:24:09.844 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-15T21:24:09.844 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-15T21:24:09.844 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-15T21:24:09.844 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-15T21:24:09.936 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-15T21:24:09.936 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-15T21:24:09.936 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-15T21:24:09.936 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-15T21:24:09.936 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-15T21:24:09.937 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-15T21:24:10.251 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-15T21:24:10.251 INFO:tasks.workunit.client.0.smithi136.stderr:1 2024-09-15T21:24:10.252 INFO:tasks.workunit.client.0.smithi136.stderr:2' 2024-09-15T21:24:10.252 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-15T21:24:10.252 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T21:24:10.252 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-15T21:24:10.476 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=274877906946 2024-09-15T21:24:10.476 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 274877906946 2024-09-15T21:24:10.476 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-274877906946' 2024-09-15T21:24:10.476 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T21:24:10.476 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-15T21:24:10.653 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=240518168582 2024-09-15T21:24:10.653 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 240518168582 2024-09-15T21:24:10.653 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-274877906946 1-240518168582' 2024-09-15T21:24:10.653 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T21:24:10.653 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-15T21:24:10.756 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=64424509465 2024-09-15T21:24:10.756 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 64424509465 2024-09-15T21:24:10.756 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-274877906946 1-240518168582 2-64424509465' 2024-09-15T21:24:10.756 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T21:24:10.757 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-274877906946 2024-09-15T21:24:10.757 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T21:24:10.757 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-15T21:24:10.758 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-274877906946 2024-09-15T21:24:10.758 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T21:24:10.759 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=274877906946 2024-09-15T21:24:10.759 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 274877906946' 2024-09-15T21:24:10.759 INFO:tasks.workunit.client.0.smithi136.stderr:waiting osd.0 seq 274877906946 2024-09-15T21:24:10.760 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T21:24:11.044 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 274877906946 -lt 274877906946 2024-09-15T21:24:11.044 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T21:24:11.045 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-240518168582 2024-09-15T21:24:11.045 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T21:24:11.046 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-15T21:24:11.046 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-240518168582 2024-09-15T21:24:11.047 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T21:24:11.047 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=240518168582 2024-09-15T21:24:11.048 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 240518168582' 2024-09-15T21:24:11.048 INFO:tasks.workunit.client.0.smithi136.stderr:waiting osd.1 seq 240518168582 2024-09-15T21:24:11.048 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-15T21:24:11.324 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 240518168582 -lt 240518168582 2024-09-15T21:24:11.324 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T21:24:11.328 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-64424509465 2024-09-15T21:24:11.328 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T21:24:11.329 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-15T21:24:11.329 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-64424509465 2024-09-15T21:24:11.329 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T21:24:11.329 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=64424509465 2024-09-15T21:24:11.329 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 64424509465' 2024-09-15T21:24:11.329 INFO:tasks.workunit.client.0.smithi136.stderr:waiting osd.2 seq 64424509465 2024-09-15T21:24:11.329 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-15T21:24:11.603 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 64424509465 -lt 64424509465 2024-09-15T21:24:11.604 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-15T21:24:11.604 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T21:24:11.604 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T21:24:11.949 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 6 == 0 2024-09-15T21:24:11.949 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-15T21:24:11.949 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-15T21:24:11.949 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-15T21:24:11.950 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T21:24:11.950 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T21:24:11.950 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T21:24:11.950 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T21:24:12.217 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=4 2024-09-15T21:24:12.217 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-15T21:24:12.217 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T21:24:12.217 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T21:24:12.560 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 = 6 2024-09-15T21:24:12.560 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 4 '!=' -1 2024-09-15T21:24:12.560 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: loop=0 2024-09-15T21:24:12.561 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1652: wait_for_clean: num_active_clean=4 2024-09-15T21:24:12.561 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-15T21:24:12.561 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 0.1 2024-09-15T21:24:12.662 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-15T21:24:12.662 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-15T21:24:12.662 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-15T21:24:12.662 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-15T21:24:12.663 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T21:24:12.663 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T21:24:12.663 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T21:24:12.663 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T21:24:12.938 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=4 2024-09-15T21:24:12.938 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-15T21:24:12.938 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T21:24:12.938 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T21:24:13.316 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 = 6 2024-09-15T21:24:13.316 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 4 '!=' 4 2024-09-15T21:24:13.316 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: get_is_making_recovery_progress 2024-09-15T21:24:13.316 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2024-09-15T21:24:13.316 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-15T21:24:13.316 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-15T21:24:13.316 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-15T21:24:13.317 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2024-09-15T21:24:13.317 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-15T21:24:13.665 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=null 2024-09-15T21:24:13.665 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test null '!=' null 2024-09-15T21:24:13.665 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1655: wait_for_clean: (( 1 >= 13 )) 2024-09-15T21:24:13.665 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-15T21:24:13.665 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 0.2 2024-09-15T21:24:13.866 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-15T21:24:13.866 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-15T21:24:13.867 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-15T21:24:13.867 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-15T21:24:13.867 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T21:24:13.867 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T21:24:13.867 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T21:24:13.867 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T21:24:14.143 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=4 2024-09-15T21:24:14.143 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-15T21:24:14.144 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T21:24:14.144 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T21:24:14.490 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 = 6 2024-09-15T21:24:14.491 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 4 '!=' 4 2024-09-15T21:24:14.491 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: get_is_making_recovery_progress 2024-09-15T21:24:14.491 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2024-09-15T21:24:14.491 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-15T21:24:14.491 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-15T21:24:14.491 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-15T21:24:14.491 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2024-09-15T21:24:14.491 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-15T21:24:14.838 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=null 2024-09-15T21:24:14.838 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test null '!=' null 2024-09-15T21:24:14.838 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1655: wait_for_clean: (( 2 >= 13 )) 2024-09-15T21:24:14.838 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-15T21:24:14.838 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 0.4 2024-09-15T21:24:15.239 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-15T21:24:15.239 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-15T21:24:15.239 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-15T21:24:15.239 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-15T21:24:15.240 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T21:24:15.240 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T21:24:15.240 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T21:24:15.240 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T21:24:15.510 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=6 2024-09-15T21:24:15.511 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-15T21:24:15.511 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T21:24:15.511 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T21:24:15.856 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 = 6 2024-09-15T21:24:15.856 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-15T21:24:15.857 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-15T21:24:15.857 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3644: corrupt_scrub_erasure: objectstore_tool td/osd-scrub-repair 2 EOBJ4 rm-attr _key2-EOBJ4 2024-09-15T21:24:15.857 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2024-09-15T21:24:15.857 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2024-09-15T21:24:15.857 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=2 2024-09-15T21:24:15.857 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2024-09-15T21:24:15.857 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 2 EOBJ4 rm-attr _key2-EOBJ4 2024-09-15T21:24:15.857 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-09-15T21:24:15.857 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2024-09-15T21:24:15.857 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=2 2024-09-15T21:24:15.857 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2024-09-15T21:24:15.857 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.2 2024-09-15T21:24:15.857 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-15T21:24:15.858 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-15T21:24:15.858 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-15T21:24:15.858 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-15T21:24:15.858 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-15T21:24:16.163 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-15T21:24:16.163 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 2 EOBJ4 rm-attr _key2-EOBJ4 2024-09-15T21:24:16.163 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-15T21:24:16.163 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-15T21:24:16.163 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=2 2024-09-15T21:24:16.163 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-15T21:24:16.163 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/2 2024-09-15T21:24:16.163 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/2 EOBJ4 rm-attr _key2-EOBJ4 2024-09-15T21:24:18.438 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 2 2024-09-15T21:24:18.438 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2024-09-15T21:24:18.439 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-15T21:24:18.439 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=2 2024-09-15T21:24:18.439 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-15T21:24:18.439 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/2 2024-09-15T21:24:18.439 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=ce4892c7-6c6d-4b4f-8e6f-07a3e11133e5 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-15T21:24:18.439 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T21:24:18.439 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T21:24:18.439 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T21:24:18.439 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/2' 2024-09-15T21:24:18.439 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/2/journal' 2024-09-15T21:24:18.439 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-15T21:24:18.439 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-15T21:24:18.439 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-15T21:24:18.440 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-15T21:24:18.440 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-15T21:24:18.440 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-15T21:24:18.440 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-15T21:24:18.440 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T21:24:18.440 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52602 2024-09-15T21:24:18.440 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.52602/$cluster-$name.asok' 2024-09-15T21:24:18.440 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.52602/$cluster-$name.asok' 2024-09-15T21:24:18.440 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-15T21:24:18.440 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-15T21:24:18.440 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-15T21:24:18.440 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T21:24:18.440 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T21:24:18.441 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T21:24:18.441 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T21:24:18.441 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-15T21:24:18.441 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-15T21:24:18.441 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/2 2024-09-15T21:24:18.442 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.2 2024-09-15T21:24:18.442 INFO:tasks.workunit.client.0.smithi136.stderr:start osd.2 2024-09-15T21:24:18.442 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 2 --fsid=ce4892c7-6c6d-4b4f-8e6f-07a3e11133e5 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.52602/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T21:24:18.442 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/2/whoami 2024-09-15T21:24:18.442 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 2 = 2 ']' 2024-09-15T21:24:18.442 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-15T21:24:18.442 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-15T21:24:18.443 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-15T21:24:18.470 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:24:18.466+0000 7f0ee19cc640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:24:18.472 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:24:18.469+0000 7f0ee19cc640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:24:18.473 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:24:18.470+0000 7f0ee19cc640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:24:18.727 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 2 2024-09-15T21:24:18.728 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-15T21:24:18.728 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=2 2024-09-15T21:24:18.728 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-15T21:24:18.728 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-15T21:24:18.728 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:24:18.728 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-15T21:24:18.728 INFO:tasks.workunit.client.0.smithi136.stderr:0 2024-09-15T21:24:18.728 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:24:18.728 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T21:24:19.017 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:24:20.018 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:24:20.018 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:24:20.018 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-15T21:24:20.018 INFO:tasks.workunit.client.0.smithi136.stderr:1 2024-09-15T21:24:20.019 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:24:20.019 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T21:24:20.308 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:24:20.332 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:24:20.329+0000 7f0ee19cc640 -1 Falling back to public interface 2024-09-15T21:24:21.308 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:24:21.309 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:24:21.309 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-15T21:24:21.309 INFO:tasks.workunit.client.0.smithi136.stderr:2 2024-09-15T21:24:21.309 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:24:21.309 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T21:24:21.593 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:24:22.594 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:24:22.594 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:24:22.595 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-15T21:24:22.595 INFO:tasks.workunit.client.0.smithi136.stderr:3 2024-09-15T21:24:22.595 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:24:22.595 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T21:24:22.886 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:24:23.383 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:24:23.380+0000 7f0ee19cc640 -1 osd.2 66 log_to_monitors true 2024-09-15T21:24:23.888 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:24:23.888 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:24:23.888 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-15T21:24:23.888 INFO:tasks.workunit.client.0.smithi136.stderr:4 2024-09-15T21:24:23.888 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:24:23.888 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T21:24:24.476 INFO:tasks.workunit.client.0.smithi136.stderr:osd.2 up in weight 1 up_from 71 up_thru 64 down_at 67 last_clean_interval [15,66) [v2:127.0.0.1:6818/89404126,v1:127.0.0.1:6819/89404126] [v2:127.0.0.1:6820/89404126,v1:127.0.0.1:6821/89404126] exists,up c090ac95-7b74-4c87-8395-5ca55c2b79b2 2024-09-15T21:24:24.476 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-15T21:24:24.476 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-15T21:24:24.476 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-15T21:24:24.476 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2024-09-15T21:24:24.476 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-15T21:24:24.477 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-15T21:24:24.477 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-15T21:24:24.477 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-15T21:24:24.477 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-15T21:24:24.477 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-15T21:24:24.478 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-15T21:24:24.478 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-15T21:24:24.478 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-15T21:24:24.569 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-15T21:24:24.569 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-15T21:24:24.569 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-15T21:24:24.569 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-15T21:24:24.569 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-15T21:24:24.569 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-15T21:24:24.873 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-15T21:24:24.873 INFO:tasks.workunit.client.0.smithi136.stderr:1 2024-09-15T21:24:24.873 INFO:tasks.workunit.client.0.smithi136.stderr:2' 2024-09-15T21:24:24.873 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-15T21:24:24.874 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T21:24:24.874 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-15T21:24:24.977 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=274877906950 2024-09-15T21:24:24.977 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 274877906950 2024-09-15T21:24:24.977 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-274877906950' 2024-09-15T21:24:24.977 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T21:24:24.977 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-15T21:24:25.153 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=240518168585 2024-09-15T21:24:25.153 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 240518168585 2024-09-15T21:24:25.153 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-274877906950 1-240518168585' 2024-09-15T21:24:25.154 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T21:24:25.154 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-15T21:24:25.256 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=304942678018 2024-09-15T21:24:25.256 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 304942678018 2024-09-15T21:24:25.256 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-274877906950 1-240518168585 2-304942678018' 2024-09-15T21:24:25.256 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T21:24:25.256 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-274877906950 2024-09-15T21:24:25.256 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T21:24:25.257 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-15T21:24:25.258 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-274877906950 2024-09-15T21:24:25.258 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T21:24:25.259 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=274877906950 2024-09-15T21:24:25.259 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 274877906950' 2024-09-15T21:24:25.259 INFO:tasks.workunit.client.0.smithi136.stderr:waiting osd.0 seq 274877906950 2024-09-15T21:24:25.259 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T21:24:25.532 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 274877906949 -lt 274877906950 2024-09-15T21:24:25.533 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-15T21:24:26.534 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-15T21:24:26.534 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T21:24:26.847 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 274877906949 -lt 274877906950 2024-09-15T21:24:26.847 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-15T21:24:27.848 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 299 -eq 0 ']' 2024-09-15T21:24:27.848 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T21:24:28.137 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 274877906950 -lt 274877906950 2024-09-15T21:24:28.137 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T21:24:28.137 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-240518168585 2024-09-15T21:24:28.137 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T21:24:28.138 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-15T21:24:28.138 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-240518168585 2024-09-15T21:24:28.140 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T21:24:28.141 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=240518168585 2024-09-15T21:24:28.141 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 240518168585' 2024-09-15T21:24:28.141 INFO:tasks.workunit.client.0.smithi136.stderr:waiting osd.1 seq 240518168585 2024-09-15T21:24:28.141 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-15T21:24:28.418 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 240518168586 -lt 240518168585 2024-09-15T21:24:28.418 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T21:24:28.418 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-304942678018 2024-09-15T21:24:28.418 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T21:24:28.420 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-15T21:24:28.420 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-304942678018 2024-09-15T21:24:28.420 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T21:24:28.421 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=304942678018 2024-09-15T21:24:28.421 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 304942678018' 2024-09-15T21:24:28.421 INFO:tasks.workunit.client.0.smithi136.stderr:waiting osd.2 seq 304942678018 2024-09-15T21:24:28.422 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-15T21:24:28.697 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 304942678018 -lt 304942678018 2024-09-15T21:24:28.698 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-15T21:24:28.698 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T21:24:28.699 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T21:24:29.041 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 6 == 0 2024-09-15T21:24:29.041 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-15T21:24:29.042 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-15T21:24:29.042 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-15T21:24:29.042 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T21:24:29.042 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T21:24:29.042 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T21:24:29.042 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T21:24:29.333 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=6 2024-09-15T21:24:29.333 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-15T21:24:29.333 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T21:24:29.333 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T21:24:29.677 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 = 6 2024-09-15T21:24:29.677 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-15T21:24:29.677 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-15T21:24:29.677 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3645: corrupt_scrub_erasure: echo -n val3-EOBJ4 2024-09-15T21:24:29.677 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3646: corrupt_scrub_erasure: objectstore_tool td/osd-scrub-repair 2 EOBJ4 set-attr _key3-EOBJ4 td/osd-scrub-repair/newval 2024-09-15T21:24:29.677 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2024-09-15T21:24:29.677 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2024-09-15T21:24:29.677 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=2 2024-09-15T21:24:29.677 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2024-09-15T21:24:29.678 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 2 EOBJ4 set-attr _key3-EOBJ4 td/osd-scrub-repair/newval 2024-09-15T21:24:29.678 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-09-15T21:24:29.678 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2024-09-15T21:24:29.678 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=2 2024-09-15T21:24:29.678 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2024-09-15T21:24:29.678 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.2 2024-09-15T21:24:29.678 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-15T21:24:29.678 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-15T21:24:29.678 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-15T21:24:29.678 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-15T21:24:29.678 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-15T21:24:29.983 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-15T21:24:29.983 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 2 EOBJ4 set-attr _key3-EOBJ4 td/osd-scrub-repair/newval 2024-09-15T21:24:29.983 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-15T21:24:29.983 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-15T21:24:29.983 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=2 2024-09-15T21:24:29.983 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-15T21:24:29.984 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/2 2024-09-15T21:24:29.984 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/2 EOBJ4 set-attr _key3-EOBJ4 td/osd-scrub-repair/newval 2024-09-15T21:24:32.411 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 2 2024-09-15T21:24:32.412 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2024-09-15T21:24:32.412 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-15T21:24:32.412 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=2 2024-09-15T21:24:32.412 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-15T21:24:32.412 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/2 2024-09-15T21:24:32.412 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=ce4892c7-6c6d-4b4f-8e6f-07a3e11133e5 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-15T21:24:32.412 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T21:24:32.412 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T21:24:32.412 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T21:24:32.412 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/2' 2024-09-15T21:24:32.413 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/2/journal' 2024-09-15T21:24:32.413 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-15T21:24:32.413 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-15T21:24:32.413 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-15T21:24:32.413 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-15T21:24:32.413 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-15T21:24:32.413 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-15T21:24:32.413 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-15T21:24:32.413 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T21:24:32.413 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52602 2024-09-15T21:24:32.413 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.52602/$cluster-$name.asok' 2024-09-15T21:24:32.413 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.52602/$cluster-$name.asok' 2024-09-15T21:24:32.414 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-15T21:24:32.414 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-15T21:24:32.414 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-15T21:24:32.414 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T21:24:32.414 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T21:24:32.414 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T21:24:32.414 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T21:24:32.414 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-15T21:24:32.414 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-15T21:24:32.414 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/2 2024-09-15T21:24:32.415 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.2 2024-09-15T21:24:32.415 INFO:tasks.workunit.client.0.smithi136.stderr:start osd.2 2024-09-15T21:24:32.415 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 2 --fsid=ce4892c7-6c6d-4b4f-8e6f-07a3e11133e5 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.52602/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T21:24:32.416 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/2/whoami 2024-09-15T21:24:32.416 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 2 = 2 ']' 2024-09-15T21:24:32.416 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-15T21:24:32.416 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-15T21:24:32.416 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-15T21:24:32.444 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:24:32.440+0000 7f23cb358640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:24:32.446 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:24:32.443+0000 7f23cb358640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:24:32.447 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:24:32.444+0000 7f23cb358640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:24:32.698 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 2 2024-09-15T21:24:32.698 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-15T21:24:32.698 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=2 2024-09-15T21:24:32.698 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-15T21:24:32.699 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-15T21:24:32.699 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:24:32.699 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-15T21:24:32.699 INFO:tasks.workunit.client.0.smithi136.stderr:0 2024-09-15T21:24:32.699 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:24:32.699 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T21:24:32.988 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:24:33.990 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:24:33.990 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:24:33.990 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-15T21:24:33.990 INFO:tasks.workunit.client.0.smithi136.stderr:1 2024-09-15T21:24:33.990 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:24:33.990 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T21:24:34.062 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:24:34.059+0000 7f23cb358640 -1 Falling back to public interface 2024-09-15T21:24:34.277 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:24:35.278 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:24:35.278 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:24:35.279 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-15T21:24:35.279 INFO:tasks.workunit.client.0.smithi136.stderr:2 2024-09-15T21:24:35.279 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:24:35.279 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T21:24:35.569 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:24:36.571 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:24:36.571 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:24:36.572 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-15T21:24:36.572 INFO:tasks.workunit.client.0.smithi136.stderr:3 2024-09-15T21:24:36.572 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:24:36.572 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T21:24:36.722 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:24:36.719+0000 7f23cb358640 -1 osd.2 73 log_to_monitors true 2024-09-15T21:24:36.909 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:24:37.911 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:24:37.911 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:24:37.911 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-15T21:24:37.911 INFO:tasks.workunit.client.0.smithi136.stderr:4 2024-09-15T21:24:37.911 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:24:37.912 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T21:24:38.255 INFO:tasks.workunit.client.0.smithi136.stderr:osd.2 up in weight 1 up_from 78 up_thru 78 down_at 74 last_clean_interval [71,73) [v2:127.0.0.1:6818/1849677986,v1:127.0.0.1:6819/1849677986] [v2:127.0.0.1:6820/1849677986,v1:127.0.0.1:6821/1849677986] exists,up c090ac95-7b74-4c87-8395-5ca55c2b79b2 2024-09-15T21:24:38.255 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-15T21:24:38.255 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-15T21:24:38.255 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-15T21:24:38.255 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2024-09-15T21:24:38.255 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-15T21:24:38.255 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-15T21:24:38.256 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-15T21:24:38.256 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-15T21:24:38.256 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-15T21:24:38.256 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-15T21:24:38.256 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-15T21:24:38.256 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-15T21:24:38.256 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-15T21:24:38.348 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-15T21:24:38.348 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-15T21:24:38.348 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-15T21:24:38.348 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-15T21:24:38.348 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-15T21:24:38.348 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-15T21:24:38.705 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-15T21:24:38.705 INFO:tasks.workunit.client.0.smithi136.stderr:1 2024-09-15T21:24:38.706 INFO:tasks.workunit.client.0.smithi136.stderr:2' 2024-09-15T21:24:38.706 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-15T21:24:38.706 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T21:24:38.706 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-15T21:24:38.809 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=274877906953 2024-09-15T21:24:38.809 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 274877906953 2024-09-15T21:24:38.809 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-274877906953' 2024-09-15T21:24:38.809 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T21:24:38.809 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-15T21:24:38.914 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=240518168589 2024-09-15T21:24:38.914 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 240518168589 2024-09-15T21:24:38.914 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-274877906953 1-240518168589' 2024-09-15T21:24:38.914 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T21:24:38.915 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-15T21:24:39.018 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=335007449090 2024-09-15T21:24:39.018 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 335007449090 2024-09-15T21:24:39.019 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-274877906953 1-240518168589 2-335007449090' 2024-09-15T21:24:39.019 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T21:24:39.019 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-274877906953 2024-09-15T21:24:39.019 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T21:24:39.020 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-15T21:24:39.020 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-274877906953 2024-09-15T21:24:39.020 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T21:24:39.021 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=274877906953 2024-09-15T21:24:39.021 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 274877906953' 2024-09-15T21:24:39.022 INFO:tasks.workunit.client.0.smithi136.stderr:waiting osd.0 seq 274877906953 2024-09-15T21:24:39.022 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T21:24:39.299 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 274877906952 -lt 274877906953 2024-09-15T21:24:39.299 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-15T21:24:40.300 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-15T21:24:40.301 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T21:24:40.586 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 274877906952 -lt 274877906953 2024-09-15T21:24:40.586 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-15T21:24:41.587 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 299 -eq 0 ']' 2024-09-15T21:24:41.588 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T21:24:41.871 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 274877906954 -lt 274877906953 2024-09-15T21:24:41.871 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T21:24:41.871 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-240518168589 2024-09-15T21:24:41.871 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T21:24:41.872 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-15T21:24:41.872 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-240518168589 2024-09-15T21:24:41.872 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T21:24:41.873 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=240518168589 2024-09-15T21:24:41.873 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 240518168589' 2024-09-15T21:24:41.873 INFO:tasks.workunit.client.0.smithi136.stderr:waiting osd.1 seq 240518168589 2024-09-15T21:24:41.874 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-15T21:24:42.148 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 240518168590 -lt 240518168589 2024-09-15T21:24:42.148 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T21:24:42.149 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-335007449090 2024-09-15T21:24:42.149 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T21:24:42.150 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-15T21:24:42.150 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-335007449090 2024-09-15T21:24:42.150 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T21:24:42.151 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=335007449090 2024-09-15T21:24:42.151 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 335007449090' 2024-09-15T21:24:42.151 INFO:tasks.workunit.client.0.smithi136.stderr:waiting osd.2 seq 335007449090 2024-09-15T21:24:42.152 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-15T21:24:42.429 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 335007449090 -lt 335007449090 2024-09-15T21:24:42.430 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-15T21:24:42.430 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T21:24:42.430 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T21:24:42.772 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 6 == 0 2024-09-15T21:24:42.773 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-15T21:24:42.773 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-15T21:24:42.774 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-15T21:24:42.774 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T21:24:42.774 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T21:24:42.774 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T21:24:42.774 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T21:24:43.048 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=6 2024-09-15T21:24:43.049 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-15T21:24:43.049 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T21:24:43.049 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T21:24:43.396 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 = 6 2024-09-15T21:24:43.396 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-15T21:24:43.396 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-15T21:24:43.396 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3647: corrupt_scrub_erasure: rm td/osd-scrub-repair/bad-val td/osd-scrub-repair/newval 2024-09-15T21:24:43.397 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3612: corrupt_scrub_erasure: for i in $(seq 1 $total_objs) 2024-09-15T21:24:43.397 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3613: corrupt_scrub_erasure: objname=EOBJ5 2024-09-15T21:24:43.398 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3614: corrupt_scrub_erasure: add_something td/osd-scrub-repair ecpool EOBJ5 2024-09-15T21:24:43.398 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-09-15T21:24:43.398 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=ecpool 2024-09-15T21:24:43.398 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=EOBJ5 2024-09-15T21:24:43.398 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-09-15T21:24:43.398 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-09-15T21:24:43.398 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-09-15T21:24:43.778 INFO:tasks.workunit.client.0.smithi136.stderr:noscrub is set 2024-09-15T21:24:43.790 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-09-15T21:24:44.172 INFO:tasks.workunit.client.0.smithi136.stderr:nodeep-scrub is set 2024-09-15T21:24:44.184 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-09-15T21:24:44.184 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-09-15T21:24:44.184 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool ecpool put EOBJ5 td/osd-scrub-repair/ORIGINAL 2024-09-15T21:24:44.426 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3616: corrupt_scrub_erasure: expr 5 % 2 2024-09-15T21:24:44.427 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3616: corrupt_scrub_erasure: local osd=1 2024-09-15T21:24:44.427 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3618: corrupt_scrub_erasure: case $i in 2024-09-15T21:24:44.427 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3652: corrupt_scrub_erasure: dd if=/dev/urandom of=td/osd-scrub-repair/CORRUPT bs=2048 count=2 2024-09-15T21:24:44.429 INFO:tasks.workunit.client.0.smithi136.stderr:2+0 records in 2024-09-15T21:24:44.429 INFO:tasks.workunit.client.0.smithi136.stderr:2+0 records out 2024-09-15T21:24:44.429 INFO:tasks.workunit.client.0.smithi136.stderr:4096 bytes (4.1 kB, 4.0 KiB) copied, 7.207e-05 s, 56.8 MB/s 2024-09-15T21:24:44.429 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3653: corrupt_scrub_erasure: objectstore_tool td/osd-scrub-repair 1 EOBJ5 set-bytes td/osd-scrub-repair/CORRUPT 2024-09-15T21:24:44.429 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2024-09-15T21:24:44.429 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2024-09-15T21:24:44.429 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=1 2024-09-15T21:24:44.429 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2024-09-15T21:24:44.430 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 EOBJ5 set-bytes td/osd-scrub-repair/CORRUPT 2024-09-15T21:24:44.430 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-09-15T21:24:44.430 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2024-09-15T21:24:44.430 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=1 2024-09-15T21:24:44.430 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2024-09-15T21:24:44.430 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-09-15T21:24:44.430 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-15T21:24:44.430 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-15T21:24:44.430 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-15T21:24:44.431 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-15T21:24:44.431 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-15T21:24:44.764 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-15T21:24:44.764 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 EOBJ5 set-bytes td/osd-scrub-repair/CORRUPT 2024-09-15T21:24:44.764 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-15T21:24:44.764 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-15T21:24:44.764 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=1 2024-09-15T21:24:44.764 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-15T21:24:44.764 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-09-15T21:24:44.764 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 EOBJ5 set-bytes td/osd-scrub-repair/CORRUPT 2024-09-15T21:24:47.049 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2024-09-15T21:24:47.049 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2024-09-15T21:24:47.049 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-15T21:24:47.049 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=1 2024-09-15T21:24:47.049 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-15T21:24:47.049 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-09-15T21:24:47.049 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=ce4892c7-6c6d-4b4f-8e6f-07a3e11133e5 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-15T21:24:47.049 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T21:24:47.049 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T21:24:47.049 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T21:24:47.049 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-09-15T21:24:47.050 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-09-15T21:24:47.050 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-15T21:24:47.050 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-15T21:24:47.050 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-15T21:24:47.050 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-15T21:24:47.050 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-15T21:24:47.050 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-15T21:24:47.050 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-15T21:24:47.050 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T21:24:47.050 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52602 2024-09-15T21:24:47.050 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.52602/$cluster-$name.asok' 2024-09-15T21:24:47.053 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.52602/$cluster-$name.asok' 2024-09-15T21:24:47.053 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-15T21:24:47.053 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-15T21:24:47.053 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-15T21:24:47.053 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T21:24:47.053 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T21:24:47.053 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T21:24:47.053 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T21:24:47.053 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-15T21:24:47.053 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-15T21:24:47.053 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-09-15T21:24:47.054 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.1 2024-09-15T21:24:47.054 INFO:tasks.workunit.client.0.smithi136.stderr:start osd.1 2024-09-15T21:24:47.054 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 1 --fsid=ce4892c7-6c6d-4b4f-8e6f-07a3e11133e5 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.52602/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T21:24:47.054 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-09-15T21:24:47.054 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 1 = 1 ']' 2024-09-15T21:24:47.054 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-15T21:24:47.054 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-15T21:24:47.054 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-15T21:24:47.081 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:24:47.078+0000 7f63fe60f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:24:47.083 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:24:47.080+0000 7f63fe60f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:24:47.084 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:24:47.081+0000 7f63fe60f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:24:47.340 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 1 2024-09-15T21:24:47.340 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-15T21:24:47.340 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2024-09-15T21:24:47.340 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-15T21:24:47.340 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-15T21:24:47.340 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:24:47.340 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-15T21:24:47.340 INFO:tasks.workunit.client.0.smithi136.stderr:0 2024-09-15T21:24:47.340 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:24:47.340 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T21:24:47.617 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:24:48.210 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:24:48.207+0000 7f63fe60f640 -1 Falling back to public interface 2024-09-15T21:24:48.619 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:24:48.619 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:24:48.619 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-15T21:24:48.619 INFO:tasks.workunit.client.0.smithi136.stderr:1 2024-09-15T21:24:48.619 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:24:48.619 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T21:24:48.903 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:24:49.905 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:24:49.905 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:24:49.905 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-15T21:24:49.905 INFO:tasks.workunit.client.0.smithi136.stderr:2 2024-09-15T21:24:49.905 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:24:49.905 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T21:24:50.205 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:24:50.931 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:24:50.929+0000 7f63fe60f640 -1 osd.1 82 log_to_monitors true 2024-09-15T21:24:51.207 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:24:51.207 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:24:51.207 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-15T21:24:51.207 INFO:tasks.workunit.client.0.smithi136.stderr:3 2024-09-15T21:24:51.207 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:24:51.207 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T21:24:51.553 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:24:52.555 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:24:52.555 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:24:52.555 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-15T21:24:52.555 INFO:tasks.workunit.client.0.smithi136.stderr:4 2024-09-15T21:24:52.555 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:24:52.556 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T21:24:52.895 INFO:tasks.workunit.client.0.smithi136.stderr:osd.1 up in weight 1 up_from 87 up_thru 87 down_at 83 last_clean_interval [56,82) [v2:127.0.0.1:6810/17759539,v1:127.0.0.1:6811/17759539] [v2:127.0.0.1:6812/17759539,v1:127.0.0.1:6813/17759539] exists,up ea0d46ed-c598-4c0c-a177-f572f3259311 2024-09-15T21:24:52.896 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-15T21:24:52.896 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-15T21:24:52.896 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-15T21:24:52.896 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2024-09-15T21:24:52.896 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-15T21:24:52.896 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-15T21:24:52.896 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-15T21:24:52.896 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-15T21:24:52.896 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-15T21:24:52.896 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-15T21:24:52.896 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-15T21:24:52.897 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-15T21:24:52.897 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-15T21:24:52.987 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-15T21:24:52.987 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-15T21:24:52.987 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-15T21:24:52.988 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-15T21:24:52.988 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-15T21:24:52.988 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-15T21:24:53.269 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-15T21:24:53.269 INFO:tasks.workunit.client.0.smithi136.stderr:1 2024-09-15T21:24:53.269 INFO:tasks.workunit.client.0.smithi136.stderr:2' 2024-09-15T21:24:53.269 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-15T21:24:53.269 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T21:24:53.269 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-15T21:24:53.420 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=274877906957 2024-09-15T21:24:53.420 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 274877906957 2024-09-15T21:24:53.420 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-274877906957' 2024-09-15T21:24:53.420 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T21:24:53.420 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-15T21:24:53.522 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=373662154754 2024-09-15T21:24:53.523 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 373662154754 2024-09-15T21:24:53.523 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-274877906957 1-373662154754' 2024-09-15T21:24:53.523 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T21:24:53.523 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-15T21:24:53.625 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=335007449094 2024-09-15T21:24:53.625 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 335007449094 2024-09-15T21:24:53.626 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-274877906957 1-373662154754 2-335007449094' 2024-09-15T21:24:53.626 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T21:24:53.626 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-274877906957 2024-09-15T21:24:53.626 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T21:24:53.626 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-15T21:24:53.627 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-274877906957 2024-09-15T21:24:53.627 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T21:24:53.628 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=274877906957 2024-09-15T21:24:53.628 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 274877906957' 2024-09-15T21:24:53.628 INFO:tasks.workunit.client.0.smithi136.stderr:waiting osd.0 seq 274877906957 2024-09-15T21:24:53.628 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T21:24:53.902 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 274877906956 -lt 274877906957 2024-09-15T21:24:53.902 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-15T21:24:54.903 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-15T21:24:54.903 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T21:24:55.188 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 274877906958 -lt 274877906957 2024-09-15T21:24:55.188 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T21:24:55.188 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-373662154754 2024-09-15T21:24:55.189 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T21:24:55.189 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-15T21:24:55.190 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-373662154754 2024-09-15T21:24:55.190 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T21:24:55.191 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=373662154754 2024-09-15T21:24:55.191 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 373662154754' 2024-09-15T21:24:55.191 INFO:tasks.workunit.client.0.smithi136.stderr:waiting osd.1 seq 373662154754 2024-09-15T21:24:55.191 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-15T21:24:55.465 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 373662154754 -lt 373662154754 2024-09-15T21:24:55.466 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T21:24:55.466 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-335007449094 2024-09-15T21:24:55.466 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T21:24:55.467 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-15T21:24:55.468 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-335007449094 2024-09-15T21:24:55.468 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T21:24:55.469 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=335007449094 2024-09-15T21:24:55.469 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 335007449094' 2024-09-15T21:24:55.469 INFO:tasks.workunit.client.0.smithi136.stderr:waiting osd.2 seq 335007449094 2024-09-15T21:24:55.469 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-15T21:24:55.744 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 335007449094 -lt 335007449094 2024-09-15T21:24:55.745 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-15T21:24:55.745 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T21:24:55.745 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T21:24:56.089 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 6 == 0 2024-09-15T21:24:56.089 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-15T21:24:56.089 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-15T21:24:56.089 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-15T21:24:56.089 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T21:24:56.089 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T21:24:56.089 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T21:24:56.089 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T21:24:56.361 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=6 2024-09-15T21:24:56.362 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-15T21:24:56.362 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T21:24:56.362 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T21:24:56.706 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 = 6 2024-09-15T21:24:56.706 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-15T21:24:56.706 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-15T21:24:56.706 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3612: corrupt_scrub_erasure: for i in $(seq 1 $total_objs) 2024-09-15T21:24:56.706 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3613: corrupt_scrub_erasure: objname=EOBJ6 2024-09-15T21:24:56.706 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3614: corrupt_scrub_erasure: add_something td/osd-scrub-repair ecpool EOBJ6 2024-09-15T21:24:56.706 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-09-15T21:24:56.706 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=ecpool 2024-09-15T21:24:56.707 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=EOBJ6 2024-09-15T21:24:56.707 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-09-15T21:24:56.707 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-09-15T21:24:56.707 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-09-15T21:24:57.182 INFO:tasks.workunit.client.0.smithi136.stderr:noscrub is set 2024-09-15T21:24:57.193 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-09-15T21:24:57.582 INFO:tasks.workunit.client.0.smithi136.stderr:nodeep-scrub is set 2024-09-15T21:24:57.593 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-09-15T21:24:57.593 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-09-15T21:24:57.594 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool ecpool put EOBJ6 td/osd-scrub-repair/ORIGINAL 2024-09-15T21:24:57.811 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3616: corrupt_scrub_erasure: expr 6 % 2 2024-09-15T21:24:57.813 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3616: corrupt_scrub_erasure: local osd=0 2024-09-15T21:24:57.813 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3618: corrupt_scrub_erasure: case $i in 2024-09-15T21:24:57.813 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3657: corrupt_scrub_erasure: objectstore_tool td/osd-scrub-repair 0 EOBJ6 rm-attr hinfo_key 2024-09-15T21:24:57.813 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2024-09-15T21:24:57.814 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2024-09-15T21:24:57.814 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=0 2024-09-15T21:24:57.814 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2024-09-15T21:24:57.814 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 EOBJ6 rm-attr hinfo_key 2024-09-15T21:24:57.814 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-09-15T21:24:57.814 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2024-09-15T21:24:57.814 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=0 2024-09-15T21:24:57.814 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2024-09-15T21:24:57.814 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2024-09-15T21:24:57.814 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-15T21:24:57.814 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-15T21:24:57.814 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-15T21:24:57.814 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-15T21:24:57.815 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-15T21:24:58.119 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-15T21:24:58.120 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 EOBJ6 rm-attr hinfo_key 2024-09-15T21:24:58.120 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-15T21:24:58.120 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-15T21:24:58.120 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=0 2024-09-15T21:24:58.120 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-15T21:24:58.120 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-09-15T21:24:58.120 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 EOBJ6 rm-attr hinfo_key 2024-09-15T21:25:00.604 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 2024-09-15T21:25:00.605 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2024-09-15T21:25:00.605 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-15T21:25:00.605 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=0 2024-09-15T21:25:00.605 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-15T21:25:00.605 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-09-15T21:25:00.605 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=ce4892c7-6c6d-4b4f-8e6f-07a3e11133e5 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-15T21:25:00.605 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T21:25:00.605 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T21:25:00.605 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T21:25:00.605 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-09-15T21:25:00.605 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-09-15T21:25:00.605 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-15T21:25:00.606 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-15T21:25:00.606 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-15T21:25:00.606 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-15T21:25:00.606 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-15T21:25:00.606 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-15T21:25:00.606 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-15T21:25:00.606 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T21:25:00.606 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52602 2024-09-15T21:25:00.606 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.52602/$cluster-$name.asok' 2024-09-15T21:25:00.606 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.52602/$cluster-$name.asok' 2024-09-15T21:25:00.606 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-15T21:25:00.606 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-15T21:25:00.606 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-15T21:25:00.606 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T21:25:00.607 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T21:25:00.607 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T21:25:00.607 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T21:25:00.607 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-15T21:25:00.607 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-15T21:25:00.607 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-09-15T21:25:00.608 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.0 2024-09-15T21:25:00.608 INFO:tasks.workunit.client.0.smithi136.stderr:start osd.0 2024-09-15T21:25:00.608 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 0 --fsid=ce4892c7-6c6d-4b4f-8e6f-07a3e11133e5 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.52602/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T21:25:00.608 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-09-15T21:25:00.608 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 0 = 0 ']' 2024-09-15T21:25:00.608 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-15T21:25:00.608 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-15T21:25:00.608 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-15T21:25:00.636 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:25:00.632+0000 7f4a30611640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:25:00.638 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:25:00.634+0000 7f4a30611640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:25:00.639 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:25:00.636+0000 7f4a30611640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:25:00.894 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 0 2024-09-15T21:25:00.895 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-15T21:25:00.895 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2024-09-15T21:25:00.895 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-15T21:25:00.895 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-15T21:25:00.895 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:25:00.895 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-15T21:25:00.895 INFO:tasks.workunit.client.0.smithi136.stderr:0 2024-09-15T21:25:00.895 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:25:00.895 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:25:01.177 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:25:01.763 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:25:01.759+0000 7f4a30611640 -1 Falling back to public interface 2024-09-15T21:25:02.179 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:25:02.179 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:25:02.179 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-15T21:25:02.179 INFO:tasks.workunit.client.0.smithi136.stderr:1 2024-09-15T21:25:02.179 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:25:02.179 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:25:02.460 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:25:03.462 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:25:03.462 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:25:03.462 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-15T21:25:03.462 INFO:tasks.workunit.client.0.smithi136.stderr:2 2024-09-15T21:25:03.462 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:25:03.462 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:25:03.748 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:25:04.674 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:25:04.671+0000 7f4a30611640 -1 osd.0 91 log_to_monitors true 2024-09-15T21:25:04.750 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:25:04.750 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:25:04.750 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-15T21:25:04.750 INFO:tasks.workunit.client.0.smithi136.stderr:3 2024-09-15T21:25:04.750 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:25:04.750 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:25:05.161 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:25:06.162 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:25:06.162 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:25:06.163 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-15T21:25:06.163 INFO:tasks.workunit.client.0.smithi136.stderr:4 2024-09-15T21:25:06.163 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:25:06.163 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:25:06.581 INFO:tasks.workunit.client.0.smithi136.stderr:osd.0 up in weight 1 up_from 96 up_thru 96 down_at 92 last_clean_interval [64,91) [v2:127.0.0.1:6802/2246448553,v1:127.0.0.1:6803/2246448553] [v2:127.0.0.1:6804/2246448553,v1:127.0.0.1:6805/2246448553] exists,up 980ffc84-a9d7-4f90-928b-2a70e4c90857 2024-09-15T21:25:06.581 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-15T21:25:06.582 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-15T21:25:06.582 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-15T21:25:06.582 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2024-09-15T21:25:06.582 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-15T21:25:06.582 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-15T21:25:06.582 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-15T21:25:06.582 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-15T21:25:06.583 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-15T21:25:06.583 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-15T21:25:06.583 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-15T21:25:06.583 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-15T21:25:06.583 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-15T21:25:06.673 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-15T21:25:06.674 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-15T21:25:06.674 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-15T21:25:06.674 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-15T21:25:06.674 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-15T21:25:06.674 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-15T21:25:06.975 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-15T21:25:06.975 INFO:tasks.workunit.client.0.smithi136.stderr:1 2024-09-15T21:25:06.975 INFO:tasks.workunit.client.0.smithi136.stderr:2' 2024-09-15T21:25:06.975 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-15T21:25:06.975 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T21:25:06.975 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-15T21:25:07.099 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=412316860418 2024-09-15T21:25:07.100 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 412316860418 2024-09-15T21:25:07.100 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-412316860418' 2024-09-15T21:25:07.100 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T21:25:07.100 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-15T21:25:07.259 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=373662154758 2024-09-15T21:25:07.260 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 373662154758 2024-09-15T21:25:07.260 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-412316860418 1-373662154758' 2024-09-15T21:25:07.260 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T21:25:07.260 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-15T21:25:07.363 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=335007449098 2024-09-15T21:25:07.363 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 335007449098 2024-09-15T21:25:07.363 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-412316860418 1-373662154758 2-335007449098' 2024-09-15T21:25:07.363 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T21:25:07.364 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-412316860418 2024-09-15T21:25:07.364 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T21:25:07.364 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-15T21:25:07.365 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-412316860418 2024-09-15T21:25:07.365 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T21:25:07.366 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=412316860418 2024-09-15T21:25:07.366 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 412316860418' 2024-09-15T21:25:07.366 INFO:tasks.workunit.client.0.smithi136.stderr:waiting osd.0 seq 412316860418 2024-09-15T21:25:07.366 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T21:25:07.641 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 1 -lt 412316860418 2024-09-15T21:25:07.641 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-15T21:25:08.642 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-15T21:25:08.643 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T21:25:08.935 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 412316860418 -lt 412316860418 2024-09-15T21:25:08.936 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T21:25:08.936 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-373662154758 2024-09-15T21:25:08.936 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T21:25:08.937 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-15T21:25:08.938 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-373662154758 2024-09-15T21:25:08.938 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T21:25:08.939 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=373662154758 2024-09-15T21:25:08.939 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 373662154758' 2024-09-15T21:25:08.939 INFO:tasks.workunit.client.0.smithi136.stderr:waiting osd.1 seq 373662154758 2024-09-15T21:25:08.939 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-15T21:25:09.216 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 373662154758 -lt 373662154758 2024-09-15T21:25:09.216 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T21:25:09.217 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-335007449098 2024-09-15T21:25:09.217 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T21:25:09.218 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-15T21:25:09.218 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-335007449098 2024-09-15T21:25:09.218 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T21:25:09.219 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=335007449098 2024-09-15T21:25:09.219 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 335007449098' 2024-09-15T21:25:09.220 INFO:tasks.workunit.client.0.smithi136.stderr:waiting osd.2 seq 335007449098 2024-09-15T21:25:09.220 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-15T21:25:09.494 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 335007449098 -lt 335007449098 2024-09-15T21:25:09.495 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-15T21:25:09.495 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T21:25:09.495 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T21:25:09.835 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 6 == 0 2024-09-15T21:25:09.835 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-15T21:25:09.835 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-15T21:25:09.835 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-15T21:25:09.835 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T21:25:09.835 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T21:25:09.835 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T21:25:09.835 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T21:25:10.104 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=6 2024-09-15T21:25:10.105 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-15T21:25:10.105 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T21:25:10.105 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T21:25:10.446 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 = 6 2024-09-15T21:25:10.446 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-15T21:25:10.446 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-15T21:25:10.446 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3658: corrupt_scrub_erasure: echo -n bad-val 2024-09-15T21:25:10.446 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3659: corrupt_scrub_erasure: objectstore_tool td/osd-scrub-repair 1 EOBJ6 set-attr hinfo_key td/osd-scrub-repair/bad-val 2024-09-15T21:25:10.446 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2024-09-15T21:25:10.446 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2024-09-15T21:25:10.446 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=1 2024-09-15T21:25:10.446 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2024-09-15T21:25:10.447 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 EOBJ6 set-attr hinfo_key td/osd-scrub-repair/bad-val 2024-09-15T21:25:10.447 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-09-15T21:25:10.447 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2024-09-15T21:25:10.447 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=1 2024-09-15T21:25:10.447 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2024-09-15T21:25:10.447 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-09-15T21:25:10.447 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-15T21:25:10.447 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-15T21:25:10.447 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-15T21:25:10.447 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-15T21:25:10.447 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-15T21:25:10.752 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-15T21:25:10.752 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 EOBJ6 set-attr hinfo_key td/osd-scrub-repair/bad-val 2024-09-15T21:25:10.752 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-15T21:25:10.752 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-15T21:25:10.752 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=1 2024-09-15T21:25:10.752 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-15T21:25:10.753 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-09-15T21:25:10.753 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 EOBJ6 set-attr hinfo_key td/osd-scrub-repair/bad-val 2024-09-15T21:25:12.925 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2024-09-15T21:25:12.925 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2024-09-15T21:25:12.925 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-15T21:25:12.926 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=1 2024-09-15T21:25:12.926 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-15T21:25:12.926 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-09-15T21:25:12.926 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=ce4892c7-6c6d-4b4f-8e6f-07a3e11133e5 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-15T21:25:12.926 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T21:25:12.926 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T21:25:12.926 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T21:25:12.926 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-09-15T21:25:12.926 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-09-15T21:25:12.926 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-15T21:25:12.926 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-15T21:25:12.926 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-15T21:25:12.926 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-15T21:25:12.926 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-15T21:25:12.927 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-15T21:25:12.927 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-15T21:25:12.927 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T21:25:12.927 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52602 2024-09-15T21:25:12.927 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.52602/$cluster-$name.asok' 2024-09-15T21:25:12.927 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.52602/$cluster-$name.asok' 2024-09-15T21:25:12.927 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-15T21:25:12.927 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-15T21:25:12.927 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-15T21:25:12.927 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T21:25:12.927 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T21:25:12.927 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T21:25:12.927 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T21:25:12.927 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-15T21:25:12.927 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-15T21:25:12.928 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-09-15T21:25:12.928 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.1 2024-09-15T21:25:12.929 INFO:tasks.workunit.client.0.smithi136.stderr:start osd.1 2024-09-15T21:25:12.929 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 1 --fsid=ce4892c7-6c6d-4b4f-8e6f-07a3e11133e5 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.52602/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T21:25:12.929 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-09-15T21:25:12.929 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 1 = 1 ']' 2024-09-15T21:25:12.929 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-15T21:25:12.929 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-15T21:25:12.929 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-15T21:25:12.957 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:25:12.954+0000 7ff9bf141640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:25:12.991 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:25:12.988+0000 7ff9bf141640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:25:12.992 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:25:12.989+0000 7ff9bf141640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:25:13.213 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 1 2024-09-15T21:25:13.213 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-15T21:25:13.213 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2024-09-15T21:25:13.213 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-15T21:25:13.213 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-15T21:25:13.213 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:25:13.213 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-15T21:25:13.213 INFO:tasks.workunit.client.0.smithi136.stderr:0 2024-09-15T21:25:13.213 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:25:13.213 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T21:25:13.488 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:25:14.140 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:25:14.138+0000 7ff9bf141640 -1 Falling back to public interface 2024-09-15T21:25:14.490 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:25:14.490 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:25:14.490 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-15T21:25:14.490 INFO:tasks.workunit.client.0.smithi136.stderr:1 2024-09-15T21:25:14.490 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:25:14.490 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T21:25:14.766 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:25:15.768 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:25:15.768 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:25:15.768 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-15T21:25:15.768 INFO:tasks.workunit.client.0.smithi136.stderr:2 2024-09-15T21:25:15.769 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:25:15.769 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T21:25:16.055 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:25:17.057 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:25:17.057 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:25:17.057 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-15T21:25:17.057 INFO:tasks.workunit.client.0.smithi136.stderr:3 2024-09-15T21:25:17.057 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:25:17.057 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T21:25:17.161 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:25:17.159+0000 7ff9bf141640 -1 osd.1 99 log_to_monitors true 2024-09-15T21:25:17.366 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:25:18.368 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:25:18.368 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:25:18.368 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-15T21:25:18.368 INFO:tasks.workunit.client.0.smithi136.stderr:4 2024-09-15T21:25:18.369 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:25:18.369 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T21:25:18.678 INFO:tasks.workunit.client.0.smithi136.stderr:osd.1 up in weight 1 up_from 104 up_thru 104 down_at 100 last_clean_interval [87,99) [v2:127.0.0.1:6810/4094729727,v1:127.0.0.1:6811/4094729727] [v2:127.0.0.1:6812/4094729727,v1:127.0.0.1:6813/4094729727] exists,up ea0d46ed-c598-4c0c-a177-f572f3259311 2024-09-15T21:25:18.678 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-15T21:25:18.678 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-15T21:25:18.678 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-15T21:25:18.678 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2024-09-15T21:25:18.678 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-15T21:25:18.678 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-15T21:25:18.678 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-15T21:25:18.679 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-15T21:25:18.679 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-15T21:25:18.679 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-15T21:25:18.679 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-15T21:25:18.679 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-15T21:25:18.680 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-15T21:25:18.771 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-15T21:25:18.772 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-15T21:25:18.772 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-15T21:25:18.772 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-15T21:25:18.772 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-15T21:25:18.772 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-15T21:25:19.158 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-15T21:25:19.159 INFO:tasks.workunit.client.0.smithi136.stderr:1 2024-09-15T21:25:19.159 INFO:tasks.workunit.client.0.smithi136.stderr:2' 2024-09-15T21:25:19.159 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-15T21:25:19.159 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T21:25:19.159 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-15T21:25:19.276 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=412316860421 2024-09-15T21:25:19.276 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 412316860421 2024-09-15T21:25:19.276 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-412316860421' 2024-09-15T21:25:19.276 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T21:25:19.276 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-15T21:25:19.380 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=446676598786 2024-09-15T21:25:19.380 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 446676598786 2024-09-15T21:25:19.380 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-412316860421 1-446676598786' 2024-09-15T21:25:19.381 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T21:25:19.381 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-15T21:25:19.483 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=335007449101 2024-09-15T21:25:19.483 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 335007449101 2024-09-15T21:25:19.483 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-412316860421 1-446676598786 2-335007449101' 2024-09-15T21:25:19.483 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T21:25:19.484 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-412316860421 2024-09-15T21:25:19.484 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T21:25:19.484 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-15T21:25:19.485 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-412316860421 2024-09-15T21:25:19.485 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T21:25:19.486 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=412316860421 2024-09-15T21:25:19.486 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 412316860421' 2024-09-15T21:25:19.486 INFO:tasks.workunit.client.0.smithi136.stderr:waiting osd.0 seq 412316860421 2024-09-15T21:25:19.487 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T21:25:19.764 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 412316860420 -lt 412316860421 2024-09-15T21:25:19.764 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-15T21:25:20.765 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-15T21:25:20.765 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T21:25:21.058 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 412316860422 -lt 412316860421 2024-09-15T21:25:21.058 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T21:25:21.058 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-446676598786 2024-09-15T21:25:21.058 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T21:25:21.059 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-15T21:25:21.060 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-446676598786 2024-09-15T21:25:21.060 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T21:25:21.061 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=446676598786 2024-09-15T21:25:21.061 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 446676598786' 2024-09-15T21:25:21.061 INFO:tasks.workunit.client.0.smithi136.stderr:waiting osd.1 seq 446676598786 2024-09-15T21:25:21.061 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-15T21:25:21.343 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 446676598786 -lt 446676598786 2024-09-15T21:25:21.343 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T21:25:21.343 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-335007449101 2024-09-15T21:25:21.343 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T21:25:21.344 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-15T21:25:21.345 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-335007449101 2024-09-15T21:25:21.345 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T21:25:21.346 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=335007449101 2024-09-15T21:25:21.346 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 335007449101' 2024-09-15T21:25:21.346 INFO:tasks.workunit.client.0.smithi136.stderr:waiting osd.2 seq 335007449101 2024-09-15T21:25:21.346 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-15T21:25:21.621 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 335007449101 -lt 335007449101 2024-09-15T21:25:21.622 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-15T21:25:21.622 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T21:25:21.622 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T21:25:21.967 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 6 == 0 2024-09-15T21:25:21.967 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-15T21:25:21.968 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-15T21:25:21.968 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-15T21:25:21.968 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T21:25:21.968 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T21:25:21.968 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T21:25:21.968 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T21:25:22.240 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=6 2024-09-15T21:25:22.241 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-15T21:25:22.241 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T21:25:22.241 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T21:25:22.584 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 = 6 2024-09-15T21:25:22.584 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-15T21:25:22.584 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-15T21:25:22.584 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3612: corrupt_scrub_erasure: for i in $(seq 1 $total_objs) 2024-09-15T21:25:22.584 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3613: corrupt_scrub_erasure: objname=EOBJ7 2024-09-15T21:25:22.584 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3614: corrupt_scrub_erasure: add_something td/osd-scrub-repair ecpool EOBJ7 2024-09-15T21:25:22.585 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-09-15T21:25:22.585 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=ecpool 2024-09-15T21:25:22.585 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=EOBJ7 2024-09-15T21:25:22.585 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-09-15T21:25:22.585 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-09-15T21:25:22.585 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-09-15T21:25:23.061 INFO:tasks.workunit.client.0.smithi136.stderr:noscrub is set 2024-09-15T21:25:23.072 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-09-15T21:25:23.453 INFO:tasks.workunit.client.0.smithi136.stderr:nodeep-scrub is set 2024-09-15T21:25:23.465 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-09-15T21:25:23.465 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-09-15T21:25:23.465 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool ecpool put EOBJ7 td/osd-scrub-repair/ORIGINAL 2024-09-15T21:25:23.657 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3616: corrupt_scrub_erasure: expr 7 % 2 2024-09-15T21:25:23.658 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3616: corrupt_scrub_erasure: local osd=1 2024-09-15T21:25:23.658 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3618: corrupt_scrub_erasure: case $i in 2024-09-15T21:25:23.658 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3663: corrupt_scrub_erasure: local payload=MAKETHISDIFFERENTFROMOTHEROBJECTS 2024-09-15T21:25:23.659 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3664: corrupt_scrub_erasure: echo MAKETHISDIFFERENTFROMOTHEROBJECTS 2024-09-15T21:25:23.659 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3665: corrupt_scrub_erasure: rados --pool ecpool put EOBJ7 td/osd-scrub-repair/DIFFERENT 2024-09-15T21:25:23.843 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3668: corrupt_scrub_erasure: objectstore_tool td/osd-scrub-repair 0 EOBJ1 get-attr hinfo_key 2024-09-15T21:25:23.843 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2024-09-15T21:25:23.843 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2024-09-15T21:25:23.843 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=0 2024-09-15T21:25:23.843 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2024-09-15T21:25:23.843 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 EOBJ1 get-attr hinfo_key 2024-09-15T21:25:23.843 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-09-15T21:25:23.843 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2024-09-15T21:25:23.843 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=0 2024-09-15T21:25:23.843 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2024-09-15T21:25:23.843 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2024-09-15T21:25:23.843 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-15T21:25:23.844 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-15T21:25:23.844 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-15T21:25:23.844 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-15T21:25:23.844 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-15T21:25:25.150 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-15T21:25:25.150 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 EOBJ1 get-attr hinfo_key 2024-09-15T21:25:25.151 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-15T21:25:25.151 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-15T21:25:25.151 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=0 2024-09-15T21:25:25.151 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-15T21:25:25.151 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-09-15T21:25:25.151 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 EOBJ1 get-attr hinfo_key 2024-09-15T21:25:26.653 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 2024-09-15T21:25:26.654 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2024-09-15T21:25:26.654 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-15T21:25:26.654 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=0 2024-09-15T21:25:26.654 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-15T21:25:26.654 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-09-15T21:25:26.654 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=ce4892c7-6c6d-4b4f-8e6f-07a3e11133e5 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-15T21:25:26.654 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T21:25:26.654 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T21:25:26.654 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T21:25:26.654 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-09-15T21:25:26.654 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-09-15T21:25:26.654 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-15T21:25:26.655 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-15T21:25:26.655 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-15T21:25:26.655 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-15T21:25:26.655 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-15T21:25:26.655 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-15T21:25:26.655 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-15T21:25:26.655 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T21:25:26.655 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52602 2024-09-15T21:25:26.655 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.52602/$cluster-$name.asok' 2024-09-15T21:25:26.657 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.52602/$cluster-$name.asok' 2024-09-15T21:25:26.657 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-15T21:25:26.657 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-15T21:25:26.657 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-15T21:25:26.657 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T21:25:26.657 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T21:25:26.657 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T21:25:26.657 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T21:25:26.657 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-15T21:25:26.657 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-15T21:25:26.657 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-09-15T21:25:26.657 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.0 2024-09-15T21:25:26.657 INFO:tasks.workunit.client.0.smithi136.stderr:start osd.0 2024-09-15T21:25:26.658 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 0 --fsid=ce4892c7-6c6d-4b4f-8e6f-07a3e11133e5 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.52602/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T21:25:26.658 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-09-15T21:25:26.658 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 0 = 0 ']' 2024-09-15T21:25:26.658 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-15T21:25:26.658 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-15T21:25:26.658 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-15T21:25:26.686 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:25:26.683+0000 7ff2d11ae640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:25:26.689 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:25:26.686+0000 7ff2d11ae640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:25:26.689 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:25:26.687+0000 7ff2d11ae640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:25:26.944 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 0 2024-09-15T21:25:26.944 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-15T21:25:26.944 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2024-09-15T21:25:26.944 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-15T21:25:26.945 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-15T21:25:26.945 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:25:26.945 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-15T21:25:26.945 INFO:tasks.workunit.client.0.smithi136.stderr:0 2024-09-15T21:25:26.945 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:25:26.945 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:25:27.223 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:25:27.812 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:25:27.809+0000 7ff2d11ae640 -1 Falling back to public interface 2024-09-15T21:25:28.224 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:25:28.224 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:25:28.224 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-15T21:25:28.224 INFO:tasks.workunit.client.0.smithi136.stderr:1 2024-09-15T21:25:28.224 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:25:28.225 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:25:28.507 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:25:29.509 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:25:29.509 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:25:29.509 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-15T21:25:29.509 INFO:tasks.workunit.client.0.smithi136.stderr:2 2024-09-15T21:25:29.509 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:25:29.509 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:25:29.812 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:25:30.431 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:25:30.428+0000 7ff2d11ae640 -1 osd.0 108 log_to_monitors true 2024-09-15T21:25:30.813 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:25:30.813 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:25:30.813 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-15T21:25:30.814 INFO:tasks.workunit.client.0.smithi136.stderr:3 2024-09-15T21:25:30.814 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:25:30.814 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:25:31.157 INFO:tasks.workunit.client.0.smithi136.stderr:osd.0 up in weight 1 up_from 113 up_thru 104 down_at 109 last_clean_interval [96,108) [v2:127.0.0.1:6802/445674539,v1:127.0.0.1:6803/445674539] [v2:127.0.0.1:6804/445674539,v1:127.0.0.1:6805/445674539] exists,up 980ffc84-a9d7-4f90-928b-2a70e4c90857 2024-09-15T21:25:31.157 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-15T21:25:31.157 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-15T21:25:31.157 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-15T21:25:31.157 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2024-09-15T21:25:31.157 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-15T21:25:31.157 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-15T21:25:31.158 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-15T21:25:31.158 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-15T21:25:31.158 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-15T21:25:31.158 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-15T21:25:31.158 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-15T21:25:31.158 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-15T21:25:31.159 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-15T21:25:31.252 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-15T21:25:31.253 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-15T21:25:31.253 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-15T21:25:31.253 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-15T21:25:31.253 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-15T21:25:31.253 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-15T21:25:31.632 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-15T21:25:31.632 INFO:tasks.workunit.client.0.smithi136.stderr:1 2024-09-15T21:25:31.632 INFO:tasks.workunit.client.0.smithi136.stderr:2' 2024-09-15T21:25:31.632 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-15T21:25:31.632 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T21:25:31.633 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-15T21:25:31.882 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=485331304450 2024-09-15T21:25:31.883 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 485331304450 2024-09-15T21:25:31.883 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-485331304450' 2024-09-15T21:25:31.883 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T21:25:31.883 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-15T21:25:32.134 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=446676598789 2024-09-15T21:25:32.134 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 446676598789 2024-09-15T21:25:32.134 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-485331304450 1-446676598789' 2024-09-15T21:25:32.134 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T21:25:32.135 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-15T21:25:32.376 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=335007449105 2024-09-15T21:25:32.376 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 335007449105 2024-09-15T21:25:32.376 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-485331304450 1-446676598789 2-335007449105' 2024-09-15T21:25:32.376 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T21:25:32.376 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-485331304450 2024-09-15T21:25:32.376 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T21:25:32.377 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-15T21:25:32.377 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-485331304450 2024-09-15T21:25:32.377 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T21:25:32.378 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=485331304450 2024-09-15T21:25:32.378 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 485331304450' 2024-09-15T21:25:32.378 INFO:tasks.workunit.client.0.smithi136.stderr:waiting osd.0 seq 485331304450 2024-09-15T21:25:32.379 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T21:25:32.824 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 1 -lt 485331304450 2024-09-15T21:25:32.825 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-15T21:25:33.826 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-15T21:25:33.826 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T21:25:34.114 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 485331304450 -lt 485331304450 2024-09-15T21:25:34.114 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T21:25:34.114 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-446676598789 2024-09-15T21:25:34.114 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T21:25:34.114 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-15T21:25:34.114 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-446676598789 2024-09-15T21:25:34.114 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T21:25:34.115 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=446676598789 2024-09-15T21:25:34.115 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 446676598789' 2024-09-15T21:25:34.115 INFO:tasks.workunit.client.0.smithi136.stderr:waiting osd.1 seq 446676598789 2024-09-15T21:25:34.115 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-15T21:25:34.393 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 446676598790 -lt 446676598789 2024-09-15T21:25:34.394 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T21:25:34.394 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-335007449105 2024-09-15T21:25:34.394 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T21:25:34.395 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-15T21:25:34.395 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-335007449105 2024-09-15T21:25:34.396 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T21:25:34.397 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=335007449105 2024-09-15T21:25:34.397 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 335007449105' 2024-09-15T21:25:34.397 INFO:tasks.workunit.client.0.smithi136.stderr:waiting osd.2 seq 335007449105 2024-09-15T21:25:34.397 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-15T21:25:34.672 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 335007449105 -lt 335007449105 2024-09-15T21:25:34.673 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-15T21:25:34.673 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T21:25:34.673 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T21:25:35.028 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 6 == 0 2024-09-15T21:25:35.029 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-15T21:25:35.029 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-15T21:25:35.029 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-15T21:25:35.029 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T21:25:35.029 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T21:25:35.029 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T21:25:35.029 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T21:25:35.311 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=3 2024-09-15T21:25:35.311 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-15T21:25:35.312 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T21:25:35.312 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T21:25:35.653 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 3 = 6 2024-09-15T21:25:35.653 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 3 '!=' -1 2024-09-15T21:25:35.653 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: loop=0 2024-09-15T21:25:35.653 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1652: wait_for_clean: num_active_clean=3 2024-09-15T21:25:35.653 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-15T21:25:35.653 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 0.1 2024-09-15T21:25:35.754 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-15T21:25:35.754 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-15T21:25:35.754 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-15T21:25:35.755 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-15T21:25:35.755 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T21:25:35.755 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T21:25:35.755 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T21:25:35.755 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T21:25:36.027 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=3 2024-09-15T21:25:36.027 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-15T21:25:36.027 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T21:25:36.027 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T21:25:36.371 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 3 = 6 2024-09-15T21:25:36.371 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 3 '!=' 3 2024-09-15T21:25:36.371 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: get_is_making_recovery_progress 2024-09-15T21:25:36.371 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2024-09-15T21:25:36.371 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-15T21:25:36.371 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-15T21:25:36.371 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-15T21:25:36.371 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2024-09-15T21:25:36.371 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-15T21:25:36.720 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=null 2024-09-15T21:25:36.721 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test null '!=' null 2024-09-15T21:25:36.721 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1655: wait_for_clean: (( 1 >= 13 )) 2024-09-15T21:25:36.721 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-15T21:25:36.721 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 0.2 2024-09-15T21:25:36.922 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-15T21:25:36.922 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-15T21:25:36.923 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-15T21:25:36.923 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-15T21:25:36.923 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T21:25:36.923 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T21:25:36.923 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T21:25:36.923 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T21:25:37.198 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=4 2024-09-15T21:25:37.198 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-15T21:25:37.199 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T21:25:37.199 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T21:25:37.542 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 = 6 2024-09-15T21:25:37.542 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 4 '!=' 3 2024-09-15T21:25:37.542 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: loop=0 2024-09-15T21:25:37.542 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1652: wait_for_clean: num_active_clean=4 2024-09-15T21:25:37.542 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-15T21:25:37.542 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 0.1 2024-09-15T21:25:37.644 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-15T21:25:37.644 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-15T21:25:37.644 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-15T21:25:37.644 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-15T21:25:37.644 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T21:25:37.644 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T21:25:37.644 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T21:25:37.644 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T21:25:37.918 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=4 2024-09-15T21:25:37.918 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-15T21:25:37.919 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T21:25:37.919 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T21:25:38.298 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 = 6 2024-09-15T21:25:38.299 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 4 '!=' 4 2024-09-15T21:25:38.299 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: get_is_making_recovery_progress 2024-09-15T21:25:38.299 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2024-09-15T21:25:38.299 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-15T21:25:38.299 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-15T21:25:38.299 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-15T21:25:38.299 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2024-09-15T21:25:38.299 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-15T21:25:38.645 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=null 2024-09-15T21:25:38.645 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test null '!=' null 2024-09-15T21:25:38.645 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1655: wait_for_clean: (( 1 >= 13 )) 2024-09-15T21:25:38.645 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-15T21:25:38.645 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 0.2 2024-09-15T21:25:38.843 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-15T21:25:38.843 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-15T21:25:38.844 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-15T21:25:38.844 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-15T21:25:38.844 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T21:25:38.844 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T21:25:38.844 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T21:25:38.844 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T21:25:39.121 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=6 2024-09-15T21:25:39.122 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-15T21:25:39.122 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T21:25:39.122 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T21:25:39.473 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 = 6 2024-09-15T21:25:39.473 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-15T21:25:39.473 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-15T21:25:39.473 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3669: corrupt_scrub_erasure: objectstore_tool td/osd-scrub-repair 0 EOBJ7 set-attr hinfo_key td/osd-scrub-repair/hinfo 2024-09-15T21:25:39.473 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2024-09-15T21:25:39.473 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2024-09-15T21:25:39.473 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=0 2024-09-15T21:25:39.473 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2024-09-15T21:25:39.473 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 EOBJ7 set-attr hinfo_key td/osd-scrub-repair/hinfo 2024-09-15T21:25:39.473 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-09-15T21:25:39.474 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2024-09-15T21:25:39.474 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=0 2024-09-15T21:25:39.474 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2024-09-15T21:25:39.474 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2024-09-15T21:25:39.474 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-15T21:25:39.474 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-15T21:25:39.474 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-15T21:25:39.474 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-15T21:25:39.474 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-15T21:25:39.779 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-15T21:25:39.779 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 EOBJ7 set-attr hinfo_key td/osd-scrub-repair/hinfo 2024-09-15T21:25:39.779 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-15T21:25:39.779 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-15T21:25:39.779 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=0 2024-09-15T21:25:39.779 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-15T21:25:39.779 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-09-15T21:25:39.779 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 EOBJ7 set-attr hinfo_key td/osd-scrub-repair/hinfo 2024-09-15T21:25:42.176 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 2024-09-15T21:25:42.177 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2024-09-15T21:25:42.177 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-15T21:25:42.177 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=0 2024-09-15T21:25:42.177 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-15T21:25:42.177 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-09-15T21:25:42.177 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=ce4892c7-6c6d-4b4f-8e6f-07a3e11133e5 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-15T21:25:42.177 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T21:25:42.177 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T21:25:42.177 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T21:25:42.177 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-09-15T21:25:42.177 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-09-15T21:25:42.177 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-15T21:25:42.178 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-15T21:25:42.178 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-15T21:25:42.178 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-15T21:25:42.178 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-15T21:25:42.178 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-15T21:25:42.178 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-15T21:25:42.178 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T21:25:42.178 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52602 2024-09-15T21:25:42.178 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.52602/$cluster-$name.asok' 2024-09-15T21:25:42.181 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.52602/$cluster-$name.asok' 2024-09-15T21:25:42.181 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-15T21:25:42.181 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-15T21:25:42.181 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-15T21:25:42.182 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T21:25:42.182 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T21:25:42.182 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T21:25:42.182 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T21:25:42.182 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-15T21:25:42.182 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-15T21:25:42.182 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-09-15T21:25:42.182 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.0 2024-09-15T21:25:42.182 INFO:tasks.workunit.client.0.smithi136.stderr:start osd.0 2024-09-15T21:25:42.182 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 0 --fsid=ce4892c7-6c6d-4b4f-8e6f-07a3e11133e5 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.52602/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T21:25:42.183 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-09-15T21:25:42.183 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 0 = 0 ']' 2024-09-15T21:25:42.183 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-15T21:25:42.183 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-15T21:25:42.183 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-15T21:25:42.210 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:25:42.207+0000 7ff676a19640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:25:42.212 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:25:42.209+0000 7ff676a19640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:25:42.214 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:25:42.211+0000 7ff676a19640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:25:42.470 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 0 2024-09-15T21:25:42.470 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-15T21:25:42.470 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2024-09-15T21:25:42.470 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-15T21:25:42.470 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-15T21:25:42.470 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:25:42.470 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-15T21:25:42.470 INFO:tasks.workunit.client.0.smithi136.stderr:0 2024-09-15T21:25:42.470 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:25:42.470 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:25:42.746 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:25:43.747 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:25:43.747 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:25:43.748 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-15T21:25:43.748 INFO:tasks.workunit.client.0.smithi136.stderr:1 2024-09-15T21:25:43.748 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:25:43.748 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:25:43.815 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:25:43.811+0000 7ff676a19640 -1 Falling back to public interface 2024-09-15T21:25:44.036 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:25:45.038 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:25:45.038 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:25:45.038 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-15T21:25:45.038 INFO:tasks.workunit.client.0.smithi136.stderr:2 2024-09-15T21:25:45.038 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:25:45.038 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:25:45.324 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:25:46.326 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:25:46.326 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:25:46.326 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-15T21:25:46.326 INFO:tasks.workunit.client.0.smithi136.stderr:3 2024-09-15T21:25:46.326 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:25:46.326 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:25:46.614 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:25:47.192 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:25:47.188+0000 7ff676a19640 -1 osd.0 116 log_to_monitors true 2024-09-15T21:25:47.615 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:25:47.616 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:25:47.616 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-15T21:25:47.616 INFO:tasks.workunit.client.0.smithi136.stderr:4 2024-09-15T21:25:47.616 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:25:47.616 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:25:47.975 INFO:tasks.workunit.client.0.smithi136.stderr:osd.0 up in weight 1 up_from 121 up_thru 113 down_at 117 last_clean_interval [113,116) [v2:127.0.0.1:6802/1432749641,v1:127.0.0.1:6803/1432749641] [v2:127.0.0.1:6804/1432749641,v1:127.0.0.1:6805/1432749641] exists,up 980ffc84-a9d7-4f90-928b-2a70e4c90857 2024-09-15T21:25:47.975 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-15T21:25:47.975 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-15T21:25:47.975 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-15T21:25:47.975 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2024-09-15T21:25:47.975 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-15T21:25:47.976 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-15T21:25:47.976 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-15T21:25:47.976 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-15T21:25:47.977 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-15T21:25:47.977 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-15T21:25:47.977 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-15T21:25:47.977 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-15T21:25:47.977 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-15T21:25:48.069 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-15T21:25:48.069 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-15T21:25:48.069 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-15T21:25:48.069 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-15T21:25:48.069 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-15T21:25:48.069 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-15T21:25:48.353 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-15T21:25:48.353 INFO:tasks.workunit.client.0.smithi136.stderr:1 2024-09-15T21:25:48.353 INFO:tasks.workunit.client.0.smithi136.stderr:2' 2024-09-15T21:25:48.353 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-15T21:25:48.353 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T21:25:48.353 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-15T21:25:48.485 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=519691042818 2024-09-15T21:25:48.485 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 519691042818 2024-09-15T21:25:48.485 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-519691042818' 2024-09-15T21:25:48.486 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T21:25:48.486 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-15T21:25:48.594 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=446676598794 2024-09-15T21:25:48.594 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 446676598794 2024-09-15T21:25:48.594 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-519691042818 1-446676598794' 2024-09-15T21:25:48.594 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T21:25:48.594 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-15T21:25:48.773 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=335007449109 2024-09-15T21:25:48.774 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 335007449109 2024-09-15T21:25:48.774 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-519691042818 1-446676598794 2-335007449109' 2024-09-15T21:25:48.774 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T21:25:48.774 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-519691042818 2024-09-15T21:25:48.774 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T21:25:48.775 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-15T21:25:48.775 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-519691042818 2024-09-15T21:25:48.776 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T21:25:48.777 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=519691042818 2024-09-15T21:25:48.777 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 519691042818' 2024-09-15T21:25:48.777 INFO:tasks.workunit.client.0.smithi136.stderr:waiting osd.0 seq 519691042818 2024-09-15T21:25:48.777 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T21:25:49.065 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 519691042818 -lt 519691042818 2024-09-15T21:25:49.066 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T21:25:49.066 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-446676598794 2024-09-15T21:25:49.066 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T21:25:49.067 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-15T21:25:49.067 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-446676598794 2024-09-15T21:25:49.067 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T21:25:49.069 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=446676598794 2024-09-15T21:25:49.069 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 446676598794' 2024-09-15T21:25:49.069 INFO:tasks.workunit.client.0.smithi136.stderr:waiting osd.1 seq 446676598794 2024-09-15T21:25:49.069 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-15T21:25:49.350 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 446676598794 -lt 446676598794 2024-09-15T21:25:49.350 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T21:25:49.351 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-335007449109 2024-09-15T21:25:49.351 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T21:25:49.352 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-15T21:25:49.352 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-335007449109 2024-09-15T21:25:49.352 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T21:25:49.353 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=335007449109 2024-09-15T21:25:49.353 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 335007449109' 2024-09-15T21:25:49.354 INFO:tasks.workunit.client.0.smithi136.stderr:waiting osd.2 seq 335007449109 2024-09-15T21:25:49.354 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-15T21:25:49.632 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 335007449109 -lt 335007449109 2024-09-15T21:25:49.632 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-15T21:25:49.632 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T21:25:49.632 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T21:25:49.980 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 6 == 0 2024-09-15T21:25:49.980 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-15T21:25:49.981 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-15T21:25:49.981 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-15T21:25:49.981 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T21:25:49.981 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T21:25:49.981 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T21:25:49.981 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T21:25:50.255 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=2 2024-09-15T21:25:50.256 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-15T21:25:50.256 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T21:25:50.256 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T21:25:50.600 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 2 = 6 2024-09-15T21:25:50.600 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 2 '!=' -1 2024-09-15T21:25:50.600 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: loop=0 2024-09-15T21:25:50.600 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1652: wait_for_clean: num_active_clean=2 2024-09-15T21:25:50.600 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-15T21:25:50.601 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 0.1 2024-09-15T21:25:50.702 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-15T21:25:50.702 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-15T21:25:50.702 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-15T21:25:50.702 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-15T21:25:50.702 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T21:25:50.702 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T21:25:50.702 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T21:25:50.703 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T21:25:50.976 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=2 2024-09-15T21:25:50.976 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-15T21:25:50.977 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T21:25:50.977 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T21:25:51.323 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 2 = 6 2024-09-15T21:25:51.323 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 2 '!=' 2 2024-09-15T21:25:51.323 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: get_is_making_recovery_progress 2024-09-15T21:25:51.323 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2024-09-15T21:25:51.323 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-15T21:25:51.323 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-15T21:25:51.323 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-15T21:25:51.323 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2024-09-15T21:25:51.323 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-15T21:25:51.667 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=null 2024-09-15T21:25:51.668 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test null '!=' null 2024-09-15T21:25:51.668 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1655: wait_for_clean: (( 1 >= 13 )) 2024-09-15T21:25:51.668 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-15T21:25:51.668 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 0.2 2024-09-15T21:25:51.869 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-15T21:25:51.869 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-15T21:25:51.869 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-15T21:25:51.869 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-15T21:25:51.869 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T21:25:51.869 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T21:25:51.869 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T21:25:51.870 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T21:25:52.157 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=2 2024-09-15T21:25:52.158 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-15T21:25:52.158 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T21:25:52.158 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T21:25:52.501 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 2 = 6 2024-09-15T21:25:52.501 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 2 '!=' 2 2024-09-15T21:25:52.501 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: get_is_making_recovery_progress 2024-09-15T21:25:52.501 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2024-09-15T21:25:52.501 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-15T21:25:52.501 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-15T21:25:52.501 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-15T21:25:52.502 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2024-09-15T21:25:52.502 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-15T21:25:52.864 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=null 2024-09-15T21:25:52.865 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test null '!=' null 2024-09-15T21:25:52.865 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1655: wait_for_clean: (( 2 >= 13 )) 2024-09-15T21:25:52.865 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-15T21:25:52.865 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 0.4 2024-09-15T21:25:53.265 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-15T21:25:53.265 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-15T21:25:53.266 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-15T21:25:53.266 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-15T21:25:53.266 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T21:25:53.266 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T21:25:53.266 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T21:25:53.266 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T21:25:53.538 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=6 2024-09-15T21:25:53.539 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-15T21:25:53.539 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T21:25:53.539 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T21:25:53.881 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 = 6 2024-09-15T21:25:53.881 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-15T21:25:53.881 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-15T21:25:53.881 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3670: corrupt_scrub_erasure: rm -f td/osd-scrub-repair/hinfo 2024-09-15T21:25:53.882 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3676: corrupt_scrub_erasure: get_pg ecpool EOBJ0 2024-09-15T21:25:53.882 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1084: get_pg: local poolname=ecpool 2024-09-15T21:25:53.882 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1085: get_pg: local objectname=EOBJ0 2024-09-15T21:25:53.882 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1087: get_pg: ceph --format json osd map ecpool EOBJ0 2024-09-15T21:25:53.882 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1087: get_pg: jq -r .pgid 2024-09-15T21:25:54.160 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3676: corrupt_scrub_erasure: local pg=3.0 2024-09-15T21:25:54.160 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3678: corrupt_scrub_erasure: pg_scrub 3.0 2024-09-15T21:25:54.160 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1883: pg_scrub: local pgid=3.0 2024-09-15T21:25:54.160 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1884: pg_scrub: get_last_scrub_stamp 3.0 2024-09-15T21:25:54.160 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=3.0 2024-09-15T21:25:54.161 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-15T21:25:54.161 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-15T21:25:54.161 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_scrub_stamp' 2024-09-15T21:25:54.431 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1884: pg_scrub: local last_scrub=2024-09-15T21:23:09.927703+0000 2024-09-15T21:25:54.431 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1885: pg_scrub: ceph pg scrub 3.0 2024-09-15T21:25:54.691 INFO:tasks.workunit.client.0.smithi136.stderr:instructing pg 3.0s0 on osd.1 to scrub 2024-09-15T21:25:54.702 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1886: pg_scrub: wait_for_scrub 3.0 2024-09-15T21:23:09.927703+0000 2024-09-15T21:25:54.702 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: local pgid=3.0 2024-09-15T21:25:54.702 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2018: wait_for_scrub: local last_scrub=2024-09-15T21:23:09.927703+0000 2024-09-15T21:25:54.702 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2019: wait_for_scrub: local sname=last_scrub_stamp 2024-09-15T21:25:54.703 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i=0 )) 2024-09-15T21:25:54.703 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-15T21:25:54.703 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 3.0 last_scrub_stamp 2024-09-15T21:25:54.703 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=3.0 2024-09-15T21:25:54.703 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-15T21:25:54.703 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-15T21:25:54.703 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_scrub_stamp' 2024-09-15T21:25:54.973 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-15T21:23:09.927703+0000 '>' 2024-09-15T21:23:09.927703+0000 2024-09-15T21:25:54.974 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-15T21:25:55.974 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-15T21:25:55.975 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-15T21:25:55.975 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 3.0 last_scrub_stamp 2024-09-15T21:25:55.975 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=3.0 2024-09-15T21:25:55.975 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-15T21:25:55.976 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-15T21:25:55.976 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_scrub_stamp' 2024-09-15T21:25:56.260 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-15T21:23:09.927703+0000 '>' 2024-09-15T21:23:09.927703+0000 2024-09-15T21:25:56.260 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-15T21:25:57.261 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-15T21:25:57.261 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-15T21:25:57.263 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 3.0 last_scrub_stamp 2024-09-15T21:25:57.263 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=3.0 2024-09-15T21:25:57.263 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-15T21:25:57.263 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-15T21:25:57.263 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_scrub_stamp' 2024-09-15T21:25:57.545 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-15T21:23:09.927703+0000 '>' 2024-09-15T21:23:09.927703+0000 2024-09-15T21:25:57.546 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-15T21:25:58.547 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-15T21:25:58.547 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-15T21:25:58.548 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 3.0 last_scrub_stamp 2024-09-15T21:25:58.548 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=3.0 2024-09-15T21:25:58.548 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-15T21:25:58.548 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-15T21:25:58.548 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_scrub_stamp' 2024-09-15T21:25:58.832 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-15T21:25:55.188382+0000 '>' 2024-09-15T21:23:09.927703+0000 2024-09-15T21:25:58.832 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2023: wait_for_scrub: return 0 2024-09-15T21:25:58.832 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3680: corrupt_scrub_erasure: rados list-inconsistent-pg ecpool 2024-09-15T21:25:58.860 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3682: corrupt_scrub_erasure: jq '. | length' td/osd-scrub-repair/json 2024-09-15T21:25:58.862 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3682: corrupt_scrub_erasure: test 1 = 1 2024-09-15T21:25:58.862 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3684: corrupt_scrub_erasure: jq -r '.[0]' td/osd-scrub-repair/json 2024-09-15T21:25:58.865 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3684: corrupt_scrub_erasure: test 3.0 = 3.0 2024-09-15T21:25:58.865 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3686: corrupt_scrub_erasure: rados list-inconsistent-obj 3.0 2024-09-15T21:25:58.893 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3688: corrupt_scrub_erasure: jq .epoch td/osd-scrub-repair/json 2024-09-15T21:25:58.895 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3688: corrupt_scrub_erasure: epoch=121 2024-09-15T21:25:58.896 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3690: corrupt_scrub_erasure: jq 'def walk(f): 2024-09-15T21:25:58.896 INFO:tasks.workunit.client.0.smithi136.stderr: . as $in 2024-09-15T21:25:58.896 INFO:tasks.workunit.client.0.smithi136.stderr: | if type == "object" then 2024-09-15T21:25:58.896 INFO:tasks.workunit.client.0.smithi136.stderr: reduce keys[] as $key 2024-09-15T21:25:58.896 INFO:tasks.workunit.client.0.smithi136.stderr: ( {}; . + { ($key): ($in[$key] | walk(f)) } ) | f 2024-09-15T21:25:58.896 INFO:tasks.workunit.client.0.smithi136.stderr: elif type == "array" then map( walk(f) ) | f 2024-09-15T21:25:58.896 INFO:tasks.workunit.client.0.smithi136.stderr: else f 2024-09-15T21:25:58.897 INFO:tasks.workunit.client.0.smithi136.stderr: end; 2024-09-15T21:25:58.897 INFO:tasks.workunit.client.0.smithi136.stderr:walk(if type == "object" then del(.mtime) else . end) 2024-09-15T21:25:58.897 INFO:tasks.workunit.client.0.smithi136.stderr:| walk(if type == "object" then del(.local_mtime) else . end) 2024-09-15T21:25:58.897 INFO:tasks.workunit.client.0.smithi136.stderr:| walk(if type == "object" then del(.last_reqid) else . end) 2024-09-15T21:25:58.897 INFO:tasks.workunit.client.0.smithi136.stderr:| walk(if type == "object" then del(.version) else . end) 2024-09-15T21:25:58.897 INFO:tasks.workunit.client.0.smithi136.stderr:| walk(if type == "object" then del(.prior_version) else . end)' 2024-09-15T21:25:58.897 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3690: corrupt_scrub_erasure: jq .inconsistents 2024-09-15T21:25:58.897 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3690: corrupt_scrub_erasure: python3 -c 'import json; import sys ; JSON=sys.stdin.read() ; ud = json.loads(JSON) ; print(json.dumps(ud, sort_keys=True, indent=2))' 2024-09-15T21:25:58.919 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4310: corrupt_scrub_erasure: jq 'def walk(f): 2024-09-15T21:25:58.919 INFO:tasks.workunit.client.0.smithi136.stderr: . as $in 2024-09-15T21:25:58.920 INFO:tasks.workunit.client.0.smithi136.stderr: | if type == "object" then 2024-09-15T21:25:58.920 INFO:tasks.workunit.client.0.smithi136.stderr: reduce keys[] as $key 2024-09-15T21:25:58.920 INFO:tasks.workunit.client.0.smithi136.stderr: ( {}; . + { ($key): ($in[$key] | walk(f)) } ) | f 2024-09-15T21:25:58.920 INFO:tasks.workunit.client.0.smithi136.stderr: elif type == "array" then map( walk(f) ) | f 2024-09-15T21:25:58.920 INFO:tasks.workunit.client.0.smithi136.stderr: else f 2024-09-15T21:25:58.920 INFO:tasks.workunit.client.0.smithi136.stderr: end; 2024-09-15T21:25:58.920 INFO:tasks.workunit.client.0.smithi136.stderr:walk(if type == "object" then del(.mtime) else . end) 2024-09-15T21:25:58.920 INFO:tasks.workunit.client.0.smithi136.stderr:| walk(if type == "object" then del(.local_mtime) else . end) 2024-09-15T21:25:58.920 INFO:tasks.workunit.client.0.smithi136.stderr:| walk(if type == "object" then del(.last_reqid) else . end) 2024-09-15T21:25:58.920 INFO:tasks.workunit.client.0.smithi136.stderr:| walk(if type == "object" then del(.version) else . end) 2024-09-15T21:25:58.920 INFO:tasks.workunit.client.0.smithi136.stderr:| walk(if type == "object" then del(.prior_version) else . end)' td/osd-scrub-repair/json 2024-09-15T21:25:58.920 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4310: corrupt_scrub_erasure: jq .inconsistents 2024-09-15T21:25:58.920 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4310: corrupt_scrub_erasure: python3 -c 'import json; import sys ; JSON=sys.stdin.read() ; ud = json.loads(JSON) ; print(json.dumps(ud, sort_keys=True, indent=2))' 2024-09-15T21:25:58.942 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4311: corrupt_scrub_erasure: multidiff td/osd-scrub-repair/checkcsjson td/osd-scrub-repair/csjson 2024-09-15T21:25:58.942 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2434: multidiff: diff td/osd-scrub-repair/checkcsjson td/osd-scrub-repair/csjson 2024-09-15T21:25:58.944 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4312: corrupt_scrub_erasure: test no = yes 2024-09-15T21:25:58.944 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4317: corrupt_scrub_erasure: test '' = yes 2024-09-15T21:25:58.944 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4322: corrupt_scrub_erasure: pg_deep_scrub 3.0 2024-09-15T21:25:58.944 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1890: pg_deep_scrub: local pgid=3.0 2024-09-15T21:25:58.944 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1891: pg_deep_scrub: get_last_scrub_stamp 3.0 last_deep_scrub_stamp 2024-09-15T21:25:58.944 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=3.0 2024-09-15T21:25:58.944 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_deep_scrub_stamp 2024-09-15T21:25:58.944 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-15T21:25:58.944 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_deep_scrub_stamp' 2024-09-15T21:25:59.218 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1891: pg_deep_scrub: local last_scrub=2024-09-15T21:23:09.927703+0000 2024-09-15T21:25:59.218 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1892: pg_deep_scrub: ceph pg deep-scrub 3.0 2024-09-15T21:25:59.480 INFO:tasks.workunit.client.0.smithi136.stderr:instructing pg 3.0s0 on osd.1 to deep-scrub 2024-09-15T21:25:59.492 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1893: pg_deep_scrub: wait_for_scrub 3.0 2024-09-15T21:23:09.927703+0000 last_deep_scrub_stamp 2024-09-15T21:25:59.492 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: local pgid=3.0 2024-09-15T21:25:59.493 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2018: wait_for_scrub: local last_scrub=2024-09-15T21:23:09.927703+0000 2024-09-15T21:25:59.493 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2019: wait_for_scrub: local sname=last_deep_scrub_stamp 2024-09-15T21:25:59.493 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i=0 )) 2024-09-15T21:25:59.493 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-15T21:25:59.493 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 3.0 last_deep_scrub_stamp 2024-09-15T21:25:59.493 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=3.0 2024-09-15T21:25:59.493 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_deep_scrub_stamp 2024-09-15T21:25:59.493 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-15T21:25:59.493 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_deep_scrub_stamp' 2024-09-15T21:25:59.765 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-15T21:23:09.927703+0000 '>' 2024-09-15T21:23:09.927703+0000 2024-09-15T21:25:59.765 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-15T21:26:00.766 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-15T21:26:00.766 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-15T21:26:00.767 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 3.0 last_deep_scrub_stamp 2024-09-15T21:26:00.767 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=3.0 2024-09-15T21:26:00.767 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_deep_scrub_stamp 2024-09-15T21:26:00.768 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-15T21:26:00.768 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_deep_scrub_stamp' 2024-09-15T21:26:01.051 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-15T21:23:09.927703+0000 '>' 2024-09-15T21:23:09.927703+0000 2024-09-15T21:26:01.051 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-15T21:26:02.054 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-15T21:26:02.054 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-15T21:26:02.054 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 3.0 last_deep_scrub_stamp 2024-09-15T21:26:02.054 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=3.0 2024-09-15T21:26:02.055 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_deep_scrub_stamp 2024-09-15T21:26:02.055 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-15T21:26:02.055 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_deep_scrub_stamp' 2024-09-15T21:26:02.336 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-15T21:23:09.927703+0000 '>' 2024-09-15T21:23:09.927703+0000 2024-09-15T21:26:02.336 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-15T21:26:03.337 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-15T21:26:03.337 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-15T21:26:03.338 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 3.0 last_deep_scrub_stamp 2024-09-15T21:26:03.338 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=3.0 2024-09-15T21:26:03.338 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_deep_scrub_stamp 2024-09-15T21:26:03.339 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-15T21:26:03.339 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_deep_scrub_stamp' 2024-09-15T21:26:03.622 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-15T21:26:00.320850+0000 '>' 2024-09-15T21:23:09.927703+0000 2024-09-15T21:26:03.622 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2023: wait_for_scrub: return 0 2024-09-15T21:26:03.622 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4324: corrupt_scrub_erasure: rados list-inconsistent-pg ecpool 2024-09-15T21:26:03.651 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4326: corrupt_scrub_erasure: jq '. | length' td/osd-scrub-repair/json 2024-09-15T21:26:03.653 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4326: corrupt_scrub_erasure: test 1 = 1 2024-09-15T21:26:03.653 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4328: corrupt_scrub_erasure: jq -r '.[0]' td/osd-scrub-repair/json 2024-09-15T21:26:03.656 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4328: corrupt_scrub_erasure: test 3.0 = 3.0 2024-09-15T21:26:03.656 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4330: corrupt_scrub_erasure: rados list-inconsistent-obj 3.0 2024-09-15T21:26:03.683 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4332: corrupt_scrub_erasure: jq .epoch td/osd-scrub-repair/json 2024-09-15T21:26:03.686 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4332: corrupt_scrub_erasure: epoch=121 2024-09-15T21:26:03.686 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4334: corrupt_scrub_erasure: '[' true = true ']' 2024-09-15T21:26:03.687 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4336: corrupt_scrub_erasure: jq 'def walk(f): 2024-09-15T21:26:03.687 INFO:tasks.workunit.client.0.smithi136.stderr: . as $in 2024-09-15T21:26:03.687 INFO:tasks.workunit.client.0.smithi136.stderr: | if type == "object" then 2024-09-15T21:26:03.687 INFO:tasks.workunit.client.0.smithi136.stderr: reduce keys[] as $key 2024-09-15T21:26:03.687 INFO:tasks.workunit.client.0.smithi136.stderr: ( {}; . + { ($key): ($in[$key] | walk(f)) } ) | f 2024-09-15T21:26:03.687 INFO:tasks.workunit.client.0.smithi136.stderr: elif type == "array" then map( walk(f) ) | f 2024-09-15T21:26:03.687 INFO:tasks.workunit.client.0.smithi136.stderr: else f 2024-09-15T21:26:03.687 INFO:tasks.workunit.client.0.smithi136.stderr: end; 2024-09-15T21:26:03.687 INFO:tasks.workunit.client.0.smithi136.stderr:walk(if type == "object" then del(.mtime) else . end) 2024-09-15T21:26:03.687 INFO:tasks.workunit.client.0.smithi136.stderr:| walk(if type == "object" then del(.local_mtime) else . end) 2024-09-15T21:26:03.687 INFO:tasks.workunit.client.0.smithi136.stderr:| walk(if type == "object" then del(.last_reqid) else . end) 2024-09-15T21:26:03.687 INFO:tasks.workunit.client.0.smithi136.stderr:| walk(if type == "object" then del(.version) else . end) 2024-09-15T21:26:03.687 INFO:tasks.workunit.client.0.smithi136.stderr:| walk(if type == "object" then del(.prior_version) else . end)' 2024-09-15T21:26:03.687 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4336: corrupt_scrub_erasure: jq .inconsistents 2024-09-15T21:26:03.688 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4336: corrupt_scrub_erasure: python3 -c 'import json; import sys ; JSON=sys.stdin.read() ; ud = json.loads(JSON) ; print(json.dumps(ud, sort_keys=True, indent=2))' 2024-09-15T21:26:03.710 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5724: corrupt_scrub_erasure: jq 'def walk(f): 2024-09-15T21:26:03.710 INFO:tasks.workunit.client.0.smithi136.stderr: . as $in 2024-09-15T21:26:03.710 INFO:tasks.workunit.client.0.smithi136.stderr: | if type == "object" then 2024-09-15T21:26:03.710 INFO:tasks.workunit.client.0.smithi136.stderr: reduce keys[] as $key 2024-09-15T21:26:03.710 INFO:tasks.workunit.client.0.smithi136.stderr: ( {}; . + { ($key): ($in[$key] | walk(f)) } ) | f 2024-09-15T21:26:03.710 INFO:tasks.workunit.client.0.smithi136.stderr: elif type == "array" then map( walk(f) ) | f 2024-09-15T21:26:03.710 INFO:tasks.workunit.client.0.smithi136.stderr: else f 2024-09-15T21:26:03.710 INFO:tasks.workunit.client.0.smithi136.stderr: end; 2024-09-15T21:26:03.710 INFO:tasks.workunit.client.0.smithi136.stderr:walk(if type == "object" then del(.mtime) else . end) 2024-09-15T21:26:03.711 INFO:tasks.workunit.client.0.smithi136.stderr:| walk(if type == "object" then del(.local_mtime) else . end) 2024-09-15T21:26:03.711 INFO:tasks.workunit.client.0.smithi136.stderr:| walk(if type == "object" then del(.last_reqid) else . end) 2024-09-15T21:26:03.711 INFO:tasks.workunit.client.0.smithi136.stderr:| walk(if type == "object" then del(.version) else . end) 2024-09-15T21:26:03.711 INFO:tasks.workunit.client.0.smithi136.stderr:| walk(if type == "object" then del(.prior_version) else . end)' td/osd-scrub-repair/json 2024-09-15T21:26:03.711 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5724: corrupt_scrub_erasure: jq .inconsistents 2024-09-15T21:26:03.711 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5724: corrupt_scrub_erasure: python3 -c 'import json; import sys ; JSON=sys.stdin.read() ; ud = json.loads(JSON) ; print(json.dumps(ud, sort_keys=True, indent=2))' 2024-09-15T21:26:03.733 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5725: corrupt_scrub_erasure: multidiff td/osd-scrub-repair/checkcsjson td/osd-scrub-repair/csjson 2024-09-15T21:26:03.733 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2434: multidiff: diff td/osd-scrub-repair/checkcsjson td/osd-scrub-repair/csjson 2024-09-15T21:26:03.733 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5726: corrupt_scrub_erasure: test no = yes 2024-09-15T21:26:03.734 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5737: corrupt_scrub_erasure: test '' = yes 2024-09-15T21:26:03.734 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5742: corrupt_scrub_erasure: ceph osd pool rm ecpool ecpool --yes-i-really-really-mean-it 2024-09-15T21:26:04.232 INFO:tasks.workunit.client.0.smithi136.stderr:pool 'ecpool' removed 2024-09-15T21:26:04.244 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:66: run: teardown td/osd-scrub-repair 2024-09-15T21:26:04.245 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/osd-scrub-repair 2024-09-15T21:26:04.245 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-15T21:26:04.245 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/osd-scrub-repair KILL 2024-09-15T21:26:04.245 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-15T21:26:04.245 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-15T21:26:04.245 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-15T21:26:04.246 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-15T21:26:04.246 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-15T21:26:04.354 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-15T21:26:04.354 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-15T21:26:04.354 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-15T21:26:04.355 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-15T21:26:04.356 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-15T21:26:04.356 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-15T21:26:04.356 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-15T21:26:04.357 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T21:26:04.357 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-15T21:26:04.358 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-15T21:26:04.358 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T21:26:04.358 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-15T21:26:04.359 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-15T21:26:04.359 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/osd-scrub-repair 2024-09-15T21:26:04.388 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-15T21:26:04.388 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T21:26:04.388 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52602 2024-09-15T21:26:04.388 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.52602 2024-09-15T21:26:04.390 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-15T21:26:04.390 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-15T21:26:04.390 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:63: run: for func in $funcs 2024-09-15T21:26:04.390 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:64: run: setup td/osd-scrub-repair 2024-09-15T21:26:04.390 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/osd-scrub-repair 2024-09-15T21:26:04.390 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/osd-scrub-repair 2024-09-15T21:26:04.390 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/osd-scrub-repair 2024-09-15T21:26:04.390 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-15T21:26:04.391 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/osd-scrub-repair KILL 2024-09-15T21:26:04.391 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-15T21:26:04.391 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-15T21:26:04.391 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-15T21:26:04.391 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-15T21:26:04.391 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-15T21:26:04.391 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-15T21:26:04.392 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-15T21:26:04.393 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-15T21:26:04.393 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-15T21:26:04.394 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-15T21:26:04.394 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-15T21:26:04.394 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-15T21:26:04.395 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T21:26:04.395 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-15T21:26:04.396 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-15T21:26:04.396 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T21:26:04.397 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-15T21:26:04.398 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-15T21:26:04.398 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/osd-scrub-repair 2024-09-15T21:26:04.399 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-15T21:26:04.399 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T21:26:04.399 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52602 2024-09-15T21:26:04.399 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.52602 2024-09-15T21:26:04.400 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-15T21:26:04.400 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-15T21:26:04.400 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/osd-scrub-repair 2024-09-15T21:26:04.402 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2024-09-15T21:26:04.402 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T21:26:04.402 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52602 2024-09-15T21:26:04.402 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.52602 2024-09-15T21:26:04.403 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2024-09-15T21:26:04.404 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 4096 -le 1024 ']' 2024-09-15T21:26:04.404 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2024-09-15T21:26:04.405 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/osd-scrub-repair 1' TERM HUP INT 2024-09-15T21:26:04.405 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:65: run: TEST_corrupt_scrub_replicated td/osd-scrub-repair 2024-09-15T21:26:04.405 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:991: TEST_corrupt_scrub_replicated: local dir=td/osd-scrub-repair 2024-09-15T21:26:04.405 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:992: TEST_corrupt_scrub_replicated: local poolname=csr_pool 2024-09-15T21:26:04.405 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:993: TEST_corrupt_scrub_replicated: local total_objs=19 2024-09-15T21:26:04.405 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:995: TEST_corrupt_scrub_replicated: run_mon td/osd-scrub-repair a --osd_pool_default_size=2 2024-09-15T21:26:04.405 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/osd-scrub-repair 2024-09-15T21:26:04.405 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2024-09-15T21:26:04.405 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2024-09-15T21:26:04.405 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2024-09-15T21:26:04.405 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/osd-scrub-repair/a 2024-09-15T21:26:04.405 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-scrub-repair/a --run-dir=td/osd-scrub-repair --osd_pool_default_size=2 2024-09-15T21:26:04.801 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2024-09-15T21:26:04.801 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-15T21:26:04.802 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-15T21:26:04.802 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-15T21:26:04.802 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T21:26:04.802 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52602 2024-09-15T21:26:04.802 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.52602/$cluster-$name.asok' 2024-09-15T21:26:04.802 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-scrub-repair/a '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.52602/$cluster-$name.asok' --mon-cluster-log-file=td/osd-scrub-repair/log --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=2 2024-09-15T21:26:04.991 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2024-09-15T21:26:04.993 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2024-09-15T21:26:04.993 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-15T21:26:04.993 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-15T21:26:04.993 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2024-09-15T21:26:04.993 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2024-09-15T21:26:04.993 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-15T21:26:04.993 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-15T21:26:04.993 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-15T21:26:04.994 INFO:tasks.workunit.client.0.smithi136.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-15T21:26:04.994 INFO:tasks.workunit.client.0.smithi136.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T21:26:04.994 INFO:tasks.workunit.client.0.smithi136.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52602 2024-09-15T21:26:04.994 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.52602/ceph-mon.a.asok 2024-09-15T21:26:04.995 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-15T21:26:04.995 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.52602/ceph-mon.a.asok config get fsid 2024-09-15T21:26:05.072 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2024-09-15T21:26:05.073 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-15T21:26:05.073 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-15T21:26:05.073 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2024-09-15T21:26:05.074 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2024-09-15T21:26:05.074 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-15T21:26:05.074 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-15T21:26:05.074 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-15T21:26:05.074 INFO:tasks.workunit.client.0.smithi136.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-15T21:26:05.074 INFO:tasks.workunit.client.0.smithi136.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T21:26:05.074 INFO:tasks.workunit.client.0.smithi136.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52602 2024-09-15T21:26:05.074 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.52602/ceph-mon.a.asok 2024-09-15T21:26:05.074 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-15T21:26:05.074 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.52602/ceph-mon.a.asok config get mon_host 2024-09-15T21:26:05.151 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:996: TEST_corrupt_scrub_replicated: run_mgr td/osd-scrub-repair x 2024-09-15T21:26:05.151 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: run_mgr: local dir=td/osd-scrub-repair 2024-09-15T21:26:05.151 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: run_mgr: shift 2024-09-15T21:26:05.151 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: run_mgr: local id=x 2024-09-15T21:26:05.151 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: run_mgr: shift 2024-09-15T21:26:05.151 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:548: run_mgr: local data=td/osd-scrub-repair/x 2024-09-15T21:26:05.151 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:550: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-15T21:26:05.959 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: get_asok_path 2024-09-15T21:26:05.960 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-15T21:26:05.960 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-15T21:26:05.960 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-15T21:26:05.960 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T21:26:05.960 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52602 2024-09-15T21:26:05.960 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.52602/$cluster-$name.asok' 2024-09-15T21:26:05.960 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-15T21:26:05.961 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-scrub-repair/x '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.52602/$cluster-$name.asok' --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-15T21:26:05.993 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:997: TEST_corrupt_scrub_replicated: run_osd td/osd-scrub-repair 0 2024-09-15T21:26:05.993 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-scrub-repair 2024-09-15T21:26:05.993 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-15T21:26:05.993 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=0 2024-09-15T21:26:05.993 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-15T21:26:05.994 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-scrub-repair/0 2024-09-15T21:26:05.994 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=ce4892c7-6c6d-4b4f-8e6f-07a3e11133e5 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-15T21:26:05.994 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T21:26:05.994 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T21:26:05.994 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T21:26:05.994 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-09-15T21:26:05.994 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-09-15T21:26:05.994 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-15T21:26:05.994 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-15T21:26:05.995 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-15T21:26:05.995 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-15T21:26:05.995 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-15T21:26:05.995 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-15T21:26:05.995 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-15T21:26:05.996 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T21:26:05.996 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52602 2024-09-15T21:26:05.996 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.52602/$cluster-$name.asok' 2024-09-15T21:26:05.996 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.52602/$cluster-$name.asok' 2024-09-15T21:26:05.996 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-15T21:26:05.996 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-15T21:26:05.996 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-15T21:26:05.996 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-15T21:26:05.996 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-15T21:26:05.996 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T21:26:05.996 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T21:26:05.996 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T21:26:05.996 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T21:26:05.996 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-15T21:26:05.997 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-15T21:26:05.997 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-scrub-repair/0 2024-09-15T21:26:05.997 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-15T21:26:05.997 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=7a68df98-8e0f-45ef-a9e9-b2a45ca8fd3f 2024-09-15T21:26:05.997 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd0 7a68df98-8e0f-45ef-a9e9-b2a45ca8fd3f' 2024-09-15T21:26:05.998 INFO:tasks.workunit.client.0.smithi136.stdout:add osd0 7a68df98-8e0f-45ef-a9e9-b2a45ca8fd3f 2024-09-15T21:26:05.998 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-15T21:26:06.014 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQDuUOdm6nmgABAAcYdQZjDwb5AhDfOp9ZHC/w== 2024-09-15T21:26:06.014 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQDuUOdm6nmgABAAcYdQZjDwb5AhDfOp9ZHC/w=="}' 2024-09-15T21:26:06.014 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 7a68df98-8e0f-45ef-a9e9-b2a45ca8fd3f -i td/osd-scrub-repair/0/new.json 2024-09-15T21:26:06.215 INFO:tasks.workunit.client.0.smithi136.stdout:0 2024-09-15T21:26:06.230 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-scrub-repair/0/new.json 2024-09-15T21:26:06.230 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 0 --fsid=ce4892c7-6c6d-4b4f-8e6f-07a3e11133e5 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.52602/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDuUOdm6nmgABAAcYdQZjDwb5AhDfOp9ZHC/w== --osd-uuid 7a68df98-8e0f-45ef-a9e9-b2a45ca8fd3f 2024-09-15T21:26:06.260 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:26:06.258+0000 7f450d431640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:26:06.266 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:26:06.264+0000 7f450d431640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:26:06.267 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:26:06.265+0000 7f450d431640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:26:06.268 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:26:06.265+0000 7f450d431640 -1 bdev(0x55ee722b3c00 td/osd-scrub-repair/0/block) open stat got: (1) Operation not permitted 2024-09-15T21:26:06.268 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:26:06.265+0000 7f450d431640 -1 bluestore(td/osd-scrub-repair/0) _read_fsid unparsable uuid 2024-09-15T21:26:12.921 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-scrub-repair/0/keyring 2024-09-15T21:26:12.921 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-15T21:26:12.926 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd0 key to auth repository 2024-09-15T21:26:12.926 INFO:tasks.workunit.client.0.smithi136.stdout:adding osd0 key to auth repository 2024-09-15T21:26:12.926 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-scrub-repair/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-15T21:26:13.292 INFO:tasks.workunit.client.0.smithi136.stdout:start osd.0 2024-09-15T21:26:13.293 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.0 2024-09-15T21:26:13.293 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 0 --fsid=ce4892c7-6c6d-4b4f-8e6f-07a3e11133e5 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.52602/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T21:26:13.293 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-15T21:26:13.294 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-15T21:26:13.294 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-15T21:26:13.320 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:26:13.317+0000 7f7942418640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:26:13.321 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:26:13.319+0000 7f7942418640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:26:13.322 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:26:13.320+0000 7f7942418640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:26:13.570 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 0 2024-09-15T21:26:13.630 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-15T21:26:13.631 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2024-09-15T21:26:13.631 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-15T21:26:13.631 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-15T21:26:13.631 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:26:13.631 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-15T21:26:13.631 INFO:tasks.workunit.client.0.smithi136.stdout:0 2024-09-15T21:26:13.632 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:26:13.632 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:26:13.847 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:26:14.846 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:26:14.846 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:26:14.847 INFO:tasks.workunit.client.0.smithi136.stdout:1 2024-09-15T21:26:14.847 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-15T21:26:14.847 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:26:14.847 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:26:14.938 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:26:14.936+0000 7f7942418640 -1 Falling back to public interface 2024-09-15T21:26:15.131 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:26:16.133 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:26:16.133 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:26:16.133 INFO:tasks.workunit.client.0.smithi136.stdout:2 2024-09-15T21:26:16.134 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-15T21:26:16.134 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:26:16.134 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:26:16.412 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:26:17.414 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:26:17.414 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:26:17.414 INFO:tasks.workunit.client.0.smithi136.stdout:3 2024-09-15T21:26:17.415 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-15T21:26:17.415 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:26:17.415 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:26:17.548 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:26:17.545+0000 7f7942418640 -1 osd.0 0 log_to_monitors true 2024-09-15T21:26:17.713 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:26:18.714 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:26:18.714 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:26:18.715 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-15T21:26:18.715 INFO:tasks.workunit.client.0.smithi136.stdout:4 2024-09-15T21:26:18.715 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:26:18.715 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:26:19.003 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:26:20.005 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:26:20.005 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:26:20.006 INFO:tasks.workunit.client.0.smithi136.stdout:5 2024-09-15T21:26:20.006 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-15T21:26:20.006 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:26:20.006 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:26:20.299 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:26:21.300 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:26:21.301 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:26:21.301 INFO:tasks.workunit.client.0.smithi136.stdout:6 2024-09-15T21:26:21.301 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-15T21:26:21.301 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:26:21.301 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:26:21.588 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:26:22.590 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:26:22.590 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:26:22.590 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-15T21:26:22.590 INFO:tasks.workunit.client.0.smithi136.stdout:7 2024-09-15T21:26:22.590 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:26:22.591 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:26:22.879 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:26:23.880 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:26:23.880 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:26:23.880 INFO:tasks.workunit.client.0.smithi136.stdout:8 2024-09-15T21:26:23.881 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-15T21:26:23.881 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:26:23.881 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:26:24.258 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:26:25.260 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:26:25.260 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:26:25.260 INFO:tasks.workunit.client.0.smithi136.stdout:9 2024-09-15T21:26:25.260 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-15T21:26:25.261 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:26:25.261 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:26:25.549 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:26:26.551 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:26:26.551 INFO:tasks.workunit.client.0.smithi136.stdout:10 2024-09-15T21:26:26.551 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:26:26.552 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-15T21:26:26.552 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:26:26.552 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:26:26.839 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:26:27.840 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:26:27.840 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:26:27.840 INFO:tasks.workunit.client.0.smithi136.stdout:11 2024-09-15T21:26:27.841 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-15T21:26:27.841 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:26:27.841 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:26:28.268 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:26:29.266 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:26:29.266 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:26:29.267 INFO:tasks.workunit.client.0.smithi136.stdout:12 2024-09-15T21:26:29.267 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-15T21:26:29.267 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:26:29.267 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:26:29.846 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:26:30.847 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:26:30.847 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:26:30.848 INFO:tasks.workunit.client.0.smithi136.stdout:13 2024-09-15T21:26:30.848 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-15T21:26:30.848 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:26:30.848 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:26:31.644 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:26:32.645 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:26:32.645 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:26:32.646 INFO:tasks.workunit.client.0.smithi136.stdout:14 2024-09-15T21:26:32.646 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-15T21:26:32.646 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:26:32.646 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:26:33.146 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:26:34.147 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:26:34.147 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:26:34.148 INFO:tasks.workunit.client.0.smithi136.stdout:15 2024-09-15T21:26:34.148 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-15T21:26:34.148 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:26:34.148 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:26:34.802 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:26:35.803 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:26:35.804 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:26:35.804 INFO:tasks.workunit.client.0.smithi136.stdout:16 2024-09-15T21:26:35.804 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-15T21:26:35.805 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:26:35.805 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:26:36.125 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:26:37.127 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:26:37.127 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:26:37.128 INFO:tasks.workunit.client.0.smithi136.stdout:17 2024-09-15T21:26:37.128 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-15T21:26:37.128 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:26:37.128 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:26:38.020 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:26:39.021 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:26:39.021 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:26:39.022 INFO:tasks.workunit.client.0.smithi136.stdout:18 2024-09-15T21:26:39.022 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 18 2024-09-15T21:26:39.022 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:26:39.022 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:26:39.372 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:26:40.373 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:26:40.373 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:26:40.374 INFO:tasks.workunit.client.0.smithi136.stdout:19 2024-09-15T21:26:40.374 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 19 2024-09-15T21:26:40.374 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:26:40.374 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:26:40.669 INFO:tasks.workunit.client.0.smithi136.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/1293266940,v1:127.0.0.1:6803/1293266940] [v2:127.0.0.1:6804/1293266940,v1:127.0.0.1:6805/1293266940] exists,up 7a68df98-8e0f-45ef-a9e9-b2a45ca8fd3f 2024-09-15T21:26:40.673 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-15T21:26:40.673 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-15T21:26:40.673 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-15T21:26:40.673 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:998: TEST_corrupt_scrub_replicated: run_osd td/osd-scrub-repair 1 2024-09-15T21:26:40.673 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-scrub-repair 2024-09-15T21:26:40.673 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-15T21:26:40.673 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=1 2024-09-15T21:26:40.673 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-15T21:26:40.673 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-scrub-repair/1 2024-09-15T21:26:40.674 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=ce4892c7-6c6d-4b4f-8e6f-07a3e11133e5 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-15T21:26:40.674 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T21:26:40.674 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T21:26:40.674 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T21:26:40.674 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-09-15T21:26:40.674 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-09-15T21:26:40.674 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-15T21:26:40.674 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-15T21:26:40.674 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-15T21:26:40.674 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-15T21:26:40.674 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-15T21:26:40.674 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-15T21:26:40.674 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-15T21:26:40.674 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T21:26:40.675 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52602 2024-09-15T21:26:40.675 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.52602/$cluster-$name.asok' 2024-09-15T21:26:40.675 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.52602/$cluster-$name.asok' 2024-09-15T21:26:40.675 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-15T21:26:40.675 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-15T21:26:40.675 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-15T21:26:40.675 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-15T21:26:40.675 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-15T21:26:40.675 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T21:26:40.675 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T21:26:40.675 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T21:26:40.675 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T21:26:40.675 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-15T21:26:40.675 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-15T21:26:40.676 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-scrub-repair/1 2024-09-15T21:26:40.676 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-15T21:26:40.676 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=9aec06d1-33ff-42ab-be71-ac761d546d96 2024-09-15T21:26:40.676 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd1 9aec06d1-33ff-42ab-be71-ac761d546d96' 2024-09-15T21:26:40.676 INFO:tasks.workunit.client.0.smithi136.stdout:add osd1 9aec06d1-33ff-42ab-be71-ac761d546d96 2024-09-15T21:26:40.677 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-15T21:26:40.693 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQAQUedmg5opKRAA9ropkAIE7H8cr975YwZFRQ== 2024-09-15T21:26:40.693 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQAQUedmg5opKRAA9ropkAIE7H8cr975YwZFRQ=="}' 2024-09-15T21:26:40.693 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 9aec06d1-33ff-42ab-be71-ac761d546d96 -i td/osd-scrub-repair/1/new.json 2024-09-15T21:26:41.024 INFO:tasks.workunit.client.0.smithi136.stdout:1 2024-09-15T21:26:41.035 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-scrub-repair/1/new.json 2024-09-15T21:26:41.036 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 1 --fsid=ce4892c7-6c6d-4b4f-8e6f-07a3e11133e5 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.52602/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAQUedmg5opKRAA9ropkAIE7H8cr975YwZFRQ== --osd-uuid 9aec06d1-33ff-42ab-be71-ac761d546d96 2024-09-15T21:26:41.065 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:26:41.062+0000 7f177703e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:26:41.100 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:26:41.097+0000 7f177703e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:26:41.101 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:26:41.098+0000 7f177703e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:26:41.101 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:26:41.099+0000 7f177703e640 -1 bdev(0x55a74c957c00 td/osd-scrub-repair/1/block) open stat got: (1) Operation not permitted 2024-09-15T21:26:41.102 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:26:41.099+0000 7f177703e640 -1 bluestore(td/osd-scrub-repair/1) _read_fsid unparsable uuid 2024-09-15T21:26:47.042 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-scrub-repair/1/keyring 2024-09-15T21:26:47.043 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-15T21:26:47.044 INFO:tasks.workunit.client.0.smithi136.stdout:adding osd1 key to auth repository 2024-09-15T21:26:47.044 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd1 key to auth repository 2024-09-15T21:26:47.044 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-scrub-repair/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-15T21:26:47.444 INFO:tasks.workunit.client.0.smithi136.stdout:start osd.1 2024-09-15T21:26:47.445 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.1 2024-09-15T21:26:47.445 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 1 --fsid=ce4892c7-6c6d-4b4f-8e6f-07a3e11133e5 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.52602/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T21:26:47.445 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-15T21:26:47.445 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-15T21:26:47.445 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-15T21:26:47.471 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:26:47.468+0000 7f3610416640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:26:47.473 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:26:47.471+0000 7f3610416640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:26:47.474 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:26:47.472+0000 7f3610416640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:26:47.721 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 1 2024-09-15T21:26:47.721 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-15T21:26:47.721 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2024-09-15T21:26:47.721 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-15T21:26:47.721 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-15T21:26:47.721 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:26:47.722 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-15T21:26:47.722 INFO:tasks.workunit.client.0.smithi136.stdout:0 2024-09-15T21:26:47.722 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:26:47.722 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T21:26:47.997 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:26:48.828 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:26:48.825+0000 7f3610416640 -1 Falling back to public interface 2024-09-15T21:26:48.998 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:26:48.999 INFO:tasks.workunit.client.0.smithi136.stdout:1 2024-09-15T21:26:48.999 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:26:48.999 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-15T21:26:48.999 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:26:48.999 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T21:26:49.278 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:26:50.280 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:26:50.280 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:26:50.280 INFO:tasks.workunit.client.0.smithi136.stdout:2 2024-09-15T21:26:50.280 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-15T21:26:50.281 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:26:50.281 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T21:26:50.566 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:26:51.567 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:26:51.567 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:26:51.567 INFO:tasks.workunit.client.0.smithi136.stdout:3 2024-09-15T21:26:51.568 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-15T21:26:51.568 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:26:51.568 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T21:26:51.617 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:26:51.614+0000 7f3610416640 -1 osd.1 0 log_to_monitors true 2024-09-15T21:26:51.852 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:26:52.854 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:26:52.854 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:26:52.854 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-15T21:26:52.854 INFO:tasks.workunit.client.0.smithi136.stdout:4 2024-09-15T21:26:52.855 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:26:52.855 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T21:26:53.142 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:26:54.143 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:26:54.144 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:26:54.144 INFO:tasks.workunit.client.0.smithi136.stdout:5 2024-09-15T21:26:54.144 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-15T21:26:54.144 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:26:54.145 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T21:26:54.435 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:26:55.436 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:26:55.436 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:26:55.437 INFO:tasks.workunit.client.0.smithi136.stdout:6 2024-09-15T21:26:55.437 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-15T21:26:55.437 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:26:55.437 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T21:26:55.755 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:26:56.756 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:26:56.756 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:26:56.756 INFO:tasks.workunit.client.0.smithi136.stdout:7 2024-09-15T21:26:56.757 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-15T21:26:56.757 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:26:56.757 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T21:26:57.093 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:26:58.095 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:26:58.095 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:26:58.095 INFO:tasks.workunit.client.0.smithi136.stdout:8 2024-09-15T21:26:58.096 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-15T21:26:58.096 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:26:58.096 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T21:26:58.400 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:26:59.402 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:26:59.404 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:26:59.404 INFO:tasks.workunit.client.0.smithi136.stdout:9 2024-09-15T21:26:59.405 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-15T21:26:59.405 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:26:59.405 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T21:26:59.732 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:27:00.733 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:27:00.733 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:27:00.734 INFO:tasks.workunit.client.0.smithi136.stdout:10 2024-09-15T21:27:00.734 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-15T21:27:00.734 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:27:00.734 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T21:27:01.021 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:27:02.022 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:27:02.022 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:27:02.022 INFO:tasks.workunit.client.0.smithi136.stdout:11 2024-09-15T21:27:02.023 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-15T21:27:02.023 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:27:02.023 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T21:27:02.409 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:27:03.410 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:27:03.410 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:27:03.410 INFO:tasks.workunit.client.0.smithi136.stdout:12 2024-09-15T21:27:03.411 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-15T21:27:03.411 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:27:03.411 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T21:27:03.953 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:27:04.955 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:27:04.955 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:27:04.955 INFO:tasks.workunit.client.0.smithi136.stdout:13 2024-09-15T21:27:04.956 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-15T21:27:04.956 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:27:04.956 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T21:27:05.439 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:27:06.441 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:27:06.441 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:27:06.441 INFO:tasks.workunit.client.0.smithi136.stdout:14 2024-09-15T21:27:06.442 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-15T21:27:06.442 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:27:06.442 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T21:27:07.055 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:27:08.056 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:27:08.057 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:27:08.057 INFO:tasks.workunit.client.0.smithi136.stdout:15 2024-09-15T21:27:08.057 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-15T21:27:08.057 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:27:08.058 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T21:27:08.863 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:27:09.864 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:27:09.864 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:27:09.865 INFO:tasks.workunit.client.0.smithi136.stdout:16 2024-09-15T21:27:09.865 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-15T21:27:09.865 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:27:09.865 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T21:27:10.150 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:27:11.151 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:27:11.152 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:27:11.152 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-15T21:27:11.152 INFO:tasks.workunit.client.0.smithi136.stdout:17 2024-09-15T21:27:11.152 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:27:11.152 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T21:27:11.999 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:27:13.000 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:27:13.000 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:27:13.001 INFO:tasks.workunit.client.0.smithi136.stdout:18 2024-09-15T21:27:13.001 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 18 2024-09-15T21:27:13.001 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:27:13.001 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T21:27:13.427 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:27:14.428 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:27:14.428 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:27:14.428 INFO:tasks.workunit.client.0.smithi136.stdout:19 2024-09-15T21:27:14.429 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 19 2024-09-15T21:27:14.429 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:27:14.429 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T21:27:14.731 INFO:tasks.workunit.client.0.smithi136.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/3997038994,v1:127.0.0.1:6811/3997038994] [v2:127.0.0.1:6812/3997038994,v1:127.0.0.1:6813/3997038994] exists,up 9aec06d1-33ff-42ab-be71-ac761d546d96 2024-09-15T21:27:14.732 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-15T21:27:14.732 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-15T21:27:14.732 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-15T21:27:14.732 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:999: TEST_corrupt_scrub_replicated: create_rbd_pool 2024-09-15T21:27:14.733 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:526: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2024-09-15T21:27:14.998 INFO:tasks.workunit.client.0.smithi136.stderr:pool 'rbd' does not exist 2024-09-15T21:27:15.010 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:527: create_rbd_pool: create_pool rbd 4 2024-09-15T21:27:15.010 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create rbd 4 2024-09-15T21:27:15.387 INFO:tasks.workunit.client.0.smithi136.stderr:pool 'rbd' created 2024-09-15T21:27:15.398 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2024-09-15T21:27:16.400 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:528: create_rbd_pool: rbd pool init rbd 2024-09-15T21:27:17.913 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1000: TEST_corrupt_scrub_replicated: wait_for_clean 2024-09-15T21:27:17.913 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-15T21:27:17.913 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-15T21:27:17.913 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-15T21:27:17.914 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-15T21:27:17.914 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-15T21:27:17.914 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-15T21:27:17.914 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-15T21:27:17.915 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-15T21:27:17.915 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-15T21:27:18.018 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-15T21:27:18.018 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-15T21:27:18.018 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-15T21:27:18.018 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-15T21:27:18.018 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-15T21:27:18.018 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-15T21:27:18.294 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-15T21:27:18.294 INFO:tasks.workunit.client.0.smithi136.stderr:1' 2024-09-15T21:27:18.294 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-15T21:27:18.294 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T21:27:18.294 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-15T21:27:18.438 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836493 2024-09-15T21:27:18.438 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836493 2024-09-15T21:27:18.438 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836493' 2024-09-15T21:27:18.438 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T21:27:18.439 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-15T21:27:18.544 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=42949672965 2024-09-15T21:27:18.544 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 42949672965 2024-09-15T21:27:18.544 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836493 1-42949672965' 2024-09-15T21:27:18.544 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T21:27:18.544 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836493 2024-09-15T21:27:18.544 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T21:27:18.545 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-15T21:27:18.546 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836493 2024-09-15T21:27:18.546 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T21:27:18.547 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836493 2024-09-15T21:27:18.547 INFO:tasks.workunit.client.0.smithi136.stdout:waiting osd.0 seq 21474836493 2024-09-15T21:27:18.547 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836493' 2024-09-15T21:27:18.547 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T21:27:18.820 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836492 -lt 21474836493 2024-09-15T21:27:18.821 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-15T21:27:19.822 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-15T21:27:19.822 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T21:27:20.107 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836493 -lt 21474836493 2024-09-15T21:27:20.107 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T21:27:20.108 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-42949672965 2024-09-15T21:27:20.108 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T21:27:20.109 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-15T21:27:20.109 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-42949672965 2024-09-15T21:27:20.109 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T21:27:20.110 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=42949672965 2024-09-15T21:27:20.110 INFO:tasks.workunit.client.0.smithi136.stdout:waiting osd.1 seq 42949672965 2024-09-15T21:27:20.111 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 42949672965' 2024-09-15T21:27:20.111 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-15T21:27:20.387 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 42949672965 -lt 42949672965 2024-09-15T21:27:20.388 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-15T21:27:20.388 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T21:27:20.388 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T21:27:20.731 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 4 == 0 2024-09-15T21:27:20.731 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-15T21:27:20.732 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-15T21:27:20.732 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-15T21:27:20.732 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T21:27:20.732 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T21:27:20.732 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T21:27:20.733 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T21:27:21.002 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=4 2024-09-15T21:27:21.002 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-15T21:27:21.003 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T21:27:21.003 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T21:27:21.364 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 = 4 2024-09-15T21:27:21.365 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-15T21:27:21.365 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-15T21:27:21.365 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1002: TEST_corrupt_scrub_replicated: create_pool foo 1 2024-09-15T21:27:21.365 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create foo 1 2024-09-15T21:27:21.703 INFO:tasks.workunit.client.0.smithi136.stderr:pool 'foo' created 2024-09-15T21:27:21.716 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2024-09-15T21:27:22.716 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1003: TEST_corrupt_scrub_replicated: create_pool csr_pool 1 1 2024-09-15T21:27:22.716 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create csr_pool 1 1 2024-09-15T21:27:23.118 INFO:tasks.workunit.client.0.smithi136.stderr:pool 'csr_pool' created 2024-09-15T21:27:23.131 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2024-09-15T21:27:24.132 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1004: TEST_corrupt_scrub_replicated: wait_for_clean 2024-09-15T21:27:24.132 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-15T21:27:24.132 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-15T21:27:24.132 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-15T21:27:24.133 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-15T21:27:24.133 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-15T21:27:24.133 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-15T21:27:24.133 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-15T21:27:24.133 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-15T21:27:24.134 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-15T21:27:24.236 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-15T21:27:24.236 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-15T21:27:24.237 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-15T21:27:24.237 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-15T21:27:24.237 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-15T21:27:24.237 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-15T21:27:24.515 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-15T21:27:24.515 INFO:tasks.workunit.client.0.smithi136.stderr:1' 2024-09-15T21:27:24.515 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-15T21:27:24.516 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T21:27:24.516 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-15T21:27:24.620 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836495 2024-09-15T21:27:24.620 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836495 2024-09-15T21:27:24.620 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836495' 2024-09-15T21:27:24.620 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T21:27:24.620 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-15T21:27:24.725 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=42949672967 2024-09-15T21:27:24.725 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 42949672967 2024-09-15T21:27:24.725 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836495 1-42949672967' 2024-09-15T21:27:24.725 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T21:27:24.725 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836495 2024-09-15T21:27:24.725 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T21:27:24.725 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-15T21:27:24.726 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836495 2024-09-15T21:27:24.726 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T21:27:24.726 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836495 2024-09-15T21:27:24.726 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836495' 2024-09-15T21:27:24.727 INFO:tasks.workunit.client.0.smithi136.stdout:waiting osd.0 seq 21474836495 2024-09-15T21:27:24.727 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T21:27:25.004 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836494 -lt 21474836495 2024-09-15T21:27:25.004 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-15T21:27:26.005 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-15T21:27:26.005 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T21:27:26.291 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836496 -lt 21474836495 2024-09-15T21:27:26.291 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T21:27:26.292 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-42949672967 2024-09-15T21:27:26.292 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T21:27:26.293 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-15T21:27:26.294 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-42949672967 2024-09-15T21:27:26.294 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T21:27:26.295 INFO:tasks.workunit.client.0.smithi136.stdout:waiting osd.1 seq 42949672967 2024-09-15T21:27:26.295 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=42949672967 2024-09-15T21:27:26.295 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 42949672967' 2024-09-15T21:27:26.295 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-15T21:27:26.609 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 42949672967 -lt 42949672967 2024-09-15T21:27:26.609 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-15T21:27:26.610 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T21:27:26.610 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T21:27:26.954 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 6 == 0 2024-09-15T21:27:26.954 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-15T21:27:26.954 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-15T21:27:26.954 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-15T21:27:26.954 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T21:27:26.955 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T21:27:26.955 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T21:27:26.955 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T21:27:27.229 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=6 2024-09-15T21:27:27.229 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-15T21:27:27.229 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T21:27:27.229 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T21:27:27.574 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 = 6 2024-09-15T21:27:27.574 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-15T21:27:27.575 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-15T21:27:27.576 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1006: TEST_corrupt_scrub_replicated: seq 1 19 2024-09-15T21:27:27.576 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1006: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2024-09-15T21:27:27.576 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1007: TEST_corrupt_scrub_replicated: objname=ROBJ1 2024-09-15T21:27:27.576 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1008: TEST_corrupt_scrub_replicated: add_something td/osd-scrub-repair csr_pool ROBJ1 2024-09-15T21:27:27.576 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-09-15T21:27:27.577 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=csr_pool 2024-09-15T21:27:27.577 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=ROBJ1 2024-09-15T21:27:27.577 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-09-15T21:27:27.577 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-09-15T21:27:27.577 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-09-15T21:27:27.952 INFO:tasks.workunit.client.0.smithi136.stderr:noscrub is set 2024-09-15T21:27:27.963 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-09-15T21:27:28.454 INFO:tasks.workunit.client.0.smithi136.stderr:nodeep-scrub is set 2024-09-15T21:27:28.466 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-09-15T21:27:28.466 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-09-15T21:27:28.466 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool csr_pool put ROBJ1 td/osd-scrub-repair/ORIGINAL 2024-09-15T21:27:28.725 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1010: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapheader ROBJ1 hdr-ROBJ1 2024-09-15T21:27:28.842 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1011: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapval ROBJ1 key-ROBJ1 val-ROBJ1 2024-09-15T21:27:28.960 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1006: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2024-09-15T21:27:28.960 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1007: TEST_corrupt_scrub_replicated: objname=ROBJ2 2024-09-15T21:27:28.960 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1008: TEST_corrupt_scrub_replicated: add_something td/osd-scrub-repair csr_pool ROBJ2 2024-09-15T21:27:28.960 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-09-15T21:27:28.960 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=csr_pool 2024-09-15T21:27:28.960 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=ROBJ2 2024-09-15T21:27:28.960 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-09-15T21:27:28.960 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-09-15T21:27:28.960 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-09-15T21:27:29.350 INFO:tasks.workunit.client.0.smithi136.stderr:noscrub is set 2024-09-15T21:27:29.363 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-09-15T21:27:29.861 INFO:tasks.workunit.client.0.smithi136.stderr:nodeep-scrub is set 2024-09-15T21:27:29.872 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-09-15T21:27:29.872 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-09-15T21:27:29.872 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool csr_pool put ROBJ2 td/osd-scrub-repair/ORIGINAL 2024-09-15T21:27:30.109 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1010: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapheader ROBJ2 hdr-ROBJ2 2024-09-15T21:27:30.206 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1011: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapval ROBJ2 key-ROBJ2 val-ROBJ2 2024-09-15T21:27:30.316 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1006: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2024-09-15T21:27:30.316 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1007: TEST_corrupt_scrub_replicated: objname=ROBJ3 2024-09-15T21:27:30.316 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1008: TEST_corrupt_scrub_replicated: add_something td/osd-scrub-repair csr_pool ROBJ3 2024-09-15T21:27:30.316 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-09-15T21:27:30.317 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=csr_pool 2024-09-15T21:27:30.317 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=ROBJ3 2024-09-15T21:27:30.317 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-09-15T21:27:30.317 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-09-15T21:27:30.317 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-09-15T21:27:30.689 INFO:tasks.workunit.client.0.smithi136.stderr:noscrub is set 2024-09-15T21:27:30.700 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-09-15T21:27:31.184 INFO:tasks.workunit.client.0.smithi136.stderr:nodeep-scrub is set 2024-09-15T21:27:31.196 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-09-15T21:27:31.196 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-09-15T21:27:31.196 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool csr_pool put ROBJ3 td/osd-scrub-repair/ORIGINAL 2024-09-15T21:27:31.513 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1010: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapheader ROBJ3 hdr-ROBJ3 2024-09-15T21:27:31.630 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1011: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapval ROBJ3 key-ROBJ3 val-ROBJ3 2024-09-15T21:27:31.715 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1006: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2024-09-15T21:27:31.715 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1007: TEST_corrupt_scrub_replicated: objname=ROBJ4 2024-09-15T21:27:31.715 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1008: TEST_corrupt_scrub_replicated: add_something td/osd-scrub-repair csr_pool ROBJ4 2024-09-15T21:27:31.715 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-09-15T21:27:31.715 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=csr_pool 2024-09-15T21:27:31.715 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=ROBJ4 2024-09-15T21:27:31.715 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-09-15T21:27:31.715 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-09-15T21:27:31.715 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-09-15T21:27:32.121 INFO:tasks.workunit.client.0.smithi136.stderr:noscrub is set 2024-09-15T21:27:32.132 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-09-15T21:27:32.540 INFO:tasks.workunit.client.0.smithi136.stderr:nodeep-scrub is set 2024-09-15T21:27:32.552 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-09-15T21:27:32.552 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-09-15T21:27:32.552 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool csr_pool put ROBJ4 td/osd-scrub-repair/ORIGINAL 2024-09-15T21:27:32.786 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1010: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapheader ROBJ4 hdr-ROBJ4 2024-09-15T21:27:32.927 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1011: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapval ROBJ4 key-ROBJ4 val-ROBJ4 2024-09-15T21:27:33.046 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1006: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2024-09-15T21:27:33.046 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1007: TEST_corrupt_scrub_replicated: objname=ROBJ5 2024-09-15T21:27:33.046 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1008: TEST_corrupt_scrub_replicated: add_something td/osd-scrub-repair csr_pool ROBJ5 2024-09-15T21:27:33.046 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-09-15T21:27:33.046 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=csr_pool 2024-09-15T21:27:33.046 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=ROBJ5 2024-09-15T21:27:33.046 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-09-15T21:27:33.046 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-09-15T21:27:33.046 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-09-15T21:27:33.419 INFO:tasks.workunit.client.0.smithi136.stderr:noscrub is set 2024-09-15T21:27:33.431 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-09-15T21:27:33.949 INFO:tasks.workunit.client.0.smithi136.stderr:nodeep-scrub is set 2024-09-15T21:27:33.960 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-09-15T21:27:33.960 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-09-15T21:27:33.961 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool csr_pool put ROBJ5 td/osd-scrub-repair/ORIGINAL 2024-09-15T21:27:34.169 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1010: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapheader ROBJ5 hdr-ROBJ5 2024-09-15T21:27:34.303 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1011: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapval ROBJ5 key-ROBJ5 val-ROBJ5 2024-09-15T21:27:34.404 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1006: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2024-09-15T21:27:34.404 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1007: TEST_corrupt_scrub_replicated: objname=ROBJ6 2024-09-15T21:27:34.405 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1008: TEST_corrupt_scrub_replicated: add_something td/osd-scrub-repair csr_pool ROBJ6 2024-09-15T21:27:34.405 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-09-15T21:27:34.405 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=csr_pool 2024-09-15T21:27:34.405 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=ROBJ6 2024-09-15T21:27:34.405 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-09-15T21:27:34.405 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-09-15T21:27:34.405 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-09-15T21:27:34.777 INFO:tasks.workunit.client.0.smithi136.stderr:noscrub is set 2024-09-15T21:27:34.789 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-09-15T21:27:35.221 INFO:tasks.workunit.client.0.smithi136.stderr:nodeep-scrub is set 2024-09-15T21:27:35.233 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-09-15T21:27:35.233 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-09-15T21:27:35.233 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool csr_pool put ROBJ6 td/osd-scrub-repair/ORIGINAL 2024-09-15T21:27:35.482 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1010: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapheader ROBJ6 hdr-ROBJ6 2024-09-15T21:27:35.692 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1011: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapval ROBJ6 key-ROBJ6 val-ROBJ6 2024-09-15T21:27:35.818 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1006: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2024-09-15T21:27:35.818 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1007: TEST_corrupt_scrub_replicated: objname=ROBJ7 2024-09-15T21:27:35.818 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1008: TEST_corrupt_scrub_replicated: add_something td/osd-scrub-repair csr_pool ROBJ7 2024-09-15T21:27:35.818 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-09-15T21:27:35.818 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=csr_pool 2024-09-15T21:27:35.819 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=ROBJ7 2024-09-15T21:27:35.819 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-09-15T21:27:35.819 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-09-15T21:27:35.819 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-09-15T21:27:36.182 INFO:tasks.workunit.client.0.smithi136.stderr:noscrub is set 2024-09-15T21:27:36.194 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-09-15T21:27:36.676 INFO:tasks.workunit.client.0.smithi136.stderr:nodeep-scrub is set 2024-09-15T21:27:36.688 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-09-15T21:27:36.688 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-09-15T21:27:36.689 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool csr_pool put ROBJ7 td/osd-scrub-repair/ORIGINAL 2024-09-15T21:27:36.930 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1010: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapheader ROBJ7 hdr-ROBJ7 2024-09-15T21:27:37.181 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1011: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapval ROBJ7 key-ROBJ7 val-ROBJ7 2024-09-15T21:27:37.324 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1006: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2024-09-15T21:27:37.324 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1007: TEST_corrupt_scrub_replicated: objname=ROBJ8 2024-09-15T21:27:37.324 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1008: TEST_corrupt_scrub_replicated: add_something td/osd-scrub-repair csr_pool ROBJ8 2024-09-15T21:27:37.325 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-09-15T21:27:37.325 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=csr_pool 2024-09-15T21:27:37.325 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=ROBJ8 2024-09-15T21:27:37.325 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-09-15T21:27:37.325 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-09-15T21:27:37.325 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-09-15T21:27:37.696 INFO:tasks.workunit.client.0.smithi136.stderr:noscrub is set 2024-09-15T21:27:37.707 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-09-15T21:27:38.190 INFO:tasks.workunit.client.0.smithi136.stderr:nodeep-scrub is set 2024-09-15T21:27:38.202 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-09-15T21:27:38.203 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-09-15T21:27:38.203 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool csr_pool put ROBJ8 td/osd-scrub-repair/ORIGINAL 2024-09-15T21:27:38.419 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1010: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapheader ROBJ8 hdr-ROBJ8 2024-09-15T21:27:38.511 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1011: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapval ROBJ8 key-ROBJ8 val-ROBJ8 2024-09-15T21:27:38.611 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1006: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2024-09-15T21:27:38.612 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1007: TEST_corrupt_scrub_replicated: objname=ROBJ9 2024-09-15T21:27:38.612 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1008: TEST_corrupt_scrub_replicated: add_something td/osd-scrub-repair csr_pool ROBJ9 2024-09-15T21:27:38.612 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-09-15T21:27:38.612 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=csr_pool 2024-09-15T21:27:38.612 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=ROBJ9 2024-09-15T21:27:38.612 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-09-15T21:27:38.612 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-09-15T21:27:38.612 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-09-15T21:27:38.976 INFO:tasks.workunit.client.0.smithi136.stderr:noscrub is set 2024-09-15T21:27:38.987 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-09-15T21:27:39.386 INFO:tasks.workunit.client.0.smithi136.stderr:nodeep-scrub is set 2024-09-15T21:27:39.398 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-09-15T21:27:39.398 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-09-15T21:27:39.398 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool csr_pool put ROBJ9 td/osd-scrub-repair/ORIGINAL 2024-09-15T21:27:39.614 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1010: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapheader ROBJ9 hdr-ROBJ9 2024-09-15T21:27:39.799 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1011: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapval ROBJ9 key-ROBJ9 val-ROBJ9 2024-09-15T21:27:39.910 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1006: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2024-09-15T21:27:39.910 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1007: TEST_corrupt_scrub_replicated: objname=ROBJ10 2024-09-15T21:27:39.910 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1008: TEST_corrupt_scrub_replicated: add_something td/osd-scrub-repair csr_pool ROBJ10 2024-09-15T21:27:39.910 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-09-15T21:27:39.910 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=csr_pool 2024-09-15T21:27:39.910 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=ROBJ10 2024-09-15T21:27:39.910 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-09-15T21:27:39.910 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-09-15T21:27:39.910 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-09-15T21:27:40.282 INFO:tasks.workunit.client.0.smithi136.stderr:noscrub is set 2024-09-15T21:27:40.294 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-09-15T21:27:40.767 INFO:tasks.workunit.client.0.smithi136.stderr:nodeep-scrub is set 2024-09-15T21:27:40.780 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-09-15T21:27:40.781 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-09-15T21:27:40.781 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool csr_pool put ROBJ10 td/osd-scrub-repair/ORIGINAL 2024-09-15T21:27:41.004 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1010: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapheader ROBJ10 hdr-ROBJ10 2024-09-15T21:27:41.138 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1011: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapval ROBJ10 key-ROBJ10 val-ROBJ10 2024-09-15T21:27:41.299 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1006: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2024-09-15T21:27:41.299 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1007: TEST_corrupt_scrub_replicated: objname=ROBJ11 2024-09-15T21:27:41.299 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1008: TEST_corrupt_scrub_replicated: add_something td/osd-scrub-repair csr_pool ROBJ11 2024-09-15T21:27:41.299 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-09-15T21:27:41.299 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=csr_pool 2024-09-15T21:27:41.299 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=ROBJ11 2024-09-15T21:27:41.300 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-09-15T21:27:41.300 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-09-15T21:27:41.300 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-09-15T21:27:41.670 INFO:tasks.workunit.client.0.smithi136.stderr:noscrub is set 2024-09-15T21:27:41.682 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-09-15T21:27:42.265 INFO:tasks.workunit.client.0.smithi136.stderr:nodeep-scrub is set 2024-09-15T21:27:42.277 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-09-15T21:27:42.277 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-09-15T21:27:42.277 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool csr_pool put ROBJ11 td/osd-scrub-repair/ORIGINAL 2024-09-15T21:27:42.493 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1010: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapheader ROBJ11 hdr-ROBJ11 2024-09-15T21:27:42.601 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1011: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapval ROBJ11 key-ROBJ11 val-ROBJ11 2024-09-15T21:27:42.694 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1006: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2024-09-15T21:27:42.695 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1007: TEST_corrupt_scrub_replicated: objname=ROBJ12 2024-09-15T21:27:42.695 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1008: TEST_corrupt_scrub_replicated: add_something td/osd-scrub-repair csr_pool ROBJ12 2024-09-15T21:27:42.695 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-09-15T21:27:42.695 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=csr_pool 2024-09-15T21:27:42.695 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=ROBJ12 2024-09-15T21:27:42.695 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-09-15T21:27:42.695 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-09-15T21:27:42.695 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-09-15T21:27:43.068 INFO:tasks.workunit.client.0.smithi136.stderr:noscrub is set 2024-09-15T21:27:43.079 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-09-15T21:27:43.453 INFO:tasks.workunit.client.0.smithi136.stderr:nodeep-scrub is set 2024-09-15T21:27:43.464 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-09-15T21:27:43.464 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-09-15T21:27:43.465 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool csr_pool put ROBJ12 td/osd-scrub-repair/ORIGINAL 2024-09-15T21:27:43.697 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1010: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapheader ROBJ12 hdr-ROBJ12 2024-09-15T21:27:43.885 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1011: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapval ROBJ12 key-ROBJ12 val-ROBJ12 2024-09-15T21:27:44.008 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1006: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2024-09-15T21:27:44.008 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1007: TEST_corrupt_scrub_replicated: objname=ROBJ13 2024-09-15T21:27:44.009 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1008: TEST_corrupt_scrub_replicated: add_something td/osd-scrub-repair csr_pool ROBJ13 2024-09-15T21:27:44.009 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-09-15T21:27:44.009 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=csr_pool 2024-09-15T21:27:44.009 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=ROBJ13 2024-09-15T21:27:44.009 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-09-15T21:27:44.009 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-09-15T21:27:44.009 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-09-15T21:27:44.381 INFO:tasks.workunit.client.0.smithi136.stderr:noscrub is set 2024-09-15T21:27:44.393 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-09-15T21:27:44.883 INFO:tasks.workunit.client.0.smithi136.stderr:nodeep-scrub is set 2024-09-15T21:27:44.895 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-09-15T21:27:44.895 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-09-15T21:27:44.895 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool csr_pool put ROBJ13 td/osd-scrub-repair/ORIGINAL 2024-09-15T21:27:45.086 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1010: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapheader ROBJ13 hdr-ROBJ13 2024-09-15T21:27:45.228 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1011: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapval ROBJ13 key-ROBJ13 val-ROBJ13 2024-09-15T21:27:45.329 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1006: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2024-09-15T21:27:45.330 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1007: TEST_corrupt_scrub_replicated: objname=ROBJ14 2024-09-15T21:27:45.330 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1008: TEST_corrupt_scrub_replicated: add_something td/osd-scrub-repair csr_pool ROBJ14 2024-09-15T21:27:45.330 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-09-15T21:27:45.330 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=csr_pool 2024-09-15T21:27:45.330 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=ROBJ14 2024-09-15T21:27:45.330 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-09-15T21:27:45.330 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-09-15T21:27:45.330 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-09-15T21:27:45.703 INFO:tasks.workunit.client.0.smithi136.stderr:noscrub is set 2024-09-15T21:27:45.714 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-09-15T21:27:46.248 INFO:tasks.workunit.client.0.smithi136.stderr:nodeep-scrub is set 2024-09-15T21:27:46.260 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-09-15T21:27:46.261 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-09-15T21:27:46.261 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool csr_pool put ROBJ14 td/osd-scrub-repair/ORIGINAL 2024-09-15T21:27:46.475 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1010: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapheader ROBJ14 hdr-ROBJ14 2024-09-15T21:27:46.635 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1011: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapval ROBJ14 key-ROBJ14 val-ROBJ14 2024-09-15T21:27:46.727 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1006: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2024-09-15T21:27:46.728 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1007: TEST_corrupt_scrub_replicated: objname=ROBJ15 2024-09-15T21:27:46.728 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1008: TEST_corrupt_scrub_replicated: add_something td/osd-scrub-repair csr_pool ROBJ15 2024-09-15T21:27:46.728 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-09-15T21:27:46.728 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=csr_pool 2024-09-15T21:27:46.728 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=ROBJ15 2024-09-15T21:27:46.728 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-09-15T21:27:46.728 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-09-15T21:27:46.728 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-09-15T21:27:47.125 INFO:tasks.workunit.client.0.smithi136.stderr:noscrub is set 2024-09-15T21:27:47.137 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-09-15T21:27:47.569 INFO:tasks.workunit.client.0.smithi136.stderr:nodeep-scrub is set 2024-09-15T21:27:47.581 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-09-15T21:27:47.581 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-09-15T21:27:47.581 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool csr_pool put ROBJ15 td/osd-scrub-repair/ORIGINAL 2024-09-15T21:27:47.832 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1010: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapheader ROBJ15 hdr-ROBJ15 2024-09-15T21:27:47.949 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1011: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapval ROBJ15 key-ROBJ15 val-ROBJ15 2024-09-15T21:27:48.067 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1006: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2024-09-15T21:27:48.067 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1007: TEST_corrupt_scrub_replicated: objname=ROBJ16 2024-09-15T21:27:48.068 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1008: TEST_corrupt_scrub_replicated: add_something td/osd-scrub-repair csr_pool ROBJ16 2024-09-15T21:27:48.068 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-09-15T21:27:48.068 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=csr_pool 2024-09-15T21:27:48.068 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=ROBJ16 2024-09-15T21:27:48.068 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-09-15T21:27:48.068 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-09-15T21:27:48.068 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-09-15T21:27:48.431 INFO:tasks.workunit.client.0.smithi136.stderr:noscrub is set 2024-09-15T21:27:48.443 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-09-15T21:27:48.942 INFO:tasks.workunit.client.0.smithi136.stderr:nodeep-scrub is set 2024-09-15T21:27:48.954 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-09-15T21:27:48.954 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-09-15T21:27:48.954 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool csr_pool put ROBJ16 td/osd-scrub-repair/ORIGINAL 2024-09-15T21:27:49.179 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1010: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapheader ROBJ16 hdr-ROBJ16 2024-09-15T21:27:49.288 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1011: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapval ROBJ16 key-ROBJ16 val-ROBJ16 2024-09-15T21:27:49.380 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1006: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2024-09-15T21:27:49.380 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1007: TEST_corrupt_scrub_replicated: objname=ROBJ17 2024-09-15T21:27:49.380 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1008: TEST_corrupt_scrub_replicated: add_something td/osd-scrub-repair csr_pool ROBJ17 2024-09-15T21:27:49.380 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-09-15T21:27:49.381 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=csr_pool 2024-09-15T21:27:49.381 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=ROBJ17 2024-09-15T21:27:49.381 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-09-15T21:27:49.381 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-09-15T21:27:49.381 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-09-15T21:27:49.753 INFO:tasks.workunit.client.0.smithi136.stderr:noscrub is set 2024-09-15T21:27:49.765 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-09-15T21:27:50.271 INFO:tasks.workunit.client.0.smithi136.stderr:nodeep-scrub is set 2024-09-15T21:27:50.283 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-09-15T21:27:50.284 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-09-15T21:27:50.284 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool csr_pool put ROBJ17 td/osd-scrub-repair/ORIGINAL 2024-09-15T21:27:50.508 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1010: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapheader ROBJ17 hdr-ROBJ17 2024-09-15T21:27:50.618 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1011: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapval ROBJ17 key-ROBJ17 val-ROBJ17 2024-09-15T21:27:50.710 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1006: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2024-09-15T21:27:50.710 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1007: TEST_corrupt_scrub_replicated: objname=ROBJ18 2024-09-15T21:27:50.711 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1008: TEST_corrupt_scrub_replicated: add_something td/osd-scrub-repair csr_pool ROBJ18 2024-09-15T21:27:50.711 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-09-15T21:27:50.711 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=csr_pool 2024-09-15T21:27:50.711 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=ROBJ18 2024-09-15T21:27:50.711 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-09-15T21:27:50.711 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-09-15T21:27:50.711 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-09-15T21:27:51.133 INFO:tasks.workunit.client.0.smithi136.stderr:noscrub is set 2024-09-15T21:27:51.145 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-09-15T21:27:51.568 INFO:tasks.workunit.client.0.smithi136.stderr:nodeep-scrub is set 2024-09-15T21:27:51.580 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-09-15T21:27:51.580 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-09-15T21:27:51.580 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool csr_pool put ROBJ18 td/osd-scrub-repair/ORIGINAL 2024-09-15T21:27:51.939 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1010: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapheader ROBJ18 hdr-ROBJ18 2024-09-15T21:27:52.064 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1011: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapval ROBJ18 key-ROBJ18 val-ROBJ18 2024-09-15T21:27:52.165 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1006: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2024-09-15T21:27:52.165 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1007: TEST_corrupt_scrub_replicated: objname=ROBJ19 2024-09-15T21:27:52.165 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1008: TEST_corrupt_scrub_replicated: add_something td/osd-scrub-repair csr_pool ROBJ19 2024-09-15T21:27:52.166 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-09-15T21:27:52.166 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=csr_pool 2024-09-15T21:27:52.166 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=ROBJ19 2024-09-15T21:27:52.166 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-09-15T21:27:52.166 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-09-15T21:27:52.166 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-09-15T21:27:52.521 INFO:tasks.workunit.client.0.smithi136.stderr:noscrub is set 2024-09-15T21:27:52.532 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-09-15T21:27:53.023 INFO:tasks.workunit.client.0.smithi136.stderr:nodeep-scrub is set 2024-09-15T21:27:53.035 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-09-15T21:27:53.035 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-09-15T21:27:53.035 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool csr_pool put ROBJ19 td/osd-scrub-repair/ORIGINAL 2024-09-15T21:27:53.335 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1010: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapheader ROBJ19 hdr-ROBJ19 2024-09-15T21:27:53.444 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1011: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapval ROBJ19 key-ROBJ19 val-ROBJ19 2024-09-15T21:27:53.612 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1015: TEST_corrupt_scrub_replicated: dd if=/dev/zero of=td/osd-scrub-repair/new.ROBJ19 bs=1024 count=1025 2024-09-15T21:27:53.614 INFO:tasks.workunit.client.0.smithi136.stderr:1025+0 records in 2024-09-15T21:27:53.615 INFO:tasks.workunit.client.0.smithi136.stderr:1025+0 records out 2024-09-15T21:27:53.615 INFO:tasks.workunit.client.0.smithi136.stderr:1049600 bytes (1.0 MB, 1.0 MiB) copied, 0.00190845 s, 550 MB/s 2024-09-15T21:27:53.615 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1016: TEST_corrupt_scrub_replicated: rados --pool csr_pool put ROBJ19 td/osd-scrub-repair/new.ROBJ19 2024-09-15T21:27:53.796 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1017: TEST_corrupt_scrub_replicated: rm -f td/osd-scrub-repair/new.ROBJ19 2024-09-15T21:27:53.798 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1019: TEST_corrupt_scrub_replicated: get_pg csr_pool ROBJ0 2024-09-15T21:27:53.798 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1084: get_pg: local poolname=csr_pool 2024-09-15T21:27:53.798 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1085: get_pg: local objectname=ROBJ0 2024-09-15T21:27:53.798 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1087: get_pg: ceph --format json osd map csr_pool ROBJ0 2024-09-15T21:27:53.798 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1087: get_pg: jq -r .pgid 2024-09-15T21:27:54.074 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1019: TEST_corrupt_scrub_replicated: local pg=3.0 2024-09-15T21:27:54.075 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1020: TEST_corrupt_scrub_replicated: get_primary csr_pool ROBJ0 2024-09-15T21:27:54.075 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1187: get_primary: local poolname=csr_pool 2024-09-15T21:27:54.075 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1188: get_primary: local objectname=ROBJ0 2024-09-15T21:27:54.075 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1190: get_primary: ceph --format json osd map csr_pool ROBJ0 2024-09-15T21:27:54.075 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1191: get_primary: jq .acting_primary 2024-09-15T21:27:54.352 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1020: TEST_corrupt_scrub_replicated: local primary=1 2024-09-15T21:27:54.353 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1023: TEST_corrupt_scrub_replicated: get_asok_path osd.0 2024-09-15T21:27:54.353 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.0 2024-09-15T21:27:54.354 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.0 ']' 2024-09-15T21:27:54.354 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-15T21:27:54.354 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T21:27:54.354 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52602 2024-09-15T21:27:54.354 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.52602/ceph-osd.0.asok 2024-09-15T21:27:54.354 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1023: TEST_corrupt_scrub_replicated: CEPH_ARGS= 2024-09-15T21:27:54.354 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1023: TEST_corrupt_scrub_replicated: ceph daemon /tmp/ceph-asok.52602/ceph-osd.0.asok config set osd_deep_scrub_update_digest_min_age 0 2024-09-15T21:27:54.431 INFO:tasks.workunit.client.0.smithi136.stdout:{ 2024-09-15T21:27:54.431 INFO:tasks.workunit.client.0.smithi136.stdout: "success": "osd_deep_scrub_update_digest_min_age = '' (not observed, change may require restart) " 2024-09-15T21:27:54.431 INFO:tasks.workunit.client.0.smithi136.stdout:} 2024-09-15T21:27:54.440 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1025: TEST_corrupt_scrub_replicated: get_asok_path osd.1 2024-09-15T21:27:54.440 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.1 2024-09-15T21:27:54.440 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.1 ']' 2024-09-15T21:27:54.440 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-15T21:27:54.440 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T21:27:54.440 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52602 2024-09-15T21:27:54.440 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.52602/ceph-osd.1.asok 2024-09-15T21:27:54.440 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1025: TEST_corrupt_scrub_replicated: CEPH_ARGS= 2024-09-15T21:27:54.441 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1025: TEST_corrupt_scrub_replicated: ceph daemon /tmp/ceph-asok.52602/ceph-osd.1.asok config set osd_deep_scrub_update_digest_min_age 0 2024-09-15T21:27:54.518 INFO:tasks.workunit.client.0.smithi136.stdout:{ 2024-09-15T21:27:54.518 INFO:tasks.workunit.client.0.smithi136.stdout: "success": "osd_deep_scrub_update_digest_min_age = '' (not observed, change may require restart) " 2024-09-15T21:27:54.518 INFO:tasks.workunit.client.0.smithi136.stdout:} 2024-09-15T21:27:54.526 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1027: TEST_corrupt_scrub_replicated: pg_deep_scrub 3.0 2024-09-15T21:27:54.526 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1890: pg_deep_scrub: local pgid=3.0 2024-09-15T21:27:54.527 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1891: pg_deep_scrub: get_last_scrub_stamp 3.0 last_deep_scrub_stamp 2024-09-15T21:27:54.527 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=3.0 2024-09-15T21:27:54.527 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_deep_scrub_stamp 2024-09-15T21:27:54.527 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-15T21:27:54.527 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_deep_scrub_stamp' 2024-09-15T21:27:54.800 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1891: pg_deep_scrub: local last_scrub=2024-09-15T21:27:23.046624+0000 2024-09-15T21:27:54.800 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1892: pg_deep_scrub: ceph pg deep-scrub 3.0 2024-09-15T21:27:55.060 INFO:tasks.workunit.client.0.smithi136.stderr:instructing pg 3.0 on osd.1 to deep-scrub 2024-09-15T21:27:55.072 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1893: pg_deep_scrub: wait_for_scrub 3.0 2024-09-15T21:27:23.046624+0000 last_deep_scrub_stamp 2024-09-15T21:27:55.072 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: local pgid=3.0 2024-09-15T21:27:55.072 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2018: wait_for_scrub: local last_scrub=2024-09-15T21:27:23.046624+0000 2024-09-15T21:27:55.072 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2019: wait_for_scrub: local sname=last_deep_scrub_stamp 2024-09-15T21:27:55.073 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i=0 )) 2024-09-15T21:27:55.073 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-15T21:27:55.073 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 3.0 last_deep_scrub_stamp 2024-09-15T21:27:55.073 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=3.0 2024-09-15T21:27:55.073 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_deep_scrub_stamp 2024-09-15T21:27:55.073 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-15T21:27:55.073 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_deep_scrub_stamp' 2024-09-15T21:27:55.366 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-15T21:27:23.046624+0000 '>' 2024-09-15T21:27:23.046624+0000 2024-09-15T21:27:55.367 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-15T21:27:56.368 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-15T21:27:56.368 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-15T21:27:56.369 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 3.0 last_deep_scrub_stamp 2024-09-15T21:27:56.369 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=3.0 2024-09-15T21:27:56.369 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_deep_scrub_stamp 2024-09-15T21:27:56.369 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-15T21:27:56.369 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_deep_scrub_stamp' 2024-09-15T21:27:56.649 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-15T21:27:23.046624+0000 '>' 2024-09-15T21:27:23.046624+0000 2024-09-15T21:27:56.649 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-15T21:27:57.650 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-15T21:27:57.651 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-15T21:27:57.651 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 3.0 last_deep_scrub_stamp 2024-09-15T21:27:57.651 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=3.0 2024-09-15T21:27:57.651 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_deep_scrub_stamp 2024-09-15T21:27:57.652 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-15T21:27:57.652 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_deep_scrub_stamp' 2024-09-15T21:27:57.932 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-15T21:27:23.046624+0000 '>' 2024-09-15T21:27:23.046624+0000 2024-09-15T21:27:57.932 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-15T21:27:58.933 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-15T21:27:58.934 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-15T21:27:58.934 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 3.0 last_deep_scrub_stamp 2024-09-15T21:27:58.934 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=3.0 2024-09-15T21:27:58.934 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_deep_scrub_stamp 2024-09-15T21:27:58.934 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-15T21:27:58.935 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_deep_scrub_stamp' 2024-09-15T21:27:59.217 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-15T21:27:23.046624+0000 '>' 2024-09-15T21:27:23.046624+0000 2024-09-15T21:27:59.217 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-15T21:28:00.218 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-15T21:28:00.218 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-15T21:28:00.219 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 3.0 last_deep_scrub_stamp 2024-09-15T21:28:00.219 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=3.0 2024-09-15T21:28:00.219 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_deep_scrub_stamp 2024-09-15T21:28:00.219 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-15T21:28:00.219 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_deep_scrub_stamp' 2024-09-15T21:28:00.501 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-15T21:27:23.046624+0000 '>' 2024-09-15T21:27:23.046624+0000 2024-09-15T21:28:00.501 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-15T21:28:01.502 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-15T21:28:01.502 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-15T21:28:01.503 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 3.0 last_deep_scrub_stamp 2024-09-15T21:28:01.503 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=3.0 2024-09-15T21:28:01.503 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_deep_scrub_stamp 2024-09-15T21:28:01.503 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-15T21:28:01.503 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_deep_scrub_stamp' 2024-09-15T21:28:01.788 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-15T21:27:56.080190+0000 '>' 2024-09-15T21:27:23.046624+0000 2024-09-15T21:28:01.788 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2023: wait_for_scrub: return 0 2024-09-15T21:28:01.788 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1029: TEST_corrupt_scrub_replicated: seq 1 19 2024-09-15T21:28:01.789 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1029: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2024-09-15T21:28:01.790 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1030: TEST_corrupt_scrub_replicated: objname=ROBJ1 2024-09-15T21:28:01.790 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1033: TEST_corrupt_scrub_replicated: expr 1 % 2 2024-09-15T21:28:01.792 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1033: TEST_corrupt_scrub_replicated: local osd=1 2024-09-15T21:28:01.792 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1035: TEST_corrupt_scrub_replicated: case $i in 2024-09-15T21:28:01.792 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1038: TEST_corrupt_scrub_replicated: local payload=UVWXYZZZ 2024-09-15T21:28:01.792 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1039: TEST_corrupt_scrub_replicated: echo UVWXYZZZ 2024-09-15T21:28:01.792 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1040: TEST_corrupt_scrub_replicated: objectstore_tool td/osd-scrub-repair 1 ROBJ1 set-bytes td/osd-scrub-repair/CORRUPT 2024-09-15T21:28:01.793 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2024-09-15T21:28:01.793 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2024-09-15T21:28:01.793 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=1 2024-09-15T21:28:01.793 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2024-09-15T21:28:01.793 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 ROBJ1 set-bytes td/osd-scrub-repair/CORRUPT 2024-09-15T21:28:01.793 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-09-15T21:28:01.793 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2024-09-15T21:28:01.793 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=1 2024-09-15T21:28:01.793 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2024-09-15T21:28:01.793 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-09-15T21:28:01.793 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-15T21:28:01.793 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-15T21:28:01.793 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-15T21:28:01.793 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-15T21:28:01.794 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-15T21:28:02.098 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-15T21:28:02.099 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 ROBJ1 set-bytes td/osd-scrub-repair/CORRUPT 2024-09-15T21:28:02.099 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-15T21:28:02.099 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-15T21:28:02.099 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=1 2024-09-15T21:28:02.099 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-15T21:28:02.099 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-09-15T21:28:02.099 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 ROBJ1 set-bytes td/osd-scrub-repair/CORRUPT 2024-09-15T21:28:03.641 INFO:tasks.workunit.client.0.smithi136.stderr:Error getting attr on : 3.0_head,#-5:00000000:::scrub_3.0:head#, (61) No data available 2024-09-15T21:28:04.202 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2024-09-15T21:28:04.202 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2024-09-15T21:28:04.203 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-15T21:28:04.203 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=1 2024-09-15T21:28:04.203 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-15T21:28:04.203 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-09-15T21:28:04.203 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=ce4892c7-6c6d-4b4f-8e6f-07a3e11133e5 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-15T21:28:04.203 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T21:28:04.203 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T21:28:04.203 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T21:28:04.203 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-09-15T21:28:04.203 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-09-15T21:28:04.203 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-15T21:28:04.203 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-15T21:28:04.204 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-15T21:28:04.204 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-15T21:28:04.204 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-15T21:28:04.204 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-15T21:28:04.204 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-15T21:28:04.204 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T21:28:04.204 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52602 2024-09-15T21:28:04.204 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.52602/$cluster-$name.asok' 2024-09-15T21:28:04.204 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.52602/$cluster-$name.asok' 2024-09-15T21:28:04.204 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-15T21:28:04.204 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-15T21:28:04.204 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-15T21:28:04.205 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T21:28:04.205 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T21:28:04.205 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T21:28:04.205 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T21:28:04.205 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-15T21:28:04.205 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-15T21:28:04.205 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-09-15T21:28:04.206 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.1 2024-09-15T21:28:04.206 INFO:tasks.workunit.client.0.smithi136.stderr:start osd.1 2024-09-15T21:28:04.206 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 1 --fsid=ce4892c7-6c6d-4b4f-8e6f-07a3e11133e5 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.52602/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T21:28:04.206 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-09-15T21:28:04.206 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 1 = 1 ']' 2024-09-15T21:28:04.206 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-15T21:28:04.207 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-15T21:28:04.207 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-15T21:28:04.233 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:28:04.229+0000 7f1b7100b640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:28:04.236 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:28:04.232+0000 7f1b7100b640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:28:04.237 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:28:04.233+0000 7f1b7100b640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:28:04.491 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 1 2024-09-15T21:28:04.491 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-15T21:28:04.491 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2024-09-15T21:28:04.492 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-15T21:28:04.492 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-15T21:28:04.492 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:28:04.492 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-15T21:28:04.492 INFO:tasks.workunit.client.0.smithi136.stderr:0 2024-09-15T21:28:04.493 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:28:04.493 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T21:28:04.768 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:28:05.594 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:28:05.590+0000 7f1b7100b640 -1 Falling back to public interface 2024-09-15T21:28:05.769 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:28:05.769 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:28:05.769 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-15T21:28:05.769 INFO:tasks.workunit.client.0.smithi136.stderr:1 2024-09-15T21:28:05.769 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:28:05.770 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T21:28:06.058 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:28:07.060 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:28:07.060 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:28:07.060 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-15T21:28:07.060 INFO:tasks.workunit.client.0.smithi136.stderr:2 2024-09-15T21:28:07.060 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:28:07.060 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T21:28:07.347 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:28:08.348 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:28:08.348 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:28:08.348 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-15T21:28:08.348 INFO:tasks.workunit.client.0.smithi136.stderr:3 2024-09-15T21:28:08.348 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:28:08.348 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T21:28:08.472 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:28:08.468+0000 7f1b7100b640 -1 osd.1 60 log_to_monitors true 2024-09-15T21:28:08.685 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:28:09.687 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:28:09.687 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:28:09.687 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-15T21:28:09.687 INFO:tasks.workunit.client.0.smithi136.stderr:4 2024-09-15T21:28:09.688 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:28:09.688 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T21:28:10.010 INFO:tasks.workunit.client.0.smithi136.stderr:osd.1 up in weight 1 up_from 64 up_thru 64 down_at 61 last_clean_interval [10,60) [v2:127.0.0.1:6810/973952255,v1:127.0.0.1:6811/973952255] [v2:127.0.0.1:6812/973952255,v1:127.0.0.1:6813/973952255] exists,up 9aec06d1-33ff-42ab-be71-ac761d546d96 2024-09-15T21:28:10.010 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-15T21:28:10.010 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-15T21:28:10.010 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-15T21:28:10.010 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2024-09-15T21:28:10.010 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-15T21:28:10.011 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-15T21:28:10.011 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-15T21:28:10.011 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-15T21:28:10.011 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-15T21:28:10.011 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-15T21:28:10.011 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-15T21:28:10.011 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-15T21:28:10.011 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-15T21:28:10.103 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-15T21:28:10.103 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-15T21:28:10.103 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-15T21:28:10.103 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-15T21:28:10.104 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-15T21:28:10.104 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-15T21:28:10.381 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-15T21:28:10.381 INFO:tasks.workunit.client.0.smithi136.stderr:1' 2024-09-15T21:28:10.381 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-15T21:28:10.381 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T21:28:10.382 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-15T21:28:10.484 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836506 2024-09-15T21:28:10.485 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836506 2024-09-15T21:28:10.485 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836506' 2024-09-15T21:28:10.485 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T21:28:10.485 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-15T21:28:10.589 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=274877906946 2024-09-15T21:28:10.589 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 274877906946 2024-09-15T21:28:10.589 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836506 1-274877906946' 2024-09-15T21:28:10.590 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T21:28:10.590 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836506 2024-09-15T21:28:10.590 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T21:28:10.591 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-15T21:28:10.591 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836506 2024-09-15T21:28:10.591 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T21:28:10.592 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836506 2024-09-15T21:28:10.592 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836506' 2024-09-15T21:28:10.592 INFO:tasks.workunit.client.0.smithi136.stderr:waiting osd.0 seq 21474836506 2024-09-15T21:28:10.593 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T21:28:10.869 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836504 -lt 21474836506 2024-09-15T21:28:10.869 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-15T21:28:11.870 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-15T21:28:11.870 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T21:28:12.151 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836506 -lt 21474836506 2024-09-15T21:28:12.152 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T21:28:12.152 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-274877906946 2024-09-15T21:28:12.152 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T21:28:12.153 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-15T21:28:12.153 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-274877906946 2024-09-15T21:28:12.154 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T21:28:12.155 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=274877906946 2024-09-15T21:28:12.155 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 274877906946' 2024-09-15T21:28:12.155 INFO:tasks.workunit.client.0.smithi136.stderr:waiting osd.1 seq 274877906946 2024-09-15T21:28:12.155 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-15T21:28:12.429 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 274877906946 -lt 274877906946 2024-09-15T21:28:12.430 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-15T21:28:12.430 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T21:28:12.430 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T21:28:12.776 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 6 == 0 2024-09-15T21:28:12.776 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-15T21:28:12.777 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-15T21:28:12.777 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-15T21:28:12.777 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T21:28:12.777 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T21:28:12.777 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T21:28:12.777 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T21:28:13.200 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=6 2024-09-15T21:28:13.201 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-15T21:28:13.201 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T21:28:13.201 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T21:28:13.540 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 = 6 2024-09-15T21:28:13.541 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-15T21:28:13.541 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-15T21:28:13.541 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1029: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2024-09-15T21:28:13.541 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1030: TEST_corrupt_scrub_replicated: objname=ROBJ2 2024-09-15T21:28:13.541 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1033: TEST_corrupt_scrub_replicated: expr 2 % 2 2024-09-15T21:28:13.544 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1033: TEST_corrupt_scrub_replicated: local osd=0 2024-09-15T21:28:13.544 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1035: TEST_corrupt_scrub_replicated: case $i in 2024-09-15T21:28:13.544 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1045: TEST_corrupt_scrub_replicated: local payload=UVWXYZ 2024-09-15T21:28:13.544 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1046: TEST_corrupt_scrub_replicated: echo UVWXYZ 2024-09-15T21:28:13.544 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1047: TEST_corrupt_scrub_replicated: objectstore_tool td/osd-scrub-repair 0 ROBJ2 set-bytes td/osd-scrub-repair/CORRUPT 2024-09-15T21:28:13.544 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2024-09-15T21:28:13.544 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2024-09-15T21:28:13.544 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=0 2024-09-15T21:28:13.545 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2024-09-15T21:28:13.545 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 ROBJ2 set-bytes td/osd-scrub-repair/CORRUPT 2024-09-15T21:28:13.545 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-09-15T21:28:13.545 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2024-09-15T21:28:13.545 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=0 2024-09-15T21:28:13.546 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2024-09-15T21:28:13.546 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2024-09-15T21:28:13.546 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-15T21:28:13.546 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-15T21:28:13.546 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-15T21:28:13.548 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-15T21:28:13.548 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-15T21:28:13.849 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-15T21:28:13.849 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 ROBJ2 set-bytes td/osd-scrub-repair/CORRUPT 2024-09-15T21:28:13.850 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-15T21:28:13.850 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-15T21:28:13.850 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=0 2024-09-15T21:28:13.850 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-15T21:28:13.850 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-09-15T21:28:13.850 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 ROBJ2 set-bytes td/osd-scrub-repair/CORRUPT 2024-09-15T21:28:16.259 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 2024-09-15T21:28:16.259 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2024-09-15T21:28:16.259 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-15T21:28:16.259 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=0 2024-09-15T21:28:16.259 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-15T21:28:16.259 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-09-15T21:28:16.259 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=ce4892c7-6c6d-4b4f-8e6f-07a3e11133e5 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-15T21:28:16.259 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T21:28:16.259 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T21:28:16.259 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T21:28:16.260 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-09-15T21:28:16.260 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-09-15T21:28:16.260 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-15T21:28:16.260 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-15T21:28:16.260 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-15T21:28:16.260 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-15T21:28:16.260 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-15T21:28:16.260 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-15T21:28:16.260 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-15T21:28:16.260 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T21:28:16.260 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52602 2024-09-15T21:28:16.261 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.52602/$cluster-$name.asok' 2024-09-15T21:28:16.261 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.52602/$cluster-$name.asok' 2024-09-15T21:28:16.261 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-15T21:28:16.261 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-15T21:28:16.261 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-15T21:28:16.261 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T21:28:16.261 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T21:28:16.261 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T21:28:16.261 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T21:28:16.261 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-15T21:28:16.261 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-15T21:28:16.262 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-09-15T21:28:16.262 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.0 2024-09-15T21:28:16.262 INFO:tasks.workunit.client.0.smithi136.stderr:start osd.0 2024-09-15T21:28:16.262 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 0 --fsid=ce4892c7-6c6d-4b4f-8e6f-07a3e11133e5 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.52602/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T21:28:16.262 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-09-15T21:28:16.262 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 0 = 0 ']' 2024-09-15T21:28:16.262 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-15T21:28:16.262 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-15T21:28:16.262 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-15T21:28:16.285 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:28:16.282+0000 7f90c3ba2640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:28:16.287 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:28:16.284+0000 7f90c3ba2640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:28:16.288 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:28:16.286+0000 7f90c3ba2640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:28:16.542 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 0 2024-09-15T21:28:16.542 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-15T21:28:16.542 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2024-09-15T21:28:16.542 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-15T21:28:16.542 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-15T21:28:16.543 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:28:16.543 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-15T21:28:16.543 INFO:tasks.workunit.client.0.smithi136.stderr:0 2024-09-15T21:28:16.543 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:28:16.543 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:28:16.817 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:28:17.420 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:28:17.418+0000 7f90c3ba2640 -1 Falling back to public interface 2024-09-15T21:28:17.819 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:28:17.819 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:28:17.819 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-15T21:28:17.819 INFO:tasks.workunit.client.0.smithi136.stderr:1 2024-09-15T21:28:17.819 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:28:17.819 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:28:18.098 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:28:19.100 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:28:19.100 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:28:19.101 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-15T21:28:19.101 INFO:tasks.workunit.client.0.smithi136.stderr:2 2024-09-15T21:28:19.101 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:28:19.101 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:28:19.390 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:28:20.030 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:28:20.028+0000 7f90c3ba2640 -1 osd.0 66 log_to_monitors true 2024-09-15T21:28:20.392 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:28:20.393 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:28:20.393 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-15T21:28:20.393 INFO:tasks.workunit.client.0.smithi136.stderr:3 2024-09-15T21:28:20.393 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:28:20.393 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:28:20.698 INFO:tasks.workunit.client.0.smithi136.stderr:osd.0 up in weight 1 up_from 70 up_thru 64 down_at 67 last_clean_interval [5,66) [v2:127.0.0.1:6802/612363166,v1:127.0.0.1:6803/612363166] [v2:127.0.0.1:6804/612363166,v1:127.0.0.1:6805/612363166] exists,up 7a68df98-8e0f-45ef-a9e9-b2a45ca8fd3f 2024-09-15T21:28:20.698 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-15T21:28:20.698 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-15T21:28:20.698 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-15T21:28:20.698 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2024-09-15T21:28:20.699 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-15T21:28:20.699 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-15T21:28:20.699 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-15T21:28:20.699 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-15T21:28:20.699 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-15T21:28:20.699 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-15T21:28:20.699 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-15T21:28:20.699 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-15T21:28:20.699 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-15T21:28:20.792 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-15T21:28:20.792 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-15T21:28:20.792 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-15T21:28:20.793 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-15T21:28:20.793 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-15T21:28:20.793 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-15T21:28:21.071 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-15T21:28:21.071 INFO:tasks.workunit.client.0.smithi136.stderr:1' 2024-09-15T21:28:21.071 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-15T21:28:21.071 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T21:28:21.071 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-15T21:28:21.223 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=300647710722 2024-09-15T21:28:21.223 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 300647710722 2024-09-15T21:28:21.223 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-300647710722' 2024-09-15T21:28:21.223 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T21:28:21.223 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-15T21:28:21.359 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=274877906949 2024-09-15T21:28:21.359 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 274877906949 2024-09-15T21:28:21.359 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-300647710722 1-274877906949' 2024-09-15T21:28:21.359 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T21:28:21.359 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-300647710722 2024-09-15T21:28:21.359 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T21:28:21.360 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-15T21:28:21.361 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-300647710722 2024-09-15T21:28:21.361 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T21:28:21.362 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=300647710722 2024-09-15T21:28:21.362 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 300647710722' 2024-09-15T21:28:21.362 INFO:tasks.workunit.client.0.smithi136.stderr:waiting osd.0 seq 300647710722 2024-09-15T21:28:21.362 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T21:28:21.637 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 1 -lt 300647710722 2024-09-15T21:28:21.637 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-15T21:28:22.638 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-15T21:28:22.638 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T21:28:22.923 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 1 -lt 300647710722 2024-09-15T21:28:22.923 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-15T21:28:23.925 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 299 -eq 0 ']' 2024-09-15T21:28:23.926 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T21:28:24.203 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 300647710722 -lt 300647710722 2024-09-15T21:28:24.204 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T21:28:24.204 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-274877906949 2024-09-15T21:28:24.204 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T21:28:24.205 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-15T21:28:24.206 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-274877906949 2024-09-15T21:28:24.206 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T21:28:24.207 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=274877906949 2024-09-15T21:28:24.207 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 274877906949' 2024-09-15T21:28:24.207 INFO:tasks.workunit.client.0.smithi136.stderr:waiting osd.1 seq 274877906949 2024-09-15T21:28:24.207 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-15T21:28:24.485 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 274877906949 -lt 274877906949 2024-09-15T21:28:24.486 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-15T21:28:24.486 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T21:28:24.486 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T21:28:24.831 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 6 == 0 2024-09-15T21:28:24.831 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-15T21:28:24.831 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-15T21:28:24.831 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-15T21:28:24.831 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T21:28:24.831 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T21:28:24.831 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T21:28:24.831 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T21:28:25.105 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=6 2024-09-15T21:28:25.106 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-15T21:28:25.106 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T21:28:25.106 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T21:28:25.447 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 = 6 2024-09-15T21:28:25.447 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-15T21:28:25.447 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-15T21:28:25.447 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1029: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2024-09-15T21:28:25.447 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1030: TEST_corrupt_scrub_replicated: objname=ROBJ3 2024-09-15T21:28:25.448 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1033: TEST_corrupt_scrub_replicated: expr 3 % 2 2024-09-15T21:28:25.451 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1033: TEST_corrupt_scrub_replicated: local osd=1 2024-09-15T21:28:25.451 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1035: TEST_corrupt_scrub_replicated: case $i in 2024-09-15T21:28:25.451 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1052: TEST_corrupt_scrub_replicated: objectstore_tool td/osd-scrub-repair 1 ROBJ3 remove 2024-09-15T21:28:25.452 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2024-09-15T21:28:25.452 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2024-09-15T21:28:25.452 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=1 2024-09-15T21:28:25.452 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2024-09-15T21:28:25.452 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 ROBJ3 remove 2024-09-15T21:28:25.453 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-09-15T21:28:25.453 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2024-09-15T21:28:25.453 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=1 2024-09-15T21:28:25.453 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2024-09-15T21:28:25.453 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-09-15T21:28:25.453 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-15T21:28:25.454 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-15T21:28:25.454 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-15T21:28:25.454 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-15T21:28:25.454 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-15T21:28:25.755 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-15T21:28:25.755 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 ROBJ3 remove 2024-09-15T21:28:25.755 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-15T21:28:25.755 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-15T21:28:25.755 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=1 2024-09-15T21:28:25.755 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-15T21:28:25.755 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-09-15T21:28:25.755 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 ROBJ3 remove 2024-09-15T21:28:27.149 INFO:tasks.workunit.client.0.smithi136.stdout:remove #3:f2a5b2a4:::ROBJ3:head# 2024-09-15T21:28:27.666 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2024-09-15T21:28:27.666 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2024-09-15T21:28:27.666 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-15T21:28:27.666 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=1 2024-09-15T21:28:27.666 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-15T21:28:27.666 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-09-15T21:28:27.666 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=ce4892c7-6c6d-4b4f-8e6f-07a3e11133e5 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-15T21:28:27.667 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T21:28:27.667 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T21:28:27.667 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T21:28:27.667 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-09-15T21:28:27.667 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-09-15T21:28:27.667 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-15T21:28:27.667 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-15T21:28:27.667 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-15T21:28:27.667 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-15T21:28:27.667 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-15T21:28:27.667 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-15T21:28:27.667 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-15T21:28:27.667 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T21:28:27.667 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52602 2024-09-15T21:28:27.668 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.52602/$cluster-$name.asok' 2024-09-15T21:28:27.668 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.52602/$cluster-$name.asok' 2024-09-15T21:28:27.668 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-15T21:28:27.668 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-15T21:28:27.668 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-15T21:28:27.668 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T21:28:27.668 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T21:28:27.668 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T21:28:27.668 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T21:28:27.668 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-15T21:28:27.668 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-15T21:28:27.668 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-09-15T21:28:27.669 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.1 2024-09-15T21:28:27.669 INFO:tasks.workunit.client.0.smithi136.stderr:start osd.1 2024-09-15T21:28:27.669 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 1 --fsid=ce4892c7-6c6d-4b4f-8e6f-07a3e11133e5 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.52602/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T21:28:27.670 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-09-15T21:28:27.670 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 1 = 1 ']' 2024-09-15T21:28:27.670 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-15T21:28:27.670 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-15T21:28:27.670 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-15T21:28:27.698 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:28:27.695+0000 7f8c16e11640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:28:27.700 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:28:27.698+0000 7f8c16e11640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:28:27.701 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:28:27.699+0000 7f8c16e11640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:28:27.954 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 1 2024-09-15T21:28:27.955 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-15T21:28:27.955 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2024-09-15T21:28:27.955 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-15T21:28:27.955 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-15T21:28:27.955 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:28:27.955 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-15T21:28:27.955 INFO:tasks.workunit.client.0.smithi136.stderr:0 2024-09-15T21:28:27.955 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:28:27.955 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T21:28:28.229 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:28:29.230 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:28:29.230 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:28:29.230 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-15T21:28:29.231 INFO:tasks.workunit.client.0.smithi136.stderr:1 2024-09-15T21:28:29.231 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:28:29.231 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T21:28:29.328 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:28:29.325+0000 7f8c16e11640 -1 Falling back to public interface 2024-09-15T21:28:29.511 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:28:30.513 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:28:30.513 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:28:30.513 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-15T21:28:30.513 INFO:tasks.workunit.client.0.smithi136.stderr:2 2024-09-15T21:28:30.513 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:28:30.513 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T21:28:30.797 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:28:31.798 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:28:31.798 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:28:31.798 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-15T21:28:31.798 INFO:tasks.workunit.client.0.smithi136.stderr:3 2024-09-15T21:28:31.798 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:28:31.798 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T21:28:32.025 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:28:32.023+0000 7f8c16e11640 -1 osd.1 71 log_to_monitors true 2024-09-15T21:28:32.081 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:28:33.082 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:28:33.082 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:28:33.083 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-15T21:28:33.083 INFO:tasks.workunit.client.0.smithi136.stderr:4 2024-09-15T21:28:33.083 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:28:33.083 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T21:28:33.369 INFO:tasks.workunit.client.0.smithi136.stderr:osd.1 up in weight 1 up_from 75 up_thru 75 down_at 72 last_clean_interval [64,71) [v2:127.0.0.1:6810/1601948888,v1:127.0.0.1:6811/1601948888] [v2:127.0.0.1:6812/1601948888,v1:127.0.0.1:6813/1601948888] exists,up 9aec06d1-33ff-42ab-be71-ac761d546d96 2024-09-15T21:28:33.369 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-15T21:28:33.370 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-15T21:28:33.370 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-15T21:28:33.370 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2024-09-15T21:28:33.370 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-15T21:28:33.370 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-15T21:28:33.370 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-15T21:28:33.370 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-15T21:28:33.371 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-15T21:28:33.371 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-15T21:28:33.371 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-15T21:28:33.371 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-15T21:28:33.371 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-15T21:28:33.463 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-15T21:28:33.463 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-15T21:28:33.463 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-15T21:28:33.463 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-15T21:28:33.463 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-15T21:28:33.463 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-15T21:28:33.738 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-15T21:28:33.738 INFO:tasks.workunit.client.0.smithi136.stderr:1' 2024-09-15T21:28:33.738 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-15T21:28:33.738 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T21:28:33.738 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-15T21:28:33.849 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=300647710725 2024-09-15T21:28:33.849 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 300647710725 2024-09-15T21:28:33.849 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-300647710725' 2024-09-15T21:28:33.849 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T21:28:33.849 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-15T21:28:33.953 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=322122547202 2024-09-15T21:28:33.953 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 322122547202 2024-09-15T21:28:33.953 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-300647710725 1-322122547202' 2024-09-15T21:28:33.953 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T21:28:33.953 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-300647710725 2024-09-15T21:28:33.953 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T21:28:33.954 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-15T21:28:33.955 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-300647710725 2024-09-15T21:28:33.955 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T21:28:33.956 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=300647710725 2024-09-15T21:28:33.956 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 300647710725' 2024-09-15T21:28:33.956 INFO:tasks.workunit.client.0.smithi136.stderr:waiting osd.0 seq 300647710725 2024-09-15T21:28:33.956 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T21:28:34.229 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 300647710724 -lt 300647710725 2024-09-15T21:28:34.229 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-15T21:28:35.230 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-15T21:28:35.230 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T21:28:35.528 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 300647710725 -lt 300647710725 2024-09-15T21:28:35.528 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T21:28:35.528 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-322122547202 2024-09-15T21:28:35.528 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T21:28:35.529 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-15T21:28:35.530 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-322122547202 2024-09-15T21:28:35.530 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T21:28:35.531 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=322122547202 2024-09-15T21:28:35.531 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 322122547202' 2024-09-15T21:28:35.531 INFO:tasks.workunit.client.0.smithi136.stderr:waiting osd.1 seq 322122547202 2024-09-15T21:28:35.531 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-15T21:28:35.807 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 322122547202 -lt 322122547202 2024-09-15T21:28:35.807 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-15T21:28:35.808 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T21:28:35.808 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T21:28:36.149 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 6 == 0 2024-09-15T21:28:36.150 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-15T21:28:36.150 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-15T21:28:36.151 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-15T21:28:36.151 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T21:28:36.151 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T21:28:36.151 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T21:28:36.151 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T21:28:36.423 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=6 2024-09-15T21:28:36.423 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-15T21:28:36.423 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T21:28:36.424 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T21:28:36.765 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 = 6 2024-09-15T21:28:36.765 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-15T21:28:36.765 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-15T21:28:36.765 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1029: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2024-09-15T21:28:36.765 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1030: TEST_corrupt_scrub_replicated: objname=ROBJ4 2024-09-15T21:28:36.765 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1033: TEST_corrupt_scrub_replicated: expr 4 % 2 2024-09-15T21:28:36.767 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1033: TEST_corrupt_scrub_replicated: local osd=0 2024-09-15T21:28:36.767 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1035: TEST_corrupt_scrub_replicated: case $i in 2024-09-15T21:28:36.767 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1057: TEST_corrupt_scrub_replicated: objectstore_tool td/osd-scrub-repair 0 ROBJ4 set-omap key-ROBJ4 td/osd-scrub-repair/CORRUPT 2024-09-15T21:28:36.767 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2024-09-15T21:28:36.767 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2024-09-15T21:28:36.767 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=0 2024-09-15T21:28:36.767 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2024-09-15T21:28:36.767 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 ROBJ4 set-omap key-ROBJ4 td/osd-scrub-repair/CORRUPT 2024-09-15T21:28:36.768 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-09-15T21:28:36.768 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2024-09-15T21:28:36.768 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=0 2024-09-15T21:28:36.768 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2024-09-15T21:28:36.768 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2024-09-15T21:28:36.768 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-15T21:28:36.768 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-15T21:28:36.768 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-15T21:28:36.768 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-15T21:28:36.768 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-15T21:28:37.074 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-15T21:28:37.074 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 ROBJ4 set-omap key-ROBJ4 td/osd-scrub-repair/CORRUPT 2024-09-15T21:28:37.074 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-15T21:28:37.074 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-15T21:28:37.074 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=0 2024-09-15T21:28:37.074 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-15T21:28:37.074 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-09-15T21:28:37.074 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 ROBJ4 set-omap key-ROBJ4 td/osd-scrub-repair/CORRUPT 2024-09-15T21:28:38.960 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 2024-09-15T21:28:38.960 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2024-09-15T21:28:38.960 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-15T21:28:38.960 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=0 2024-09-15T21:28:38.960 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-15T21:28:38.960 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-09-15T21:28:38.960 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=ce4892c7-6c6d-4b4f-8e6f-07a3e11133e5 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-15T21:28:38.960 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T21:28:38.960 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T21:28:38.960 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T21:28:38.961 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-09-15T21:28:38.961 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-09-15T21:28:38.961 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-15T21:28:38.961 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-15T21:28:38.961 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-15T21:28:38.961 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-15T21:28:38.961 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-15T21:28:38.961 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-15T21:28:38.961 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-15T21:28:38.961 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T21:28:38.961 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52602 2024-09-15T21:28:38.961 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.52602/$cluster-$name.asok' 2024-09-15T21:28:38.961 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.52602/$cluster-$name.asok' 2024-09-15T21:28:38.962 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-15T21:28:38.962 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-15T21:28:38.962 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-15T21:28:38.962 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T21:28:38.962 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T21:28:38.962 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T21:28:38.962 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T21:28:38.962 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-15T21:28:38.962 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-15T21:28:38.962 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-09-15T21:28:38.962 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.0 2024-09-15T21:28:38.962 INFO:tasks.workunit.client.0.smithi136.stderr:start osd.0 2024-09-15T21:28:38.962 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 0 --fsid=ce4892c7-6c6d-4b4f-8e6f-07a3e11133e5 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.52602/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T21:28:38.963 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-09-15T21:28:38.963 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 0 = 0 ']' 2024-09-15T21:28:38.963 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-15T21:28:38.963 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-15T21:28:38.963 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-15T21:28:38.990 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:28:38.987+0000 7f0e70a0d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:28:38.992 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:28:38.990+0000 7f0e70a0d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:28:38.994 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:28:38.992+0000 7f0e70a0d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:28:39.265 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 0 2024-09-15T21:28:39.265 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-15T21:28:39.266 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2024-09-15T21:28:39.266 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-15T21:28:39.266 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-15T21:28:39.266 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:28:39.266 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-15T21:28:39.266 INFO:tasks.workunit.client.0.smithi136.stderr:0 2024-09-15T21:28:39.266 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:28:39.266 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:28:39.544 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:28:40.116 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:28:40.114+0000 7f0e70a0d640 -1 Falling back to public interface 2024-09-15T21:28:40.546 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:28:40.546 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:28:40.546 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-15T21:28:40.546 INFO:tasks.workunit.client.0.smithi136.stderr:1 2024-09-15T21:28:40.546 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:28:40.546 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:28:40.827 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:28:41.828 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:28:41.828 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:28:41.828 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-15T21:28:41.829 INFO:tasks.workunit.client.0.smithi136.stderr:2 2024-09-15T21:28:41.829 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:28:41.829 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:28:42.119 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:28:42.744 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:28:42.741+0000 7f0e70a0d640 -1 osd.0 76 log_to_monitors true 2024-09-15T21:28:43.120 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:28:43.121 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:28:43.121 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-15T21:28:43.121 INFO:tasks.workunit.client.0.smithi136.stderr:3 2024-09-15T21:28:43.121 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:28:43.121 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:28:43.481 INFO:tasks.workunit.client.0.smithi136.stderr:osd.0 up in weight 1 up_from 80 up_thru 75 down_at 77 last_clean_interval [70,76) [v2:127.0.0.1:6802/2321090940,v1:127.0.0.1:6803/2321090940] [v2:127.0.0.1:6804/2321090940,v1:127.0.0.1:6805/2321090940] exists,up 7a68df98-8e0f-45ef-a9e9-b2a45ca8fd3f 2024-09-15T21:28:43.482 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-15T21:28:43.482 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-15T21:28:43.482 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-15T21:28:43.482 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2024-09-15T21:28:43.482 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-15T21:28:43.482 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-15T21:28:43.482 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-15T21:28:43.482 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-15T21:28:43.482 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-15T21:28:43.482 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-15T21:28:43.482 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-15T21:28:43.483 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-15T21:28:43.483 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-15T21:28:43.575 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-15T21:28:43.575 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-15T21:28:43.575 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-15T21:28:43.575 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-15T21:28:43.575 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-15T21:28:43.575 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-15T21:28:43.854 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-15T21:28:43.855 INFO:tasks.workunit.client.0.smithi136.stderr:1' 2024-09-15T21:28:43.855 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-15T21:28:43.855 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T21:28:43.855 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-15T21:28:43.962 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=343597383682 2024-09-15T21:28:43.962 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 343597383682 2024-09-15T21:28:43.962 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-343597383682' 2024-09-15T21:28:43.962 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T21:28:43.962 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-15T21:28:44.110 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=322122547205 2024-09-15T21:28:44.110 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 322122547205 2024-09-15T21:28:44.110 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-343597383682 1-322122547205' 2024-09-15T21:28:44.110 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T21:28:44.110 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-343597383682 2024-09-15T21:28:44.110 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T21:28:44.111 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-15T21:28:44.112 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-343597383682 2024-09-15T21:28:44.112 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T21:28:44.113 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=343597383682 2024-09-15T21:28:44.113 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 343597383682' 2024-09-15T21:28:44.113 INFO:tasks.workunit.client.0.smithi136.stderr:waiting osd.0 seq 343597383682 2024-09-15T21:28:44.113 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T21:28:44.388 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 1 -lt 343597383682 2024-09-15T21:28:44.388 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-15T21:28:45.389 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-15T21:28:45.389 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T21:28:45.675 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 343597383682 -lt 343597383682 2024-09-15T21:28:45.676 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T21:28:45.676 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-322122547205 2024-09-15T21:28:45.676 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T21:28:45.677 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-15T21:28:45.677 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-322122547205 2024-09-15T21:28:45.678 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T21:28:45.679 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=322122547205 2024-09-15T21:28:45.679 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 322122547205' 2024-09-15T21:28:45.679 INFO:tasks.workunit.client.0.smithi136.stderr:waiting osd.1 seq 322122547205 2024-09-15T21:28:45.679 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-15T21:28:45.957 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 322122547205 -lt 322122547205 2024-09-15T21:28:45.958 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-15T21:28:45.958 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T21:28:45.958 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T21:28:46.302 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 6 == 0 2024-09-15T21:28:46.302 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-15T21:28:46.303 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-15T21:28:46.303 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-15T21:28:46.303 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T21:28:46.303 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T21:28:46.303 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T21:28:46.303 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T21:28:46.577 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=6 2024-09-15T21:28:46.578 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-15T21:28:46.578 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T21:28:46.578 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T21:28:46.921 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 = 6 2024-09-15T21:28:46.921 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-15T21:28:46.922 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-15T21:28:46.922 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1029: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2024-09-15T21:28:46.922 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1030: TEST_corrupt_scrub_replicated: objname=ROBJ5 2024-09-15T21:28:46.922 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1033: TEST_corrupt_scrub_replicated: expr 5 % 2 2024-09-15T21:28:46.924 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1033: TEST_corrupt_scrub_replicated: local osd=1 2024-09-15T21:28:46.924 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1035: TEST_corrupt_scrub_replicated: case $i in 2024-09-15T21:28:46.924 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1062: TEST_corrupt_scrub_replicated: objectstore_tool td/osd-scrub-repair 1 ROBJ5 rm-omap key-ROBJ5 2024-09-15T21:28:46.925 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2024-09-15T21:28:46.925 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2024-09-15T21:28:46.925 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=1 2024-09-15T21:28:46.925 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2024-09-15T21:28:46.925 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 ROBJ5 rm-omap key-ROBJ5 2024-09-15T21:28:46.926 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-09-15T21:28:46.926 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2024-09-15T21:28:46.926 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=1 2024-09-15T21:28:46.926 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2024-09-15T21:28:46.926 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-09-15T21:28:46.926 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-15T21:28:46.927 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-15T21:28:46.927 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-15T21:28:46.927 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-15T21:28:46.927 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-15T21:28:47.231 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-15T21:28:47.231 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 ROBJ5 rm-omap key-ROBJ5 2024-09-15T21:28:47.231 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-15T21:28:47.231 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-15T21:28:47.231 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=1 2024-09-15T21:28:47.231 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-15T21:28:47.231 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-09-15T21:28:47.231 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 ROBJ5 rm-omap key-ROBJ5 2024-09-15T21:28:49.011 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2024-09-15T21:28:49.012 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2024-09-15T21:28:49.012 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-15T21:28:49.012 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=1 2024-09-15T21:28:49.012 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-15T21:28:49.012 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-09-15T21:28:49.012 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=ce4892c7-6c6d-4b4f-8e6f-07a3e11133e5 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-15T21:28:49.012 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T21:28:49.012 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T21:28:49.012 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T21:28:49.012 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-09-15T21:28:49.012 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-09-15T21:28:49.013 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-15T21:28:49.013 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-15T21:28:49.013 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-15T21:28:49.013 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-15T21:28:49.013 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-15T21:28:49.013 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-15T21:28:49.013 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-15T21:28:49.013 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T21:28:49.013 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52602 2024-09-15T21:28:49.013 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.52602/$cluster-$name.asok' 2024-09-15T21:28:49.013 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.52602/$cluster-$name.asok' 2024-09-15T21:28:49.013 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-15T21:28:49.013 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-15T21:28:49.013 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-15T21:28:49.014 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T21:28:49.014 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T21:28:49.014 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T21:28:49.014 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T21:28:49.014 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-15T21:28:49.014 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-15T21:28:49.014 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-09-15T21:28:49.014 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.1 2024-09-15T21:28:49.014 INFO:tasks.workunit.client.0.smithi136.stderr:start osd.1 2024-09-15T21:28:49.014 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 1 --fsid=ce4892c7-6c6d-4b4f-8e6f-07a3e11133e5 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.52602/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T21:28:49.014 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-09-15T21:28:49.015 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 1 = 1 ']' 2024-09-15T21:28:49.015 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-15T21:28:49.015 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-15T21:28:49.015 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-15T21:28:49.042 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:28:49.039+0000 7fce51dcc640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:28:49.066 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:28:49.063+0000 7fce51dcc640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:28:49.067 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:28:49.065+0000 7fce51dcc640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:28:49.317 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 1 2024-09-15T21:28:49.317 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-15T21:28:49.317 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2024-09-15T21:28:49.317 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-15T21:28:49.317 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-15T21:28:49.317 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:28:49.317 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-15T21:28:49.317 INFO:tasks.workunit.client.0.smithi136.stderr:0 2024-09-15T21:28:49.318 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:28:49.318 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T21:28:49.592 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:28:50.594 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:28:50.594 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:28:50.594 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-15T21:28:50.594 INFO:tasks.workunit.client.0.smithi136.stderr:1 2024-09-15T21:28:50.594 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:28:50.594 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T21:28:50.676 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:28:50.672+0000 7fce51dcc640 -1 Falling back to public interface 2024-09-15T21:28:50.879 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:28:51.881 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:28:51.881 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:28:51.881 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-15T21:28:51.881 INFO:tasks.workunit.client.0.smithi136.stderr:2 2024-09-15T21:28:51.881 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:28:51.881 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T21:28:52.169 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:28:53.170 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:28:53.171 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:28:53.171 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-15T21:28:53.171 INFO:tasks.workunit.client.0.smithi136.stderr:3 2024-09-15T21:28:53.171 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:28:53.171 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T21:28:53.454 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:28:53.500 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:28:53.496+0000 7fce51dcc640 -1 osd.1 81 log_to_monitors true 2024-09-15T21:28:54.456 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:28:54.456 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:28:54.456 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-15T21:28:54.456 INFO:tasks.workunit.client.0.smithi136.stderr:4 2024-09-15T21:28:54.456 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:28:54.456 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T21:28:54.749 INFO:tasks.workunit.client.0.smithi136.stderr:osd.1 up in weight 1 up_from 85 up_thru 85 down_at 82 last_clean_interval [75,81) [v2:127.0.0.1:6810/876396541,v1:127.0.0.1:6811/876396541] [v2:127.0.0.1:6812/876396541,v1:127.0.0.1:6813/876396541] exists,up 9aec06d1-33ff-42ab-be71-ac761d546d96 2024-09-15T21:28:54.749 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-15T21:28:54.749 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-15T21:28:54.749 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-15T21:28:54.749 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2024-09-15T21:28:54.749 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-15T21:28:54.749 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-15T21:28:54.749 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-15T21:28:54.749 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-15T21:28:54.749 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-15T21:28:54.749 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-15T21:28:54.750 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-15T21:28:54.750 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-15T21:28:54.750 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-15T21:28:54.842 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-15T21:28:54.842 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-15T21:28:54.842 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-15T21:28:54.842 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-15T21:28:54.843 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-15T21:28:54.843 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-15T21:28:55.126 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-15T21:28:55.126 INFO:tasks.workunit.client.0.smithi136.stderr:1' 2024-09-15T21:28:55.126 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-15T21:28:55.126 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T21:28:55.126 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-15T21:28:55.363 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=343597383685 2024-09-15T21:28:55.363 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 343597383685 2024-09-15T21:28:55.363 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-343597383685' 2024-09-15T21:28:55.363 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T21:28:55.363 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-15T21:28:55.467 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=365072220162 2024-09-15T21:28:55.467 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 365072220162 2024-09-15T21:28:55.467 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-343597383685 1-365072220162' 2024-09-15T21:28:55.467 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T21:28:55.467 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-343597383685 2024-09-15T21:28:55.467 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T21:28:55.468 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-15T21:28:55.469 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-343597383685 2024-09-15T21:28:55.469 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T21:28:55.470 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=343597383685 2024-09-15T21:28:55.470 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 343597383685' 2024-09-15T21:28:55.470 INFO:tasks.workunit.client.0.smithi136.stderr:waiting osd.0 seq 343597383685 2024-09-15T21:28:55.470 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T21:28:55.744 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 343597383684 -lt 343597383685 2024-09-15T21:28:55.744 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-15T21:28:56.745 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-15T21:28:56.746 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T21:28:57.032 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 343597383684 -lt 343597383685 2024-09-15T21:28:57.033 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-15T21:28:58.033 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 299 -eq 0 ']' 2024-09-15T21:28:58.034 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T21:28:58.315 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 343597383685 -lt 343597383685 2024-09-15T21:28:58.315 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T21:28:58.316 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-365072220162 2024-09-15T21:28:58.316 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T21:28:58.317 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-15T21:28:58.317 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-365072220162 2024-09-15T21:28:58.317 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T21:28:58.319 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=365072220162 2024-09-15T21:28:58.319 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 365072220162' 2024-09-15T21:28:58.319 INFO:tasks.workunit.client.0.smithi136.stderr:waiting osd.1 seq 365072220162 2024-09-15T21:28:58.319 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-15T21:28:58.597 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 365072220162 -lt 365072220162 2024-09-15T21:28:58.597 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-15T21:28:58.597 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T21:28:58.597 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T21:28:58.939 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 6 == 0 2024-09-15T21:28:58.939 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-15T21:28:58.939 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-15T21:28:58.939 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-15T21:28:58.939 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T21:28:58.940 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T21:28:58.940 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T21:28:58.940 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T21:28:59.281 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=6 2024-09-15T21:28:59.281 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-15T21:28:59.282 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T21:28:59.282 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T21:28:59.625 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 = 6 2024-09-15T21:28:59.625 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-15T21:28:59.625 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-15T21:28:59.625 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1029: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2024-09-15T21:28:59.625 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1030: TEST_corrupt_scrub_replicated: objname=ROBJ6 2024-09-15T21:28:59.625 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1033: TEST_corrupt_scrub_replicated: expr 6 % 2 2024-09-15T21:28:59.627 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1033: TEST_corrupt_scrub_replicated: local osd=0 2024-09-15T21:28:59.628 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1035: TEST_corrupt_scrub_replicated: case $i in 2024-09-15T21:28:59.628 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1067: TEST_corrupt_scrub_replicated: echo extra 2024-09-15T21:28:59.628 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1068: TEST_corrupt_scrub_replicated: objectstore_tool td/osd-scrub-repair 0 ROBJ6 set-omap key2-ROBJ6 td/osd-scrub-repair/extra-val 2024-09-15T21:28:59.628 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2024-09-15T21:28:59.628 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2024-09-15T21:28:59.628 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=0 2024-09-15T21:28:59.628 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2024-09-15T21:28:59.628 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 ROBJ6 set-omap key2-ROBJ6 td/osd-scrub-repair/extra-val 2024-09-15T21:28:59.628 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-09-15T21:28:59.628 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2024-09-15T21:28:59.628 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=0 2024-09-15T21:28:59.628 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2024-09-15T21:28:59.628 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2024-09-15T21:28:59.629 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-15T21:28:59.629 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-15T21:28:59.629 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-15T21:28:59.629 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-15T21:28:59.629 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-15T21:28:59.934 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-15T21:28:59.934 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 ROBJ6 set-omap key2-ROBJ6 td/osd-scrub-repair/extra-val 2024-09-15T21:28:59.934 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-15T21:28:59.935 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-15T21:28:59.935 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=0 2024-09-15T21:28:59.935 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-15T21:28:59.935 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-09-15T21:28:59.935 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 ROBJ6 set-omap key2-ROBJ6 td/osd-scrub-repair/extra-val 2024-09-15T21:29:01.908 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 2024-09-15T21:29:01.908 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2024-09-15T21:29:01.908 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-15T21:29:01.908 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=0 2024-09-15T21:29:01.909 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-15T21:29:01.909 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-09-15T21:29:01.909 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=ce4892c7-6c6d-4b4f-8e6f-07a3e11133e5 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-15T21:29:01.909 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T21:29:01.909 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T21:29:01.909 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T21:29:01.909 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-09-15T21:29:01.909 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-09-15T21:29:01.909 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-15T21:29:01.909 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-15T21:29:01.909 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-15T21:29:01.909 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-15T21:29:01.909 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-15T21:29:01.910 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-15T21:29:01.910 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-15T21:29:01.910 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T21:29:01.910 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52602 2024-09-15T21:29:01.910 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.52602/$cluster-$name.asok' 2024-09-15T21:29:01.911 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.52602/$cluster-$name.asok' 2024-09-15T21:29:01.911 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-15T21:29:01.912 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-15T21:29:01.912 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-15T21:29:01.912 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T21:29:01.912 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T21:29:01.912 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T21:29:01.912 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T21:29:01.912 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-15T21:29:01.912 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-15T21:29:01.912 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-09-15T21:29:01.912 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.0 2024-09-15T21:29:01.912 INFO:tasks.workunit.client.0.smithi136.stderr:start osd.0 2024-09-15T21:29:01.912 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 0 --fsid=ce4892c7-6c6d-4b4f-8e6f-07a3e11133e5 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.52602/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T21:29:01.913 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-09-15T21:29:01.913 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 0 = 0 ']' 2024-09-15T21:29:01.913 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-15T21:29:01.913 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-15T21:29:01.913 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-15T21:29:01.941 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:29:01.938+0000 7f410ea28640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:29:01.943 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:29:01.941+0000 7f410ea28640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:29:01.944 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:29:01.942+0000 7f410ea28640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:29:02.197 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 0 2024-09-15T21:29:02.197 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-15T21:29:02.197 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2024-09-15T21:29:02.197 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-15T21:29:02.197 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-15T21:29:02.197 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:29:02.197 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-15T21:29:02.197 INFO:tasks.workunit.client.0.smithi136.stderr:0 2024-09-15T21:29:02.197 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:29:02.197 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:29:02.473 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:29:03.300 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:29:03.298+0000 7f410ea28640 -1 Falling back to public interface 2024-09-15T21:29:03.475 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:29:03.475 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:29:03.475 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-15T21:29:03.475 INFO:tasks.workunit.client.0.smithi136.stderr:1 2024-09-15T21:29:03.475 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:29:03.475 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:29:03.757 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:29:04.760 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:29:04.760 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:29:04.760 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-15T21:29:04.760 INFO:tasks.workunit.client.0.smithi136.stderr:2 2024-09-15T21:29:04.760 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:29:04.760 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:29:05.046 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:29:06.048 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:29:06.048 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:29:06.049 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-15T21:29:06.049 INFO:tasks.workunit.client.0.smithi136.stderr:3 2024-09-15T21:29:06.049 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:29:06.049 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:29:06.149 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:29:06.146+0000 7f410ea28640 -1 osd.0 87 log_to_monitors true 2024-09-15T21:29:06.431 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:29:07.432 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:29:07.433 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:29:07.433 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-15T21:29:07.433 INFO:tasks.workunit.client.0.smithi136.stderr:4 2024-09-15T21:29:07.433 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:29:07.433 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:29:07.766 INFO:tasks.workunit.client.0.smithi136.stderr:osd.0 up in weight 1 up_from 91 up_thru 91 down_at 88 last_clean_interval [80,87) [v2:127.0.0.1:6802/1256222021,v1:127.0.0.1:6803/1256222021] [v2:127.0.0.1:6804/1256222021,v1:127.0.0.1:6805/1256222021] exists,up 7a68df98-8e0f-45ef-a9e9-b2a45ca8fd3f 2024-09-15T21:29:07.766 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-15T21:29:07.766 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-15T21:29:07.766 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-15T21:29:07.766 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2024-09-15T21:29:07.767 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-15T21:29:07.767 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-15T21:29:07.767 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-15T21:29:07.767 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-15T21:29:07.767 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-15T21:29:07.767 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-15T21:29:07.768 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-15T21:29:07.768 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-15T21:29:07.768 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-15T21:29:07.860 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-15T21:29:07.861 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-15T21:29:07.861 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-15T21:29:07.861 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-15T21:29:07.861 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-15T21:29:07.861 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-15T21:29:08.136 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-15T21:29:08.136 INFO:tasks.workunit.client.0.smithi136.stderr:1' 2024-09-15T21:29:08.136 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-15T21:29:08.136 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T21:29:08.136 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-15T21:29:08.239 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=390842023938 2024-09-15T21:29:08.239 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 390842023938 2024-09-15T21:29:08.239 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-390842023938' 2024-09-15T21:29:08.240 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T21:29:08.240 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-15T21:29:08.342 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=365072220165 2024-09-15T21:29:08.343 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 365072220165 2024-09-15T21:29:08.343 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-390842023938 1-365072220165' 2024-09-15T21:29:08.343 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T21:29:08.343 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-390842023938 2024-09-15T21:29:08.343 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T21:29:08.344 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-15T21:29:08.345 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-390842023938 2024-09-15T21:29:08.345 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T21:29:08.346 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=390842023938 2024-09-15T21:29:08.346 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 390842023938' 2024-09-15T21:29:08.346 INFO:tasks.workunit.client.0.smithi136.stderr:waiting osd.0 seq 390842023938 2024-09-15T21:29:08.346 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T21:29:08.619 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 1 -lt 390842023938 2024-09-15T21:29:08.619 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-15T21:29:09.620 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-15T21:29:09.621 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T21:29:09.908 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 390842023938 -lt 390842023938 2024-09-15T21:29:09.908 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T21:29:09.908 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-365072220165 2024-09-15T21:29:09.909 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T21:29:09.909 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-15T21:29:09.910 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-365072220165 2024-09-15T21:29:09.910 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T21:29:09.911 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=365072220165 2024-09-15T21:29:09.911 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 365072220165' 2024-09-15T21:29:09.911 INFO:tasks.workunit.client.0.smithi136.stderr:waiting osd.1 seq 365072220165 2024-09-15T21:29:09.911 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-15T21:29:10.237 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 365072220166 -lt 365072220165 2024-09-15T21:29:10.238 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-15T21:29:10.238 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T21:29:10.238 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T21:29:10.623 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 6 == 0 2024-09-15T21:29:10.624 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-15T21:29:10.624 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-15T21:29:10.624 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-15T21:29:10.624 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T21:29:10.624 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T21:29:10.624 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T21:29:10.624 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T21:29:10.897 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=6 2024-09-15T21:29:10.898 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-15T21:29:10.898 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T21:29:10.898 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T21:29:11.252 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 = 6 2024-09-15T21:29:11.252 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-15T21:29:11.252 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-15T21:29:11.252 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1069: TEST_corrupt_scrub_replicated: rm td/osd-scrub-repair/extra-val 2024-09-15T21:29:11.253 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1029: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2024-09-15T21:29:11.253 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1030: TEST_corrupt_scrub_replicated: objname=ROBJ7 2024-09-15T21:29:11.253 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1033: TEST_corrupt_scrub_replicated: expr 7 % 2 2024-09-15T21:29:11.255 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1033: TEST_corrupt_scrub_replicated: local osd=1 2024-09-15T21:29:11.256 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1035: TEST_corrupt_scrub_replicated: case $i in 2024-09-15T21:29:11.256 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1074: TEST_corrupt_scrub_replicated: echo -n newheader 2024-09-15T21:29:11.256 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1075: TEST_corrupt_scrub_replicated: objectstore_tool td/osd-scrub-repair 1 ROBJ7 set-omaphdr td/osd-scrub-repair/hdr 2024-09-15T21:29:11.256 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2024-09-15T21:29:11.256 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2024-09-15T21:29:11.256 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=1 2024-09-15T21:29:11.256 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2024-09-15T21:29:11.256 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 ROBJ7 set-omaphdr td/osd-scrub-repair/hdr 2024-09-15T21:29:11.256 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-09-15T21:29:11.256 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2024-09-15T21:29:11.256 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=1 2024-09-15T21:29:11.256 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2024-09-15T21:29:11.256 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-09-15T21:29:11.257 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-15T21:29:11.257 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-15T21:29:11.257 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-15T21:29:11.257 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-15T21:29:11.257 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-15T21:29:11.562 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-15T21:29:11.562 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 ROBJ7 set-omaphdr td/osd-scrub-repair/hdr 2024-09-15T21:29:11.562 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-15T21:29:11.562 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-15T21:29:11.562 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=1 2024-09-15T21:29:11.562 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-15T21:29:11.562 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-09-15T21:29:11.562 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 ROBJ7 set-omaphdr td/osd-scrub-repair/hdr 2024-09-15T21:29:13.284 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2024-09-15T21:29:13.284 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2024-09-15T21:29:13.284 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-15T21:29:13.284 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=1 2024-09-15T21:29:13.284 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-15T21:29:13.284 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-09-15T21:29:13.284 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=ce4892c7-6c6d-4b4f-8e6f-07a3e11133e5 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-15T21:29:13.284 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T21:29:13.284 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T21:29:13.284 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T21:29:13.284 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-09-15T21:29:13.285 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-09-15T21:29:13.285 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-15T21:29:13.285 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-15T21:29:13.285 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-15T21:29:13.285 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-15T21:29:13.285 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-15T21:29:13.285 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-15T21:29:13.285 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-15T21:29:13.285 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T21:29:13.285 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52602 2024-09-15T21:29:13.285 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.52602/$cluster-$name.asok' 2024-09-15T21:29:13.285 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.52602/$cluster-$name.asok' 2024-09-15T21:29:13.286 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-15T21:29:13.286 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-15T21:29:13.286 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-15T21:29:13.286 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T21:29:13.286 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T21:29:13.286 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T21:29:13.286 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T21:29:13.286 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-15T21:29:13.286 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-15T21:29:13.286 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-09-15T21:29:13.286 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.1 2024-09-15T21:29:13.287 INFO:tasks.workunit.client.0.smithi136.stderr:start osd.1 2024-09-15T21:29:13.287 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 1 --fsid=ce4892c7-6c6d-4b4f-8e6f-07a3e11133e5 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.52602/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T21:29:13.287 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-09-15T21:29:13.287 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 1 = 1 ']' 2024-09-15T21:29:13.288 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-15T21:29:13.288 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-15T21:29:13.288 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-15T21:29:13.314 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:29:13.312+0000 7f9b109dd640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:29:13.317 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:29:13.314+0000 7f9b109dd640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:29:13.318 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:29:13.315+0000 7f9b109dd640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:29:13.570 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 1 2024-09-15T21:29:13.570 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-15T21:29:13.570 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2024-09-15T21:29:13.570 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-15T21:29:13.570 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-15T21:29:13.571 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:29:13.571 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-15T21:29:13.571 INFO:tasks.workunit.client.0.smithi136.stderr:0 2024-09-15T21:29:13.571 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:29:13.571 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T21:29:13.844 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:29:14.846 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:29:14.846 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:29:14.846 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-15T21:29:14.846 INFO:tasks.workunit.client.0.smithi136.stderr:1 2024-09-15T21:29:14.846 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:29:14.847 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T21:29:14.933 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:29:14.931+0000 7f9b109dd640 -1 Falling back to public interface 2024-09-15T21:29:15.156 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:29:16.158 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:29:16.158 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:29:16.158 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-15T21:29:16.158 INFO:tasks.workunit.client.0.smithi136.stderr:2 2024-09-15T21:29:16.158 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:29:16.158 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T21:29:16.441 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:29:17.442 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:29:17.442 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:29:17.443 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-15T21:29:17.443 INFO:tasks.workunit.client.0.smithi136.stderr:3 2024-09-15T21:29:17.443 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:29:17.443 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T21:29:17.748 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:29:18.115 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:29:18.112+0000 7f9b109dd640 -1 osd.1 92 log_to_monitors true 2024-09-15T21:29:18.749 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:29:18.749 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:29:18.749 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-15T21:29:18.749 INFO:tasks.workunit.client.0.smithi136.stderr:4 2024-09-15T21:29:18.750 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:29:18.750 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T21:29:19.075 INFO:tasks.workunit.client.0.smithi136.stderr:osd.1 up in weight 1 up_from 96 up_thru 96 down_at 93 last_clean_interval [85,92) [v2:127.0.0.1:6810/2527180184,v1:127.0.0.1:6811/2527180184] [v2:127.0.0.1:6812/2527180184,v1:127.0.0.1:6813/2527180184] exists,up 9aec06d1-33ff-42ab-be71-ac761d546d96 2024-09-15T21:29:19.076 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-15T21:29:19.076 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-15T21:29:19.076 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-15T21:29:19.076 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2024-09-15T21:29:19.076 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-15T21:29:19.076 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-15T21:29:19.076 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-15T21:29:19.076 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-15T21:29:19.077 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-15T21:29:19.077 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-15T21:29:19.077 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-15T21:29:19.077 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-15T21:29:19.077 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-15T21:29:19.170 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-15T21:29:19.170 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-15T21:29:19.170 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-15T21:29:19.170 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-15T21:29:19.170 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-15T21:29:19.170 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-15T21:29:19.448 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-15T21:29:19.448 INFO:tasks.workunit.client.0.smithi136.stderr:1' 2024-09-15T21:29:19.448 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-15T21:29:19.448 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T21:29:19.449 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-15T21:29:19.551 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=390842023941 2024-09-15T21:29:19.551 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 390842023941 2024-09-15T21:29:19.551 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-390842023941' 2024-09-15T21:29:19.552 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T21:29:19.552 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-15T21:29:19.654 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=412316860418 2024-09-15T21:29:19.654 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 412316860418 2024-09-15T21:29:19.654 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-390842023941 1-412316860418' 2024-09-15T21:29:19.654 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T21:29:19.654 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-390842023941 2024-09-15T21:29:19.655 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T21:29:19.655 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-15T21:29:19.656 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-390842023941 2024-09-15T21:29:19.656 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T21:29:19.657 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=390842023941 2024-09-15T21:29:19.657 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 390842023941' 2024-09-15T21:29:19.657 INFO:tasks.workunit.client.0.smithi136.stderr:waiting osd.0 seq 390842023941 2024-09-15T21:29:19.657 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T21:29:19.932 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 390842023940 -lt 390842023941 2024-09-15T21:29:19.932 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-15T21:29:20.933 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-15T21:29:20.933 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T21:29:21.224 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 390842023941 -lt 390842023941 2024-09-15T21:29:21.224 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T21:29:21.224 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-412316860418 2024-09-15T21:29:21.224 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T21:29:21.225 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-15T21:29:21.226 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-412316860418 2024-09-15T21:29:21.226 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T21:29:21.227 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=412316860418 2024-09-15T21:29:21.227 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 412316860418' 2024-09-15T21:29:21.227 INFO:tasks.workunit.client.0.smithi136.stderr:waiting osd.1 seq 412316860418 2024-09-15T21:29:21.227 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-15T21:29:21.538 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 412316860418 -lt 412316860418 2024-09-15T21:29:21.538 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-15T21:29:21.539 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T21:29:21.539 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T21:29:21.887 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 6 == 0 2024-09-15T21:29:21.887 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-15T21:29:21.887 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-15T21:29:21.887 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-15T21:29:21.887 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T21:29:21.887 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T21:29:21.887 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T21:29:21.887 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T21:29:22.159 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=6 2024-09-15T21:29:22.159 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-15T21:29:22.159 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T21:29:22.159 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T21:29:22.499 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 = 6 2024-09-15T21:29:22.500 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-15T21:29:22.500 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-15T21:29:22.500 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1076: TEST_corrupt_scrub_replicated: rm td/osd-scrub-repair/hdr 2024-09-15T21:29:22.500 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1029: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2024-09-15T21:29:22.501 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1030: TEST_corrupt_scrub_replicated: objname=ROBJ8 2024-09-15T21:29:22.501 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1033: TEST_corrupt_scrub_replicated: expr 8 % 2 2024-09-15T21:29:22.502 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1033: TEST_corrupt_scrub_replicated: local osd=0 2024-09-15T21:29:22.502 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1035: TEST_corrupt_scrub_replicated: case $i in 2024-09-15T21:29:22.502 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1080: TEST_corrupt_scrub_replicated: rados --pool csr_pool setxattr ROBJ8 key1-ROBJ8 val1-ROBJ8 2024-09-15T21:29:22.621 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1081: TEST_corrupt_scrub_replicated: rados --pool csr_pool setxattr ROBJ8 key2-ROBJ8 val2-ROBJ8 2024-09-15T21:29:22.748 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1084: TEST_corrupt_scrub_replicated: echo -n bad-val 2024-09-15T21:29:22.749 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1085: TEST_corrupt_scrub_replicated: objectstore_tool td/osd-scrub-repair 0 ROBJ8 set-attr _key1-ROBJ8 td/osd-scrub-repair/bad-val 2024-09-15T21:29:22.749 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2024-09-15T21:29:22.749 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2024-09-15T21:29:22.749 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=0 2024-09-15T21:29:22.749 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2024-09-15T21:29:22.749 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 ROBJ8 set-attr _key1-ROBJ8 td/osd-scrub-repair/bad-val 2024-09-15T21:29:22.749 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-09-15T21:29:22.749 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2024-09-15T21:29:22.749 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=0 2024-09-15T21:29:22.749 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2024-09-15T21:29:22.749 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2024-09-15T21:29:22.749 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-15T21:29:22.750 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-15T21:29:22.750 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-15T21:29:22.750 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-15T21:29:22.750 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-15T21:29:23.055 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-15T21:29:23.055 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 ROBJ8 set-attr _key1-ROBJ8 td/osd-scrub-repair/bad-val 2024-09-15T21:29:23.055 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-15T21:29:23.055 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-15T21:29:23.055 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=0 2024-09-15T21:29:23.055 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-15T21:29:23.055 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-09-15T21:29:23.055 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 ROBJ8 set-attr _key1-ROBJ8 td/osd-scrub-repair/bad-val 2024-09-15T21:29:24.620 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 2024-09-15T21:29:24.620 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2024-09-15T21:29:24.620 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-15T21:29:24.620 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=0 2024-09-15T21:29:24.620 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-15T21:29:24.620 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-09-15T21:29:24.620 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=ce4892c7-6c6d-4b4f-8e6f-07a3e11133e5 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-15T21:29:24.620 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T21:29:24.620 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T21:29:24.621 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T21:29:24.621 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-09-15T21:29:24.621 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-09-15T21:29:24.621 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-15T21:29:24.621 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-15T21:29:24.621 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-15T21:29:24.621 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-15T21:29:24.621 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-15T21:29:24.621 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-15T21:29:24.621 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-15T21:29:24.621 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T21:29:24.621 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52602 2024-09-15T21:29:24.621 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.52602/$cluster-$name.asok' 2024-09-15T21:29:24.621 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.52602/$cluster-$name.asok' 2024-09-15T21:29:24.622 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-15T21:29:24.622 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-15T21:29:24.622 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-15T21:29:24.622 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T21:29:24.622 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T21:29:24.622 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T21:29:24.622 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T21:29:24.622 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-15T21:29:24.622 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-15T21:29:24.622 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-09-15T21:29:24.623 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.0 2024-09-15T21:29:24.623 INFO:tasks.workunit.client.0.smithi136.stderr:start osd.0 2024-09-15T21:29:24.623 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 0 --fsid=ce4892c7-6c6d-4b4f-8e6f-07a3e11133e5 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.52602/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T21:29:24.623 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-09-15T21:29:24.623 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 0 = 0 ']' 2024-09-15T21:29:24.623 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-15T21:29:24.623 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-15T21:29:24.624 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-15T21:29:24.650 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:29:24.647+0000 7faffdb02640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:29:24.652 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:29:24.649+0000 7faffdb02640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:29:24.653 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:29:24.651+0000 7faffdb02640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:29:24.906 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 0 2024-09-15T21:29:24.906 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-15T21:29:24.906 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2024-09-15T21:29:24.906 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-15T21:29:24.906 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-15T21:29:24.906 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:29:24.906 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-15T21:29:24.906 INFO:tasks.workunit.client.0.smithi136.stderr:0 2024-09-15T21:29:24.907 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:29:24.907 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:29:25.200 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:29:25.770 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:29:25.768+0000 7faffdb02640 -1 Falling back to public interface 2024-09-15T21:29:26.201 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:29:26.202 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:29:26.202 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-15T21:29:26.202 INFO:tasks.workunit.client.0.smithi136.stderr:1 2024-09-15T21:29:26.202 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:29:26.202 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:29:26.489 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:29:27.491 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:29:27.491 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:29:27.491 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-15T21:29:27.491 INFO:tasks.workunit.client.0.smithi136.stderr:2 2024-09-15T21:29:27.491 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:29:27.491 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:29:27.788 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:29:28.301 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:29:28.298+0000 7faffdb02640 -1 osd.0 97 log_to_monitors true 2024-09-15T21:29:28.789 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:29:28.789 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:29:28.790 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-15T21:29:28.790 INFO:tasks.workunit.client.0.smithi136.stderr:3 2024-09-15T21:29:28.790 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:29:28.790 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:29:29.124 INFO:tasks.workunit.client.0.smithi136.stderr:osd.0 up in weight 1 up_from 101 up_thru 101 down_at 98 last_clean_interval [91,97) [v2:127.0.0.1:6802/3884357174,v1:127.0.0.1:6803/3884357174] [v2:127.0.0.1:6804/3884357174,v1:127.0.0.1:6805/3884357174] exists,up 7a68df98-8e0f-45ef-a9e9-b2a45ca8fd3f 2024-09-15T21:29:29.124 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-15T21:29:29.124 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-15T21:29:29.124 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-15T21:29:29.124 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2024-09-15T21:29:29.124 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-15T21:29:29.124 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-15T21:29:29.124 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-15T21:29:29.124 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-15T21:29:29.124 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-15T21:29:29.124 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-15T21:29:29.125 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-15T21:29:29.125 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-15T21:29:29.125 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-15T21:29:29.217 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-15T21:29:29.217 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-15T21:29:29.217 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-15T21:29:29.217 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-15T21:29:29.217 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-15T21:29:29.217 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-15T21:29:29.667 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-15T21:29:29.667 INFO:tasks.workunit.client.0.smithi136.stderr:1' 2024-09-15T21:29:29.667 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-15T21:29:29.667 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T21:29:29.667 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-15T21:29:29.821 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=433791696898 2024-09-15T21:29:29.821 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 433791696898 2024-09-15T21:29:29.821 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-433791696898' 2024-09-15T21:29:29.821 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T21:29:29.821 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-15T21:29:29.923 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=412316860421 2024-09-15T21:29:29.923 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 412316860421 2024-09-15T21:29:29.924 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-433791696898 1-412316860421' 2024-09-15T21:29:29.924 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T21:29:29.924 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-433791696898 2024-09-15T21:29:29.924 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T21:29:29.925 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-15T21:29:29.925 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-433791696898 2024-09-15T21:29:29.926 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T21:29:29.926 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=433791696898 2024-09-15T21:29:29.927 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 433791696898' 2024-09-15T21:29:29.927 INFO:tasks.workunit.client.0.smithi136.stderr:waiting osd.0 seq 433791696898 2024-09-15T21:29:29.927 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T21:29:30.201 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 1 -lt 433791696898 2024-09-15T21:29:30.201 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-15T21:29:31.202 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-15T21:29:31.202 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T21:29:31.486 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 433791696898 -lt 433791696898 2024-09-15T21:29:31.487 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T21:29:31.487 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-412316860421 2024-09-15T21:29:31.487 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T21:29:31.488 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-15T21:29:31.489 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-412316860421 2024-09-15T21:29:31.489 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T21:29:31.490 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=412316860421 2024-09-15T21:29:31.490 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 412316860421' 2024-09-15T21:29:31.490 INFO:tasks.workunit.client.0.smithi136.stderr:waiting osd.1 seq 412316860421 2024-09-15T21:29:31.490 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-15T21:29:31.807 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 412316860421 -lt 412316860421 2024-09-15T21:29:31.808 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-15T21:29:31.808 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T21:29:31.808 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T21:29:32.153 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 6 == 0 2024-09-15T21:29:32.153 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-15T21:29:32.153 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-15T21:29:32.154 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-15T21:29:32.154 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T21:29:32.154 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T21:29:32.154 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T21:29:32.154 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T21:29:32.428 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=6 2024-09-15T21:29:32.428 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-15T21:29:32.429 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T21:29:32.429 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T21:29:32.773 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 = 6 2024-09-15T21:29:32.773 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-15T21:29:32.773 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-15T21:29:32.773 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1086: TEST_corrupt_scrub_replicated: objectstore_tool td/osd-scrub-repair 1 ROBJ8 rm-attr _key2-ROBJ8 2024-09-15T21:29:32.773 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2024-09-15T21:29:32.774 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2024-09-15T21:29:32.774 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=1 2024-09-15T21:29:32.774 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2024-09-15T21:29:32.774 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 ROBJ8 rm-attr _key2-ROBJ8 2024-09-15T21:29:32.774 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-09-15T21:29:32.774 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2024-09-15T21:29:32.774 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=1 2024-09-15T21:29:32.774 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2024-09-15T21:29:32.774 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-09-15T21:29:32.774 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-15T21:29:32.774 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-15T21:29:32.774 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-15T21:29:32.774 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-15T21:29:32.774 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-15T21:29:33.079 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-15T21:29:33.079 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 ROBJ8 rm-attr _key2-ROBJ8 2024-09-15T21:29:33.080 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-15T21:29:33.080 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-15T21:29:33.080 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=1 2024-09-15T21:29:33.080 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-15T21:29:33.080 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-09-15T21:29:33.080 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 ROBJ8 rm-attr _key2-ROBJ8 2024-09-15T21:29:34.967 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2024-09-15T21:29:34.967 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2024-09-15T21:29:34.967 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-15T21:29:34.968 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=1 2024-09-15T21:29:34.968 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-15T21:29:34.968 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-09-15T21:29:34.968 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=ce4892c7-6c6d-4b4f-8e6f-07a3e11133e5 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-15T21:29:34.968 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T21:29:34.968 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T21:29:34.968 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T21:29:34.968 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-09-15T21:29:34.968 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-09-15T21:29:34.968 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-15T21:29:34.968 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-15T21:29:34.968 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-15T21:29:34.969 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-15T21:29:34.969 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-15T21:29:34.969 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-15T21:29:34.969 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-15T21:29:34.969 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T21:29:34.969 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52602 2024-09-15T21:29:34.969 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.52602/$cluster-$name.asok' 2024-09-15T21:29:34.969 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.52602/$cluster-$name.asok' 2024-09-15T21:29:34.969 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-15T21:29:34.969 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-15T21:29:34.969 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-15T21:29:34.969 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T21:29:34.969 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T21:29:34.970 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T21:29:34.970 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T21:29:34.970 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-15T21:29:34.970 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-15T21:29:34.970 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-09-15T21:29:34.971 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.1 2024-09-15T21:29:34.971 INFO:tasks.workunit.client.0.smithi136.stderr:start osd.1 2024-09-15T21:29:34.971 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 1 --fsid=ce4892c7-6c6d-4b4f-8e6f-07a3e11133e5 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.52602/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T21:29:34.971 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-09-15T21:29:34.971 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 1 = 1 ']' 2024-09-15T21:29:34.971 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-15T21:29:34.971 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-15T21:29:34.971 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-15T21:29:34.999 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:29:34.996+0000 7faf47032640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:29:35.001 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:29:34.999+0000 7faf47032640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:29:35.003 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:29:35.000+0000 7faf47032640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:29:35.265 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 1 2024-09-15T21:29:35.265 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-15T21:29:35.265 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2024-09-15T21:29:35.265 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-15T21:29:35.265 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-15T21:29:35.265 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:29:35.265 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-15T21:29:35.266 INFO:tasks.workunit.client.0.smithi136.stderr:0 2024-09-15T21:29:35.266 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:29:35.266 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T21:29:35.541 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:29:36.542 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:29:36.543 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:29:36.543 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-15T21:29:36.543 INFO:tasks.workunit.client.0.smithi136.stderr:1 2024-09-15T21:29:36.543 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:29:36.543 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T21:29:36.617 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:29:36.615+0000 7faf47032640 -1 Falling back to public interface 2024-09-15T21:29:36.830 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:29:37.831 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:29:37.832 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:29:37.832 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-15T21:29:37.832 INFO:tasks.workunit.client.0.smithi136.stderr:2 2024-09-15T21:29:37.832 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:29:37.832 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T21:29:38.118 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:29:39.120 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:29:39.120 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:29:39.120 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-15T21:29:39.120 INFO:tasks.workunit.client.0.smithi136.stderr:3 2024-09-15T21:29:39.120 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:29:39.120 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T21:29:39.404 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:29:39.756 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:29:39.753+0000 7faf47032640 -1 osd.1 103 log_to_monitors true 2024-09-15T21:29:40.406 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:29:40.406 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:29:40.406 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-15T21:29:40.406 INFO:tasks.workunit.client.0.smithi136.stderr:4 2024-09-15T21:29:40.406 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:29:40.406 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T21:29:40.731 INFO:tasks.workunit.client.0.smithi136.stderr:osd.1 up in weight 1 up_from 107 up_thru 102 down_at 104 last_clean_interval [96,103) [v2:127.0.0.1:6810/4087845224,v1:127.0.0.1:6811/4087845224] [v2:127.0.0.1:6812/4087845224,v1:127.0.0.1:6813/4087845224] exists,up 9aec06d1-33ff-42ab-be71-ac761d546d96 2024-09-15T21:29:40.731 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-15T21:29:40.731 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-15T21:29:40.731 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-15T21:29:40.731 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2024-09-15T21:29:40.732 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-15T21:29:40.732 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-15T21:29:40.732 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-15T21:29:40.732 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-15T21:29:40.732 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-15T21:29:40.733 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-15T21:29:40.733 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-15T21:29:40.733 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-15T21:29:40.733 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-15T21:29:40.825 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-15T21:29:40.826 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-15T21:29:40.826 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-15T21:29:40.826 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-15T21:29:40.826 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-15T21:29:40.826 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-15T21:29:41.300 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-15T21:29:41.301 INFO:tasks.workunit.client.0.smithi136.stderr:1' 2024-09-15T21:29:41.301 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-15T21:29:41.301 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T21:29:41.301 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-15T21:29:41.404 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=433791696901 2024-09-15T21:29:41.404 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 433791696901 2024-09-15T21:29:41.404 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-433791696901' 2024-09-15T21:29:41.404 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T21:29:41.405 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-15T21:29:41.507 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=459561500674 2024-09-15T21:29:41.507 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 459561500674 2024-09-15T21:29:41.507 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-433791696901 1-459561500674' 2024-09-15T21:29:41.508 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T21:29:41.508 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-433791696901 2024-09-15T21:29:41.508 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T21:29:41.509 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-15T21:29:41.509 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-433791696901 2024-09-15T21:29:41.510 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T21:29:41.510 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=433791696901 2024-09-15T21:29:41.510 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 433791696901' 2024-09-15T21:29:41.510 INFO:tasks.workunit.client.0.smithi136.stderr:waiting osd.0 seq 433791696901 2024-09-15T21:29:41.511 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T21:29:41.785 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 433791696900 -lt 433791696901 2024-09-15T21:29:41.786 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-15T21:29:42.787 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-15T21:29:42.787 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T21:29:43.085 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 433791696900 -lt 433791696901 2024-09-15T21:29:43.085 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-15T21:29:44.086 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 299 -eq 0 ']' 2024-09-15T21:29:44.086 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T21:29:44.372 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 433791696901 -lt 433791696901 2024-09-15T21:29:44.372 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T21:29:44.372 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-459561500674 2024-09-15T21:29:44.372 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T21:29:44.373 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-15T21:29:44.374 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-459561500674 2024-09-15T21:29:44.374 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T21:29:44.375 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=459561500674 2024-09-15T21:29:44.375 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 459561500674' 2024-09-15T21:29:44.375 INFO:tasks.workunit.client.0.smithi136.stderr:waiting osd.1 seq 459561500674 2024-09-15T21:29:44.375 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-15T21:29:44.652 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 459561500674 -lt 459561500674 2024-09-15T21:29:44.652 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-15T21:29:44.652 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T21:29:44.652 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T21:29:45.000 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 6 == 0 2024-09-15T21:29:45.001 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-15T21:29:45.001 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-15T21:29:45.001 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-15T21:29:45.001 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T21:29:45.001 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T21:29:45.001 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T21:29:45.001 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T21:29:45.296 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=6 2024-09-15T21:29:45.296 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-15T21:29:45.296 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T21:29:45.296 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T21:29:45.638 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 = 6 2024-09-15T21:29:45.639 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-15T21:29:45.639 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-15T21:29:45.639 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1087: TEST_corrupt_scrub_replicated: echo -n val3-ROBJ8 2024-09-15T21:29:45.639 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1088: TEST_corrupt_scrub_replicated: objectstore_tool td/osd-scrub-repair 1 ROBJ8 set-attr _key3-ROBJ8 td/osd-scrub-repair/newval 2024-09-15T21:29:45.639 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2024-09-15T21:29:45.639 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2024-09-15T21:29:45.639 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=1 2024-09-15T21:29:45.639 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2024-09-15T21:29:45.639 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 ROBJ8 set-attr _key3-ROBJ8 td/osd-scrub-repair/newval 2024-09-15T21:29:45.639 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-09-15T21:29:45.639 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2024-09-15T21:29:45.639 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=1 2024-09-15T21:29:45.640 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2024-09-15T21:29:45.640 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-09-15T21:29:45.640 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-15T21:29:45.640 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-15T21:29:45.640 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-15T21:29:45.640 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-15T21:29:45.640 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-15T21:29:45.945 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-15T21:29:45.945 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 ROBJ8 set-attr _key3-ROBJ8 td/osd-scrub-repair/newval 2024-09-15T21:29:45.946 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-15T21:29:45.946 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-15T21:29:45.946 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=1 2024-09-15T21:29:45.946 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-15T21:29:45.946 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-09-15T21:29:45.946 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 ROBJ8 set-attr _key3-ROBJ8 td/osd-scrub-repair/newval 2024-09-15T21:29:48.370 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2024-09-15T21:29:48.370 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2024-09-15T21:29:48.370 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-15T21:29:48.370 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=1 2024-09-15T21:29:48.370 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-15T21:29:48.371 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-09-15T21:29:48.371 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=ce4892c7-6c6d-4b4f-8e6f-07a3e11133e5 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-15T21:29:48.371 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T21:29:48.371 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T21:29:48.371 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T21:29:48.371 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-09-15T21:29:48.371 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-09-15T21:29:48.371 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-15T21:29:48.371 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-15T21:29:48.371 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-15T21:29:48.371 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-15T21:29:48.371 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-15T21:29:48.371 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-15T21:29:48.371 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-15T21:29:48.372 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T21:29:48.372 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52602 2024-09-15T21:29:48.372 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.52602/$cluster-$name.asok' 2024-09-15T21:29:48.372 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.52602/$cluster-$name.asok' 2024-09-15T21:29:48.372 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-15T21:29:48.372 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-15T21:29:48.372 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-15T21:29:48.372 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T21:29:48.372 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T21:29:48.372 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T21:29:48.372 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T21:29:48.372 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-15T21:29:48.372 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-15T21:29:48.373 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-09-15T21:29:48.373 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.1 2024-09-15T21:29:48.373 INFO:tasks.workunit.client.0.smithi136.stderr:start osd.1 2024-09-15T21:29:48.373 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 1 --fsid=ce4892c7-6c6d-4b4f-8e6f-07a3e11133e5 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.52602/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T21:29:48.374 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-09-15T21:29:48.374 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 1 = 1 ']' 2024-09-15T21:29:48.374 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-15T21:29:48.374 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-15T21:29:48.374 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-15T21:29:48.401 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:29:48.398+0000 7f943312e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:29:48.403 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:29:48.401+0000 7f943312e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:29:48.405 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:29:48.402+0000 7f943312e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:29:48.657 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 1 2024-09-15T21:29:48.658 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-15T21:29:48.658 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2024-09-15T21:29:48.658 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-15T21:29:48.658 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-15T21:29:48.658 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:29:48.659 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-15T21:29:48.659 INFO:tasks.workunit.client.0.smithi136.stderr:0 2024-09-15T21:29:48.659 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:29:48.659 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T21:29:48.965 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:29:49.523 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:29:49.521+0000 7f943312e640 -1 Falling back to public interface 2024-09-15T21:29:49.967 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:29:49.967 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:29:49.967 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-15T21:29:49.968 INFO:tasks.workunit.client.0.smithi136.stderr:1 2024-09-15T21:29:49.968 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:29:49.968 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T21:29:50.253 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:29:51.254 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:29:51.254 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:29:51.254 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-15T21:29:51.254 INFO:tasks.workunit.client.0.smithi136.stderr:2 2024-09-15T21:29:51.254 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:29:51.255 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T21:29:51.539 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:29:52.152 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:29:52.149+0000 7f943312e640 -1 osd.1 108 log_to_monitors true 2024-09-15T21:29:52.540 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:29:52.541 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:29:52.541 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-15T21:29:52.541 INFO:tasks.workunit.client.0.smithi136.stderr:3 2024-09-15T21:29:52.541 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:29:52.541 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T21:29:52.848 INFO:tasks.workunit.client.0.smithi136.stderr:osd.1 up in weight 1 up_from 112 up_thru 107 down_at 109 last_clean_interval [107,108) [v2:127.0.0.1:6810/744916680,v1:127.0.0.1:6811/744916680] [v2:127.0.0.1:6812/744916680,v1:127.0.0.1:6813/744916680] exists,up 9aec06d1-33ff-42ab-be71-ac761d546d96 2024-09-15T21:29:52.848 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-15T21:29:52.848 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-15T21:29:52.848 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-15T21:29:52.848 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2024-09-15T21:29:52.848 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-15T21:29:52.848 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-15T21:29:52.848 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-15T21:29:52.848 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-15T21:29:52.849 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-15T21:29:52.849 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-15T21:29:52.849 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-15T21:29:52.849 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-15T21:29:52.849 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-15T21:29:52.941 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-15T21:29:52.941 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-15T21:29:52.942 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-15T21:29:52.942 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-15T21:29:52.942 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-15T21:29:52.942 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-15T21:29:53.278 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-15T21:29:53.278 INFO:tasks.workunit.client.0.smithi136.stderr:1' 2024-09-15T21:29:53.278 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-15T21:29:53.278 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T21:29:53.278 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-15T21:29:53.486 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=433791696904 2024-09-15T21:29:53.486 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 433791696904 2024-09-15T21:29:53.486 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-433791696904' 2024-09-15T21:29:53.486 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T21:29:53.486 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-15T21:29:53.605 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=481036337154 2024-09-15T21:29:53.605 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 481036337154 2024-09-15T21:29:53.605 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-433791696904 1-481036337154' 2024-09-15T21:29:53.605 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T21:29:53.606 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-433791696904 2024-09-15T21:29:53.606 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T21:29:53.606 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-15T21:29:53.607 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-433791696904 2024-09-15T21:29:53.607 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T21:29:53.608 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=433791696904 2024-09-15T21:29:53.608 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 433791696904' 2024-09-15T21:29:53.609 INFO:tasks.workunit.client.0.smithi136.stderr:waiting osd.0 seq 433791696904 2024-09-15T21:29:53.609 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T21:29:53.953 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 433791696903 -lt 433791696904 2024-09-15T21:29:53.953 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-15T21:29:54.954 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-15T21:29:54.954 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T21:29:55.244 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 433791696905 -lt 433791696904 2024-09-15T21:29:55.245 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T21:29:55.245 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-481036337154 2024-09-15T21:29:55.245 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T21:29:55.246 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-15T21:29:55.247 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-481036337154 2024-09-15T21:29:55.247 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T21:29:55.248 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=481036337154 2024-09-15T21:29:55.248 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 481036337154' 2024-09-15T21:29:55.248 INFO:tasks.workunit.client.0.smithi136.stderr:waiting osd.1 seq 481036337154 2024-09-15T21:29:55.248 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-15T21:29:55.535 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 481036337154 -lt 481036337154 2024-09-15T21:29:55.535 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-15T21:29:55.535 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T21:29:55.535 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T21:29:55.876 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 6 == 0 2024-09-15T21:29:55.876 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-15T21:29:55.876 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-15T21:29:55.876 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-15T21:29:55.876 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T21:29:55.876 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T21:29:55.876 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T21:29:55.876 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T21:29:56.147 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=0 2024-09-15T21:29:56.148 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-15T21:29:56.148 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T21:29:56.148 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T21:29:56.493 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 0 = 6 2024-09-15T21:29:56.493 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 0 '!=' -1 2024-09-15T21:29:56.493 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: loop=0 2024-09-15T21:29:56.493 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1652: wait_for_clean: num_active_clean=0 2024-09-15T21:29:56.493 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-15T21:29:56.493 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 0.1 2024-09-15T21:29:56.594 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-15T21:29:56.594 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-15T21:29:56.595 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-15T21:29:56.595 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-15T21:29:56.595 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T21:29:56.595 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T21:29:56.595 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T21:29:56.595 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T21:29:56.867 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=0 2024-09-15T21:29:56.868 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-15T21:29:56.868 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T21:29:56.868 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T21:29:57.219 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 0 = 6 2024-09-15T21:29:57.219 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 0 '!=' 0 2024-09-15T21:29:57.219 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: get_is_making_recovery_progress 2024-09-15T21:29:57.219 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2024-09-15T21:29:57.220 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-15T21:29:57.220 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-15T21:29:57.220 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-15T21:29:57.220 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2024-09-15T21:29:57.220 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-15T21:29:57.568 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=null 2024-09-15T21:29:57.568 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test null '!=' null 2024-09-15T21:29:57.568 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1655: wait_for_clean: (( 1 >= 13 )) 2024-09-15T21:29:57.568 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-15T21:29:57.569 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 0.2 2024-09-15T21:29:57.770 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-15T21:29:57.770 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-15T21:29:57.770 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-15T21:29:57.770 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-15T21:29:57.770 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T21:29:57.770 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T21:29:57.770 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T21:29:57.770 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T21:29:58.039 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=0 2024-09-15T21:29:58.039 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-15T21:29:58.039 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T21:29:58.039 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T21:29:58.383 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 0 = 6 2024-09-15T21:29:58.383 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 0 '!=' 0 2024-09-15T21:29:58.383 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: get_is_making_recovery_progress 2024-09-15T21:29:58.383 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2024-09-15T21:29:58.383 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-15T21:29:58.383 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-15T21:29:58.383 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-15T21:29:58.383 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2024-09-15T21:29:58.383 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-15T21:29:58.727 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=null 2024-09-15T21:29:58.727 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test null '!=' null 2024-09-15T21:29:58.727 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1655: wait_for_clean: (( 2 >= 13 )) 2024-09-15T21:29:58.727 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-15T21:29:58.727 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 0.4 2024-09-15T21:29:59.128 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-15T21:29:59.128 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-15T21:29:59.130 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-15T21:29:59.130 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-15T21:29:59.130 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T21:29:59.130 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T21:29:59.130 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T21:29:59.130 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T21:29:59.400 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=6 2024-09-15T21:29:59.401 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-15T21:29:59.401 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T21:29:59.401 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T21:29:59.743 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 = 6 2024-09-15T21:29:59.743 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-15T21:29:59.743 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-15T21:29:59.743 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1089: TEST_corrupt_scrub_replicated: rm td/osd-scrub-repair/bad-val td/osd-scrub-repair/newval 2024-09-15T21:29:59.743 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1029: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2024-09-15T21:29:59.744 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1030: TEST_corrupt_scrub_replicated: objname=ROBJ9 2024-09-15T21:29:59.744 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1033: TEST_corrupt_scrub_replicated: expr 9 % 2 2024-09-15T21:29:59.746 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1033: TEST_corrupt_scrub_replicated: local osd=1 2024-09-15T21:29:59.746 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1035: TEST_corrupt_scrub_replicated: case $i in 2024-09-15T21:29:59.746 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1093: TEST_corrupt_scrub_replicated: objectstore_tool td/osd-scrub-repair 1 ROBJ9 get-attr _ 2024-09-15T21:29:59.746 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2024-09-15T21:29:59.746 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2024-09-15T21:29:59.746 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=1 2024-09-15T21:29:59.747 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2024-09-15T21:29:59.747 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 ROBJ9 get-attr _ 2024-09-15T21:29:59.747 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-09-15T21:29:59.747 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2024-09-15T21:29:59.747 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=1 2024-09-15T21:29:59.747 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2024-09-15T21:29:59.747 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-09-15T21:29:59.747 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-15T21:29:59.747 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-15T21:29:59.748 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-15T21:29:59.748 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-15T21:29:59.748 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-15T21:30:00.053 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-15T21:30:00.053 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 ROBJ9 get-attr _ 2024-09-15T21:30:00.053 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-15T21:30:00.053 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-15T21:30:00.053 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=1 2024-09-15T21:30:00.053 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-15T21:30:00.053 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-09-15T21:30:00.053 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 ROBJ9 get-attr _ 2024-09-15T21:30:02.295 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2024-09-15T21:30:02.296 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2024-09-15T21:30:02.296 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-15T21:30:02.296 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=1 2024-09-15T21:30:02.296 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-15T21:30:02.296 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-09-15T21:30:02.296 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=ce4892c7-6c6d-4b4f-8e6f-07a3e11133e5 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-15T21:30:02.296 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T21:30:02.296 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T21:30:02.296 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T21:30:02.296 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-09-15T21:30:02.296 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-09-15T21:30:02.296 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-15T21:30:02.296 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-15T21:30:02.297 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-15T21:30:02.297 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-15T21:30:02.297 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-15T21:30:02.297 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-15T21:30:02.297 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-15T21:30:02.297 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T21:30:02.297 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52602 2024-09-15T21:30:02.297 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.52602/$cluster-$name.asok' 2024-09-15T21:30:02.297 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.52602/$cluster-$name.asok' 2024-09-15T21:30:02.297 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-15T21:30:02.297 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-15T21:30:02.297 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-15T21:30:02.297 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T21:30:02.297 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T21:30:02.298 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T21:30:02.298 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T21:30:02.298 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-15T21:30:02.298 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-15T21:30:02.298 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-09-15T21:30:02.299 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.1 2024-09-15T21:30:02.299 INFO:tasks.workunit.client.0.smithi136.stderr:start osd.1 2024-09-15T21:30:02.299 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 1 --fsid=ce4892c7-6c6d-4b4f-8e6f-07a3e11133e5 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.52602/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T21:30:02.299 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-09-15T21:30:02.299 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 1 = 1 ']' 2024-09-15T21:30:02.299 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-15T21:30:02.299 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-15T21:30:02.299 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-15T21:30:02.327 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:30:02.324+0000 7fadc720b640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:30:02.329 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:30:02.327+0000 7fadc720b640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:30:02.330 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:30:02.328+0000 7fadc720b640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:30:02.584 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 1 2024-09-15T21:30:02.584 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-15T21:30:02.584 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2024-09-15T21:30:02.584 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-15T21:30:02.585 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-15T21:30:02.585 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:30:02.585 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-15T21:30:02.585 INFO:tasks.workunit.client.0.smithi136.stderr:0 2024-09-15T21:30:02.585 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:30:02.585 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T21:30:02.902 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:30:03.449 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:30:03.447+0000 7fadc720b640 -1 Falling back to public interface 2024-09-15T21:30:03.904 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:30:03.904 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:30:03.904 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-15T21:30:03.904 INFO:tasks.workunit.client.0.smithi136.stderr:1 2024-09-15T21:30:03.904 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:30:03.904 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T21:30:04.189 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:30:05.190 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:30:05.190 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:30:05.191 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-15T21:30:05.191 INFO:tasks.workunit.client.0.smithi136.stderr:2 2024-09-15T21:30:05.191 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:30:05.191 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T21:30:05.476 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:30:06.004 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:30:06.002+0000 7fadc720b640 -1 osd.1 113 log_to_monitors true 2024-09-15T21:30:06.478 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:30:06.478 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:30:06.478 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-15T21:30:06.478 INFO:tasks.workunit.client.0.smithi136.stderr:3 2024-09-15T21:30:06.478 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:30:06.478 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T21:30:06.829 INFO:tasks.workunit.client.0.smithi136.stderr:osd.1 up in weight 1 up_from 117 up_thru 112 down_at 114 last_clean_interval [112,113) [v2:127.0.0.1:6810/3614667686,v1:127.0.0.1:6811/3614667686] [v2:127.0.0.1:6812/3614667686,v1:127.0.0.1:6813/3614667686] exists,up 9aec06d1-33ff-42ab-be71-ac761d546d96 2024-09-15T21:30:06.830 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-15T21:30:06.830 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-15T21:30:06.830 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-15T21:30:06.830 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2024-09-15T21:30:06.830 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-15T21:30:06.830 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-15T21:30:06.830 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-15T21:30:06.830 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-15T21:30:06.830 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-15T21:30:06.830 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-15T21:30:06.831 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-15T21:30:06.831 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-15T21:30:06.831 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-15T21:30:06.923 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-15T21:30:06.923 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-15T21:30:06.923 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-15T21:30:06.923 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-15T21:30:06.923 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-15T21:30:06.923 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-15T21:30:07.273 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-15T21:30:07.273 INFO:tasks.workunit.client.0.smithi136.stderr:1' 2024-09-15T21:30:07.273 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-15T21:30:07.273 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T21:30:07.274 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-15T21:30:07.492 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=433791696908 2024-09-15T21:30:07.492 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 433791696908 2024-09-15T21:30:07.492 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-433791696908' 2024-09-15T21:30:07.492 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T21:30:07.493 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-15T21:30:07.598 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=502511173634 2024-09-15T21:30:07.599 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 502511173634 2024-09-15T21:30:07.599 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-433791696908 1-502511173634' 2024-09-15T21:30:07.599 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T21:30:07.599 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-433791696908 2024-09-15T21:30:07.599 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T21:30:07.600 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-15T21:30:07.601 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-433791696908 2024-09-15T21:30:07.601 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T21:30:07.602 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=433791696908 2024-09-15T21:30:07.602 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 433791696908' 2024-09-15T21:30:07.602 INFO:tasks.workunit.client.0.smithi136.stderr:waiting osd.0 seq 433791696908 2024-09-15T21:30:07.602 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T21:30:07.925 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 433791696907 -lt 433791696908 2024-09-15T21:30:07.925 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-15T21:30:08.926 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-15T21:30:08.926 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T21:30:09.217 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 433791696909 -lt 433791696908 2024-09-15T21:30:09.217 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T21:30:09.217 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-502511173634 2024-09-15T21:30:09.217 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T21:30:09.218 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-15T21:30:09.219 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-502511173634 2024-09-15T21:30:09.219 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T21:30:09.220 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=502511173634 2024-09-15T21:30:09.220 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 502511173634' 2024-09-15T21:30:09.220 INFO:tasks.workunit.client.0.smithi136.stderr:waiting osd.1 seq 502511173634 2024-09-15T21:30:09.220 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-15T21:30:09.498 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 502511173634 -lt 502511173634 2024-09-15T21:30:09.498 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-15T21:30:09.499 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T21:30:09.499 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T21:30:09.843 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 6 == 0 2024-09-15T21:30:09.843 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-15T21:30:09.844 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-15T21:30:09.844 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-15T21:30:09.844 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T21:30:09.844 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T21:30:09.844 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T21:30:09.844 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T21:30:10.116 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=6 2024-09-15T21:30:10.117 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-15T21:30:10.117 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T21:30:10.117 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T21:30:10.463 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 = 6 2024-09-15T21:30:10.463 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-15T21:30:10.463 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-15T21:30:10.463 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1094: TEST_corrupt_scrub_replicated: echo -n D 2024-09-15T21:30:10.463 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1095: TEST_corrupt_scrub_replicated: rados --pool csr_pool put ROBJ9 td/osd-scrub-repair/change 2024-09-15T21:30:10.613 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1096: TEST_corrupt_scrub_replicated: objectstore_tool td/osd-scrub-repair 1 ROBJ9 set-attr _ td/osd-scrub-repair/robj9-oi 2024-09-15T21:30:10.614 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2024-09-15T21:30:10.614 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2024-09-15T21:30:10.614 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=1 2024-09-15T21:30:10.614 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2024-09-15T21:30:10.614 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 ROBJ9 set-attr _ td/osd-scrub-repair/robj9-oi 2024-09-15T21:30:10.614 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-09-15T21:30:10.614 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2024-09-15T21:30:10.614 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=1 2024-09-15T21:30:10.614 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2024-09-15T21:30:10.614 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-09-15T21:30:10.614 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-15T21:30:10.614 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-15T21:30:10.615 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-15T21:30:10.615 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-15T21:30:10.615 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-15T21:30:10.920 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-15T21:30:10.920 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 ROBJ9 set-attr _ td/osd-scrub-repair/robj9-oi 2024-09-15T21:30:10.921 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-15T21:30:10.921 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-15T21:30:10.921 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=1 2024-09-15T21:30:10.921 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-15T21:30:10.921 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-09-15T21:30:10.921 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 ROBJ9 set-attr _ td/osd-scrub-repair/robj9-oi 2024-09-15T21:30:12.726 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2024-09-15T21:30:12.726 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2024-09-15T21:30:12.726 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-15T21:30:12.726 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=1 2024-09-15T21:30:12.726 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-15T21:30:12.726 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-09-15T21:30:12.726 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=ce4892c7-6c6d-4b4f-8e6f-07a3e11133e5 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-15T21:30:12.726 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T21:30:12.726 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T21:30:12.727 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T21:30:12.727 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-09-15T21:30:12.727 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-09-15T21:30:12.727 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-15T21:30:12.727 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-15T21:30:12.727 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-15T21:30:12.727 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-15T21:30:12.727 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-15T21:30:12.727 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-15T21:30:12.727 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-15T21:30:12.727 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T21:30:12.727 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52602 2024-09-15T21:30:12.727 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.52602/$cluster-$name.asok' 2024-09-15T21:30:12.729 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.52602/$cluster-$name.asok' 2024-09-15T21:30:12.729 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-15T21:30:12.729 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-15T21:30:12.729 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-15T21:30:12.729 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T21:30:12.729 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T21:30:12.729 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T21:30:12.729 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T21:30:12.729 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-15T21:30:12.730 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-15T21:30:12.730 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-09-15T21:30:12.730 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.1 2024-09-15T21:30:12.730 INFO:tasks.workunit.client.0.smithi136.stderr:start osd.1 2024-09-15T21:30:12.730 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 1 --fsid=ce4892c7-6c6d-4b4f-8e6f-07a3e11133e5 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.52602/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T21:30:12.730 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-09-15T21:30:12.730 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 1 = 1 ']' 2024-09-15T21:30:12.731 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-15T21:30:12.731 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-15T21:30:12.731 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-15T21:30:12.759 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:30:12.756+0000 7f7761631640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:30:12.761 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:30:12.759+0000 7f7761631640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:30:12.762 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:30:12.760+0000 7f7761631640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:30:13.019 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 1 2024-09-15T21:30:13.019 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-15T21:30:13.019 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2024-09-15T21:30:13.020 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-15T21:30:13.020 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-15T21:30:13.020 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:30:13.020 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-15T21:30:13.020 INFO:tasks.workunit.client.0.smithi136.stderr:0 2024-09-15T21:30:13.020 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:30:13.020 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T21:30:13.298 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:30:13.870 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:30:13.868+0000 7f7761631640 -1 Falling back to public interface 2024-09-15T21:30:14.300 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:30:14.300 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:30:14.300 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-15T21:30:14.300 INFO:tasks.workunit.client.0.smithi136.stderr:1 2024-09-15T21:30:14.300 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:30:14.300 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T21:30:14.586 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:30:15.588 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:30:15.588 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:30:15.588 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-15T21:30:15.588 INFO:tasks.workunit.client.0.smithi136.stderr:2 2024-09-15T21:30:15.588 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:30:15.588 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T21:30:15.881 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:30:16.419 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:30:16.416+0000 7f7761631640 -1 osd.1 119 log_to_monitors true 2024-09-15T21:30:16.883 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:30:16.883 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:30:16.883 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-15T21:30:16.883 INFO:tasks.workunit.client.0.smithi136.stderr:3 2024-09-15T21:30:16.883 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:30:16.884 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T21:30:17.291 INFO:tasks.workunit.client.0.smithi136.stderr:osd.1 up in weight 1 up_from 123 up_thru 123 down_at 120 last_clean_interval [117,119) [v2:127.0.0.1:6810/108521004,v1:127.0.0.1:6811/108521004] [v2:127.0.0.1:6812/108521004,v1:127.0.0.1:6813/108521004] exists,up 9aec06d1-33ff-42ab-be71-ac761d546d96 2024-09-15T21:30:17.292 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-15T21:30:17.292 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-15T21:30:17.292 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-15T21:30:17.292 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2024-09-15T21:30:17.292 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-15T21:30:17.292 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-15T21:30:17.292 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-15T21:30:17.292 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-15T21:30:17.293 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-15T21:30:17.293 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-15T21:30:17.293 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-15T21:30:17.293 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-15T21:30:17.293 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-15T21:30:17.385 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-15T21:30:17.386 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-15T21:30:17.386 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-15T21:30:17.386 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-15T21:30:17.386 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-15T21:30:17.386 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-15T21:30:17.664 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-15T21:30:17.664 INFO:tasks.workunit.client.0.smithi136.stderr:1' 2024-09-15T21:30:17.664 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-15T21:30:17.664 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T21:30:17.664 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-15T21:30:17.809 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=433791696911 2024-09-15T21:30:17.810 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 433791696911 2024-09-15T21:30:17.810 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-433791696911' 2024-09-15T21:30:17.810 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T21:30:17.810 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-15T21:30:17.911 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=528280977410 2024-09-15T21:30:17.911 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 528280977410 2024-09-15T21:30:17.911 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-433791696911 1-528280977410' 2024-09-15T21:30:17.912 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T21:30:17.912 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-433791696911 2024-09-15T21:30:17.912 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T21:30:17.913 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-15T21:30:17.913 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-433791696911 2024-09-15T21:30:17.914 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T21:30:17.914 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=433791696911 2024-09-15T21:30:17.915 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 433791696911' 2024-09-15T21:30:17.915 INFO:tasks.workunit.client.0.smithi136.stderr:waiting osd.0 seq 433791696911 2024-09-15T21:30:17.915 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T21:30:18.190 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 433791696910 -lt 433791696911 2024-09-15T21:30:18.190 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-15T21:30:19.191 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-15T21:30:19.191 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T21:30:19.477 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 433791696912 -lt 433791696911 2024-09-15T21:30:19.477 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T21:30:19.477 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-528280977410 2024-09-15T21:30:19.477 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T21:30:19.478 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-15T21:30:19.479 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-528280977410 2024-09-15T21:30:19.479 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T21:30:19.480 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=528280977410 2024-09-15T21:30:19.480 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 528280977410' 2024-09-15T21:30:19.480 INFO:tasks.workunit.client.0.smithi136.stderr:waiting osd.1 seq 528280977410 2024-09-15T21:30:19.480 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-15T21:30:19.758 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 528280977410 -lt 528280977410 2024-09-15T21:30:19.758 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-15T21:30:19.758 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T21:30:19.758 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T21:30:20.101 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 6 == 0 2024-09-15T21:30:20.101 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-15T21:30:20.102 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-15T21:30:20.102 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-15T21:30:20.102 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T21:30:20.102 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T21:30:20.102 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T21:30:20.102 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T21:30:20.376 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=6 2024-09-15T21:30:20.376 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-15T21:30:20.376 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T21:30:20.376 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T21:30:20.718 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 = 6 2024-09-15T21:30:20.718 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-15T21:30:20.718 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-15T21:30:20.718 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1097: TEST_corrupt_scrub_replicated: rm td/osd-scrub-repair/oi td/osd-scrub-repair/change 2024-09-15T21:30:20.719 INFO:tasks.workunit.client.0.smithi136.stderr:rm: cannot remove 'td/osd-scrub-repair/oi': No such file or directory 2024-09-15T21:30:20.719 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1029: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2024-09-15T21:30:20.719 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1030: TEST_corrupt_scrub_replicated: objname=ROBJ10 2024-09-15T21:30:20.719 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1033: TEST_corrupt_scrub_replicated: expr 10 % 2 2024-09-15T21:30:20.721 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1033: TEST_corrupt_scrub_replicated: local osd=0 2024-09-15T21:30:20.721 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1035: TEST_corrupt_scrub_replicated: case $i in 2024-09-15T21:30:20.721 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1029: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2024-09-15T21:30:20.721 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1030: TEST_corrupt_scrub_replicated: objname=ROBJ11 2024-09-15T21:30:20.721 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1033: TEST_corrupt_scrub_replicated: expr 11 % 2 2024-09-15T21:30:20.722 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1033: TEST_corrupt_scrub_replicated: local osd=1 2024-09-15T21:30:20.722 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1035: TEST_corrupt_scrub_replicated: case $i in 2024-09-15T21:30:20.722 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1029: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2024-09-15T21:30:20.722 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1030: TEST_corrupt_scrub_replicated: objname=ROBJ12 2024-09-15T21:30:20.722 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1033: TEST_corrupt_scrub_replicated: expr 12 % 2 2024-09-15T21:30:20.723 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1033: TEST_corrupt_scrub_replicated: local osd=0 2024-09-15T21:30:20.723 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1035: TEST_corrupt_scrub_replicated: case $i in 2024-09-15T21:30:20.723 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1029: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2024-09-15T21:30:20.723 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1030: TEST_corrupt_scrub_replicated: objname=ROBJ13 2024-09-15T21:30:20.724 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1033: TEST_corrupt_scrub_replicated: expr 13 % 2 2024-09-15T21:30:20.724 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1033: TEST_corrupt_scrub_replicated: local osd=1 2024-09-15T21:30:20.724 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1035: TEST_corrupt_scrub_replicated: case $i in 2024-09-15T21:30:20.725 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1029: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2024-09-15T21:30:20.725 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1030: TEST_corrupt_scrub_replicated: objname=ROBJ14 2024-09-15T21:30:20.725 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1033: TEST_corrupt_scrub_replicated: expr 14 % 2 2024-09-15T21:30:20.727 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1033: TEST_corrupt_scrub_replicated: local osd=0 2024-09-15T21:30:20.727 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1035: TEST_corrupt_scrub_replicated: case $i in 2024-09-15T21:30:20.727 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1106: TEST_corrupt_scrub_replicated: echo -n bad-val 2024-09-15T21:30:20.727 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1107: TEST_corrupt_scrub_replicated: objectstore_tool td/osd-scrub-repair 0 ROBJ14 set-attr _ td/osd-scrub-repair/bad-val 2024-09-15T21:30:20.727 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2024-09-15T21:30:20.727 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2024-09-15T21:30:20.727 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=0 2024-09-15T21:30:20.727 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2024-09-15T21:30:20.727 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 ROBJ14 set-attr _ td/osd-scrub-repair/bad-val 2024-09-15T21:30:20.728 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-09-15T21:30:20.728 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2024-09-15T21:30:20.728 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=0 2024-09-15T21:30:20.728 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2024-09-15T21:30:20.728 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2024-09-15T21:30:20.728 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-15T21:30:20.728 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-15T21:30:20.728 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-15T21:30:20.728 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-15T21:30:20.728 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-15T21:30:21.034 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-15T21:30:21.034 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 ROBJ14 set-attr _ td/osd-scrub-repair/bad-val 2024-09-15T21:30:21.034 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-15T21:30:21.034 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-15T21:30:21.034 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=0 2024-09-15T21:30:21.034 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-15T21:30:21.034 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-09-15T21:30:21.034 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 ROBJ14 set-attr _ td/osd-scrub-repair/bad-val 2024-09-15T21:30:22.826 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 2024-09-15T21:30:22.826 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2024-09-15T21:30:22.826 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-15T21:30:22.826 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=0 2024-09-15T21:30:22.826 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-15T21:30:22.826 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-09-15T21:30:22.826 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=ce4892c7-6c6d-4b4f-8e6f-07a3e11133e5 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-15T21:30:22.827 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T21:30:22.827 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T21:30:22.827 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T21:30:22.827 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-09-15T21:30:22.827 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-09-15T21:30:22.827 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-15T21:30:22.827 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-15T21:30:22.827 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-15T21:30:22.827 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-15T21:30:22.827 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-15T21:30:22.827 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-15T21:30:22.827 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-15T21:30:22.828 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T21:30:22.828 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52602 2024-09-15T21:30:22.828 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.52602/$cluster-$name.asok' 2024-09-15T21:30:22.828 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.52602/$cluster-$name.asok' 2024-09-15T21:30:22.828 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-15T21:30:22.828 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-15T21:30:22.828 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-15T21:30:22.828 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T21:30:22.828 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T21:30:22.828 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T21:30:22.828 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T21:30:22.828 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-15T21:30:22.828 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-15T21:30:22.828 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-09-15T21:30:22.830 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.0 2024-09-15T21:30:22.830 INFO:tasks.workunit.client.0.smithi136.stderr:start osd.0 2024-09-15T21:30:22.830 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 0 --fsid=ce4892c7-6c6d-4b4f-8e6f-07a3e11133e5 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.52602/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T21:30:22.830 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-09-15T21:30:22.830 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 0 = 0 ']' 2024-09-15T21:30:22.830 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-15T21:30:22.830 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-15T21:30:22.830 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-15T21:30:22.857 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:30:22.854+0000 7ff967c0f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:30:22.859 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:30:22.857+0000 7ff967c0f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:30:22.861 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:30:22.858+0000 7ff967c0f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:30:23.113 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 0 2024-09-15T21:30:23.113 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-15T21:30:23.113 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2024-09-15T21:30:23.113 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-15T21:30:23.113 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-15T21:30:23.113 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:30:23.114 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-15T21:30:23.114 INFO:tasks.workunit.client.0.smithi136.stderr:0 2024-09-15T21:30:23.114 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:30:23.114 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:30:23.403 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:30:23.988 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:30:23.986+0000 7ff967c0f640 -1 Falling back to public interface 2024-09-15T21:30:24.404 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:30:24.404 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:30:24.404 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-15T21:30:24.405 INFO:tasks.workunit.client.0.smithi136.stderr:1 2024-09-15T21:30:24.405 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:30:24.405 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:30:24.690 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:30:25.692 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:30:25.692 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:30:25.692 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-15T21:30:25.692 INFO:tasks.workunit.client.0.smithi136.stderr:2 2024-09-15T21:30:25.692 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:30:25.692 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:30:25.979 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:30:26.616 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:30:26.612+0000 7ff967c0f640 -1 osd.0 124 log_to_monitors true 2024-09-15T21:30:26.980 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:30:26.980 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:30:26.981 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-15T21:30:26.981 INFO:tasks.workunit.client.0.smithi136.stderr:3 2024-09-15T21:30:26.981 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:30:26.981 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:30:27.272 INFO:tasks.workunit.client.0.smithi136.stderr:osd.0 up in weight 1 up_from 128 up_thru 123 down_at 125 last_clean_interval [101,124) [v2:127.0.0.1:6802/214913542,v1:127.0.0.1:6803/214913542] [v2:127.0.0.1:6804/214913542,v1:127.0.0.1:6805/214913542] exists,up 7a68df98-8e0f-45ef-a9e9-b2a45ca8fd3f 2024-09-15T21:30:27.272 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-15T21:30:27.272 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-15T21:30:27.272 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-15T21:30:27.272 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2024-09-15T21:30:27.272 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-15T21:30:27.273 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-15T21:30:27.273 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-15T21:30:27.273 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-15T21:30:27.273 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-15T21:30:27.273 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-15T21:30:27.273 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-15T21:30:27.273 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-15T21:30:27.273 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-15T21:30:27.365 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-15T21:30:27.366 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-15T21:30:27.366 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-15T21:30:27.366 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-15T21:30:27.366 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-15T21:30:27.366 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-15T21:30:27.731 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-15T21:30:27.731 INFO:tasks.workunit.client.0.smithi136.stderr:1' 2024-09-15T21:30:27.731 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-15T21:30:27.731 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T21:30:27.732 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-15T21:30:27.920 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=549755813890 2024-09-15T21:30:27.920 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 549755813890 2024-09-15T21:30:27.920 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-549755813890' 2024-09-15T21:30:27.920 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T21:30:27.920 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-15T21:30:28.024 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=528280977413 2024-09-15T21:30:28.024 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 528280977413 2024-09-15T21:30:28.024 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-549755813890 1-528280977413' 2024-09-15T21:30:28.024 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T21:30:28.024 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-549755813890 2024-09-15T21:30:28.024 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T21:30:28.025 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-15T21:30:28.026 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-549755813890 2024-09-15T21:30:28.026 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T21:30:28.027 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=549755813890 2024-09-15T21:30:28.027 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 549755813890' 2024-09-15T21:30:28.027 INFO:tasks.workunit.client.0.smithi136.stderr:waiting osd.0 seq 549755813890 2024-09-15T21:30:28.027 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T21:30:28.374 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 1 -lt 549755813890 2024-09-15T21:30:28.374 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-15T21:30:29.375 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-15T21:30:29.375 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T21:30:29.661 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 549755813890 -lt 549755813890 2024-09-15T21:30:29.661 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T21:30:29.661 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-528280977413 2024-09-15T21:30:29.661 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T21:30:29.663 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-15T21:30:29.663 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-528280977413 2024-09-15T21:30:29.663 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T21:30:29.664 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=528280977413 2024-09-15T21:30:29.664 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 528280977413' 2024-09-15T21:30:29.664 INFO:tasks.workunit.client.0.smithi136.stderr:waiting osd.1 seq 528280977413 2024-09-15T21:30:29.665 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-15T21:30:29.940 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 528280977413 -lt 528280977413 2024-09-15T21:30:29.941 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-15T21:30:29.941 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T21:30:29.941 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T21:30:30.285 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 6 == 0 2024-09-15T21:30:30.285 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-15T21:30:30.285 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-15T21:30:30.285 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-15T21:30:30.285 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T21:30:30.285 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T21:30:30.285 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T21:30:30.285 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T21:30:30.555 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=2 2024-09-15T21:30:30.556 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-15T21:30:30.556 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T21:30:30.556 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T21:30:30.899 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 2 = 6 2024-09-15T21:30:30.899 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 2 '!=' -1 2024-09-15T21:30:30.899 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: loop=0 2024-09-15T21:30:30.899 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1652: wait_for_clean: num_active_clean=2 2024-09-15T21:30:30.899 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-15T21:30:30.899 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 0.1 2024-09-15T21:30:31.001 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-15T21:30:31.004 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-15T21:30:31.004 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-15T21:30:31.004 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-15T21:30:31.004 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T21:30:31.004 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T21:30:31.005 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T21:30:31.005 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T21:30:31.275 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=2 2024-09-15T21:30:31.275 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-15T21:30:31.275 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T21:30:31.275 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T21:30:31.617 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 2 = 6 2024-09-15T21:30:31.618 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 2 '!=' 2 2024-09-15T21:30:31.618 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: get_is_making_recovery_progress 2024-09-15T21:30:31.618 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2024-09-15T21:30:31.618 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-15T21:30:31.618 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-15T21:30:31.618 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-15T21:30:31.618 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2024-09-15T21:30:31.619 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-15T21:30:31.961 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=null 2024-09-15T21:30:31.961 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test null '!=' null 2024-09-15T21:30:31.961 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1655: wait_for_clean: (( 1 >= 13 )) 2024-09-15T21:30:31.961 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-15T21:30:31.961 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 0.2 2024-09-15T21:30:32.161 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-15T21:30:32.161 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-15T21:30:32.162 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-15T21:30:32.162 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-15T21:30:32.162 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T21:30:32.163 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T21:30:32.163 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T21:30:32.163 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T21:30:32.443 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=2 2024-09-15T21:30:32.444 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-15T21:30:32.444 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T21:30:32.444 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T21:30:32.786 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 2 = 6 2024-09-15T21:30:32.786 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 2 '!=' 2 2024-09-15T21:30:32.786 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: get_is_making_recovery_progress 2024-09-15T21:30:32.786 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2024-09-15T21:30:32.786 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-15T21:30:32.786 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-15T21:30:32.786 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-15T21:30:32.786 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2024-09-15T21:30:32.786 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-15T21:30:33.144 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=null 2024-09-15T21:30:33.144 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test null '!=' null 2024-09-15T21:30:33.144 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1655: wait_for_clean: (( 2 >= 13 )) 2024-09-15T21:30:33.144 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-15T21:30:33.144 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 0.4 2024-09-15T21:30:33.545 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-15T21:30:33.545 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-15T21:30:33.545 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-15T21:30:33.546 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-15T21:30:33.546 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T21:30:33.546 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T21:30:33.546 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T21:30:33.546 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T21:30:33.819 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=6 2024-09-15T21:30:33.819 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-15T21:30:33.820 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T21:30:33.820 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T21:30:34.182 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 = 6 2024-09-15T21:30:34.182 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-15T21:30:34.182 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-15T21:30:34.182 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1108: TEST_corrupt_scrub_replicated: objectstore_tool td/osd-scrub-repair 1 ROBJ14 rm-attr _ 2024-09-15T21:30:34.182 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2024-09-15T21:30:34.182 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2024-09-15T21:30:34.183 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=1 2024-09-15T21:30:34.183 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2024-09-15T21:30:34.183 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 ROBJ14 rm-attr _ 2024-09-15T21:30:34.183 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-09-15T21:30:34.183 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2024-09-15T21:30:34.183 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=1 2024-09-15T21:30:34.183 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2024-09-15T21:30:34.183 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-09-15T21:30:34.183 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-15T21:30:34.183 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-15T21:30:34.183 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-15T21:30:34.183 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-15T21:30:34.183 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-15T21:30:34.488 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-15T21:30:34.489 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 ROBJ14 rm-attr _ 2024-09-15T21:30:34.489 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-15T21:30:34.489 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-15T21:30:34.489 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=1 2024-09-15T21:30:34.489 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-15T21:30:34.489 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-09-15T21:30:34.489 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 ROBJ14 rm-attr _ 2024-09-15T21:30:36.465 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2024-09-15T21:30:36.465 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2024-09-15T21:30:36.466 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-15T21:30:36.466 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=1 2024-09-15T21:30:36.466 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-15T21:30:36.466 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-09-15T21:30:36.466 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=ce4892c7-6c6d-4b4f-8e6f-07a3e11133e5 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-15T21:30:36.466 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T21:30:36.466 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T21:30:36.466 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T21:30:36.466 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-09-15T21:30:36.466 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-09-15T21:30:36.466 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-15T21:30:36.466 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-15T21:30:36.466 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-15T21:30:36.466 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-15T21:30:36.467 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-15T21:30:36.467 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-15T21:30:36.467 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-15T21:30:36.467 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T21:30:36.467 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52602 2024-09-15T21:30:36.467 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.52602/$cluster-$name.asok' 2024-09-15T21:30:36.469 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.52602/$cluster-$name.asok' 2024-09-15T21:30:36.469 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-15T21:30:36.469 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-15T21:30:36.469 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-15T21:30:36.469 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T21:30:36.469 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T21:30:36.469 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T21:30:36.469 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T21:30:36.469 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-15T21:30:36.469 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-15T21:30:36.469 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-09-15T21:30:36.469 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.1 2024-09-15T21:30:36.469 INFO:tasks.workunit.client.0.smithi136.stderr:start osd.1 2024-09-15T21:30:36.470 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 1 --fsid=ce4892c7-6c6d-4b4f-8e6f-07a3e11133e5 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.52602/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T21:30:36.470 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-09-15T21:30:36.470 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 1 = 1 ']' 2024-09-15T21:30:36.470 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-15T21:30:36.470 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-15T21:30:36.470 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-15T21:30:36.498 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:30:36.495+0000 7fd1a3613640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:30:36.501 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:30:36.498+0000 7fd1a3613640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:30:36.502 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:30:36.500+0000 7fd1a3613640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:30:36.754 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 1 2024-09-15T21:30:36.754 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-15T21:30:36.754 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2024-09-15T21:30:36.754 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-15T21:30:36.754 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-15T21:30:36.754 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:30:36.754 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-15T21:30:36.755 INFO:tasks.workunit.client.0.smithi136.stderr:0 2024-09-15T21:30:36.755 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:30:36.755 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T21:30:37.033 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:30:37.851 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:30:37.849+0000 7fd1a3613640 -1 Falling back to public interface 2024-09-15T21:30:38.034 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:30:38.034 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:30:38.034 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-15T21:30:38.034 INFO:tasks.workunit.client.0.smithi136.stderr:1 2024-09-15T21:30:38.035 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:30:38.035 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T21:30:38.315 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:30:39.316 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:30:39.316 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:30:39.317 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-15T21:30:39.317 INFO:tasks.workunit.client.0.smithi136.stderr:2 2024-09-15T21:30:39.317 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:30:39.317 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T21:30:39.596 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:30:40.598 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:30:40.598 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:30:40.598 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-15T21:30:40.598 INFO:tasks.workunit.client.0.smithi136.stderr:3 2024-09-15T21:30:40.598 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:30:40.598 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T21:30:40.880 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:30:40.922 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:30:40.919+0000 7fd1a3613640 -1 osd.1 129 log_to_monitors true 2024-09-15T21:30:41.882 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:30:41.882 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:30:41.882 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-15T21:30:41.882 INFO:tasks.workunit.client.0.smithi136.stderr:4 2024-09-15T21:30:41.882 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:30:41.882 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T21:30:42.190 INFO:tasks.workunit.client.0.smithi136.stderr:osd.1 up in weight 1 up_from 133 up_thru 133 down_at 130 last_clean_interval [123,129) [v2:127.0.0.1:6810/2365292533,v1:127.0.0.1:6811/2365292533] [v2:127.0.0.1:6812/2365292533,v1:127.0.0.1:6813/2365292533] exists,up 9aec06d1-33ff-42ab-be71-ac761d546d96 2024-09-15T21:30:42.191 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-15T21:30:42.191 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-15T21:30:42.191 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-15T21:30:42.191 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2024-09-15T21:30:42.191 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-15T21:30:42.191 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-15T21:30:42.191 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-15T21:30:42.191 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-15T21:30:42.192 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-15T21:30:42.192 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-15T21:30:42.192 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-15T21:30:42.192 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-15T21:30:42.192 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-15T21:30:42.284 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-15T21:30:42.284 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-15T21:30:42.284 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-15T21:30:42.284 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-15T21:30:42.284 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-15T21:30:42.284 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-15T21:30:42.708 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-15T21:30:42.709 INFO:tasks.workunit.client.0.smithi136.stderr:1' 2024-09-15T21:30:42.709 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-15T21:30:42.709 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T21:30:42.709 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-15T21:30:42.817 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=549755813894 2024-09-15T21:30:42.817 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 549755813894 2024-09-15T21:30:42.817 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-549755813894' 2024-09-15T21:30:42.817 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T21:30:42.817 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-15T21:30:42.959 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=571230650370 2024-09-15T21:30:42.959 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 571230650370 2024-09-15T21:30:42.959 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-549755813894 1-571230650370' 2024-09-15T21:30:42.959 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T21:30:42.960 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-549755813894 2024-09-15T21:30:42.960 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T21:30:42.961 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-15T21:30:42.961 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-549755813894 2024-09-15T21:30:42.962 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T21:30:42.962 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=549755813894 2024-09-15T21:30:42.962 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 549755813894' 2024-09-15T21:30:42.963 INFO:tasks.workunit.client.0.smithi136.stderr:waiting osd.0 seq 549755813894 2024-09-15T21:30:42.963 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T21:30:43.238 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 549755813894 -lt 549755813894 2024-09-15T21:30:43.239 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T21:30:43.239 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-571230650370 2024-09-15T21:30:43.239 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T21:30:43.240 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-15T21:30:43.240 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-571230650370 2024-09-15T21:30:43.241 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T21:30:43.241 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=571230650370 2024-09-15T21:30:43.242 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 571230650370' 2024-09-15T21:30:43.242 INFO:tasks.workunit.client.0.smithi136.stderr:waiting osd.1 seq 571230650370 2024-09-15T21:30:43.242 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-15T21:30:43.558 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 571230650370 -lt 571230650370 2024-09-15T21:30:43.559 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-15T21:30:43.559 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T21:30:43.559 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T21:30:43.906 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 6 == 0 2024-09-15T21:30:43.906 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-15T21:30:43.906 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-15T21:30:43.907 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-15T21:30:43.907 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T21:30:43.907 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T21:30:43.907 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T21:30:43.907 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T21:30:44.179 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=6 2024-09-15T21:30:44.179 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-15T21:30:44.180 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T21:30:44.180 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T21:30:44.519 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 = 6 2024-09-15T21:30:44.519 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-15T21:30:44.519 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-15T21:30:44.519 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1109: TEST_corrupt_scrub_replicated: rm td/osd-scrub-repair/bad-val 2024-09-15T21:30:44.520 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1029: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2024-09-15T21:30:44.520 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1030: TEST_corrupt_scrub_replicated: objname=ROBJ15 2024-09-15T21:30:44.520 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1033: TEST_corrupt_scrub_replicated: expr 15 % 2 2024-09-15T21:30:44.522 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1033: TEST_corrupt_scrub_replicated: local osd=1 2024-09-15T21:30:44.523 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1035: TEST_corrupt_scrub_replicated: case $i in 2024-09-15T21:30:44.523 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1113: TEST_corrupt_scrub_replicated: objectstore_tool td/osd-scrub-repair 1 ROBJ15 rm-attr _ 2024-09-15T21:30:44.523 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2024-09-15T21:30:44.523 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2024-09-15T21:30:44.523 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=1 2024-09-15T21:30:44.523 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2024-09-15T21:30:44.523 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 ROBJ15 rm-attr _ 2024-09-15T21:30:44.523 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-09-15T21:30:44.523 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2024-09-15T21:30:44.523 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=1 2024-09-15T21:30:44.523 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2024-09-15T21:30:44.523 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-09-15T21:30:44.523 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-15T21:30:44.524 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-15T21:30:44.524 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-15T21:30:44.524 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-15T21:30:44.524 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-15T21:30:44.828 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-15T21:30:44.829 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 ROBJ15 rm-attr _ 2024-09-15T21:30:44.829 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-15T21:30:44.829 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-15T21:30:44.829 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=1 2024-09-15T21:30:44.829 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-15T21:30:44.829 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-09-15T21:30:44.829 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 ROBJ15 rm-attr _ 2024-09-15T21:30:46.075 INFO:tasks.workunit.client.0.smithi136.stderr:Error getting attr on : 3.0_head,#3:c0c86b1d:::ROBJ14:head#, (61) No data available 2024-09-15T21:30:46.596 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2024-09-15T21:30:46.596 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2024-09-15T21:30:46.596 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-15T21:30:46.596 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=1 2024-09-15T21:30:46.597 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-15T21:30:46.597 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-09-15T21:30:46.597 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=ce4892c7-6c6d-4b4f-8e6f-07a3e11133e5 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-15T21:30:46.597 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T21:30:46.597 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T21:30:46.597 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T21:30:46.597 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-09-15T21:30:46.597 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-09-15T21:30:46.597 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-15T21:30:46.597 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-15T21:30:46.597 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-15T21:30:46.597 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-15T21:30:46.597 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-15T21:30:46.597 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-15T21:30:46.598 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-15T21:30:46.598 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T21:30:46.598 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52602 2024-09-15T21:30:46.598 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.52602/$cluster-$name.asok' 2024-09-15T21:30:46.598 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.52602/$cluster-$name.asok' 2024-09-15T21:30:46.598 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-15T21:30:46.598 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-15T21:30:46.598 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-15T21:30:46.598 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T21:30:46.598 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T21:30:46.598 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T21:30:46.598 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T21:30:46.598 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-15T21:30:46.599 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-15T21:30:46.599 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-09-15T21:30:46.599 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.1 2024-09-15T21:30:46.600 INFO:tasks.workunit.client.0.smithi136.stderr:start osd.1 2024-09-15T21:30:46.600 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 1 --fsid=ce4892c7-6c6d-4b4f-8e6f-07a3e11133e5 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.52602/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T21:30:46.600 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-09-15T21:30:46.600 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 1 = 1 ']' 2024-09-15T21:30:46.600 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-15T21:30:46.600 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-15T21:30:46.600 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-15T21:30:46.628 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:30:46.625+0000 7f2cf5a14640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:30:46.630 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:30:46.628+0000 7f2cf5a14640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:30:46.631 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:30:46.629+0000 7f2cf5a14640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:30:46.883 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 1 2024-09-15T21:30:46.884 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-15T21:30:46.884 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2024-09-15T21:30:46.884 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-15T21:30:46.884 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-15T21:30:46.884 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:30:46.884 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-15T21:30:46.884 INFO:tasks.workunit.client.0.smithi136.stderr:0 2024-09-15T21:30:46.884 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:30:46.884 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T21:30:47.163 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:30:47.989 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:30:47.986+0000 7f2cf5a14640 -1 Falling back to public interface 2024-09-15T21:30:48.164 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:30:48.165 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:30:48.165 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-15T21:30:48.165 INFO:tasks.workunit.client.0.smithi136.stderr:1 2024-09-15T21:30:48.165 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:30:48.165 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T21:30:48.451 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:30:49.453 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:30:49.453 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:30:49.453 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-15T21:30:49.454 INFO:tasks.workunit.client.0.smithi136.stderr:2 2024-09-15T21:30:49.454 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:30:49.454 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T21:30:49.740 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:30:50.537 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:30:50.534+0000 7f2cf5a14640 -1 osd.1 135 log_to_monitors true 2024-09-15T21:30:50.741 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:30:50.742 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:30:50.742 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-15T21:30:50.742 INFO:tasks.workunit.client.0.smithi136.stderr:3 2024-09-15T21:30:50.742 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:30:50.742 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T21:30:51.305 INFO:tasks.workunit.client.0.smithi136.stderr:osd.1 up in weight 1 up_from 139 up_thru 133 down_at 136 last_clean_interval [133,135) [v2:127.0.0.1:6810/2711530035,v1:127.0.0.1:6811/2711530035] [v2:127.0.0.1:6812/2711530035,v1:127.0.0.1:6813/2711530035] exists,up 9aec06d1-33ff-42ab-be71-ac761d546d96 2024-09-15T21:30:51.306 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-15T21:30:51.306 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-15T21:30:51.306 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-15T21:30:51.306 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2024-09-15T21:30:51.306 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-15T21:30:51.306 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-15T21:30:51.306 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-15T21:30:51.306 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-15T21:30:51.306 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-15T21:30:51.306 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-15T21:30:51.307 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-15T21:30:51.307 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-15T21:30:51.307 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-15T21:30:51.400 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-15T21:30:51.400 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-15T21:30:51.400 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-15T21:30:51.400 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-15T21:30:51.400 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-15T21:30:51.400 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-15T21:30:51.755 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-15T21:30:51.755 INFO:tasks.workunit.client.0.smithi136.stderr:1' 2024-09-15T21:30:51.755 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-15T21:30:51.755 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T21:30:51.755 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-15T21:30:51.899 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=549755813896 2024-09-15T21:30:51.899 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 549755813896 2024-09-15T21:30:51.899 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-549755813896' 2024-09-15T21:30:51.899 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T21:30:51.900 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-15T21:30:52.014 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=597000454146 2024-09-15T21:30:52.014 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 597000454146 2024-09-15T21:30:52.014 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-549755813896 1-597000454146' 2024-09-15T21:30:52.014 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T21:30:52.015 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-549755813896 2024-09-15T21:30:52.015 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T21:30:52.015 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-15T21:30:52.016 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-549755813896 2024-09-15T21:30:52.016 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T21:30:52.017 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=549755813896 2024-09-15T21:30:52.017 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 549755813896' 2024-09-15T21:30:52.017 INFO:tasks.workunit.client.0.smithi136.stderr:waiting osd.0 seq 549755813896 2024-09-15T21:30:52.018 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T21:30:52.296 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 549755813895 -lt 549755813896 2024-09-15T21:30:52.296 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-15T21:30:53.297 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-15T21:30:53.298 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T21:30:53.584 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 549755813897 -lt 549755813896 2024-09-15T21:30:53.584 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T21:30:53.584 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-597000454146 2024-09-15T21:30:53.585 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T21:30:53.585 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-15T21:30:53.586 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-597000454146 2024-09-15T21:30:53.586 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T21:30:53.587 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=597000454146 2024-09-15T21:30:53.587 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 597000454146' 2024-09-15T21:30:53.588 INFO:tasks.workunit.client.0.smithi136.stderr:waiting osd.1 seq 597000454146 2024-09-15T21:30:53.588 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-15T21:30:53.868 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 597000454146 -lt 597000454146 2024-09-15T21:30:53.869 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-15T21:30:53.869 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T21:30:53.869 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T21:30:54.216 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 6 == 0 2024-09-15T21:30:54.216 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-15T21:30:54.217 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-15T21:30:54.217 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-15T21:30:54.217 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T21:30:54.217 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T21:30:54.217 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T21:30:54.217 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T21:30:54.488 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=6 2024-09-15T21:30:54.489 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-15T21:30:54.489 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T21:30:54.489 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T21:30:54.831 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 = 6 2024-09-15T21:30:54.831 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-15T21:30:54.831 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-15T21:30:54.831 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1029: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2024-09-15T21:30:54.831 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1030: TEST_corrupt_scrub_replicated: objname=ROBJ16 2024-09-15T21:30:54.831 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1033: TEST_corrupt_scrub_replicated: expr 16 % 2 2024-09-15T21:30:54.833 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1033: TEST_corrupt_scrub_replicated: local osd=0 2024-09-15T21:30:54.833 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1035: TEST_corrupt_scrub_replicated: case $i in 2024-09-15T21:30:54.833 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1117: TEST_corrupt_scrub_replicated: objectstore_tool td/osd-scrub-repair 0 ROBJ16 rm-attr snapset 2024-09-15T21:30:54.834 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2024-09-15T21:30:54.834 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2024-09-15T21:30:54.834 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=0 2024-09-15T21:30:54.834 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2024-09-15T21:30:54.834 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 ROBJ16 rm-attr snapset 2024-09-15T21:30:54.834 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-09-15T21:30:54.834 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2024-09-15T21:30:54.834 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=0 2024-09-15T21:30:54.834 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2024-09-15T21:30:54.834 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2024-09-15T21:30:54.834 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-15T21:30:54.834 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-15T21:30:54.834 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-15T21:30:54.835 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-15T21:30:54.835 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-15T21:30:55.140 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-15T21:30:55.140 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 ROBJ16 rm-attr snapset 2024-09-15T21:30:55.140 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-15T21:30:55.140 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-15T21:30:55.140 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=0 2024-09-15T21:30:55.140 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-15T21:30:55.140 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-09-15T21:30:55.140 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 ROBJ16 rm-attr snapset 2024-09-15T21:30:56.492 INFO:tasks.workunit.client.0.smithi136.stderr:Error decoding attr on : 3.0_head,#3:c0c86b1d:::ROBJ14:head#, (22) Invalid argument 2024-09-15T21:30:57.259 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 2024-09-15T21:30:57.259 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2024-09-15T21:30:57.259 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-15T21:30:57.259 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=0 2024-09-15T21:30:57.259 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-15T21:30:57.259 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-09-15T21:30:57.259 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=ce4892c7-6c6d-4b4f-8e6f-07a3e11133e5 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-15T21:30:57.259 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T21:30:57.259 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T21:30:57.259 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T21:30:57.260 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-09-15T21:30:57.260 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-09-15T21:30:57.260 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-15T21:30:57.260 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-15T21:30:57.260 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-15T21:30:57.260 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-15T21:30:57.260 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-15T21:30:57.260 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-15T21:30:57.260 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-15T21:30:57.260 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T21:30:57.260 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52602 2024-09-15T21:30:57.260 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.52602/$cluster-$name.asok' 2024-09-15T21:30:57.260 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.52602/$cluster-$name.asok' 2024-09-15T21:30:57.260 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-15T21:30:57.261 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-15T21:30:57.261 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-15T21:30:57.261 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T21:30:57.261 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T21:30:57.261 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T21:30:57.261 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T21:30:57.261 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-15T21:30:57.261 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-15T21:30:57.261 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-09-15T21:30:57.262 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.0 2024-09-15T21:30:57.262 INFO:tasks.workunit.client.0.smithi136.stderr:start osd.0 2024-09-15T21:30:57.262 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 0 --fsid=ce4892c7-6c6d-4b4f-8e6f-07a3e11133e5 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.52602/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T21:30:57.262 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-09-15T21:30:57.263 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 0 = 0 ']' 2024-09-15T21:30:57.263 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-15T21:30:57.263 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-15T21:30:57.263 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-15T21:30:57.290 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:30:57.288+0000 7f2cde82e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:30:57.292 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:30:57.290+0000 7f2cde82e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:30:57.294 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:30:57.291+0000 7f2cde82e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:30:57.550 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 0 2024-09-15T21:30:57.550 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-15T21:30:57.550 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2024-09-15T21:30:57.551 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-15T21:30:57.551 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-15T21:30:57.551 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:30:57.551 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-15T21:30:57.551 INFO:tasks.workunit.client.0.smithi136.stderr:0 2024-09-15T21:30:57.551 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:30:57.551 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:30:57.828 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:30:58.668 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:30:58.666+0000 7f2cde82e640 -1 Falling back to public interface 2024-09-15T21:30:58.830 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:30:58.830 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:30:58.830 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-15T21:30:58.830 INFO:tasks.workunit.client.0.smithi136.stderr:1 2024-09-15T21:30:58.830 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:30:58.830 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:30:59.114 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:31:00.116 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:31:00.116 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:31:00.116 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-15T21:31:00.116 INFO:tasks.workunit.client.0.smithi136.stderr:2 2024-09-15T21:31:00.116 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:31:00.116 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:31:00.402 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:31:01.355 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:31:01.353+0000 7f2cde82e640 -1 osd.0 140 log_to_monitors true 2024-09-15T21:31:01.404 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:31:01.404 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:31:01.404 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-15T21:31:01.404 INFO:tasks.workunit.client.0.smithi136.stderr:3 2024-09-15T21:31:01.404 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:31:01.405 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:31:01.735 INFO:tasks.workunit.client.0.smithi136.stderr:osd.0 up in weight 1 up_from 144 up_thru 139 down_at 141 last_clean_interval [128,140) [v2:127.0.0.1:6802/2875341239,v1:127.0.0.1:6803/2875341239] [v2:127.0.0.1:6804/2875341239,v1:127.0.0.1:6805/2875341239] exists,up 7a68df98-8e0f-45ef-a9e9-b2a45ca8fd3f 2024-09-15T21:31:01.735 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-15T21:31:01.735 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-15T21:31:01.735 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-15T21:31:01.735 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2024-09-15T21:31:01.735 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-15T21:31:01.735 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-15T21:31:01.735 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-15T21:31:01.735 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-15T21:31:01.735 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-15T21:31:01.735 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-15T21:31:01.736 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-15T21:31:01.736 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-15T21:31:01.736 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-15T21:31:01.828 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-15T21:31:01.828 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-15T21:31:01.828 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-15T21:31:01.828 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-15T21:31:01.828 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-15T21:31:01.828 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-15T21:31:02.185 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-15T21:31:02.185 INFO:tasks.workunit.client.0.smithi136.stderr:1' 2024-09-15T21:31:02.185 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-15T21:31:02.185 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T21:31:02.185 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-15T21:31:02.288 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=618475290626 2024-09-15T21:31:02.288 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 618475290626 2024-09-15T21:31:02.288 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-618475290626' 2024-09-15T21:31:02.288 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T21:31:02.288 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-15T21:31:02.392 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=597000454149 2024-09-15T21:31:02.392 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 597000454149 2024-09-15T21:31:02.392 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-618475290626 1-597000454149' 2024-09-15T21:31:02.393 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T21:31:02.393 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-618475290626 2024-09-15T21:31:02.393 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T21:31:02.393 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-15T21:31:02.394 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-618475290626 2024-09-15T21:31:02.394 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T21:31:02.396 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=618475290626 2024-09-15T21:31:02.396 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 618475290626' 2024-09-15T21:31:02.396 INFO:tasks.workunit.client.0.smithi136.stderr:waiting osd.0 seq 618475290626 2024-09-15T21:31:02.396 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T21:31:02.674 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 549755813897 -lt 618475290626 2024-09-15T21:31:02.674 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-15T21:31:03.675 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-15T21:31:03.676 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T21:31:03.964 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 618475290626 -lt 618475290626 2024-09-15T21:31:03.964 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T21:31:03.964 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-597000454149 2024-09-15T21:31:03.964 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T21:31:03.965 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-15T21:31:03.966 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-597000454149 2024-09-15T21:31:03.966 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T21:31:03.967 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=597000454149 2024-09-15T21:31:03.967 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 597000454149' 2024-09-15T21:31:03.967 INFO:tasks.workunit.client.0.smithi136.stderr:waiting osd.1 seq 597000454149 2024-09-15T21:31:03.967 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-15T21:31:04.243 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 597000454149 -lt 597000454149 2024-09-15T21:31:04.244 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-15T21:31:04.244 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T21:31:04.244 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T21:31:04.588 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 6 == 0 2024-09-15T21:31:04.588 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-15T21:31:04.588 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-15T21:31:04.588 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-15T21:31:04.588 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T21:31:04.588 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T21:31:04.589 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T21:31:04.589 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T21:31:04.861 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=0 2024-09-15T21:31:04.862 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-15T21:31:04.862 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T21:31:04.862 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T21:31:05.219 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 0 = 6 2024-09-15T21:31:05.219 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 0 '!=' -1 2024-09-15T21:31:05.219 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: loop=0 2024-09-15T21:31:05.219 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1652: wait_for_clean: num_active_clean=0 2024-09-15T21:31:05.219 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-15T21:31:05.219 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 0.1 2024-09-15T21:31:05.320 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-15T21:31:05.320 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-15T21:31:05.321 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-15T21:31:05.321 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-15T21:31:05.321 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T21:31:05.321 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T21:31:05.321 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T21:31:05.321 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T21:31:05.593 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=0 2024-09-15T21:31:05.593 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-15T21:31:05.593 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T21:31:05.593 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T21:31:05.936 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 0 = 6 2024-09-15T21:31:05.936 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 0 '!=' 0 2024-09-15T21:31:05.936 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: get_is_making_recovery_progress 2024-09-15T21:31:05.936 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2024-09-15T21:31:05.936 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-15T21:31:05.937 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-15T21:31:05.937 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-15T21:31:05.937 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2024-09-15T21:31:05.937 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-15T21:31:06.281 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=null 2024-09-15T21:31:06.281 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test null '!=' null 2024-09-15T21:31:06.281 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1655: wait_for_clean: (( 1 >= 13 )) 2024-09-15T21:31:06.281 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-15T21:31:06.281 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 0.2 2024-09-15T21:31:06.482 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-15T21:31:06.482 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-15T21:31:06.483 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-15T21:31:06.483 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-15T21:31:06.483 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T21:31:06.483 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T21:31:06.483 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T21:31:06.483 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T21:31:06.750 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=0 2024-09-15T21:31:06.751 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-15T21:31:06.751 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T21:31:06.751 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T21:31:07.104 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 0 = 6 2024-09-15T21:31:07.104 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 0 '!=' 0 2024-09-15T21:31:07.104 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: get_is_making_recovery_progress 2024-09-15T21:31:07.104 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2024-09-15T21:31:07.104 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-15T21:31:07.104 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-15T21:31:07.104 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-15T21:31:07.104 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2024-09-15T21:31:07.104 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-15T21:31:07.472 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=null 2024-09-15T21:31:07.473 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test null '!=' null 2024-09-15T21:31:07.473 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1655: wait_for_clean: (( 2 >= 13 )) 2024-09-15T21:31:07.473 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-15T21:31:07.473 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 0.4 2024-09-15T21:31:07.873 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-15T21:31:07.873 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-15T21:31:07.874 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-15T21:31:07.874 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-15T21:31:07.874 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T21:31:07.874 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T21:31:07.874 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T21:31:07.874 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T21:31:08.147 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=6 2024-09-15T21:31:08.148 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-15T21:31:08.148 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T21:31:08.148 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T21:31:08.499 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 = 6 2024-09-15T21:31:08.499 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-15T21:31:08.499 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-15T21:31:08.500 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1118: TEST_corrupt_scrub_replicated: echo -n bad-val 2024-09-15T21:31:08.500 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1119: TEST_corrupt_scrub_replicated: objectstore_tool td/osd-scrub-repair 1 ROBJ16 set-attr snapset td/osd-scrub-repair/bad-val 2024-09-15T21:31:08.500 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2024-09-15T21:31:08.500 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2024-09-15T21:31:08.500 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=1 2024-09-15T21:31:08.500 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2024-09-15T21:31:08.500 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 ROBJ16 set-attr snapset td/osd-scrub-repair/bad-val 2024-09-15T21:31:08.500 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-09-15T21:31:08.500 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2024-09-15T21:31:08.500 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=1 2024-09-15T21:31:08.500 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2024-09-15T21:31:08.500 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-09-15T21:31:08.500 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-15T21:31:08.500 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-15T21:31:08.501 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-15T21:31:08.501 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-15T21:31:08.501 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-15T21:31:08.806 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-15T21:31:08.806 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 ROBJ16 set-attr snapset td/osd-scrub-repair/bad-val 2024-09-15T21:31:08.806 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-15T21:31:08.806 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-15T21:31:08.806 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=1 2024-09-15T21:31:08.806 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-15T21:31:08.806 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-09-15T21:31:08.806 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 ROBJ16 set-attr snapset td/osd-scrub-repair/bad-val 2024-09-15T21:31:10.034 INFO:tasks.workunit.client.0.smithi136.stderr:Error getting attr on : 3.0_head,#3:30259878:::ROBJ15:head#, (61) No data available 2024-09-15T21:31:10.034 INFO:tasks.workunit.client.0.smithi136.stderr:Error getting attr on : 3.0_head,#3:c0c86b1d:::ROBJ14:head#, (61) No data available 2024-09-15T21:31:10.551 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2024-09-15T21:31:10.551 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2024-09-15T21:31:10.551 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-15T21:31:10.551 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=1 2024-09-15T21:31:10.551 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-15T21:31:10.551 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-09-15T21:31:10.552 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=ce4892c7-6c6d-4b4f-8e6f-07a3e11133e5 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-15T21:31:10.552 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T21:31:10.552 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T21:31:10.552 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T21:31:10.552 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-09-15T21:31:10.552 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-09-15T21:31:10.552 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-15T21:31:10.552 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-15T21:31:10.552 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-15T21:31:10.552 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-15T21:31:10.552 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-15T21:31:10.552 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-15T21:31:10.552 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-15T21:31:10.553 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T21:31:10.553 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52602 2024-09-15T21:31:10.553 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.52602/$cluster-$name.asok' 2024-09-15T21:31:10.553 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.52602/$cluster-$name.asok' 2024-09-15T21:31:10.553 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-15T21:31:10.553 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-15T21:31:10.553 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-15T21:31:10.553 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T21:31:10.553 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T21:31:10.553 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T21:31:10.553 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T21:31:10.553 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-15T21:31:10.553 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-15T21:31:10.553 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-09-15T21:31:10.554 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.1 2024-09-15T21:31:10.555 INFO:tasks.workunit.client.0.smithi136.stderr:start osd.1 2024-09-15T21:31:10.555 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 1 --fsid=ce4892c7-6c6d-4b4f-8e6f-07a3e11133e5 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.52602/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T21:31:10.555 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-09-15T21:31:10.555 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 1 = 1 ']' 2024-09-15T21:31:10.555 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-15T21:31:10.555 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-15T21:31:10.555 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-15T21:31:10.583 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:31:10.580+0000 7f6e62c1c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:31:10.585 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:31:10.582+0000 7f6e62c1c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:31:10.586 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:31:10.584+0000 7f6e62c1c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:31:10.840 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 1 2024-09-15T21:31:10.840 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-15T21:31:10.841 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2024-09-15T21:31:10.841 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-15T21:31:10.841 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-15T21:31:10.841 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:31:10.841 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-15T21:31:10.841 INFO:tasks.workunit.client.0.smithi136.stderr:0 2024-09-15T21:31:10.841 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:31:10.841 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T21:31:11.119 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:31:12.121 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:31:12.121 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:31:12.121 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-15T21:31:12.121 INFO:tasks.workunit.client.0.smithi136.stderr:1 2024-09-15T21:31:12.121 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:31:12.121 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T21:31:12.405 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:31:12.440 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:31:12.437+0000 7f6e62c1c640 -1 Falling back to public interface 2024-09-15T21:31:13.407 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:31:13.407 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:31:13.407 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-15T21:31:13.407 INFO:tasks.workunit.client.0.smithi136.stderr:2 2024-09-15T21:31:13.407 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:31:13.408 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T21:31:13.694 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:31:14.695 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:31:14.696 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:31:14.696 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-15T21:31:14.696 INFO:tasks.workunit.client.0.smithi136.stderr:3 2024-09-15T21:31:14.696 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:31:14.696 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T21:31:14.983 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:31:15.160 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:31:15.154+0000 7f6e62c1c640 -1 osd.1 146 log_to_monitors true 2024-09-15T21:31:15.984 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:31:15.985 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:31:15.985 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-15T21:31:15.985 INFO:tasks.workunit.client.0.smithi136.stderr:4 2024-09-15T21:31:15.985 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:31:15.985 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T21:31:16.386 INFO:tasks.workunit.client.0.smithi136.stderr:osd.1 up in weight 1 up_from 150 up_thru 150 down_at 147 last_clean_interval [139,146) [v2:127.0.0.1:6810/3631841700,v1:127.0.0.1:6811/3631841700] [v2:127.0.0.1:6812/3631841700,v1:127.0.0.1:6813/3631841700] exists,up 9aec06d1-33ff-42ab-be71-ac761d546d96 2024-09-15T21:31:16.386 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-15T21:31:16.386 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-15T21:31:16.387 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-15T21:31:16.387 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2024-09-15T21:31:16.387 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-15T21:31:16.387 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-15T21:31:16.387 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-15T21:31:16.387 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-15T21:31:16.387 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-15T21:31:16.387 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-15T21:31:16.387 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-15T21:31:16.387 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-15T21:31:16.387 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-15T21:31:16.476 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-15T21:31:16.476 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-15T21:31:16.476 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-15T21:31:16.476 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-15T21:31:16.476 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-15T21:31:16.476 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-15T21:31:16.758 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-15T21:31:16.758 INFO:tasks.workunit.client.0.smithi136.stderr:1' 2024-09-15T21:31:16.758 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-15T21:31:16.758 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T21:31:16.758 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-15T21:31:16.919 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=618475290630 2024-09-15T21:31:16.919 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 618475290630 2024-09-15T21:31:16.919 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-618475290630' 2024-09-15T21:31:16.919 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T21:31:16.919 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-15T21:31:17.022 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=644245094402 2024-09-15T21:31:17.022 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 644245094402 2024-09-15T21:31:17.022 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-618475290630 1-644245094402' 2024-09-15T21:31:17.022 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T21:31:17.022 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-618475290630 2024-09-15T21:31:17.023 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T21:31:17.023 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-15T21:31:17.024 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-618475290630 2024-09-15T21:31:17.024 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T21:31:17.025 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=618475290630 2024-09-15T21:31:17.025 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 618475290630' 2024-09-15T21:31:17.025 INFO:tasks.workunit.client.0.smithi136.stderr:waiting osd.0 seq 618475290630 2024-09-15T21:31:17.025 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T21:31:17.303 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 618475290630 -lt 618475290630 2024-09-15T21:31:17.303 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T21:31:17.303 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-644245094402 2024-09-15T21:31:17.304 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T21:31:17.305 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-15T21:31:17.305 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-644245094402 2024-09-15T21:31:17.305 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T21:31:17.306 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=644245094402 2024-09-15T21:31:17.306 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 644245094402' 2024-09-15T21:31:17.306 INFO:tasks.workunit.client.0.smithi136.stderr:waiting osd.1 seq 644245094402 2024-09-15T21:31:17.307 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-15T21:31:17.583 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 1 -lt 644245094402 2024-09-15T21:31:17.583 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-15T21:31:18.584 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-15T21:31:18.584 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-15T21:31:18.870 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 1 -lt 644245094402 2024-09-15T21:31:18.870 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-15T21:31:19.871 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 299 -eq 0 ']' 2024-09-15T21:31:19.872 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-15T21:31:20.157 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 644245094402 -lt 644245094402 2024-09-15T21:31:20.158 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-15T21:31:20.158 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T21:31:20.158 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T21:31:20.501 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 6 == 0 2024-09-15T21:31:20.501 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-15T21:31:20.502 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-15T21:31:20.502 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-15T21:31:20.502 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T21:31:20.502 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T21:31:20.502 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T21:31:20.502 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T21:31:20.775 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=6 2024-09-15T21:31:20.776 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-15T21:31:20.776 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T21:31:20.776 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T21:31:21.124 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 = 6 2024-09-15T21:31:21.124 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-15T21:31:21.125 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-15T21:31:21.125 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1029: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2024-09-15T21:31:21.125 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1030: TEST_corrupt_scrub_replicated: objname=ROBJ17 2024-09-15T21:31:21.125 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1033: TEST_corrupt_scrub_replicated: expr 17 % 2 2024-09-15T21:31:21.127 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1033: TEST_corrupt_scrub_replicated: local osd=1 2024-09-15T21:31:21.127 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1035: TEST_corrupt_scrub_replicated: case $i in 2024-09-15T21:31:21.127 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1124: TEST_corrupt_scrub_replicated: local payload=ROBJ17 2024-09-15T21:31:21.127 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1125: TEST_corrupt_scrub_replicated: echo ROBJ17 2024-09-15T21:31:21.127 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1126: TEST_corrupt_scrub_replicated: objectstore_tool td/osd-scrub-repair 0 ROBJ17 set-bytes td/osd-scrub-repair/new.ROBJ17 2024-09-15T21:31:21.127 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2024-09-15T21:31:21.127 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2024-09-15T21:31:21.127 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=0 2024-09-15T21:31:21.127 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2024-09-15T21:31:21.127 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 ROBJ17 set-bytes td/osd-scrub-repair/new.ROBJ17 2024-09-15T21:31:21.128 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-09-15T21:31:21.128 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2024-09-15T21:31:21.128 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=0 2024-09-15T21:31:21.128 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2024-09-15T21:31:21.128 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2024-09-15T21:31:21.128 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-15T21:31:21.128 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-15T21:31:21.128 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-15T21:31:21.128 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-15T21:31:21.128 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-15T21:31:21.433 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-15T21:31:21.433 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 ROBJ17 set-bytes td/osd-scrub-repair/new.ROBJ17 2024-09-15T21:31:21.433 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-15T21:31:21.434 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-15T21:31:21.434 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=0 2024-09-15T21:31:21.434 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-15T21:31:21.434 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-09-15T21:31:21.434 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 ROBJ17 set-bytes td/osd-scrub-repair/new.ROBJ17 2024-09-15T21:31:23.003 INFO:tasks.workunit.client.0.smithi136.stderr:Error decoding attr on : 3.0_head,#3:c0c86b1d:::ROBJ14:head#, (22) Invalid argument 2024-09-15T21:31:23.848 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 2024-09-15T21:31:23.848 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2024-09-15T21:31:23.848 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-15T21:31:23.849 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=0 2024-09-15T21:31:23.849 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-15T21:31:23.849 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-09-15T21:31:23.849 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=ce4892c7-6c6d-4b4f-8e6f-07a3e11133e5 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-15T21:31:23.849 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T21:31:23.849 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T21:31:23.849 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T21:31:23.849 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-09-15T21:31:23.849 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-09-15T21:31:23.849 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-15T21:31:23.849 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-15T21:31:23.850 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-15T21:31:23.850 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-15T21:31:23.850 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-15T21:31:23.850 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-15T21:31:23.850 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-15T21:31:23.850 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T21:31:23.850 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52602 2024-09-15T21:31:23.850 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.52602/$cluster-$name.asok' 2024-09-15T21:31:23.850 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.52602/$cluster-$name.asok' 2024-09-15T21:31:23.850 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-15T21:31:23.850 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-15T21:31:23.850 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-15T21:31:23.850 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T21:31:23.851 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T21:31:23.851 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T21:31:23.851 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T21:31:23.851 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-15T21:31:23.851 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-15T21:31:23.851 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-09-15T21:31:23.851 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.0 2024-09-15T21:31:23.851 INFO:tasks.workunit.client.0.smithi136.stderr:start osd.0 2024-09-15T21:31:23.852 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 0 --fsid=ce4892c7-6c6d-4b4f-8e6f-07a3e11133e5 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.52602/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T21:31:23.852 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-09-15T21:31:23.852 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 0 = 0 ']' 2024-09-15T21:31:23.852 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-15T21:31:23.852 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-15T21:31:23.853 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-15T21:31:23.879 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:31:23.876+0000 7fdc22830640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:31:23.881 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:31:23.878+0000 7fdc22830640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:31:23.882 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:31:23.880+0000 7fdc22830640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:31:24.136 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 0 2024-09-15T21:31:24.136 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-15T21:31:24.137 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2024-09-15T21:31:24.137 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-15T21:31:24.137 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-15T21:31:24.137 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:31:24.137 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-15T21:31:24.137 INFO:tasks.workunit.client.0.smithi136.stderr:0 2024-09-15T21:31:24.137 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:31:24.137 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:31:24.415 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:31:25.006 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:31:25.004+0000 7fdc22830640 -1 Falling back to public interface 2024-09-15T21:31:25.417 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:31:25.417 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:31:25.417 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-15T21:31:25.417 INFO:tasks.workunit.client.0.smithi136.stderr:1 2024-09-15T21:31:25.417 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:31:25.417 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:31:25.699 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:31:26.701 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:31:26.701 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:31:26.701 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-15T21:31:26.701 INFO:tasks.workunit.client.0.smithi136.stderr:2 2024-09-15T21:31:26.701 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:31:26.701 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:31:26.993 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:31:27.735 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:31:27.733+0000 7fdc22830640 -1 osd.0 152 log_to_monitors true 2024-09-15T21:31:27.994 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:31:27.994 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:31:27.995 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-15T21:31:27.995 INFO:tasks.workunit.client.0.smithi136.stderr:3 2024-09-15T21:31:27.995 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:31:27.995 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:31:28.406 INFO:tasks.workunit.client.0.smithi136.stderr:osd.0 up in weight 1 up_from 156 up_thru 151 down_at 153 last_clean_interval [144,152) [v2:127.0.0.1:6802/1257238916,v1:127.0.0.1:6803/1257238916] [v2:127.0.0.1:6804/1257238916,v1:127.0.0.1:6805/1257238916] exists,up 7a68df98-8e0f-45ef-a9e9-b2a45ca8fd3f 2024-09-15T21:31:28.406 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-15T21:31:28.406 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-15T21:31:28.406 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-15T21:31:28.406 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2024-09-15T21:31:28.406 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-15T21:31:28.406 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-15T21:31:28.406 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-15T21:31:28.406 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-15T21:31:28.407 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-15T21:31:28.407 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-15T21:31:28.407 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-15T21:31:28.407 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-15T21:31:28.407 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-15T21:31:28.500 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-15T21:31:28.500 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-15T21:31:28.501 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-15T21:31:28.501 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-15T21:31:28.501 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-15T21:31:28.501 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-15T21:31:28.852 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-15T21:31:28.853 INFO:tasks.workunit.client.0.smithi136.stderr:1' 2024-09-15T21:31:28.853 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-15T21:31:28.853 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T21:31:28.853 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-15T21:31:29.088 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=670014898178 2024-09-15T21:31:29.088 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 670014898178 2024-09-15T21:31:29.088 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-670014898178' 2024-09-15T21:31:29.088 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T21:31:29.088 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-15T21:31:29.220 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=644245094405 2024-09-15T21:31:29.221 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 644245094405 2024-09-15T21:31:29.221 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-670014898178 1-644245094405' 2024-09-15T21:31:29.221 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T21:31:29.221 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-670014898178 2024-09-15T21:31:29.221 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T21:31:29.222 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-15T21:31:29.222 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-670014898178 2024-09-15T21:31:29.222 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T21:31:29.223 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=670014898178 2024-09-15T21:31:29.223 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 670014898178' 2024-09-15T21:31:29.224 INFO:tasks.workunit.client.0.smithi136.stderr:waiting osd.0 seq 670014898178 2024-09-15T21:31:29.224 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T21:31:29.641 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 1 -lt 670014898178 2024-09-15T21:31:29.641 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-15T21:31:30.641 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-15T21:31:30.642 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T21:31:30.931 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 1 -lt 670014898178 2024-09-15T21:31:30.931 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-15T21:31:31.933 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 299 -eq 0 ']' 2024-09-15T21:31:31.933 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T21:31:32.219 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 670014898178 -lt 670014898178 2024-09-15T21:31:32.220 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T21:31:32.220 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-644245094405 2024-09-15T21:31:32.220 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T21:31:32.221 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-15T21:31:32.222 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-644245094405 2024-09-15T21:31:32.222 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T21:31:32.223 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=644245094405 2024-09-15T21:31:32.223 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 644245094405' 2024-09-15T21:31:32.223 INFO:tasks.workunit.client.0.smithi136.stderr:waiting osd.1 seq 644245094405 2024-09-15T21:31:32.223 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-15T21:31:32.502 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 644245094406 -lt 644245094405 2024-09-15T21:31:32.502 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-15T21:31:32.502 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T21:31:32.502 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T21:31:32.899 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 6 == 0 2024-09-15T21:31:32.899 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-15T21:31:32.900 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-15T21:31:32.900 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-15T21:31:32.900 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T21:31:32.900 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T21:31:32.900 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T21:31:32.900 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T21:31:33.174 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=3 2024-09-15T21:31:33.175 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-15T21:31:33.175 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T21:31:33.175 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T21:31:33.515 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 3 = 6 2024-09-15T21:31:33.515 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 3 '!=' -1 2024-09-15T21:31:33.515 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: loop=0 2024-09-15T21:31:33.515 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1652: wait_for_clean: num_active_clean=3 2024-09-15T21:31:33.515 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-15T21:31:33.515 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 0.1 2024-09-15T21:31:33.616 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-15T21:31:33.616 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-15T21:31:33.617 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-15T21:31:33.617 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-15T21:31:33.617 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T21:31:33.617 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T21:31:33.617 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T21:31:33.617 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T21:31:33.889 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=3 2024-09-15T21:31:33.889 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-15T21:31:33.889 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T21:31:33.889 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T21:31:34.235 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 3 = 6 2024-09-15T21:31:34.235 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 3 '!=' 3 2024-09-15T21:31:34.235 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: get_is_making_recovery_progress 2024-09-15T21:31:34.236 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2024-09-15T21:31:34.236 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-15T21:31:34.236 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-15T21:31:34.236 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-15T21:31:34.236 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2024-09-15T21:31:34.236 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-15T21:31:34.604 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=null 2024-09-15T21:31:34.604 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test null '!=' null 2024-09-15T21:31:34.604 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1655: wait_for_clean: (( 1 >= 13 )) 2024-09-15T21:31:34.604 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-15T21:31:34.604 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 0.2 2024-09-15T21:31:34.804 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-15T21:31:34.805 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-15T21:31:34.805 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-15T21:31:34.805 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-15T21:31:34.806 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T21:31:34.806 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T21:31:34.806 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T21:31:34.806 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T21:31:35.077 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=6 2024-09-15T21:31:35.078 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-15T21:31:35.078 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T21:31:35.078 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T21:31:35.426 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 = 6 2024-09-15T21:31:35.426 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-15T21:31:35.427 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-15T21:31:35.427 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1127: TEST_corrupt_scrub_replicated: objectstore_tool td/osd-scrub-repair 1 ROBJ17 set-bytes td/osd-scrub-repair/new.ROBJ17 2024-09-15T21:31:35.427 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2024-09-15T21:31:35.427 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2024-09-15T21:31:35.427 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=1 2024-09-15T21:31:35.427 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2024-09-15T21:31:35.427 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 ROBJ17 set-bytes td/osd-scrub-repair/new.ROBJ17 2024-09-15T21:31:35.427 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-09-15T21:31:35.427 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2024-09-15T21:31:35.427 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=1 2024-09-15T21:31:35.427 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2024-09-15T21:31:35.427 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-09-15T21:31:35.427 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-15T21:31:35.428 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-15T21:31:35.428 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-15T21:31:35.428 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-15T21:31:35.428 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-15T21:31:35.733 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-15T21:31:35.733 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 ROBJ17 set-bytes td/osd-scrub-repair/new.ROBJ17 2024-09-15T21:31:35.733 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-15T21:31:35.733 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-15T21:31:35.733 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=1 2024-09-15T21:31:35.733 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-15T21:31:35.733 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-09-15T21:31:35.733 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 ROBJ17 set-bytes td/osd-scrub-repair/new.ROBJ17 2024-09-15T21:31:36.990 INFO:tasks.workunit.client.0.smithi136.stderr:Error getting attr on : 3.0_head,#3:30259878:::ROBJ15:head#, (61) No data available 2024-09-15T21:31:36.990 INFO:tasks.workunit.client.0.smithi136.stderr:Error getting attr on : 3.0_head,#3:c0c86b1d:::ROBJ14:head#, (61) No data available 2024-09-15T21:31:37.539 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2024-09-15T21:31:37.540 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2024-09-15T21:31:37.540 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-15T21:31:37.540 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=1 2024-09-15T21:31:37.540 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-15T21:31:37.540 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-09-15T21:31:37.540 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=ce4892c7-6c6d-4b4f-8e6f-07a3e11133e5 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-15T21:31:37.540 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T21:31:37.540 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T21:31:37.540 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T21:31:37.540 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-09-15T21:31:37.541 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-09-15T21:31:37.541 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-15T21:31:37.541 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-15T21:31:37.541 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-15T21:31:37.541 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-15T21:31:37.541 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-15T21:31:37.541 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-15T21:31:37.541 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-15T21:31:37.541 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T21:31:37.541 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52602 2024-09-15T21:31:37.541 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.52602/$cluster-$name.asok' 2024-09-15T21:31:37.541 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.52602/$cluster-$name.asok' 2024-09-15T21:31:37.541 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-15T21:31:37.541 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-15T21:31:37.542 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-15T21:31:37.542 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T21:31:37.542 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T21:31:37.542 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T21:31:37.542 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T21:31:37.542 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-15T21:31:37.542 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-15T21:31:37.542 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-09-15T21:31:37.543 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.1 2024-09-15T21:31:37.543 INFO:tasks.workunit.client.0.smithi136.stderr:start osd.1 2024-09-15T21:31:37.543 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 1 --fsid=ce4892c7-6c6d-4b4f-8e6f-07a3e11133e5 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.52602/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T21:31:37.543 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-09-15T21:31:37.543 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 1 = 1 ']' 2024-09-15T21:31:37.543 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-15T21:31:37.543 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-15T21:31:37.544 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-15T21:31:37.571 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:31:37.569+0000 7fa6f97d9640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:31:37.574 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:31:37.571+0000 7fa6f97d9640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:31:37.575 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:31:37.573+0000 7fa6f97d9640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:31:37.830 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 1 2024-09-15T21:31:37.830 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-15T21:31:37.830 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2024-09-15T21:31:37.830 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-15T21:31:37.830 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-15T21:31:37.830 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:31:37.830 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-15T21:31:37.830 INFO:tasks.workunit.client.0.smithi136.stderr:0 2024-09-15T21:31:37.830 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:31:37.831 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T21:31:38.109 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:31:39.111 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:31:39.111 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:31:39.111 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-15T21:31:39.111 INFO:tasks.workunit.client.0.smithi136.stderr:1 2024-09-15T21:31:39.111 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:31:39.111 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T21:31:39.182 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:31:39.180+0000 7fa6f97d9640 -1 Falling back to public interface 2024-09-15T21:31:39.408 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:31:40.410 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:31:40.410 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:31:40.410 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-15T21:31:40.410 INFO:tasks.workunit.client.0.smithi136.stderr:2 2024-09-15T21:31:40.410 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:31:40.410 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T21:31:40.693 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:31:41.694 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:31:41.695 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:31:41.695 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-15T21:31:41.695 INFO:tasks.workunit.client.0.smithi136.stderr:3 2024-09-15T21:31:41.695 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:31:41.695 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T21:31:41.834 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:31:41.832+0000 7fa6f97d9640 -1 osd.1 157 log_to_monitors true 2024-09-15T21:31:42.022 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:31:43.023 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:31:43.023 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:31:43.024 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-15T21:31:43.024 INFO:tasks.workunit.client.0.smithi136.stderr:4 2024-09-15T21:31:43.024 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:31:43.024 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T21:31:43.320 INFO:tasks.workunit.client.0.smithi136.stderr:osd.1 up in weight 1 up_from 161 up_thru 161 down_at 158 last_clean_interval [150,157) [v2:127.0.0.1:6810/3465354686,v1:127.0.0.1:6811/3465354686] [v2:127.0.0.1:6812/3465354686,v1:127.0.0.1:6813/3465354686] exists,up 9aec06d1-33ff-42ab-be71-ac761d546d96 2024-09-15T21:31:43.320 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-15T21:31:43.320 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-15T21:31:43.320 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-15T21:31:43.320 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2024-09-15T21:31:43.320 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-15T21:31:43.320 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-15T21:31:43.320 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-15T21:31:43.320 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-15T21:31:43.321 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-15T21:31:43.321 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-15T21:31:43.321 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-15T21:31:43.321 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-15T21:31:43.321 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-15T21:31:43.414 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-15T21:31:43.414 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-15T21:31:43.414 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-15T21:31:43.414 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-15T21:31:43.414 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-15T21:31:43.414 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-15T21:31:43.727 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-15T21:31:43.727 INFO:tasks.workunit.client.0.smithi136.stderr:1' 2024-09-15T21:31:43.727 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-15T21:31:43.727 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T21:31:43.727 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-15T21:31:43.870 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=670014898182 2024-09-15T21:31:43.871 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 670014898182 2024-09-15T21:31:43.871 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-670014898182' 2024-09-15T21:31:43.871 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T21:31:43.871 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-15T21:31:43.973 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=691489734658 2024-09-15T21:31:43.973 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 691489734658 2024-09-15T21:31:43.973 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-670014898182 1-691489734658' 2024-09-15T21:31:43.973 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T21:31:43.974 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-670014898182 2024-09-15T21:31:43.974 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T21:31:43.974 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-15T21:31:43.975 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-670014898182 2024-09-15T21:31:43.975 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T21:31:43.976 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=670014898182 2024-09-15T21:31:43.976 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 670014898182' 2024-09-15T21:31:43.977 INFO:tasks.workunit.client.0.smithi136.stderr:waiting osd.0 seq 670014898182 2024-09-15T21:31:43.977 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T21:31:44.252 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 670014898180 -lt 670014898182 2024-09-15T21:31:44.253 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-15T21:31:45.254 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-15T21:31:45.254 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T21:31:45.532 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 670014898182 -lt 670014898182 2024-09-15T21:31:45.532 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T21:31:45.533 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-691489734658 2024-09-15T21:31:45.533 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T21:31:45.534 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-15T21:31:45.535 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-691489734658 2024-09-15T21:31:45.535 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T21:31:45.535 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=691489734658 2024-09-15T21:31:45.536 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 691489734658' 2024-09-15T21:31:45.536 INFO:tasks.workunit.client.0.smithi136.stderr:waiting osd.1 seq 691489734658 2024-09-15T21:31:45.536 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-15T21:31:45.810 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 691489734658 -lt 691489734658 2024-09-15T21:31:45.810 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-15T21:31:45.811 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T21:31:45.811 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T21:31:46.160 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 6 == 0 2024-09-15T21:31:46.160 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-15T21:31:46.160 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-15T21:31:46.160 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-15T21:31:46.160 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T21:31:46.160 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T21:31:46.160 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T21:31:46.161 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T21:31:46.433 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=6 2024-09-15T21:31:46.433 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-15T21:31:46.434 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T21:31:46.434 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T21:31:46.775 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 = 6 2024-09-15T21:31:46.775 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-15T21:31:46.775 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-15T21:31:46.775 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1029: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2024-09-15T21:31:46.775 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1030: TEST_corrupt_scrub_replicated: objname=ROBJ18 2024-09-15T21:31:46.775 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1033: TEST_corrupt_scrub_replicated: expr 18 % 2 2024-09-15T21:31:46.777 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1033: TEST_corrupt_scrub_replicated: local osd=0 2024-09-15T21:31:46.777 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1035: TEST_corrupt_scrub_replicated: case $i in 2024-09-15T21:31:46.778 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1132: TEST_corrupt_scrub_replicated: local payload=ROBJ18 2024-09-15T21:31:46.778 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1133: TEST_corrupt_scrub_replicated: echo ROBJ18 2024-09-15T21:31:46.778 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1134: TEST_corrupt_scrub_replicated: objectstore_tool td/osd-scrub-repair 0 ROBJ18 set-bytes td/osd-scrub-repair/new.ROBJ18 2024-09-15T21:31:46.778 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2024-09-15T21:31:46.778 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2024-09-15T21:31:46.778 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=0 2024-09-15T21:31:46.778 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2024-09-15T21:31:46.778 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 ROBJ18 set-bytes td/osd-scrub-repair/new.ROBJ18 2024-09-15T21:31:46.778 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-09-15T21:31:46.778 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2024-09-15T21:31:46.778 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=0 2024-09-15T21:31:46.778 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2024-09-15T21:31:46.778 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2024-09-15T21:31:46.779 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-15T21:31:46.779 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-15T21:31:46.779 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-15T21:31:46.779 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-15T21:31:46.779 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-15T21:31:47.084 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-15T21:31:47.084 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 ROBJ18 set-bytes td/osd-scrub-repair/new.ROBJ18 2024-09-15T21:31:47.084 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-15T21:31:47.084 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-15T21:31:47.085 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=0 2024-09-15T21:31:47.085 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-15T21:31:47.085 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-09-15T21:31:47.085 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 ROBJ18 set-bytes td/osd-scrub-repair/new.ROBJ18 2024-09-15T21:31:48.436 INFO:tasks.workunit.client.0.smithi136.stderr:Error decoding attr on : 3.0_head,#3:c0c86b1d:::ROBJ14:head#, (22) Invalid argument 2024-09-15T21:31:49.038 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 2024-09-15T21:31:49.038 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2024-09-15T21:31:49.038 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-15T21:31:49.038 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=0 2024-09-15T21:31:49.038 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-15T21:31:49.038 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-09-15T21:31:49.038 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=ce4892c7-6c6d-4b4f-8e6f-07a3e11133e5 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-15T21:31:49.039 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T21:31:49.039 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T21:31:49.039 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T21:31:49.039 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-09-15T21:31:49.039 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-09-15T21:31:49.039 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-15T21:31:49.039 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-15T21:31:49.039 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-15T21:31:49.039 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-15T21:31:49.039 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-15T21:31:49.039 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-15T21:31:49.039 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-15T21:31:49.039 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T21:31:49.040 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52602 2024-09-15T21:31:49.040 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.52602/$cluster-$name.asok' 2024-09-15T21:31:49.040 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.52602/$cluster-$name.asok' 2024-09-15T21:31:49.040 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-15T21:31:49.040 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-15T21:31:49.040 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-15T21:31:49.040 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T21:31:49.040 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T21:31:49.040 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T21:31:49.040 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T21:31:49.040 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-15T21:31:49.040 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-15T21:31:49.040 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-09-15T21:31:49.041 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.0 2024-09-15T21:31:49.041 INFO:tasks.workunit.client.0.smithi136.stderr:start osd.0 2024-09-15T21:31:49.041 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 0 --fsid=ce4892c7-6c6d-4b4f-8e6f-07a3e11133e5 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.52602/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T21:31:49.042 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-09-15T21:31:49.042 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 0 = 0 ']' 2024-09-15T21:31:49.042 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-15T21:31:49.042 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-15T21:31:49.042 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-15T21:31:49.069 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:31:49.067+0000 7fe7fca10640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:31:49.104 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:31:49.102+0000 7fe7fca10640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:31:49.105 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:31:49.103+0000 7fe7fca10640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:31:49.355 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 0 2024-09-15T21:31:49.355 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-15T21:31:49.356 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2024-09-15T21:31:49.356 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-15T21:31:49.356 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-15T21:31:49.356 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:31:49.356 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-15T21:31:49.356 INFO:tasks.workunit.client.0.smithi136.stderr:0 2024-09-15T21:31:49.356 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:31:49.356 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:31:49.633 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:31:50.226 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:31:50.224+0000 7fe7fca10640 -1 Falling back to public interface 2024-09-15T21:31:50.635 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:31:50.635 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:31:50.635 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-15T21:31:50.635 INFO:tasks.workunit.client.0.smithi136.stderr:1 2024-09-15T21:31:50.635 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:31:50.635 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:31:50.921 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:31:51.923 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:31:51.923 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:31:51.923 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-15T21:31:51.923 INFO:tasks.workunit.client.0.smithi136.stderr:2 2024-09-15T21:31:51.923 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:31:51.923 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:31:52.213 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:31:52.852 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:31:52.850+0000 7fe7fca10640 -1 osd.0 163 log_to_monitors true 2024-09-15T21:31:53.215 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:31:53.215 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:31:53.215 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-15T21:31:53.215 INFO:tasks.workunit.client.0.smithi136.stderr:3 2024-09-15T21:31:53.215 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:31:53.215 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:31:53.512 INFO:tasks.workunit.client.0.smithi136.stderr:osd.0 up in weight 1 up_from 167 up_thru 161 down_at 164 last_clean_interval [156,163) [v2:127.0.0.1:6802/3193570841,v1:127.0.0.1:6803/3193570841] [v2:127.0.0.1:6804/3193570841,v1:127.0.0.1:6805/3193570841] exists,up 7a68df98-8e0f-45ef-a9e9-b2a45ca8fd3f 2024-09-15T21:31:53.512 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-15T21:31:53.512 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-15T21:31:53.512 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-15T21:31:53.512 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2024-09-15T21:31:53.512 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-15T21:31:53.512 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-15T21:31:53.513 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-15T21:31:53.513 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-15T21:31:53.513 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-15T21:31:53.513 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-15T21:31:53.513 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-15T21:31:53.514 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-15T21:31:53.514 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-15T21:31:53.606 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-15T21:31:53.607 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-15T21:31:53.607 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-15T21:31:53.607 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-15T21:31:53.607 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-15T21:31:53.607 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-15T21:31:53.939 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-15T21:31:53.939 INFO:tasks.workunit.client.0.smithi136.stderr:1' 2024-09-15T21:31:53.939 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-15T21:31:53.939 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T21:31:53.939 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-15T21:31:54.121 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=717259538434 2024-09-15T21:31:54.121 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 717259538434 2024-09-15T21:31:54.121 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-717259538434' 2024-09-15T21:31:54.121 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T21:31:54.121 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-15T21:31:54.293 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=691489734661 2024-09-15T21:31:54.293 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 691489734661 2024-09-15T21:31:54.293 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-717259538434 1-691489734661' 2024-09-15T21:31:54.293 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T21:31:54.294 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-717259538434 2024-09-15T21:31:54.294 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T21:31:54.295 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-15T21:31:54.295 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-717259538434 2024-09-15T21:31:54.295 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T21:31:54.296 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=717259538434 2024-09-15T21:31:54.297 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 717259538434' 2024-09-15T21:31:54.297 INFO:tasks.workunit.client.0.smithi136.stderr:waiting osd.0 seq 717259538434 2024-09-15T21:31:54.297 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T21:31:54.665 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 670014898182 -lt 717259538434 2024-09-15T21:31:54.665 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-15T21:31:55.666 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-15T21:31:55.666 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T21:31:55.954 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 717259538434 -lt 717259538434 2024-09-15T21:31:55.954 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T21:31:55.955 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-691489734661 2024-09-15T21:31:55.955 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T21:31:55.955 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-15T21:31:55.956 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-691489734661 2024-09-15T21:31:55.956 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T21:31:55.957 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=691489734661 2024-09-15T21:31:55.957 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 691489734661' 2024-09-15T21:31:55.957 INFO:tasks.workunit.client.0.smithi136.stderr:waiting osd.1 seq 691489734661 2024-09-15T21:31:55.958 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-15T21:31:56.234 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 691489734661 -lt 691489734661 2024-09-15T21:31:56.235 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-15T21:31:56.235 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T21:31:56.235 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T21:31:56.578 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 6 == 0 2024-09-15T21:31:56.578 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-15T21:31:56.578 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-15T21:31:56.578 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-15T21:31:56.579 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T21:31:56.579 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T21:31:56.579 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T21:31:56.579 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T21:31:56.849 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=2 2024-09-15T21:31:56.850 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-15T21:31:56.850 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T21:31:56.850 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T21:31:57.199 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 2 = 6 2024-09-15T21:31:57.199 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 2 '!=' -1 2024-09-15T21:31:57.199 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: loop=0 2024-09-15T21:31:57.199 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1652: wait_for_clean: num_active_clean=2 2024-09-15T21:31:57.199 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-15T21:31:57.199 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 0.1 2024-09-15T21:31:57.300 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-15T21:31:57.300 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-15T21:31:57.300 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-15T21:31:57.301 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-15T21:31:57.301 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T21:31:57.301 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T21:31:57.301 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T21:31:57.301 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T21:31:57.586 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=2 2024-09-15T21:31:57.587 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-15T21:31:57.587 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T21:31:57.587 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T21:31:57.933 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 2 = 6 2024-09-15T21:31:57.933 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 2 '!=' 2 2024-09-15T21:31:57.933 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: get_is_making_recovery_progress 2024-09-15T21:31:57.933 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2024-09-15T21:31:57.933 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-15T21:31:57.933 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-15T21:31:57.933 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-15T21:31:57.933 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2024-09-15T21:31:57.934 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-15T21:31:58.277 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=null 2024-09-15T21:31:58.277 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test null '!=' null 2024-09-15T21:31:58.277 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1655: wait_for_clean: (( 1 >= 13 )) 2024-09-15T21:31:58.277 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-15T21:31:58.278 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 0.2 2024-09-15T21:31:58.478 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-15T21:31:58.478 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-15T21:31:58.479 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-15T21:31:58.479 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-15T21:31:58.479 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T21:31:58.479 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T21:31:58.479 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T21:31:58.479 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T21:31:58.750 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=2 2024-09-15T21:31:58.751 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-15T21:31:58.751 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T21:31:58.751 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T21:31:59.112 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 2 = 6 2024-09-15T21:31:59.112 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 2 '!=' 2 2024-09-15T21:31:59.112 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: get_is_making_recovery_progress 2024-09-15T21:31:59.112 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2024-09-15T21:31:59.112 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-15T21:31:59.112 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-15T21:31:59.112 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-15T21:31:59.113 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2024-09-15T21:31:59.113 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-15T21:31:59.478 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=null 2024-09-15T21:31:59.478 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test null '!=' null 2024-09-15T21:31:59.478 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1655: wait_for_clean: (( 2 >= 13 )) 2024-09-15T21:31:59.478 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-15T21:31:59.478 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 0.4 2024-09-15T21:31:59.879 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-15T21:31:59.879 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-15T21:31:59.880 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-15T21:31:59.880 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-15T21:31:59.880 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T21:31:59.880 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T21:31:59.880 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T21:31:59.880 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T21:32:00.153 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=6 2024-09-15T21:32:00.154 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-15T21:32:00.154 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T21:32:00.154 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T21:32:00.505 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 = 6 2024-09-15T21:32:00.505 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-15T21:32:00.505 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-15T21:32:00.505 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1135: TEST_corrupt_scrub_replicated: objectstore_tool td/osd-scrub-repair 1 ROBJ18 set-bytes td/osd-scrub-repair/new.ROBJ18 2024-09-15T21:32:00.505 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2024-09-15T21:32:00.505 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2024-09-15T21:32:00.505 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=1 2024-09-15T21:32:00.505 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2024-09-15T21:32:00.505 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 ROBJ18 set-bytes td/osd-scrub-repair/new.ROBJ18 2024-09-15T21:32:00.505 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-09-15T21:32:00.505 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2024-09-15T21:32:00.505 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=1 2024-09-15T21:32:00.506 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2024-09-15T21:32:00.506 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-09-15T21:32:00.506 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-15T21:32:00.506 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-15T21:32:00.506 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-15T21:32:00.506 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-15T21:32:00.506 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-15T21:32:00.811 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-15T21:32:00.811 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 ROBJ18 set-bytes td/osd-scrub-repair/new.ROBJ18 2024-09-15T21:32:00.811 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-15T21:32:00.811 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-15T21:32:00.811 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=1 2024-09-15T21:32:00.811 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-15T21:32:00.811 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-09-15T21:32:00.811 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 ROBJ18 set-bytes td/osd-scrub-repair/new.ROBJ18 2024-09-15T21:32:01.822 INFO:tasks.workunit.client.0.smithi136.stderr:Error getting attr on : 3.0_head,#3:30259878:::ROBJ15:head#, (61) No data available 2024-09-15T21:32:01.822 INFO:tasks.workunit.client.0.smithi136.stderr:Error getting attr on : 3.0_head,#3:c0c86b1d:::ROBJ14:head#, (61) No data available 2024-09-15T21:32:02.366 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2024-09-15T21:32:02.366 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2024-09-15T21:32:02.366 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-15T21:32:02.366 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=1 2024-09-15T21:32:02.366 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-15T21:32:02.366 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-09-15T21:32:02.366 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=ce4892c7-6c6d-4b4f-8e6f-07a3e11133e5 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-15T21:32:02.366 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T21:32:02.366 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T21:32:02.366 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T21:32:02.366 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-09-15T21:32:02.367 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-09-15T21:32:02.367 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-15T21:32:02.367 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-15T21:32:02.367 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-15T21:32:02.367 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-15T21:32:02.367 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-15T21:32:02.367 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-15T21:32:02.367 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-15T21:32:02.367 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T21:32:02.367 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52602 2024-09-15T21:32:02.367 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.52602/$cluster-$name.asok' 2024-09-15T21:32:02.367 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.52602/$cluster-$name.asok' 2024-09-15T21:32:02.367 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-15T21:32:02.367 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-15T21:32:02.368 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-15T21:32:02.368 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T21:32:02.368 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T21:32:02.368 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T21:32:02.368 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T21:32:02.368 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-15T21:32:02.368 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-15T21:32:02.368 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-09-15T21:32:02.369 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.1 2024-09-15T21:32:02.369 INFO:tasks.workunit.client.0.smithi136.stderr:start osd.1 2024-09-15T21:32:02.369 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 1 --fsid=ce4892c7-6c6d-4b4f-8e6f-07a3e11133e5 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.52602/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T21:32:02.369 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-09-15T21:32:02.369 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 1 = 1 ']' 2024-09-15T21:32:02.370 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-15T21:32:02.370 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-15T21:32:02.370 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-15T21:32:02.397 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:32:02.393+0000 7f44b500e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:32:02.400 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:32:02.396+0000 7f44b500e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:32:02.400 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:32:02.397+0000 7f44b500e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:32:02.658 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 1 2024-09-15T21:32:02.658 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-15T21:32:02.658 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2024-09-15T21:32:02.658 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-15T21:32:02.658 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-15T21:32:02.658 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:32:02.658 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-15T21:32:02.658 INFO:tasks.workunit.client.0.smithi136.stderr:0 2024-09-15T21:32:02.658 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:32:02.658 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T21:32:02.935 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:32:03.524 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:32:03.521+0000 7f44b500e640 -1 Falling back to public interface 2024-09-15T21:32:03.937 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:32:03.937 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:32:03.937 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-15T21:32:03.937 INFO:tasks.workunit.client.0.smithi136.stderr:1 2024-09-15T21:32:03.937 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:32:03.938 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T21:32:04.222 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:32:05.224 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:32:05.224 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:32:05.224 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-15T21:32:05.225 INFO:tasks.workunit.client.0.smithi136.stderr:2 2024-09-15T21:32:05.225 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:32:05.225 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T21:32:05.536 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:32:05.920 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:32:05.917+0000 7f44b500e640 -1 osd.1 169 log_to_monitors true 2024-09-15T21:32:06.537 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:32:06.537 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:32:06.537 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-15T21:32:06.537 INFO:tasks.workunit.client.0.smithi136.stderr:3 2024-09-15T21:32:06.537 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:32:06.537 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T21:32:06.958 INFO:tasks.workunit.client.0.smithi136.stderr:osd.1 up in weight 1 up_from 173 up_thru 173 down_at 170 last_clean_interval [161,169) [v2:127.0.0.1:6810/1145338380,v1:127.0.0.1:6811/1145338380] [v2:127.0.0.1:6812/1145338380,v1:127.0.0.1:6813/1145338380] exists,up 9aec06d1-33ff-42ab-be71-ac761d546d96 2024-09-15T21:32:06.958 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-15T21:32:06.958 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-15T21:32:06.958 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-15T21:32:06.958 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2024-09-15T21:32:06.958 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-15T21:32:06.958 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-15T21:32:06.958 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-15T21:32:06.958 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-15T21:32:06.958 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-15T21:32:06.958 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-15T21:32:06.958 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-15T21:32:06.959 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-15T21:32:06.959 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-15T21:32:07.051 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-15T21:32:07.051 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-15T21:32:07.051 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-15T21:32:07.051 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-15T21:32:07.051 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-15T21:32:07.051 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-15T21:32:07.498 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-15T21:32:07.498 INFO:tasks.workunit.client.0.smithi136.stderr:1' 2024-09-15T21:32:07.498 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-15T21:32:07.498 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T21:32:07.498 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-15T21:32:07.602 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=717259538437 2024-09-15T21:32:07.602 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 717259538437 2024-09-15T21:32:07.602 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-717259538437' 2024-09-15T21:32:07.603 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T21:32:07.603 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-15T21:32:07.707 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=743029342210 2024-09-15T21:32:07.707 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 743029342210 2024-09-15T21:32:07.708 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-717259538437 1-743029342210' 2024-09-15T21:32:07.708 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T21:32:07.708 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-717259538437 2024-09-15T21:32:07.708 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T21:32:07.708 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-15T21:32:07.709 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-717259538437 2024-09-15T21:32:07.709 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T21:32:07.710 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=717259538437 2024-09-15T21:32:07.710 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 717259538437' 2024-09-15T21:32:07.710 INFO:tasks.workunit.client.0.smithi136.stderr:waiting osd.0 seq 717259538437 2024-09-15T21:32:07.711 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T21:32:07.986 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 717259538436 -lt 717259538437 2024-09-15T21:32:07.986 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-15T21:32:08.987 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-15T21:32:08.987 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T21:32:09.304 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 717259538438 -lt 717259538437 2024-09-15T21:32:09.304 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T21:32:09.305 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-743029342210 2024-09-15T21:32:09.305 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T21:32:09.306 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-15T21:32:09.306 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-743029342210 2024-09-15T21:32:09.307 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T21:32:09.308 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=743029342210 2024-09-15T21:32:09.308 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 743029342210' 2024-09-15T21:32:09.308 INFO:tasks.workunit.client.0.smithi136.stderr:waiting osd.1 seq 743029342210 2024-09-15T21:32:09.308 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-15T21:32:09.586 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 743029342210 -lt 743029342210 2024-09-15T21:32:09.586 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-15T21:32:09.586 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T21:32:09.586 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T21:32:09.929 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 6 == 0 2024-09-15T21:32:09.929 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-15T21:32:09.930 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-15T21:32:09.930 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-15T21:32:09.930 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T21:32:09.930 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T21:32:09.930 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T21:32:09.930 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T21:32:10.204 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=6 2024-09-15T21:32:10.205 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-15T21:32:10.205 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T21:32:10.205 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T21:32:10.550 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 = 6 2024-09-15T21:32:10.550 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-15T21:32:10.550 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-15T21:32:10.550 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1137: TEST_corrupt_scrub_replicated: objectstore_tool td/osd-scrub-repair 1 ROBJ18 corrupt-info 2024-09-15T21:32:10.550 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2024-09-15T21:32:10.550 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2024-09-15T21:32:10.550 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=1 2024-09-15T21:32:10.551 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2024-09-15T21:32:10.551 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 ROBJ18 corrupt-info 2024-09-15T21:32:10.551 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-09-15T21:32:10.551 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2024-09-15T21:32:10.551 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=1 2024-09-15T21:32:10.551 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2024-09-15T21:32:10.551 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-09-15T21:32:10.551 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-15T21:32:10.551 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-15T21:32:10.551 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-15T21:32:10.551 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-15T21:32:10.551 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-15T21:32:10.856 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-15T21:32:10.856 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 ROBJ18 corrupt-info 2024-09-15T21:32:10.856 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-15T21:32:10.857 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-15T21:32:10.857 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=1 2024-09-15T21:32:10.857 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-15T21:32:10.857 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-09-15T21:32:10.857 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 ROBJ18 corrupt-info 2024-09-15T21:32:11.878 INFO:tasks.workunit.client.0.smithi136.stderr:Error getting attr on : 3.0_head,#3:30259878:::ROBJ15:head#, (61) No data available 2024-09-15T21:32:11.878 INFO:tasks.workunit.client.0.smithi136.stderr:Error getting attr on : 3.0_head,#3:c0c86b1d:::ROBJ14:head#, (61) No data available 2024-09-15T21:32:12.396 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2024-09-15T21:32:12.396 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2024-09-15T21:32:12.396 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-15T21:32:12.396 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=1 2024-09-15T21:32:12.396 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-15T21:32:12.396 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-09-15T21:32:12.397 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=ce4892c7-6c6d-4b4f-8e6f-07a3e11133e5 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-15T21:32:12.397 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T21:32:12.397 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T21:32:12.397 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T21:32:12.397 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-09-15T21:32:12.397 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-09-15T21:32:12.397 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-15T21:32:12.397 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-15T21:32:12.397 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-15T21:32:12.397 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-15T21:32:12.397 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-15T21:32:12.397 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-15T21:32:12.398 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-15T21:32:12.398 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T21:32:12.398 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52602 2024-09-15T21:32:12.398 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.52602/$cluster-$name.asok' 2024-09-15T21:32:12.398 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.52602/$cluster-$name.asok' 2024-09-15T21:32:12.398 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-15T21:32:12.398 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-15T21:32:12.398 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-15T21:32:12.398 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T21:32:12.398 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T21:32:12.398 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T21:32:12.398 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T21:32:12.398 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-15T21:32:12.398 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-15T21:32:12.399 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-09-15T21:32:12.400 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.1 2024-09-15T21:32:12.400 INFO:tasks.workunit.client.0.smithi136.stderr:start osd.1 2024-09-15T21:32:12.400 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 1 --fsid=ce4892c7-6c6d-4b4f-8e6f-07a3e11133e5 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.52602/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T21:32:12.400 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-09-15T21:32:12.400 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 1 = 1 ']' 2024-09-15T21:32:12.401 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-15T21:32:12.401 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-15T21:32:12.401 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-15T21:32:12.427 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:32:12.425+0000 7f713c3a6640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:32:12.429 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:32:12.427+0000 7f713c3a6640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:32:12.430 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:32:12.428+0000 7f713c3a6640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:32:12.685 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 1 2024-09-15T21:32:12.685 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-15T21:32:12.685 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2024-09-15T21:32:12.685 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-15T21:32:12.685 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-15T21:32:12.685 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:32:12.685 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-15T21:32:12.685 INFO:tasks.workunit.client.0.smithi136.stderr:0 2024-09-15T21:32:12.686 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:32:12.686 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T21:32:12.959 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:32:13.532 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:32:13.530+0000 7f713c3a6640 -1 Falling back to public interface 2024-09-15T21:32:13.961 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:32:13.961 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:32:13.961 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-15T21:32:13.961 INFO:tasks.workunit.client.0.smithi136.stderr:1 2024-09-15T21:32:13.961 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:32:13.961 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T21:32:14.249 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:32:15.250 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:32:15.250 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:32:15.251 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-15T21:32:15.251 INFO:tasks.workunit.client.0.smithi136.stderr:2 2024-09-15T21:32:15.251 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:32:15.251 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T21:32:15.552 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:32:15.913 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:32:15.910+0000 7f713c3a6640 -1 osd.1 175 log_to_monitors true 2024-09-15T21:32:16.554 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:32:16.554 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:32:16.554 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-15T21:32:16.554 INFO:tasks.workunit.client.0.smithi136.stderr:3 2024-09-15T21:32:16.554 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:32:16.555 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T21:32:16.872 INFO:tasks.workunit.client.0.smithi136.stderr:osd.1 up in weight 1 up_from 179 up_thru 179 down_at 176 last_clean_interval [173,175) [v2:127.0.0.1:6810/2399599615,v1:127.0.0.1:6811/2399599615] [v2:127.0.0.1:6812/2399599615,v1:127.0.0.1:6813/2399599615] exists,up 9aec06d1-33ff-42ab-be71-ac761d546d96 2024-09-15T21:32:16.872 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-15T21:32:16.872 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-15T21:32:16.872 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-15T21:32:16.872 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2024-09-15T21:32:16.872 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-15T21:32:16.872 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-15T21:32:16.872 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-15T21:32:16.872 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-15T21:32:16.873 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-15T21:32:16.873 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-15T21:32:16.873 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-15T21:32:16.873 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-15T21:32:16.873 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-15T21:32:16.966 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-15T21:32:16.966 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-15T21:32:16.966 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-15T21:32:16.966 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-15T21:32:16.966 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-15T21:32:16.966 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-15T21:32:17.480 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-15T21:32:17.480 INFO:tasks.workunit.client.0.smithi136.stderr:1' 2024-09-15T21:32:17.480 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-15T21:32:17.480 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T21:32:17.481 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-15T21:32:17.585 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=717259538440 2024-09-15T21:32:17.585 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 717259538440 2024-09-15T21:32:17.585 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-717259538440' 2024-09-15T21:32:17.585 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T21:32:17.585 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-15T21:32:17.712 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=768799145986 2024-09-15T21:32:17.712 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 768799145986 2024-09-15T21:32:17.712 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-717259538440 1-768799145986' 2024-09-15T21:32:17.712 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T21:32:17.713 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-717259538440 2024-09-15T21:32:17.713 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T21:32:17.713 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-15T21:32:17.714 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-717259538440 2024-09-15T21:32:17.714 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T21:32:17.715 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=717259538440 2024-09-15T21:32:17.715 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 717259538440' 2024-09-15T21:32:17.715 INFO:tasks.workunit.client.0.smithi136.stderr:waiting osd.0 seq 717259538440 2024-09-15T21:32:17.716 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T21:32:17.993 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 717259538439 -lt 717259538440 2024-09-15T21:32:17.993 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-15T21:32:18.994 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-15T21:32:18.994 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T21:32:19.319 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 717259538441 -lt 717259538440 2024-09-15T21:32:19.319 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T21:32:19.319 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-768799145986 2024-09-15T21:32:19.319 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T21:32:19.320 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-15T21:32:19.321 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-768799145986 2024-09-15T21:32:19.321 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T21:32:19.322 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=768799145986 2024-09-15T21:32:19.322 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 768799145986' 2024-09-15T21:32:19.322 INFO:tasks.workunit.client.0.smithi136.stderr:waiting osd.1 seq 768799145986 2024-09-15T21:32:19.322 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-15T21:32:19.600 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 768799145986 -lt 768799145986 2024-09-15T21:32:19.601 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-15T21:32:19.601 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T21:32:19.601 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T21:32:19.974 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 6 == 0 2024-09-15T21:32:19.974 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-15T21:32:19.974 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-15T21:32:19.975 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-15T21:32:19.975 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T21:32:19.975 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T21:32:19.975 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T21:32:19.975 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T21:32:20.246 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=6 2024-09-15T21:32:20.247 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-15T21:32:20.247 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T21:32:20.247 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T21:32:20.593 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 = 6 2024-09-15T21:32:20.593 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-15T21:32:20.593 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-15T21:32:20.593 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1029: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2024-09-15T21:32:20.593 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1030: TEST_corrupt_scrub_replicated: objname=ROBJ19 2024-09-15T21:32:20.593 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1033: TEST_corrupt_scrub_replicated: expr 19 % 2 2024-09-15T21:32:20.594 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1033: TEST_corrupt_scrub_replicated: local osd=1 2024-09-15T21:32:20.594 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1035: TEST_corrupt_scrub_replicated: case $i in 2024-09-15T21:32:20.595 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1146: TEST_corrupt_scrub_replicated: get_pg csr_pool ROBJ0 2024-09-15T21:32:20.595 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1084: get_pg: local poolname=csr_pool 2024-09-15T21:32:20.595 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1085: get_pg: local objectname=ROBJ0 2024-09-15T21:32:20.595 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1087: get_pg: ceph --format json osd map csr_pool ROBJ0 2024-09-15T21:32:20.595 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1087: get_pg: jq -r .pgid 2024-09-15T21:32:20.872 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1146: TEST_corrupt_scrub_replicated: local pg=3.0 2024-09-15T21:32:20.872 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1148: TEST_corrupt_scrub_replicated: ceph tell 'osd.*' injectargs -- --osd-max-object-size=1048576 2024-09-15T21:32:20.966 INFO:tasks.workunit.client.0.smithi136.stdout:osd.0: {} 2024-09-15T21:32:20.967 INFO:tasks.workunit.client.0.smithi136.stderr:osd.0: osd_delete_sleep = '' osd_delete_sleep_hdd = '' osd_delete_sleep_hybrid = '' osd_delete_sleep_ssd = '' osd_max_backfills = '' osd_max_object_size = '' osd_recovery_max_active = '' osd_recovery_max_active_hdd = '' osd_recovery_max_active_ssd = '' osd_recovery_sleep = '' osd_recovery_sleep_hdd = '' osd_recovery_sleep_hybrid = '' osd_recovery_sleep_ssd = '' osd_scrub_sleep = '' osd_snap_trim_sleep = '' osd_snap_trim_sleep_hdd = '' osd_snap_trim_sleep_hybrid = '' osd_snap_trim_sleep_ssd = '' 2024-09-15T21:32:20.983 INFO:tasks.workunit.client.0.smithi136.stdout:osd.1: {} 2024-09-15T21:32:20.983 INFO:tasks.workunit.client.0.smithi136.stderr:osd.1: osd_delete_sleep = '' osd_delete_sleep_hdd = '' osd_delete_sleep_hybrid = '' osd_delete_sleep_ssd = '' osd_max_backfills = '' osd_max_object_size = '' osd_recovery_max_active = '' osd_recovery_max_active_hdd = '' osd_recovery_max_active_ssd = '' osd_recovery_sleep = '' osd_recovery_sleep_hdd = '' osd_recovery_sleep_hybrid = '' osd_recovery_sleep_ssd = '' osd_scrub_sleep = '' osd_snap_trim_sleep = '' osd_snap_trim_sleep_hdd = '' osd_snap_trim_sleep_hybrid = '' osd_snap_trim_sleep_ssd = '' 2024-09-15T21:32:20.993 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1150: TEST_corrupt_scrub_replicated: inject_eio rep data csr_pool ROBJ11 td/osd-scrub-repair 0 2024-09-15T21:32:20.994 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2402: inject_eio: local pooltype=rep 2024-09-15T21:32:20.994 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2403: inject_eio: shift 2024-09-15T21:32:20.994 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2404: inject_eio: local which=data 2024-09-15T21:32:20.994 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2405: inject_eio: shift 2024-09-15T21:32:20.994 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2406: inject_eio: local poolname=csr_pool 2024-09-15T21:32:20.994 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2407: inject_eio: shift 2024-09-15T21:32:20.994 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2408: inject_eio: local objname=ROBJ11 2024-09-15T21:32:20.994 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2409: inject_eio: shift 2024-09-15T21:32:20.994 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2410: inject_eio: local dir=td/osd-scrub-repair 2024-09-15T21:32:20.994 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2411: inject_eio: shift 2024-09-15T21:32:20.994 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2412: inject_eio: local shard_id=0 2024-09-15T21:32:20.994 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2413: inject_eio: shift 2024-09-15T21:32:20.994 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: get_osds csr_pool ROBJ11 2024-09-15T21:32:20.994 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1012: get_osds: local poolname=csr_pool 2024-09-15T21:32:20.995 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1013: get_osds: local objectname=ROBJ11 2024-09-15T21:32:20.995 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: ceph --format json osd map csr_pool ROBJ11 2024-09-15T21:32:20.995 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: jq '.acting | .[]' 2024-09-15T21:32:21.305 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: local 'osds=1 2024-09-15T21:32:21.305 INFO:tasks.workunit.client.0.smithi136.stderr:0' 2024-09-15T21:32:21.306 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1018: get_osds: echo 1 0 2024-09-15T21:32:21.306 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: initial_osds=('1' '0') 2024-09-15T21:32:21.306 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: local -a initial_osds 2024-09-15T21:32:21.306 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2416: inject_eio: local osd_id=1 2024-09-15T21:32:21.306 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2417: inject_eio: '[' rep '!=' ec ']' 2024-09-15T21:32:21.306 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2418: inject_eio: shard_id= 2024-09-15T21:32:21.307 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2420: inject_eio: cat td/osd-scrub-repair/1/type 2024-09-15T21:32:21.309 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2420: inject_eio: type=bluestore 2024-09-15T21:32:21.309 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2421: inject_eio: set_config osd 1 bluestore_debug_inject_read_err true 2024-09-15T21:32:21.309 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1152: set_config: local daemon=osd 2024-09-15T21:32:21.309 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1153: set_config: local id=1 2024-09-15T21:32:21.309 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1154: set_config: local config=bluestore_debug_inject_read_err 2024-09-15T21:32:21.309 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1155: set_config: local value=true 2024-09-15T21:32:21.310 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: jq 'has("success")' 2024-09-15T21:32:21.310 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: get_asok_path osd.1 2024-09-15T21:32:21.310 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.1 2024-09-15T21:32:21.310 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.1 ']' 2024-09-15T21:32:21.310 INFO:tasks.workunit.client.0.smithi136.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-15T21:32:21.310 INFO:tasks.workunit.client.0.smithi136.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T21:32:21.310 INFO:tasks.workunit.client.0.smithi136.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52602 2024-09-15T21:32:21.310 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.52602/ceph-osd.1.asok 2024-09-15T21:32:21.310 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.52602/ceph-osd.1.asok config set bluestore_debug_inject_read_err true 2024-09-15T21:32:21.395 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: test true == true 2024-09-15T21:32:21.395 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2422: inject_eio: local loop=0 2024-09-15T21:32:21.396 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2424: inject_eio: grep -q Invalid 2024-09-15T21:32:21.396 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: get_asok_path osd.1 2024-09-15T21:32:21.396 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.1 2024-09-15T21:32:21.396 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.1 ']' 2024-09-15T21:32:21.396 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-15T21:32:21.396 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T21:32:21.396 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52602 2024-09-15T21:32:21.396 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.52602/ceph-osd.1.asok 2024-09-15T21:32:21.397 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: CEPH_ARGS= 2024-09-15T21:32:21.397 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: ceph --admin-daemon /tmp/ceph-asok.52602/ceph-osd.1.asok injectdataerr csr_pool ROBJ11 2024-09-15T21:32:21.484 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1151: TEST_corrupt_scrub_replicated: inject_eio rep mdata csr_pool ROBJ12 td/osd-scrub-repair 1 2024-09-15T21:32:21.484 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2402: inject_eio: local pooltype=rep 2024-09-15T21:32:21.484 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2403: inject_eio: shift 2024-09-15T21:32:21.484 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2404: inject_eio: local which=mdata 2024-09-15T21:32:21.484 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2405: inject_eio: shift 2024-09-15T21:32:21.484 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2406: inject_eio: local poolname=csr_pool 2024-09-15T21:32:21.485 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2407: inject_eio: shift 2024-09-15T21:32:21.485 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2408: inject_eio: local objname=ROBJ12 2024-09-15T21:32:21.485 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2409: inject_eio: shift 2024-09-15T21:32:21.485 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2410: inject_eio: local dir=td/osd-scrub-repair 2024-09-15T21:32:21.485 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2411: inject_eio: shift 2024-09-15T21:32:21.485 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2412: inject_eio: local shard_id=1 2024-09-15T21:32:21.485 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2413: inject_eio: shift 2024-09-15T21:32:21.485 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: get_osds csr_pool ROBJ12 2024-09-15T21:32:21.485 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1012: get_osds: local poolname=csr_pool 2024-09-15T21:32:21.485 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1013: get_osds: local objectname=ROBJ12 2024-09-15T21:32:21.485 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: ceph --format json osd map csr_pool ROBJ12 2024-09-15T21:32:21.485 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: jq '.acting | .[]' 2024-09-15T21:32:21.761 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: local 'osds=1 2024-09-15T21:32:21.762 INFO:tasks.workunit.client.0.smithi136.stderr:0' 2024-09-15T21:32:21.762 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1018: get_osds: echo 1 0 2024-09-15T21:32:21.762 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: initial_osds=('1' '0') 2024-09-15T21:32:21.762 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: local -a initial_osds 2024-09-15T21:32:21.762 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2416: inject_eio: local osd_id=0 2024-09-15T21:32:21.762 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2417: inject_eio: '[' rep '!=' ec ']' 2024-09-15T21:32:21.762 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2418: inject_eio: shard_id= 2024-09-15T21:32:21.762 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2420: inject_eio: cat td/osd-scrub-repair/0/type 2024-09-15T21:32:21.763 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2420: inject_eio: type=bluestore 2024-09-15T21:32:21.763 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2421: inject_eio: set_config osd 0 bluestore_debug_inject_read_err true 2024-09-15T21:32:21.763 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1152: set_config: local daemon=osd 2024-09-15T21:32:21.763 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1153: set_config: local id=0 2024-09-15T21:32:21.763 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1154: set_config: local config=bluestore_debug_inject_read_err 2024-09-15T21:32:21.763 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1155: set_config: local value=true 2024-09-15T21:32:21.764 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: jq 'has("success")' 2024-09-15T21:32:21.764 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: get_asok_path osd.0 2024-09-15T21:32:21.764 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.0 2024-09-15T21:32:21.765 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.0 ']' 2024-09-15T21:32:21.765 INFO:tasks.workunit.client.0.smithi136.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-15T21:32:21.765 INFO:tasks.workunit.client.0.smithi136.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T21:32:21.765 INFO:tasks.workunit.client.0.smithi136.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52602 2024-09-15T21:32:21.765 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.52602/ceph-osd.0.asok 2024-09-15T21:32:21.765 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.52602/ceph-osd.0.asok config set bluestore_debug_inject_read_err true 2024-09-15T21:32:21.850 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: test true == true 2024-09-15T21:32:21.851 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2422: inject_eio: local loop=0 2024-09-15T21:32:21.852 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2424: inject_eio: grep -q Invalid 2024-09-15T21:32:21.852 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: get_asok_path osd.0 2024-09-15T21:32:21.852 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.0 2024-09-15T21:32:21.852 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.0 ']' 2024-09-15T21:32:21.852 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-15T21:32:21.852 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T21:32:21.852 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52602 2024-09-15T21:32:21.852 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.52602/ceph-osd.0.asok 2024-09-15T21:32:21.853 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: CEPH_ARGS= 2024-09-15T21:32:21.853 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: ceph --admin-daemon /tmp/ceph-asok.52602/ceph-osd.0.asok injectmdataerr csr_pool ROBJ12 2024-09-15T21:32:21.939 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1152: TEST_corrupt_scrub_replicated: inject_eio rep mdata csr_pool ROBJ13 td/osd-scrub-repair 1 2024-09-15T21:32:21.939 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2402: inject_eio: local pooltype=rep 2024-09-15T21:32:21.939 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2403: inject_eio: shift 2024-09-15T21:32:21.940 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2404: inject_eio: local which=mdata 2024-09-15T21:32:21.940 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2405: inject_eio: shift 2024-09-15T21:32:21.940 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2406: inject_eio: local poolname=csr_pool 2024-09-15T21:32:21.940 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2407: inject_eio: shift 2024-09-15T21:32:21.940 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2408: inject_eio: local objname=ROBJ13 2024-09-15T21:32:21.940 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2409: inject_eio: shift 2024-09-15T21:32:21.940 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2410: inject_eio: local dir=td/osd-scrub-repair 2024-09-15T21:32:21.940 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2411: inject_eio: shift 2024-09-15T21:32:21.940 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2412: inject_eio: local shard_id=1 2024-09-15T21:32:21.940 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2413: inject_eio: shift 2024-09-15T21:32:21.940 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: get_osds csr_pool ROBJ13 2024-09-15T21:32:21.940 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1012: get_osds: local poolname=csr_pool 2024-09-15T21:32:21.940 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1013: get_osds: local objectname=ROBJ13 2024-09-15T21:32:21.941 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: ceph --format json osd map csr_pool ROBJ13 2024-09-15T21:32:21.941 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: jq '.acting | .[]' 2024-09-15T21:32:22.218 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: local 'osds=1 2024-09-15T21:32:22.218 INFO:tasks.workunit.client.0.smithi136.stderr:0' 2024-09-15T21:32:22.218 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1018: get_osds: echo 1 0 2024-09-15T21:32:22.219 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: initial_osds=('1' '0') 2024-09-15T21:32:22.219 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: local -a initial_osds 2024-09-15T21:32:22.219 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2416: inject_eio: local osd_id=0 2024-09-15T21:32:22.219 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2417: inject_eio: '[' rep '!=' ec ']' 2024-09-15T21:32:22.219 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2418: inject_eio: shard_id= 2024-09-15T21:32:22.219 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2420: inject_eio: cat td/osd-scrub-repair/0/type 2024-09-15T21:32:22.221 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2420: inject_eio: type=bluestore 2024-09-15T21:32:22.221 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2421: inject_eio: set_config osd 0 bluestore_debug_inject_read_err true 2024-09-15T21:32:22.221 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1152: set_config: local daemon=osd 2024-09-15T21:32:22.221 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1153: set_config: local id=0 2024-09-15T21:32:22.221 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1154: set_config: local config=bluestore_debug_inject_read_err 2024-09-15T21:32:22.221 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1155: set_config: local value=true 2024-09-15T21:32:22.221 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: jq 'has("success")' 2024-09-15T21:32:22.222 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: get_asok_path osd.0 2024-09-15T21:32:22.222 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.0 2024-09-15T21:32:22.222 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.0 ']' 2024-09-15T21:32:22.222 INFO:tasks.workunit.client.0.smithi136.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-15T21:32:22.222 INFO:tasks.workunit.client.0.smithi136.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T21:32:22.222 INFO:tasks.workunit.client.0.smithi136.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52602 2024-09-15T21:32:22.222 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.52602/ceph-osd.0.asok 2024-09-15T21:32:22.222 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.52602/ceph-osd.0.asok config set bluestore_debug_inject_read_err true 2024-09-15T21:32:22.306 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: test true == true 2024-09-15T21:32:22.306 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2422: inject_eio: local loop=0 2024-09-15T21:32:22.307 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2424: inject_eio: grep -q Invalid 2024-09-15T21:32:22.307 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: get_asok_path osd.0 2024-09-15T21:32:22.307 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.0 2024-09-15T21:32:22.307 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.0 ']' 2024-09-15T21:32:22.307 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-15T21:32:22.307 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T21:32:22.307 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52602 2024-09-15T21:32:22.307 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.52602/ceph-osd.0.asok 2024-09-15T21:32:22.308 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: CEPH_ARGS= 2024-09-15T21:32:22.308 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: ceph --admin-daemon /tmp/ceph-asok.52602/ceph-osd.0.asok injectmdataerr csr_pool ROBJ13 2024-09-15T21:32:22.395 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1153: TEST_corrupt_scrub_replicated: inject_eio rep data csr_pool ROBJ13 td/osd-scrub-repair 0 2024-09-15T21:32:22.395 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2402: inject_eio: local pooltype=rep 2024-09-15T21:32:22.395 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2403: inject_eio: shift 2024-09-15T21:32:22.396 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2404: inject_eio: local which=data 2024-09-15T21:32:22.396 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2405: inject_eio: shift 2024-09-15T21:32:22.396 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2406: inject_eio: local poolname=csr_pool 2024-09-15T21:32:22.396 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2407: inject_eio: shift 2024-09-15T21:32:22.396 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2408: inject_eio: local objname=ROBJ13 2024-09-15T21:32:22.396 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2409: inject_eio: shift 2024-09-15T21:32:22.396 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2410: inject_eio: local dir=td/osd-scrub-repair 2024-09-15T21:32:22.396 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2411: inject_eio: shift 2024-09-15T21:32:22.396 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2412: inject_eio: local shard_id=0 2024-09-15T21:32:22.396 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2413: inject_eio: shift 2024-09-15T21:32:22.396 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: get_osds csr_pool ROBJ13 2024-09-15T21:32:22.397 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1012: get_osds: local poolname=csr_pool 2024-09-15T21:32:22.397 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1013: get_osds: local objectname=ROBJ13 2024-09-15T21:32:22.397 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: ceph --format json osd map csr_pool ROBJ13 2024-09-15T21:32:22.397 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: jq '.acting | .[]' 2024-09-15T21:32:22.690 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: local 'osds=1 2024-09-15T21:32:22.690 INFO:tasks.workunit.client.0.smithi136.stderr:0' 2024-09-15T21:32:22.690 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1018: get_osds: echo 1 0 2024-09-15T21:32:22.690 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: initial_osds=('1' '0') 2024-09-15T21:32:22.690 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: local -a initial_osds 2024-09-15T21:32:22.690 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2416: inject_eio: local osd_id=1 2024-09-15T21:32:22.690 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2417: inject_eio: '[' rep '!=' ec ']' 2024-09-15T21:32:22.691 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2418: inject_eio: shard_id= 2024-09-15T21:32:22.691 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2420: inject_eio: cat td/osd-scrub-repair/1/type 2024-09-15T21:32:22.693 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2420: inject_eio: type=bluestore 2024-09-15T21:32:22.693 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2421: inject_eio: set_config osd 1 bluestore_debug_inject_read_err true 2024-09-15T21:32:22.693 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1152: set_config: local daemon=osd 2024-09-15T21:32:22.694 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1153: set_config: local id=1 2024-09-15T21:32:22.694 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1154: set_config: local config=bluestore_debug_inject_read_err 2024-09-15T21:32:22.694 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1155: set_config: local value=true 2024-09-15T21:32:22.694 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: jq 'has("success")' 2024-09-15T21:32:22.694 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: get_asok_path osd.1 2024-09-15T21:32:22.694 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.1 2024-09-15T21:32:22.694 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.1 ']' 2024-09-15T21:32:22.694 INFO:tasks.workunit.client.0.smithi136.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-15T21:32:22.695 INFO:tasks.workunit.client.0.smithi136.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T21:32:22.695 INFO:tasks.workunit.client.0.smithi136.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52602 2024-09-15T21:32:22.695 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.52602/ceph-osd.1.asok 2024-09-15T21:32:22.695 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.52602/ceph-osd.1.asok config set bluestore_debug_inject_read_err true 2024-09-15T21:32:22.777 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: test true == true 2024-09-15T21:32:22.777 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2422: inject_eio: local loop=0 2024-09-15T21:32:22.778 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2424: inject_eio: grep -q Invalid 2024-09-15T21:32:22.778 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: get_asok_path osd.1 2024-09-15T21:32:22.779 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.1 2024-09-15T21:32:22.779 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.1 ']' 2024-09-15T21:32:22.779 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-15T21:32:22.779 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T21:32:22.779 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52602 2024-09-15T21:32:22.779 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.52602/ceph-osd.1.asok 2024-09-15T21:32:22.779 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: CEPH_ARGS= 2024-09-15T21:32:22.779 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: ceph --admin-daemon /tmp/ceph-asok.52602/ceph-osd.1.asok injectdataerr csr_pool ROBJ13 2024-09-15T21:32:22.865 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1155: TEST_corrupt_scrub_replicated: pg_scrub 3.0 2024-09-15T21:32:22.865 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1883: pg_scrub: local pgid=3.0 2024-09-15T21:32:22.866 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1884: pg_scrub: get_last_scrub_stamp 3.0 2024-09-15T21:32:22.866 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=3.0 2024-09-15T21:32:22.866 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-15T21:32:22.866 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-15T21:32:22.866 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_scrub_stamp' 2024-09-15T21:32:23.139 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1884: pg_scrub: local last_scrub=2024-09-15T21:27:56.080190+0000 2024-09-15T21:32:23.139 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1885: pg_scrub: ceph pg scrub 3.0 2024-09-15T21:32:23.408 INFO:tasks.workunit.client.0.smithi136.stderr:instructing pg 3.0 on osd.1 to scrub 2024-09-15T21:32:23.420 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1886: pg_scrub: wait_for_scrub 3.0 2024-09-15T21:27:56.080190+0000 2024-09-15T21:32:23.420 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: local pgid=3.0 2024-09-15T21:32:23.420 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2018: wait_for_scrub: local last_scrub=2024-09-15T21:27:56.080190+0000 2024-09-15T21:32:23.420 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2019: wait_for_scrub: local sname=last_scrub_stamp 2024-09-15T21:32:23.420 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i=0 )) 2024-09-15T21:32:23.421 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-15T21:32:23.421 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 3.0 last_scrub_stamp 2024-09-15T21:32:23.421 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=3.0 2024-09-15T21:32:23.421 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-15T21:32:23.422 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-15T21:32:23.422 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_scrub_stamp' 2024-09-15T21:32:23.692 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-15T21:27:56.080190+0000 '>' 2024-09-15T21:27:56.080190+0000 2024-09-15T21:32:23.692 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-15T21:32:24.697 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-15T21:32:24.697 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-15T21:32:24.697 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 3.0 last_scrub_stamp 2024-09-15T21:32:24.697 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=3.0 2024-09-15T21:32:24.697 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-15T21:32:24.697 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-15T21:32:24.697 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_scrub_stamp' 2024-09-15T21:32:24.978 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-15T21:27:56.080190+0000 '>' 2024-09-15T21:27:56.080190+0000 2024-09-15T21:32:24.978 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-15T21:32:25.979 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-15T21:32:25.979 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-15T21:32:25.980 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 3.0 last_scrub_stamp 2024-09-15T21:32:25.980 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=3.0 2024-09-15T21:32:25.980 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-15T21:32:25.980 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-15T21:32:25.980 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_scrub_stamp' 2024-09-15T21:32:26.260 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-15T21:27:56.080190+0000 '>' 2024-09-15T21:27:56.080190+0000 2024-09-15T21:32:26.260 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-15T21:32:27.261 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-15T21:32:27.261 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-15T21:32:27.262 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 3.0 last_scrub_stamp 2024-09-15T21:32:27.262 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=3.0 2024-09-15T21:32:27.262 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-15T21:32:27.262 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-15T21:32:27.262 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_scrub_stamp' 2024-09-15T21:32:27.558 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-15T21:32:23.996281+0000 '>' 2024-09-15T21:27:56.080190+0000 2024-09-15T21:32:27.559 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2023: wait_for_scrub: return 0 2024-09-15T21:32:27.559 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1157: TEST_corrupt_scrub_replicated: ERRORS=0 2024-09-15T21:32:27.559 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1158: TEST_corrupt_scrub_replicated: declare -a s_err_strings 2024-09-15T21:32:27.559 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1159: TEST_corrupt_scrub_replicated: err_strings[0]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:30259878:::ROBJ15:head : candidate had a missing info key' 2024-09-15T21:32:27.559 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1160: TEST_corrupt_scrub_replicated: err_strings[1]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid 3:33aca486:::ROBJ18:head : object info inconsistent ' 2024-09-15T21:32:27.559 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1161: TEST_corrupt_scrub_replicated: err_strings[2]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:5c7b2c47:::ROBJ16:head : candidate had a corrupt snapset' 2024-09-15T21:32:27.559 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1162: TEST_corrupt_scrub_replicated: err_strings[3]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 0 soid 3:5c7b2c47:::ROBJ16:head : candidate had a missing snapset key' 2024-09-15T21:32:27.559 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1163: TEST_corrupt_scrub_replicated: err_strings[4]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid 3:5c7b2c47:::ROBJ16:head : failed to pick suitable object info' 2024-09-15T21:32:27.559 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1164: TEST_corrupt_scrub_replicated: err_strings[5]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid 3:86586531:::ROBJ8:head : attr value mismatch '\''_key1-ROBJ8'\'', attr name mismatch '\''_key3-ROBJ8'\'', attr name mismatch '\''_key2-ROBJ8'\''' 2024-09-15T21:32:27.559 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1165: TEST_corrupt_scrub_replicated: err_strings[6]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 0 soid 3:bc819597:::ROBJ12:head : candidate had a stat error' 2024-09-15T21:32:27.559 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1166: TEST_corrupt_scrub_replicated: err_strings[7]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:c0c86b1d:::ROBJ14:head : candidate had a missing info key' 2024-09-15T21:32:27.559 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1167: TEST_corrupt_scrub_replicated: err_strings[8]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 0 soid 3:c0c86b1d:::ROBJ14:head : candidate had a corrupt info' 2024-09-15T21:32:27.559 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1168: TEST_corrupt_scrub_replicated: err_strings[9]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid 3:c0c86b1d:::ROBJ14:head : failed to pick suitable object info' 2024-09-15T21:32:27.560 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1169: TEST_corrupt_scrub_replicated: err_strings[10]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:ce3f1d6a:::ROBJ1:head : candidate size 9 info size 7 mismatch' 2024-09-15T21:32:27.560 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1170: TEST_corrupt_scrub_replicated: err_strings[11]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:ce3f1d6a:::ROBJ1:head : size 9 != size 7 from auth oi 3:ce3f1d6a:::ROBJ1:head[(][0-9]*'\''[0-9]* osd.1.0:[0-9]* dirty|omap|data_digest|omap_digest s 7 uv 3 dd 2ddbf8f5 od f5fba2c6 alloc_hint [[]0 0 0[]][)], size 9 != size 7 from shard 0' 2024-09-15T21:32:27.560 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1171: TEST_corrupt_scrub_replicated: err_strings[12]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 0 soid 3:d60617f9:::ROBJ13:head : candidate had a stat error' 2024-09-15T21:32:27.560 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1172: TEST_corrupt_scrub_replicated: err_strings[13]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 3:f2a5b2a4:::ROBJ3:head : missing' 2024-09-15T21:32:27.560 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1173: TEST_corrupt_scrub_replicated: err_strings[14]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:ffdb2004:::ROBJ9:head : candidate size 1 info size 7 mismatch' 2024-09-15T21:32:27.561 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1174: TEST_corrupt_scrub_replicated: err_strings[15]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:ffdb2004:::ROBJ9:head : object info inconsistent ' 2024-09-15T21:32:27.561 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1175: TEST_corrupt_scrub_replicated: err_strings[16]='log_channel[(]cluster[)] log [[]ERR[]] : scrub [0-9]*[.]0 3:c0c86b1d:::ROBJ14:head : no '\''_'\'' attr' 2024-09-15T21:32:27.561 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1176: TEST_corrupt_scrub_replicated: err_strings[17]='log_channel[(]cluster[)] log [[]ERR[]] : scrub [0-9]*[.]0 3:5c7b2c47:::ROBJ16:head : can'\''t decode '\''snapset'\'' attr .* v=3 cannot decode .* Malformed input' 2024-09-15T21:32:27.561 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1177: TEST_corrupt_scrub_replicated: err_strings[18]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 scrub : stat mismatch, got 19/19 objects, 0/0 clones, 18/19 dirty, 18/19 omap, 0/0 pinned, 0/0 hit_set_archive, 0/0 whiteouts, 1049713/1049720 bytes, 0/0 manifest objects, 0/0 hit_set_archive bytes.' 2024-09-15T21:32:27.561 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1178: TEST_corrupt_scrub_replicated: err_strings[19]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 scrub 1 missing, 8 inconsistent objects' 2024-09-15T21:32:27.561 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1179: TEST_corrupt_scrub_replicated: err_strings[20]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 scrub 18 errors' 2024-09-15T21:32:27.561 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1180: TEST_corrupt_scrub_replicated: err_strings[21]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid 3:123a5f55:::ROBJ19:head : size 1049600 > 1048576 is too large' 2024-09-15T21:32:27.561 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1182: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-09-15T21:32:27.561 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1184: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:30259878:::ROBJ15:head : candidate had a missing info key' td/osd-scrub-repair/osd.1.log 2024-09-15T21:32:27.566 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1182: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-09-15T21:32:27.566 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1184: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid 3:33aca486:::ROBJ18:head : object info inconsistent ' td/osd-scrub-repair/osd.1.log 2024-09-15T21:32:27.576 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1182: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-09-15T21:32:27.576 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1184: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:5c7b2c47:::ROBJ16:head : candidate had a corrupt snapset' td/osd-scrub-repair/osd.1.log 2024-09-15T21:32:27.585 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1182: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-09-15T21:32:27.585 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1184: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 0 soid 3:5c7b2c47:::ROBJ16:head : candidate had a missing snapset key' td/osd-scrub-repair/osd.1.log 2024-09-15T21:32:27.594 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1182: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-09-15T21:32:27.594 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1184: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid 3:5c7b2c47:::ROBJ16:head : failed to pick suitable object info' td/osd-scrub-repair/osd.1.log 2024-09-15T21:32:27.604 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1182: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-09-15T21:32:27.604 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1184: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid 3:86586531:::ROBJ8:head : attr value mismatch '\''_key1-ROBJ8'\'', attr name mismatch '\''_key3-ROBJ8'\'', attr name mismatch '\''_key2-ROBJ8'\''' td/osd-scrub-repair/osd.1.log 2024-09-15T21:32:27.612 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1182: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-09-15T21:32:27.612 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1184: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 0 soid 3:bc819597:::ROBJ12:head : candidate had a stat error' td/osd-scrub-repair/osd.1.log 2024-09-15T21:32:27.621 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1182: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-09-15T21:32:27.622 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1184: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:c0c86b1d:::ROBJ14:head : candidate had a missing info key' td/osd-scrub-repair/osd.1.log 2024-09-15T21:32:27.630 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1182: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-09-15T21:32:27.631 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1184: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 0 soid 3:c0c86b1d:::ROBJ14:head : candidate had a corrupt info' td/osd-scrub-repair/osd.1.log 2024-09-15T21:32:27.640 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1182: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-09-15T21:32:27.640 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1184: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid 3:c0c86b1d:::ROBJ14:head : failed to pick suitable object info' td/osd-scrub-repair/osd.1.log 2024-09-15T21:32:27.650 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1182: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-09-15T21:32:27.650 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1184: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:ce3f1d6a:::ROBJ1:head : candidate size 9 info size 7 mismatch' td/osd-scrub-repair/osd.1.log 2024-09-15T21:32:27.658 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1182: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-09-15T21:32:27.658 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1184: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:ce3f1d6a:::ROBJ1:head : size 9 != size 7 from auth oi 3:ce3f1d6a:::ROBJ1:head[(][0-9]*'\''[0-9]* osd.1.0:[0-9]* dirty|omap|data_digest|omap_digest s 7 uv 3 dd 2ddbf8f5 od f5fba2c6 alloc_hint [[]0 0 0[]][)], size 9 != size 7 from shard 0' td/osd-scrub-repair/osd.1.log 2024-09-15T21:32:27.668 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1182: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-09-15T21:32:27.668 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1184: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 0 soid 3:d60617f9:::ROBJ13:head : candidate had a stat error' td/osd-scrub-repair/osd.1.log 2024-09-15T21:32:27.677 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1182: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-09-15T21:32:27.677 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1184: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 3:f2a5b2a4:::ROBJ3:head : missing' td/osd-scrub-repair/osd.1.log 2024-09-15T21:32:27.687 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1182: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-09-15T21:32:27.687 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1184: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:ffdb2004:::ROBJ9:head : candidate size 1 info size 7 mismatch' td/osd-scrub-repair/osd.1.log 2024-09-15T21:32:27.697 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1182: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-09-15T21:32:27.697 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1184: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:ffdb2004:::ROBJ9:head : object info inconsistent ' td/osd-scrub-repair/osd.1.log 2024-09-15T21:32:27.706 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1182: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-09-15T21:32:27.706 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1184: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : scrub [0-9]*[.]0 3:c0c86b1d:::ROBJ14:head : no '\''_'\'' attr' td/osd-scrub-repair/osd.1.log 2024-09-15T21:32:27.717 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1182: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-09-15T21:32:27.717 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1184: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : scrub [0-9]*[.]0 3:5c7b2c47:::ROBJ16:head : can'\''t decode '\''snapset'\'' attr .* v=3 cannot decode .* Malformed input' td/osd-scrub-repair/osd.1.log 2024-09-15T21:32:27.727 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1182: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-09-15T21:32:27.727 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1184: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 scrub : stat mismatch, got 19/19 objects, 0/0 clones, 18/19 dirty, 18/19 omap, 0/0 pinned, 0/0 hit_set_archive, 0/0 whiteouts, 1049713/1049720 bytes, 0/0 manifest objects, 0/0 hit_set_archive bytes.' td/osd-scrub-repair/osd.1.log 2024-09-15T21:32:27.736 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1182: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-09-15T21:32:27.736 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1184: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 scrub 1 missing, 8 inconsistent objects' td/osd-scrub-repair/osd.1.log 2024-09-15T21:32:27.746 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1182: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-09-15T21:32:27.746 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1184: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 scrub 18 errors' td/osd-scrub-repair/osd.1.log 2024-09-15T21:32:27.758 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1182: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-09-15T21:32:27.758 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1184: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid 3:123a5f55:::ROBJ19:head : size 1049600 > 1048576 is too large' td/osd-scrub-repair/osd.1.log 2024-09-15T21:32:27.768 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1191: TEST_corrupt_scrub_replicated: rados list-inconsistent-pg csr_pool 2024-09-15T21:32:27.796 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1193: TEST_corrupt_scrub_replicated: jq '. | length' td/osd-scrub-repair/json 2024-09-15T21:32:27.798 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1193: TEST_corrupt_scrub_replicated: test 1 = 1 2024-09-15T21:32:27.798 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1195: TEST_corrupt_scrub_replicated: jq -r '.[0]' td/osd-scrub-repair/json 2024-09-15T21:32:27.801 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1195: TEST_corrupt_scrub_replicated: test 3.0 = 3.0 2024-09-15T21:32:27.801 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1197: TEST_corrupt_scrub_replicated: rados list-inconsistent-obj 3.0 2024-09-15T21:32:27.829 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1199: TEST_corrupt_scrub_replicated: jq .epoch td/osd-scrub-repair/json 2024-09-15T21:32:27.831 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1199: TEST_corrupt_scrub_replicated: epoch=179 2024-09-15T21:32:27.832 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1201: TEST_corrupt_scrub_replicated: jq 'def walk(f): 2024-09-15T21:32:27.832 INFO:tasks.workunit.client.0.smithi136.stderr: . as $in 2024-09-15T21:32:27.832 INFO:tasks.workunit.client.0.smithi136.stderr: | if type == "object" then 2024-09-15T21:32:27.832 INFO:tasks.workunit.client.0.smithi136.stderr: reduce keys[] as $key 2024-09-15T21:32:27.832 INFO:tasks.workunit.client.0.smithi136.stderr: ( {}; . + { ($key): ($in[$key] | walk(f)) } ) | f 2024-09-15T21:32:27.832 INFO:tasks.workunit.client.0.smithi136.stderr: elif type == "array" then map( walk(f) ) | f 2024-09-15T21:32:27.832 INFO:tasks.workunit.client.0.smithi136.stderr: else f 2024-09-15T21:32:27.832 INFO:tasks.workunit.client.0.smithi136.stderr: end; 2024-09-15T21:32:27.832 INFO:tasks.workunit.client.0.smithi136.stderr:walk(if type == "object" then del(.mtime) else . end) 2024-09-15T21:32:27.832 INFO:tasks.workunit.client.0.smithi136.stderr:| walk(if type == "object" then del(.local_mtime) else . end) 2024-09-15T21:32:27.832 INFO:tasks.workunit.client.0.smithi136.stderr:| walk(if type == "object" then del(.last_reqid) else . end) 2024-09-15T21:32:27.832 INFO:tasks.workunit.client.0.smithi136.stderr:| walk(if type == "object" then del(.version) else . end) 2024-09-15T21:32:27.832 INFO:tasks.workunit.client.0.smithi136.stderr:| walk(if type == "object" then del(.prior_version) else . end)' 2024-09-15T21:32:27.833 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1201: TEST_corrupt_scrub_replicated: jq .inconsistents 2024-09-15T21:32:27.833 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1201: TEST_corrupt_scrub_replicated: python3 -c 'import json; import sys ; JSON=sys.stdin.read() ; ud = json.loads(JSON) ; print(json.dumps(ud, sort_keys=True, indent=2))' 2024-09-15T21:32:27.855 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2082: TEST_corrupt_scrub_replicated: jq 'def walk(f): 2024-09-15T21:32:27.855 INFO:tasks.workunit.client.0.smithi136.stderr: . as $in 2024-09-15T21:32:27.855 INFO:tasks.workunit.client.0.smithi136.stderr: | if type == "object" then 2024-09-15T21:32:27.855 INFO:tasks.workunit.client.0.smithi136.stderr: reduce keys[] as $key 2024-09-15T21:32:27.856 INFO:tasks.workunit.client.0.smithi136.stderr: ( {}; . + { ($key): ($in[$key] | walk(f)) } ) | f 2024-09-15T21:32:27.856 INFO:tasks.workunit.client.0.smithi136.stderr: elif type == "array" then map( walk(f) ) | f 2024-09-15T21:32:27.856 INFO:tasks.workunit.client.0.smithi136.stderr: else f 2024-09-15T21:32:27.856 INFO:tasks.workunit.client.0.smithi136.stderr: end; 2024-09-15T21:32:27.856 INFO:tasks.workunit.client.0.smithi136.stderr:walk(if type == "object" then del(.mtime) else . end) 2024-09-15T21:32:27.856 INFO:tasks.workunit.client.0.smithi136.stderr:| walk(if type == "object" then del(.local_mtime) else . end) 2024-09-15T21:32:27.856 INFO:tasks.workunit.client.0.smithi136.stderr:| walk(if type == "object" then del(.last_reqid) else . end) 2024-09-15T21:32:27.856 INFO:tasks.workunit.client.0.smithi136.stderr:| walk(if type == "object" then del(.version) else . end) 2024-09-15T21:32:27.856 INFO:tasks.workunit.client.0.smithi136.stderr:| walk(if type == "object" then del(.prior_version) else . end)' td/osd-scrub-repair/json 2024-09-15T21:32:27.856 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2082: TEST_corrupt_scrub_replicated: jq .inconsistents 2024-09-15T21:32:27.856 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2082: TEST_corrupt_scrub_replicated: python3 -c 'import json; import sys ; JSON=sys.stdin.read() ; ud = json.loads(JSON) ; print(json.dumps(ud, sort_keys=True, indent=2))' 2024-09-15T21:32:27.878 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2083: TEST_corrupt_scrub_replicated: multidiff td/osd-scrub-repair/checkcsjson td/osd-scrub-repair/csjson 2024-09-15T21:32:27.878 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2434: multidiff: diff td/osd-scrub-repair/checkcsjson td/osd-scrub-repair/csjson 2024-09-15T21:32:27.879 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2084: TEST_corrupt_scrub_replicated: test no = yes 2024-09-15T21:32:27.879 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2089: TEST_corrupt_scrub_replicated: test '' = yes 2024-09-15T21:32:27.879 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2094: TEST_corrupt_scrub_replicated: objname=ROBJ9 2024-09-15T21:32:27.879 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2096: TEST_corrupt_scrub_replicated: echo -n ZZZ 2024-09-15T21:32:27.879 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2097: TEST_corrupt_scrub_replicated: rados --pool csr_pool put ROBJ9 td/osd-scrub-repair/change 2024-09-15T21:32:27.963 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2099: TEST_corrupt_scrub_replicated: objectstore_tool td/osd-scrub-repair 0 ROBJ9 set-attr _ td/osd-scrub-repair/robj9-oi 2024-09-15T21:32:27.963 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2024-09-15T21:32:27.963 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2024-09-15T21:32:27.963 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=0 2024-09-15T21:32:27.963 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2024-09-15T21:32:27.963 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 ROBJ9 set-attr _ td/osd-scrub-repair/robj9-oi 2024-09-15T21:32:27.964 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-09-15T21:32:27.964 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2024-09-15T21:32:27.964 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=0 2024-09-15T21:32:27.964 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2024-09-15T21:32:27.964 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2024-09-15T21:32:27.964 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-15T21:32:27.964 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-15T21:32:27.964 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-15T21:32:27.964 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-15T21:32:27.964 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-15T21:32:28.269 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-15T21:32:28.269 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 ROBJ9 set-attr _ td/osd-scrub-repair/robj9-oi 2024-09-15T21:32:28.269 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-15T21:32:28.269 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-15T21:32:28.269 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=0 2024-09-15T21:32:28.270 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-15T21:32:28.270 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-09-15T21:32:28.270 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 ROBJ9 set-attr _ td/osd-scrub-repair/robj9-oi 2024-09-15T21:32:29.771 INFO:tasks.workunit.client.0.smithi136.stderr:Error decoding attr on : 3.0_head,#3:c0c86b1d:::ROBJ14:head#, (22) Invalid argument 2024-09-15T21:32:30.289 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 2024-09-15T21:32:30.289 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2024-09-15T21:32:30.289 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-15T21:32:30.289 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=0 2024-09-15T21:32:30.289 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-15T21:32:30.289 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-09-15T21:32:30.290 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=ce4892c7-6c6d-4b4f-8e6f-07a3e11133e5 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-15T21:32:30.290 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T21:32:30.290 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T21:32:30.290 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T21:32:30.290 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-09-15T21:32:30.290 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-09-15T21:32:30.290 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-15T21:32:30.290 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-15T21:32:30.290 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-15T21:32:30.290 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-15T21:32:30.290 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-15T21:32:30.290 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-15T21:32:30.290 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-15T21:32:30.291 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T21:32:30.291 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52602 2024-09-15T21:32:30.291 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.52602/$cluster-$name.asok' 2024-09-15T21:32:30.291 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.52602/$cluster-$name.asok' 2024-09-15T21:32:30.291 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-15T21:32:30.291 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-15T21:32:30.291 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-15T21:32:30.291 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T21:32:30.291 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T21:32:30.291 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T21:32:30.291 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T21:32:30.291 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-15T21:32:30.291 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-15T21:32:30.292 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-09-15T21:32:30.292 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.0 2024-09-15T21:32:30.293 INFO:tasks.workunit.client.0.smithi136.stderr:start osd.0 2024-09-15T21:32:30.293 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 0 --fsid=ce4892c7-6c6d-4b4f-8e6f-07a3e11133e5 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.52602/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T21:32:30.293 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-09-15T21:32:30.293 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 0 = 0 ']' 2024-09-15T21:32:30.293 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-15T21:32:30.293 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-15T21:32:30.293 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-15T21:32:30.320 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:32:30.318+0000 7f0c47e0d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:32:30.323 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:32:30.321+0000 7f0c47e0d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:32:30.324 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:32:30.322+0000 7f0c47e0d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:32:30.576 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 0 2024-09-15T21:32:30.576 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-15T21:32:30.576 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2024-09-15T21:32:30.576 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-15T21:32:30.576 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-15T21:32:30.576 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:32:30.576 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-15T21:32:30.576 INFO:tasks.workunit.client.0.smithi136.stderr:0 2024-09-15T21:32:30.576 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:32:30.576 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:32:30.849 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:32:31.686 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:32:31.684+0000 7f0c47e0d640 -1 Falling back to public interface 2024-09-15T21:32:31.851 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:32:31.851 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:32:31.851 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-15T21:32:31.851 INFO:tasks.workunit.client.0.smithi136.stderr:1 2024-09-15T21:32:31.851 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:32:31.851 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:32:32.137 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:32:33.138 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:32:33.138 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:32:33.139 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-15T21:32:33.139 INFO:tasks.workunit.client.0.smithi136.stderr:2 2024-09-15T21:32:33.139 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:32:33.139 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:32:33.420 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:32:34.422 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:32:34.422 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:32:34.423 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-15T21:32:34.423 INFO:tasks.workunit.client.0.smithi136.stderr:3 2024-09-15T21:32:34.423 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:32:34.423 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:32:34.504 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:32:34.502+0000 7f0c47e0d640 -1 osd.0 181 log_to_monitors true 2024-09-15T21:32:34.717 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:32:35.718 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:32:35.718 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:32:35.718 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-15T21:32:35.718 INFO:tasks.workunit.client.0.smithi136.stderr:4 2024-09-15T21:32:35.719 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:32:35.719 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:32:36.056 INFO:tasks.workunit.client.0.smithi136.stderr:osd.0 up in weight 1 up_from 185 up_thru 185 down_at 182 last_clean_interval [167,181) [v2:127.0.0.1:6802/2141588538,v1:127.0.0.1:6803/2141588538] [v2:127.0.0.1:6804/2141588538,v1:127.0.0.1:6805/2141588538] exists,up 7a68df98-8e0f-45ef-a9e9-b2a45ca8fd3f 2024-09-15T21:32:36.056 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-15T21:32:36.056 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-15T21:32:36.056 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-15T21:32:36.056 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2024-09-15T21:32:36.057 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-15T21:32:36.057 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-15T21:32:36.057 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-15T21:32:36.057 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-15T21:32:36.057 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-15T21:32:36.057 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-15T21:32:36.058 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-15T21:32:36.058 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-15T21:32:36.058 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-15T21:32:36.150 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-15T21:32:36.150 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-15T21:32:36.150 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-15T21:32:36.151 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-15T21:32:36.151 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-15T21:32:36.151 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-15T21:32:36.426 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-15T21:32:36.426 INFO:tasks.workunit.client.0.smithi136.stderr:1' 2024-09-15T21:32:36.426 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-15T21:32:36.426 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T21:32:36.426 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-15T21:32:36.530 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=794568949762 2024-09-15T21:32:36.530 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 794568949762 2024-09-15T21:32:36.530 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-794568949762' 2024-09-15T21:32:36.530 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T21:32:36.531 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-15T21:32:36.633 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=768799145991 2024-09-15T21:32:36.633 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 768799145991 2024-09-15T21:32:36.633 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-794568949762 1-768799145991' 2024-09-15T21:32:36.633 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T21:32:36.634 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-794568949762 2024-09-15T21:32:36.634 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T21:32:36.634 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-15T21:32:36.635 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-794568949762 2024-09-15T21:32:36.635 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T21:32:36.636 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=794568949762 2024-09-15T21:32:36.636 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 794568949762' 2024-09-15T21:32:36.636 INFO:tasks.workunit.client.0.smithi136.stderr:waiting osd.0 seq 794568949762 2024-09-15T21:32:36.637 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T21:32:36.909 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 1 -lt 794568949762 2024-09-15T21:32:36.909 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-15T21:32:37.910 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-15T21:32:37.911 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T21:32:38.194 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 794568949762 -lt 794568949762 2024-09-15T21:32:38.194 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T21:32:38.195 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-768799145991 2024-09-15T21:32:38.195 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T21:32:38.195 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-15T21:32:38.196 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-768799145991 2024-09-15T21:32:38.196 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T21:32:38.197 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=768799145991 2024-09-15T21:32:38.197 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 768799145991' 2024-09-15T21:32:38.197 INFO:tasks.workunit.client.0.smithi136.stderr:waiting osd.1 seq 768799145991 2024-09-15T21:32:38.198 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-15T21:32:38.474 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 768799145991 -lt 768799145991 2024-09-15T21:32:38.475 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-15T21:32:38.475 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T21:32:38.475 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T21:32:38.820 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 6 == 0 2024-09-15T21:32:38.820 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-15T21:32:38.821 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-15T21:32:38.821 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-15T21:32:38.821 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T21:32:38.821 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T21:32:38.821 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T21:32:38.821 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T21:32:39.095 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=6 2024-09-15T21:32:39.095 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-15T21:32:39.095 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T21:32:39.095 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T21:32:39.444 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 = 6 2024-09-15T21:32:39.445 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-15T21:32:39.445 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-15T21:32:39.445 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2100: TEST_corrupt_scrub_replicated: rm td/osd-scrub-repair/oi td/osd-scrub-repair/change 2024-09-15T21:32:39.446 INFO:tasks.workunit.client.0.smithi136.stderr:rm: cannot remove 'td/osd-scrub-repair/oi': No such file or directory 2024-09-15T21:32:39.447 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2102: TEST_corrupt_scrub_replicated: objname=ROBJ10 2024-09-15T21:32:39.447 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2103: TEST_corrupt_scrub_replicated: objectstore_tool td/osd-scrub-repair 1 ROBJ10 get-attr _ 2024-09-15T21:32:39.447 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2024-09-15T21:32:39.447 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2024-09-15T21:32:39.447 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=1 2024-09-15T21:32:39.447 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2024-09-15T21:32:39.447 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 ROBJ10 get-attr _ 2024-09-15T21:32:39.447 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-09-15T21:32:39.447 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2024-09-15T21:32:39.447 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=1 2024-09-15T21:32:39.447 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2024-09-15T21:32:39.447 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-09-15T21:32:39.447 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-15T21:32:39.447 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-15T21:32:39.448 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-15T21:32:39.448 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-15T21:32:39.448 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-15T21:32:39.753 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-15T21:32:39.754 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 ROBJ10 get-attr _ 2024-09-15T21:32:39.754 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-15T21:32:39.754 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-15T21:32:39.754 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=1 2024-09-15T21:32:39.754 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-15T21:32:39.754 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-09-15T21:32:39.754 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 ROBJ10 get-attr _ 2024-09-15T21:32:40.940 INFO:tasks.workunit.client.0.smithi136.stderr:Error getting attr on : 3.0_head,#3:30259878:::ROBJ15:head#, (61) No data available 2024-09-15T21:32:40.940 INFO:tasks.workunit.client.0.smithi136.stderr:Error getting attr on : 3.0_head,#3:c0c86b1d:::ROBJ14:head#, (61) No data available 2024-09-15T21:32:41.457 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2024-09-15T21:32:41.458 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2024-09-15T21:32:41.458 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-15T21:32:41.458 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=1 2024-09-15T21:32:41.458 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-15T21:32:41.458 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-09-15T21:32:41.458 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=ce4892c7-6c6d-4b4f-8e6f-07a3e11133e5 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-15T21:32:41.458 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T21:32:41.458 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T21:32:41.458 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T21:32:41.458 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-09-15T21:32:41.458 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-09-15T21:32:41.458 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-15T21:32:41.459 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-15T21:32:41.459 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-15T21:32:41.459 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-15T21:32:41.459 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-15T21:32:41.459 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-15T21:32:41.459 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-15T21:32:41.459 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T21:32:41.459 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52602 2024-09-15T21:32:41.459 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.52602/$cluster-$name.asok' 2024-09-15T21:32:41.459 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.52602/$cluster-$name.asok' 2024-09-15T21:32:41.459 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-15T21:32:41.459 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-15T21:32:41.459 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-15T21:32:41.459 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T21:32:41.460 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T21:32:41.460 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T21:32:41.460 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T21:32:41.460 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-15T21:32:41.460 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-15T21:32:41.460 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-09-15T21:32:41.461 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.1 2024-09-15T21:32:41.461 INFO:tasks.workunit.client.0.smithi136.stderr:start osd.1 2024-09-15T21:32:41.461 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 1 --fsid=ce4892c7-6c6d-4b4f-8e6f-07a3e11133e5 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.52602/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T21:32:41.461 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-09-15T21:32:41.461 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 1 = 1 ']' 2024-09-15T21:32:41.461 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-15T21:32:41.461 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-15T21:32:41.461 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-15T21:32:41.489 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:32:41.486+0000 7f309260d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:32:41.491 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:32:41.489+0000 7f309260d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:32:41.492 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:32:41.490+0000 7f309260d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:32:41.746 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 1 2024-09-15T21:32:41.746 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-15T21:32:41.746 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2024-09-15T21:32:41.746 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-15T21:32:41.746 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-15T21:32:41.746 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:32:41.746 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-15T21:32:41.746 INFO:tasks.workunit.client.0.smithi136.stderr:0 2024-09-15T21:32:41.747 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:32:41.747 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T21:32:42.022 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:32:42.619 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:32:42.617+0000 7f309260d640 -1 Falling back to public interface 2024-09-15T21:32:43.024 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:32:43.024 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:32:43.024 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-15T21:32:43.024 INFO:tasks.workunit.client.0.smithi136.stderr:1 2024-09-15T21:32:43.024 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:32:43.024 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T21:32:43.314 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:32:44.315 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:32:44.316 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:32:44.316 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-15T21:32:44.316 INFO:tasks.workunit.client.0.smithi136.stderr:2 2024-09-15T21:32:44.316 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:32:44.316 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T21:32:44.602 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:32:45.470 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:32:45.467+0000 7f309260d640 -1 osd.1 187 log_to_monitors true 2024-09-15T21:32:45.604 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:32:45.604 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:32:45.604 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-15T21:32:45.604 INFO:tasks.workunit.client.0.smithi136.stderr:3 2024-09-15T21:32:45.604 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:32:45.604 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T21:32:45.985 INFO:tasks.workunit.client.0.smithi136.stderr:osd.1 up in weight 1 up_from 191 up_thru 185 down_at 188 last_clean_interval [179,187) [v2:127.0.0.1:6810/2293044877,v1:127.0.0.1:6811/2293044877] [v2:127.0.0.1:6812/2293044877,v1:127.0.0.1:6813/2293044877] exists,up 9aec06d1-33ff-42ab-be71-ac761d546d96 2024-09-15T21:32:45.985 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-15T21:32:45.986 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-15T21:32:45.986 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-15T21:32:45.986 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2024-09-15T21:32:45.986 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-15T21:32:45.986 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-15T21:32:45.986 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-15T21:32:45.986 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-15T21:32:45.986 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-15T21:32:45.986 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-15T21:32:45.986 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-15T21:32:45.986 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-15T21:32:45.987 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-15T21:32:46.078 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-15T21:32:46.078 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-15T21:32:46.078 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-15T21:32:46.078 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-15T21:32:46.078 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-15T21:32:46.078 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-15T21:32:46.586 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-15T21:32:46.586 INFO:tasks.workunit.client.0.smithi136.stderr:1' 2024-09-15T21:32:46.586 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-15T21:32:46.586 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T21:32:46.586 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-15T21:32:46.806 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=794568949765 2024-09-15T21:32:46.807 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 794568949765 2024-09-15T21:32:46.807 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-794568949765' 2024-09-15T21:32:46.807 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T21:32:46.807 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-15T21:32:46.978 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=820338753538 2024-09-15T21:32:46.978 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 820338753538 2024-09-15T21:32:46.978 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-794568949765 1-820338753538' 2024-09-15T21:32:46.978 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T21:32:46.979 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-794568949765 2024-09-15T21:32:46.979 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T21:32:46.979 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-15T21:32:46.980 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-794568949765 2024-09-15T21:32:46.980 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T21:32:46.981 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=794568949765 2024-09-15T21:32:46.981 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 794568949765' 2024-09-15T21:32:46.981 INFO:tasks.workunit.client.0.smithi136.stderr:waiting osd.0 seq 794568949765 2024-09-15T21:32:46.982 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T21:32:47.258 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 794568949765 -lt 794568949765 2024-09-15T21:32:47.259 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T21:32:47.259 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-820338753538 2024-09-15T21:32:47.259 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T21:32:47.260 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-15T21:32:47.261 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-820338753538 2024-09-15T21:32:47.261 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T21:32:47.262 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=820338753538 2024-09-15T21:32:47.262 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 820338753538' 2024-09-15T21:32:47.262 INFO:tasks.workunit.client.0.smithi136.stderr:waiting osd.1 seq 820338753538 2024-09-15T21:32:47.262 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-15T21:32:47.541 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 820338753538 -lt 820338753538 2024-09-15T21:32:47.541 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-15T21:32:47.542 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T21:32:47.542 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T21:32:47.889 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 6 == 0 2024-09-15T21:32:47.889 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-15T21:32:47.890 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-15T21:32:47.890 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-15T21:32:47.890 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T21:32:47.890 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T21:32:47.890 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T21:32:47.890 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T21:32:48.165 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=3 2024-09-15T21:32:48.165 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-15T21:32:48.166 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T21:32:48.166 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T21:32:48.507 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 3 = 6 2024-09-15T21:32:48.508 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 3 '!=' -1 2024-09-15T21:32:48.508 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: loop=0 2024-09-15T21:32:48.508 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1652: wait_for_clean: num_active_clean=3 2024-09-15T21:32:48.508 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-15T21:32:48.508 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 0.1 2024-09-15T21:32:48.609 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-15T21:32:48.609 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-15T21:32:48.610 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-15T21:32:48.610 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-15T21:32:48.610 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T21:32:48.610 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T21:32:48.610 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T21:32:48.610 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T21:32:48.878 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=3 2024-09-15T21:32:48.879 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-15T21:32:48.879 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T21:32:48.879 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T21:32:49.236 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 3 = 6 2024-09-15T21:32:49.237 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 3 '!=' 3 2024-09-15T21:32:49.237 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: get_is_making_recovery_progress 2024-09-15T21:32:49.237 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2024-09-15T21:32:49.237 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-15T21:32:49.237 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-15T21:32:49.237 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-15T21:32:49.238 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2024-09-15T21:32:49.238 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-15T21:32:49.590 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=null 2024-09-15T21:32:49.590 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test null '!=' null 2024-09-15T21:32:49.590 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1655: wait_for_clean: (( 1 >= 13 )) 2024-09-15T21:32:49.590 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-15T21:32:49.591 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 0.2 2024-09-15T21:32:49.791 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-15T21:32:49.791 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-15T21:32:49.792 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-15T21:32:49.793 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-15T21:32:49.793 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T21:32:49.793 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T21:32:49.793 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T21:32:49.793 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T21:32:50.060 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=3 2024-09-15T21:32:50.060 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-15T21:32:50.061 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T21:32:50.061 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T21:32:50.401 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 3 = 6 2024-09-15T21:32:50.402 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 3 '!=' 3 2024-09-15T21:32:50.402 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: get_is_making_recovery_progress 2024-09-15T21:32:50.402 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2024-09-15T21:32:50.402 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-15T21:32:50.402 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-15T21:32:50.402 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-15T21:32:50.402 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2024-09-15T21:32:50.402 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-15T21:32:50.745 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=null 2024-09-15T21:32:50.745 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test null '!=' null 2024-09-15T21:32:50.746 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1655: wait_for_clean: (( 2 >= 13 )) 2024-09-15T21:32:50.746 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-15T21:32:50.746 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 0.4 2024-09-15T21:32:51.146 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-15T21:32:51.146 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-15T21:32:51.147 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-15T21:32:51.147 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-15T21:32:51.147 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T21:32:51.147 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T21:32:51.147 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T21:32:51.147 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T21:32:51.421 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=6 2024-09-15T21:32:51.421 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-15T21:32:51.421 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T21:32:51.421 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T21:32:51.766 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 = 6 2024-09-15T21:32:51.766 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-15T21:32:51.766 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-15T21:32:51.766 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2104: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapval ROBJ10 key2-ROBJ10 val2-ROBJ10 2024-09-15T21:32:51.903 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2105: TEST_corrupt_scrub_replicated: objectstore_tool td/osd-scrub-repair 0 ROBJ10 set-attr _ td/osd-scrub-repair/oi 2024-09-15T21:32:51.903 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2024-09-15T21:32:51.903 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2024-09-15T21:32:51.903 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=0 2024-09-15T21:32:51.903 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2024-09-15T21:32:51.903 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 ROBJ10 set-attr _ td/osd-scrub-repair/oi 2024-09-15T21:32:51.903 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-09-15T21:32:51.904 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2024-09-15T21:32:51.904 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=0 2024-09-15T21:32:51.904 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2024-09-15T21:32:51.904 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2024-09-15T21:32:51.904 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-15T21:32:51.904 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-15T21:32:51.904 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-15T21:32:51.904 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-15T21:32:51.904 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-15T21:32:52.210 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-15T21:32:52.211 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 ROBJ10 set-attr _ td/osd-scrub-repair/oi 2024-09-15T21:32:52.211 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-15T21:32:52.211 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-15T21:32:52.211 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=0 2024-09-15T21:32:52.211 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-15T21:32:52.211 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-09-15T21:32:52.212 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 ROBJ10 set-attr _ td/osd-scrub-repair/oi 2024-09-15T21:32:53.682 INFO:tasks.workunit.client.0.smithi136.stderr:Error decoding attr on : 3.0_head,#3:c0c86b1d:::ROBJ14:head#, (22) Invalid argument 2024-09-15T21:32:54.204 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 2024-09-15T21:32:54.204 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2024-09-15T21:32:54.205 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-15T21:32:54.205 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=0 2024-09-15T21:32:54.205 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-15T21:32:54.205 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-09-15T21:32:54.205 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=ce4892c7-6c6d-4b4f-8e6f-07a3e11133e5 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-15T21:32:54.205 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T21:32:54.205 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T21:32:54.205 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T21:32:54.205 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-09-15T21:32:54.205 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-09-15T21:32:54.205 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-15T21:32:54.206 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-15T21:32:54.206 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-15T21:32:54.206 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-15T21:32:54.206 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-15T21:32:54.206 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-15T21:32:54.206 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-15T21:32:54.206 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T21:32:54.206 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52602 2024-09-15T21:32:54.206 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.52602/$cluster-$name.asok' 2024-09-15T21:32:54.206 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.52602/$cluster-$name.asok' 2024-09-15T21:32:54.207 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-15T21:32:54.207 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-15T21:32:54.207 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-15T21:32:54.207 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T21:32:54.207 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T21:32:54.207 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T21:32:54.207 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T21:32:54.207 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-15T21:32:54.207 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-15T21:32:54.207 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-09-15T21:32:54.207 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.0 2024-09-15T21:32:54.207 INFO:tasks.workunit.client.0.smithi136.stderr:start osd.0 2024-09-15T21:32:54.208 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 0 --fsid=ce4892c7-6c6d-4b4f-8e6f-07a3e11133e5 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.52602/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T21:32:54.209 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-09-15T21:32:54.209 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 0 = 0 ']' 2024-09-15T21:32:54.209 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-15T21:32:54.209 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-15T21:32:54.209 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-15T21:32:54.235 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:32:54.233+0000 7f363200c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:32:54.237 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:32:54.235+0000 7f363200c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:32:54.239 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:32:54.236+0000 7f363200c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:32:54.494 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 0 2024-09-15T21:32:54.494 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-15T21:32:54.494 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2024-09-15T21:32:54.494 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-15T21:32:54.494 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-15T21:32:54.495 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:32:54.495 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-15T21:32:54.495 INFO:tasks.workunit.client.0.smithi136.stderr:0 2024-09-15T21:32:54.495 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:32:54.495 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:32:54.771 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:32:55.361 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:32:55.359+0000 7f363200c640 -1 Falling back to public interface 2024-09-15T21:32:55.772 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:32:55.772 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:32:55.773 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-15T21:32:55.773 INFO:tasks.workunit.client.0.smithi136.stderr:1 2024-09-15T21:32:55.773 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:32:55.773 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:32:56.057 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:32:57.058 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:32:57.059 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:32:57.059 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-15T21:32:57.059 INFO:tasks.workunit.client.0.smithi136.stderr:2 2024-09-15T21:32:57.059 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:32:57.059 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:32:57.381 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:32:58.036 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:32:58.033+0000 7f363200c640 -1 osd.0 193 log_to_monitors true 2024-09-15T21:32:58.383 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:32:58.383 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:32:58.383 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-15T21:32:58.383 INFO:tasks.workunit.client.0.smithi136.stderr:3 2024-09-15T21:32:58.383 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:32:58.383 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:32:58.745 INFO:tasks.workunit.client.0.smithi136.stderr:osd.0 up in weight 1 up_from 197 up_thru 191 down_at 194 last_clean_interval [185,193) [v2:127.0.0.1:6802/194192594,v1:127.0.0.1:6803/194192594] [v2:127.0.0.1:6804/194192594,v1:127.0.0.1:6805/194192594] exists,up 7a68df98-8e0f-45ef-a9e9-b2a45ca8fd3f 2024-09-15T21:32:58.745 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-15T21:32:58.745 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-15T21:32:58.745 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-15T21:32:58.745 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2024-09-15T21:32:58.745 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-15T21:32:58.746 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-15T21:32:58.746 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-15T21:32:58.746 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-15T21:32:58.746 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-15T21:32:58.746 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-15T21:32:58.746 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-15T21:32:58.746 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-15T21:32:58.746 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-15T21:32:58.840 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-15T21:32:58.840 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-15T21:32:58.840 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-15T21:32:58.840 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-15T21:32:58.840 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-15T21:32:58.840 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-15T21:32:59.240 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-15T21:32:59.241 INFO:tasks.workunit.client.0.smithi136.stderr:1' 2024-09-15T21:32:59.241 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-15T21:32:59.241 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T21:32:59.241 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-15T21:32:59.453 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=846108557314 2024-09-15T21:32:59.453 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 846108557314 2024-09-15T21:32:59.454 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-846108557314' 2024-09-15T21:32:59.454 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T21:32:59.454 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-15T21:32:59.579 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=820338753541 2024-09-15T21:32:59.579 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 820338753541 2024-09-15T21:32:59.579 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-846108557314 1-820338753541' 2024-09-15T21:32:59.579 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T21:32:59.579 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-846108557314 2024-09-15T21:32:59.580 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T21:32:59.580 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-15T21:32:59.581 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-846108557314 2024-09-15T21:32:59.581 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T21:32:59.582 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=846108557314 2024-09-15T21:32:59.582 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 846108557314' 2024-09-15T21:32:59.582 INFO:tasks.workunit.client.0.smithi136.stderr:waiting osd.0 seq 846108557314 2024-09-15T21:32:59.582 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T21:32:59.860 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 1 -lt 846108557314 2024-09-15T21:32:59.860 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-15T21:33:00.861 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-15T21:33:00.863 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T21:33:01.173 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 846108557314 -lt 846108557314 2024-09-15T21:33:01.174 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T21:33:01.174 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-820338753541 2024-09-15T21:33:01.174 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T21:33:01.175 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-15T21:33:01.176 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-820338753541 2024-09-15T21:33:01.176 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T21:33:01.177 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=820338753541 2024-09-15T21:33:01.177 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 820338753541' 2024-09-15T21:33:01.177 INFO:tasks.workunit.client.0.smithi136.stderr:waiting osd.1 seq 820338753541 2024-09-15T21:33:01.177 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-15T21:33:01.454 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 820338753542 -lt 820338753541 2024-09-15T21:33:01.455 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-15T21:33:01.455 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T21:33:01.455 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T21:33:01.797 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 6 == 0 2024-09-15T21:33:01.797 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-15T21:33:01.798 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-15T21:33:01.798 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-15T21:33:01.798 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T21:33:01.798 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T21:33:01.798 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T21:33:01.798 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T21:33:02.071 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=6 2024-09-15T21:33:02.071 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-15T21:33:02.072 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T21:33:02.072 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T21:33:02.417 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 = 6 2024-09-15T21:33:02.417 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-15T21:33:02.417 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-15T21:33:02.417 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2106: TEST_corrupt_scrub_replicated: objectstore_tool td/osd-scrub-repair 1 ROBJ10 set-attr _ td/osd-scrub-repair/oi 2024-09-15T21:33:02.417 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2024-09-15T21:33:02.417 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2024-09-15T21:33:02.418 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=1 2024-09-15T21:33:02.418 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2024-09-15T21:33:02.418 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 ROBJ10 set-attr _ td/osd-scrub-repair/oi 2024-09-15T21:33:02.418 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-09-15T21:33:02.418 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2024-09-15T21:33:02.418 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=1 2024-09-15T21:33:02.418 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2024-09-15T21:33:02.418 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-09-15T21:33:02.418 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-15T21:33:02.418 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-15T21:33:02.418 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-15T21:33:02.418 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-15T21:33:02.418 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-15T21:33:02.723 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-15T21:33:02.723 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 ROBJ10 set-attr _ td/osd-scrub-repair/oi 2024-09-15T21:33:02.723 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-15T21:33:02.723 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-15T21:33:02.723 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=1 2024-09-15T21:33:02.724 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-15T21:33:02.724 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-09-15T21:33:02.724 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 ROBJ10 set-attr _ td/osd-scrub-repair/oi 2024-09-15T21:33:03.778 INFO:tasks.workunit.client.0.smithi136.stderr:Error getting attr on : 3.0_head,#3:30259878:::ROBJ15:head#, (61) No data available 2024-09-15T21:33:03.778 INFO:tasks.workunit.client.0.smithi136.stderr:Error getting attr on : 3.0_head,#3:c0c86b1d:::ROBJ14:head#, (61) No data available 2024-09-15T21:33:04.295 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2024-09-15T21:33:04.295 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2024-09-15T21:33:04.295 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-15T21:33:04.295 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=1 2024-09-15T21:33:04.295 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-15T21:33:04.295 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-09-15T21:33:04.295 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=ce4892c7-6c6d-4b4f-8e6f-07a3e11133e5 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-15T21:33:04.295 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T21:33:04.295 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T21:33:04.295 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T21:33:04.295 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-09-15T21:33:04.296 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-09-15T21:33:04.296 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-15T21:33:04.296 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-15T21:33:04.296 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-15T21:33:04.296 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-15T21:33:04.296 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-15T21:33:04.296 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-15T21:33:04.296 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-15T21:33:04.296 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T21:33:04.296 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52602 2024-09-15T21:33:04.296 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.52602/$cluster-$name.asok' 2024-09-15T21:33:04.298 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.52602/$cluster-$name.asok' 2024-09-15T21:33:04.298 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-15T21:33:04.298 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-15T21:33:04.298 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-15T21:33:04.298 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T21:33:04.298 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T21:33:04.298 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T21:33:04.298 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T21:33:04.298 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-15T21:33:04.298 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-15T21:33:04.298 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-09-15T21:33:04.298 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.1 2024-09-15T21:33:04.299 INFO:tasks.workunit.client.0.smithi136.stderr:start osd.1 2024-09-15T21:33:04.299 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 1 --fsid=ce4892c7-6c6d-4b4f-8e6f-07a3e11133e5 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.52602/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T21:33:04.299 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-09-15T21:33:04.299 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 1 = 1 ']' 2024-09-15T21:33:04.299 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-15T21:33:04.299 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-15T21:33:04.299 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-15T21:33:04.327 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:33:04.324+0000 7f02d4233640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:33:04.329 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:33:04.326+0000 7f02d4233640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:33:04.330 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:33:04.328+0000 7f02d4233640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:33:04.587 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 1 2024-09-15T21:33:04.587 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-15T21:33:04.587 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2024-09-15T21:33:04.587 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-15T21:33:04.588 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-15T21:33:04.588 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:33:04.588 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-15T21:33:04.588 INFO:tasks.workunit.client.0.smithi136.stderr:0 2024-09-15T21:33:04.588 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:33:04.588 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T21:33:04.864 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:33:05.230 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:33:05.228+0000 7f02d4233640 -1 Falling back to public interface 2024-09-15T21:33:05.865 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:33:05.865 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:33:05.866 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-15T21:33:05.866 INFO:tasks.workunit.client.0.smithi136.stderr:1 2024-09-15T21:33:05.866 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:33:05.866 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T21:33:06.155 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:33:07.157 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:33:07.157 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:33:07.157 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-15T21:33:07.157 INFO:tasks.workunit.client.0.smithi136.stderr:2 2024-09-15T21:33:07.157 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:33:07.157 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T21:33:07.435 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:33:07.865 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:33:07.862+0000 7f02d4233640 -1 osd.1 199 log_to_monitors true 2024-09-15T21:33:08.437 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:33:08.437 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:33:08.437 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-15T21:33:08.437 INFO:tasks.workunit.client.0.smithi136.stderr:3 2024-09-15T21:33:08.437 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:33:08.437 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T21:33:08.832 INFO:tasks.workunit.client.0.smithi136.stderr:osd.1 up in weight 1 up_from 203 up_thru 203 down_at 200 last_clean_interval [191,199) [v2:127.0.0.1:6810/1430947547,v1:127.0.0.1:6811/1430947547] [v2:127.0.0.1:6812/1430947547,v1:127.0.0.1:6813/1430947547] exists,up 9aec06d1-33ff-42ab-be71-ac761d546d96 2024-09-15T21:33:08.833 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-15T21:33:08.833 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-15T21:33:08.833 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-15T21:33:08.833 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2024-09-15T21:33:08.833 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-15T21:33:08.833 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-15T21:33:08.833 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-15T21:33:08.833 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-15T21:33:08.833 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-15T21:33:08.833 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-15T21:33:08.833 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-15T21:33:08.833 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-15T21:33:08.833 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-15T21:33:08.926 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-15T21:33:08.926 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-15T21:33:08.926 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-15T21:33:08.927 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-15T21:33:08.927 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-15T21:33:08.927 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-15T21:33:09.206 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-15T21:33:09.206 INFO:tasks.workunit.client.0.smithi136.stderr:1' 2024-09-15T21:33:09.206 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-15T21:33:09.206 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T21:33:09.206 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-15T21:33:09.414 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=846108557317 2024-09-15T21:33:09.414 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 846108557317 2024-09-15T21:33:09.414 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-846108557317' 2024-09-15T21:33:09.414 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T21:33:09.415 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-15T21:33:09.531 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=871878361090 2024-09-15T21:33:09.532 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 871878361090 2024-09-15T21:33:09.532 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-846108557317 1-871878361090' 2024-09-15T21:33:09.532 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T21:33:09.532 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-846108557317 2024-09-15T21:33:09.532 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T21:33:09.533 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-15T21:33:09.533 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-846108557317 2024-09-15T21:33:09.533 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T21:33:09.534 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=846108557317 2024-09-15T21:33:09.535 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 846108557317' 2024-09-15T21:33:09.535 INFO:tasks.workunit.client.0.smithi136.stderr:waiting osd.0 seq 846108557317 2024-09-15T21:33:09.535 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T21:33:09.811 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 846108557316 -lt 846108557317 2024-09-15T21:33:09.812 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-15T21:33:10.813 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-15T21:33:10.813 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T21:33:11.126 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 846108557316 -lt 846108557317 2024-09-15T21:33:11.126 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-15T21:33:12.128 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 299 -eq 0 ']' 2024-09-15T21:33:12.128 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T21:33:12.414 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 846108557317 -lt 846108557317 2024-09-15T21:33:12.415 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T21:33:12.415 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-871878361090 2024-09-15T21:33:12.415 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T21:33:12.416 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-15T21:33:12.417 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-871878361090 2024-09-15T21:33:12.417 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T21:33:12.418 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=871878361090 2024-09-15T21:33:12.418 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 871878361090' 2024-09-15T21:33:12.418 INFO:tasks.workunit.client.0.smithi136.stderr:waiting osd.1 seq 871878361090 2024-09-15T21:33:12.418 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-15T21:33:12.695 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 871878361090 -lt 871878361090 2024-09-15T21:33:12.695 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-15T21:33:12.696 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T21:33:12.696 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T21:33:13.039 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 6 == 0 2024-09-15T21:33:13.039 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-15T21:33:13.039 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-15T21:33:13.039 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-15T21:33:13.039 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T21:33:13.039 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T21:33:13.039 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T21:33:13.039 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T21:33:13.310 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=6 2024-09-15T21:33:13.311 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-15T21:33:13.311 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T21:33:13.311 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T21:33:13.656 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 = 6 2024-09-15T21:33:13.656 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-15T21:33:13.656 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-15T21:33:13.657 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2107: TEST_corrupt_scrub_replicated: rm td/osd-scrub-repair/oi 2024-09-15T21:33:13.659 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2109: TEST_corrupt_scrub_replicated: inject_eio rep data csr_pool ROBJ11 td/osd-scrub-repair 0 2024-09-15T21:33:13.659 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2402: inject_eio: local pooltype=rep 2024-09-15T21:33:13.659 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2403: inject_eio: shift 2024-09-15T21:33:13.659 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2404: inject_eio: local which=data 2024-09-15T21:33:13.659 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2405: inject_eio: shift 2024-09-15T21:33:13.659 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2406: inject_eio: local poolname=csr_pool 2024-09-15T21:33:13.659 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2407: inject_eio: shift 2024-09-15T21:33:13.659 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2408: inject_eio: local objname=ROBJ11 2024-09-15T21:33:13.659 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2409: inject_eio: shift 2024-09-15T21:33:13.659 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2410: inject_eio: local dir=td/osd-scrub-repair 2024-09-15T21:33:13.659 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2411: inject_eio: shift 2024-09-15T21:33:13.660 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2412: inject_eio: local shard_id=0 2024-09-15T21:33:13.660 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2413: inject_eio: shift 2024-09-15T21:33:13.660 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: get_osds csr_pool ROBJ11 2024-09-15T21:33:13.660 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1012: get_osds: local poolname=csr_pool 2024-09-15T21:33:13.660 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1013: get_osds: local objectname=ROBJ11 2024-09-15T21:33:13.660 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: ceph --format json osd map csr_pool ROBJ11 2024-09-15T21:33:13.660 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: jq '.acting | .[]' 2024-09-15T21:33:13.937 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: local 'osds=1 2024-09-15T21:33:13.937 INFO:tasks.workunit.client.0.smithi136.stderr:0' 2024-09-15T21:33:13.937 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1018: get_osds: echo 1 0 2024-09-15T21:33:13.937 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: initial_osds=('1' '0') 2024-09-15T21:33:13.937 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: local -a initial_osds 2024-09-15T21:33:13.937 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2416: inject_eio: local osd_id=1 2024-09-15T21:33:13.938 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2417: inject_eio: '[' rep '!=' ec ']' 2024-09-15T21:33:13.938 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2418: inject_eio: shard_id= 2024-09-15T21:33:13.938 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2420: inject_eio: cat td/osd-scrub-repair/1/type 2024-09-15T21:33:13.939 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2420: inject_eio: type=bluestore 2024-09-15T21:33:13.939 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2421: inject_eio: set_config osd 1 bluestore_debug_inject_read_err true 2024-09-15T21:33:13.939 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1152: set_config: local daemon=osd 2024-09-15T21:33:13.939 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1153: set_config: local id=1 2024-09-15T21:33:13.939 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1154: set_config: local config=bluestore_debug_inject_read_err 2024-09-15T21:33:13.939 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1155: set_config: local value=true 2024-09-15T21:33:13.939 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: jq 'has("success")' 2024-09-15T21:33:13.939 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: get_asok_path osd.1 2024-09-15T21:33:13.940 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.1 2024-09-15T21:33:13.940 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.1 ']' 2024-09-15T21:33:13.940 INFO:tasks.workunit.client.0.smithi136.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-15T21:33:13.940 INFO:tasks.workunit.client.0.smithi136.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T21:33:13.940 INFO:tasks.workunit.client.0.smithi136.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52602 2024-09-15T21:33:13.941 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.52602/ceph-osd.1.asok 2024-09-15T21:33:13.941 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.52602/ceph-osd.1.asok config set bluestore_debug_inject_read_err true 2024-09-15T21:33:14.026 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: test true == true 2024-09-15T21:33:14.026 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2422: inject_eio: local loop=0 2024-09-15T21:33:14.027 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2424: inject_eio: grep -q Invalid 2024-09-15T21:33:14.027 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: get_asok_path osd.1 2024-09-15T21:33:14.027 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.1 2024-09-15T21:33:14.027 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.1 ']' 2024-09-15T21:33:14.027 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-15T21:33:14.027 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T21:33:14.027 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52602 2024-09-15T21:33:14.027 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.52602/ceph-osd.1.asok 2024-09-15T21:33:14.028 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: CEPH_ARGS= 2024-09-15T21:33:14.028 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: ceph --admin-daemon /tmp/ceph-asok.52602/ceph-osd.1.asok injectdataerr csr_pool ROBJ11 2024-09-15T21:33:14.116 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2110: TEST_corrupt_scrub_replicated: inject_eio rep mdata csr_pool ROBJ12 td/osd-scrub-repair 1 2024-09-15T21:33:14.116 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2402: inject_eio: local pooltype=rep 2024-09-15T21:33:14.116 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2403: inject_eio: shift 2024-09-15T21:33:14.116 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2404: inject_eio: local which=mdata 2024-09-15T21:33:14.116 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2405: inject_eio: shift 2024-09-15T21:33:14.116 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2406: inject_eio: local poolname=csr_pool 2024-09-15T21:33:14.116 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2407: inject_eio: shift 2024-09-15T21:33:14.116 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2408: inject_eio: local objname=ROBJ12 2024-09-15T21:33:14.116 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2409: inject_eio: shift 2024-09-15T21:33:14.116 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2410: inject_eio: local dir=td/osd-scrub-repair 2024-09-15T21:33:14.116 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2411: inject_eio: shift 2024-09-15T21:33:14.117 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2412: inject_eio: local shard_id=1 2024-09-15T21:33:14.117 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2413: inject_eio: shift 2024-09-15T21:33:14.117 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: get_osds csr_pool ROBJ12 2024-09-15T21:33:14.117 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1012: get_osds: local poolname=csr_pool 2024-09-15T21:33:14.117 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1013: get_osds: local objectname=ROBJ12 2024-09-15T21:33:14.117 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: ceph --format json osd map csr_pool ROBJ12 2024-09-15T21:33:14.117 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: jq '.acting | .[]' 2024-09-15T21:33:14.434 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: local 'osds=1 2024-09-15T21:33:14.434 INFO:tasks.workunit.client.0.smithi136.stderr:0' 2024-09-15T21:33:14.434 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1018: get_osds: echo 1 0 2024-09-15T21:33:14.434 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: initial_osds=('1' '0') 2024-09-15T21:33:14.434 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: local -a initial_osds 2024-09-15T21:33:14.434 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2416: inject_eio: local osd_id=0 2024-09-15T21:33:14.434 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2417: inject_eio: '[' rep '!=' ec ']' 2024-09-15T21:33:14.434 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2418: inject_eio: shard_id= 2024-09-15T21:33:14.434 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2420: inject_eio: cat td/osd-scrub-repair/0/type 2024-09-15T21:33:14.435 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2420: inject_eio: type=bluestore 2024-09-15T21:33:14.435 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2421: inject_eio: set_config osd 0 bluestore_debug_inject_read_err true 2024-09-15T21:33:14.435 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1152: set_config: local daemon=osd 2024-09-15T21:33:14.436 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1153: set_config: local id=0 2024-09-15T21:33:14.436 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1154: set_config: local config=bluestore_debug_inject_read_err 2024-09-15T21:33:14.436 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1155: set_config: local value=true 2024-09-15T21:33:14.437 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: jq 'has("success")' 2024-09-15T21:33:14.437 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: get_asok_path osd.0 2024-09-15T21:33:14.437 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.0 2024-09-15T21:33:14.437 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.0 ']' 2024-09-15T21:33:14.437 INFO:tasks.workunit.client.0.smithi136.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-15T21:33:14.437 INFO:tasks.workunit.client.0.smithi136.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T21:33:14.437 INFO:tasks.workunit.client.0.smithi136.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52602 2024-09-15T21:33:14.437 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.52602/ceph-osd.0.asok 2024-09-15T21:33:14.437 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.52602/ceph-osd.0.asok config set bluestore_debug_inject_read_err true 2024-09-15T21:33:14.522 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: test true == true 2024-09-15T21:33:14.522 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2422: inject_eio: local loop=0 2024-09-15T21:33:14.523 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2424: inject_eio: grep -q Invalid 2024-09-15T21:33:14.524 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: get_asok_path osd.0 2024-09-15T21:33:14.524 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.0 2024-09-15T21:33:14.524 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.0 ']' 2024-09-15T21:33:14.524 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-15T21:33:14.524 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T21:33:14.524 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52602 2024-09-15T21:33:14.524 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.52602/ceph-osd.0.asok 2024-09-15T21:33:14.524 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: CEPH_ARGS= 2024-09-15T21:33:14.525 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: ceph --admin-daemon /tmp/ceph-asok.52602/ceph-osd.0.asok injectmdataerr csr_pool ROBJ12 2024-09-15T21:33:14.613 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2111: TEST_corrupt_scrub_replicated: inject_eio rep mdata csr_pool ROBJ13 td/osd-scrub-repair 1 2024-09-15T21:33:14.613 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2402: inject_eio: local pooltype=rep 2024-09-15T21:33:14.613 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2403: inject_eio: shift 2024-09-15T21:33:14.613 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2404: inject_eio: local which=mdata 2024-09-15T21:33:14.613 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2405: inject_eio: shift 2024-09-15T21:33:14.613 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2406: inject_eio: local poolname=csr_pool 2024-09-15T21:33:14.614 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2407: inject_eio: shift 2024-09-15T21:33:14.614 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2408: inject_eio: local objname=ROBJ13 2024-09-15T21:33:14.614 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2409: inject_eio: shift 2024-09-15T21:33:14.614 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2410: inject_eio: local dir=td/osd-scrub-repair 2024-09-15T21:33:14.614 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2411: inject_eio: shift 2024-09-15T21:33:14.614 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2412: inject_eio: local shard_id=1 2024-09-15T21:33:14.614 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2413: inject_eio: shift 2024-09-15T21:33:14.614 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: get_osds csr_pool ROBJ13 2024-09-15T21:33:14.614 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1012: get_osds: local poolname=csr_pool 2024-09-15T21:33:14.614 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1013: get_osds: local objectname=ROBJ13 2024-09-15T21:33:14.614 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: ceph --format json osd map csr_pool ROBJ13 2024-09-15T21:33:14.614 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: jq '.acting | .[]' 2024-09-15T21:33:14.889 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: local 'osds=1 2024-09-15T21:33:14.889 INFO:tasks.workunit.client.0.smithi136.stderr:0' 2024-09-15T21:33:14.890 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1018: get_osds: echo 1 0 2024-09-15T21:33:14.890 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: initial_osds=('1' '0') 2024-09-15T21:33:14.890 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: local -a initial_osds 2024-09-15T21:33:14.890 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2416: inject_eio: local osd_id=0 2024-09-15T21:33:14.890 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2417: inject_eio: '[' rep '!=' ec ']' 2024-09-15T21:33:14.890 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2418: inject_eio: shard_id= 2024-09-15T21:33:14.890 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2420: inject_eio: cat td/osd-scrub-repair/0/type 2024-09-15T21:33:14.891 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2420: inject_eio: type=bluestore 2024-09-15T21:33:14.891 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2421: inject_eio: set_config osd 0 bluestore_debug_inject_read_err true 2024-09-15T21:33:14.891 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1152: set_config: local daemon=osd 2024-09-15T21:33:14.892 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1153: set_config: local id=0 2024-09-15T21:33:14.892 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1154: set_config: local config=bluestore_debug_inject_read_err 2024-09-15T21:33:14.892 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1155: set_config: local value=true 2024-09-15T21:33:14.892 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: jq 'has("success")' 2024-09-15T21:33:14.892 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: get_asok_path osd.0 2024-09-15T21:33:14.892 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.0 2024-09-15T21:33:14.892 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.0 ']' 2024-09-15T21:33:14.893 INFO:tasks.workunit.client.0.smithi136.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-15T21:33:14.893 INFO:tasks.workunit.client.0.smithi136.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T21:33:14.893 INFO:tasks.workunit.client.0.smithi136.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52602 2024-09-15T21:33:14.893 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.52602/ceph-osd.0.asok 2024-09-15T21:33:14.893 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.52602/ceph-osd.0.asok config set bluestore_debug_inject_read_err true 2024-09-15T21:33:14.978 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: test true == true 2024-09-15T21:33:14.978 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2422: inject_eio: local loop=0 2024-09-15T21:33:14.979 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2424: inject_eio: grep -q Invalid 2024-09-15T21:33:14.979 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: get_asok_path osd.0 2024-09-15T21:33:14.980 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.0 2024-09-15T21:33:14.980 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.0 ']' 2024-09-15T21:33:14.980 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-15T21:33:14.980 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T21:33:14.980 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52602 2024-09-15T21:33:14.980 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.52602/ceph-osd.0.asok 2024-09-15T21:33:14.980 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: CEPH_ARGS= 2024-09-15T21:33:14.980 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: ceph --admin-daemon /tmp/ceph-asok.52602/ceph-osd.0.asok injectmdataerr csr_pool ROBJ13 2024-09-15T21:33:15.069 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2112: TEST_corrupt_scrub_replicated: inject_eio rep data csr_pool ROBJ13 td/osd-scrub-repair 0 2024-09-15T21:33:15.069 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2402: inject_eio: local pooltype=rep 2024-09-15T21:33:15.069 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2403: inject_eio: shift 2024-09-15T21:33:15.069 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2404: inject_eio: local which=data 2024-09-15T21:33:15.069 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2405: inject_eio: shift 2024-09-15T21:33:15.069 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2406: inject_eio: local poolname=csr_pool 2024-09-15T21:33:15.069 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2407: inject_eio: shift 2024-09-15T21:33:15.069 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2408: inject_eio: local objname=ROBJ13 2024-09-15T21:33:15.069 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2409: inject_eio: shift 2024-09-15T21:33:15.069 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2410: inject_eio: local dir=td/osd-scrub-repair 2024-09-15T21:33:15.070 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2411: inject_eio: shift 2024-09-15T21:33:15.070 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2412: inject_eio: local shard_id=0 2024-09-15T21:33:15.070 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2413: inject_eio: shift 2024-09-15T21:33:15.070 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: get_osds csr_pool ROBJ13 2024-09-15T21:33:15.070 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1012: get_osds: local poolname=csr_pool 2024-09-15T21:33:15.070 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1013: get_osds: local objectname=ROBJ13 2024-09-15T21:33:15.070 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: ceph --format json osd map csr_pool ROBJ13 2024-09-15T21:33:15.070 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: jq '.acting | .[]' 2024-09-15T21:33:15.348 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: local 'osds=1 2024-09-15T21:33:15.348 INFO:tasks.workunit.client.0.smithi136.stderr:0' 2024-09-15T21:33:15.348 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1018: get_osds: echo 1 0 2024-09-15T21:33:15.348 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: initial_osds=('1' '0') 2024-09-15T21:33:15.348 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: local -a initial_osds 2024-09-15T21:33:15.348 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2416: inject_eio: local osd_id=1 2024-09-15T21:33:15.348 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2417: inject_eio: '[' rep '!=' ec ']' 2024-09-15T21:33:15.349 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2418: inject_eio: shard_id= 2024-09-15T21:33:15.349 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2420: inject_eio: cat td/osd-scrub-repair/1/type 2024-09-15T21:33:15.350 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2420: inject_eio: type=bluestore 2024-09-15T21:33:15.351 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2421: inject_eio: set_config osd 1 bluestore_debug_inject_read_err true 2024-09-15T21:33:15.351 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1152: set_config: local daemon=osd 2024-09-15T21:33:15.351 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1153: set_config: local id=1 2024-09-15T21:33:15.351 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1154: set_config: local config=bluestore_debug_inject_read_err 2024-09-15T21:33:15.351 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1155: set_config: local value=true 2024-09-15T21:33:15.351 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: jq 'has("success")' 2024-09-15T21:33:15.351 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: get_asok_path osd.1 2024-09-15T21:33:15.351 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.1 2024-09-15T21:33:15.351 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.1 ']' 2024-09-15T21:33:15.351 INFO:tasks.workunit.client.0.smithi136.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-15T21:33:15.351 INFO:tasks.workunit.client.0.smithi136.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T21:33:15.351 INFO:tasks.workunit.client.0.smithi136.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52602 2024-09-15T21:33:15.352 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.52602/ceph-osd.1.asok 2024-09-15T21:33:15.352 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.52602/ceph-osd.1.asok config set bluestore_debug_inject_read_err true 2024-09-15T21:33:15.436 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: test true == true 2024-09-15T21:33:15.436 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2422: inject_eio: local loop=0 2024-09-15T21:33:15.437 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2424: inject_eio: grep -q Invalid 2024-09-15T21:33:15.437 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: get_asok_path osd.1 2024-09-15T21:33:15.437 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.1 2024-09-15T21:33:15.437 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.1 ']' 2024-09-15T21:33:15.437 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-15T21:33:15.437 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T21:33:15.437 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52602 2024-09-15T21:33:15.437 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.52602/ceph-osd.1.asok 2024-09-15T21:33:15.438 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: CEPH_ARGS= 2024-09-15T21:33:15.438 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: ceph --admin-daemon /tmp/ceph-asok.52602/ceph-osd.1.asok injectdataerr csr_pool ROBJ13 2024-09-15T21:33:15.523 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2115: TEST_corrupt_scrub_replicated: ceph tell 'osd.*' injectargs -- --osd-max-object-size=134217728 2024-09-15T21:33:15.617 INFO:tasks.workunit.client.0.smithi136.stdout:osd.0: {} 2024-09-15T21:33:15.618 INFO:tasks.workunit.client.0.smithi136.stderr:osd.0: osd_max_object_size = '' 2024-09-15T21:33:15.634 INFO:tasks.workunit.client.0.smithi136.stdout:osd.1: {} 2024-09-15T21:33:15.634 INFO:tasks.workunit.client.0.smithi136.stderr:osd.1: osd_max_object_size = '' 2024-09-15T21:33:15.643 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2117: TEST_corrupt_scrub_replicated: pg_deep_scrub 3.0 2024-09-15T21:33:15.643 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1890: pg_deep_scrub: local pgid=3.0 2024-09-15T21:33:15.644 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1891: pg_deep_scrub: get_last_scrub_stamp 3.0 last_deep_scrub_stamp 2024-09-15T21:33:15.644 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=3.0 2024-09-15T21:33:15.644 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_deep_scrub_stamp 2024-09-15T21:33:15.644 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-15T21:33:15.644 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_deep_scrub_stamp' 2024-09-15T21:33:15.939 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1891: pg_deep_scrub: local last_scrub=2024-09-15T21:27:56.080190+0000 2024-09-15T21:33:15.939 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1892: pg_deep_scrub: ceph pg deep-scrub 3.0 2024-09-15T21:33:16.201 INFO:tasks.workunit.client.0.smithi136.stderr:instructing pg 3.0 on osd.1 to deep-scrub 2024-09-15T21:33:16.213 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1893: pg_deep_scrub: wait_for_scrub 3.0 2024-09-15T21:27:56.080190+0000 last_deep_scrub_stamp 2024-09-15T21:33:16.213 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: local pgid=3.0 2024-09-15T21:33:16.214 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2018: wait_for_scrub: local last_scrub=2024-09-15T21:27:56.080190+0000 2024-09-15T21:33:16.214 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2019: wait_for_scrub: local sname=last_deep_scrub_stamp 2024-09-15T21:33:16.214 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i=0 )) 2024-09-15T21:33:16.214 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-15T21:33:16.214 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 3.0 last_deep_scrub_stamp 2024-09-15T21:33:16.214 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=3.0 2024-09-15T21:33:16.214 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_deep_scrub_stamp 2024-09-15T21:33:16.214 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-15T21:33:16.214 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_deep_scrub_stamp' 2024-09-15T21:33:16.484 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-15T21:27:56.080190+0000 '>' 2024-09-15T21:27:56.080190+0000 2024-09-15T21:33:16.484 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-15T21:33:17.488 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-15T21:33:17.488 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-15T21:33:17.489 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 3.0 last_deep_scrub_stamp 2024-09-15T21:33:17.489 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=3.0 2024-09-15T21:33:17.489 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_deep_scrub_stamp 2024-09-15T21:33:17.489 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-15T21:33:17.489 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_deep_scrub_stamp' 2024-09-15T21:33:17.766 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-15T21:27:56.080190+0000 '>' 2024-09-15T21:27:56.080190+0000 2024-09-15T21:33:17.766 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-15T21:33:18.767 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-15T21:33:18.767 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-15T21:33:18.768 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 3.0 last_deep_scrub_stamp 2024-09-15T21:33:18.768 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=3.0 2024-09-15T21:33:18.768 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_deep_scrub_stamp 2024-09-15T21:33:18.769 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-15T21:33:18.769 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_deep_scrub_stamp' 2024-09-15T21:33:19.054 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-15T21:33:17.249222+0000 '>' 2024-09-15T21:27:56.080190+0000 2024-09-15T21:33:19.054 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2023: wait_for_scrub: return 0 2024-09-15T21:33:19.054 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2119: TEST_corrupt_scrub_replicated: err_strings=() 2024-09-15T21:33:19.054 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2120: TEST_corrupt_scrub_replicated: err_strings[0]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:30259878:::ROBJ15:head : candidate had a missing info key' 2024-09-15T21:33:19.055 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2121: TEST_corrupt_scrub_replicated: err_strings[1]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 0 soid 3:33aca486:::ROBJ18:head : data_digest 0xbd89c912 != data_digest 0x2ddbf8f5 from auth oi 3:33aca486:::ROBJ18:head[(][0-9]*'\''[0-9]* osd.1.0:[0-9]* dirty|omap|data_digest|omap_digest s 7 uv 54 dd 2ddbf8f5 od ddc3680f alloc_hint [[]0 0 255[]][)], object info inconsistent ' 2024-09-15T21:33:19.055 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2122: TEST_corrupt_scrub_replicated: err_strings[2]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:33aca486:::ROBJ18:head : data_digest 0xbd89c912 != data_digest 0x2ddbf8f5 from auth oi 3:33aca486:::ROBJ18:head[(][0-9]*'\''[0-9]* osd.1.0:[0-9]* dirty|omap|data_digest|omap_digest s 7 uv 54 dd 2ddbf8f5 od ddc3680f alloc_hint [[]0 0 255[]][)]' 2024-09-15T21:33:19.055 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2123: TEST_corrupt_scrub_replicated: err_strings[3]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid 3:33aca486:::ROBJ18:head : failed to pick suitable auth object' 2024-09-15T21:33:19.055 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2124: TEST_corrupt_scrub_replicated: err_strings[4]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:5c7b2c47:::ROBJ16:head : candidate had a corrupt snapset' 2024-09-15T21:33:19.055 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2125: TEST_corrupt_scrub_replicated: err_strings[5]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 0 soid 3:5c7b2c47:::ROBJ16:head : candidate had a missing snapset key' 2024-09-15T21:33:19.055 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2126: TEST_corrupt_scrub_replicated: err_strings[6]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid 3:5c7b2c47:::ROBJ16:head : failed to pick suitable object info' 2024-09-15T21:33:19.055 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2127: TEST_corrupt_scrub_replicated: err_strings[7]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid 3:86586531:::ROBJ8:head : attr value mismatch '\''_key1-ROBJ8'\'', attr name mismatch '\''_key3-ROBJ8'\'', attr name mismatch '\''_key2-ROBJ8'\''' 2024-09-15T21:33:19.055 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2128: TEST_corrupt_scrub_replicated: err_strings[8]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:87abbf36:::ROBJ11:head : candidate had a read error' 2024-09-15T21:33:19.055 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2129: TEST_corrupt_scrub_replicated: err_strings[9]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 0 soid 3:8aa5320e:::ROBJ17:head : data_digest 0x5af0c3ef != data_digest 0x2ddbf8f5 from auth oi 3:8aa5320e:::ROBJ17:head[(][0-9]*'\''[0-9]* osd.1.0:[0-9]* dirty|omap|data_digest|omap_digest s 7 uv 51 dd 2ddbf8f5 od e9572720 alloc_hint [[]0 0 0[]][)]' 2024-09-15T21:33:19.055 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2130: TEST_corrupt_scrub_replicated: err_strings[10]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:8aa5320e:::ROBJ17:head : data_digest 0x5af0c3ef != data_digest 0x2ddbf8f5 from auth oi 3:8aa5320e:::ROBJ17:head[(][0-9]*'\''[0-9]* osd.1.0:[0-9]* dirty|omap|data_digest|omap_digest s 7 uv 51 dd 2ddbf8f5 od e9572720 alloc_hint [[]0 0 0[]][)]' 2024-09-15T21:33:19.055 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2131: TEST_corrupt_scrub_replicated: err_strings[11]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid 3:8aa5320e:::ROBJ17:head : failed to pick suitable auth object' 2024-09-15T21:33:19.056 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2132: TEST_corrupt_scrub_replicated: err_strings[12]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid 3:8b55fa4b:::ROBJ7:head : omap_digest 0xefced57a != omap_digest 0x6a73cc07 from shard 1' 2024-09-15T21:33:19.056 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2133: TEST_corrupt_scrub_replicated: err_strings[13]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:8b55fa4b:::ROBJ7:head : omap_digest 0x6a73cc07 != omap_digest 0xefced57a from auth oi 3:8b55fa4b:::ROBJ7:head[(][0-9]*'\''[0-9]* osd.1.0:[0-9]* dirty|omap|data_digest|omap_digest s 7 uv 21 dd 2ddbf8f5 od efced57a alloc_hint [[]0 0 0[]][)]' 2024-09-15T21:33:19.056 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2134: TEST_corrupt_scrub_replicated: err_strings[14]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 0 soid 3:a53c12e8:::ROBJ6:head : omap_digest 0x689ee887 != omap_digest 0x179c919f from shard 1, omap_digest 0x689ee887 != omap_digest 0x179c919f from auth oi 3:a53c12e8:::ROBJ6:head[(][0-9]*'\''[0-9]* osd.1.0:[0-9]* dirty|omap|data_digest|omap_digest s 7 uv 18 dd 2ddbf8f5 od 179c919f alloc_hint [[]0 0 0[]][)]' 2024-09-15T21:33:19.056 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2135: TEST_corrupt_scrub_replicated: err_strings[15]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 0 soid 3:b1f19cbd:::ROBJ10:head : omap_digest 0xa8dd5adc != omap_digest 0xc2025a24 from auth oi 3:b1f19cbd:::ROBJ10:head[(][0-9]*'\''[0-9]* osd.1.0:[0-9]* dirty|omap|data_digest|omap_digest s 7 uv 30 dd 2ddbf8f5 od c2025a24 alloc_hint [[]0 0 0[]][)]' 2024-09-15T21:33:19.056 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2136: TEST_corrupt_scrub_replicated: err_strings[16]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:b1f19cbd:::ROBJ10:head : omap_digest 0xa8dd5adc != omap_digest 0xc2025a24 from auth oi 3:b1f19cbd:::ROBJ10:head[(][0-9]*'\''[0-9]* osd.1.0:[0-9]* dirty|omap|data_digest|omap_digest s 7 uv 30 dd 2ddbf8f5 od c2025a24 alloc_hint [[]0 0 0[]][)]' 2024-09-15T21:33:19.056 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2137: TEST_corrupt_scrub_replicated: err_strings[17]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid 3:b1f19cbd:::ROBJ10:head : failed to pick suitable auth object' 2024-09-15T21:33:19.056 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2138: TEST_corrupt_scrub_replicated: err_strings[18]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 0 soid 3:bc819597:::ROBJ12:head : candidate had a stat error' 2024-09-15T21:33:19.056 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2139: TEST_corrupt_scrub_replicated: err_strings[19]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:c0c86b1d:::ROBJ14:head : candidate had a missing info key' 2024-09-15T21:33:19.056 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2140: TEST_corrupt_scrub_replicated: err_strings[20]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 0 soid 3:c0c86b1d:::ROBJ14:head : candidate had a corrupt info' 2024-09-15T21:33:19.056 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2141: TEST_corrupt_scrub_replicated: err_strings[21]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid 3:c0c86b1d:::ROBJ14:head : failed to pick suitable object info' 2024-09-15T21:33:19.056 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2142: TEST_corrupt_scrub_replicated: err_strings[22]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:ce3f1d6a:::ROBJ1:head : candidate size 9 info size 7 mismatch' 2024-09-15T21:33:19.057 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2143: TEST_corrupt_scrub_replicated: err_strings[23]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:ce3f1d6a:::ROBJ1:head : data_digest 0x2d4a11c2 != data_digest 0x2ddbf8f5 from shard 0, data_digest 0x2d4a11c2 != data_digest 0x2ddbf8f5 from auth oi 3:ce3f1d6a:::ROBJ1:head[(][0-9]*'\''[0-9]* osd.1.0:[0-9]* dirty|omap|data_digest|omap_digest s 7 uv 3 dd 2ddbf8f5 od f5fba2c6 alloc_hint [[]0 0 0[]][)], size 9 != size 7 from auth oi 3:ce3f1d6a:::ROBJ1:head[(][0-9]*'\''[0-9]* osd.1.0:[0-9]* dirty|omap|data_digest|omap_digest s 7 uv 3 dd 2ddbf8f5 od f5fba2c6 alloc_hint [[]0 0 0[]][)], size 9 != size 7 from shard 0' 2024-09-15T21:33:19.057 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2144: TEST_corrupt_scrub_replicated: err_strings[24]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:d60617f9:::ROBJ13:head : candidate had a read error' 2024-09-15T21:33:19.057 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2145: TEST_corrupt_scrub_replicated: err_strings[25]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 0 soid 3:d60617f9:::ROBJ13:head : candidate had a stat error' 2024-09-15T21:33:19.057 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2146: TEST_corrupt_scrub_replicated: err_strings[26]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid 3:d60617f9:::ROBJ13:head : failed to pick suitable object info' 2024-09-15T21:33:19.057 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2147: TEST_corrupt_scrub_replicated: err_strings[27]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 0 soid 3:e97ce31e:::ROBJ2:head : data_digest 0x578a4830 != data_digest 0x2ddbf8f5 from shard 1, data_digest 0x578a4830 != data_digest 0x2ddbf8f5 from auth oi 3:e97ce31e:::ROBJ2:head[(][0-9]*'\''[0-9]* osd.1.0:[0-9]* dirty|omap|data_digest|omap_digest s 7 uv 6 dd 2ddbf8f5 od f8e11918 alloc_hint [[]0 0 0[]][)]' 2024-09-15T21:33:19.057 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2148: TEST_corrupt_scrub_replicated: err_strings[28]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 3:f2a5b2a4:::ROBJ3:head : missing' 2024-09-15T21:33:19.057 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2149: TEST_corrupt_scrub_replicated: err_strings[29]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 0 soid 3:f4981d31:::ROBJ4:head : omap_digest 0xd7178dfe != omap_digest 0xe2d46ea4 from shard 1, omap_digest 0xd7178dfe != omap_digest 0xe2d46ea4 from auth oi 3:f4981d31:::ROBJ4:head[(][0-9]*'\''[0-9]* osd.1.0:[0-9]* dirty|omap|data_digest|omap_digest s 7 uv 12 dd 2ddbf8f5 od e2d46ea4 alloc_hint [[]0 0 0[]][)]' 2024-09-15T21:33:19.057 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2150: TEST_corrupt_scrub_replicated: err_strings[30]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid 3:f4bfd4d1:::ROBJ5:head : omap_digest 0x1a862a41 != omap_digest 0x6cac8f6 from shard 1' 2024-09-15T21:33:19.057 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2151: TEST_corrupt_scrub_replicated: err_strings[31]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:f4bfd4d1:::ROBJ5:head : omap_digest 0x6cac8f6 != omap_digest 0x1a862a41 from auth oi 3:f4bfd4d1:::ROBJ5:head[(][0-9]*'\''[0-9]* osd.1.0:[0-9]* dirty|omap|data_digest|omap_digest s 7 uv 15 dd 2ddbf8f5 od 1a862a41 alloc_hint [[]0 0 0[]][)]' 2024-09-15T21:33:19.057 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2152: TEST_corrupt_scrub_replicated: err_strings[32]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 0 soid 3:ffdb2004:::ROBJ9:head : candidate size 3 info size 7 mismatch' 2024-09-15T21:33:19.057 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2153: TEST_corrupt_scrub_replicated: err_strings[33]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 0 soid 3:ffdb2004:::ROBJ9:head : object info inconsistent ' 2024-09-15T21:33:19.057 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2154: TEST_corrupt_scrub_replicated: err_strings[34]='log_channel[(]cluster[)] log [[]ERR[]] : deep-scrub [0-9]*[.]0 3:c0c86b1d:::ROBJ14:head : no '\''_'\'' attr' 2024-09-15T21:33:19.057 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2155: TEST_corrupt_scrub_replicated: err_strings[35]='log_channel[(]cluster[)] log [[]ERR[]] : deep-scrub [0-9]*[.]0 3:5c7b2c47:::ROBJ16:head : can'\''t decode '\''snapset'\'' attr .* v=3 cannot decode .* Malformed input' 2024-09-15T21:33:19.058 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2156: TEST_corrupt_scrub_replicated: err_strings[36]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 deep-scrub : stat mismatch, got 19/19 objects, 0/0 clones, 18/19 dirty, 18/19 omap, 0/0 pinned, 0/0 hit_set_archive, 0/0 whiteouts, 1049715/1049716 bytes, 0/0 manifest objects, 0/0 hit_set_archive bytes.' 2024-09-15T21:33:19.058 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2157: TEST_corrupt_scrub_replicated: err_strings[37]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 deep-scrub 1 missing, 11 inconsistent objects' 2024-09-15T21:33:19.058 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2158: TEST_corrupt_scrub_replicated: err_strings[38]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 deep-scrub 35 errors' 2024-09-15T21:33:19.058 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2160: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-09-15T21:33:19.058 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2162: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:30259878:::ROBJ15:head : candidate had a missing info key' td/osd-scrub-repair/osd.1.log 2024-09-15T21:33:19.059 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2160: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-09-15T21:33:19.059 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2162: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 0 soid 3:33aca486:::ROBJ18:head : data_digest 0xbd89c912 != data_digest 0x2ddbf8f5 from auth oi 3:33aca486:::ROBJ18:head[(][0-9]*'\''[0-9]* osd.1.0:[0-9]* dirty|omap|data_digest|omap_digest s 7 uv 54 dd 2ddbf8f5 od ddc3680f alloc_hint [[]0 0 255[]][)], object info inconsistent ' td/osd-scrub-repair/osd.1.log 2024-09-15T21:33:19.069 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2160: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-09-15T21:33:19.069 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2162: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:33aca486:::ROBJ18:head : data_digest 0xbd89c912 != data_digest 0x2ddbf8f5 from auth oi 3:33aca486:::ROBJ18:head[(][0-9]*'\''[0-9]* osd.1.0:[0-9]* dirty|omap|data_digest|omap_digest s 7 uv 54 dd 2ddbf8f5 od ddc3680f alloc_hint [[]0 0 255[]][)]' td/osd-scrub-repair/osd.1.log 2024-09-15T21:33:19.079 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2160: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-09-15T21:33:19.079 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2162: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid 3:33aca486:::ROBJ18:head : failed to pick suitable auth object' td/osd-scrub-repair/osd.1.log 2024-09-15T21:33:19.089 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2160: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-09-15T21:33:19.089 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2162: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:5c7b2c47:::ROBJ16:head : candidate had a corrupt snapset' td/osd-scrub-repair/osd.1.log 2024-09-15T21:33:19.099 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2160: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-09-15T21:33:19.099 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2162: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 0 soid 3:5c7b2c47:::ROBJ16:head : candidate had a missing snapset key' td/osd-scrub-repair/osd.1.log 2024-09-15T21:33:19.108 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2160: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-09-15T21:33:19.108 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2162: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid 3:5c7b2c47:::ROBJ16:head : failed to pick suitable object info' td/osd-scrub-repair/osd.1.log 2024-09-15T21:33:19.117 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2160: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-09-15T21:33:19.118 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2162: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid 3:86586531:::ROBJ8:head : attr value mismatch '\''_key1-ROBJ8'\'', attr name mismatch '\''_key3-ROBJ8'\'', attr name mismatch '\''_key2-ROBJ8'\''' td/osd-scrub-repair/osd.1.log 2024-09-15T21:33:19.125 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2160: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-09-15T21:33:19.126 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2162: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:87abbf36:::ROBJ11:head : candidate had a read error' td/osd-scrub-repair/osd.1.log 2024-09-15T21:33:19.136 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2160: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-09-15T21:33:19.136 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2162: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 0 soid 3:8aa5320e:::ROBJ17:head : data_digest 0x5af0c3ef != data_digest 0x2ddbf8f5 from auth oi 3:8aa5320e:::ROBJ17:head[(][0-9]*'\''[0-9]* osd.1.0:[0-9]* dirty|omap|data_digest|omap_digest s 7 uv 51 dd 2ddbf8f5 od e9572720 alloc_hint [[]0 0 0[]][)]' td/osd-scrub-repair/osd.1.log 2024-09-15T21:33:19.146 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2160: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-09-15T21:33:19.147 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2162: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:8aa5320e:::ROBJ17:head : data_digest 0x5af0c3ef != data_digest 0x2ddbf8f5 from auth oi 3:8aa5320e:::ROBJ17:head[(][0-9]*'\''[0-9]* osd.1.0:[0-9]* dirty|omap|data_digest|omap_digest s 7 uv 51 dd 2ddbf8f5 od e9572720 alloc_hint [[]0 0 0[]][)]' td/osd-scrub-repair/osd.1.log 2024-09-15T21:33:19.156 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2160: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-09-15T21:33:19.156 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2162: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid 3:8aa5320e:::ROBJ17:head : failed to pick suitable auth object' td/osd-scrub-repair/osd.1.log 2024-09-15T21:33:19.166 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2160: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-09-15T21:33:19.166 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2162: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid 3:8b55fa4b:::ROBJ7:head : omap_digest 0xefced57a != omap_digest 0x6a73cc07 from shard 1' td/osd-scrub-repair/osd.1.log 2024-09-15T21:33:19.177 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2160: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-09-15T21:33:19.177 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2162: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:8b55fa4b:::ROBJ7:head : omap_digest 0x6a73cc07 != omap_digest 0xefced57a from auth oi 3:8b55fa4b:::ROBJ7:head[(][0-9]*'\''[0-9]* osd.1.0:[0-9]* dirty|omap|data_digest|omap_digest s 7 uv 21 dd 2ddbf8f5 od efced57a alloc_hint [[]0 0 0[]][)]' td/osd-scrub-repair/osd.1.log 2024-09-15T21:33:19.186 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2160: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-09-15T21:33:19.187 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2162: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 0 soid 3:a53c12e8:::ROBJ6:head : omap_digest 0x689ee887 != omap_digest 0x179c919f from shard 1, omap_digest 0x689ee887 != omap_digest 0x179c919f from auth oi 3:a53c12e8:::ROBJ6:head[(][0-9]*'\''[0-9]* osd.1.0:[0-9]* dirty|omap|data_digest|omap_digest s 7 uv 18 dd 2ddbf8f5 od 179c919f alloc_hint [[]0 0 0[]][)]' td/osd-scrub-repair/osd.1.log 2024-09-15T21:33:19.195 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2160: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-09-15T21:33:19.196 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2162: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 0 soid 3:b1f19cbd:::ROBJ10:head : omap_digest 0xa8dd5adc != omap_digest 0xc2025a24 from auth oi 3:b1f19cbd:::ROBJ10:head[(][0-9]*'\''[0-9]* osd.1.0:[0-9]* dirty|omap|data_digest|omap_digest s 7 uv 30 dd 2ddbf8f5 od c2025a24 alloc_hint [[]0 0 0[]][)]' td/osd-scrub-repair/osd.1.log 2024-09-15T21:33:19.206 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2160: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-09-15T21:33:19.206 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2162: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:b1f19cbd:::ROBJ10:head : omap_digest 0xa8dd5adc != omap_digest 0xc2025a24 from auth oi 3:b1f19cbd:::ROBJ10:head[(][0-9]*'\''[0-9]* osd.1.0:[0-9]* dirty|omap|data_digest|omap_digest s 7 uv 30 dd 2ddbf8f5 od c2025a24 alloc_hint [[]0 0 0[]][)]' td/osd-scrub-repair/osd.1.log 2024-09-15T21:33:19.216 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2160: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-09-15T21:33:19.216 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2162: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid 3:b1f19cbd:::ROBJ10:head : failed to pick suitable auth object' td/osd-scrub-repair/osd.1.log 2024-09-15T21:33:19.226 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2160: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-09-15T21:33:19.226 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2162: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 0 soid 3:bc819597:::ROBJ12:head : candidate had a stat error' td/osd-scrub-repair/osd.1.log 2024-09-15T21:33:19.236 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2160: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-09-15T21:33:19.236 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2162: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:c0c86b1d:::ROBJ14:head : candidate had a missing info key' td/osd-scrub-repair/osd.1.log 2024-09-15T21:33:19.245 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2160: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-09-15T21:33:19.245 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2162: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 0 soid 3:c0c86b1d:::ROBJ14:head : candidate had a corrupt info' td/osd-scrub-repair/osd.1.log 2024-09-15T21:33:19.255 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2160: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-09-15T21:33:19.255 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2162: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid 3:c0c86b1d:::ROBJ14:head : failed to pick suitable object info' td/osd-scrub-repair/osd.1.log 2024-09-15T21:33:19.263 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2160: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-09-15T21:33:19.264 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2162: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:ce3f1d6a:::ROBJ1:head : candidate size 9 info size 7 mismatch' td/osd-scrub-repair/osd.1.log 2024-09-15T21:33:19.272 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2160: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-09-15T21:33:19.272 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2162: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:ce3f1d6a:::ROBJ1:head : data_digest 0x2d4a11c2 != data_digest 0x2ddbf8f5 from shard 0, data_digest 0x2d4a11c2 != data_digest 0x2ddbf8f5 from auth oi 3:ce3f1d6a:::ROBJ1:head[(][0-9]*'\''[0-9]* osd.1.0:[0-9]* dirty|omap|data_digest|omap_digest s 7 uv 3 dd 2ddbf8f5 od f5fba2c6 alloc_hint [[]0 0 0[]][)], size 9 != size 7 from auth oi 3:ce3f1d6a:::ROBJ1:head[(][0-9]*'\''[0-9]* osd.1.0:[0-9]* dirty|omap|data_digest|omap_digest s 7 uv 3 dd 2ddbf8f5 od f5fba2c6 alloc_hint [[]0 0 0[]][)], size 9 != size 7 from shard 0' td/osd-scrub-repair/osd.1.log 2024-09-15T21:33:19.282 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2160: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-09-15T21:33:19.282 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2162: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:d60617f9:::ROBJ13:head : candidate had a read error' td/osd-scrub-repair/osd.1.log 2024-09-15T21:33:19.292 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2160: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-09-15T21:33:19.292 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2162: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 0 soid 3:d60617f9:::ROBJ13:head : candidate had a stat error' td/osd-scrub-repair/osd.1.log 2024-09-15T21:33:19.301 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2160: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-09-15T21:33:19.302 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2162: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid 3:d60617f9:::ROBJ13:head : failed to pick suitable object info' td/osd-scrub-repair/osd.1.log 2024-09-15T21:33:19.312 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2160: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-09-15T21:33:19.312 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2162: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 0 soid 3:e97ce31e:::ROBJ2:head : data_digest 0x578a4830 != data_digest 0x2ddbf8f5 from shard 1, data_digest 0x578a4830 != data_digest 0x2ddbf8f5 from auth oi 3:e97ce31e:::ROBJ2:head[(][0-9]*'\''[0-9]* osd.1.0:[0-9]* dirty|omap|data_digest|omap_digest s 7 uv 6 dd 2ddbf8f5 od f8e11918 alloc_hint [[]0 0 0[]][)]' td/osd-scrub-repair/osd.1.log 2024-09-15T21:33:19.321 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2160: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-09-15T21:33:19.321 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2162: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 3:f2a5b2a4:::ROBJ3:head : missing' td/osd-scrub-repair/osd.1.log 2024-09-15T21:33:19.332 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2160: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-09-15T21:33:19.333 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2162: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 0 soid 3:f4981d31:::ROBJ4:head : omap_digest 0xd7178dfe != omap_digest 0xe2d46ea4 from shard 1, omap_digest 0xd7178dfe != omap_digest 0xe2d46ea4 from auth oi 3:f4981d31:::ROBJ4:head[(][0-9]*'\''[0-9]* osd.1.0:[0-9]* dirty|omap|data_digest|omap_digest s 7 uv 12 dd 2ddbf8f5 od e2d46ea4 alloc_hint [[]0 0 0[]][)]' td/osd-scrub-repair/osd.1.log 2024-09-15T21:33:19.341 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2160: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-09-15T21:33:19.341 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2162: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid 3:f4bfd4d1:::ROBJ5:head : omap_digest 0x1a862a41 != omap_digest 0x6cac8f6 from shard 1' td/osd-scrub-repair/osd.1.log 2024-09-15T21:33:19.352 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2160: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-09-15T21:33:19.352 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2162: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:f4bfd4d1:::ROBJ5:head : omap_digest 0x6cac8f6 != omap_digest 0x1a862a41 from auth oi 3:f4bfd4d1:::ROBJ5:head[(][0-9]*'\''[0-9]* osd.1.0:[0-9]* dirty|omap|data_digest|omap_digest s 7 uv 15 dd 2ddbf8f5 od 1a862a41 alloc_hint [[]0 0 0[]][)]' td/osd-scrub-repair/osd.1.log 2024-09-15T21:33:19.362 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2160: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-09-15T21:33:19.362 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2162: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 0 soid 3:ffdb2004:::ROBJ9:head : candidate size 3 info size 7 mismatch' td/osd-scrub-repair/osd.1.log 2024-09-15T21:33:19.372 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2160: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-09-15T21:33:19.372 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2162: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 0 soid 3:ffdb2004:::ROBJ9:head : object info inconsistent ' td/osd-scrub-repair/osd.1.log 2024-09-15T21:33:19.383 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2160: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-09-15T21:33:19.383 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2162: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : deep-scrub [0-9]*[.]0 3:c0c86b1d:::ROBJ14:head : no '\''_'\'' attr' td/osd-scrub-repair/osd.1.log 2024-09-15T21:33:19.396 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2160: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-09-15T21:33:19.396 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2162: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : deep-scrub [0-9]*[.]0 3:5c7b2c47:::ROBJ16:head : can'\''t decode '\''snapset'\'' attr .* v=3 cannot decode .* Malformed input' td/osd-scrub-repair/osd.1.log 2024-09-15T21:33:19.407 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2160: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-09-15T21:33:19.407 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2162: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 deep-scrub : stat mismatch, got 19/19 objects, 0/0 clones, 18/19 dirty, 18/19 omap, 0/0 pinned, 0/0 hit_set_archive, 0/0 whiteouts, 1049715/1049716 bytes, 0/0 manifest objects, 0/0 hit_set_archive bytes.' td/osd-scrub-repair/osd.1.log 2024-09-15T21:33:19.416 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2160: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-09-15T21:33:19.417 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2162: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 deep-scrub 1 missing, 11 inconsistent objects' td/osd-scrub-repair/osd.1.log 2024-09-15T21:33:19.428 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2160: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-09-15T21:33:19.428 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2162: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 deep-scrub 35 errors' td/osd-scrub-repair/osd.1.log 2024-09-15T21:33:19.441 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2169: TEST_corrupt_scrub_replicated: rados list-inconsistent-pg csr_pool 2024-09-15T21:33:19.472 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2171: TEST_corrupt_scrub_replicated: jq '. | length' td/osd-scrub-repair/json 2024-09-15T21:33:19.474 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2171: TEST_corrupt_scrub_replicated: test 1 = 1 2024-09-15T21:33:19.474 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2173: TEST_corrupt_scrub_replicated: jq -r '.[0]' td/osd-scrub-repair/json 2024-09-15T21:33:19.476 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2173: TEST_corrupt_scrub_replicated: test 3.0 = 3.0 2024-09-15T21:33:19.476 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2175: TEST_corrupt_scrub_replicated: rados list-inconsistent-obj 3.0 2024-09-15T21:33:19.505 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2177: TEST_corrupt_scrub_replicated: jq .epoch td/osd-scrub-repair/json 2024-09-15T21:33:19.507 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2177: TEST_corrupt_scrub_replicated: epoch=203 2024-09-15T21:33:19.508 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2179: TEST_corrupt_scrub_replicated: jq 'def walk(f): 2024-09-15T21:33:19.508 INFO:tasks.workunit.client.0.smithi136.stderr: . as $in 2024-09-15T21:33:19.508 INFO:tasks.workunit.client.0.smithi136.stderr: | if type == "object" then 2024-09-15T21:33:19.508 INFO:tasks.workunit.client.0.smithi136.stderr: reduce keys[] as $key 2024-09-15T21:33:19.508 INFO:tasks.workunit.client.0.smithi136.stderr: ( {}; . + { ($key): ($in[$key] | walk(f)) } ) | f 2024-09-15T21:33:19.508 INFO:tasks.workunit.client.0.smithi136.stderr: elif type == "array" then map( walk(f) ) | f 2024-09-15T21:33:19.508 INFO:tasks.workunit.client.0.smithi136.stderr: else f 2024-09-15T21:33:19.508 INFO:tasks.workunit.client.0.smithi136.stderr: end; 2024-09-15T21:33:19.508 INFO:tasks.workunit.client.0.smithi136.stderr:walk(if type == "object" then del(.mtime) else . end) 2024-09-15T21:33:19.509 INFO:tasks.workunit.client.0.smithi136.stderr:| walk(if type == "object" then del(.local_mtime) else . end) 2024-09-15T21:33:19.509 INFO:tasks.workunit.client.0.smithi136.stderr:| walk(if type == "object" then del(.last_reqid) else . end) 2024-09-15T21:33:19.509 INFO:tasks.workunit.client.0.smithi136.stderr:| walk(if type == "object" then del(.version) else . end) 2024-09-15T21:33:19.509 INFO:tasks.workunit.client.0.smithi136.stderr:| walk(if type == "object" then del(.prior_version) else . end)' 2024-09-15T21:33:19.509 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2179: TEST_corrupt_scrub_replicated: jq .inconsistents 2024-09-15T21:33:19.509 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2179: TEST_corrupt_scrub_replicated: python3 -c 'import json; import sys ; JSON=sys.stdin.read() ; ud = json.loads(JSON) ; print(json.dumps(ud, sort_keys=True, indent=2))' 2024-09-15T21:33:19.532 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3558: TEST_corrupt_scrub_replicated: jq 'def walk(f): 2024-09-15T21:33:19.532 INFO:tasks.workunit.client.0.smithi136.stderr: . as $in 2024-09-15T21:33:19.533 INFO:tasks.workunit.client.0.smithi136.stderr: | if type == "object" then 2024-09-15T21:33:19.533 INFO:tasks.workunit.client.0.smithi136.stderr: reduce keys[] as $key 2024-09-15T21:33:19.533 INFO:tasks.workunit.client.0.smithi136.stderr: ( {}; . + { ($key): ($in[$key] | walk(f)) } ) | f 2024-09-15T21:33:19.533 INFO:tasks.workunit.client.0.smithi136.stderr: elif type == "array" then map( walk(f) ) | f 2024-09-15T21:33:19.533 INFO:tasks.workunit.client.0.smithi136.stderr: else f 2024-09-15T21:33:19.533 INFO:tasks.workunit.client.0.smithi136.stderr: end; 2024-09-15T21:33:19.533 INFO:tasks.workunit.client.0.smithi136.stderr:walk(if type == "object" then del(.mtime) else . end) 2024-09-15T21:33:19.533 INFO:tasks.workunit.client.0.smithi136.stderr:| walk(if type == "object" then del(.local_mtime) else . end) 2024-09-15T21:33:19.533 INFO:tasks.workunit.client.0.smithi136.stderr:| walk(if type == "object" then del(.last_reqid) else . end) 2024-09-15T21:33:19.534 INFO:tasks.workunit.client.0.smithi136.stderr:| walk(if type == "object" then del(.version) else . end) 2024-09-15T21:33:19.534 INFO:tasks.workunit.client.0.smithi136.stderr:| walk(if type == "object" then del(.prior_version) else . end)' td/osd-scrub-repair/json 2024-09-15T21:33:19.534 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3558: TEST_corrupt_scrub_replicated: jq .inconsistents 2024-09-15T21:33:19.534 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3558: TEST_corrupt_scrub_replicated: python3 -c 'import json; import sys ; JSON=sys.stdin.read() ; ud = json.loads(JSON) ; print(json.dumps(ud, sort_keys=True, indent=2))' 2024-09-15T21:33:19.555 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3559: TEST_corrupt_scrub_replicated: multidiff td/osd-scrub-repair/checkcsjson td/osd-scrub-repair/csjson 2024-09-15T21:33:19.555 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2434: multidiff: diff td/osd-scrub-repair/checkcsjson td/osd-scrub-repair/csjson 2024-09-15T21:33:19.556 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3560: TEST_corrupt_scrub_replicated: test no = yes 2024-09-15T21:33:19.556 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3565: TEST_corrupt_scrub_replicated: test '' = yes 2024-09-15T21:33:19.556 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3570: TEST_corrupt_scrub_replicated: repair 3.0 2024-09-15T21:33:19.557 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1846: repair: local pgid=3.0 2024-09-15T21:33:19.557 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1847: repair: get_last_scrub_stamp 3.0 2024-09-15T21:33:19.557 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=3.0 2024-09-15T21:33:19.557 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-15T21:33:19.558 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-15T21:33:19.558 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_scrub_stamp' 2024-09-15T21:33:19.828 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1847: repair: local last_scrub=2024-09-15T21:33:17.249222+0000 2024-09-15T21:33:19.828 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1848: repair: ceph pg repair 3.0 2024-09-15T21:33:20.090 INFO:tasks.workunit.client.0.smithi136.stderr:instructing pg 3.0 on osd.1 to repair 2024-09-15T21:33:20.102 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1849: repair: wait_for_scrub 3.0 2024-09-15T21:33:17.249222+0000 2024-09-15T21:33:20.102 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: local pgid=3.0 2024-09-15T21:33:20.102 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2018: wait_for_scrub: local last_scrub=2024-09-15T21:33:17.249222+0000 2024-09-15T21:33:20.102 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2019: wait_for_scrub: local sname=last_scrub_stamp 2024-09-15T21:33:20.102 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i=0 )) 2024-09-15T21:33:20.102 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-15T21:33:20.102 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 3.0 last_scrub_stamp 2024-09-15T21:33:20.103 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=3.0 2024-09-15T21:33:20.103 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-15T21:33:20.103 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-15T21:33:20.103 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_scrub_stamp' 2024-09-15T21:33:20.378 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-15T21:33:17.249222+0000 '>' 2024-09-15T21:33:17.249222+0000 2024-09-15T21:33:20.378 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-15T21:33:21.379 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-15T21:33:21.379 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-15T21:33:21.380 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 3.0 last_scrub_stamp 2024-09-15T21:33:21.380 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=3.0 2024-09-15T21:33:21.380 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-15T21:33:21.380 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-15T21:33:21.380 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_scrub_stamp' 2024-09-15T21:33:21.662 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-15T21:33:17.249222+0000 '>' 2024-09-15T21:33:17.249222+0000 2024-09-15T21:33:21.662 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-15T21:33:22.663 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-15T21:33:22.663 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-15T21:33:22.664 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 3.0 last_scrub_stamp 2024-09-15T21:33:22.664 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=3.0 2024-09-15T21:33:22.664 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-15T21:33:22.664 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-15T21:33:22.665 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_scrub_stamp' 2024-09-15T21:33:22.948 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-15T21:33:17.249222+0000 '>' 2024-09-15T21:33:17.249222+0000 2024-09-15T21:33:22.948 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-15T21:33:23.949 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-15T21:33:23.949 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-15T21:33:23.950 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 3.0 last_scrub_stamp 2024-09-15T21:33:23.950 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=3.0 2024-09-15T21:33:23.950 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-15T21:33:23.950 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-15T21:33:23.951 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_scrub_stamp' 2024-09-15T21:33:24.237 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-15T21:33:17.249222+0000 '>' 2024-09-15T21:33:17.249222+0000 2024-09-15T21:33:24.237 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-15T21:33:25.238 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-15T21:33:25.238 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-15T21:33:25.239 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 3.0 last_scrub_stamp 2024-09-15T21:33:25.239 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=3.0 2024-09-15T21:33:25.239 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-15T21:33:25.239 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-15T21:33:25.239 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_scrub_stamp' 2024-09-15T21:33:25.516 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-15T21:33:22.065978+0000 '>' 2024-09-15T21:33:17.249222+0000 2024-09-15T21:33:25.516 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2023: wait_for_scrub: return 0 2024-09-15T21:33:25.516 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1850: repair: sleep 2 2024-09-15T21:33:27.517 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3571: TEST_corrupt_scrub_replicated: wait_for_clean 2024-09-15T21:33:27.518 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-15T21:33:27.518 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-15T21:33:27.518 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-15T21:33:27.518 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-15T21:33:27.519 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-15T21:33:27.519 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-15T21:33:27.519 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-15T21:33:27.519 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-15T21:33:27.519 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-15T21:33:27.624 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-15T21:33:27.624 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-15T21:33:27.624 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-15T21:33:27.624 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-15T21:33:27.625 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-15T21:33:27.625 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-15T21:33:27.901 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-15T21:33:27.901 INFO:tasks.workunit.client.0.smithi136.stderr:1' 2024-09-15T21:33:27.901 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-15T21:33:27.901 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T21:33:27.901 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-15T21:33:28.003 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=846108557321 2024-09-15T21:33:28.003 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 846108557321 2024-09-15T21:33:28.003 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-846108557321' 2024-09-15T21:33:28.003 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T21:33:28.004 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-15T21:33:28.108 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=871878361095 2024-09-15T21:33:28.108 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 871878361095 2024-09-15T21:33:28.109 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-846108557321 1-871878361095' 2024-09-15T21:33:28.109 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T21:33:28.109 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-846108557321 2024-09-15T21:33:28.109 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T21:33:28.110 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-15T21:33:28.111 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-846108557321 2024-09-15T21:33:28.111 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T21:33:28.112 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=846108557321 2024-09-15T21:33:28.112 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 846108557321' 2024-09-15T21:33:28.112 INFO:tasks.workunit.client.0.smithi136.stdout:waiting osd.0 seq 846108557321 2024-09-15T21:33:28.112 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T21:33:28.388 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 846108557320 -lt 846108557321 2024-09-15T21:33:28.389 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-15T21:33:29.390 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-15T21:33:29.390 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T21:33:29.674 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 846108557322 -lt 846108557321 2024-09-15T21:33:29.675 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T21:33:29.675 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-871878361095 2024-09-15T21:33:29.675 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T21:33:29.676 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-15T21:33:29.677 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-871878361095 2024-09-15T21:33:29.677 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T21:33:29.678 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=871878361095 2024-09-15T21:33:29.678 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 871878361095' 2024-09-15T21:33:29.678 INFO:tasks.workunit.client.0.smithi136.stdout:waiting osd.1 seq 871878361095 2024-09-15T21:33:29.678 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-15T21:33:29.955 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 871878361095 -lt 871878361095 2024-09-15T21:33:29.956 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-15T21:33:29.956 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T21:33:29.956 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T21:33:30.299 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 6 == 0 2024-09-15T21:33:30.299 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-15T21:33:30.300 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-15T21:33:30.300 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-15T21:33:30.300 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T21:33:30.300 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T21:33:30.300 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T21:33:30.300 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T21:33:30.576 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=6 2024-09-15T21:33:30.576 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-15T21:33:30.577 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T21:33:30.577 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T21:33:30.918 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 = 6 2024-09-15T21:33:30.918 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-15T21:33:30.918 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-15T21:33:30.918 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3574: TEST_corrupt_scrub_replicated: timeout 30 rados -p csr_pool get ROBJ17 td/osd-scrub-repair/robj17.out 2024-09-15T21:33:30.949 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3575: TEST_corrupt_scrub_replicated: timeout 30 rados -p csr_pool get ROBJ18 td/osd-scrub-repair/robj18.out 2024-09-15T21:33:30.985 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3577: TEST_corrupt_scrub_replicated: diff -q td/osd-scrub-repair/new.ROBJ17 td/osd-scrub-repair/robj17.out 2024-09-15T21:33:30.986 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3578: TEST_corrupt_scrub_replicated: rm -f td/osd-scrub-repair/new.ROBJ17 td/osd-scrub-repair/robj17.out 2024-09-15T21:33:30.987 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3579: TEST_corrupt_scrub_replicated: diff -q td/osd-scrub-repair/new.ROBJ18 td/osd-scrub-repair/robj18.out 2024-09-15T21:33:30.988 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3580: TEST_corrupt_scrub_replicated: rm -f td/osd-scrub-repair/new.ROBJ18 td/osd-scrub-repair/robj18.out 2024-09-15T21:33:30.989 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3582: TEST_corrupt_scrub_replicated: '[' 0 '!=' 0 ']' 2024-09-15T21:33:30.989 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3588: TEST_corrupt_scrub_replicated: ceph osd pool rm csr_pool csr_pool --yes-i-really-really-mean-it 2024-09-15T21:33:31.465 INFO:tasks.workunit.client.0.smithi136.stderr:pool 'csr_pool' removed 2024-09-15T21:33:31.481 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:66: run: teardown td/osd-scrub-repair 2024-09-15T21:33:31.481 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/osd-scrub-repair 2024-09-15T21:33:31.481 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-15T21:33:31.481 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/osd-scrub-repair KILL 2024-09-15T21:33:31.481 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-15T21:33:31.481 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-15T21:33:31.481 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-15T21:33:31.481 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-15T21:33:31.481 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-15T21:33:31.584 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-15T21:33:31.585 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-15T21:33:31.586 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-15T21:33:31.586 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-15T21:33:31.587 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-15T21:33:31.587 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-15T21:33:31.588 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-15T21:33:31.589 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T21:33:31.589 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-15T21:33:31.589 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-15T21:33:31.589 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T21:33:31.590 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-15T21:33:31.591 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-15T21:33:31.591 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/osd-scrub-repair 2024-09-15T21:33:31.628 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-15T21:33:31.628 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T21:33:31.628 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52602 2024-09-15T21:33:31.628 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.52602 2024-09-15T21:33:31.630 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-15T21:33:31.630 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-15T21:33:31.630 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:63: run: for func in $funcs 2024-09-15T21:33:31.630 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:64: run: setup td/osd-scrub-repair 2024-09-15T21:33:31.630 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/osd-scrub-repair 2024-09-15T21:33:31.630 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/osd-scrub-repair 2024-09-15T21:33:31.630 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/osd-scrub-repair 2024-09-15T21:33:31.630 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-15T21:33:31.630 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/osd-scrub-repair KILL 2024-09-15T21:33:31.630 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-15T21:33:31.631 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-15T21:33:31.631 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-15T21:33:31.631 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-15T21:33:31.631 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-15T21:33:31.631 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-15T21:33:31.632 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-15T21:33:31.633 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-15T21:33:31.633 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-15T21:33:31.634 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-15T21:33:31.634 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-15T21:33:31.635 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-15T21:33:31.636 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T21:33:31.636 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-15T21:33:31.636 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-15T21:33:31.636 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T21:33:31.637 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-15T21:33:31.638 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-15T21:33:31.638 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/osd-scrub-repair 2024-09-15T21:33:31.639 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-15T21:33:31.639 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T21:33:31.639 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52602 2024-09-15T21:33:31.639 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.52602 2024-09-15T21:33:31.640 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-15T21:33:31.640 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-15T21:33:31.640 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/osd-scrub-repair 2024-09-15T21:33:31.642 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2024-09-15T21:33:31.642 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T21:33:31.642 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52602 2024-09-15T21:33:31.642 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.52602 2024-09-15T21:33:31.644 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2024-09-15T21:33:31.644 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 4096 -le 1024 ']' 2024-09-15T21:33:31.645 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2024-09-15T21:33:31.645 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/osd-scrub-repair 1' TERM HUP INT 2024-09-15T21:33:31.645 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:65: run: TEST_corrupt_snapset_scrub_rep td/osd-scrub-repair 2024-09-15T21:33:31.645 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5934: TEST_corrupt_snapset_scrub_rep: local dir=td/osd-scrub-repair 2024-09-15T21:33:31.645 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5935: TEST_corrupt_snapset_scrub_rep: local poolname=csr_pool 2024-09-15T21:33:31.645 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5936: TEST_corrupt_snapset_scrub_rep: local total_objs=2 2024-09-15T21:33:31.645 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5938: TEST_corrupt_snapset_scrub_rep: run_mon td/osd-scrub-repair a --osd_pool_default_size=2 2024-09-15T21:33:31.645 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/osd-scrub-repair 2024-09-15T21:33:31.645 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2024-09-15T21:33:31.645 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2024-09-15T21:33:31.645 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2024-09-15T21:33:31.645 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/osd-scrub-repair/a 2024-09-15T21:33:31.645 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-scrub-repair/a --run-dir=td/osd-scrub-repair --osd_pool_default_size=2 2024-09-15T21:33:32.081 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2024-09-15T21:33:32.081 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-15T21:33:32.081 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-15T21:33:32.082 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-15T21:33:32.082 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T21:33:32.082 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52602 2024-09-15T21:33:32.082 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.52602/$cluster-$name.asok' 2024-09-15T21:33:32.082 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-scrub-repair/a '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.52602/$cluster-$name.asok' --mon-cluster-log-file=td/osd-scrub-repair/log --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=2 2024-09-15T21:33:32.307 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2024-09-15T21:33:32.308 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2024-09-15T21:33:32.308 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-15T21:33:32.308 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-15T21:33:32.308 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2024-09-15T21:33:32.309 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2024-09-15T21:33:32.309 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-15T21:33:32.309 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-15T21:33:32.309 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-15T21:33:32.310 INFO:tasks.workunit.client.0.smithi136.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-15T21:33:32.310 INFO:tasks.workunit.client.0.smithi136.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T21:33:32.310 INFO:tasks.workunit.client.0.smithi136.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52602 2024-09-15T21:33:32.310 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.52602/ceph-mon.a.asok 2024-09-15T21:33:32.310 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-15T21:33:32.310 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.52602/ceph-mon.a.asok config get fsid 2024-09-15T21:33:32.392 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2024-09-15T21:33:32.392 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-15T21:33:32.392 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-15T21:33:32.392 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2024-09-15T21:33:32.392 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2024-09-15T21:33:32.392 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-15T21:33:32.392 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-15T21:33:32.392 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-15T21:33:32.393 INFO:tasks.workunit.client.0.smithi136.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-15T21:33:32.393 INFO:tasks.workunit.client.0.smithi136.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T21:33:32.393 INFO:tasks.workunit.client.0.smithi136.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52602 2024-09-15T21:33:32.393 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.52602/ceph-mon.a.asok 2024-09-15T21:33:32.394 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-15T21:33:32.394 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.52602/ceph-mon.a.asok config get mon_host 2024-09-15T21:33:32.472 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5939: TEST_corrupt_snapset_scrub_rep: run_mgr td/osd-scrub-repair x 2024-09-15T21:33:32.472 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: run_mgr: local dir=td/osd-scrub-repair 2024-09-15T21:33:32.472 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: run_mgr: shift 2024-09-15T21:33:32.472 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: run_mgr: local id=x 2024-09-15T21:33:32.472 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: run_mgr: shift 2024-09-15T21:33:32.472 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:548: run_mgr: local data=td/osd-scrub-repair/x 2024-09-15T21:33:32.472 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:550: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-15T21:33:33.283 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: get_asok_path 2024-09-15T21:33:33.283 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-15T21:33:33.283 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-15T21:33:33.283 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-15T21:33:33.283 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T21:33:33.284 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52602 2024-09-15T21:33:33.284 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.52602/$cluster-$name.asok' 2024-09-15T21:33:33.284 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-15T21:33:33.285 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-scrub-repair/x '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.52602/$cluster-$name.asok' --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-15T21:33:33.315 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5940: TEST_corrupt_snapset_scrub_rep: run_osd td/osd-scrub-repair 0 2024-09-15T21:33:33.315 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-scrub-repair 2024-09-15T21:33:33.316 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-15T21:33:33.316 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=0 2024-09-15T21:33:33.316 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-15T21:33:33.316 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-scrub-repair/0 2024-09-15T21:33:33.316 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=ce4892c7-6c6d-4b4f-8e6f-07a3e11133e5 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-15T21:33:33.316 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T21:33:33.316 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T21:33:33.316 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T21:33:33.316 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-09-15T21:33:33.316 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-09-15T21:33:33.316 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-15T21:33:33.316 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-15T21:33:33.317 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-15T21:33:33.317 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-15T21:33:33.317 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-15T21:33:33.317 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-15T21:33:33.317 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-15T21:33:33.317 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T21:33:33.317 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52602 2024-09-15T21:33:33.317 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.52602/$cluster-$name.asok' 2024-09-15T21:33:33.317 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.52602/$cluster-$name.asok' 2024-09-15T21:33:33.317 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-15T21:33:33.317 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-15T21:33:33.317 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-15T21:33:33.317 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-15T21:33:33.318 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-15T21:33:33.318 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T21:33:33.318 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T21:33:33.318 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T21:33:33.318 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T21:33:33.318 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-15T21:33:33.318 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-15T21:33:33.318 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-scrub-repair/0 2024-09-15T21:33:33.318 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-15T21:33:33.319 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=69b3cfa1-62b4-4f5d-aa79-48f162232a77 2024-09-15T21:33:33.319 INFO:tasks.workunit.client.0.smithi136.stdout:add osd0 69b3cfa1-62b4-4f5d-aa79-48f162232a77 2024-09-15T21:33:33.319 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd0 69b3cfa1-62b4-4f5d-aa79-48f162232a77' 2024-09-15T21:33:33.319 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-15T21:33:33.335 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQCtUudm//3bExAAZzYkLTo/qyA5dqHnoUEK1Q== 2024-09-15T21:33:33.336 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQCtUudm//3bExAAZzYkLTo/qyA5dqHnoUEK1Q=="}' 2024-09-15T21:33:33.336 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 69b3cfa1-62b4-4f5d-aa79-48f162232a77 -i td/osd-scrub-repair/0/new.json 2024-09-15T21:33:33.539 INFO:tasks.workunit.client.0.smithi136.stdout:0 2024-09-15T21:33:33.549 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-scrub-repair/0/new.json 2024-09-15T21:33:33.550 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 0 --fsid=ce4892c7-6c6d-4b4f-8e6f-07a3e11133e5 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.52602/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCtUudm//3bExAAZzYkLTo/qyA5dqHnoUEK1Q== --osd-uuid 69b3cfa1-62b4-4f5d-aa79-48f162232a77 2024-09-15T21:33:33.579 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:33:33.576+0000 7fd45d81c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:33:33.590 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:33:33.586+0000 7fd45d81c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:33:33.591 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:33:33.588+0000 7fd45d81c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:33:33.591 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:33:33.588+0000 7fd45d81c640 -1 bdev(0x559d92e4bc00 td/osd-scrub-repair/0/block) open stat got: (1) Operation not permitted 2024-09-15T21:33:33.591 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:33:33.588+0000 7fd45d81c640 -1 bluestore(td/osd-scrub-repair/0) _read_fsid unparsable uuid 2024-09-15T21:33:39.026 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-scrub-repair/0/keyring 2024-09-15T21:33:39.026 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-15T21:33:39.027 INFO:tasks.workunit.client.0.smithi136.stdout:adding osd0 key to auth repository 2024-09-15T21:33:39.028 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd0 key to auth repository 2024-09-15T21:33:39.028 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-scrub-repair/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-15T21:33:39.389 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.0 2024-09-15T21:33:39.390 INFO:tasks.workunit.client.0.smithi136.stdout:start osd.0 2024-09-15T21:33:39.390 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 0 --fsid=ce4892c7-6c6d-4b4f-8e6f-07a3e11133e5 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.52602/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T21:33:39.390 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-15T21:33:39.390 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-15T21:33:39.390 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-15T21:33:39.417 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:33:39.414+0000 7fa8f4d89640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:33:39.418 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:33:39.416+0000 7fa8f4d89640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:33:39.420 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:33:39.417+0000 7fa8f4d89640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:33:39.669 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 0 2024-09-15T21:33:39.669 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-15T21:33:39.669 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2024-09-15T21:33:39.669 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-15T21:33:39.669 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-15T21:33:39.669 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:33:39.669 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-15T21:33:39.670 INFO:tasks.workunit.client.0.smithi136.stdout:0 2024-09-15T21:33:39.670 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:33:39.670 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:33:39.944 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:33:40.945 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:33:40.945 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:33:40.945 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-15T21:33:40.945 INFO:tasks.workunit.client.0.smithi136.stdout:1 2024-09-15T21:33:40.946 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:33:40.946 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:33:41.028 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:33:41.026+0000 7fa8f4d89640 -1 Falling back to public interface 2024-09-15T21:33:41.230 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:33:42.231 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:33:42.231 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:33:42.231 INFO:tasks.workunit.client.0.smithi136.stdout:2 2024-09-15T21:33:42.232 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-15T21:33:42.232 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:33:42.232 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:33:42.525 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:33:43.526 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:33:43.526 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:33:43.526 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-15T21:33:43.526 INFO:tasks.workunit.client.0.smithi136.stdout:3 2024-09-15T21:33:43.527 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:33:43.527 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:33:43.684 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:33:43.682+0000 7fa8f4d89640 -1 osd.0 0 log_to_monitors true 2024-09-15T21:33:43.851 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:33:44.853 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:33:44.853 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:33:44.854 INFO:tasks.workunit.client.0.smithi136.stdout:4 2024-09-15T21:33:44.854 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-15T21:33:44.855 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:33:44.855 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:33:45.141 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:33:46.143 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:33:46.143 INFO:tasks.workunit.client.0.smithi136.stdout:5 2024-09-15T21:33:46.143 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:33:46.144 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-15T21:33:46.144 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:33:46.144 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:33:46.437 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:33:47.438 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:33:47.439 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:33:47.439 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-15T21:33:47.439 INFO:tasks.workunit.client.0.smithi136.stdout:6 2024-09-15T21:33:47.439 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:33:47.439 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:33:47.726 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:33:48.728 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:33:48.729 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:33:48.729 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-15T21:33:48.729 INFO:tasks.workunit.client.0.smithi136.stdout:7 2024-09-15T21:33:48.730 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:33:48.730 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:33:49.018 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:33:50.021 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:33:50.021 INFO:tasks.workunit.client.0.smithi136.stdout:8 2024-09-15T21:33:50.022 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:33:50.022 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-15T21:33:50.022 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:33:50.022 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:33:50.310 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:33:51.311 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:33:51.311 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:33:51.312 INFO:tasks.workunit.client.0.smithi136.stdout:9 2024-09-15T21:33:51.313 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-15T21:33:51.313 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:33:51.313 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:33:51.664 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:33:52.665 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:33:52.665 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:33:52.666 INFO:tasks.workunit.client.0.smithi136.stdout:10 2024-09-15T21:33:52.666 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-15T21:33:52.666 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:33:52.666 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:33:52.955 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:33:53.956 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:33:53.956 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:33:53.957 INFO:tasks.workunit.client.0.smithi136.stdout:11 2024-09-15T21:33:53.957 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-15T21:33:53.957 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:33:53.957 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:33:54.247 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:33:55.249 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:33:55.249 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:33:55.249 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-15T21:33:55.249 INFO:tasks.workunit.client.0.smithi136.stdout:12 2024-09-15T21:33:55.250 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:33:55.250 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:33:55.902 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:33:56.904 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:33:56.904 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:33:56.904 INFO:tasks.workunit.client.0.smithi136.stdout:13 2024-09-15T21:33:56.905 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-15T21:33:56.905 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:33:56.905 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:33:57.837 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:33:58.837 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:33:58.838 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:33:58.838 INFO:tasks.workunit.client.0.smithi136.stdout:14 2024-09-15T21:33:58.838 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-15T21:33:58.838 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:33:58.839 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:33:59.215 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:34:00.216 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:34:00.216 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:34:00.217 INFO:tasks.workunit.client.0.smithi136.stdout:15 2024-09-15T21:34:00.217 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-15T21:34:00.217 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:34:00.217 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:34:00.721 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:34:01.723 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:34:01.723 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:34:01.723 INFO:tasks.workunit.client.0.smithi136.stdout:16 2024-09-15T21:34:01.724 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-15T21:34:01.724 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:34:01.724 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:34:02.543 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:34:03.544 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:34:03.544 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:34:03.545 INFO:tasks.workunit.client.0.smithi136.stdout:17 2024-09-15T21:34:03.545 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-15T21:34:03.545 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:34:03.546 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:34:04.395 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:34:05.396 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:34:05.396 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:34:05.396 INFO:tasks.workunit.client.0.smithi136.stdout:18 2024-09-15T21:34:05.397 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 18 2024-09-15T21:34:05.397 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:34:05.397 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:34:05.877 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:34:06.878 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:34:06.878 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:34:06.879 INFO:tasks.workunit.client.0.smithi136.stdout:19 2024-09-15T21:34:06.879 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 19 2024-09-15T21:34:06.879 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:34:06.879 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:34:07.183 INFO:tasks.workunit.client.0.smithi136.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/1471914542,v1:127.0.0.1:6803/1471914542] [v2:127.0.0.1:6804/1471914542,v1:127.0.0.1:6805/1471914542] exists,up 69b3cfa1-62b4-4f5d-aa79-48f162232a77 2024-09-15T21:34:07.187 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-15T21:34:07.187 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-15T21:34:07.187 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-15T21:34:07.188 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5941: TEST_corrupt_snapset_scrub_rep: run_osd td/osd-scrub-repair 1 2024-09-15T21:34:07.188 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-scrub-repair 2024-09-15T21:34:07.188 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-15T21:34:07.188 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=1 2024-09-15T21:34:07.188 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-15T21:34:07.188 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-scrub-repair/1 2024-09-15T21:34:07.188 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=ce4892c7-6c6d-4b4f-8e6f-07a3e11133e5 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-15T21:34:07.188 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T21:34:07.188 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T21:34:07.188 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T21:34:07.188 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-09-15T21:34:07.188 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-09-15T21:34:07.188 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-15T21:34:07.189 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-15T21:34:07.189 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-15T21:34:07.189 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-15T21:34:07.189 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-15T21:34:07.189 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-15T21:34:07.189 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-15T21:34:07.189 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T21:34:07.189 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52602 2024-09-15T21:34:07.189 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.52602/$cluster-$name.asok' 2024-09-15T21:34:07.189 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.52602/$cluster-$name.asok' 2024-09-15T21:34:07.189 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-15T21:34:07.189 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-15T21:34:07.189 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-15T21:34:07.190 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-15T21:34:07.190 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-15T21:34:07.190 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T21:34:07.190 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T21:34:07.190 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T21:34:07.190 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T21:34:07.190 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-15T21:34:07.190 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-15T21:34:07.190 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-scrub-repair/1 2024-09-15T21:34:07.190 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-15T21:34:07.190 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=b7c965e4-5a20-45ae-b9d3-a4f700ce5474 2024-09-15T21:34:07.191 INFO:tasks.workunit.client.0.smithi136.stdout:add osd1 b7c965e4-5a20-45ae-b9d3-a4f700ce5474 2024-09-15T21:34:07.191 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd1 b7c965e4-5a20-45ae-b9d3-a4f700ce5474' 2024-09-15T21:34:07.191 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-15T21:34:07.207 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQDPUudmTQ8wDBAAeIvbY0H8bBN9dVgPiW+wjg== 2024-09-15T21:34:07.207 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQDPUudmTQ8wDBAAeIvbY0H8bBN9dVgPiW+wjg=="}' 2024-09-15T21:34:07.207 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new b7c965e4-5a20-45ae-b9d3-a4f700ce5474 -i td/osd-scrub-repair/1/new.json 2024-09-15T21:34:07.574 INFO:tasks.workunit.client.0.smithi136.stdout:1 2024-09-15T21:34:07.586 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-scrub-repair/1/new.json 2024-09-15T21:34:07.587 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 1 --fsid=ce4892c7-6c6d-4b4f-8e6f-07a3e11133e5 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.52602/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDPUudmTQ8wDBAAeIvbY0H8bBN9dVgPiW+wjg== --osd-uuid b7c965e4-5a20-45ae-b9d3-a4f700ce5474 2024-09-15T21:34:07.615 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:34:07.613+0000 7fed22887640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:34:07.659 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:34:07.657+0000 7fed22887640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:34:07.660 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:34:07.658+0000 7fed22887640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:34:07.660 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:34:07.658+0000 7fed22887640 -1 bdev(0x564459d5a800 td/osd-scrub-repair/1/block) open stat got: (1) Operation not permitted 2024-09-15T21:34:07.660 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:34:07.658+0000 7fed22887640 -1 bluestore(td/osd-scrub-repair/1) _read_fsid unparsable uuid 2024-09-15T21:34:13.351 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-scrub-repair/1/keyring 2024-09-15T21:34:13.352 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-15T21:34:13.353 INFO:tasks.workunit.client.0.smithi136.stdout:adding osd1 key to auth repository 2024-09-15T21:34:13.353 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd1 key to auth repository 2024-09-15T21:34:13.353 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-scrub-repair/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-15T21:34:13.751 INFO:tasks.workunit.client.0.smithi136.stdout:start osd.1 2024-09-15T21:34:13.752 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.1 2024-09-15T21:34:13.752 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 1 --fsid=ce4892c7-6c6d-4b4f-8e6f-07a3e11133e5 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.52602/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T21:34:13.752 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-15T21:34:13.752 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-15T21:34:13.752 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-15T21:34:13.778 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:34:13.776+0000 7fc6f920d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:34:13.780 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:34:13.778+0000 7fc6f920d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:34:13.781 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:34:13.779+0000 7fc6f920d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:34:14.031 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 1 2024-09-15T21:34:14.031 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-15T21:34:14.031 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2024-09-15T21:34:14.031 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-15T21:34:14.031 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-15T21:34:14.031 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:34:14.031 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-15T21:34:14.031 INFO:tasks.workunit.client.0.smithi136.stdout:0 2024-09-15T21:34:14.032 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:34:14.032 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T21:34:14.307 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:34:15.143 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:34:15.141+0000 7fc6f920d640 -1 Falling back to public interface 2024-09-15T21:34:15.308 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:34:15.308 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:34:15.309 INFO:tasks.workunit.client.0.smithi136.stdout:1 2024-09-15T21:34:15.309 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-15T21:34:15.309 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:34:15.309 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T21:34:15.617 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:34:16.618 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:34:16.618 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:34:16.619 INFO:tasks.workunit.client.0.smithi136.stdout:2 2024-09-15T21:34:16.619 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-15T21:34:16.619 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:34:16.619 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T21:34:16.905 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:34:17.907 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:34:17.907 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:34:17.907 INFO:tasks.workunit.client.0.smithi136.stdout:3 2024-09-15T21:34:17.907 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-15T21:34:17.908 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:34:17.908 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T21:34:18.068 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:34:18.066+0000 7fc6f920d640 -1 osd.1 0 log_to_monitors true 2024-09-15T21:34:18.275 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:34:19.276 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:34:19.276 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:34:19.276 INFO:tasks.workunit.client.0.smithi136.stdout:4 2024-09-15T21:34:19.277 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-15T21:34:19.277 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:34:19.277 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T21:34:19.591 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:34:20.592 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:34:20.593 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:34:20.593 INFO:tasks.workunit.client.0.smithi136.stdout:5 2024-09-15T21:34:20.593 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-15T21:34:20.593 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:34:20.594 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T21:34:20.973 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:34:21.974 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:34:21.974 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:34:21.974 INFO:tasks.workunit.client.0.smithi136.stdout:6 2024-09-15T21:34:21.975 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-15T21:34:21.975 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:34:21.975 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T21:34:22.266 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:34:23.267 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:34:23.268 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:34:23.268 INFO:tasks.workunit.client.0.smithi136.stdout:7 2024-09-15T21:34:23.269 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-15T21:34:23.269 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:34:23.269 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T21:34:23.623 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:34:24.625 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:34:24.625 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:34:24.625 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-15T21:34:24.625 INFO:tasks.workunit.client.0.smithi136.stdout:8 2024-09-15T21:34:24.626 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:34:24.626 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T21:34:24.971 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:34:25.972 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:34:25.972 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:34:25.973 INFO:tasks.workunit.client.0.smithi136.stdout:9 2024-09-15T21:34:25.973 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-15T21:34:25.973 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:34:25.973 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T21:34:26.262 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:34:27.264 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:34:27.264 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:34:27.264 INFO:tasks.workunit.client.0.smithi136.stdout:10 2024-09-15T21:34:27.265 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-15T21:34:27.265 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:34:27.265 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T21:34:27.716 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:34:28.717 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:34:28.717 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:34:28.717 INFO:tasks.workunit.client.0.smithi136.stdout:11 2024-09-15T21:34:28.718 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-15T21:34:28.718 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:34:28.718 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T21:34:29.122 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:34:30.124 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:34:30.124 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:34:30.124 INFO:tasks.workunit.client.0.smithi136.stdout:12 2024-09-15T21:34:30.124 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-15T21:34:30.125 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:34:30.125 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T21:34:30.415 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:34:31.416 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:34:31.416 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:34:31.417 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-15T21:34:31.417 INFO:tasks.workunit.client.0.smithi136.stdout:13 2024-09-15T21:34:31.417 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:34:31.417 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T21:34:32.327 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:34:33.329 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:34:33.329 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:34:33.329 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-15T21:34:33.329 INFO:tasks.workunit.client.0.smithi136.stdout:14 2024-09-15T21:34:33.330 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:34:33.330 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T21:34:33.834 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:34:34.836 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:34:34.836 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:34:34.836 INFO:tasks.workunit.client.0.smithi136.stdout:15 2024-09-15T21:34:34.837 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-15T21:34:34.837 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:34:34.837 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T21:34:35.401 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:34:36.402 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:34:36.402 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:34:36.403 INFO:tasks.workunit.client.0.smithi136.stdout:16 2024-09-15T21:34:36.403 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-15T21:34:36.403 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:34:36.403 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T21:34:37.029 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:34:38.030 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:34:38.030 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:34:38.031 INFO:tasks.workunit.client.0.smithi136.stdout:17 2024-09-15T21:34:38.031 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-15T21:34:38.031 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:34:38.031 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T21:34:38.581 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:34:39.583 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:34:39.583 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:34:39.583 INFO:tasks.workunit.client.0.smithi136.stdout:18 2024-09-15T21:34:39.584 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 18 2024-09-15T21:34:39.584 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:34:39.584 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T21:34:40.039 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:34:41.040 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:34:41.040 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:34:41.041 INFO:tasks.workunit.client.0.smithi136.stdout:19 2024-09-15T21:34:41.041 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 19 2024-09-15T21:34:41.041 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:34:41.041 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T21:34:41.400 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:34:42.401 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:34:42.402 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:34:42.402 INFO:tasks.workunit.client.0.smithi136.stdout:20 2024-09-15T21:34:42.402 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 20 2024-09-15T21:34:42.402 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:34:42.402 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T21:34:42.686 INFO:tasks.workunit.client.0.smithi136.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/2407761254,v1:127.0.0.1:6811/2407761254] [v2:127.0.0.1:6812/2407761254,v1:127.0.0.1:6813/2407761254] exists,up b7c965e4-5a20-45ae-b9d3-a4f700ce5474 2024-09-15T21:34:42.687 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-15T21:34:42.687 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-15T21:34:42.687 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-15T21:34:42.687 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5942: TEST_corrupt_snapset_scrub_rep: create_rbd_pool 2024-09-15T21:34:42.687 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:526: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2024-09-15T21:34:42.956 INFO:tasks.workunit.client.0.smithi136.stderr:pool 'rbd' does not exist 2024-09-15T21:34:42.967 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:527: create_rbd_pool: create_pool rbd 4 2024-09-15T21:34:42.967 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create rbd 4 2024-09-15T21:34:43.336 INFO:tasks.workunit.client.0.smithi136.stderr:pool 'rbd' created 2024-09-15T21:34:43.348 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2024-09-15T21:34:44.349 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:528: create_rbd_pool: rbd pool init rbd 2024-09-15T21:34:46.466 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5943: TEST_corrupt_snapset_scrub_rep: wait_for_clean 2024-09-15T21:34:46.466 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-15T21:34:46.467 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-15T21:34:46.467 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-15T21:34:46.467 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-15T21:34:46.467 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-15T21:34:46.467 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-15T21:34:46.468 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-15T21:34:46.468 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-15T21:34:46.468 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-15T21:34:46.574 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-15T21:34:46.574 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-15T21:34:46.575 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-15T21:34:46.575 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-15T21:34:46.575 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-15T21:34:46.575 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-15T21:34:46.849 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-15T21:34:46.850 INFO:tasks.workunit.client.0.smithi136.stderr:1' 2024-09-15T21:34:46.850 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-15T21:34:46.850 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T21:34:46.850 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-15T21:34:46.952 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836494 2024-09-15T21:34:46.952 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836494 2024-09-15T21:34:46.952 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836494' 2024-09-15T21:34:46.953 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T21:34:46.953 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-15T21:34:47.055 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=42949672964 2024-09-15T21:34:47.055 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 42949672964 2024-09-15T21:34:47.056 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836494 1-42949672964' 2024-09-15T21:34:47.056 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T21:34:47.056 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836494 2024-09-15T21:34:47.056 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T21:34:47.057 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-15T21:34:47.057 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836494 2024-09-15T21:34:47.058 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T21:34:47.059 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836494 2024-09-15T21:34:47.059 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836494' 2024-09-15T21:34:47.059 INFO:tasks.workunit.client.0.smithi136.stdout:waiting osd.0 seq 21474836494 2024-09-15T21:34:47.059 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T21:34:47.335 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836493 -lt 21474836494 2024-09-15T21:34:47.335 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-15T21:34:48.336 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-15T21:34:48.337 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T21:34:48.614 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836495 -lt 21474836494 2024-09-15T21:34:48.614 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T21:34:48.615 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-42949672964 2024-09-15T21:34:48.615 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T21:34:48.616 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-15T21:34:48.616 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-42949672964 2024-09-15T21:34:48.616 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T21:34:48.617 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=42949672964 2024-09-15T21:34:48.617 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 42949672964' 2024-09-15T21:34:48.618 INFO:tasks.workunit.client.0.smithi136.stdout:waiting osd.1 seq 42949672964 2024-09-15T21:34:48.618 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-15T21:34:48.893 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 42949672964 -lt 42949672964 2024-09-15T21:34:48.894 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-15T21:34:48.894 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T21:34:48.895 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T21:34:49.240 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 4 == 0 2024-09-15T21:34:49.240 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-15T21:34:49.240 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-15T21:34:49.240 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-15T21:34:49.241 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T21:34:49.241 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T21:34:49.241 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T21:34:49.241 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T21:34:49.513 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=4 2024-09-15T21:34:49.514 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-15T21:34:49.514 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T21:34:49.515 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T21:34:49.859 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 = 4 2024-09-15T21:34:49.860 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-15T21:34:49.860 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-15T21:34:49.860 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5945: TEST_corrupt_snapset_scrub_rep: create_pool foo 1 2024-09-15T21:34:49.860 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create foo 1 2024-09-15T21:34:50.237 INFO:tasks.workunit.client.0.smithi136.stderr:pool 'foo' created 2024-09-15T21:34:50.249 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2024-09-15T21:34:51.251 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5946: TEST_corrupt_snapset_scrub_rep: create_pool csr_pool 1 1 2024-09-15T21:34:51.251 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create csr_pool 1 1 2024-09-15T21:34:51.694 INFO:tasks.workunit.client.0.smithi136.stderr:pool 'csr_pool' created 2024-09-15T21:34:51.706 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2024-09-15T21:34:52.708 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5947: TEST_corrupt_snapset_scrub_rep: wait_for_clean 2024-09-15T21:34:52.708 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-15T21:34:52.708 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-15T21:34:52.708 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-15T21:34:52.708 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-15T21:34:52.709 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-15T21:34:52.709 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-15T21:34:52.709 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-15T21:34:52.709 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-15T21:34:52.709 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-15T21:34:52.817 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-15T21:34:52.817 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-15T21:34:52.817 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-15T21:34:52.817 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-15T21:34:52.817 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-15T21:34:52.817 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-15T21:34:53.090 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-15T21:34:53.090 INFO:tasks.workunit.client.0.smithi136.stderr:1' 2024-09-15T21:34:53.090 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-15T21:34:53.090 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T21:34:53.091 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-15T21:34:53.204 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836497 2024-09-15T21:34:53.204 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836497 2024-09-15T21:34:53.204 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836497' 2024-09-15T21:34:53.204 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T21:34:53.204 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-15T21:34:53.308 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=42949672967 2024-09-15T21:34:53.309 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 42949672967 2024-09-15T21:34:53.309 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836497 1-42949672967' 2024-09-15T21:34:53.309 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T21:34:53.309 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836497 2024-09-15T21:34:53.309 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T21:34:53.310 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-15T21:34:53.311 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836497 2024-09-15T21:34:53.311 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T21:34:53.312 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836497 2024-09-15T21:34:53.312 INFO:tasks.workunit.client.0.smithi136.stdout:waiting osd.0 seq 21474836497 2024-09-15T21:34:53.312 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836497' 2024-09-15T21:34:53.312 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T21:34:53.602 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836497 -lt 21474836497 2024-09-15T21:34:53.603 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T21:34:53.603 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-42949672967 2024-09-15T21:34:53.603 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T21:34:53.604 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-15T21:34:53.605 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-42949672967 2024-09-15T21:34:53.605 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T21:34:53.606 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=42949672967 2024-09-15T21:34:53.606 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 42949672967' 2024-09-15T21:34:53.607 INFO:tasks.workunit.client.0.smithi136.stdout:waiting osd.1 seq 42949672967 2024-09-15T21:34:53.607 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-15T21:34:53.883 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 42949672966 -lt 42949672967 2024-09-15T21:34:53.883 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-15T21:34:54.885 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-15T21:34:54.885 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-15T21:34:55.171 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 42949672966 -lt 42949672967 2024-09-15T21:34:55.171 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-15T21:34:56.172 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 299 -eq 0 ']' 2024-09-15T21:34:56.173 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-15T21:34:56.459 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 42949672967 -lt 42949672967 2024-09-15T21:34:56.459 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-15T21:34:56.460 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T21:34:56.460 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T21:34:56.806 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 6 == 0 2024-09-15T21:34:56.806 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-15T21:34:56.806 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-15T21:34:56.807 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-15T21:34:56.807 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T21:34:56.807 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T21:34:56.807 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T21:34:56.807 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T21:34:57.078 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=6 2024-09-15T21:34:57.079 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-15T21:34:57.079 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T21:34:57.079 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T21:34:57.426 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 = 6 2024-09-15T21:34:57.426 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-15T21:34:57.427 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-15T21:34:57.427 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5949: TEST_corrupt_snapset_scrub_rep: seq 1 2 2024-09-15T21:34:57.428 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5949: TEST_corrupt_snapset_scrub_rep: for i in $(seq 1 $total_objs) 2024-09-15T21:34:57.428 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5950: TEST_corrupt_snapset_scrub_rep: objname=ROBJ1 2024-09-15T21:34:57.428 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5951: TEST_corrupt_snapset_scrub_rep: add_something td/osd-scrub-repair csr_pool ROBJ1 2024-09-15T21:34:57.428 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-09-15T21:34:57.428 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=csr_pool 2024-09-15T21:34:57.428 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=ROBJ1 2024-09-15T21:34:57.428 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-09-15T21:34:57.429 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-09-15T21:34:57.429 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-09-15T21:34:57.798 INFO:tasks.workunit.client.0.smithi136.stderr:noscrub is set 2024-09-15T21:34:57.810 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-09-15T21:34:58.242 INFO:tasks.workunit.client.0.smithi136.stderr:nodeep-scrub is set 2024-09-15T21:34:58.254 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-09-15T21:34:58.254 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-09-15T21:34:58.254 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool csr_pool put ROBJ1 td/osd-scrub-repair/ORIGINAL 2024-09-15T21:34:58.493 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5953: TEST_corrupt_snapset_scrub_rep: rados --pool csr_pool setomapheader ROBJ1 hdr-ROBJ1 2024-09-15T21:34:58.621 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5954: TEST_corrupt_snapset_scrub_rep: rados --pool csr_pool setomapval ROBJ1 key-ROBJ1 val-ROBJ1 2024-09-15T21:34:58.706 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5949: TEST_corrupt_snapset_scrub_rep: for i in $(seq 1 $total_objs) 2024-09-15T21:34:58.706 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5950: TEST_corrupt_snapset_scrub_rep: objname=ROBJ2 2024-09-15T21:34:58.706 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5951: TEST_corrupt_snapset_scrub_rep: add_something td/osd-scrub-repair csr_pool ROBJ2 2024-09-15T21:34:58.706 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-09-15T21:34:58.707 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=csr_pool 2024-09-15T21:34:58.707 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=ROBJ2 2024-09-15T21:34:58.707 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-09-15T21:34:58.707 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-09-15T21:34:58.707 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-09-15T21:34:59.070 INFO:tasks.workunit.client.0.smithi136.stderr:noscrub is set 2024-09-15T21:34:59.082 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-09-15T21:34:59.506 INFO:tasks.workunit.client.0.smithi136.stderr:nodeep-scrub is set 2024-09-15T21:34:59.518 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-09-15T21:34:59.518 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-09-15T21:34:59.518 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool csr_pool put ROBJ2 td/osd-scrub-repair/ORIGINAL 2024-09-15T21:34:59.768 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5953: TEST_corrupt_snapset_scrub_rep: rados --pool csr_pool setomapheader ROBJ2 hdr-ROBJ2 2024-09-15T21:34:59.885 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5954: TEST_corrupt_snapset_scrub_rep: rados --pool csr_pool setomapval ROBJ2 key-ROBJ2 val-ROBJ2 2024-09-15T21:34:59.978 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5957: TEST_corrupt_snapset_scrub_rep: get_pg csr_pool ROBJ0 2024-09-15T21:34:59.979 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1084: get_pg: local poolname=csr_pool 2024-09-15T21:34:59.979 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1085: get_pg: local objectname=ROBJ0 2024-09-15T21:34:59.979 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1087: get_pg: ceph --format json osd map csr_pool ROBJ0 2024-09-15T21:34:59.979 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1087: get_pg: jq -r .pgid 2024-09-15T21:35:00.257 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5957: TEST_corrupt_snapset_scrub_rep: local pg=3.0 2024-09-15T21:35:00.258 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5958: TEST_corrupt_snapset_scrub_rep: get_primary csr_pool ROBJ0 2024-09-15T21:35:00.258 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1187: get_primary: local poolname=csr_pool 2024-09-15T21:35:00.258 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1188: get_primary: local objectname=ROBJ0 2024-09-15T21:35:00.258 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1190: get_primary: ceph --format json osd map csr_pool ROBJ0 2024-09-15T21:35:00.258 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1191: get_primary: jq .acting_primary 2024-09-15T21:35:00.535 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5958: TEST_corrupt_snapset_scrub_rep: local primary=1 2024-09-15T21:35:00.535 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5960: TEST_corrupt_snapset_scrub_rep: rados -p csr_pool mksnap snap1 2024-09-15T21:35:00.744 INFO:tasks.workunit.client.0.smithi136.stdout:created pool csr_pool snap snap1 2024-09-15T21:35:00.746 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5961: TEST_corrupt_snapset_scrub_rep: echo -n head_of_snapshot_data 2024-09-15T21:35:00.747 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5963: TEST_corrupt_snapset_scrub_rep: seq 1 2 2024-09-15T21:35:00.748 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5963: TEST_corrupt_snapset_scrub_rep: for i in $(seq 1 $total_objs) 2024-09-15T21:35:00.748 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5964: TEST_corrupt_snapset_scrub_rep: objname=ROBJ1 2024-09-15T21:35:00.748 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5967: TEST_corrupt_snapset_scrub_rep: expr 1 % 2 2024-09-15T21:35:00.749 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5967: TEST_corrupt_snapset_scrub_rep: local osd=1 2024-09-15T21:35:00.749 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5969: TEST_corrupt_snapset_scrub_rep: case $i in 2024-09-15T21:35:00.750 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5971: TEST_corrupt_snapset_scrub_rep: rados --pool csr_pool put ROBJ1 td/osd-scrub-repair/change 2024-09-15T21:35:00.998 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5972: TEST_corrupt_snapset_scrub_rep: objectstore_tool td/osd-scrub-repair 1 --head ROBJ1 clear-snapset corrupt 2024-09-15T21:35:00.998 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2024-09-15T21:35:00.999 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2024-09-15T21:35:00.999 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=1 2024-09-15T21:35:00.999 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2024-09-15T21:35:00.999 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 --head ROBJ1 clear-snapset corrupt 2024-09-15T21:35:00.999 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-09-15T21:35:00.999 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2024-09-15T21:35:00.999 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=1 2024-09-15T21:35:00.999 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2024-09-15T21:35:00.999 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-09-15T21:35:00.999 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-15T21:35:00.999 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-15T21:35:00.999 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-15T21:35:00.999 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-15T21:35:01.000 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-15T21:35:01.305 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-15T21:35:01.305 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 --head ROBJ1 clear-snapset corrupt 2024-09-15T21:35:01.305 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-15T21:35:01.306 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-15T21:35:01.306 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=1 2024-09-15T21:35:01.306 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-15T21:35:01.306 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-09-15T21:35:01.306 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 --head ROBJ1 clear-snapset corrupt 2024-09-15T21:35:03.356 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2024-09-15T21:35:03.356 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2024-09-15T21:35:03.356 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-15T21:35:03.357 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=1 2024-09-15T21:35:03.357 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-15T21:35:03.357 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-09-15T21:35:03.357 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=ce4892c7-6c6d-4b4f-8e6f-07a3e11133e5 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-15T21:35:03.357 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T21:35:03.357 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T21:35:03.357 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T21:35:03.357 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-09-15T21:35:03.357 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-09-15T21:35:03.357 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-15T21:35:03.357 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-15T21:35:03.357 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-15T21:35:03.358 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-15T21:35:03.358 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-15T21:35:03.358 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-15T21:35:03.358 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-15T21:35:03.358 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T21:35:03.358 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52602 2024-09-15T21:35:03.358 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.52602/$cluster-$name.asok' 2024-09-15T21:35:03.358 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.52602/$cluster-$name.asok' 2024-09-15T21:35:03.358 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-15T21:35:03.358 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-15T21:35:03.358 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-15T21:35:03.358 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T21:35:03.358 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T21:35:03.359 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T21:35:03.359 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T21:35:03.359 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-15T21:35:03.359 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-15T21:35:03.359 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-09-15T21:35:03.360 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.1 2024-09-15T21:35:03.360 INFO:tasks.workunit.client.0.smithi136.stderr:start osd.1 2024-09-15T21:35:03.360 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 1 --fsid=ce4892c7-6c6d-4b4f-8e6f-07a3e11133e5 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.52602/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T21:35:03.360 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-09-15T21:35:03.360 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 1 = 1 ']' 2024-09-15T21:35:03.360 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-15T21:35:03.360 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-15T21:35:03.361 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-15T21:35:03.387 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:35:03.385+0000 7f958a60f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:35:03.414 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:35:03.412+0000 7f958a60f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:35:03.415 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:35:03.413+0000 7f958a60f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:35:03.673 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 1 2024-09-15T21:35:03.673 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-15T21:35:03.673 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2024-09-15T21:35:03.673 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-15T21:35:03.673 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-15T21:35:03.673 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:35:03.673 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-15T21:35:03.673 INFO:tasks.workunit.client.0.smithi136.stderr:0 2024-09-15T21:35:03.673 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:35:03.673 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T21:35:03.949 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:35:04.530 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:35:04.528+0000 7f958a60f640 -1 Falling back to public interface 2024-09-15T21:35:04.950 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:35:04.950 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:35:04.950 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-15T21:35:04.951 INFO:tasks.workunit.client.0.smithi136.stderr:1 2024-09-15T21:35:04.951 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:35:04.951 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T21:35:05.235 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:35:06.237 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:35:06.237 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:35:06.237 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-15T21:35:06.238 INFO:tasks.workunit.client.0.smithi136.stderr:2 2024-09-15T21:35:06.238 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:35:06.238 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T21:35:06.527 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:35:07.212 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:35:07.208+0000 7f958a60f640 -1 osd.1 32 log_to_monitors true 2024-09-15T21:35:07.529 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:35:07.529 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:35:07.529 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-15T21:35:07.529 INFO:tasks.workunit.client.0.smithi136.stderr:3 2024-09-15T21:35:07.529 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:35:07.529 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T21:35:07.817 INFO:tasks.workunit.client.0.smithi136.stderr:osd.1 up in weight 1 up_from 36 up_thru 25 down_at 33 last_clean_interval [10,32) [v2:127.0.0.1:6810/3897922551,v1:127.0.0.1:6811/3897922551] [v2:127.0.0.1:6812/3897922551,v1:127.0.0.1:6813/3897922551] exists,up b7c965e4-5a20-45ae-b9d3-a4f700ce5474 2024-09-15T21:35:07.817 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-15T21:35:07.817 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-15T21:35:07.817 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-15T21:35:07.817 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2024-09-15T21:35:07.817 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-15T21:35:07.818 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-15T21:35:07.818 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-15T21:35:07.818 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-15T21:35:07.818 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-15T21:35:07.818 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-15T21:35:07.819 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-15T21:35:07.819 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-15T21:35:07.819 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-15T21:35:07.914 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-15T21:35:07.914 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-15T21:35:07.914 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-15T21:35:07.915 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-15T21:35:07.915 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-15T21:35:07.915 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-15T21:35:08.451 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-15T21:35:08.452 INFO:tasks.workunit.client.0.smithi136.stderr:1' 2024-09-15T21:35:08.452 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-15T21:35:08.452 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T21:35:08.452 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-15T21:35:08.692 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836501 2024-09-15T21:35:08.692 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836501 2024-09-15T21:35:08.692 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836501' 2024-09-15T21:35:08.692 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T21:35:08.692 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-15T21:35:08.796 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=154618822658 2024-09-15T21:35:08.796 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 154618822658 2024-09-15T21:35:08.796 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836501 1-154618822658' 2024-09-15T21:35:08.796 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T21:35:08.797 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836501 2024-09-15T21:35:08.797 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T21:35:08.797 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-15T21:35:08.798 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836501 2024-09-15T21:35:08.798 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T21:35:08.799 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836501 2024-09-15T21:35:08.799 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836501' 2024-09-15T21:35:08.799 INFO:tasks.workunit.client.0.smithi136.stderr:waiting osd.0 seq 21474836501 2024-09-15T21:35:08.800 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T21:35:09.078 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836500 -lt 21474836501 2024-09-15T21:35:09.078 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-15T21:35:10.079 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-15T21:35:10.079 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T21:35:10.369 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836501 -lt 21474836501 2024-09-15T21:35:10.372 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T21:35:10.373 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-154618822658 2024-09-15T21:35:10.374 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T21:35:10.374 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-15T21:35:10.374 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-154618822658 2024-09-15T21:35:10.374 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T21:35:10.374 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=154618822658 2024-09-15T21:35:10.374 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 154618822658' 2024-09-15T21:35:10.374 INFO:tasks.workunit.client.0.smithi136.stderr:waiting osd.1 seq 154618822658 2024-09-15T21:35:10.374 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-15T21:35:10.649 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 154618822658 -lt 154618822658 2024-09-15T21:35:10.650 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-15T21:35:10.650 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T21:35:10.650 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T21:35:10.999 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 6 == 0 2024-09-15T21:35:10.999 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-15T21:35:10.999 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-15T21:35:10.999 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-15T21:35:10.999 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T21:35:10.999 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T21:35:10.999 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T21:35:11.000 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T21:35:11.274 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=3 2024-09-15T21:35:11.274 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-15T21:35:11.275 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T21:35:11.275 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T21:35:11.644 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 3 = 6 2024-09-15T21:35:11.644 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 3 '!=' -1 2024-09-15T21:35:11.644 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: loop=0 2024-09-15T21:35:11.644 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1652: wait_for_clean: num_active_clean=3 2024-09-15T21:35:11.644 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-15T21:35:11.644 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 0.1 2024-09-15T21:35:11.745 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-15T21:35:11.746 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-15T21:35:11.746 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-15T21:35:11.746 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-15T21:35:11.746 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T21:35:11.746 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T21:35:11.746 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T21:35:11.746 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T21:35:12.021 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=3 2024-09-15T21:35:12.021 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-15T21:35:12.021 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T21:35:12.021 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T21:35:12.364 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 3 = 6 2024-09-15T21:35:12.364 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 3 '!=' 3 2024-09-15T21:35:12.364 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: get_is_making_recovery_progress 2024-09-15T21:35:12.364 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2024-09-15T21:35:12.364 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-15T21:35:12.364 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-15T21:35:12.364 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-15T21:35:12.365 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2024-09-15T21:35:12.365 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-15T21:35:12.744 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=null 2024-09-15T21:35:12.744 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test null '!=' null 2024-09-15T21:35:12.744 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1655: wait_for_clean: (( 1 >= 13 )) 2024-09-15T21:35:12.744 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-15T21:35:12.744 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 0.2 2024-09-15T21:35:12.945 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-15T21:35:12.945 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-15T21:35:12.946 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-15T21:35:12.946 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-15T21:35:12.946 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T21:35:12.946 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T21:35:12.946 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T21:35:12.946 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T21:35:13.218 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=3 2024-09-15T21:35:13.219 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-15T21:35:13.219 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T21:35:13.219 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T21:35:13.564 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 3 = 6 2024-09-15T21:35:13.565 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 3 '!=' 3 2024-09-15T21:35:13.565 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: get_is_making_recovery_progress 2024-09-15T21:35:13.565 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2024-09-15T21:35:13.565 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-15T21:35:13.565 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-15T21:35:13.565 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-15T21:35:13.565 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2024-09-15T21:35:13.565 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-15T21:35:13.932 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=null 2024-09-15T21:35:13.932 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test null '!=' null 2024-09-15T21:35:13.932 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1655: wait_for_clean: (( 2 >= 13 )) 2024-09-15T21:35:13.932 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-15T21:35:13.933 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 0.4 2024-09-15T21:35:14.333 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-15T21:35:14.333 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-15T21:35:14.334 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-15T21:35:14.334 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-15T21:35:14.334 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T21:35:14.334 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T21:35:14.334 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T21:35:14.334 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T21:35:14.609 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=6 2024-09-15T21:35:14.609 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-15T21:35:14.609 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T21:35:14.609 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T21:35:14.958 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 = 6 2024-09-15T21:35:14.958 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-15T21:35:14.959 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-15T21:35:14.959 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5963: TEST_corrupt_snapset_scrub_rep: for i in $(seq 1 $total_objs) 2024-09-15T21:35:14.959 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5964: TEST_corrupt_snapset_scrub_rep: objname=ROBJ2 2024-09-15T21:35:14.959 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5967: TEST_corrupt_snapset_scrub_rep: expr 2 % 2 2024-09-15T21:35:14.960 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5967: TEST_corrupt_snapset_scrub_rep: local osd=0 2024-09-15T21:35:14.960 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5969: TEST_corrupt_snapset_scrub_rep: case $i in 2024-09-15T21:35:14.960 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5976: TEST_corrupt_snapset_scrub_rep: rados --pool csr_pool put ROBJ2 td/osd-scrub-repair/change 2024-09-15T21:35:15.096 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5977: TEST_corrupt_snapset_scrub_rep: objectstore_tool td/osd-scrub-repair 0 --head ROBJ2 clear-snapset corrupt 2024-09-15T21:35:15.096 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2024-09-15T21:35:15.096 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2024-09-15T21:35:15.096 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=0 2024-09-15T21:35:15.097 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2024-09-15T21:35:15.097 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 --head ROBJ2 clear-snapset corrupt 2024-09-15T21:35:15.097 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-09-15T21:35:15.097 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2024-09-15T21:35:15.097 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=0 2024-09-15T21:35:15.097 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2024-09-15T21:35:15.097 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2024-09-15T21:35:15.097 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-15T21:35:15.097 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-15T21:35:15.097 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-15T21:35:15.097 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-15T21:35:15.097 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-15T21:35:15.403 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-15T21:35:15.403 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 --head ROBJ2 clear-snapset corrupt 2024-09-15T21:35:15.403 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-15T21:35:15.403 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-15T21:35:15.403 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=0 2024-09-15T21:35:15.403 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-15T21:35:15.403 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-09-15T21:35:15.404 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 --head ROBJ2 clear-snapset corrupt 2024-09-15T21:35:17.411 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 2024-09-15T21:35:17.412 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2024-09-15T21:35:17.412 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-15T21:35:17.412 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=0 2024-09-15T21:35:17.412 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-15T21:35:17.412 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-09-15T21:35:17.412 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=ce4892c7-6c6d-4b4f-8e6f-07a3e11133e5 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-15T21:35:17.412 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T21:35:17.413 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T21:35:17.413 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T21:35:17.413 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-09-15T21:35:17.413 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-09-15T21:35:17.413 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-15T21:35:17.413 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-15T21:35:17.413 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-15T21:35:17.413 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-15T21:35:17.413 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-15T21:35:17.414 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-15T21:35:17.414 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-15T21:35:17.414 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T21:35:17.414 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52602 2024-09-15T21:35:17.414 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.52602/$cluster-$name.asok' 2024-09-15T21:35:17.414 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.52602/$cluster-$name.asok' 2024-09-15T21:35:17.414 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-15T21:35:17.414 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-15T21:35:17.414 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-15T21:35:17.415 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T21:35:17.415 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T21:35:17.415 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T21:35:17.415 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T21:35:17.415 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-15T21:35:17.415 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-15T21:35:17.415 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-09-15T21:35:17.415 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.0 2024-09-15T21:35:17.415 INFO:tasks.workunit.client.0.smithi136.stderr:start osd.0 2024-09-15T21:35:17.415 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 0 --fsid=ce4892c7-6c6d-4b4f-8e6f-07a3e11133e5 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.52602/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T21:35:17.415 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-09-15T21:35:17.416 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 0 = 0 ']' 2024-09-15T21:35:17.416 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-15T21:35:17.416 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-15T21:35:17.417 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-15T21:35:17.441 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:35:17.439+0000 7f18a61d7640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:35:17.461 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:35:17.458+0000 7f18a61d7640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:35:17.462 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:35:17.460+0000 7f18a61d7640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:35:17.727 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 0 2024-09-15T21:35:17.727 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-15T21:35:17.728 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2024-09-15T21:35:17.728 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-15T21:35:17.728 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-15T21:35:17.728 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:35:17.728 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-15T21:35:17.728 INFO:tasks.workunit.client.0.smithi136.stderr:0 2024-09-15T21:35:17.728 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:35:17.728 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:35:18.003 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:35:18.575 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:35:18.573+0000 7f18a61d7640 -1 Falling back to public interface 2024-09-15T21:35:19.004 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:35:19.004 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:35:19.005 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-15T21:35:19.005 INFO:tasks.workunit.client.0.smithi136.stderr:1 2024-09-15T21:35:19.005 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:35:19.005 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:35:19.294 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:35:20.295 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:35:20.295 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:35:20.295 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-15T21:35:20.295 INFO:tasks.workunit.client.0.smithi136.stderr:2 2024-09-15T21:35:20.296 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:35:20.296 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:35:20.582 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:35:21.583 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:35:21.584 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:35:21.584 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-15T21:35:21.584 INFO:tasks.workunit.client.0.smithi136.stderr:3 2024-09-15T21:35:21.584 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:35:21.584 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:35:21.592 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:35:21.590+0000 7f18a61d7640 -1 osd.0 37 log_to_monitors true 2024-09-15T21:35:21.912 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:35:22.914 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:35:22.914 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:35:22.914 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-15T21:35:22.914 INFO:tasks.workunit.client.0.smithi136.stderr:4 2024-09-15T21:35:22.914 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:35:22.915 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:35:23.198 INFO:tasks.workunit.client.0.smithi136.stderr:osd.0 up in weight 1 up_from 41 up_thru 41 down_at 38 last_clean_interval [5,37) [v2:127.0.0.1:6802/2747655591,v1:127.0.0.1:6803/2747655591] [v2:127.0.0.1:6804/2747655591,v1:127.0.0.1:6805/2747655591] exists,up 69b3cfa1-62b4-4f5d-aa79-48f162232a77 2024-09-15T21:35:23.199 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-15T21:35:23.199 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-15T21:35:23.199 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-15T21:35:23.199 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2024-09-15T21:35:23.199 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-15T21:35:23.199 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-15T21:35:23.199 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-15T21:35:23.199 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-15T21:35:23.200 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-15T21:35:23.200 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-15T21:35:23.200 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-15T21:35:23.200 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-15T21:35:23.200 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-15T21:35:23.295 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-15T21:35:23.296 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-15T21:35:23.296 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-15T21:35:23.296 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-15T21:35:23.296 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-15T21:35:23.296 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-15T21:35:23.573 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-15T21:35:23.573 INFO:tasks.workunit.client.0.smithi136.stderr:1' 2024-09-15T21:35:23.573 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-15T21:35:23.574 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T21:35:23.574 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-15T21:35:23.692 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=176093659138 2024-09-15T21:35:23.692 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 176093659138 2024-09-15T21:35:23.692 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-176093659138' 2024-09-15T21:35:23.692 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T21:35:23.693 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-15T21:35:23.796 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=154618822662 2024-09-15T21:35:23.796 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 154618822662 2024-09-15T21:35:23.796 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-176093659138 1-154618822662' 2024-09-15T21:35:23.796 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T21:35:23.797 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-176093659138 2024-09-15T21:35:23.797 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T21:35:23.797 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-15T21:35:23.798 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-176093659138 2024-09-15T21:35:23.798 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T21:35:23.800 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=176093659138 2024-09-15T21:35:23.800 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 176093659138' 2024-09-15T21:35:23.800 INFO:tasks.workunit.client.0.smithi136.stderr:waiting osd.0 seq 176093659138 2024-09-15T21:35:23.800 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T21:35:24.077 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 1 -lt 176093659138 2024-09-15T21:35:24.077 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-15T21:35:25.078 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-15T21:35:25.078 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T21:35:25.387 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 1 -lt 176093659138 2024-09-15T21:35:25.387 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-15T21:35:26.388 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 299 -eq 0 ']' 2024-09-15T21:35:26.388 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T21:35:26.668 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 176093659138 -lt 176093659138 2024-09-15T21:35:26.668 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T21:35:26.669 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-154618822662 2024-09-15T21:35:26.669 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T21:35:26.670 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-15T21:35:26.670 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-154618822662 2024-09-15T21:35:26.670 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T21:35:26.671 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=154618822662 2024-09-15T21:35:26.672 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 154618822662' 2024-09-15T21:35:26.672 INFO:tasks.workunit.client.0.smithi136.stderr:waiting osd.1 seq 154618822662 2024-09-15T21:35:26.672 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-15T21:35:26.948 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 154618822662 -lt 154618822662 2024-09-15T21:35:26.948 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-15T21:35:26.948 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T21:35:26.948 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T21:35:27.292 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 6 == 0 2024-09-15T21:35:27.292 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-15T21:35:27.292 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-15T21:35:27.293 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-15T21:35:27.293 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T21:35:27.293 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T21:35:27.293 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T21:35:27.293 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T21:35:27.567 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=6 2024-09-15T21:35:27.567 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-15T21:35:27.567 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T21:35:27.568 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T21:35:27.913 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 = 6 2024-09-15T21:35:27.913 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-15T21:35:27.913 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-15T21:35:27.914 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5982: TEST_corrupt_snapset_scrub_rep: rm td/osd-scrub-repair/change 2024-09-15T21:35:27.914 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5984: TEST_corrupt_snapset_scrub_rep: pg_scrub 3.0 2024-09-15T21:35:27.914 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1883: pg_scrub: local pgid=3.0 2024-09-15T21:35:27.915 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1884: pg_scrub: get_last_scrub_stamp 3.0 2024-09-15T21:35:27.915 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=3.0 2024-09-15T21:35:27.915 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-15T21:35:27.915 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-15T21:35:27.915 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_scrub_stamp' 2024-09-15T21:35:28.188 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1884: pg_scrub: local last_scrub=2024-09-15T21:34:51.600242+0000 2024-09-15T21:35:28.188 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1885: pg_scrub: ceph pg scrub 3.0 2024-09-15T21:35:28.449 INFO:tasks.workunit.client.0.smithi136.stderr:instructing pg 3.0 on osd.1 to scrub 2024-09-15T21:35:28.461 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1886: pg_scrub: wait_for_scrub 3.0 2024-09-15T21:34:51.600242+0000 2024-09-15T21:35:28.461 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: local pgid=3.0 2024-09-15T21:35:28.461 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2018: wait_for_scrub: local last_scrub=2024-09-15T21:34:51.600242+0000 2024-09-15T21:35:28.461 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2019: wait_for_scrub: local sname=last_scrub_stamp 2024-09-15T21:35:28.461 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i=0 )) 2024-09-15T21:35:28.461 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-15T21:35:28.461 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 3.0 last_scrub_stamp 2024-09-15T21:35:28.461 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=3.0 2024-09-15T21:35:28.461 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-15T21:35:28.461 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-15T21:35:28.461 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_scrub_stamp' 2024-09-15T21:35:28.732 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-15T21:34:51.600242+0000 '>' 2024-09-15T21:34:51.600242+0000 2024-09-15T21:35:28.732 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-15T21:35:29.733 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-15T21:35:29.734 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-15T21:35:29.734 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 3.0 last_scrub_stamp 2024-09-15T21:35:29.734 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=3.0 2024-09-15T21:35:29.734 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-15T21:35:29.735 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-15T21:35:29.735 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_scrub_stamp' 2024-09-15T21:35:30.017 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-15T21:34:51.600242+0000 '>' 2024-09-15T21:34:51.600242+0000 2024-09-15T21:35:30.017 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-15T21:35:31.018 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-15T21:35:31.018 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-15T21:35:31.019 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 3.0 last_scrub_stamp 2024-09-15T21:35:31.019 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=3.0 2024-09-15T21:35:31.019 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-15T21:35:31.019 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-15T21:35:31.019 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_scrub_stamp' 2024-09-15T21:35:31.299 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-15T21:34:51.600242+0000 '>' 2024-09-15T21:34:51.600242+0000 2024-09-15T21:35:31.300 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-15T21:35:32.301 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-15T21:35:32.301 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-15T21:35:32.302 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 3.0 last_scrub_stamp 2024-09-15T21:35:32.302 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=3.0 2024-09-15T21:35:32.302 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-15T21:35:32.302 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-15T21:35:32.302 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_scrub_stamp' 2024-09-15T21:35:32.581 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-15T21:34:51.600242+0000 '>' 2024-09-15T21:34:51.600242+0000 2024-09-15T21:35:32.581 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-15T21:35:33.582 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-15T21:35:33.582 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-15T21:35:33.583 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 3.0 last_scrub_stamp 2024-09-15T21:35:33.583 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=3.0 2024-09-15T21:35:33.583 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-15T21:35:33.583 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-15T21:35:33.583 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_scrub_stamp' 2024-09-15T21:35:33.864 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-15T21:35:28.492617+0000 '>' 2024-09-15T21:34:51.600242+0000 2024-09-15T21:35:33.864 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2023: wait_for_scrub: return 0 2024-09-15T21:35:33.864 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5986: TEST_corrupt_snapset_scrub_rep: rados list-inconsistent-pg csr_pool 2024-09-15T21:35:33.892 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5988: TEST_corrupt_snapset_scrub_rep: jq '. | length' td/osd-scrub-repair/json 2024-09-15T21:35:33.894 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5988: TEST_corrupt_snapset_scrub_rep: test 1 = 1 2024-09-15T21:35:33.894 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5990: TEST_corrupt_snapset_scrub_rep: jq -r '.[0]' td/osd-scrub-repair/json 2024-09-15T21:35:33.897 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5990: TEST_corrupt_snapset_scrub_rep: test 3.0 = 3.0 2024-09-15T21:35:33.897 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5992: TEST_corrupt_snapset_scrub_rep: rados list-inconsistent-obj 3.0 2024-09-15T21:35:33.925 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5994: TEST_corrupt_snapset_scrub_rep: jq 'def walk(f): 2024-09-15T21:35:33.925 INFO:tasks.workunit.client.0.smithi136.stderr: . as $in 2024-09-15T21:35:33.925 INFO:tasks.workunit.client.0.smithi136.stderr: | if type == "object" then 2024-09-15T21:35:33.925 INFO:tasks.workunit.client.0.smithi136.stderr: reduce keys[] as $key 2024-09-15T21:35:33.925 INFO:tasks.workunit.client.0.smithi136.stderr: ( {}; . + { ($key): ($in[$key] | walk(f)) } ) | f 2024-09-15T21:35:33.925 INFO:tasks.workunit.client.0.smithi136.stderr: elif type == "array" then map( walk(f) ) | f 2024-09-15T21:35:33.925 INFO:tasks.workunit.client.0.smithi136.stderr: else f 2024-09-15T21:35:33.925 INFO:tasks.workunit.client.0.smithi136.stderr: end; 2024-09-15T21:35:33.925 INFO:tasks.workunit.client.0.smithi136.stderr:walk(if type == "object" then del(.mtime) else . end) 2024-09-15T21:35:33.926 INFO:tasks.workunit.client.0.smithi136.stderr:| walk(if type == "object" then del(.local_mtime) else . end) 2024-09-15T21:35:33.926 INFO:tasks.workunit.client.0.smithi136.stderr:| walk(if type == "object" then del(.last_reqid) else . end) 2024-09-15T21:35:33.926 INFO:tasks.workunit.client.0.smithi136.stderr:| walk(if type == "object" then del(.version) else . end) 2024-09-15T21:35:33.926 INFO:tasks.workunit.client.0.smithi136.stderr:| walk(if type == "object" then del(.prior_version) else . end)' 2024-09-15T21:35:33.926 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5994: TEST_corrupt_snapset_scrub_rep: jq .inconsistents 2024-09-15T21:35:33.926 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5994: TEST_corrupt_snapset_scrub_rep: python3 -c 'import json; import sys ; JSON=sys.stdin.read() ; ud = json.loads(JSON) ; print(json.dumps(ud, sort_keys=True, indent=2))' 2024-09-15T21:35:33.947 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6160: TEST_corrupt_snapset_scrub_rep: jq 'def walk(f): 2024-09-15T21:35:33.948 INFO:tasks.workunit.client.0.smithi136.stderr: . as $in 2024-09-15T21:35:33.948 INFO:tasks.workunit.client.0.smithi136.stderr: | if type == "object" then 2024-09-15T21:35:33.948 INFO:tasks.workunit.client.0.smithi136.stderr: reduce keys[] as $key 2024-09-15T21:35:33.948 INFO:tasks.workunit.client.0.smithi136.stderr: ( {}; . + { ($key): ($in[$key] | walk(f)) } ) | f 2024-09-15T21:35:33.948 INFO:tasks.workunit.client.0.smithi136.stderr: elif type == "array" then map( walk(f) ) | f 2024-09-15T21:35:33.948 INFO:tasks.workunit.client.0.smithi136.stderr: else f 2024-09-15T21:35:33.948 INFO:tasks.workunit.client.0.smithi136.stderr: end; 2024-09-15T21:35:33.948 INFO:tasks.workunit.client.0.smithi136.stderr:walk(if type == "object" then del(.mtime) else . end) 2024-09-15T21:35:33.948 INFO:tasks.workunit.client.0.smithi136.stderr:| walk(if type == "object" then del(.local_mtime) else . end) 2024-09-15T21:35:33.948 INFO:tasks.workunit.client.0.smithi136.stderr:| walk(if type == "object" then del(.last_reqid) else . end) 2024-09-15T21:35:33.948 INFO:tasks.workunit.client.0.smithi136.stderr:| walk(if type == "object" then del(.version) else . end) 2024-09-15T21:35:33.948 INFO:tasks.workunit.client.0.smithi136.stderr:| walk(if type == "object" then del(.prior_version) else . end)' td/osd-scrub-repair/json 2024-09-15T21:35:33.948 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6160: TEST_corrupt_snapset_scrub_rep: jq .inconsistents 2024-09-15T21:35:33.948 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6160: TEST_corrupt_snapset_scrub_rep: python3 -c 'import json; import sys ; JSON=sys.stdin.read() ; ud = json.loads(JSON) ; print(json.dumps(ud, sort_keys=True, indent=2))' 2024-09-15T21:35:33.969 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6161: TEST_corrupt_snapset_scrub_rep: multidiff td/osd-scrub-repair/checkcsjson td/osd-scrub-repair/csjson 2024-09-15T21:35:33.969 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2434: multidiff: diff td/osd-scrub-repair/checkcsjson td/osd-scrub-repair/csjson 2024-09-15T21:35:33.971 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6162: TEST_corrupt_snapset_scrub_rep: test no = yes 2024-09-15T21:35:33.971 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6167: TEST_corrupt_snapset_scrub_rep: test '' = yes 2024-09-15T21:35:33.971 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6172: TEST_corrupt_snapset_scrub_rep: ERRORS=0 2024-09-15T21:35:33.971 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6173: TEST_corrupt_snapset_scrub_rep: declare -a err_strings 2024-09-15T21:35:33.971 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6174: TEST_corrupt_snapset_scrub_rep: err_strings[0]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid [0-9]*:.*:::ROBJ1:head : snapset inconsistent' 2024-09-15T21:35:33.971 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6175: TEST_corrupt_snapset_scrub_rep: err_strings[1]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid [0-9]*:.*:::ROBJ2:head : snapset inconsistent' 2024-09-15T21:35:33.971 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6176: TEST_corrupt_snapset_scrub_rep: err_strings[2]='log_channel[(]cluster[)] log [[]ERR[]] : scrub [0-9]*[.]0 [0-9]*:.*:::ROBJ1:1 : is an unexpected clone' 2024-09-15T21:35:33.971 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6177: TEST_corrupt_snapset_scrub_rep: err_strings[3]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 scrub : stat mismatch, got 3/4 objects, 1/2 clones, 3/4 dirty, 3/4 omap, 0/0 pinned, 0/0 hit_set_archive, 0/0 whiteouts, 49/56 bytes, 0/0 manifest objects, 0/0 hit_set_archive bytes.' 2024-09-15T21:35:33.972 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6178: TEST_corrupt_snapset_scrub_rep: err_strings[4]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 scrub 0 missing, 2 inconsistent objects' 2024-09-15T21:35:33.972 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6179: TEST_corrupt_snapset_scrub_rep: err_strings[5]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 scrub 4 errors' 2024-09-15T21:35:33.972 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6181: TEST_corrupt_snapset_scrub_rep: for err_string in "${err_strings[@]}" 2024-09-15T21:35:33.972 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6183: TEST_corrupt_snapset_scrub_rep: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid [0-9]*:.*:::ROBJ1:head : snapset inconsistent' td/osd-scrub-repair/osd.1.log 2024-09-15T21:35:33.973 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6181: TEST_corrupt_snapset_scrub_rep: for err_string in "${err_strings[@]}" 2024-09-15T21:35:33.973 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6183: TEST_corrupt_snapset_scrub_rep: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid [0-9]*:.*:::ROBJ2:head : snapset inconsistent' td/osd-scrub-repair/osd.1.log 2024-09-15T21:35:33.976 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6181: TEST_corrupt_snapset_scrub_rep: for err_string in "${err_strings[@]}" 2024-09-15T21:35:33.976 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6183: TEST_corrupt_snapset_scrub_rep: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : scrub [0-9]*[.]0 [0-9]*:.*:::ROBJ1:1 : is an unexpected clone' td/osd-scrub-repair/osd.1.log 2024-09-15T21:35:33.979 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6181: TEST_corrupt_snapset_scrub_rep: for err_string in "${err_strings[@]}" 2024-09-15T21:35:33.979 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6183: TEST_corrupt_snapset_scrub_rep: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 scrub : stat mismatch, got 3/4 objects, 1/2 clones, 3/4 dirty, 3/4 omap, 0/0 pinned, 0/0 hit_set_archive, 0/0 whiteouts, 49/56 bytes, 0/0 manifest objects, 0/0 hit_set_archive bytes.' td/osd-scrub-repair/osd.1.log 2024-09-15T21:35:33.981 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6181: TEST_corrupt_snapset_scrub_rep: for err_string in "${err_strings[@]}" 2024-09-15T21:35:33.981 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6183: TEST_corrupt_snapset_scrub_rep: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 scrub 0 missing, 2 inconsistent objects' td/osd-scrub-repair/osd.1.log 2024-09-15T21:35:33.984 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6181: TEST_corrupt_snapset_scrub_rep: for err_string in "${err_strings[@]}" 2024-09-15T21:35:33.984 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6183: TEST_corrupt_snapset_scrub_rep: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 scrub 4 errors' td/osd-scrub-repair/osd.1.log 2024-09-15T21:35:33.986 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6190: TEST_corrupt_snapset_scrub_rep: '[' 0 '!=' 0 ']' 2024-09-15T21:35:33.986 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6196: TEST_corrupt_snapset_scrub_rep: ceph osd pool rm csr_pool csr_pool --yes-i-really-really-mean-it 2024-09-15T21:35:34.408 INFO:tasks.workunit.client.0.smithi136.stderr:pool 'csr_pool' removed 2024-09-15T21:35:34.420 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:66: run: teardown td/osd-scrub-repair 2024-09-15T21:35:34.420 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/osd-scrub-repair 2024-09-15T21:35:34.420 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-15T21:35:34.420 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/osd-scrub-repair KILL 2024-09-15T21:35:34.421 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-15T21:35:34.421 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-15T21:35:34.421 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-15T21:35:34.421 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-15T21:35:34.421 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-15T21:35:34.528 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-15T21:35:34.528 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-15T21:35:34.529 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-15T21:35:34.530 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-15T21:35:34.531 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-15T21:35:34.531 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-15T21:35:34.531 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-15T21:35:34.532 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T21:35:34.532 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-15T21:35:34.533 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-15T21:35:34.533 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T21:35:34.533 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-15T21:35:34.534 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-15T21:35:34.534 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/osd-scrub-repair 2024-09-15T21:35:34.548 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-15T21:35:34.548 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T21:35:34.548 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52602 2024-09-15T21:35:34.548 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.52602 2024-09-15T21:35:34.550 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-15T21:35:34.550 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-15T21:35:34.550 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:63: run: for func in $funcs 2024-09-15T21:35:34.550 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:64: run: setup td/osd-scrub-repair 2024-09-15T21:35:34.550 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/osd-scrub-repair 2024-09-15T21:35:34.550 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/osd-scrub-repair 2024-09-15T21:35:34.550 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/osd-scrub-repair 2024-09-15T21:35:34.550 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-15T21:35:34.550 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/osd-scrub-repair KILL 2024-09-15T21:35:34.550 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-15T21:35:34.550 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-15T21:35:34.551 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-15T21:35:34.551 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-15T21:35:34.551 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-15T21:35:34.551 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-15T21:35:34.552 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-15T21:35:34.553 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-15T21:35:34.553 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-15T21:35:34.554 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-15T21:35:34.554 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-15T21:35:34.554 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-15T21:35:34.555 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T21:35:34.555 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-15T21:35:34.556 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-15T21:35:34.556 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T21:35:34.557 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-15T21:35:34.558 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-15T21:35:34.558 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/osd-scrub-repair 2024-09-15T21:35:34.559 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-15T21:35:34.559 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T21:35:34.559 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52602 2024-09-15T21:35:34.560 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.52602 2024-09-15T21:35:34.560 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-15T21:35:34.560 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-15T21:35:34.560 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/osd-scrub-repair 2024-09-15T21:35:34.562 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2024-09-15T21:35:34.562 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T21:35:34.562 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52602 2024-09-15T21:35:34.562 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.52602 2024-09-15T21:35:34.564 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2024-09-15T21:35:34.564 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 4096 -le 1024 ']' 2024-09-15T21:35:34.565 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2024-09-15T21:35:34.565 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/osd-scrub-repair 1' TERM HUP INT 2024-09-15T21:35:34.565 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:65: run: TEST_list_missing_erasure_coded_appends td/osd-scrub-repair 2024-09-15T21:35:34.565 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:978: TEST_list_missing_erasure_coded_appends: list_missing_erasure_coded td/osd-scrub-repair false 2024-09-15T21:35:34.565 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:916: list_missing_erasure_coded: local dir=td/osd-scrub-repair 2024-09-15T21:35:34.565 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:917: list_missing_erasure_coded: local allow_overwrites=false 2024-09-15T21:35:34.565 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:918: list_missing_erasure_coded: local poolname=ecpool 2024-09-15T21:35:34.565 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:920: list_missing_erasure_coded: run_mon td/osd-scrub-repair a 2024-09-15T21:35:34.565 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/osd-scrub-repair 2024-09-15T21:35:34.565 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2024-09-15T21:35:34.565 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2024-09-15T21:35:34.566 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2024-09-15T21:35:34.566 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/osd-scrub-repair/a 2024-09-15T21:35:34.566 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-scrub-repair/a --run-dir=td/osd-scrub-repair 2024-09-15T21:35:35.025 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2024-09-15T21:35:35.025 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-15T21:35:35.025 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-15T21:35:35.025 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-15T21:35:35.025 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T21:35:35.025 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52602 2024-09-15T21:35:35.025 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.52602/$cluster-$name.asok' 2024-09-15T21:35:35.026 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-scrub-repair/a '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.52602/$cluster-$name.asok' --mon-cluster-log-file=td/osd-scrub-repair/log --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-09-15T21:35:35.225 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2024-09-15T21:35:35.227 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2024-09-15T21:35:35.227 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-15T21:35:35.227 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-15T21:35:35.227 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2024-09-15T21:35:35.227 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2024-09-15T21:35:35.227 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-15T21:35:35.227 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-15T21:35:35.227 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-15T21:35:35.228 INFO:tasks.workunit.client.0.smithi136.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-15T21:35:35.228 INFO:tasks.workunit.client.0.smithi136.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T21:35:35.229 INFO:tasks.workunit.client.0.smithi136.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52602 2024-09-15T21:35:35.229 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.52602/ceph-mon.a.asok 2024-09-15T21:35:35.229 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-15T21:35:35.229 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.52602/ceph-mon.a.asok config get fsid 2024-09-15T21:35:35.309 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2024-09-15T21:35:35.309 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-15T21:35:35.309 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-15T21:35:35.309 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2024-09-15T21:35:35.309 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2024-09-15T21:35:35.309 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-15T21:35:35.309 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-15T21:35:35.309 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-15T21:35:35.310 INFO:tasks.workunit.client.0.smithi136.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-15T21:35:35.310 INFO:tasks.workunit.client.0.smithi136.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T21:35:35.310 INFO:tasks.workunit.client.0.smithi136.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52602 2024-09-15T21:35:35.310 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.52602/ceph-mon.a.asok 2024-09-15T21:35:35.310 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-15T21:35:35.310 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.52602/ceph-mon.a.asok config get mon_host 2024-09-15T21:35:35.387 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:921: list_missing_erasure_coded: run_mgr td/osd-scrub-repair x 2024-09-15T21:35:35.388 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: run_mgr: local dir=td/osd-scrub-repair 2024-09-15T21:35:35.388 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: run_mgr: shift 2024-09-15T21:35:35.388 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: run_mgr: local id=x 2024-09-15T21:35:35.388 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: run_mgr: shift 2024-09-15T21:35:35.388 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:548: run_mgr: local data=td/osd-scrub-repair/x 2024-09-15T21:35:35.388 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:550: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-15T21:35:36.202 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: get_asok_path 2024-09-15T21:35:36.202 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-15T21:35:36.202 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-15T21:35:36.202 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-15T21:35:36.202 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T21:35:36.202 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52602 2024-09-15T21:35:36.202 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.52602/$cluster-$name.asok' 2024-09-15T21:35:36.203 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-15T21:35:36.204 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-scrub-repair/x '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.52602/$cluster-$name.asok' --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-15T21:35:36.231 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:922: list_missing_erasure_coded: seq 0 2 2024-09-15T21:35:36.235 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:922: list_missing_erasure_coded: for id in $(seq 0 2) 2024-09-15T21:35:36.235 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:923: list_missing_erasure_coded: run_osd td/osd-scrub-repair 0 2024-09-15T21:35:36.235 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-scrub-repair 2024-09-15T21:35:36.235 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-15T21:35:36.235 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=0 2024-09-15T21:35:36.235 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-15T21:35:36.235 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-scrub-repair/0 2024-09-15T21:35:36.235 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=ce4892c7-6c6d-4b4f-8e6f-07a3e11133e5 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-15T21:35:36.235 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T21:35:36.236 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T21:35:36.236 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T21:35:36.236 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-09-15T21:35:36.236 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-09-15T21:35:36.236 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-15T21:35:36.236 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-15T21:35:36.236 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-15T21:35:36.236 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-15T21:35:36.236 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-15T21:35:36.236 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-15T21:35:36.236 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-15T21:35:36.236 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T21:35:36.237 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52602 2024-09-15T21:35:36.237 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.52602/$cluster-$name.asok' 2024-09-15T21:35:36.237 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.52602/$cluster-$name.asok' 2024-09-15T21:35:36.237 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-15T21:35:36.237 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-15T21:35:36.237 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-15T21:35:36.237 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-15T21:35:36.237 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-15T21:35:36.237 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T21:35:36.237 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T21:35:36.237 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T21:35:36.237 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T21:35:36.238 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-15T21:35:36.238 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-15T21:35:36.238 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-scrub-repair/0 2024-09-15T21:35:36.238 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-15T21:35:36.238 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=8f9fc7c0-96fe-4bba-9896-4ad11fa25dc1 2024-09-15T21:35:36.238 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd0 8f9fc7c0-96fe-4bba-9896-4ad11fa25dc1' 2024-09-15T21:35:36.239 INFO:tasks.workunit.client.0.smithi136.stdout:add osd0 8f9fc7c0-96fe-4bba-9896-4ad11fa25dc1 2024-09-15T21:35:36.239 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-15T21:35:36.256 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQAoU+dmRbMbDxAAFYLxi09GTMnfWGBtx3KKLQ== 2024-09-15T21:35:36.256 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQAoU+dmRbMbDxAAFYLxi09GTMnfWGBtx3KKLQ=="}' 2024-09-15T21:35:36.256 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 8f9fc7c0-96fe-4bba-9896-4ad11fa25dc1 -i td/osd-scrub-repair/0/new.json 2024-09-15T21:35:36.475 INFO:tasks.workunit.client.0.smithi136.stdout:0 2024-09-15T21:35:36.485 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-scrub-repair/0/new.json 2024-09-15T21:35:36.486 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 0 --fsid=ce4892c7-6c6d-4b4f-8e6f-07a3e11133e5 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.52602/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAoU+dmRbMbDxAAFYLxi09GTMnfWGBtx3KKLQ== --osd-uuid 8f9fc7c0-96fe-4bba-9896-4ad11fa25dc1 2024-09-15T21:35:36.515 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:35:36.513+0000 7fb0a7fcc640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:35:36.526 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:35:36.524+0000 7fb0a7fcc640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:35:36.527 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:35:36.525+0000 7fb0a7fcc640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:35:36.527 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:35:36.525+0000 7fb0a7fcc640 -1 bdev(0x5626fdd37c00 td/osd-scrub-repair/0/block) open stat got: (1) Operation not permitted 2024-09-15T21:35:36.527 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:35:36.525+0000 7fb0a7fcc640 -1 bluestore(td/osd-scrub-repair/0) _read_fsid unparsable uuid 2024-09-15T21:35:42.299 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-scrub-repair/0/keyring 2024-09-15T21:35:42.300 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-15T21:35:42.300 INFO:tasks.workunit.client.0.smithi136.stdout:adding osd0 key to auth repository 2024-09-15T21:35:42.301 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd0 key to auth repository 2024-09-15T21:35:42.301 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-scrub-repair/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-15T21:35:42.686 INFO:tasks.workunit.client.0.smithi136.stdout:start osd.0 2024-09-15T21:35:42.687 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.0 2024-09-15T21:35:42.687 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 0 --fsid=ce4892c7-6c6d-4b4f-8e6f-07a3e11133e5 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.52602/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T21:35:42.687 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-15T21:35:42.687 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-15T21:35:42.687 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-15T21:35:42.714 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:35:42.711+0000 7f0e5262e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:35:42.716 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:35:42.714+0000 7f0e5262e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:35:42.717 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:35:42.715+0000 7f0e5262e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:35:42.964 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 0 2024-09-15T21:35:42.964 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-15T21:35:42.964 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2024-09-15T21:35:42.964 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-15T21:35:42.964 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-15T21:35:42.964 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:35:42.965 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-15T21:35:42.965 INFO:tasks.workunit.client.0.smithi136.stdout:0 2024-09-15T21:35:42.965 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:35:42.965 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:35:43.242 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:35:44.076 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:35:44.072+0000 7f0e5262e640 -1 Falling back to public interface 2024-09-15T21:35:44.242 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:35:44.242 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:35:44.243 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-15T21:35:44.243 INFO:tasks.workunit.client.0.smithi136.stdout:1 2024-09-15T21:35:44.243 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:35:44.243 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:35:44.528 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:35:45.530 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:35:45.530 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:35:45.530 INFO:tasks.workunit.client.0.smithi136.stdout:2 2024-09-15T21:35:45.530 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-15T21:35:45.531 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:35:45.531 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:35:45.815 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:35:46.717 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:35:46.714+0000 7f0e5262e640 -1 osd.0 0 log_to_monitors true 2024-09-15T21:35:46.817 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:35:46.817 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:35:46.818 INFO:tasks.workunit.client.0.smithi136.stdout:3 2024-09-15T21:35:46.818 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-15T21:35:46.818 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:35:46.818 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:35:47.180 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:35:48.182 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:35:48.182 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:35:48.182 INFO:tasks.workunit.client.0.smithi136.stdout:4 2024-09-15T21:35:48.183 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-15T21:35:48.183 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:35:48.183 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:35:48.518 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:35:49.520 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:35:49.520 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:35:49.520 INFO:tasks.workunit.client.0.smithi136.stdout:5 2024-09-15T21:35:49.520 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-15T21:35:49.521 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:35:49.521 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:35:49.809 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:35:50.810 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:35:50.811 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:35:50.811 INFO:tasks.workunit.client.0.smithi136.stdout:6 2024-09-15T21:35:50.811 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-15T21:35:50.811 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:35:50.811 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:35:51.101 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:35:52.102 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:35:52.102 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:35:52.102 INFO:tasks.workunit.client.0.smithi136.stdout:7 2024-09-15T21:35:52.103 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-15T21:35:52.103 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:35:52.103 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:35:52.483 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:35:53.485 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:35:53.485 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:35:53.485 INFO:tasks.workunit.client.0.smithi136.stdout:8 2024-09-15T21:35:53.486 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-15T21:35:53.486 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:35:53.486 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:35:53.772 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:35:54.773 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:35:54.773 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:35:54.774 INFO:tasks.workunit.client.0.smithi136.stdout:9 2024-09-15T21:35:54.774 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-15T21:35:54.774 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:35:54.774 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:35:55.061 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:35:56.063 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:35:56.063 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:35:56.063 INFO:tasks.workunit.client.0.smithi136.stdout:10 2024-09-15T21:35:56.064 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-15T21:35:56.064 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:35:56.064 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:35:56.352 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:35:57.353 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:35:57.353 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:35:57.353 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-15T21:35:57.354 INFO:tasks.workunit.client.0.smithi136.stdout:11 2024-09-15T21:35:57.354 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:35:57.354 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:35:57.705 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:35:58.706 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:35:58.707 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:35:58.707 INFO:tasks.workunit.client.0.smithi136.stdout:12 2024-09-15T21:35:58.708 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-15T21:35:58.708 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:35:58.708 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:35:59.132 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:36:00.133 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:36:00.133 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:36:00.134 INFO:tasks.workunit.client.0.smithi136.stdout:13 2024-09-15T21:36:00.134 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-15T21:36:00.134 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:36:00.134 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:36:00.877 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:36:01.878 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:36:01.878 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:36:01.879 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-15T21:36:01.879 INFO:tasks.workunit.client.0.smithi136.stdout:14 2024-09-15T21:36:01.879 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:36:01.879 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:36:02.322 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:36:03.323 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:36:03.324 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:36:03.324 INFO:tasks.workunit.client.0.smithi136.stdout:15 2024-09-15T21:36:03.324 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-15T21:36:03.325 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:36:03.325 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:36:03.697 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:36:04.698 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:36:04.698 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:36:04.698 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-15T21:36:04.699 INFO:tasks.workunit.client.0.smithi136.stdout:16 2024-09-15T21:36:04.699 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:36:04.699 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:36:05.491 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:36:06.492 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:36:06.492 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:36:06.493 INFO:tasks.workunit.client.0.smithi136.stdout:17 2024-09-15T21:36:06.493 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-15T21:36:06.493 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:36:06.493 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:36:07.166 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:36:08.168 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:36:08.168 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:36:08.168 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 18 2024-09-15T21:36:08.168 INFO:tasks.workunit.client.0.smithi136.stdout:18 2024-09-15T21:36:08.169 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:36:08.169 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:36:09.047 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:36:10.049 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:36:10.049 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:36:10.049 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 19 2024-09-15T21:36:10.049 INFO:tasks.workunit.client.0.smithi136.stdout:19 2024-09-15T21:36:10.050 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:36:10.050 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:36:10.420 INFO:tasks.workunit.client.0.smithi136.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/3231336434,v1:127.0.0.1:6803/3231336434] [v2:127.0.0.1:6804/3231336434,v1:127.0.0.1:6805/3231336434] exists,up 8f9fc7c0-96fe-4bba-9896-4ad11fa25dc1 2024-09-15T21:36:10.424 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-15T21:36:10.424 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-15T21:36:10.424 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-15T21:36:10.424 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:922: list_missing_erasure_coded: for id in $(seq 0 2) 2024-09-15T21:36:10.424 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:923: list_missing_erasure_coded: run_osd td/osd-scrub-repair 1 2024-09-15T21:36:10.424 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-scrub-repair 2024-09-15T21:36:10.424 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-15T21:36:10.424 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=1 2024-09-15T21:36:10.424 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-15T21:36:10.424 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-scrub-repair/1 2024-09-15T21:36:10.425 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=ce4892c7-6c6d-4b4f-8e6f-07a3e11133e5 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-15T21:36:10.425 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T21:36:10.425 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T21:36:10.425 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T21:36:10.425 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-09-15T21:36:10.428 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-09-15T21:36:10.429 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-15T21:36:10.429 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-15T21:36:10.429 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-15T21:36:10.429 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-15T21:36:10.429 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-15T21:36:10.429 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-15T21:36:10.429 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-15T21:36:10.429 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T21:36:10.429 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52602 2024-09-15T21:36:10.429 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.52602/$cluster-$name.asok' 2024-09-15T21:36:10.429 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.52602/$cluster-$name.asok' 2024-09-15T21:36:10.430 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-15T21:36:10.430 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-15T21:36:10.430 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-15T21:36:10.430 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-15T21:36:10.430 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-15T21:36:10.430 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T21:36:10.430 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T21:36:10.430 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T21:36:10.430 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T21:36:10.430 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-15T21:36:10.430 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-15T21:36:10.430 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-scrub-repair/1 2024-09-15T21:36:10.430 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-15T21:36:10.431 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=be3362ad-a83f-47e7-9fcb-4faae0dc2193 2024-09-15T21:36:10.431 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd1 be3362ad-a83f-47e7-9fcb-4faae0dc2193' 2024-09-15T21:36:10.431 INFO:tasks.workunit.client.0.smithi136.stdout:add osd1 be3362ad-a83f-47e7-9fcb-4faae0dc2193 2024-09-15T21:36:10.432 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-15T21:36:10.443 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQBKU+dmKgBPGhAAfpXw8wqcrpHrMpXEvUJbqg== 2024-09-15T21:36:10.443 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQBKU+dmKgBPGhAAfpXw8wqcrpHrMpXEvUJbqg=="}' 2024-09-15T21:36:10.443 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new be3362ad-a83f-47e7-9fcb-4faae0dc2193 -i td/osd-scrub-repair/1/new.json 2024-09-15T21:36:10.800 INFO:tasks.workunit.client.0.smithi136.stdout:1 2024-09-15T21:36:10.811 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-scrub-repair/1/new.json 2024-09-15T21:36:10.812 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 1 --fsid=ce4892c7-6c6d-4b4f-8e6f-07a3e11133e5 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.52602/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBKU+dmKgBPGhAAfpXw8wqcrpHrMpXEvUJbqg== --osd-uuid be3362ad-a83f-47e7-9fcb-4faae0dc2193 2024-09-15T21:36:10.842 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:36:10.839+0000 7fe5eb351640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:36:10.885 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:36:10.883+0000 7fe5eb351640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:36:10.886 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:36:10.884+0000 7fe5eb351640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:36:10.886 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:36:10.884+0000 7fe5eb351640 -1 bdev(0x55a1e93adc00 td/osd-scrub-repair/1/block) open stat got: (1) Operation not permitted 2024-09-15T21:36:10.886 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:36:10.884+0000 7fe5eb351640 -1 bluestore(td/osd-scrub-repair/1) _read_fsid unparsable uuid 2024-09-15T21:36:16.538 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-scrub-repair/1/keyring 2024-09-15T21:36:16.538 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-15T21:36:16.539 INFO:tasks.workunit.client.0.smithi136.stdout:adding osd1 key to auth repository 2024-09-15T21:36:16.539 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd1 key to auth repository 2024-09-15T21:36:16.539 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-scrub-repair/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-15T21:36:16.905 INFO:tasks.workunit.client.0.smithi136.stdout:start osd.1 2024-09-15T21:36:16.905 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.1 2024-09-15T21:36:16.906 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 1 --fsid=ce4892c7-6c6d-4b4f-8e6f-07a3e11133e5 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.52602/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T21:36:16.906 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-15T21:36:16.906 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-15T21:36:16.906 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-15T21:36:16.932 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:36:16.930+0000 7f5f83e19640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:36:16.934 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:36:16.932+0000 7f5f83e19640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:36:16.935 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:36:16.933+0000 7f5f83e19640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:36:17.180 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 1 2024-09-15T21:36:17.180 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-15T21:36:17.180 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2024-09-15T21:36:17.180 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-15T21:36:17.181 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-15T21:36:17.181 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:36:17.181 INFO:tasks.workunit.client.0.smithi136.stdout:0 2024-09-15T21:36:17.181 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-15T21:36:17.181 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:36:17.181 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T21:36:17.454 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:36:18.299 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:36:18.297+0000 7f5f83e19640 -1 Falling back to public interface 2024-09-15T21:36:18.455 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:36:18.455 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:36:18.456 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-15T21:36:18.456 INFO:tasks.workunit.client.0.smithi136.stdout:1 2024-09-15T21:36:18.456 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:36:18.456 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T21:36:18.738 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:36:19.739 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:36:19.740 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:36:19.740 INFO:tasks.workunit.client.0.smithi136.stdout:2 2024-09-15T21:36:19.740 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-15T21:36:19.741 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:36:19.741 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T21:36:20.025 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:36:20.974 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:36:20.972+0000 7f5f83e19640 -1 osd.1 0 log_to_monitors true 2024-09-15T21:36:21.026 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:36:21.026 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:36:21.026 INFO:tasks.workunit.client.0.smithi136.stdout:3 2024-09-15T21:36:21.027 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-15T21:36:21.027 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:36:21.027 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T21:36:21.501 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:36:22.502 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:36:22.502 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:36:22.502 INFO:tasks.workunit.client.0.smithi136.stdout:4 2024-09-15T21:36:22.503 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-15T21:36:22.503 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:36:22.503 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T21:36:22.973 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:36:23.974 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:36:23.975 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:36:23.975 INFO:tasks.workunit.client.0.smithi136.stdout:5 2024-09-15T21:36:23.975 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-15T21:36:23.975 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:36:23.976 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T21:36:24.303 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:36:25.304 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:36:25.304 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:36:25.304 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-15T21:36:25.305 INFO:tasks.workunit.client.0.smithi136.stdout:6 2024-09-15T21:36:25.305 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:36:25.305 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T21:36:25.617 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:36:26.618 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:36:26.618 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:36:26.618 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-15T21:36:26.618 INFO:tasks.workunit.client.0.smithi136.stdout:7 2024-09-15T21:36:26.619 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:36:26.619 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T21:36:26.959 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:36:27.960 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:36:27.960 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:36:27.961 INFO:tasks.workunit.client.0.smithi136.stdout:8 2024-09-15T21:36:27.961 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-15T21:36:27.961 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:36:27.961 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T21:36:28.294 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:36:29.295 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:36:29.296 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:36:29.296 INFO:tasks.workunit.client.0.smithi136.stdout:9 2024-09-15T21:36:29.297 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-15T21:36:29.297 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:36:29.298 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T21:36:29.647 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:36:30.648 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:36:30.648 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:36:30.648 INFO:tasks.workunit.client.0.smithi136.stdout:10 2024-09-15T21:36:30.649 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-15T21:36:30.649 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:36:30.649 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T21:36:31.058 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:36:32.059 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:36:32.059 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:36:32.060 INFO:tasks.workunit.client.0.smithi136.stdout:11 2024-09-15T21:36:32.060 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-15T21:36:32.060 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:36:32.060 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T21:36:32.349 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:36:33.350 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:36:33.350 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:36:33.351 INFO:tasks.workunit.client.0.smithi136.stdout:12 2024-09-15T21:36:33.351 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-15T21:36:33.351 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:36:33.351 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T21:36:33.636 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:36:34.638 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:36:34.638 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:36:34.638 INFO:tasks.workunit.client.0.smithi136.stdout:13 2024-09-15T21:36:34.639 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-15T21:36:34.639 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:36:34.639 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T21:36:35.401 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:36:36.402 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:36:36.402 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:36:36.402 INFO:tasks.workunit.client.0.smithi136.stdout:14 2024-09-15T21:36:36.403 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-15T21:36:36.403 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:36:36.403 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T21:36:37.278 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:36:38.279 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:36:38.280 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:36:38.280 INFO:tasks.workunit.client.0.smithi136.stdout:15 2024-09-15T21:36:38.281 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-15T21:36:38.281 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:36:38.281 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T21:36:38.721 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:36:39.723 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:36:39.723 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:36:39.723 INFO:tasks.workunit.client.0.smithi136.stdout:16 2024-09-15T21:36:39.724 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-15T21:36:39.724 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:36:39.724 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T21:36:40.258 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:36:41.257 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:36:41.257 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:36:41.257 INFO:tasks.workunit.client.0.smithi136.stdout:17 2024-09-15T21:36:41.258 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-15T21:36:41.258 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:36:41.258 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T21:36:41.568 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:36:42.569 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:36:42.569 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:36:42.570 INFO:tasks.workunit.client.0.smithi136.stdout:18 2024-09-15T21:36:42.570 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 18 2024-09-15T21:36:42.570 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:36:42.570 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T21:36:43.346 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:36:44.349 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:36:44.349 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:36:44.349 INFO:tasks.workunit.client.0.smithi136.stdout:19 2024-09-15T21:36:44.350 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 19 2024-09-15T21:36:44.350 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:36:44.350 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T21:36:44.644 INFO:tasks.workunit.client.0.smithi136.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/1806463188,v1:127.0.0.1:6811/1806463188] [v2:127.0.0.1:6812/1806463188,v1:127.0.0.1:6813/1806463188] exists,up be3362ad-a83f-47e7-9fcb-4faae0dc2193 2024-09-15T21:36:44.647 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-15T21:36:44.647 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-15T21:36:44.648 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-15T21:36:44.648 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:922: list_missing_erasure_coded: for id in $(seq 0 2) 2024-09-15T21:36:44.648 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:923: list_missing_erasure_coded: run_osd td/osd-scrub-repair 2 2024-09-15T21:36:44.648 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-scrub-repair 2024-09-15T21:36:44.648 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-15T21:36:44.648 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=2 2024-09-15T21:36:44.648 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-15T21:36:44.648 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-scrub-repair/2 2024-09-15T21:36:44.648 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=ce4892c7-6c6d-4b4f-8e6f-07a3e11133e5 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-15T21:36:44.648 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T21:36:44.648 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T21:36:44.648 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T21:36:44.649 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/2' 2024-09-15T21:36:44.649 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/2/journal' 2024-09-15T21:36:44.649 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-15T21:36:44.649 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-15T21:36:44.649 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-15T21:36:44.649 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-15T21:36:44.649 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-15T21:36:44.649 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-15T21:36:44.649 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-15T21:36:44.649 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T21:36:44.649 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52602 2024-09-15T21:36:44.649 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.52602/$cluster-$name.asok' 2024-09-15T21:36:44.649 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.52602/$cluster-$name.asok' 2024-09-15T21:36:44.649 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-15T21:36:44.650 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-15T21:36:44.650 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-15T21:36:44.650 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-15T21:36:44.650 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-15T21:36:44.650 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T21:36:44.650 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T21:36:44.650 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T21:36:44.650 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T21:36:44.650 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-15T21:36:44.650 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-15T21:36:44.650 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-scrub-repair/2 2024-09-15T21:36:44.650 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-15T21:36:44.651 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=7cd8e98a-b1b9-4812-b1c1-25f756bcad7f 2024-09-15T21:36:44.651 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd2 7cd8e98a-b1b9-4812-b1c1-25f756bcad7f' 2024-09-15T21:36:44.651 INFO:tasks.workunit.client.0.smithi136.stdout:add osd2 7cd8e98a-b1b9-4812-b1c1-25f756bcad7f 2024-09-15T21:36:44.651 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-15T21:36:44.667 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQBsU+dm1D+iJxAAi5Wv0OlnZZRBdYRXzZfyMQ== 2024-09-15T21:36:44.667 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQBsU+dm1D+iJxAAi5Wv0OlnZZRBdYRXzZfyMQ=="}' 2024-09-15T21:36:44.667 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 7cd8e98a-b1b9-4812-b1c1-25f756bcad7f -i td/osd-scrub-repair/2/new.json 2024-09-15T21:36:45.141 INFO:tasks.workunit.client.0.smithi136.stdout:2 2024-09-15T21:36:45.152 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-scrub-repair/2/new.json 2024-09-15T21:36:45.153 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 2 --fsid=ce4892c7-6c6d-4b4f-8e6f-07a3e11133e5 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.52602/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBsU+dm1D+iJxAAi5Wv0OlnZZRBdYRXzZfyMQ== --osd-uuid 7cd8e98a-b1b9-4812-b1c1-25f756bcad7f 2024-09-15T21:36:45.182 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:36:45.180+0000 7efca740d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:36:45.242 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:36:45.240+0000 7efca740d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:36:45.243 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:36:45.241+0000 7efca740d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:36:45.244 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:36:45.242+0000 7efca740d640 -1 bdev(0x55dd12fbbc00 td/osd-scrub-repair/2/block) open stat got: (1) Operation not permitted 2024-09-15T21:36:45.244 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:36:45.242+0000 7efca740d640 -1 bluestore(td/osd-scrub-repair/2) _read_fsid unparsable uuid 2024-09-15T21:36:51.664 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-scrub-repair/2/keyring 2024-09-15T21:36:51.664 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-15T21:36:51.665 INFO:tasks.workunit.client.0.smithi136.stdout:adding osd2 key to auth repository 2024-09-15T21:36:51.665 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd2 key to auth repository 2024-09-15T21:36:51.665 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-scrub-repair/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-15T21:36:52.023 INFO:tasks.workunit.client.0.smithi136.stdout:start osd.2 2024-09-15T21:36:52.023 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.2 2024-09-15T21:36:52.023 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 2 --fsid=ce4892c7-6c6d-4b4f-8e6f-07a3e11133e5 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.52602/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T21:36:52.023 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-15T21:36:52.024 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-15T21:36:52.024 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-15T21:36:52.050 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:36:52.048+0000 7f182b4dc640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:36:52.052 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:36:52.050+0000 7f182b4dc640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:36:52.053 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:36:52.051+0000 7f182b4dc640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:36:52.302 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 2 2024-09-15T21:36:52.302 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-15T21:36:52.302 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=2 2024-09-15T21:36:52.302 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-15T21:36:52.302 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-15T21:36:52.303 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:36:52.303 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-15T21:36:52.303 INFO:tasks.workunit.client.0.smithi136.stdout:0 2024-09-15T21:36:52.303 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:36:52.303 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T21:36:52.587 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:36:53.418 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:36:53.416+0000 7f182b4dc640 -1 Falling back to public interface 2024-09-15T21:36:53.588 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:36:53.588 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:36:53.588 INFO:tasks.workunit.client.0.smithi136.stdout:1 2024-09-15T21:36:53.589 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-15T21:36:53.589 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:36:53.589 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T21:36:53.875 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:36:54.877 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:36:54.877 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:36:54.877 INFO:tasks.workunit.client.0.smithi136.stdout:2 2024-09-15T21:36:54.878 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-15T21:36:54.878 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:36:54.878 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T21:36:55.165 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:36:56.167 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:36:56.167 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:36:56.167 INFO:tasks.workunit.client.0.smithi136.stdout:3 2024-09-15T21:36:56.167 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-15T21:36:56.168 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:36:56.168 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T21:36:56.449 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:36:56.447+0000 7f182b4dc640 -1 osd.2 0 log_to_monitors true 2024-09-15T21:36:56.502 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:36:57.503 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:36:57.503 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:36:57.504 INFO:tasks.workunit.client.0.smithi136.stdout:4 2024-09-15T21:36:57.504 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-15T21:36:57.504 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:36:57.504 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T21:36:57.792 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:36:58.794 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:36:58.794 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:36:58.794 INFO:tasks.workunit.client.0.smithi136.stdout:5 2024-09-15T21:36:58.795 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-15T21:36:58.795 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:36:58.795 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T21:36:59.111 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:37:00.112 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:37:00.112 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:37:00.112 INFO:tasks.workunit.client.0.smithi136.stdout:6 2024-09-15T21:37:00.113 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-15T21:37:00.113 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:37:00.113 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T21:37:00.403 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:37:01.405 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:37:01.405 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:37:01.405 INFO:tasks.workunit.client.0.smithi136.stdout:7 2024-09-15T21:37:01.405 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-15T21:37:01.405 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:37:01.406 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T21:37:01.692 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:37:02.693 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:37:02.693 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:37:02.694 INFO:tasks.workunit.client.0.smithi136.stdout:8 2024-09-15T21:37:02.694 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-15T21:37:02.694 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:37:02.694 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T21:37:02.983 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:37:03.984 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:37:03.985 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:37:03.985 INFO:tasks.workunit.client.0.smithi136.stdout:9 2024-09-15T21:37:03.985 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-15T21:37:03.985 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:37:03.986 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T21:37:04.392 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:37:05.394 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:37:05.394 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:37:05.394 INFO:tasks.workunit.client.0.smithi136.stdout:10 2024-09-15T21:37:05.394 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-15T21:37:05.395 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:37:05.395 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T21:37:05.680 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:37:06.682 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:37:06.682 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:37:06.682 INFO:tasks.workunit.client.0.smithi136.stdout:11 2024-09-15T21:37:06.683 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-15T21:37:06.683 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:37:06.683 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T21:37:06.986 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:37:07.987 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:37:07.988 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:37:07.988 INFO:tasks.workunit.client.0.smithi136.stdout:12 2024-09-15T21:37:07.988 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-15T21:37:07.988 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:37:07.989 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T21:37:08.777 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:37:09.778 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:37:09.778 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:37:09.778 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-15T21:37:09.778 INFO:tasks.workunit.client.0.smithi136.stdout:13 2024-09-15T21:37:09.779 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:37:09.779 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T21:37:10.318 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:37:11.320 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:37:11.320 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:37:11.320 INFO:tasks.workunit.client.0.smithi136.stdout:14 2024-09-15T21:37:11.321 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-15T21:37:11.321 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:37:11.321 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T21:37:11.970 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:37:12.972 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:37:12.972 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:37:12.972 INFO:tasks.workunit.client.0.smithi136.stdout:15 2024-09-15T21:37:12.973 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-15T21:37:12.973 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:37:12.973 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T21:37:13.360 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:37:14.361 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:37:14.362 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:37:14.362 INFO:tasks.workunit.client.0.smithi136.stdout:16 2024-09-15T21:37:14.362 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-15T21:37:14.362 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:37:14.363 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T21:37:15.472 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:37:16.474 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:37:16.474 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:37:16.475 INFO:tasks.workunit.client.0.smithi136.stdout:17 2024-09-15T21:37:16.475 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-15T21:37:16.475 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:37:16.475 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T21:37:17.392 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:37:18.394 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:37:18.394 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:37:18.394 INFO:tasks.workunit.client.0.smithi136.stdout:18 2024-09-15T21:37:18.395 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 18 2024-09-15T21:37:18.395 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:37:18.395 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T21:37:18.885 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:37:19.886 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:37:19.887 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:37:19.887 INFO:tasks.workunit.client.0.smithi136.stdout:19 2024-09-15T21:37:19.887 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 19 2024-09-15T21:37:19.888 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:37:19.888 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T21:37:20.163 INFO:tasks.workunit.client.0.smithi136.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/922002709,v1:127.0.0.1:6819/922002709] [v2:127.0.0.1:6820/922002709,v1:127.0.0.1:6821/922002709] exists,up 7cd8e98a-b1b9-4812-b1c1-25f756bcad7f 2024-09-15T21:37:20.164 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-15T21:37:20.164 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-15T21:37:20.164 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-15T21:37:20.165 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:925: list_missing_erasure_coded: create_rbd_pool 2024-09-15T21:37:20.165 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:526: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2024-09-15T21:37:20.453 INFO:tasks.workunit.client.0.smithi136.stderr:pool 'rbd' does not exist 2024-09-15T21:37:20.465 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:527: create_rbd_pool: create_pool rbd 4 2024-09-15T21:37:20.465 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create rbd 4 2024-09-15T21:37:20.880 INFO:tasks.workunit.client.0.smithi136.stderr:pool 'rbd' created 2024-09-15T21:37:20.891 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2024-09-15T21:37:21.893 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:528: create_rbd_pool: rbd pool init rbd 2024-09-15T21:37:23.494 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:926: list_missing_erasure_coded: wait_for_clean 2024-09-15T21:37:23.495 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-15T21:37:23.495 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-15T21:37:23.495 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-15T21:37:23.496 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-15T21:37:23.496 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-15T21:37:23.496 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-15T21:37:23.496 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-15T21:37:23.496 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-15T21:37:23.496 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-15T21:37:23.601 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-15T21:37:23.601 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-15T21:37:23.601 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-15T21:37:23.602 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-15T21:37:23.602 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-15T21:37:23.602 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-15T21:37:23.877 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-15T21:37:23.877 INFO:tasks.workunit.client.0.smithi136.stderr:1 2024-09-15T21:37:23.877 INFO:tasks.workunit.client.0.smithi136.stderr:2' 2024-09-15T21:37:23.877 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-15T21:37:23.877 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T21:37:23.877 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-15T21:37:23.980 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836500 2024-09-15T21:37:23.980 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836500 2024-09-15T21:37:23.980 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836500' 2024-09-15T21:37:23.980 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T21:37:23.980 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-15T21:37:24.084 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=42949672973 2024-09-15T21:37:24.084 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 42949672973 2024-09-15T21:37:24.084 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836500 1-42949672973' 2024-09-15T21:37:24.084 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T21:37:24.085 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-15T21:37:24.188 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=64424509446 2024-09-15T21:37:24.188 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 64424509446 2024-09-15T21:37:24.188 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836500 1-42949672973 2-64424509446' 2024-09-15T21:37:24.188 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T21:37:24.189 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836500 2024-09-15T21:37:24.189 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T21:37:24.189 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-15T21:37:24.190 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836500 2024-09-15T21:37:24.190 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T21:37:24.191 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836500 2024-09-15T21:37:24.191 INFO:tasks.workunit.client.0.smithi136.stdout:waiting osd.0 seq 21474836500 2024-09-15T21:37:24.192 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836500' 2024-09-15T21:37:24.192 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T21:37:24.471 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836500 -lt 21474836500 2024-09-15T21:37:24.471 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T21:37:24.471 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-42949672973 2024-09-15T21:37:24.471 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T21:37:24.472 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-15T21:37:24.473 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-42949672973 2024-09-15T21:37:24.473 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T21:37:24.474 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=42949672973 2024-09-15T21:37:24.474 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 42949672973' 2024-09-15T21:37:24.475 INFO:tasks.workunit.client.0.smithi136.stdout:waiting osd.1 seq 42949672973 2024-09-15T21:37:24.475 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-15T21:37:24.752 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 42949672973 -lt 42949672973 2024-09-15T21:37:24.752 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T21:37:24.752 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-64424509446 2024-09-15T21:37:24.752 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T21:37:24.753 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-15T21:37:24.754 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-64424509446 2024-09-15T21:37:24.754 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T21:37:24.755 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=64424509446 2024-09-15T21:37:24.755 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 64424509446' 2024-09-15T21:37:24.755 INFO:tasks.workunit.client.0.smithi136.stdout:waiting osd.2 seq 64424509446 2024-09-15T21:37:24.755 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-15T21:37:25.031 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 64424509445 -lt 64424509446 2024-09-15T21:37:25.031 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-15T21:37:26.032 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-15T21:37:26.032 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-15T21:37:26.343 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 64424509447 -lt 64424509446 2024-09-15T21:37:26.344 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-15T21:37:26.344 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T21:37:26.344 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T21:37:26.694 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 4 == 0 2024-09-15T21:37:26.695 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-15T21:37:26.695 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-15T21:37:26.695 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-15T21:37:26.695 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T21:37:26.695 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T21:37:26.695 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T21:37:26.695 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T21:37:26.976 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=4 2024-09-15T21:37:26.976 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-15T21:37:26.977 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T21:37:26.977 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T21:37:27.348 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 = 4 2024-09-15T21:37:27.349 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-15T21:37:27.349 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-15T21:37:27.349 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:928: list_missing_erasure_coded: create_ec_pool ecpool false k=2 m=1 2024-09-15T21:37:27.349 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2443: create_ec_pool: local pool_name=ecpool 2024-09-15T21:37:27.349 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2444: create_ec_pool: shift 2024-09-15T21:37:27.349 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2445: create_ec_pool: local allow_overwrites=false 2024-09-15T21:37:27.349 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2446: create_ec_pool: shift 2024-09-15T21:37:27.349 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2448: create_ec_pool: ceph osd erasure-code-profile set myprofile crush-failure-domain=osd k=2 m=1 2024-09-15T21:37:27.741 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2450: create_ec_pool: create_pool ecpool 1 1 erasure myprofile 2024-09-15T21:37:27.741 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create ecpool 1 1 erasure myprofile 2024-09-15T21:37:28.432 INFO:tasks.workunit.client.0.smithi136.stderr:pool 'ecpool' created 2024-09-15T21:37:28.445 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2024-09-15T21:37:29.446 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2452: create_ec_pool: '[' false = true ']' 2024-09-15T21:37:29.447 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2456: create_ec_pool: wait_for_clean 2024-09-15T21:37:29.447 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-15T21:37:29.447 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-15T21:37:29.447 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-15T21:37:29.448 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-15T21:37:29.448 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-15T21:37:29.448 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-15T21:37:29.448 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-15T21:37:29.448 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-15T21:37:29.448 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-15T21:37:29.554 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-15T21:37:29.554 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-15T21:37:29.554 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-15T21:37:29.555 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-15T21:37:29.555 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-15T21:37:29.555 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-15T21:37:29.860 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-15T21:37:29.860 INFO:tasks.workunit.client.0.smithi136.stderr:1 2024-09-15T21:37:29.860 INFO:tasks.workunit.client.0.smithi136.stderr:2' 2024-09-15T21:37:29.860 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-15T21:37:29.860 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T21:37:29.860 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-15T21:37:29.964 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836502 2024-09-15T21:37:29.964 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836502 2024-09-15T21:37:29.964 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836502' 2024-09-15T21:37:29.965 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T21:37:29.965 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-15T21:37:30.068 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=42949672976 2024-09-15T21:37:30.068 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 42949672976 2024-09-15T21:37:30.068 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836502 1-42949672976' 2024-09-15T21:37:30.068 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T21:37:30.068 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-15T21:37:30.170 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=64424509449 2024-09-15T21:37:30.170 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 64424509449 2024-09-15T21:37:30.170 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836502 1-42949672976 2-64424509449' 2024-09-15T21:37:30.170 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T21:37:30.171 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836502 2024-09-15T21:37:30.171 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T21:37:30.171 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-15T21:37:30.172 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836502 2024-09-15T21:37:30.172 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T21:37:30.173 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836502 2024-09-15T21:37:30.173 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836502' 2024-09-15T21:37:30.173 INFO:tasks.workunit.client.0.smithi136.stdout:waiting osd.0 seq 21474836502 2024-09-15T21:37:30.174 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T21:37:30.457 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836502 -lt 21474836502 2024-09-15T21:37:30.457 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T21:37:30.458 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-42949672976 2024-09-15T21:37:30.458 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T21:37:30.458 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-15T21:37:30.459 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-42949672976 2024-09-15T21:37:30.459 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T21:37:30.460 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=42949672976 2024-09-15T21:37:30.460 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 42949672976' 2024-09-15T21:37:30.461 INFO:tasks.workunit.client.0.smithi136.stdout:waiting osd.1 seq 42949672976 2024-09-15T21:37:30.461 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-15T21:37:30.736 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 42949672976 -lt 42949672976 2024-09-15T21:37:30.736 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T21:37:30.737 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-64424509449 2024-09-15T21:37:30.737 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T21:37:30.738 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-15T21:37:30.739 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-64424509449 2024-09-15T21:37:30.739 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T21:37:30.740 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=64424509449 2024-09-15T21:37:30.740 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 64424509449' 2024-09-15T21:37:30.740 INFO:tasks.workunit.client.0.smithi136.stdout:waiting osd.2 seq 64424509449 2024-09-15T21:37:30.740 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-15T21:37:31.023 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 64424509449 -lt 64424509449 2024-09-15T21:37:31.023 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-15T21:37:31.024 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T21:37:31.024 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T21:37:31.371 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 5 == 0 2024-09-15T21:37:31.371 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-15T21:37:31.371 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-15T21:37:31.371 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-15T21:37:31.371 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T21:37:31.371 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T21:37:31.371 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T21:37:31.372 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T21:37:31.642 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=5 2024-09-15T21:37:31.643 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-15T21:37:31.643 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T21:37:31.643 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T21:37:31.986 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 = 5 2024-09-15T21:37:31.987 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-15T21:37:31.987 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-15T21:37:31.987 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2457: create_ec_pool: return 0 2024-09-15T21:37:31.987 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:931: list_missing_erasure_coded: add_something td/osd-scrub-repair ecpool MOBJ0 2024-09-15T21:37:31.987 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-09-15T21:37:31.987 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=ecpool 2024-09-15T21:37:31.987 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=MOBJ0 2024-09-15T21:37:31.987 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-09-15T21:37:31.987 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-09-15T21:37:31.987 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-09-15T21:37:32.460 INFO:tasks.workunit.client.0.smithi136.stderr:noscrub is set 2024-09-15T21:37:32.472 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-09-15T21:37:32.988 INFO:tasks.workunit.client.0.smithi136.stderr:nodeep-scrub is set 2024-09-15T21:37:33.000 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-09-15T21:37:33.000 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-09-15T21:37:33.000 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool ecpool put MOBJ0 td/osd-scrub-repair/ORIGINAL 2024-09-15T21:37:33.259 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:932: list_missing_erasure_coded: get_osds ecpool MOBJ0 2024-09-15T21:37:33.259 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1012: get_osds: local poolname=ecpool 2024-09-15T21:37:33.260 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1013: get_osds: local objectname=MOBJ0 2024-09-15T21:37:33.260 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: ceph --format json osd map ecpool MOBJ0 2024-09-15T21:37:33.260 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: jq '.acting | .[]' 2024-09-15T21:37:33.543 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: local 'osds=2 2024-09-15T21:37:33.543 INFO:tasks.workunit.client.0.smithi136.stderr:1 2024-09-15T21:37:33.543 INFO:tasks.workunit.client.0.smithi136.stderr:0' 2024-09-15T21:37:33.543 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1018: get_osds: echo 2 1 0 2024-09-15T21:37:33.543 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:932: list_missing_erasure_coded: osds0=('2' '1' '0') 2024-09-15T21:37:33.544 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:932: list_missing_erasure_coded: local -a osds0 2024-09-15T21:37:33.544 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:935: list_missing_erasure_coded: add_something td/osd-scrub-repair ecpool MOBJ1 2024-09-15T21:37:33.544 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-09-15T21:37:33.544 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=ecpool 2024-09-15T21:37:33.545 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=MOBJ1 2024-09-15T21:37:33.545 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-09-15T21:37:33.545 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-09-15T21:37:33.545 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-09-15T21:37:33.925 INFO:tasks.workunit.client.0.smithi136.stderr:noscrub is set 2024-09-15T21:37:33.937 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-09-15T21:37:34.359 INFO:tasks.workunit.client.0.smithi136.stderr:nodeep-scrub is set 2024-09-15T21:37:34.372 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-09-15T21:37:34.372 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-09-15T21:37:34.372 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool ecpool put MOBJ1 td/osd-scrub-repair/ORIGINAL 2024-09-15T21:37:34.647 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:936: list_missing_erasure_coded: get_osds ecpool MOBJ1 2024-09-15T21:37:34.648 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1012: get_osds: local poolname=ecpool 2024-09-15T21:37:34.648 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1013: get_osds: local objectname=MOBJ1 2024-09-15T21:37:34.648 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: ceph --format json osd map ecpool MOBJ1 2024-09-15T21:37:34.648 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: jq '.acting | .[]' 2024-09-15T21:37:34.956 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: local 'osds=2 2024-09-15T21:37:34.956 INFO:tasks.workunit.client.0.smithi136.stderr:1 2024-09-15T21:37:34.956 INFO:tasks.workunit.client.0.smithi136.stderr:0' 2024-09-15T21:37:34.956 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1018: get_osds: echo 2 1 0 2024-09-15T21:37:34.956 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:936: list_missing_erasure_coded: osds1=('2' '1' '0') 2024-09-15T21:37:34.956 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:936: list_missing_erasure_coded: local -a osds1 2024-09-15T21:37:34.957 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:939: list_missing_erasure_coded: seq 0 2 2024-09-15T21:37:34.957 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:939: list_missing_erasure_coded: for id in $(seq 0 2) 2024-09-15T21:37:34.957 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:940: list_missing_erasure_coded: kill_daemons td/osd-scrub-repair TERM osd.0 2024-09-15T21:37:34.958 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-15T21:37:34.958 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-15T21:37:34.958 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-15T21:37:34.958 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-15T21:37:34.958 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-15T21:37:35.265 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-15T21:37:35.265 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:939: list_missing_erasure_coded: for id in $(seq 0 2) 2024-09-15T21:37:35.265 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:940: list_missing_erasure_coded: kill_daemons td/osd-scrub-repair TERM osd.1 2024-09-15T21:37:35.266 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-15T21:37:35.266 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-15T21:37:35.266 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-15T21:37:35.266 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-15T21:37:35.266 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-15T21:37:36.574 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-15T21:37:36.574 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:939: list_missing_erasure_coded: for id in $(seq 0 2) 2024-09-15T21:37:36.574 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:940: list_missing_erasure_coded: kill_daemons td/osd-scrub-repair TERM osd.2 2024-09-15T21:37:36.575 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-15T21:37:36.575 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-15T21:37:36.575 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-15T21:37:36.575 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-15T21:37:36.575 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-15T21:37:36.883 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-15T21:37:36.884 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:943: list_missing_erasure_coded: id=2 2024-09-15T21:37:36.884 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:944: list_missing_erasure_coded: ceph-objectstore-tool --data-path td/osd-scrub-repair/2 MOBJ0 remove 2024-09-15T21:37:38.234 INFO:tasks.workunit.client.0.smithi136.stdout:remove 0#2:a9030505:::MOBJ0:head# 2024-09-15T21:37:38.750 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:946: list_missing_erasure_coded: id=1 2024-09-15T21:37:38.750 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:947: list_missing_erasure_coded: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 MOBJ0 remove 2024-09-15T21:37:40.138 INFO:tasks.workunit.client.0.smithi136.stdout:remove 1#2:a9030505:::MOBJ0:head# 2024-09-15T21:37:40.653 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:950: list_missing_erasure_coded: id=1 2024-09-15T21:37:40.653 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:951: list_missing_erasure_coded: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 MOBJ1 remove 2024-09-15T21:37:41.800 INFO:tasks.workunit.client.0.smithi136.stdout:remove 1#2:8dd16d55:::MOBJ1:head# 2024-09-15T21:37:42.315 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:953: list_missing_erasure_coded: id=0 2024-09-15T21:37:42.316 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:954: list_missing_erasure_coded: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 MOBJ1 remove 2024-09-15T21:37:43.695 INFO:tasks.workunit.client.0.smithi136.stdout:remove 2#2:8dd16d55:::MOBJ1:head# 2024-09-15T21:37:44.210 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:957: list_missing_erasure_coded: seq 0 2 2024-09-15T21:37:44.214 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:957: list_missing_erasure_coded: for id in $(seq 0 2) 2024-09-15T21:37:44.214 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:958: list_missing_erasure_coded: activate_osd td/osd-scrub-repair 0 2024-09-15T21:37:44.214 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2024-09-15T21:37:44.214 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-15T21:37:44.214 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=0 2024-09-15T21:37:44.214 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-15T21:37:44.215 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-09-15T21:37:44.215 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=ce4892c7-6c6d-4b4f-8e6f-07a3e11133e5 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-15T21:37:44.215 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T21:37:44.215 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T21:37:44.215 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T21:37:44.215 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-09-15T21:37:44.215 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-09-15T21:37:44.215 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-15T21:37:44.215 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-15T21:37:44.215 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-15T21:37:44.215 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-15T21:37:44.215 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-15T21:37:44.215 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-15T21:37:44.215 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-15T21:37:44.216 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T21:37:44.216 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52602 2024-09-15T21:37:44.216 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.52602/$cluster-$name.asok' 2024-09-15T21:37:44.216 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.52602/$cluster-$name.asok' 2024-09-15T21:37:44.216 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-15T21:37:44.216 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-15T21:37:44.216 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-15T21:37:44.216 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T21:37:44.216 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T21:37:44.216 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T21:37:44.216 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T21:37:44.216 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-15T21:37:44.216 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-15T21:37:44.217 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-09-15T21:37:44.217 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.0 2024-09-15T21:37:44.218 INFO:tasks.workunit.client.0.smithi136.stderr:start osd.0 2024-09-15T21:37:44.218 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 0 --fsid=ce4892c7-6c6d-4b4f-8e6f-07a3e11133e5 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.52602/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T21:37:44.218 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-09-15T21:37:44.218 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 0 = 0 ']' 2024-09-15T21:37:44.218 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-15T21:37:44.218 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-15T21:37:44.218 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-15T21:37:44.245 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:37:44.243+0000 7fc85cc31640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:37:44.262 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:37:44.260+0000 7fc85cc31640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:37:44.263 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:37:44.261+0000 7fc85cc31640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:37:44.504 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 0 2024-09-15T21:37:44.504 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-15T21:37:44.504 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2024-09-15T21:37:44.504 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-15T21:37:44.504 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-15T21:37:44.504 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:37:44.504 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-15T21:37:44.504 INFO:tasks.workunit.client.0.smithi136.stderr:0 2024-09-15T21:37:44.504 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:37:44.504 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:37:44.782 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:37:45.783 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:37:45.783 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:37:45.783 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-15T21:37:45.783 INFO:tasks.workunit.client.0.smithi136.stderr:1 2024-09-15T21:37:45.784 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:37:45.784 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:37:45.873 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:37:45.871+0000 7fc85cc31640 -1 Falling back to public interface 2024-09-15T21:37:46.069 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:37:47.071 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:37:47.071 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:37:47.071 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-15T21:37:47.071 INFO:tasks.workunit.client.0.smithi136.stderr:2 2024-09-15T21:37:47.071 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:37:47.071 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:37:47.360 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:37:48.361 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:37:48.361 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:37:48.361 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-15T21:37:48.361 INFO:tasks.workunit.client.0.smithi136.stderr:3 2024-09-15T21:37:48.362 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:37:48.362 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:37:48.437 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:37:48.435+0000 7fc85cc31640 -1 osd.0 32 log_to_monitors true 2024-09-15T21:37:48.677 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:37:49.679 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:37:49.679 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:37:49.679 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-15T21:37:49.679 INFO:tasks.workunit.client.0.smithi136.stderr:4 2024-09-15T21:37:49.679 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:37:49.679 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:37:49.965 INFO:tasks.workunit.client.0.smithi136.stderr:osd.0 up in weight 1 up_from 38 up_thru 38 down_at 33 last_clean_interval [5,32) [v2:127.0.0.1:6802/245881690,v1:127.0.0.1:6803/245881690] [v2:127.0.0.1:6804/245881690,v1:127.0.0.1:6805/245881690] exists,up 8f9fc7c0-96fe-4bba-9896-4ad11fa25dc1 2024-09-15T21:37:49.965 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-15T21:37:49.965 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-15T21:37:49.965 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-15T21:37:49.965 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:957: list_missing_erasure_coded: for id in $(seq 0 2) 2024-09-15T21:37:49.965 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:958: list_missing_erasure_coded: activate_osd td/osd-scrub-repair 1 2024-09-15T21:37:49.965 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2024-09-15T21:37:49.966 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-15T21:37:49.966 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=1 2024-09-15T21:37:49.966 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-15T21:37:49.966 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-09-15T21:37:49.966 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=ce4892c7-6c6d-4b4f-8e6f-07a3e11133e5 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-15T21:37:49.966 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T21:37:49.966 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T21:37:49.966 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T21:37:49.966 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-09-15T21:37:49.966 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-09-15T21:37:49.967 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-15T21:37:49.967 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-15T21:37:49.967 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-15T21:37:49.967 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-15T21:37:49.967 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-15T21:37:49.967 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-15T21:37:49.967 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-15T21:37:49.967 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T21:37:49.967 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52602 2024-09-15T21:37:49.967 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.52602/$cluster-$name.asok' 2024-09-15T21:37:49.967 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.52602/$cluster-$name.asok' 2024-09-15T21:37:49.967 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-15T21:37:49.967 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-15T21:37:49.968 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-15T21:37:49.968 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T21:37:49.968 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T21:37:49.968 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T21:37:49.968 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T21:37:49.968 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-15T21:37:49.968 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-15T21:37:49.968 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-09-15T21:37:49.968 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.1 2024-09-15T21:37:49.968 INFO:tasks.workunit.client.0.smithi136.stderr:start osd.1 2024-09-15T21:37:49.969 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 1 --fsid=ce4892c7-6c6d-4b4f-8e6f-07a3e11133e5 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.52602/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T21:37:49.969 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-09-15T21:37:49.969 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 1 = 1 ']' 2024-09-15T21:37:49.969 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-15T21:37:49.969 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-15T21:37:49.969 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-15T21:37:49.991 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:37:49.989+0000 7ff623064640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:37:49.993 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:37:49.991+0000 7ff623064640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:37:49.994 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:37:49.992+0000 7ff623064640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:37:50.278 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 1 2024-09-15T21:37:50.278 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-15T21:37:50.278 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2024-09-15T21:37:50.278 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-15T21:37:50.278 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-15T21:37:50.278 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:37:50.278 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-15T21:37:50.278 INFO:tasks.workunit.client.0.smithi136.stderr:0 2024-09-15T21:37:50.278 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:37:50.278 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T21:37:50.558 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:37:51.340 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:37:51.338+0000 7ff623064640 -1 Falling back to public interface 2024-09-15T21:37:51.561 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:37:51.561 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:37:51.561 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-15T21:37:51.561 INFO:tasks.workunit.client.0.smithi136.stderr:1 2024-09-15T21:37:51.561 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:37:51.561 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T21:37:51.849 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:37:52.851 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:37:52.851 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:37:52.852 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-15T21:37:52.852 INFO:tasks.workunit.client.0.smithi136.stderr:2 2024-09-15T21:37:52.852 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:37:52.852 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T21:37:53.139 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:37:54.140 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:37:54.140 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:37:54.140 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-15T21:37:54.141 INFO:tasks.workunit.client.0.smithi136.stderr:3 2024-09-15T21:37:54.141 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:37:54.141 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T21:37:54.426 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:37:54.646 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:37:54.644+0000 7ff623064640 -1 osd.1 33 log_to_monitors true 2024-09-15T21:37:55.428 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:37:55.428 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:37:55.428 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-15T21:37:55.428 INFO:tasks.workunit.client.0.smithi136.stderr:4 2024-09-15T21:37:55.428 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:37:55.428 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T21:37:55.823 INFO:tasks.workunit.client.0.smithi136.stderr:osd.1 up in weight 1 up_from 41 up_thru 41 down_at 34 last_clean_interval [10,33) [v2:127.0.0.1:6810/1488270013,v1:127.0.0.1:6811/1488270013] [v2:127.0.0.1:6812/1488270013,v1:127.0.0.1:6813/1488270013] exists,up be3362ad-a83f-47e7-9fcb-4faae0dc2193 2024-09-15T21:37:55.823 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-15T21:37:55.823 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-15T21:37:55.823 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-15T21:37:55.823 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:957: list_missing_erasure_coded: for id in $(seq 0 2) 2024-09-15T21:37:55.823 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:958: list_missing_erasure_coded: activate_osd td/osd-scrub-repair 2 2024-09-15T21:37:55.824 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2024-09-15T21:37:55.824 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-15T21:37:55.824 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=2 2024-09-15T21:37:55.824 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-15T21:37:55.824 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/2 2024-09-15T21:37:55.824 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=ce4892c7-6c6d-4b4f-8e6f-07a3e11133e5 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-15T21:37:55.824 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T21:37:55.824 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T21:37:55.824 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T21:37:55.824 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/2' 2024-09-15T21:37:55.824 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/2/journal' 2024-09-15T21:37:55.824 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-15T21:37:55.824 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-15T21:37:55.824 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-15T21:37:55.825 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-15T21:37:55.825 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-15T21:37:55.825 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-15T21:37:55.825 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-15T21:37:55.825 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T21:37:55.825 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52602 2024-09-15T21:37:55.825 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.52602/$cluster-$name.asok' 2024-09-15T21:37:55.825 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.52602/$cluster-$name.asok' 2024-09-15T21:37:55.825 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-15T21:37:55.825 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-15T21:37:55.825 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-15T21:37:55.825 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T21:37:55.825 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T21:37:55.825 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T21:37:55.826 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T21:37:55.826 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-15T21:37:55.826 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-15T21:37:55.826 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/2 2024-09-15T21:37:55.826 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.2 2024-09-15T21:37:55.826 INFO:tasks.workunit.client.0.smithi136.stderr:start osd.2 2024-09-15T21:37:55.826 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 2 --fsid=ce4892c7-6c6d-4b4f-8e6f-07a3e11133e5 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.52602/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T21:37:55.826 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/2/whoami 2024-09-15T21:37:55.827 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 2 = 2 ']' 2024-09-15T21:37:55.827 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-15T21:37:55.827 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-15T21:37:55.827 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-15T21:37:55.850 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:37:55.848+0000 7f92b4030640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:37:55.959 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:37:55.957+0000 7f92b4030640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:37:55.960 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:37:55.958+0000 7f92b4030640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:37:56.162 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 2 2024-09-15T21:37:56.163 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-15T21:37:56.163 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=2 2024-09-15T21:37:56.163 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-15T21:37:56.163 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-15T21:37:56.163 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:37:56.163 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-15T21:37:56.163 INFO:tasks.workunit.client.0.smithi136.stderr:0 2024-09-15T21:37:56.163 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:37:56.163 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T21:37:56.447 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:37:57.321 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:37:57.319+0000 7f92b4030640 -1 Falling back to public interface 2024-09-15T21:37:57.449 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:37:57.449 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:37:57.449 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-15T21:37:57.449 INFO:tasks.workunit.client.0.smithi136.stderr:1 2024-09-15T21:37:57.449 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:37:57.449 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T21:37:57.738 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:37:58.740 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:37:58.740 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:37:58.740 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-15T21:37:58.740 INFO:tasks.workunit.client.0.smithi136.stderr:2 2024-09-15T21:37:58.740 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:37:58.740 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T21:37:59.019 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:38:00.021 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:38:00.021 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:38:00.021 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-15T21:38:00.022 INFO:tasks.workunit.client.0.smithi136.stderr:3 2024-09-15T21:38:00.022 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:38:00.022 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T21:38:00.026 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:38:00.024+0000 7f92b4030640 -1 osd.2 35 log_to_monitors true 2024-09-15T21:38:00.328 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:38:01.329 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:38:01.329 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:38:01.329 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-15T21:38:01.330 INFO:tasks.workunit.client.0.smithi136.stderr:4 2024-09-15T21:38:01.330 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:38:01.330 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T21:38:01.619 INFO:tasks.workunit.client.0.smithi136.stderr:osd.2 up in weight 1 up_from 45 up_thru 45 down_at 36 last_clean_interval [15,35) [v2:127.0.0.1:6818/2989779627,v1:127.0.0.1:6819/2989779627] [v2:127.0.0.1:6820/2989779627,v1:127.0.0.1:6821/2989779627] exists,up 7cd8e98a-b1b9-4812-b1c1-25f756bcad7f 2024-09-15T21:38:01.620 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-15T21:38:01.620 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-15T21:38:01.620 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-15T21:38:01.620 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:960: list_missing_erasure_coded: create_rbd_pool 2024-09-15T21:38:01.620 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:526: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2024-09-15T21:38:01.991 INFO:tasks.workunit.client.0.smithi136.stderr:pool 'rbd' removed 2024-09-15T21:38:02.003 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:527: create_rbd_pool: create_pool rbd 4 2024-09-15T21:38:02.003 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create rbd 4 2024-09-15T21:38:02.502 INFO:tasks.workunit.client.0.smithi136.stderr:pool 'rbd' created 2024-09-15T21:38:02.513 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2024-09-15T21:38:03.515 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:528: create_rbd_pool: rbd pool init rbd 2024-09-15T21:38:05.370 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:961: list_missing_erasure_coded: wait_for_clean 2024-09-15T21:38:05.370 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-15T21:38:05.370 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-15T21:38:05.370 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-15T21:38:05.370 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-15T21:38:05.371 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-15T21:38:05.371 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-15T21:38:05.371 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-15T21:38:05.371 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-15T21:38:05.371 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-15T21:38:05.476 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-15T21:38:05.477 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-15T21:38:05.477 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-15T21:38:05.477 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-15T21:38:05.477 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-15T21:38:05.477 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-15T21:38:05.757 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-15T21:38:05.758 INFO:tasks.workunit.client.0.smithi136.stderr:1 2024-09-15T21:38:05.758 INFO:tasks.workunit.client.0.smithi136.stderr:2' 2024-09-15T21:38:05.758 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-15T21:38:05.758 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T21:38:05.758 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-15T21:38:05.861 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=163208757253 2024-09-15T21:38:05.861 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 163208757253 2024-09-15T21:38:05.862 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-163208757253' 2024-09-15T21:38:05.862 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T21:38:05.862 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-15T21:38:05.965 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=176093659140 2024-09-15T21:38:05.966 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 176093659140 2024-09-15T21:38:05.966 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-163208757253 1-176093659140' 2024-09-15T21:38:05.966 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T21:38:05.966 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-15T21:38:06.070 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=193273528323 2024-09-15T21:38:06.070 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 193273528323 2024-09-15T21:38:06.070 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-163208757253 1-176093659140 2-193273528323' 2024-09-15T21:38:06.070 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T21:38:06.071 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-163208757253 2024-09-15T21:38:06.071 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T21:38:06.072 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-15T21:38:06.072 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-163208757253 2024-09-15T21:38:06.073 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T21:38:06.074 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=163208757253 2024-09-15T21:38:06.074 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 163208757253' 2024-09-15T21:38:06.074 INFO:tasks.workunit.client.0.smithi136.stdout:waiting osd.0 seq 163208757253 2024-09-15T21:38:06.074 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T21:38:06.353 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 163208757253 -lt 163208757253 2024-09-15T21:38:06.353 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T21:38:06.353 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-176093659140 2024-09-15T21:38:06.353 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T21:38:06.354 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-15T21:38:06.355 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-176093659140 2024-09-15T21:38:06.355 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T21:38:06.356 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=176093659140 2024-09-15T21:38:06.356 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 176093659140' 2024-09-15T21:38:06.356 INFO:tasks.workunit.client.0.smithi136.stdout:waiting osd.1 seq 176093659140 2024-09-15T21:38:06.357 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-15T21:38:06.634 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 176093659140 -lt 176093659140 2024-09-15T21:38:06.634 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T21:38:06.635 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-193273528323 2024-09-15T21:38:06.635 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T21:38:06.636 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-15T21:38:06.636 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-193273528323 2024-09-15T21:38:06.636 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T21:38:06.637 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=193273528323 2024-09-15T21:38:06.637 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 193273528323' 2024-09-15T21:38:06.638 INFO:tasks.workunit.client.0.smithi136.stdout:waiting osd.2 seq 193273528323 2024-09-15T21:38:06.638 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-15T21:38:06.915 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 193273528323 -lt 193273528323 2024-09-15T21:38:06.915 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-15T21:38:06.915 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T21:38:06.915 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T21:38:07.264 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 5 == 0 2024-09-15T21:38:07.264 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-15T21:38:07.264 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-15T21:38:07.265 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-15T21:38:07.265 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T21:38:07.265 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T21:38:07.265 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T21:38:07.265 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T21:38:07.538 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=5 2024-09-15T21:38:07.539 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-15T21:38:07.539 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T21:38:07.539 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T21:38:07.885 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 = 5 2024-09-15T21:38:07.885 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-15T21:38:07.885 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-15T21:38:07.885 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:964: list_missing_erasure_coded: get_pg ecpool MOBJ0 2024-09-15T21:38:07.885 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1084: get_pg: local poolname=ecpool 2024-09-15T21:38:07.885 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1085: get_pg: local objectname=MOBJ0 2024-09-15T21:38:07.885 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1087: get_pg: ceph --format json osd map ecpool MOBJ0 2024-09-15T21:38:07.886 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1087: get_pg: jq -r .pgid 2024-09-15T21:38:08.164 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:964: list_missing_erasure_coded: local pg=2.0 2024-09-15T21:38:08.164 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:968: list_missing_erasure_coded: repair 2.0 2024-09-15T21:38:08.164 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1846: repair: local pgid=2.0 2024-09-15T21:38:08.164 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1847: repair: get_last_scrub_stamp 2.0 2024-09-15T21:38:08.164 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2024-09-15T21:38:08.164 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-15T21:38:08.165 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-15T21:38:08.165 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-09-15T21:38:08.437 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1847: repair: local last_scrub=2024-09-15T21:37:28.338753+0000 2024-09-15T21:38:08.438 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1848: repair: ceph pg repair 2.0 2024-09-15T21:38:08.698 INFO:tasks.workunit.client.0.smithi136.stderr:instructing pg 2.0s0 on osd.2 to repair 2024-09-15T21:38:08.710 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1849: repair: wait_for_scrub 2.0 2024-09-15T21:37:28.338753+0000 2024-09-15T21:38:08.710 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: local pgid=2.0 2024-09-15T21:38:08.710 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2018: wait_for_scrub: local last_scrub=2024-09-15T21:37:28.338753+0000 2024-09-15T21:38:08.710 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2019: wait_for_scrub: local sname=last_scrub_stamp 2024-09-15T21:38:08.710 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i=0 )) 2024-09-15T21:38:08.710 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-15T21:38:08.710 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-09-15T21:38:08.710 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2024-09-15T21:38:08.710 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-15T21:38:08.711 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-15T21:38:08.711 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-09-15T21:38:08.981 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-15T21:37:28.338753+0000 '>' 2024-09-15T21:37:28.338753+0000 2024-09-15T21:38:08.981 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-15T21:38:09.982 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-15T21:38:09.982 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-15T21:38:09.983 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-09-15T21:38:09.983 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2024-09-15T21:38:09.983 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-15T21:38:09.983 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-15T21:38:09.983 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-09-15T21:38:10.260 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-15T21:37:28.338753+0000 '>' 2024-09-15T21:37:28.338753+0000 2024-09-15T21:38:10.260 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-15T21:38:11.261 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-15T21:38:11.261 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-15T21:38:11.262 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-09-15T21:38:11.262 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2024-09-15T21:38:11.262 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-15T21:38:11.262 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-15T21:38:11.262 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-09-15T21:38:11.546 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-15T21:37:28.338753+0000 '>' 2024-09-15T21:37:28.338753+0000 2024-09-15T21:38:11.546 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-15T21:38:12.547 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-15T21:38:12.548 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-15T21:38:12.549 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-09-15T21:38:12.549 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2024-09-15T21:38:12.549 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-15T21:38:12.549 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-15T21:38:12.549 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-09-15T21:38:12.833 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-15T21:38:08.990247+0000 '>' 2024-09-15T21:37:28.338753+0000 2024-09-15T21:38:12.833 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2023: wait_for_scrub: return 0 2024-09-15T21:38:12.833 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1850: repair: sleep 2 2024-09-15T21:38:14.831 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:970: list_missing_erasure_coded: seq 0 120 2024-09-15T21:38:14.832 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:970: list_missing_erasure_coded: for i in $(seq 0 120) 2024-09-15T21:38:14.832 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:971: list_missing_erasure_coded: '[' 0 -lt 60 ']' 2024-09-15T21:38:14.833 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:972: list_missing_erasure_coded: ceph pg 2.0 list_unfound 2024-09-15T21:38:14.833 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:972: list_missing_erasure_coded: egrep 'MOBJ0|MOBJ1' 2024-09-15T21:38:14.834 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:972: list_missing_erasure_coded: wc -l 2024-09-15T21:38:14.950 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:972: list_missing_erasure_coded: matches=2 2024-09-15T21:38:14.950 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:973: list_missing_erasure_coded: '[' 2 -eq 2 ']' 2024-09-15T21:38:14.950 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:973: list_missing_erasure_coded: break 2024-09-15T21:38:14.950 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:66: run: teardown td/osd-scrub-repair 2024-09-15T21:38:14.950 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/osd-scrub-repair 2024-09-15T21:38:14.950 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-15T21:38:14.950 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/osd-scrub-repair KILL 2024-09-15T21:38:14.951 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-15T21:38:14.951 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-15T21:38:14.951 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-15T21:38:14.951 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-15T21:38:14.951 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-15T21:38:15.059 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-15T21:38:15.059 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-15T21:38:15.060 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-15T21:38:15.060 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-15T21:38:15.061 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-15T21:38:15.062 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-15T21:38:15.062 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-15T21:38:15.063 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T21:38:15.063 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-15T21:38:15.063 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-15T21:38:15.063 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T21:38:15.064 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-15T21:38:15.065 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-15T21:38:15.065 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/osd-scrub-repair 2024-09-15T21:38:15.083 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-15T21:38:15.083 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T21:38:15.083 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52602 2024-09-15T21:38:15.083 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.52602 2024-09-15T21:38:15.085 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-15T21:38:15.085 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-15T21:38:15.085 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:63: run: for func in $funcs 2024-09-15T21:38:15.085 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:64: run: setup td/osd-scrub-repair 2024-09-15T21:38:15.085 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/osd-scrub-repair 2024-09-15T21:38:15.085 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/osd-scrub-repair 2024-09-15T21:38:15.085 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/osd-scrub-repair 2024-09-15T21:38:15.085 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-15T21:38:15.085 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/osd-scrub-repair KILL 2024-09-15T21:38:15.085 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-15T21:38:15.085 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-15T21:38:15.086 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-15T21:38:15.086 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-15T21:38:15.086 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-15T21:38:15.086 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-15T21:38:15.087 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-15T21:38:15.088 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-15T21:38:15.088 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-15T21:38:15.089 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-15T21:38:15.089 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-15T21:38:15.089 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-15T21:38:15.090 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T21:38:15.091 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-15T21:38:15.091 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-15T21:38:15.091 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T21:38:15.092 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-15T21:38:15.093 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-15T21:38:15.093 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/osd-scrub-repair 2024-09-15T21:38:15.094 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-15T21:38:15.094 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T21:38:15.094 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52602 2024-09-15T21:38:15.094 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.52602 2024-09-15T21:38:15.095 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-15T21:38:15.095 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-15T21:38:15.095 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/osd-scrub-repair 2024-09-15T21:38:15.097 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2024-09-15T21:38:15.097 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T21:38:15.097 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52602 2024-09-15T21:38:15.097 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.52602 2024-09-15T21:38:15.099 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2024-09-15T21:38:15.099 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 4096 -le 1024 ']' 2024-09-15T21:38:15.100 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2024-09-15T21:38:15.100 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/osd-scrub-repair 1' TERM HUP INT 2024-09-15T21:38:15.100 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:65: run: TEST_list_missing_erasure_coded_overwrites td/osd-scrub-repair 2024-09-15T21:38:15.100 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:982: TEST_list_missing_erasure_coded_overwrites: '[' true = true ']' 2024-09-15T21:38:15.100 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:983: TEST_list_missing_erasure_coded_overwrites: list_missing_erasure_coded td/osd-scrub-repair true 2024-09-15T21:38:15.100 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:916: list_missing_erasure_coded: local dir=td/osd-scrub-repair 2024-09-15T21:38:15.100 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:917: list_missing_erasure_coded: local allow_overwrites=true 2024-09-15T21:38:15.100 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:918: list_missing_erasure_coded: local poolname=ecpool 2024-09-15T21:38:15.100 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:920: list_missing_erasure_coded: run_mon td/osd-scrub-repair a 2024-09-15T21:38:15.100 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/osd-scrub-repair 2024-09-15T21:38:15.100 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2024-09-15T21:38:15.100 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2024-09-15T21:38:15.101 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2024-09-15T21:38:15.101 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/osd-scrub-repair/a 2024-09-15T21:38:15.101 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-scrub-repair/a --run-dir=td/osd-scrub-repair 2024-09-15T21:38:15.576 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2024-09-15T21:38:15.576 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-15T21:38:15.576 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-15T21:38:15.576 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-15T21:38:15.576 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T21:38:15.576 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52602 2024-09-15T21:38:15.577 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.52602/$cluster-$name.asok' 2024-09-15T21:38:15.577 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-scrub-repair/a '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.52602/$cluster-$name.asok' --mon-cluster-log-file=td/osd-scrub-repair/log --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-09-15T21:38:15.793 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2024-09-15T21:38:15.794 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2024-09-15T21:38:15.794 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-15T21:38:15.795 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-15T21:38:15.795 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2024-09-15T21:38:15.795 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2024-09-15T21:38:15.796 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-15T21:38:15.796 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-15T21:38:15.796 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-15T21:38:15.796 INFO:tasks.workunit.client.0.smithi136.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-15T21:38:15.796 INFO:tasks.workunit.client.0.smithi136.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T21:38:15.796 INFO:tasks.workunit.client.0.smithi136.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52602 2024-09-15T21:38:15.796 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.52602/ceph-mon.a.asok 2024-09-15T21:38:15.796 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-15T21:38:15.797 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.52602/ceph-mon.a.asok config get fsid 2024-09-15T21:38:15.877 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2024-09-15T21:38:15.878 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-15T21:38:15.878 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-15T21:38:15.878 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2024-09-15T21:38:15.878 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2024-09-15T21:38:15.878 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-15T21:38:15.878 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-15T21:38:15.878 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-15T21:38:15.878 INFO:tasks.workunit.client.0.smithi136.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-15T21:38:15.878 INFO:tasks.workunit.client.0.smithi136.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T21:38:15.878 INFO:tasks.workunit.client.0.smithi136.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52602 2024-09-15T21:38:15.878 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.52602/ceph-mon.a.asok 2024-09-15T21:38:15.879 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-15T21:38:15.879 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.52602/ceph-mon.a.asok config get mon_host 2024-09-15T21:38:15.957 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:921: list_missing_erasure_coded: run_mgr td/osd-scrub-repair x 2024-09-15T21:38:15.957 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: run_mgr: local dir=td/osd-scrub-repair 2024-09-15T21:38:15.957 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: run_mgr: shift 2024-09-15T21:38:15.957 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: run_mgr: local id=x 2024-09-15T21:38:15.957 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: run_mgr: shift 2024-09-15T21:38:15.957 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:548: run_mgr: local data=td/osd-scrub-repair/x 2024-09-15T21:38:15.957 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:550: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-15T21:38:16.770 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: get_asok_path 2024-09-15T21:38:16.770 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-15T21:38:16.770 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-15T21:38:16.771 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-15T21:38:16.771 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T21:38:16.771 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52602 2024-09-15T21:38:16.771 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.52602/$cluster-$name.asok' 2024-09-15T21:38:16.771 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-15T21:38:16.772 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-scrub-repair/x '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.52602/$cluster-$name.asok' --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-15T21:38:16.800 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:922: list_missing_erasure_coded: seq 0 2 2024-09-15T21:38:16.804 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:922: list_missing_erasure_coded: for id in $(seq 0 2) 2024-09-15T21:38:16.804 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:923: list_missing_erasure_coded: run_osd td/osd-scrub-repair 0 2024-09-15T21:38:16.804 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-scrub-repair 2024-09-15T21:38:16.804 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-15T21:38:16.804 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=0 2024-09-15T21:38:16.804 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-15T21:38:16.805 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-scrub-repair/0 2024-09-15T21:38:16.805 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=ce4892c7-6c6d-4b4f-8e6f-07a3e11133e5 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-15T21:38:16.805 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T21:38:16.805 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T21:38:16.805 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T21:38:16.805 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-09-15T21:38:16.805 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-09-15T21:38:16.805 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-15T21:38:16.805 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-15T21:38:16.805 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-15T21:38:16.805 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-15T21:38:16.805 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-15T21:38:16.805 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-15T21:38:16.806 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-15T21:38:16.806 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T21:38:16.806 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52602 2024-09-15T21:38:16.806 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.52602/$cluster-$name.asok' 2024-09-15T21:38:16.806 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.52602/$cluster-$name.asok' 2024-09-15T21:38:16.806 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-15T21:38:16.806 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-15T21:38:16.806 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-15T21:38:16.806 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-15T21:38:16.806 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-15T21:38:16.806 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T21:38:16.806 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T21:38:16.806 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T21:38:16.806 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T21:38:16.807 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-15T21:38:16.807 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-15T21:38:16.807 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-scrub-repair/0 2024-09-15T21:38:16.807 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-15T21:38:16.807 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=e1ef6b83-4aa7-4e52-ab1d-d8b427f7daaa 2024-09-15T21:38:16.807 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd0 e1ef6b83-4aa7-4e52-ab1d-d8b427f7daaa' 2024-09-15T21:38:16.808 INFO:tasks.workunit.client.0.smithi136.stdout:add osd0 e1ef6b83-4aa7-4e52-ab1d-d8b427f7daaa 2024-09-15T21:38:16.808 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-15T21:38:16.825 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQDIU+dm1OsEMRAASATh1vxEhEimE9QvpYYR9A== 2024-09-15T21:38:16.825 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQDIU+dm1OsEMRAASATh1vxEhEimE9QvpYYR9A=="}' 2024-09-15T21:38:16.825 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new e1ef6b83-4aa7-4e52-ab1d-d8b427f7daaa -i td/osd-scrub-repair/0/new.json 2024-09-15T21:38:17.025 INFO:tasks.workunit.client.0.smithi136.stdout:0 2024-09-15T21:38:17.035 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-scrub-repair/0/new.json 2024-09-15T21:38:17.036 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 0 --fsid=ce4892c7-6c6d-4b4f-8e6f-07a3e11133e5 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.52602/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDIU+dm1OsEMRAASATh1vxEhEimE9QvpYYR9A== --osd-uuid e1ef6b83-4aa7-4e52-ab1d-d8b427f7daaa 2024-09-15T21:38:17.066 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:38:17.063+0000 7fdbb67c1640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:38:17.076 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:38:17.073+0000 7fdbb67c1640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:38:17.077 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:38:17.074+0000 7fdbb67c1640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:38:17.078 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:38:17.075+0000 7fdbb67c1640 -1 bdev(0x55a6c1a45c00 td/osd-scrub-repair/0/block) open stat got: (1) Operation not permitted 2024-09-15T21:38:17.078 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:38:17.075+0000 7fdbb67c1640 -1 bluestore(td/osd-scrub-repair/0) _read_fsid unparsable uuid 2024-09-15T21:38:22.797 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-scrub-repair/0/keyring 2024-09-15T21:38:22.797 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-15T21:38:22.798 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd0 key to auth repository 2024-09-15T21:38:22.798 INFO:tasks.workunit.client.0.smithi136.stdout:adding osd0 key to auth repository 2024-09-15T21:38:22.798 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-scrub-repair/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-15T21:38:23.163 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.0 2024-09-15T21:38:23.163 INFO:tasks.workunit.client.0.smithi136.stdout:start osd.0 2024-09-15T21:38:23.164 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 0 --fsid=ce4892c7-6c6d-4b4f-8e6f-07a3e11133e5 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.52602/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T21:38:23.164 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-15T21:38:23.164 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-15T21:38:23.164 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-15T21:38:23.190 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:38:23.188+0000 7f4e3ce15640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:38:23.192 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:38:23.190+0000 7f4e3ce15640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:38:23.193 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:38:23.191+0000 7f4e3ce15640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:38:23.441 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 0 2024-09-15T21:38:23.441 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-15T21:38:23.442 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2024-09-15T21:38:23.442 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-15T21:38:23.442 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-15T21:38:23.442 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:38:23.442 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-15T21:38:23.442 INFO:tasks.workunit.client.0.smithi136.stdout:0 2024-09-15T21:38:23.442 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:38:23.443 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:38:23.715 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:38:24.560 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:38:24.558+0000 7f4e3ce15640 -1 Falling back to public interface 2024-09-15T21:38:24.716 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:38:24.716 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:38:24.717 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-15T21:38:24.717 INFO:tasks.workunit.client.0.smithi136.stdout:1 2024-09-15T21:38:24.717 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:38:24.717 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:38:24.995 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:38:25.997 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:38:25.997 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:38:25.997 INFO:tasks.workunit.client.0.smithi136.stdout:2 2024-09-15T21:38:25.997 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-15T21:38:25.997 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:38:25.998 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:38:26.289 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:38:27.289 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:38:27.290 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:38:27.290 INFO:tasks.workunit.client.0.smithi136.stdout:3 2024-09-15T21:38:27.290 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-15T21:38:27.291 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:38:27.291 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:38:27.437 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:38:27.435+0000 7f4e3ce15640 -1 osd.0 0 log_to_monitors true 2024-09-15T21:38:27.612 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:38:28.613 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:38:28.613 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:38:28.614 INFO:tasks.workunit.client.0.smithi136.stdout:4 2024-09-15T21:38:28.614 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-15T21:38:28.614 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:38:28.614 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:38:28.899 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:38:29.900 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:38:29.901 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:38:29.901 INFO:tasks.workunit.client.0.smithi136.stdout:5 2024-09-15T21:38:29.901 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-15T21:38:29.901 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:38:29.901 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:38:30.222 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:38:31.224 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:38:31.224 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:38:31.225 INFO:tasks.workunit.client.0.smithi136.stdout:6 2024-09-15T21:38:31.225 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-15T21:38:31.225 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:38:31.225 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:38:31.511 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:38:32.512 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:38:32.512 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:38:32.512 INFO:tasks.workunit.client.0.smithi136.stdout:7 2024-09-15T21:38:32.513 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-15T21:38:32.513 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:38:32.513 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:38:32.799 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:38:33.801 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:38:33.801 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:38:33.801 INFO:tasks.workunit.client.0.smithi136.stdout:8 2024-09-15T21:38:33.802 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-15T21:38:33.802 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:38:33.802 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:38:34.089 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:38:35.090 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:38:35.090 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:38:35.091 INFO:tasks.workunit.client.0.smithi136.stdout:9 2024-09-15T21:38:35.091 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-15T21:38:35.091 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:38:35.091 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:38:35.426 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:38:36.428 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:38:36.428 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:38:36.428 INFO:tasks.workunit.client.0.smithi136.stdout:10 2024-09-15T21:38:36.429 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-15T21:38:36.429 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:38:36.429 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:38:36.717 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:38:37.718 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:38:37.718 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:38:37.718 INFO:tasks.workunit.client.0.smithi136.stdout:11 2024-09-15T21:38:37.719 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-15T21:38:37.719 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:38:37.719 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:38:38.042 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:38:39.043 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:38:39.043 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:38:39.043 INFO:tasks.workunit.client.0.smithi136.stdout:12 2024-09-15T21:38:39.044 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-15T21:38:39.044 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:38:39.044 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:38:39.638 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:38:40.640 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:38:40.640 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:38:40.640 INFO:tasks.workunit.client.0.smithi136.stdout:13 2024-09-15T21:38:40.641 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-15T21:38:40.641 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:38:40.641 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:38:41.407 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:38:42.408 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:38:42.408 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:38:42.409 INFO:tasks.workunit.client.0.smithi136.stdout:14 2024-09-15T21:38:42.409 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-15T21:38:42.409 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:38:42.409 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:38:42.983 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:38:43.984 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:38:43.985 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:38:43.985 INFO:tasks.workunit.client.0.smithi136.stdout:15 2024-09-15T21:38:43.985 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-15T21:38:43.985 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:38:43.986 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:38:44.771 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:38:45.773 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:38:45.773 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:38:45.773 INFO:tasks.workunit.client.0.smithi136.stdout:16 2024-09-15T21:38:45.774 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-15T21:38:45.774 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:38:45.774 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:38:46.389 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:38:47.391 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:38:47.391 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:38:47.391 INFO:tasks.workunit.client.0.smithi136.stdout:17 2024-09-15T21:38:47.391 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-15T21:38:47.392 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:38:47.392 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:38:47.699 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:38:48.700 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:38:48.700 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:38:48.701 INFO:tasks.workunit.client.0.smithi136.stdout:18 2024-09-15T21:38:48.701 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 18 2024-09-15T21:38:48.701 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:38:48.701 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:38:49.366 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:38:50.368 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:38:50.368 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:38:50.368 INFO:tasks.workunit.client.0.smithi136.stdout:19 2024-09-15T21:38:50.369 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 19 2024-09-15T21:38:50.369 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:38:50.369 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:38:50.657 INFO:tasks.workunit.client.0.smithi136.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/3980678764,v1:127.0.0.1:6803/3980678764] [v2:127.0.0.1:6804/3980678764,v1:127.0.0.1:6805/3980678764] exists,up e1ef6b83-4aa7-4e52-ab1d-d8b427f7daaa 2024-09-15T21:38:50.661 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-15T21:38:50.662 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-15T21:38:50.662 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-15T21:38:50.662 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:922: list_missing_erasure_coded: for id in $(seq 0 2) 2024-09-15T21:38:50.662 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:923: list_missing_erasure_coded: run_osd td/osd-scrub-repair 1 2024-09-15T21:38:50.662 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-scrub-repair 2024-09-15T21:38:50.662 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-15T21:38:50.662 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=1 2024-09-15T21:38:50.662 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-15T21:38:50.662 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-scrub-repair/1 2024-09-15T21:38:50.662 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=ce4892c7-6c6d-4b4f-8e6f-07a3e11133e5 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-15T21:38:50.662 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T21:38:50.662 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T21:38:50.662 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T21:38:50.662 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-09-15T21:38:50.663 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-09-15T21:38:50.663 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-15T21:38:50.663 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-15T21:38:50.663 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-15T21:38:50.663 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-15T21:38:50.663 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-15T21:38:50.663 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-15T21:38:50.663 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-15T21:38:50.663 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T21:38:50.664 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52602 2024-09-15T21:38:50.664 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.52602/$cluster-$name.asok' 2024-09-15T21:38:50.664 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.52602/$cluster-$name.asok' 2024-09-15T21:38:50.664 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-15T21:38:50.664 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-15T21:38:50.664 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-15T21:38:50.664 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-15T21:38:50.664 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-15T21:38:50.664 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T21:38:50.664 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T21:38:50.664 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T21:38:50.664 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T21:38:50.664 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-15T21:38:50.665 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-15T21:38:50.665 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-scrub-repair/1 2024-09-15T21:38:50.665 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-15T21:38:50.665 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=3dfe3945-eee5-4128-81ad-96e70f53c913 2024-09-15T21:38:50.665 INFO:tasks.workunit.client.0.smithi136.stdout:add osd1 3dfe3945-eee5-4128-81ad-96e70f53c913 2024-09-15T21:38:50.665 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd1 3dfe3945-eee5-4128-81ad-96e70f53c913' 2024-09-15T21:38:50.665 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-15T21:38:50.680 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQDqU+dmB3JyKBAAE9BQCWcPyRh64rRh75aC4Q== 2024-09-15T21:38:50.681 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQDqU+dmB3JyKBAAE9BQCWcPyRh64rRh75aC4Q=="}' 2024-09-15T21:38:50.681 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 3dfe3945-eee5-4128-81ad-96e70f53c913 -i td/osd-scrub-repair/1/new.json 2024-09-15T21:38:51.105 INFO:tasks.workunit.client.0.smithi136.stdout:1 2024-09-15T21:38:51.117 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-scrub-repair/1/new.json 2024-09-15T21:38:51.118 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 1 --fsid=ce4892c7-6c6d-4b4f-8e6f-07a3e11133e5 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.52602/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDqU+dmB3JyKBAAE9BQCWcPyRh64rRh75aC4Q== --osd-uuid 3dfe3945-eee5-4128-81ad-96e70f53c913 2024-09-15T21:38:51.147 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:38:51.145+0000 7f2f28b3e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:38:51.190 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:38:51.188+0000 7f2f28b3e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:38:51.191 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:38:51.189+0000 7f2f28b3e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:38:51.191 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:38:51.189+0000 7f2f28b3e640 -1 bdev(0x55b4bbdc3c00 td/osd-scrub-repair/1/block) open stat got: (1) Operation not permitted 2024-09-15T21:38:51.191 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:38:51.189+0000 7f2f28b3e640 -1 bluestore(td/osd-scrub-repair/1) _read_fsid unparsable uuid 2024-09-15T21:38:57.085 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-scrub-repair/1/keyring 2024-09-15T21:38:57.085 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-15T21:38:57.086 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd1 key to auth repository 2024-09-15T21:38:57.086 INFO:tasks.workunit.client.0.smithi136.stdout:adding osd1 key to auth repository 2024-09-15T21:38:57.086 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-scrub-repair/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-15T21:38:57.451 INFO:tasks.workunit.client.0.smithi136.stdout:start osd.1 2024-09-15T21:38:57.452 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.1 2024-09-15T21:38:57.452 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 1 --fsid=ce4892c7-6c6d-4b4f-8e6f-07a3e11133e5 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.52602/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T21:38:57.452 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-15T21:38:57.452 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-15T21:38:57.452 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-15T21:38:57.478 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:38:57.476+0000 7ff26b2f1640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:38:57.480 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:38:57.478+0000 7ff26b2f1640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:38:57.481 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:38:57.479+0000 7ff26b2f1640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:38:57.728 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 1 2024-09-15T21:38:57.729 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-15T21:38:57.729 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2024-09-15T21:38:57.729 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-15T21:38:57.729 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-15T21:38:57.729 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:38:57.729 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-15T21:38:57.729 INFO:tasks.workunit.client.0.smithi136.stdout:0 2024-09-15T21:38:57.730 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:38:57.730 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T21:38:58.004 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:38:58.868 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:38:58.866+0000 7ff26b2f1640 -1 Falling back to public interface 2024-09-15T21:38:59.005 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:38:59.005 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:38:59.006 INFO:tasks.workunit.client.0.smithi136.stdout:1 2024-09-15T21:38:59.006 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-15T21:38:59.006 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:38:59.006 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T21:38:59.290 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:39:00.292 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:39:00.292 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:39:00.292 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-15T21:39:00.292 INFO:tasks.workunit.client.0.smithi136.stdout:2 2024-09-15T21:39:00.293 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:39:00.293 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T21:39:00.577 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:39:01.579 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:39:01.579 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:39:01.579 INFO:tasks.workunit.client.0.smithi136.stdout:3 2024-09-15T21:39:01.580 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-15T21:39:01.580 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:39:01.580 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T21:39:01.866 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:39:02.062 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:39:02.059+0000 7ff26b2f1640 -1 osd.1 0 log_to_monitors true 2024-09-15T21:39:02.867 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:39:02.867 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:39:02.867 INFO:tasks.workunit.client.0.smithi136.stdout:4 2024-09-15T21:39:02.868 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-15T21:39:02.868 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:39:02.868 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T21:39:03.355 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:39:04.357 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:39:04.357 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:39:04.357 INFO:tasks.workunit.client.0.smithi136.stdout:5 2024-09-15T21:39:04.358 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-15T21:39:04.358 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:39:04.358 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T21:39:04.699 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:39:05.700 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:39:05.700 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:39:05.701 INFO:tasks.workunit.client.0.smithi136.stdout:6 2024-09-15T21:39:05.701 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-15T21:39:05.701 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:39:05.701 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T21:39:06.068 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:39:07.069 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:39:07.069 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:39:07.070 INFO:tasks.workunit.client.0.smithi136.stdout:7 2024-09-15T21:39:07.070 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-15T21:39:07.070 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:39:07.070 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T21:39:07.357 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:39:08.358 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:39:08.358 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:39:08.359 INFO:tasks.workunit.client.0.smithi136.stdout:8 2024-09-15T21:39:08.359 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-15T21:39:08.359 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:39:08.359 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T21:39:08.678 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:39:09.680 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:39:09.680 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:39:09.680 INFO:tasks.workunit.client.0.smithi136.stdout:9 2024-09-15T21:39:09.681 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-15T21:39:09.681 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:39:09.681 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T21:39:10.093 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:39:11.094 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:39:11.095 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:39:11.095 INFO:tasks.workunit.client.0.smithi136.stdout:10 2024-09-15T21:39:11.095 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-15T21:39:11.095 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:39:11.096 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T21:39:11.501 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:39:12.505 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:39:12.505 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:39:12.506 INFO:tasks.workunit.client.0.smithi136.stdout:11 2024-09-15T21:39:12.506 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-15T21:39:12.506 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:39:12.506 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T21:39:12.791 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:39:13.792 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:39:13.793 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:39:13.793 INFO:tasks.workunit.client.0.smithi136.stdout:12 2024-09-15T21:39:13.793 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-15T21:39:13.793 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:39:13.794 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T21:39:14.086 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:39:15.087 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:39:15.088 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:39:15.088 INFO:tasks.workunit.client.0.smithi136.stdout:13 2024-09-15T21:39:15.088 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-15T21:39:15.089 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:39:15.089 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T21:39:15.838 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:39:16.840 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:39:16.840 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:39:16.840 INFO:tasks.workunit.client.0.smithi136.stdout:14 2024-09-15T21:39:16.841 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-15T21:39:16.841 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:39:16.841 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T21:39:17.723 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:39:18.724 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:39:18.724 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:39:18.725 INFO:tasks.workunit.client.0.smithi136.stdout:15 2024-09-15T21:39:18.726 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-15T21:39:18.726 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:39:18.726 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T21:39:19.235 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:39:20.237 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:39:20.237 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:39:20.237 INFO:tasks.workunit.client.0.smithi136.stdout:16 2024-09-15T21:39:20.238 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-15T21:39:20.238 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:39:20.238 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T21:39:20.918 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:39:21.920 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:39:21.920 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:39:21.920 INFO:tasks.workunit.client.0.smithi136.stdout:17 2024-09-15T21:39:21.921 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-15T21:39:21.921 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:39:21.921 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T21:39:22.296 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:39:23.297 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:39:23.297 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:39:23.298 INFO:tasks.workunit.client.0.smithi136.stdout:18 2024-09-15T21:39:23.298 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 18 2024-09-15T21:39:23.298 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:39:23.298 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T21:39:23.965 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:39:24.966 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:39:24.966 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:39:24.966 INFO:tasks.workunit.client.0.smithi136.stdout:19 2024-09-15T21:39:24.967 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 19 2024-09-15T21:39:24.967 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:39:24.967 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T21:39:25.301 INFO:tasks.workunit.client.0.smithi136.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/198532873,v1:127.0.0.1:6811/198532873] [v2:127.0.0.1:6812/198532873,v1:127.0.0.1:6813/198532873] exists,up 3dfe3945-eee5-4128-81ad-96e70f53c913 2024-09-15T21:39:25.305 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-15T21:39:25.305 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-15T21:39:25.305 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-15T21:39:25.305 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:922: list_missing_erasure_coded: for id in $(seq 0 2) 2024-09-15T21:39:25.305 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:923: list_missing_erasure_coded: run_osd td/osd-scrub-repair 2 2024-09-15T21:39:25.306 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-scrub-repair 2024-09-15T21:39:25.306 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-15T21:39:25.306 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=2 2024-09-15T21:39:25.306 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-15T21:39:25.306 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-scrub-repair/2 2024-09-15T21:39:25.306 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=ce4892c7-6c6d-4b4f-8e6f-07a3e11133e5 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-15T21:39:25.306 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T21:39:25.306 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T21:39:25.306 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T21:39:25.306 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/2' 2024-09-15T21:39:25.306 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/2/journal' 2024-09-15T21:39:25.306 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-15T21:39:25.306 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-15T21:39:25.306 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-15T21:39:25.307 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-15T21:39:25.307 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-15T21:39:25.307 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-15T21:39:25.307 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-15T21:39:25.307 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T21:39:25.307 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52602 2024-09-15T21:39:25.307 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.52602/$cluster-$name.asok' 2024-09-15T21:39:25.307 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.52602/$cluster-$name.asok' 2024-09-15T21:39:25.307 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-15T21:39:25.307 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-15T21:39:25.307 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-15T21:39:25.307 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-15T21:39:25.307 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-15T21:39:25.308 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T21:39:25.308 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T21:39:25.308 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T21:39:25.308 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T21:39:25.308 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-15T21:39:25.308 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-15T21:39:25.308 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-scrub-repair/2 2024-09-15T21:39:25.308 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-15T21:39:25.309 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=09a13e62-8ce3-4a2f-aa56-0d45c974cc80 2024-09-15T21:39:25.309 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd2 09a13e62-8ce3-4a2f-aa56-0d45c974cc80' 2024-09-15T21:39:25.310 INFO:tasks.workunit.client.0.smithi136.stdout:add osd2 09a13e62-8ce3-4a2f-aa56-0d45c974cc80 2024-09-15T21:39:25.310 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-15T21:39:25.326 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQANVOdm79ZQExAAVWKNp2Z3R9lPrMy+iG8iHA== 2024-09-15T21:39:25.326 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQANVOdm79ZQExAAVWKNp2Z3R9lPrMy+iG8iHA=="}' 2024-09-15T21:39:25.326 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 09a13e62-8ce3-4a2f-aa56-0d45c974cc80 -i td/osd-scrub-repair/2/new.json 2024-09-15T21:39:25.642 INFO:tasks.workunit.client.0.smithi136.stdout:2 2024-09-15T21:39:25.654 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-scrub-repair/2/new.json 2024-09-15T21:39:25.655 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 2 --fsid=ce4892c7-6c6d-4b4f-8e6f-07a3e11133e5 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.52602/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQANVOdm79ZQExAAVWKNp2Z3R9lPrMy+iG8iHA== --osd-uuid 09a13e62-8ce3-4a2f-aa56-0d45c974cc80 2024-09-15T21:39:25.683 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:39:25.681+0000 7fe7555b2640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:39:25.727 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:39:25.725+0000 7fe7555b2640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:39:25.728 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:39:25.726+0000 7fe7555b2640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:39:25.728 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:39:25.726+0000 7fe7555b2640 -1 bdev(0x56236f88dc00 td/osd-scrub-repair/2/block) open stat got: (1) Operation not permitted 2024-09-15T21:39:25.728 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:39:25.726+0000 7fe7555b2640 -1 bluestore(td/osd-scrub-repair/2) _read_fsid unparsable uuid 2024-09-15T21:39:32.403 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-scrub-repair/2/keyring 2024-09-15T21:39:32.403 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-15T21:39:32.404 INFO:tasks.workunit.client.0.smithi136.stdout:adding osd2 key to auth repository 2024-09-15T21:39:32.405 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd2 key to auth repository 2024-09-15T21:39:32.405 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-scrub-repair/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-15T21:39:32.768 INFO:tasks.workunit.client.0.smithi136.stdout:start osd.2 2024-09-15T21:39:32.769 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.2 2024-09-15T21:39:32.769 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 2 --fsid=ce4892c7-6c6d-4b4f-8e6f-07a3e11133e5 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.52602/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T21:39:32.769 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-15T21:39:32.769 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-15T21:39:32.769 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-15T21:39:32.795 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:39:32.793+0000 7fe927341640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:39:32.797 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:39:32.795+0000 7fe927341640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:39:32.798 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:39:32.796+0000 7fe927341640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:39:33.050 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 2 2024-09-15T21:39:33.051 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-15T21:39:33.051 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=2 2024-09-15T21:39:33.051 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-15T21:39:33.051 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-15T21:39:33.051 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:39:33.051 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-15T21:39:33.051 INFO:tasks.workunit.client.0.smithi136.stdout:0 2024-09-15T21:39:33.052 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:39:33.052 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T21:39:33.326 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:39:34.186 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:39:34.184+0000 7fe927341640 -1 Falling back to public interface 2024-09-15T21:39:34.328 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:39:34.328 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:39:34.328 INFO:tasks.workunit.client.0.smithi136.stdout:1 2024-09-15T21:39:34.329 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-15T21:39:34.329 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:39:34.329 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T21:39:34.612 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:39:35.613 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:39:35.613 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:39:35.614 INFO:tasks.workunit.client.0.smithi136.stdout:2 2024-09-15T21:39:35.614 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-15T21:39:35.614 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:39:35.614 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T21:39:35.898 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:39:36.899 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:39:36.899 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:39:36.899 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-15T21:39:36.899 INFO:tasks.workunit.client.0.smithi136.stdout:3 2024-09-15T21:39:36.900 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:39:36.900 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T21:39:36.925 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:39:36.923+0000 7fe927341640 -1 osd.2 0 log_to_monitors true 2024-09-15T21:39:37.301 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:39:38.313 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:39:38.313 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:39:38.313 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-15T21:39:38.313 INFO:tasks.workunit.client.0.smithi136.stdout:4 2024-09-15T21:39:38.314 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:39:38.314 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T21:39:38.598 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:39:39.600 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:39:39.600 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:39:39.600 INFO:tasks.workunit.client.0.smithi136.stdout:5 2024-09-15T21:39:39.601 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-15T21:39:39.601 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:39:39.602 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T21:39:39.971 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:39:40.973 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:39:40.973 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:39:40.973 INFO:tasks.workunit.client.0.smithi136.stdout:6 2024-09-15T21:39:40.974 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-15T21:39:40.974 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:39:40.974 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T21:39:41.361 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:39:42.363 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:39:42.363 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:39:42.363 INFO:tasks.workunit.client.0.smithi136.stdout:7 2024-09-15T21:39:42.364 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-15T21:39:42.364 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:39:42.364 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T21:39:42.650 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:39:43.652 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:39:43.652 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:39:43.652 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-15T21:39:43.653 INFO:tasks.workunit.client.0.smithi136.stdout:8 2024-09-15T21:39:43.653 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:39:43.653 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T21:39:43.989 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:39:44.990 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:39:44.990 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:39:44.990 INFO:tasks.workunit.client.0.smithi136.stdout:9 2024-09-15T21:39:44.991 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-15T21:39:44.991 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:39:44.991 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T21:39:45.279 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:39:46.280 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:39:46.280 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:39:46.281 INFO:tasks.workunit.client.0.smithi136.stdout:10 2024-09-15T21:39:46.281 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-15T21:39:46.281 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:39:46.281 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T21:39:46.615 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:39:47.617 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:39:47.617 INFO:tasks.workunit.client.0.smithi136.stdout:11 2024-09-15T21:39:47.618 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:39:47.618 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-15T21:39:47.618 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:39:47.618 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T21:39:47.973 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:39:48.975 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:39:48.975 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:39:48.975 INFO:tasks.workunit.client.0.smithi136.stdout:12 2024-09-15T21:39:48.976 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-15T21:39:48.976 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:39:48.977 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T21:39:49.803 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:39:50.804 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:39:50.804 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:39:50.805 INFO:tasks.workunit.client.0.smithi136.stdout:13 2024-09-15T21:39:50.805 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-15T21:39:50.805 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:39:50.805 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T21:39:51.514 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:39:52.515 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:39:52.515 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:39:52.515 INFO:tasks.workunit.client.0.smithi136.stdout:14 2024-09-15T21:39:52.516 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-15T21:39:52.516 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:39:52.516 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T21:39:53.109 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:39:54.111 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:39:54.111 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:39:54.111 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-15T21:39:54.111 INFO:tasks.workunit.client.0.smithi136.stdout:15 2024-09-15T21:39:54.112 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:39:54.112 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T21:39:54.726 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:39:55.727 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:39:55.727 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:39:55.727 INFO:tasks.workunit.client.0.smithi136.stdout:16 2024-09-15T21:39:55.728 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-15T21:39:55.728 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:39:55.728 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T21:39:56.258 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:39:57.260 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:39:57.260 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:39:57.260 INFO:tasks.workunit.client.0.smithi136.stdout:17 2024-09-15T21:39:57.261 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-15T21:39:57.261 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:39:57.261 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T21:39:58.022 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:39:59.023 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:39:59.023 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:39:59.024 INFO:tasks.workunit.client.0.smithi136.stdout:18 2024-09-15T21:39:59.024 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 18 2024-09-15T21:39:59.024 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:39:59.025 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T21:39:59.558 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:40:00.559 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:40:00.560 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:40:00.560 INFO:tasks.workunit.client.0.smithi136.stdout:19 2024-09-15T21:40:00.560 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 19 2024-09-15T21:40:00.561 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:40:00.561 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T21:40:00.849 INFO:tasks.workunit.client.0.smithi136.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/1467895190,v1:127.0.0.1:6819/1467895190] [v2:127.0.0.1:6820/1467895190,v1:127.0.0.1:6821/1467895190] exists,up 09a13e62-8ce3-4a2f-aa56-0d45c974cc80 2024-09-15T21:40:00.849 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-15T21:40:00.850 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-15T21:40:00.850 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-15T21:40:00.850 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:925: list_missing_erasure_coded: create_rbd_pool 2024-09-15T21:40:00.850 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:526: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2024-09-15T21:40:01.135 INFO:tasks.workunit.client.0.smithi136.stderr:pool 'rbd' does not exist 2024-09-15T21:40:01.147 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:527: create_rbd_pool: create_pool rbd 4 2024-09-15T21:40:01.147 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create rbd 4 2024-09-15T21:40:01.527 INFO:tasks.workunit.client.0.smithi136.stderr:pool 'rbd' created 2024-09-15T21:40:01.538 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2024-09-15T21:40:02.540 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:528: create_rbd_pool: rbd pool init rbd 2024-09-15T21:40:04.637 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:926: list_missing_erasure_coded: wait_for_clean 2024-09-15T21:40:04.637 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-15T21:40:04.638 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-15T21:40:04.638 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-15T21:40:04.638 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-15T21:40:04.639 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-15T21:40:04.639 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-15T21:40:04.639 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-15T21:40:04.639 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-15T21:40:04.639 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-15T21:40:04.745 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-15T21:40:04.745 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-15T21:40:04.746 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-15T21:40:04.746 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-15T21:40:04.746 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-15T21:40:04.746 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-15T21:40:05.038 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-15T21:40:05.039 INFO:tasks.workunit.client.0.smithi136.stderr:1 2024-09-15T21:40:05.039 INFO:tasks.workunit.client.0.smithi136.stderr:2' 2024-09-15T21:40:05.039 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-15T21:40:05.039 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T21:40:05.039 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-15T21:40:05.176 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836496 2024-09-15T21:40:05.176 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836496 2024-09-15T21:40:05.176 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836496' 2024-09-15T21:40:05.176 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T21:40:05.176 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-15T21:40:05.279 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=42949672973 2024-09-15T21:40:05.279 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 42949672973 2024-09-15T21:40:05.279 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836496 1-42949672973' 2024-09-15T21:40:05.279 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T21:40:05.279 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-15T21:40:05.383 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=64424509444 2024-09-15T21:40:05.383 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 64424509444 2024-09-15T21:40:05.383 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836496 1-42949672973 2-64424509444' 2024-09-15T21:40:05.383 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T21:40:05.384 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836496 2024-09-15T21:40:05.384 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T21:40:05.384 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-15T21:40:05.385 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836496 2024-09-15T21:40:05.385 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T21:40:05.386 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836496 2024-09-15T21:40:05.386 INFO:tasks.workunit.client.0.smithi136.stdout:waiting osd.0 seq 21474836496 2024-09-15T21:40:05.387 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836496' 2024-09-15T21:40:05.387 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T21:40:05.662 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836495 -lt 21474836496 2024-09-15T21:40:05.662 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-15T21:40:06.663 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-15T21:40:06.664 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T21:40:06.995 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836497 -lt 21474836496 2024-09-15T21:40:06.995 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T21:40:06.995 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-42949672973 2024-09-15T21:40:06.995 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T21:40:06.996 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-15T21:40:06.997 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-42949672973 2024-09-15T21:40:06.997 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T21:40:06.998 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=42949672973 2024-09-15T21:40:06.998 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 42949672973' 2024-09-15T21:40:06.998 INFO:tasks.workunit.client.0.smithi136.stdout:waiting osd.1 seq 42949672973 2024-09-15T21:40:06.999 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-15T21:40:07.277 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 42949672974 -lt 42949672973 2024-09-15T21:40:07.277 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T21:40:07.278 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-64424509444 2024-09-15T21:40:07.278 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T21:40:07.279 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-15T21:40:07.280 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-64424509444 2024-09-15T21:40:07.280 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T21:40:07.280 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=64424509444 2024-09-15T21:40:07.281 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 64424509444' 2024-09-15T21:40:07.281 INFO:tasks.workunit.client.0.smithi136.stdout:waiting osd.2 seq 64424509444 2024-09-15T21:40:07.281 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-15T21:40:07.559 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 64424509445 -lt 64424509444 2024-09-15T21:40:07.560 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-15T21:40:07.560 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T21:40:07.560 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T21:40:07.909 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 4 == 0 2024-09-15T21:40:07.909 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-15T21:40:07.909 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-15T21:40:07.910 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-15T21:40:07.910 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T21:40:07.910 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T21:40:07.910 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T21:40:07.910 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T21:40:08.180 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=4 2024-09-15T21:40:08.181 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-15T21:40:08.181 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T21:40:08.181 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T21:40:08.524 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 = 4 2024-09-15T21:40:08.524 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-15T21:40:08.524 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-15T21:40:08.524 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:928: list_missing_erasure_coded: create_ec_pool ecpool true k=2 m=1 2024-09-15T21:40:08.524 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2443: create_ec_pool: local pool_name=ecpool 2024-09-15T21:40:08.524 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2444: create_ec_pool: shift 2024-09-15T21:40:08.524 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2445: create_ec_pool: local allow_overwrites=true 2024-09-15T21:40:08.524 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2446: create_ec_pool: shift 2024-09-15T21:40:08.524 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2448: create_ec_pool: ceph osd erasure-code-profile set myprofile crush-failure-domain=osd k=2 m=1 2024-09-15T21:40:09.072 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2450: create_ec_pool: create_pool ecpool 1 1 erasure myprofile 2024-09-15T21:40:09.072 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create ecpool 1 1 erasure myprofile 2024-09-15T21:40:09.662 INFO:tasks.workunit.client.0.smithi136.stderr:pool 'ecpool' created 2024-09-15T21:40:09.668 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2024-09-15T21:40:10.668 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2452: create_ec_pool: '[' true = true ']' 2024-09-15T21:40:10.668 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2453: create_ec_pool: ceph osd pool set ecpool allow_ec_overwrites true 2024-09-15T21:40:11.087 INFO:tasks.workunit.client.0.smithi136.stderr:set pool 2 allow_ec_overwrites to true 2024-09-15T21:40:11.099 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2456: create_ec_pool: wait_for_clean 2024-09-15T21:40:11.099 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-15T21:40:11.100 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-15T21:40:11.100 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-15T21:40:11.100 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-15T21:40:11.100 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-15T21:40:11.100 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-15T21:40:11.101 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-15T21:40:11.101 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-15T21:40:11.101 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-15T21:40:11.197 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-15T21:40:11.197 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-15T21:40:11.198 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-15T21:40:11.198 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-15T21:40:11.198 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-15T21:40:11.198 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-15T21:40:11.472 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-15T21:40:11.472 INFO:tasks.workunit.client.0.smithi136.stderr:1 2024-09-15T21:40:11.472 INFO:tasks.workunit.client.0.smithi136.stderr:2' 2024-09-15T21:40:11.473 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-15T21:40:11.473 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T21:40:11.473 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-15T21:40:11.576 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836499 2024-09-15T21:40:11.576 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836499 2024-09-15T21:40:11.576 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836499' 2024-09-15T21:40:11.576 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T21:40:11.577 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-15T21:40:11.680 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=42949672976 2024-09-15T21:40:11.680 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 42949672976 2024-09-15T21:40:11.680 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836499 1-42949672976' 2024-09-15T21:40:11.680 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T21:40:11.681 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-15T21:40:11.793 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=64424509447 2024-09-15T21:40:11.793 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 64424509447 2024-09-15T21:40:11.793 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836499 1-42949672976 2-64424509447' 2024-09-15T21:40:11.793 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T21:40:11.793 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836499 2024-09-15T21:40:11.793 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T21:40:11.794 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-15T21:40:11.795 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836499 2024-09-15T21:40:11.795 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T21:40:11.796 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836499 2024-09-15T21:40:11.796 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836499' 2024-09-15T21:40:11.796 INFO:tasks.workunit.client.0.smithi136.stdout:waiting osd.0 seq 21474836499 2024-09-15T21:40:11.796 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T21:40:12.099 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836497 -lt 21474836499 2024-09-15T21:40:12.099 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-15T21:40:13.100 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-15T21:40:13.100 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T21:40:13.378 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836499 -lt 21474836499 2024-09-15T21:40:13.379 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T21:40:13.379 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-42949672976 2024-09-15T21:40:13.379 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T21:40:13.379 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-15T21:40:13.380 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-42949672976 2024-09-15T21:40:13.380 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T21:40:13.381 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=42949672976 2024-09-15T21:40:13.381 INFO:tasks.workunit.client.0.smithi136.stdout:waiting osd.1 seq 42949672976 2024-09-15T21:40:13.382 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 42949672976' 2024-09-15T21:40:13.382 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-15T21:40:13.658 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 42949672976 -lt 42949672976 2024-09-15T21:40:13.658 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T21:40:13.659 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-64424509447 2024-09-15T21:40:13.659 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T21:40:13.659 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-15T21:40:13.660 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-64424509447 2024-09-15T21:40:13.660 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T21:40:13.661 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=64424509447 2024-09-15T21:40:13.662 INFO:tasks.workunit.client.0.smithi136.stdout:waiting osd.2 seq 64424509447 2024-09-15T21:40:13.662 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 64424509447' 2024-09-15T21:40:13.662 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-15T21:40:13.942 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 64424509447 -lt 64424509447 2024-09-15T21:40:13.942 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-15T21:40:13.943 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T21:40:13.943 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T21:40:14.290 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 5 == 0 2024-09-15T21:40:14.290 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-15T21:40:14.290 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-15T21:40:14.290 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-15T21:40:14.291 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T21:40:14.291 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T21:40:14.291 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T21:40:14.291 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T21:40:14.563 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=5 2024-09-15T21:40:14.563 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-15T21:40:14.563 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T21:40:14.563 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T21:40:14.929 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 = 5 2024-09-15T21:40:14.929 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-15T21:40:14.929 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-15T21:40:14.929 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2457: create_ec_pool: return 0 2024-09-15T21:40:14.930 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:931: list_missing_erasure_coded: add_something td/osd-scrub-repair ecpool MOBJ0 2024-09-15T21:40:14.930 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-09-15T21:40:14.930 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=ecpool 2024-09-15T21:40:14.930 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=MOBJ0 2024-09-15T21:40:14.930 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-09-15T21:40:14.930 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-09-15T21:40:14.930 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-09-15T21:40:15.308 INFO:tasks.workunit.client.0.smithi136.stderr:noscrub is set 2024-09-15T21:40:15.319 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-09-15T21:40:15.842 INFO:tasks.workunit.client.0.smithi136.stderr:nodeep-scrub is set 2024-09-15T21:40:15.853 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-09-15T21:40:15.853 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-09-15T21:40:15.854 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool ecpool put MOBJ0 td/osd-scrub-repair/ORIGINAL 2024-09-15T21:40:16.116 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:932: list_missing_erasure_coded: get_osds ecpool MOBJ0 2024-09-15T21:40:16.116 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1012: get_osds: local poolname=ecpool 2024-09-15T21:40:16.116 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1013: get_osds: local objectname=MOBJ0 2024-09-15T21:40:16.116 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: ceph --format json osd map ecpool MOBJ0 2024-09-15T21:40:16.116 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: jq '.acting | .[]' 2024-09-15T21:40:16.397 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: local 'osds=2 2024-09-15T21:40:16.398 INFO:tasks.workunit.client.0.smithi136.stderr:1 2024-09-15T21:40:16.398 INFO:tasks.workunit.client.0.smithi136.stderr:0' 2024-09-15T21:40:16.398 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1018: get_osds: echo 2 1 0 2024-09-15T21:40:16.398 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:932: list_missing_erasure_coded: osds0=('2' '1' '0') 2024-09-15T21:40:16.398 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:932: list_missing_erasure_coded: local -a osds0 2024-09-15T21:40:16.398 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:935: list_missing_erasure_coded: add_something td/osd-scrub-repair ecpool MOBJ1 2024-09-15T21:40:16.398 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-09-15T21:40:16.398 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=ecpool 2024-09-15T21:40:16.398 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=MOBJ1 2024-09-15T21:40:16.398 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-09-15T21:40:16.398 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-09-15T21:40:16.398 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-09-15T21:40:16.773 INFO:tasks.workunit.client.0.smithi136.stderr:noscrub is set 2024-09-15T21:40:16.785 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-09-15T21:40:17.191 INFO:tasks.workunit.client.0.smithi136.stderr:nodeep-scrub is set 2024-09-15T21:40:17.203 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-09-15T21:40:17.203 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-09-15T21:40:17.203 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool ecpool put MOBJ1 td/osd-scrub-repair/ORIGINAL 2024-09-15T21:40:17.472 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:936: list_missing_erasure_coded: get_osds ecpool MOBJ1 2024-09-15T21:40:17.472 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1012: get_osds: local poolname=ecpool 2024-09-15T21:40:17.472 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1013: get_osds: local objectname=MOBJ1 2024-09-15T21:40:17.472 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: ceph --format json osd map ecpool MOBJ1 2024-09-15T21:40:17.472 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: jq '.acting | .[]' 2024-09-15T21:40:17.751 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: local 'osds=2 2024-09-15T21:40:17.751 INFO:tasks.workunit.client.0.smithi136.stderr:1 2024-09-15T21:40:17.751 INFO:tasks.workunit.client.0.smithi136.stderr:0' 2024-09-15T21:40:17.751 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1018: get_osds: echo 2 1 0 2024-09-15T21:40:17.751 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:936: list_missing_erasure_coded: osds1=('2' '1' '0') 2024-09-15T21:40:17.751 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:936: list_missing_erasure_coded: local -a osds1 2024-09-15T21:40:17.752 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:939: list_missing_erasure_coded: seq 0 2 2024-09-15T21:40:17.752 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:939: list_missing_erasure_coded: for id in $(seq 0 2) 2024-09-15T21:40:17.752 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:940: list_missing_erasure_coded: kill_daemons td/osd-scrub-repair TERM osd.0 2024-09-15T21:40:17.753 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-15T21:40:17.753 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-15T21:40:17.753 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-15T21:40:17.753 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-15T21:40:17.753 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-15T21:40:18.060 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-15T21:40:18.060 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:939: list_missing_erasure_coded: for id in $(seq 0 2) 2024-09-15T21:40:18.061 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:940: list_missing_erasure_coded: kill_daemons td/osd-scrub-repair TERM osd.1 2024-09-15T21:40:18.061 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-15T21:40:18.061 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-15T21:40:18.061 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-15T21:40:18.061 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-15T21:40:18.061 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-15T21:40:19.369 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-15T21:40:19.369 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:939: list_missing_erasure_coded: for id in $(seq 0 2) 2024-09-15T21:40:19.370 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:940: list_missing_erasure_coded: kill_daemons td/osd-scrub-repair TERM osd.2 2024-09-15T21:40:19.370 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-15T21:40:19.370 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-15T21:40:19.370 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-15T21:40:19.370 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-15T21:40:19.371 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-15T21:40:19.679 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-15T21:40:19.679 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:943: list_missing_erasure_coded: id=2 2024-09-15T21:40:19.679 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:944: list_missing_erasure_coded: ceph-objectstore-tool --data-path td/osd-scrub-repair/2 MOBJ0 remove 2024-09-15T21:40:21.393 INFO:tasks.workunit.client.0.smithi136.stdout:remove 0#2:a9030505:::MOBJ0:head# 2024-09-15T21:40:21.907 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:946: list_missing_erasure_coded: id=1 2024-09-15T21:40:21.907 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:947: list_missing_erasure_coded: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 MOBJ0 remove 2024-09-15T21:40:23.423 INFO:tasks.workunit.client.0.smithi136.stdout:remove 1#2:a9030505:::MOBJ0:head# 2024-09-15T21:40:23.937 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:950: list_missing_erasure_coded: id=1 2024-09-15T21:40:23.937 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:951: list_missing_erasure_coded: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 MOBJ1 remove 2024-09-15T21:40:25.122 INFO:tasks.workunit.client.0.smithi136.stdout:remove 1#2:8dd16d55:::MOBJ1:head# 2024-09-15T21:40:25.640 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:953: list_missing_erasure_coded: id=0 2024-09-15T21:40:25.640 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:954: list_missing_erasure_coded: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 MOBJ1 remove 2024-09-15T21:40:27.181 INFO:tasks.workunit.client.0.smithi136.stdout:remove 2#2:8dd16d55:::MOBJ1:head# 2024-09-15T21:40:27.696 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:957: list_missing_erasure_coded: seq 0 2 2024-09-15T21:40:27.698 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:957: list_missing_erasure_coded: for id in $(seq 0 2) 2024-09-15T21:40:27.698 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:958: list_missing_erasure_coded: activate_osd td/osd-scrub-repair 0 2024-09-15T21:40:27.698 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2024-09-15T21:40:27.698 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-15T21:40:27.699 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=0 2024-09-15T21:40:27.699 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-15T21:40:27.699 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-09-15T21:40:27.699 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=ce4892c7-6c6d-4b4f-8e6f-07a3e11133e5 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-15T21:40:27.699 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T21:40:27.699 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T21:40:27.699 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T21:40:27.699 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-09-15T21:40:27.699 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-09-15T21:40:27.699 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-15T21:40:27.699 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-15T21:40:27.699 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-15T21:40:27.699 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-15T21:40:27.699 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-15T21:40:27.700 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-15T21:40:27.700 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-15T21:40:27.700 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T21:40:27.700 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52602 2024-09-15T21:40:27.700 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.52602/$cluster-$name.asok' 2024-09-15T21:40:27.701 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.52602/$cluster-$name.asok' 2024-09-15T21:40:27.702 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-15T21:40:27.702 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-15T21:40:27.702 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-15T21:40:27.702 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T21:40:27.702 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T21:40:27.702 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T21:40:27.702 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T21:40:27.702 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-15T21:40:27.702 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-15T21:40:27.702 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-09-15T21:40:27.702 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.0 2024-09-15T21:40:27.702 INFO:tasks.workunit.client.0.smithi136.stderr:start osd.0 2024-09-15T21:40:27.702 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 0 --fsid=ce4892c7-6c6d-4b4f-8e6f-07a3e11133e5 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.52602/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T21:40:27.703 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-09-15T21:40:27.703 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 0 = 0 ']' 2024-09-15T21:40:27.703 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-15T21:40:27.703 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-15T21:40:27.703 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-15T21:40:27.731 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:40:27.729+0000 7fd6f6030640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:40:27.733 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:40:27.731+0000 7fd6f6030640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:40:27.734 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:40:27.733+0000 7fd6f6030640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:40:27.983 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 0 2024-09-15T21:40:27.983 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-15T21:40:27.984 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2024-09-15T21:40:27.984 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-15T21:40:27.984 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-15T21:40:27.984 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:40:27.984 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-15T21:40:27.984 INFO:tasks.workunit.client.0.smithi136.stderr:0 2024-09-15T21:40:27.984 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:40:27.984 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:40:28.261 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:40:28.862 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:40:28.860+0000 7fd6f6030640 -1 Falling back to public interface 2024-09-15T21:40:29.263 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:40:29.263 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:40:29.263 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-15T21:40:29.263 INFO:tasks.workunit.client.0.smithi136.stderr:1 2024-09-15T21:40:29.263 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:40:29.263 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:40:29.543 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:40:30.544 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:40:30.545 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:40:30.545 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-15T21:40:30.545 INFO:tasks.workunit.client.0.smithi136.stderr:2 2024-09-15T21:40:30.545 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:40:30.545 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:40:30.860 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:40:31.425 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:40:31.423+0000 7fd6f6030640 -1 osd.0 34 log_to_monitors true 2024-09-15T21:40:31.862 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:40:31.863 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:40:31.863 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-15T21:40:31.863 INFO:tasks.workunit.client.0.smithi136.stderr:3 2024-09-15T21:40:31.863 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:40:31.863 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:40:32.214 INFO:tasks.workunit.client.0.smithi136.stderr:osd.0 up in weight 1 up_from 40 up_thru 40 down_at 35 last_clean_interval [5,34) [v2:127.0.0.1:6802/3280738906,v1:127.0.0.1:6803/3280738906] [v2:127.0.0.1:6804/3280738906,v1:127.0.0.1:6805/3280738906] exists,up e1ef6b83-4aa7-4e52-ab1d-d8b427f7daaa 2024-09-15T21:40:32.214 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-15T21:40:32.214 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-15T21:40:32.214 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-15T21:40:32.214 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:957: list_missing_erasure_coded: for id in $(seq 0 2) 2024-09-15T21:40:32.214 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:958: list_missing_erasure_coded: activate_osd td/osd-scrub-repair 1 2024-09-15T21:40:32.214 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2024-09-15T21:40:32.214 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-15T21:40:32.215 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=1 2024-09-15T21:40:32.215 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-15T21:40:32.215 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-09-15T21:40:32.215 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=ce4892c7-6c6d-4b4f-8e6f-07a3e11133e5 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-15T21:40:32.215 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T21:40:32.215 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T21:40:32.215 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T21:40:32.215 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-09-15T21:40:32.215 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-09-15T21:40:32.215 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-15T21:40:32.215 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-15T21:40:32.215 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-15T21:40:32.215 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-15T21:40:32.215 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-15T21:40:32.215 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-15T21:40:32.216 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-15T21:40:32.216 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T21:40:32.216 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52602 2024-09-15T21:40:32.216 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.52602/$cluster-$name.asok' 2024-09-15T21:40:32.216 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.52602/$cluster-$name.asok' 2024-09-15T21:40:32.216 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-15T21:40:32.216 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-15T21:40:32.216 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-15T21:40:32.217 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T21:40:32.217 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T21:40:32.217 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T21:40:32.217 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T21:40:32.217 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-15T21:40:32.217 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-15T21:40:32.217 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-09-15T21:40:32.217 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.1 2024-09-15T21:40:32.217 INFO:tasks.workunit.client.0.smithi136.stderr:start osd.1 2024-09-15T21:40:32.217 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 1 --fsid=ce4892c7-6c6d-4b4f-8e6f-07a3e11133e5 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.52602/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T21:40:32.218 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-09-15T21:40:32.218 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 1 = 1 ']' 2024-09-15T21:40:32.218 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-15T21:40:32.218 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-15T21:40:32.218 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-15T21:40:32.241 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:40:32.238+0000 7fb4bd2b0640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:40:32.256 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:40:32.254+0000 7fb4bd2b0640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:40:32.260 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:40:32.255+0000 7fb4bd2b0640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:40:32.494 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 1 2024-09-15T21:40:32.494 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-15T21:40:32.494 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2024-09-15T21:40:32.494 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-15T21:40:32.495 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-15T21:40:32.495 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:40:32.495 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-15T21:40:32.495 INFO:tasks.workunit.client.0.smithi136.stderr:0 2024-09-15T21:40:32.495 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:40:32.495 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T21:40:32.771 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:40:33.773 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:40:33.773 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:40:33.773 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-15T21:40:33.773 INFO:tasks.workunit.client.0.smithi136.stderr:1 2024-09-15T21:40:33.773 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:40:33.773 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T21:40:33.881 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:40:33.879+0000 7fb4bd2b0640 -1 Falling back to public interface 2024-09-15T21:40:34.057 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:40:35.059 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:40:35.059 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:40:35.059 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-15T21:40:35.059 INFO:tasks.workunit.client.0.smithi136.stderr:2 2024-09-15T21:40:35.059 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:40:35.059 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T21:40:35.337 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:40:36.339 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:40:36.339 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:40:36.339 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-15T21:40:36.339 INFO:tasks.workunit.client.0.smithi136.stderr:3 2024-09-15T21:40:36.339 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:40:36.339 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T21:40:36.622 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:40:36.763 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:40:36.760+0000 7fb4bd2b0640 -1 osd.1 35 log_to_monitors true 2024-09-15T21:40:37.623 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:40:37.623 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:40:37.624 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-15T21:40:37.624 INFO:tasks.workunit.client.0.smithi136.stderr:4 2024-09-15T21:40:37.624 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:40:37.624 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T21:40:37.909 INFO:tasks.workunit.client.0.smithi136.stderr:osd.1 up in weight 1 up_from 43 up_thru 43 down_at 36 last_clean_interval [10,35) [v2:127.0.0.1:6810/132090963,v1:127.0.0.1:6811/132090963] [v2:127.0.0.1:6812/132090963,v1:127.0.0.1:6813/132090963] exists,up 3dfe3945-eee5-4128-81ad-96e70f53c913 2024-09-15T21:40:37.909 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-15T21:40:37.909 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-15T21:40:37.910 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-15T21:40:37.910 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:957: list_missing_erasure_coded: for id in $(seq 0 2) 2024-09-15T21:40:37.910 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:958: list_missing_erasure_coded: activate_osd td/osd-scrub-repair 2 2024-09-15T21:40:37.910 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2024-09-15T21:40:37.910 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-15T21:40:37.910 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=2 2024-09-15T21:40:37.910 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-15T21:40:37.911 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/2 2024-09-15T21:40:37.911 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=ce4892c7-6c6d-4b4f-8e6f-07a3e11133e5 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-15T21:40:37.911 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T21:40:37.911 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T21:40:37.911 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T21:40:37.911 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/2' 2024-09-15T21:40:37.911 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/2/journal' 2024-09-15T21:40:37.911 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-15T21:40:37.911 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-15T21:40:37.911 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-15T21:40:37.911 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-15T21:40:37.911 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-15T21:40:37.911 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-15T21:40:37.911 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-15T21:40:37.912 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T21:40:37.912 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52602 2024-09-15T21:40:37.912 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.52602/$cluster-$name.asok' 2024-09-15T21:40:37.912 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.52602/$cluster-$name.asok' 2024-09-15T21:40:37.912 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-15T21:40:37.912 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-15T21:40:37.912 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-15T21:40:37.912 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T21:40:37.912 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T21:40:37.912 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T21:40:37.912 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T21:40:37.912 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-15T21:40:37.912 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-15T21:40:37.913 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/2 2024-09-15T21:40:37.913 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.2 2024-09-15T21:40:37.913 INFO:tasks.workunit.client.0.smithi136.stderr:start osd.2 2024-09-15T21:40:37.913 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 2 --fsid=ce4892c7-6c6d-4b4f-8e6f-07a3e11133e5 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.52602/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T21:40:37.913 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/2/whoami 2024-09-15T21:40:37.914 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 2 = 2 ']' 2024-09-15T21:40:37.914 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-15T21:40:37.914 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-15T21:40:37.914 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-15T21:40:37.936 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:40:37.933+0000 7f9dfd329640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:40:38.045 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:40:38.042+0000 7f9dfd329640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:40:38.046 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:40:38.043+0000 7f9dfd329640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:40:38.283 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 2 2024-09-15T21:40:38.283 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-15T21:40:38.283 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=2 2024-09-15T21:40:38.284 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-15T21:40:38.284 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-15T21:40:38.284 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:40:38.284 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-15T21:40:38.284 INFO:tasks.workunit.client.0.smithi136.stderr:0 2024-09-15T21:40:38.284 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:40:38.284 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T21:40:38.698 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:40:39.148 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:40:39.145+0000 7f9dfd329640 -1 Falling back to public interface 2024-09-15T21:40:39.700 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:40:39.700 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:40:39.700 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-15T21:40:39.700 INFO:tasks.workunit.client.0.smithi136.stderr:1 2024-09-15T21:40:39.700 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:40:39.700 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T21:40:39.986 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:40:40.987 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:40:40.988 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:40:40.988 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-15T21:40:40.988 INFO:tasks.workunit.client.0.smithi136.stderr:2 2024-09-15T21:40:40.988 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:40:40.988 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T21:40:41.277 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:40:41.766 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:40:41.764+0000 7f9dfd329640 -1 osd.2 37 log_to_monitors true 2024-09-15T21:40:42.278 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:40:42.278 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:40:42.279 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-15T21:40:42.279 INFO:tasks.workunit.client.0.smithi136.stderr:3 2024-09-15T21:40:42.279 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:40:42.279 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T21:40:42.595 INFO:tasks.workunit.client.0.smithi136.stderr:osd.2 up in weight 1 up_from 47 up_thru 36 down_at 38 last_clean_interval [15,37) [v2:127.0.0.1:6818/1204569946,v1:127.0.0.1:6819/1204569946] [v2:127.0.0.1:6820/1204569946,v1:127.0.0.1:6821/1204569946] exists,up 09a13e62-8ce3-4a2f-aa56-0d45c974cc80 2024-09-15T21:40:42.595 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-15T21:40:42.595 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-15T21:40:42.595 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-15T21:40:42.595 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:960: list_missing_erasure_coded: create_rbd_pool 2024-09-15T21:40:42.595 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:526: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2024-09-15T21:40:43.363 INFO:tasks.workunit.client.0.smithi136.stderr:pool 'rbd' removed 2024-09-15T21:40:43.376 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:527: create_rbd_pool: create_pool rbd 4 2024-09-15T21:40:43.376 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create rbd 4 2024-09-15T21:40:43.773 INFO:tasks.workunit.client.0.smithi136.stderr:pool 'rbd' created 2024-09-15T21:40:43.785 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2024-09-15T21:40:44.787 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:528: create_rbd_pool: rbd pool init rbd 2024-09-15T21:40:46.433 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:961: list_missing_erasure_coded: wait_for_clean 2024-09-15T21:40:46.433 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-15T21:40:46.433 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-15T21:40:46.433 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-15T21:40:46.434 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-15T21:40:46.434 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-15T21:40:46.434 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-15T21:40:46.435 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-15T21:40:46.435 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-15T21:40:46.435 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-15T21:40:46.539 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-15T21:40:46.540 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-15T21:40:46.540 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-15T21:40:46.540 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-15T21:40:46.540 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-15T21:40:46.540 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-15T21:40:46.816 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-15T21:40:46.816 INFO:tasks.workunit.client.0.smithi136.stderr:1 2024-09-15T21:40:46.816 INFO:tasks.workunit.client.0.smithi136.stderr:2' 2024-09-15T21:40:46.816 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-15T21:40:46.816 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T21:40:46.817 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-15T21:40:46.922 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=171798691845 2024-09-15T21:40:46.923 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 171798691845 2024-09-15T21:40:46.923 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-171798691845' 2024-09-15T21:40:46.923 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T21:40:46.923 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-15T21:40:47.029 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=184683593731 2024-09-15T21:40:47.029 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 184683593731 2024-09-15T21:40:47.029 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-171798691845 1-184683593731' 2024-09-15T21:40:47.029 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T21:40:47.029 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-15T21:40:47.130 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=201863462915 2024-09-15T21:40:47.130 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 201863462915 2024-09-15T21:40:47.130 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-171798691845 1-184683593731 2-201863462915' 2024-09-15T21:40:47.130 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T21:40:47.131 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-171798691845 2024-09-15T21:40:47.131 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T21:40:47.131 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-15T21:40:47.132 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-171798691845 2024-09-15T21:40:47.132 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T21:40:47.133 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=171798691845 2024-09-15T21:40:47.133 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 171798691845' 2024-09-15T21:40:47.134 INFO:tasks.workunit.client.0.smithi136.stdout:waiting osd.0 seq 171798691845 2024-09-15T21:40:47.134 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T21:40:47.411 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 171798691844 -lt 171798691845 2024-09-15T21:40:47.411 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-15T21:40:48.412 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-15T21:40:48.413 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T21:40:48.697 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 171798691844 -lt 171798691845 2024-09-15T21:40:48.697 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-15T21:40:49.698 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 299 -eq 0 ']' 2024-09-15T21:40:49.699 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T21:40:49.980 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 171798691845 -lt 171798691845 2024-09-15T21:40:49.980 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T21:40:49.980 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-184683593731 2024-09-15T21:40:49.980 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T21:40:49.981 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-15T21:40:49.982 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-184683593731 2024-09-15T21:40:49.982 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T21:40:49.983 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=184683593731 2024-09-15T21:40:49.983 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 184683593731' 2024-09-15T21:40:49.983 INFO:tasks.workunit.client.0.smithi136.stdout:waiting osd.1 seq 184683593731 2024-09-15T21:40:49.984 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-15T21:40:50.258 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 184683593732 -lt 184683593731 2024-09-15T21:40:50.258 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T21:40:50.259 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-201863462915 2024-09-15T21:40:50.259 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T21:40:50.260 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-15T21:40:50.261 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-201863462915 2024-09-15T21:40:50.261 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T21:40:50.262 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=201863462915 2024-09-15T21:40:50.262 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 201863462915' 2024-09-15T21:40:50.262 INFO:tasks.workunit.client.0.smithi136.stdout:waiting osd.2 seq 201863462915 2024-09-15T21:40:50.262 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-15T21:40:50.539 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 201863462915 -lt 201863462915 2024-09-15T21:40:50.539 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-15T21:40:50.540 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T21:40:50.540 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T21:40:50.883 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 5 == 0 2024-09-15T21:40:50.883 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-15T21:40:50.883 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-15T21:40:50.883 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-15T21:40:50.883 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T21:40:50.883 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T21:40:50.884 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T21:40:50.884 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T21:40:51.179 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=5 2024-09-15T21:40:51.180 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-15T21:40:51.180 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T21:40:51.180 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T21:40:51.523 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 = 5 2024-09-15T21:40:51.524 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-15T21:40:51.524 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-15T21:40:51.524 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:964: list_missing_erasure_coded: get_pg ecpool MOBJ0 2024-09-15T21:40:51.524 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1084: get_pg: local poolname=ecpool 2024-09-15T21:40:51.524 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1085: get_pg: local objectname=MOBJ0 2024-09-15T21:40:51.524 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1087: get_pg: ceph --format json osd map ecpool MOBJ0 2024-09-15T21:40:51.524 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1087: get_pg: jq -r .pgid 2024-09-15T21:40:51.804 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:964: list_missing_erasure_coded: local pg=2.0 2024-09-15T21:40:51.804 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:968: list_missing_erasure_coded: repair 2.0 2024-09-15T21:40:51.804 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1846: repair: local pgid=2.0 2024-09-15T21:40:51.804 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1847: repair: get_last_scrub_stamp 2.0 2024-09-15T21:40:51.804 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2024-09-15T21:40:51.804 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-15T21:40:51.804 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-15T21:40:51.804 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-09-15T21:40:52.080 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1847: repair: local last_scrub=2024-09-15T21:40:09.543997+0000 2024-09-15T21:40:52.080 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1848: repair: ceph pg repair 2.0 2024-09-15T21:40:52.368 INFO:tasks.workunit.client.0.smithi136.stderr:instructing pg 2.0s0 on osd.2 to repair 2024-09-15T21:40:52.381 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1849: repair: wait_for_scrub 2.0 2024-09-15T21:40:09.543997+0000 2024-09-15T21:40:52.381 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: local pgid=2.0 2024-09-15T21:40:52.381 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2018: wait_for_scrub: local last_scrub=2024-09-15T21:40:09.543997+0000 2024-09-15T21:40:52.381 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2019: wait_for_scrub: local sname=last_scrub_stamp 2024-09-15T21:40:52.381 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i=0 )) 2024-09-15T21:40:52.381 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-15T21:40:52.382 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-09-15T21:40:52.382 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2024-09-15T21:40:52.382 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-15T21:40:52.382 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-15T21:40:52.382 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-09-15T21:40:52.651 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-15T21:40:09.543997+0000 '>' 2024-09-15T21:40:09.543997+0000 2024-09-15T21:40:52.652 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-15T21:40:53.653 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-15T21:40:53.653 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-15T21:40:53.654 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-09-15T21:40:53.654 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2024-09-15T21:40:53.654 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-15T21:40:53.654 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-15T21:40:53.654 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-09-15T21:40:53.935 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-15T21:40:09.543997+0000 '>' 2024-09-15T21:40:09.543997+0000 2024-09-15T21:40:53.935 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-15T21:40:54.936 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-15T21:40:54.936 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-15T21:40:54.937 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-09-15T21:40:54.937 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2024-09-15T21:40:54.937 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-15T21:40:54.937 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-15T21:40:54.937 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-09-15T21:40:55.222 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-15T21:40:09.543997+0000 '>' 2024-09-15T21:40:09.543997+0000 2024-09-15T21:40:55.222 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-15T21:40:56.226 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-15T21:40:56.226 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-15T21:40:56.226 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-09-15T21:40:56.226 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2024-09-15T21:40:56.226 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-15T21:40:56.226 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-15T21:40:56.226 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-09-15T21:40:56.505 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-15T21:40:09.543997+0000 '>' 2024-09-15T21:40:09.543997+0000 2024-09-15T21:40:56.505 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-15T21:40:57.507 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-15T21:40:57.507 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-15T21:40:57.508 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-09-15T21:40:57.508 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2024-09-15T21:40:57.508 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-15T21:40:57.508 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-15T21:40:57.508 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-09-15T21:40:57.795 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-15T21:40:09.543997+0000 '>' 2024-09-15T21:40:09.543997+0000 2024-09-15T21:40:57.795 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-15T21:40:58.796 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-15T21:40:58.796 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-15T21:40:58.797 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-09-15T21:40:58.797 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2024-09-15T21:40:58.797 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-15T21:40:58.797 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-15T21:40:58.797 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-09-15T21:40:59.102 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-15T21:40:52.925938+0000 '>' 2024-09-15T21:40:09.543997+0000 2024-09-15T21:40:59.102 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2023: wait_for_scrub: return 0 2024-09-15T21:40:59.102 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1850: repair: sleep 2 2024-09-15T21:41:01.104 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:970: list_missing_erasure_coded: seq 0 120 2024-09-15T21:41:01.105 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:970: list_missing_erasure_coded: for i in $(seq 0 120) 2024-09-15T21:41:01.105 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:971: list_missing_erasure_coded: '[' 0 -lt 60 ']' 2024-09-15T21:41:01.106 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:972: list_missing_erasure_coded: ceph pg 2.0 list_unfound 2024-09-15T21:41:01.106 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:972: list_missing_erasure_coded: egrep 'MOBJ0|MOBJ1' 2024-09-15T21:41:01.106 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:972: list_missing_erasure_coded: wc -l 2024-09-15T21:41:01.223 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:972: list_missing_erasure_coded: matches=2 2024-09-15T21:41:01.223 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:973: list_missing_erasure_coded: '[' 2 -eq 2 ']' 2024-09-15T21:41:01.223 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:973: list_missing_erasure_coded: break 2024-09-15T21:41:01.223 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:66: run: teardown td/osd-scrub-repair 2024-09-15T21:41:01.223 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/osd-scrub-repair 2024-09-15T21:41:01.223 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-15T21:41:01.223 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/osd-scrub-repair KILL 2024-09-15T21:41:01.223 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-15T21:41:01.223 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-15T21:41:01.224 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-15T21:41:01.224 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-15T21:41:01.224 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-15T21:41:01.331 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-15T21:41:01.332 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-15T21:41:01.332 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-15T21:41:01.333 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-15T21:41:01.334 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-15T21:41:01.334 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-15T21:41:01.334 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-15T21:41:01.335 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T21:41:01.335 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-15T21:41:01.336 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-15T21:41:01.336 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T21:41:01.337 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-15T21:41:01.338 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-15T21:41:01.338 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/osd-scrub-repair 2024-09-15T21:41:01.355 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-15T21:41:01.356 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T21:41:01.356 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52602 2024-09-15T21:41:01.356 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.52602 2024-09-15T21:41:01.358 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-15T21:41:01.358 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-15T21:41:01.358 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:63: run: for func in $funcs 2024-09-15T21:41:01.358 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:64: run: setup td/osd-scrub-repair 2024-09-15T21:41:01.358 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/osd-scrub-repair 2024-09-15T21:41:01.358 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/osd-scrub-repair 2024-09-15T21:41:01.358 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/osd-scrub-repair 2024-09-15T21:41:01.358 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-15T21:41:01.358 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/osd-scrub-repair KILL 2024-09-15T21:41:01.358 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-15T21:41:01.358 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-15T21:41:01.358 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-15T21:41:01.358 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-15T21:41:01.359 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-15T21:41:01.359 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-15T21:41:01.360 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-15T21:41:01.361 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-15T21:41:01.361 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-15T21:41:01.362 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-15T21:41:01.362 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-15T21:41:01.362 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-15T21:41:01.363 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T21:41:01.363 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-15T21:41:01.364 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-15T21:41:01.364 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T21:41:01.365 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-15T21:41:01.366 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-15T21:41:01.366 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/osd-scrub-repair 2024-09-15T21:41:01.367 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-15T21:41:01.367 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T21:41:01.367 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52602 2024-09-15T21:41:01.367 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.52602 2024-09-15T21:41:01.368 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-15T21:41:01.368 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-15T21:41:01.368 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/osd-scrub-repair 2024-09-15T21:41:01.370 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2024-09-15T21:41:01.370 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T21:41:01.370 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52602 2024-09-15T21:41:01.370 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.52602 2024-09-15T21:41:01.373 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2024-09-15T21:41:01.373 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 4096 -le 1024 ']' 2024-09-15T21:41:01.373 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2024-09-15T21:41:01.373 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/osd-scrub-repair 1' TERM HUP INT 2024-09-15T21:41:01.373 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:65: run: TEST_periodic_scrub_replicated td/osd-scrub-repair 2024-09-15T21:41:01.373 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5760: TEST_periodic_scrub_replicated: local dir=td/osd-scrub-repair 2024-09-15T21:41:01.373 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5761: TEST_periodic_scrub_replicated: local poolname=psr_pool 2024-09-15T21:41:01.373 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5762: TEST_periodic_scrub_replicated: local objname=POBJ 2024-09-15T21:41:01.373 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5763: TEST_periodic_scrub_replicated: return 0 2024-09-15T21:41:01.373 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:66: run: teardown td/osd-scrub-repair 2024-09-15T21:41:01.373 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/osd-scrub-repair 2024-09-15T21:41:01.373 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-15T21:41:01.374 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/osd-scrub-repair KILL 2024-09-15T21:41:01.374 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-15T21:41:01.374 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-15T21:41:01.374 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-15T21:41:01.374 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-15T21:41:01.374 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-15T21:41:01.374 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-15T21:41:01.374 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-15T21:41:01.375 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-15T21:41:01.376 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-15T21:41:01.380 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-15T21:41:01.380 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-15T21:41:01.382 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-15T21:41:01.382 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T21:41:01.382 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-15T21:41:01.382 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-15T21:41:01.382 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T21:41:01.382 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-15T21:41:01.382 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-15T21:41:01.382 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/osd-scrub-repair 2024-09-15T21:41:01.382 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-15T21:41:01.382 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T21:41:01.382 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52602 2024-09-15T21:41:01.383 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.52602 2024-09-15T21:41:01.384 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-15T21:41:01.384 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-15T21:41:01.384 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:63: run: for func in $funcs 2024-09-15T21:41:01.385 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:64: run: setup td/osd-scrub-repair 2024-09-15T21:41:01.385 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/osd-scrub-repair 2024-09-15T21:41:01.385 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/osd-scrub-repair 2024-09-15T21:41:01.385 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/osd-scrub-repair 2024-09-15T21:41:01.385 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-15T21:41:01.385 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/osd-scrub-repair KILL 2024-09-15T21:41:01.385 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-15T21:41:01.385 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-15T21:41:01.385 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-15T21:41:01.385 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-15T21:41:01.385 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-15T21:41:01.386 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-15T21:41:01.386 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-15T21:41:01.387 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-15T21:41:01.387 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-15T21:41:01.388 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-15T21:41:01.388 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-15T21:41:01.389 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-15T21:41:01.389 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T21:41:01.390 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-15T21:41:01.390 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-15T21:41:01.390 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T21:41:01.391 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-15T21:41:01.392 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-15T21:41:01.392 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/osd-scrub-repair 2024-09-15T21:41:01.393 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-15T21:41:01.393 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T21:41:01.393 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52602 2024-09-15T21:41:01.393 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.52602 2024-09-15T21:41:01.394 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-15T21:41:01.394 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-15T21:41:01.394 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/osd-scrub-repair 2024-09-15T21:41:01.396 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2024-09-15T21:41:01.396 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T21:41:01.396 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52602 2024-09-15T21:41:01.396 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.52602 2024-09-15T21:41:01.399 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2024-09-15T21:41:01.399 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 4096 -le 1024 ']' 2024-09-15T21:41:01.399 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2024-09-15T21:41:01.399 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/osd-scrub-repair 1' TERM HUP INT 2024-09-15T21:41:01.399 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:65: run: TEST_repair_stats td/osd-scrub-repair 2024-09-15T21:41:01.399 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:654: TEST_repair_stats: local dir=td/osd-scrub-repair 2024-09-15T21:41:01.399 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:655: TEST_repair_stats: local poolname=testpool 2024-09-15T21:41:01.399 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:656: TEST_repair_stats: local OSDS=2 2024-09-15T21:41:01.399 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:657: TEST_repair_stats: local OBJS=30 2024-09-15T21:41:01.399 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:659: TEST_repair_stats: local REPAIRS=20 2024-09-15T21:41:01.399 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:662: TEST_repair_stats: run_mon td/osd-scrub-repair a 2024-09-15T21:41:01.399 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/osd-scrub-repair 2024-09-15T21:41:01.399 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2024-09-15T21:41:01.399 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2024-09-15T21:41:01.400 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2024-09-15T21:41:01.400 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/osd-scrub-repair/a 2024-09-15T21:41:01.400 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-scrub-repair/a --run-dir=td/osd-scrub-repair 2024-09-15T21:41:01.870 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2024-09-15T21:41:01.870 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-15T21:41:01.870 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-15T21:41:01.870 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-15T21:41:01.870 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T21:41:01.870 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52602 2024-09-15T21:41:01.870 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.52602/$cluster-$name.asok' 2024-09-15T21:41:01.870 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-scrub-repair/a '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.52602/$cluster-$name.asok' --mon-cluster-log-file=td/osd-scrub-repair/log --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-09-15T21:41:02.045 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2024-09-15T21:41:02.047 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2024-09-15T21:41:02.047 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-15T21:41:02.047 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-15T21:41:02.047 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2024-09-15T21:41:02.047 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2024-09-15T21:41:02.047 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-15T21:41:02.047 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-15T21:41:02.047 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-15T21:41:02.048 INFO:tasks.workunit.client.0.smithi136.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-15T21:41:02.049 INFO:tasks.workunit.client.0.smithi136.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T21:41:02.049 INFO:tasks.workunit.client.0.smithi136.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52602 2024-09-15T21:41:02.049 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.52602/ceph-mon.a.asok 2024-09-15T21:41:02.049 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-15T21:41:02.049 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.52602/ceph-mon.a.asok config get fsid 2024-09-15T21:41:02.127 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2024-09-15T21:41:02.127 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-15T21:41:02.127 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-15T21:41:02.127 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2024-09-15T21:41:02.127 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2024-09-15T21:41:02.128 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-15T21:41:02.128 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-15T21:41:02.128 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-15T21:41:02.128 INFO:tasks.workunit.client.0.smithi136.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-15T21:41:02.129 INFO:tasks.workunit.client.0.smithi136.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T21:41:02.129 INFO:tasks.workunit.client.0.smithi136.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52602 2024-09-15T21:41:02.129 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.52602/ceph-mon.a.asok 2024-09-15T21:41:02.129 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-15T21:41:02.129 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.52602/ceph-mon.a.asok config get mon_host 2024-09-15T21:41:02.207 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:663: TEST_repair_stats: run_mgr td/osd-scrub-repair x 2024-09-15T21:41:02.208 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: run_mgr: local dir=td/osd-scrub-repair 2024-09-15T21:41:02.208 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: run_mgr: shift 2024-09-15T21:41:02.208 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: run_mgr: local id=x 2024-09-15T21:41:02.208 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: run_mgr: shift 2024-09-15T21:41:02.208 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:548: run_mgr: local data=td/osd-scrub-repair/x 2024-09-15T21:41:02.208 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:550: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-15T21:41:03.013 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: get_asok_path 2024-09-15T21:41:03.013 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-15T21:41:03.013 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-15T21:41:03.014 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-15T21:41:03.014 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T21:41:03.014 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52602 2024-09-15T21:41:03.014 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.52602/$cluster-$name.asok' 2024-09-15T21:41:03.014 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-15T21:41:03.015 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-scrub-repair/x '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.52602/$cluster-$name.asok' --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-15T21:41:03.042 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:665: TEST_repair_stats: local 'ceph_osd_args=--osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0' 2024-09-15T21:41:03.043 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:666: TEST_repair_stats: expr 2 - 1 2024-09-15T21:41:03.044 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:666: TEST_repair_stats: seq 0 1 2024-09-15T21:41:03.048 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:666: TEST_repair_stats: for id in $(seq 0 $(expr $OSDS - 1)) 2024-09-15T21:41:03.048 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:667: TEST_repair_stats: run_osd td/osd-scrub-repair 0 --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-09-15T21:41:03.048 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-scrub-repair 2024-09-15T21:41:03.048 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-15T21:41:03.048 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=0 2024-09-15T21:41:03.049 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-15T21:41:03.049 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-scrub-repair/0 2024-09-15T21:41:03.049 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=ce4892c7-6c6d-4b4f-8e6f-07a3e11133e5 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-15T21:41:03.049 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T21:41:03.049 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T21:41:03.049 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T21:41:03.049 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-09-15T21:41:03.049 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-09-15T21:41:03.049 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-15T21:41:03.049 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-15T21:41:03.049 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-15T21:41:03.049 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-15T21:41:03.049 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-15T21:41:03.050 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-15T21:41:03.050 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-15T21:41:03.050 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T21:41:03.050 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52602 2024-09-15T21:41:03.050 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.52602/$cluster-$name.asok' 2024-09-15T21:41:03.050 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.52602/$cluster-$name.asok' 2024-09-15T21:41:03.050 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-15T21:41:03.050 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-15T21:41:03.050 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-15T21:41:03.050 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-15T21:41:03.050 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-15T21:41:03.050 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T21:41:03.050 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T21:41:03.050 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T21:41:03.051 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T21:41:03.051 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-15T21:41:03.051 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+='--osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0' 2024-09-15T21:41:03.051 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-scrub-repair/0 2024-09-15T21:41:03.051 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-15T21:41:03.051 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=26207316-4540-4f5c-9936-c8113d190460 2024-09-15T21:41:03.051 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd0 26207316-4540-4f5c-9936-c8113d190460' 2024-09-15T21:41:03.052 INFO:tasks.workunit.client.0.smithi136.stdout:add osd0 26207316-4540-4f5c-9936-c8113d190460 2024-09-15T21:41:03.052 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-15T21:41:03.068 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQBvVOdm68rwAxAAqvzJgv6B9DKR2LFxq8VRSA== 2024-09-15T21:41:03.068 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQBvVOdm68rwAxAAqvzJgv6B9DKR2LFxq8VRSA=="}' 2024-09-15T21:41:03.068 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 26207316-4540-4f5c-9936-c8113d190460 -i td/osd-scrub-repair/0/new.json 2024-09-15T21:41:03.269 INFO:tasks.workunit.client.0.smithi136.stdout:0 2024-09-15T21:41:03.279 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-scrub-repair/0/new.json 2024-09-15T21:41:03.280 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 0 --fsid=ce4892c7-6c6d-4b4f-8e6f-07a3e11133e5 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.52602/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --mkfs --key AQBvVOdm68rwAxAAqvzJgv6B9DKR2LFxq8VRSA== --osd-uuid 26207316-4540-4f5c-9936-c8113d190460 2024-09-15T21:41:03.311 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:41:03.309+0000 7ff56b00c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:41:03.320 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:41:03.318+0000 7ff56b00c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:41:03.321 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:41:03.319+0000 7ff56b00c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:41:03.321 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:41:03.319+0000 7ff56b00c640 -1 bdev(0x55b4a8a15c00 td/osd-scrub-repair/0/block) open stat got: (1) Operation not permitted 2024-09-15T21:41:03.321 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:41:03.320+0000 7ff56b00c640 -1 bluestore(td/osd-scrub-repair/0) _read_fsid unparsable uuid 2024-09-15T21:41:08.852 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-scrub-repair/0/keyring 2024-09-15T21:41:08.852 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-15T21:41:08.853 INFO:tasks.workunit.client.0.smithi136.stdout:adding osd0 key to auth repository 2024-09-15T21:41:08.854 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd0 key to auth repository 2024-09-15T21:41:08.854 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-scrub-repair/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-15T21:41:09.247 INFO:tasks.workunit.client.0.smithi136.stdout:start osd.0 2024-09-15T21:41:09.248 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.0 2024-09-15T21:41:09.248 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 0 --fsid=ce4892c7-6c6d-4b4f-8e6f-07a3e11133e5 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.52602/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-09-15T21:41:09.248 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-15T21:41:09.248 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-15T21:41:09.248 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-15T21:41:09.275 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:41:09.274+0000 7f3863f02640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:41:09.278 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:41:09.276+0000 7f3863f02640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:41:09.279 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:41:09.277+0000 7f3863f02640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:41:09.527 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 0 2024-09-15T21:41:09.528 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-15T21:41:09.528 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2024-09-15T21:41:09.528 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-15T21:41:09.528 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-15T21:41:09.528 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:41:09.528 INFO:tasks.workunit.client.0.smithi136.stdout:0 2024-09-15T21:41:09.529 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-15T21:41:09.529 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:41:09.529 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:41:09.802 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:41:10.404 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:41:10.402+0000 7f3863f02640 -1 Falling back to public interface 2024-09-15T21:41:10.803 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:41:10.804 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:41:10.804 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-15T21:41:10.804 INFO:tasks.workunit.client.0.smithi136.stdout:1 2024-09-15T21:41:10.805 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:41:10.805 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:41:11.088 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:41:12.089 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:41:12.089 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:41:12.090 INFO:tasks.workunit.client.0.smithi136.stdout:2 2024-09-15T21:41:12.090 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-15T21:41:12.090 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:41:12.090 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:41:12.379 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:41:13.318 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:41:13.316+0000 7f3863f02640 -1 osd.0 0 log_to_monitors true 2024-09-15T21:41:13.380 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:41:13.381 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:41:13.381 INFO:tasks.workunit.client.0.smithi136.stdout:3 2024-09-15T21:41:13.381 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-15T21:41:13.381 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:41:13.382 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:41:13.775 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:41:14.777 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:41:14.777 INFO:tasks.workunit.client.0.smithi136.stdout:4 2024-09-15T21:41:14.777 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:41:14.778 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-15T21:41:14.778 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:41:14.778 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:41:15.071 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:41:16.072 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:41:16.072 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:41:16.072 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-15T21:41:16.073 INFO:tasks.workunit.client.0.smithi136.stdout:5 2024-09-15T21:41:16.073 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:41:16.073 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:41:16.358 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:41:17.359 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:41:17.359 INFO:tasks.workunit.client.0.smithi136.stdout:6 2024-09-15T21:41:17.360 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:41:17.360 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-15T21:41:17.360 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:41:17.360 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:41:17.649 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:41:18.650 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:41:18.651 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:41:18.651 INFO:tasks.workunit.client.0.smithi136.stdout:7 2024-09-15T21:41:18.651 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-15T21:41:18.651 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:41:18.651 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:41:18.937 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:41:19.939 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:41:19.939 INFO:tasks.workunit.client.0.smithi136.stdout:8 2024-09-15T21:41:19.940 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:41:19.940 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-15T21:41:19.940 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:41:19.940 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:41:20.288 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:41:21.290 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:41:21.290 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:41:21.290 INFO:tasks.workunit.client.0.smithi136.stdout:9 2024-09-15T21:41:21.291 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-15T21:41:21.291 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:41:21.291 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:41:21.573 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:41:22.575 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:41:22.575 INFO:tasks.workunit.client.0.smithi136.stdout:10 2024-09-15T21:41:22.575 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:41:22.576 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-15T21:41:22.576 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:41:22.576 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:41:22.887 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:41:23.889 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:41:23.889 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:41:23.889 INFO:tasks.workunit.client.0.smithi136.stdout:11 2024-09-15T21:41:23.890 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-15T21:41:23.890 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:41:23.890 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:41:24.267 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:41:25.268 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:41:25.268 INFO:tasks.workunit.client.0.smithi136.stdout:12 2024-09-15T21:41:25.269 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:41:25.269 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-15T21:41:25.269 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:41:25.269 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:41:25.564 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:41:26.565 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:41:26.565 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:41:26.565 INFO:tasks.workunit.client.0.smithi136.stdout:13 2024-09-15T21:41:26.566 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-15T21:41:26.566 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:41:26.566 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:41:27.145 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:41:28.147 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:41:28.147 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:41:28.147 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-15T21:41:28.147 INFO:tasks.workunit.client.0.smithi136.stdout:14 2024-09-15T21:41:28.148 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:41:28.148 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:41:28.843 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:41:29.844 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:41:29.844 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:41:29.844 INFO:tasks.workunit.client.0.smithi136.stdout:15 2024-09-15T21:41:29.845 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-15T21:41:29.845 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:41:29.845 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:41:30.293 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:41:31.294 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:41:31.294 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:41:31.295 INFO:tasks.workunit.client.0.smithi136.stdout:16 2024-09-15T21:41:31.295 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-15T21:41:31.295 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:41:31.295 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:41:32.123 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:41:33.125 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:41:33.125 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:41:33.125 INFO:tasks.workunit.client.0.smithi136.stdout:17 2024-09-15T21:41:33.125 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-15T21:41:33.125 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:41:33.126 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:41:33.725 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:41:34.726 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:41:34.727 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:41:34.727 INFO:tasks.workunit.client.0.smithi136.stdout:18 2024-09-15T21:41:34.727 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 18 2024-09-15T21:41:34.727 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:41:34.727 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:41:35.242 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:41:36.243 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:41:36.243 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:41:36.243 INFO:tasks.workunit.client.0.smithi136.stdout:19 2024-09-15T21:41:36.244 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 19 2024-09-15T21:41:36.244 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:41:36.244 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:41:36.548 INFO:tasks.workunit.client.0.smithi136.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/2604923145,v1:127.0.0.1:6803/2604923145] [v2:127.0.0.1:6804/2604923145,v1:127.0.0.1:6805/2604923145] exists,up 26207316-4540-4f5c-9936-c8113d190460 2024-09-15T21:41:36.552 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-15T21:41:36.552 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-15T21:41:36.552 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-15T21:41:36.552 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:666: TEST_repair_stats: for id in $(seq 0 $(expr $OSDS - 1)) 2024-09-15T21:41:36.552 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:667: TEST_repair_stats: run_osd td/osd-scrub-repair 1 --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-09-15T21:41:36.552 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-scrub-repair 2024-09-15T21:41:36.553 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-15T21:41:36.553 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=1 2024-09-15T21:41:36.553 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-15T21:41:36.553 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-scrub-repair/1 2024-09-15T21:41:36.553 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=ce4892c7-6c6d-4b4f-8e6f-07a3e11133e5 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-15T21:41:36.553 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T21:41:36.553 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T21:41:36.553 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T21:41:36.553 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-09-15T21:41:36.553 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-09-15T21:41:36.553 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-15T21:41:36.553 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-15T21:41:36.553 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-15T21:41:36.553 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-15T21:41:36.554 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-15T21:41:36.554 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-15T21:41:36.554 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-15T21:41:36.554 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T21:41:36.554 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52602 2024-09-15T21:41:36.554 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.52602/$cluster-$name.asok' 2024-09-15T21:41:36.554 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.52602/$cluster-$name.asok' 2024-09-15T21:41:36.554 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-15T21:41:36.554 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-15T21:41:36.554 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-15T21:41:36.554 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-15T21:41:36.554 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-15T21:41:36.554 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T21:41:36.555 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T21:41:36.555 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T21:41:36.555 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T21:41:36.555 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-15T21:41:36.555 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+='--osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0' 2024-09-15T21:41:36.555 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-scrub-repair/1 2024-09-15T21:41:36.555 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-15T21:41:36.555 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=1f119f37-cbbe-4de1-9593-d6efe8897aeb 2024-09-15T21:41:36.556 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd1 1f119f37-cbbe-4de1-9593-d6efe8897aeb' 2024-09-15T21:41:36.556 INFO:tasks.workunit.client.0.smithi136.stdout:add osd1 1f119f37-cbbe-4de1-9593-d6efe8897aeb 2024-09-15T21:41:36.556 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-15T21:41:36.571 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQCQVOdm1JD1IRAANcI5iqzopUFI90Ihz2LIAg== 2024-09-15T21:41:36.572 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQCQVOdm1JD1IRAANcI5iqzopUFI90Ihz2LIAg=="}' 2024-09-15T21:41:36.572 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 1f119f37-cbbe-4de1-9593-d6efe8897aeb -i td/osd-scrub-repair/1/new.json 2024-09-15T21:41:36.887 INFO:tasks.workunit.client.0.smithi136.stdout:1 2024-09-15T21:41:36.898 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-scrub-repair/1/new.json 2024-09-15T21:41:36.899 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 1 --fsid=ce4892c7-6c6d-4b4f-8e6f-07a3e11133e5 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.52602/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --mkfs --key AQCQVOdm1JD1IRAANcI5iqzopUFI90Ihz2LIAg== --osd-uuid 1f119f37-cbbe-4de1-9593-d6efe8897aeb 2024-09-15T21:41:36.930 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:41:36.928+0000 7f3262502640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:41:36.963 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:41:36.961+0000 7f3262502640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:41:36.964 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:41:36.962+0000 7f3262502640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:41:36.964 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:41:36.962+0000 7f3262502640 -1 bdev(0x563038005c00 td/osd-scrub-repair/1/block) open stat got: (1) Operation not permitted 2024-09-15T21:41:36.964 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:41:36.962+0000 7f3262502640 -1 bluestore(td/osd-scrub-repair/1) _read_fsid unparsable uuid 2024-09-15T21:41:42.691 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-scrub-repair/1/keyring 2024-09-15T21:41:42.691 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-15T21:41:42.691 INFO:tasks.workunit.client.0.smithi136.stdout:adding osd1 key to auth repository 2024-09-15T21:41:42.692 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd1 key to auth repository 2024-09-15T21:41:42.692 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-scrub-repair/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-15T21:41:43.048 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.1 2024-09-15T21:41:43.048 INFO:tasks.workunit.client.0.smithi136.stdout:start osd.1 2024-09-15T21:41:43.049 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 1 --fsid=ce4892c7-6c6d-4b4f-8e6f-07a3e11133e5 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.52602/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-09-15T21:41:43.049 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-15T21:41:43.049 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-15T21:41:43.049 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-15T21:41:43.077 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:41:43.075+0000 7f9193a28640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:41:43.078 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:41:43.077+0000 7f9193a28640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:41:43.080 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:41:43.078+0000 7f9193a28640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:41:43.325 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 1 2024-09-15T21:41:43.325 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-15T21:41:43.325 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2024-09-15T21:41:43.325 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-15T21:41:43.326 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-15T21:41:43.326 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:41:43.326 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-15T21:41:43.326 INFO:tasks.workunit.client.0.smithi136.stdout:0 2024-09-15T21:41:43.326 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:41:43.327 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T21:41:43.602 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:41:44.603 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:41:44.603 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:41:44.604 INFO:tasks.workunit.client.0.smithi136.stdout:1 2024-09-15T21:41:44.604 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-15T21:41:44.604 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:41:44.604 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T21:41:44.891 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:41:44.938 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:41:44.937+0000 7f9193a28640 -1 Falling back to public interface 2024-09-15T21:41:45.892 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:41:45.893 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:41:45.893 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-15T21:41:45.893 INFO:tasks.workunit.client.0.smithi136.stdout:2 2024-09-15T21:41:45.893 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:41:45.893 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T21:41:46.174 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:41:47.176 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:41:47.176 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:41:47.176 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-15T21:41:47.176 INFO:tasks.workunit.client.0.smithi136.stdout:3 2024-09-15T21:41:47.177 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:41:47.177 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T21:41:47.461 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:41:47.905 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:41:47.900+0000 7f9193a28640 -1 osd.1 0 log_to_monitors true 2024-09-15T21:41:48.462 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:41:48.463 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:41:48.463 INFO:tasks.workunit.client.0.smithi136.stdout:4 2024-09-15T21:41:48.463 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-15T21:41:48.463 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:41:48.463 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T21:41:48.893 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:41:49.894 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:41:49.895 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:41:49.895 INFO:tasks.workunit.client.0.smithi136.stdout:5 2024-09-15T21:41:49.895 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-15T21:41:49.895 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:41:49.895 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T21:41:50.175 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:41:51.176 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:41:51.177 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:41:51.177 INFO:tasks.workunit.client.0.smithi136.stdout:6 2024-09-15T21:41:51.178 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-15T21:41:51.178 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:41:51.178 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T21:41:51.467 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:41:52.468 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:41:52.469 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:41:52.469 INFO:tasks.workunit.client.0.smithi136.stdout:7 2024-09-15T21:41:52.469 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-15T21:41:52.469 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:41:52.469 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T21:41:52.755 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:41:53.757 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:41:53.757 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:41:53.757 INFO:tasks.workunit.client.0.smithi136.stdout:8 2024-09-15T21:41:53.758 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-15T21:41:53.758 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:41:53.758 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T21:41:54.054 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:41:55.055 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:41:55.055 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:41:55.055 INFO:tasks.workunit.client.0.smithi136.stdout:9 2024-09-15T21:41:55.056 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-15T21:41:55.056 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:41:55.056 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T21:41:55.345 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:41:56.347 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:41:56.347 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:41:56.348 INFO:tasks.workunit.client.0.smithi136.stdout:10 2024-09-15T21:41:56.348 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-15T21:41:56.349 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:41:56.349 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T21:41:56.770 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:41:57.772 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:41:57.772 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:41:57.772 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-15T21:41:57.773 INFO:tasks.workunit.client.0.smithi136.stdout:11 2024-09-15T21:41:57.773 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:41:57.773 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T21:41:58.063 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:41:59.065 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:41:59.065 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:41:59.065 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-15T21:41:59.065 INFO:tasks.workunit.client.0.smithi136.stdout:12 2024-09-15T21:41:59.066 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:41:59.066 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T21:41:59.455 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:42:00.457 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:42:00.457 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:42:00.457 INFO:tasks.workunit.client.0.smithi136.stdout:13 2024-09-15T21:42:00.458 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-15T21:42:00.458 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:42:00.458 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T21:42:01.109 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:42:02.111 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:42:02.111 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:42:02.111 INFO:tasks.workunit.client.0.smithi136.stdout:14 2024-09-15T21:42:02.112 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-15T21:42:02.112 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:42:02.112 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T21:42:02.669 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:42:03.671 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:42:03.671 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:42:03.671 INFO:tasks.workunit.client.0.smithi136.stdout:15 2024-09-15T21:42:03.672 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-15T21:42:03.672 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:42:03.672 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T21:42:04.036 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:42:05.038 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:42:05.038 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:42:05.038 INFO:tasks.workunit.client.0.smithi136.stdout:16 2024-09-15T21:42:05.039 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-15T21:42:05.039 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:42:05.039 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T21:42:05.480 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:42:06.481 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:42:06.481 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:42:06.481 INFO:tasks.workunit.client.0.smithi136.stdout:17 2024-09-15T21:42:06.482 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-15T21:42:06.482 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:42:06.482 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T21:42:07.046 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:42:08.048 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:42:08.048 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:42:08.048 INFO:tasks.workunit.client.0.smithi136.stdout:18 2024-09-15T21:42:08.048 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 18 2024-09-15T21:42:08.049 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:42:08.049 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T21:42:08.644 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:42:09.645 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:42:09.645 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:42:09.645 INFO:tasks.workunit.client.0.smithi136.stdout:19 2024-09-15T21:42:09.646 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 19 2024-09-15T21:42:09.646 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:42:09.646 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T21:42:10.328 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:42:11.330 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:42:11.330 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:42:11.330 INFO:tasks.workunit.client.0.smithi136.stdout:20 2024-09-15T21:42:11.331 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 20 2024-09-15T21:42:11.331 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:42:11.331 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T21:42:11.620 INFO:tasks.workunit.client.0.smithi136.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/1137942921,v1:127.0.0.1:6811/1137942921] [v2:127.0.0.1:6812/1137942921,v1:127.0.0.1:6813/1137942921] exists,up 1f119f37-cbbe-4de1-9593-d6efe8897aeb 2024-09-15T21:42:11.621 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-15T21:42:11.621 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-15T21:42:11.621 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-15T21:42:11.621 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:670: TEST_repair_stats: create_pool testpool 1 1 2024-09-15T21:42:11.621 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create testpool 1 1 2024-09-15T21:42:11.989 INFO:tasks.workunit.client.0.smithi136.stderr:pool 'testpool' created 2024-09-15T21:42:12.000 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2024-09-15T21:42:13.001 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:671: TEST_repair_stats: ceph osd pool set testpool size 2 2024-09-15T21:42:13.468 INFO:tasks.workunit.client.0.smithi136.stderr:set pool 1 size to 2 2024-09-15T21:42:13.481 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:672: TEST_repair_stats: wait_for_clean 2024-09-15T21:42:13.481 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-15T21:42:13.481 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-15T21:42:13.481 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-15T21:42:13.481 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-15T21:42:13.482 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-15T21:42:13.482 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-15T21:42:13.482 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-15T21:42:13.482 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-15T21:42:13.482 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-15T21:42:13.578 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-15T21:42:13.578 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-15T21:42:13.578 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-15T21:42:13.578 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-15T21:42:13.578 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-15T21:42:13.578 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-15T21:42:13.980 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-15T21:42:13.980 INFO:tasks.workunit.client.0.smithi136.stderr:1' 2024-09-15T21:42:13.980 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-15T21:42:13.980 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T21:42:13.980 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-15T21:42:14.083 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836491 2024-09-15T21:42:14.083 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836491 2024-09-15T21:42:14.083 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836491' 2024-09-15T21:42:14.083 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T21:42:14.083 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-15T21:42:14.191 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=42949672970 2024-09-15T21:42:14.191 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 42949672970 2024-09-15T21:42:14.191 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836491 1-42949672970' 2024-09-15T21:42:14.191 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T21:42:14.192 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836491 2024-09-15T21:42:14.192 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T21:42:14.192 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-15T21:42:14.193 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836491 2024-09-15T21:42:14.193 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T21:42:14.194 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836491 2024-09-15T21:42:14.194 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836491' 2024-09-15T21:42:14.195 INFO:tasks.workunit.client.0.smithi136.stdout:waiting osd.0 seq 21474836491 2024-09-15T21:42:14.195 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T21:42:14.637 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836490 -lt 21474836491 2024-09-15T21:42:14.637 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-15T21:42:15.638 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-15T21:42:15.638 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T21:42:15.923 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836491 -lt 21474836491 2024-09-15T21:42:15.923 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T21:42:15.923 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-42949672970 2024-09-15T21:42:15.924 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T21:42:15.924 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-15T21:42:15.927 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-42949672970 2024-09-15T21:42:15.927 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T21:42:15.927 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=42949672970 2024-09-15T21:42:15.927 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 42949672970' 2024-09-15T21:42:15.927 INFO:tasks.workunit.client.0.smithi136.stdout:waiting osd.1 seq 42949672970 2024-09-15T21:42:15.927 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-15T21:42:16.202 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 42949672970 -lt 42949672970 2024-09-15T21:42:16.203 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-15T21:42:16.203 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T21:42:16.203 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T21:42:16.548 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 1 == 0 2024-09-15T21:42:16.548 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-15T21:42:16.549 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-15T21:42:16.549 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-15T21:42:16.549 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T21:42:16.549 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T21:42:16.549 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T21:42:16.549 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T21:42:16.822 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=0 2024-09-15T21:42:16.823 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-15T21:42:16.823 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T21:42:16.823 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T21:42:17.166 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 0 = 1 2024-09-15T21:42:17.166 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 0 '!=' -1 2024-09-15T21:42:17.166 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: loop=0 2024-09-15T21:42:17.166 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1652: wait_for_clean: num_active_clean=0 2024-09-15T21:42:17.166 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-15T21:42:17.166 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 0.1 2024-09-15T21:42:17.266 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-15T21:42:17.267 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-15T21:42:17.267 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-15T21:42:17.268 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-15T21:42:17.268 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T21:42:17.268 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T21:42:17.268 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T21:42:17.268 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T21:42:17.554 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=0 2024-09-15T21:42:17.554 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-15T21:42:17.554 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T21:42:17.555 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T21:42:17.898 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 0 = 1 2024-09-15T21:42:17.898 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 0 '!=' 0 2024-09-15T21:42:17.898 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: get_is_making_recovery_progress 2024-09-15T21:42:17.898 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2024-09-15T21:42:17.898 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-15T21:42:17.898 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-15T21:42:17.898 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-15T21:42:17.899 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2024-09-15T21:42:17.899 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-15T21:42:18.243 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=null 2024-09-15T21:42:18.243 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test null '!=' null 2024-09-15T21:42:18.243 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1655: wait_for_clean: (( 1 >= 13 )) 2024-09-15T21:42:18.243 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-15T21:42:18.243 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 0.2 2024-09-15T21:42:18.444 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-15T21:42:18.444 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-15T21:42:18.445 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-15T21:42:18.445 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-15T21:42:18.445 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T21:42:18.445 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T21:42:18.445 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T21:42:18.445 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T21:42:18.718 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=0 2024-09-15T21:42:18.719 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-15T21:42:18.719 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T21:42:18.719 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T21:42:19.065 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 0 = 1 2024-09-15T21:42:19.066 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 0 '!=' 0 2024-09-15T21:42:19.066 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: get_is_making_recovery_progress 2024-09-15T21:42:19.066 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2024-09-15T21:42:19.066 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-15T21:42:19.066 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-15T21:42:19.066 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-15T21:42:19.066 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2024-09-15T21:42:19.066 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-15T21:42:19.405 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=null 2024-09-15T21:42:19.405 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test null '!=' null 2024-09-15T21:42:19.406 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1655: wait_for_clean: (( 2 >= 13 )) 2024-09-15T21:42:19.406 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-15T21:42:19.406 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 0.4 2024-09-15T21:42:19.806 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-15T21:42:19.806 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-15T21:42:19.807 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-15T21:42:19.807 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-15T21:42:19.807 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T21:42:19.807 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T21:42:19.807 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T21:42:19.807 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T21:42:20.078 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=1 2024-09-15T21:42:20.079 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-15T21:42:20.079 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T21:42:20.079 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T21:42:20.423 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 = 1 2024-09-15T21:42:20.423 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-15T21:42:20.423 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-15T21:42:20.424 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:675: TEST_repair_stats: local payload=ABCDEF 2024-09-15T21:42:20.424 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:676: TEST_repair_stats: echo ABCDEF 2024-09-15T21:42:20.424 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:677: TEST_repair_stats: seq 1 30 2024-09-15T21:42:20.424 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:677: TEST_repair_stats: for i in $(seq 1 $OBJS) 2024-09-15T21:42:20.424 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:679: TEST_repair_stats: rados --pool testpool put obj1 td/osd-scrub-repair/ORIGINAL 2024-09-15T21:42:20.566 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:677: TEST_repair_stats: for i in $(seq 1 $OBJS) 2024-09-15T21:42:20.566 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:679: TEST_repair_stats: rados --pool testpool put obj2 td/osd-scrub-repair/ORIGINAL 2024-09-15T21:42:20.725 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:677: TEST_repair_stats: for i in $(seq 1 $OBJS) 2024-09-15T21:42:20.725 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:679: TEST_repair_stats: rados --pool testpool put obj3 td/osd-scrub-repair/ORIGINAL 2024-09-15T21:42:20.859 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:677: TEST_repair_stats: for i in $(seq 1 $OBJS) 2024-09-15T21:42:20.859 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:679: TEST_repair_stats: rados --pool testpool put obj4 td/osd-scrub-repair/ORIGINAL 2024-09-15T21:42:20.985 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:677: TEST_repair_stats: for i in $(seq 1 $OBJS) 2024-09-15T21:42:20.985 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:679: TEST_repair_stats: rados --pool testpool put obj5 td/osd-scrub-repair/ORIGINAL 2024-09-15T21:42:21.127 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:677: TEST_repair_stats: for i in $(seq 1 $OBJS) 2024-09-15T21:42:21.127 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:679: TEST_repair_stats: rados --pool testpool put obj6 td/osd-scrub-repair/ORIGINAL 2024-09-15T21:42:21.278 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:677: TEST_repair_stats: for i in $(seq 1 $OBJS) 2024-09-15T21:42:21.278 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:679: TEST_repair_stats: rados --pool testpool put obj7 td/osd-scrub-repair/ORIGINAL 2024-09-15T21:42:21.412 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:677: TEST_repair_stats: for i in $(seq 1 $OBJS) 2024-09-15T21:42:21.412 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:679: TEST_repair_stats: rados --pool testpool put obj8 td/osd-scrub-repair/ORIGINAL 2024-09-15T21:42:21.546 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:677: TEST_repair_stats: for i in $(seq 1 $OBJS) 2024-09-15T21:42:21.546 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:679: TEST_repair_stats: rados --pool testpool put obj9 td/osd-scrub-repair/ORIGINAL 2024-09-15T21:42:21.671 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:677: TEST_repair_stats: for i in $(seq 1 $OBJS) 2024-09-15T21:42:21.671 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:679: TEST_repair_stats: rados --pool testpool put obj10 td/osd-scrub-repair/ORIGINAL 2024-09-15T21:42:21.805 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:677: TEST_repair_stats: for i in $(seq 1 $OBJS) 2024-09-15T21:42:21.805 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:679: TEST_repair_stats: rados --pool testpool put obj11 td/osd-scrub-repair/ORIGINAL 2024-09-15T21:42:21.930 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:677: TEST_repair_stats: for i in $(seq 1 $OBJS) 2024-09-15T21:42:21.931 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:679: TEST_repair_stats: rados --pool testpool put obj12 td/osd-scrub-repair/ORIGINAL 2024-09-15T21:42:22.056 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:677: TEST_repair_stats: for i in $(seq 1 $OBJS) 2024-09-15T21:42:22.056 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:679: TEST_repair_stats: rados --pool testpool put obj13 td/osd-scrub-repair/ORIGINAL 2024-09-15T21:42:22.182 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:677: TEST_repair_stats: for i in $(seq 1 $OBJS) 2024-09-15T21:42:22.182 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:679: TEST_repair_stats: rados --pool testpool put obj14 td/osd-scrub-repair/ORIGINAL 2024-09-15T21:42:22.308 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:677: TEST_repair_stats: for i in $(seq 1 $OBJS) 2024-09-15T21:42:22.308 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:679: TEST_repair_stats: rados --pool testpool put obj15 td/osd-scrub-repair/ORIGINAL 2024-09-15T21:42:22.467 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:677: TEST_repair_stats: for i in $(seq 1 $OBJS) 2024-09-15T21:42:22.467 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:679: TEST_repair_stats: rados --pool testpool put obj16 td/osd-scrub-repair/ORIGINAL 2024-09-15T21:42:22.600 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:677: TEST_repair_stats: for i in $(seq 1 $OBJS) 2024-09-15T21:42:22.600 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:679: TEST_repair_stats: rados --pool testpool put obj17 td/osd-scrub-repair/ORIGINAL 2024-09-15T21:42:22.734 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:677: TEST_repair_stats: for i in $(seq 1 $OBJS) 2024-09-15T21:42:22.734 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:679: TEST_repair_stats: rados --pool testpool put obj18 td/osd-scrub-repair/ORIGINAL 2024-09-15T21:42:22.859 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:677: TEST_repair_stats: for i in $(seq 1 $OBJS) 2024-09-15T21:42:22.860 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:679: TEST_repair_stats: rados --pool testpool put obj19 td/osd-scrub-repair/ORIGINAL 2024-09-15T21:42:22.977 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:677: TEST_repair_stats: for i in $(seq 1 $OBJS) 2024-09-15T21:42:22.977 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:679: TEST_repair_stats: rados --pool testpool put obj20 td/osd-scrub-repair/ORIGINAL 2024-09-15T21:42:23.111 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:677: TEST_repair_stats: for i in $(seq 1 $OBJS) 2024-09-15T21:42:23.111 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:679: TEST_repair_stats: rados --pool testpool put obj21 td/osd-scrub-repair/ORIGINAL 2024-09-15T21:42:23.253 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:677: TEST_repair_stats: for i in $(seq 1 $OBJS) 2024-09-15T21:42:23.253 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:679: TEST_repair_stats: rados --pool testpool put obj22 td/osd-scrub-repair/ORIGINAL 2024-09-15T21:42:23.387 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:677: TEST_repair_stats: for i in $(seq 1 $OBJS) 2024-09-15T21:42:23.388 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:679: TEST_repair_stats: rados --pool testpool put obj23 td/osd-scrub-repair/ORIGINAL 2024-09-15T21:42:23.513 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:677: TEST_repair_stats: for i in $(seq 1 $OBJS) 2024-09-15T21:42:23.513 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:679: TEST_repair_stats: rados --pool testpool put obj24 td/osd-scrub-repair/ORIGINAL 2024-09-15T21:42:23.638 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:677: TEST_repair_stats: for i in $(seq 1 $OBJS) 2024-09-15T21:42:23.638 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:679: TEST_repair_stats: rados --pool testpool put obj25 td/osd-scrub-repair/ORIGINAL 2024-09-15T21:42:23.756 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:677: TEST_repair_stats: for i in $(seq 1 $OBJS) 2024-09-15T21:42:23.756 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:679: TEST_repair_stats: rados --pool testpool put obj26 td/osd-scrub-repair/ORIGINAL 2024-09-15T21:42:24.090 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:677: TEST_repair_stats: for i in $(seq 1 $OBJS) 2024-09-15T21:42:24.090 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:679: TEST_repair_stats: rados --pool testpool put obj27 td/osd-scrub-repair/ORIGINAL 2024-09-15T21:42:24.216 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:677: TEST_repair_stats: for i in $(seq 1 $OBJS) 2024-09-15T21:42:24.216 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:679: TEST_repair_stats: rados --pool testpool put obj28 td/osd-scrub-repair/ORIGINAL 2024-09-15T21:42:24.342 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:677: TEST_repair_stats: for i in $(seq 1 $OBJS) 2024-09-15T21:42:24.343 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:679: TEST_repair_stats: rados --pool testpool put obj29 td/osd-scrub-repair/ORIGINAL 2024-09-15T21:42:24.466 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:677: TEST_repair_stats: for i in $(seq 1 $OBJS) 2024-09-15T21:42:24.467 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:679: TEST_repair_stats: rados --pool testpool put obj30 td/osd-scrub-repair/ORIGINAL 2024-09-15T21:42:24.602 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:684: TEST_repair_stats: get_not_primary testpool obj1 2024-09-15T21:42:24.602 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1220: get_not_primary: local poolname=testpool 2024-09-15T21:42:24.602 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1221: get_not_primary: local objectname=obj1 2024-09-15T21:42:24.602 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1223: get_not_primary: get_primary testpool obj1 2024-09-15T21:42:24.602 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1187: get_primary: local poolname=testpool 2024-09-15T21:42:24.602 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1188: get_primary: local objectname=obj1 2024-09-15T21:42:24.602 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1190: get_primary: ceph --format json osd map testpool obj1 2024-09-15T21:42:24.602 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1191: get_primary: jq .acting_primary 2024-09-15T21:42:24.880 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1223: get_not_primary: local primary=1 2024-09-15T21:42:24.880 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1224: get_not_primary: ceph --format json osd map testpool obj1 2024-09-15T21:42:24.881 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1225: get_not_primary: jq '.acting | map(select (. != 1)) | .[0]' 2024-09-15T21:42:25.158 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:684: TEST_repair_stats: local other=0 2024-09-15T21:42:25.159 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:685: TEST_repair_stats: get_pg testpool obj1 2024-09-15T21:42:25.159 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1084: get_pg: local poolname=testpool 2024-09-15T21:42:25.159 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1085: get_pg: local objectname=obj1 2024-09-15T21:42:25.159 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1087: get_pg: ceph --format json osd map testpool obj1 2024-09-15T21:42:25.159 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1087: get_pg: jq -r .pgid 2024-09-15T21:42:25.437 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:685: TEST_repair_stats: local pgid=1.0 2024-09-15T21:42:25.437 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:686: TEST_repair_stats: get_primary testpool obj1 2024-09-15T21:42:25.437 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1187: get_primary: local poolname=testpool 2024-09-15T21:42:25.438 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1188: get_primary: local objectname=obj1 2024-09-15T21:42:25.438 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1190: get_primary: ceph --format json osd map testpool obj1 2024-09-15T21:42:25.438 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1191: get_primary: jq .acting_primary 2024-09-15T21:42:25.714 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:686: TEST_repair_stats: local primary=1 2024-09-15T21:42:25.714 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:688: TEST_repair_stats: kill_daemons td/osd-scrub-repair TERM osd.0 2024-09-15T21:42:25.715 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-15T21:42:25.715 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-15T21:42:25.715 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-15T21:42:25.715 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-15T21:42:25.715 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-15T21:42:26.025 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-15T21:42:26.025 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:689: TEST_repair_stats: kill_daemons td/osd-scrub-repair TERM osd.1 2024-09-15T21:42:26.026 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-15T21:42:26.026 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-15T21:42:26.026 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-15T21:42:26.026 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-15T21:42:26.026 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-15T21:42:26.328 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-15T21:42:26.329 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:690: TEST_repair_stats: seq 1 20 2024-09-15T21:42:26.330 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:690: TEST_repair_stats: for i in $(seq 1 $REPAIRS) 2024-09-15T21:42:26.330 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:693: TEST_repair_stats: expr 1 % 2 2024-09-15T21:42:26.332 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:693: TEST_repair_stats: OSD=1 2024-09-15T21:42:26.332 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:694: TEST_repair_stats: _objectstore_tool_nodown td/osd-scrub-repair 1 obj1 remove 2024-09-15T21:42:26.332 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-15T21:42:26.332 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-15T21:42:26.332 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=1 2024-09-15T21:42:26.332 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-15T21:42:26.333 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-09-15T21:42:26.333 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 obj1 remove 2024-09-15T21:42:27.818 INFO:tasks.workunit.client.0.smithi136.stdout:remove #1:ff7b1f36:::obj1:head# 2024-09-15T21:42:28.335 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:690: TEST_repair_stats: for i in $(seq 1 $REPAIRS) 2024-09-15T21:42:28.336 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:693: TEST_repair_stats: expr 2 % 2 2024-09-15T21:42:28.337 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:693: TEST_repair_stats: OSD=0 2024-09-15T21:42:28.338 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:694: TEST_repair_stats: _objectstore_tool_nodown td/osd-scrub-repair 0 obj2 remove 2024-09-15T21:42:28.338 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-15T21:42:28.338 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-15T21:42:28.338 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=0 2024-09-15T21:42:28.338 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-15T21:42:28.338 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-09-15T21:42:28.338 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 obj2 remove 2024-09-15T21:42:29.805 INFO:tasks.workunit.client.0.smithi136.stdout:remove #1:104778fc:::obj2:head# 2024-09-15T21:42:30.320 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:690: TEST_repair_stats: for i in $(seq 1 $REPAIRS) 2024-09-15T21:42:30.320 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:693: TEST_repair_stats: expr 3 % 2 2024-09-15T21:42:30.322 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:693: TEST_repair_stats: OSD=1 2024-09-15T21:42:30.322 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:694: TEST_repair_stats: _objectstore_tool_nodown td/osd-scrub-repair 1 obj3 remove 2024-09-15T21:42:30.322 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-15T21:42:30.322 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-15T21:42:30.322 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=1 2024-09-15T21:42:30.322 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-15T21:42:30.323 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-09-15T21:42:30.323 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 obj3 remove 2024-09-15T21:42:31.541 INFO:tasks.workunit.client.0.smithi136.stdout:remove #1:8dd16f86:::obj3:head# 2024-09-15T21:42:32.055 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:690: TEST_repair_stats: for i in $(seq 1 $REPAIRS) 2024-09-15T21:42:32.056 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:693: TEST_repair_stats: expr 4 % 2 2024-09-15T21:42:32.058 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:693: TEST_repair_stats: OSD=0 2024-09-15T21:42:32.058 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:694: TEST_repair_stats: _objectstore_tool_nodown td/osd-scrub-repair 0 obj4 remove 2024-09-15T21:42:32.058 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-15T21:42:32.058 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-15T21:42:32.058 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=0 2024-09-15T21:42:32.058 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-15T21:42:32.058 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-09-15T21:42:32.058 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 obj4 remove 2024-09-15T21:42:33.084 INFO:tasks.workunit.client.0.smithi136.stdout:remove #1:0ee9ae15:::obj4:head# 2024-09-15T21:42:33.603 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:690: TEST_repair_stats: for i in $(seq 1 $REPAIRS) 2024-09-15T21:42:33.603 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:693: TEST_repair_stats: expr 5 % 2 2024-09-15T21:42:33.603 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:693: TEST_repair_stats: OSD=1 2024-09-15T21:42:33.603 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:694: TEST_repair_stats: _objectstore_tool_nodown td/osd-scrub-repair 1 obj5 remove 2024-09-15T21:42:33.603 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-15T21:42:33.603 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-15T21:42:33.604 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=1 2024-09-15T21:42:33.604 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-15T21:42:33.604 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-09-15T21:42:33.604 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 obj5 remove 2024-09-15T21:42:34.562 INFO:tasks.workunit.client.0.smithi136.stdout:remove #1:c52c9666:::obj5:head# 2024-09-15T21:42:35.083 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:690: TEST_repair_stats: for i in $(seq 1 $REPAIRS) 2024-09-15T21:42:35.084 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:693: TEST_repair_stats: expr 6 % 2 2024-09-15T21:42:35.085 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:693: TEST_repair_stats: OSD=0 2024-09-15T21:42:35.086 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:694: TEST_repair_stats: _objectstore_tool_nodown td/osd-scrub-repair 0 obj6 remove 2024-09-15T21:42:35.086 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-15T21:42:35.086 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-15T21:42:35.086 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=0 2024-09-15T21:42:35.086 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-15T21:42:35.086 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-09-15T21:42:35.086 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 obj6 remove 2024-09-15T21:42:36.056 INFO:tasks.workunit.client.0.smithi136.stdout:remove #1:233a42c1:::obj6:head# 2024-09-15T21:42:36.572 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:690: TEST_repair_stats: for i in $(seq 1 $REPAIRS) 2024-09-15T21:42:36.573 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:693: TEST_repair_stats: expr 7 % 2 2024-09-15T21:42:36.575 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:693: TEST_repair_stats: OSD=1 2024-09-15T21:42:36.575 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:694: TEST_repair_stats: _objectstore_tool_nodown td/osd-scrub-repair 1 obj7 remove 2024-09-15T21:42:36.575 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-15T21:42:36.575 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-15T21:42:36.575 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=1 2024-09-15T21:42:36.575 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-15T21:42:36.575 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-09-15T21:42:36.575 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 obj7 remove 2024-09-15T21:42:37.558 INFO:tasks.workunit.client.0.smithi136.stdout:remove #1:9a09113a:::obj7:head# 2024-09-15T21:42:38.078 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:690: TEST_repair_stats: for i in $(seq 1 $REPAIRS) 2024-09-15T21:42:38.078 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:693: TEST_repair_stats: expr 8 % 2 2024-09-15T21:42:38.080 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:693: TEST_repair_stats: OSD=0 2024-09-15T21:42:38.080 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:694: TEST_repair_stats: _objectstore_tool_nodown td/osd-scrub-repair 0 obj8 remove 2024-09-15T21:42:38.080 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-15T21:42:38.080 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-15T21:42:38.081 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=0 2024-09-15T21:42:38.081 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-15T21:42:38.081 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-09-15T21:42:38.081 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 obj8 remove 2024-09-15T21:42:39.410 INFO:tasks.workunit.client.0.smithi136.stdout:remove #1:a557efb1:::obj8:head# 2024-09-15T21:42:39.925 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:690: TEST_repair_stats: for i in $(seq 1 $REPAIRS) 2024-09-15T21:42:39.925 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:693: TEST_repair_stats: expr 9 % 2 2024-09-15T21:42:39.928 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:693: TEST_repair_stats: OSD=1 2024-09-15T21:42:39.928 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:694: TEST_repair_stats: _objectstore_tool_nodown td/osd-scrub-repair 1 obj9 remove 2024-09-15T21:42:39.928 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-15T21:42:39.928 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-15T21:42:39.928 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=1 2024-09-15T21:42:39.928 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-15T21:42:39.928 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-09-15T21:42:39.928 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 obj9 remove 2024-09-15T21:42:40.845 INFO:tasks.workunit.client.0.smithi136.stdout:remove #1:fe351e27:::obj9:head# 2024-09-15T21:42:41.363 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:690: TEST_repair_stats: for i in $(seq 1 $REPAIRS) 2024-09-15T21:42:41.363 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:693: TEST_repair_stats: expr 10 % 2 2024-09-15T21:42:41.365 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:693: TEST_repair_stats: OSD=0 2024-09-15T21:42:41.365 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:694: TEST_repair_stats: _objectstore_tool_nodown td/osd-scrub-repair 0 obj10 remove 2024-09-15T21:42:41.365 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-15T21:42:41.365 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-15T21:42:41.366 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=0 2024-09-15T21:42:41.366 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-15T21:42:41.366 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-09-15T21:42:41.366 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 obj10 remove 2024-09-15T21:42:42.313 INFO:tasks.workunit.client.0.smithi136.stdout:remove #1:d1337354:::obj10:head# 2024-09-15T21:42:42.828 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:690: TEST_repair_stats: for i in $(seq 1 $REPAIRS) 2024-09-15T21:42:42.828 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:693: TEST_repair_stats: expr 11 % 2 2024-09-15T21:42:42.830 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:693: TEST_repair_stats: OSD=1 2024-09-15T21:42:42.830 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:694: TEST_repair_stats: _objectstore_tool_nodown td/osd-scrub-repair 1 obj11 remove 2024-09-15T21:42:42.830 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-15T21:42:42.830 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-15T21:42:42.830 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=1 2024-09-15T21:42:42.831 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-15T21:42:42.831 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-09-15T21:42:42.831 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 obj11 remove 2024-09-15T21:42:44.007 INFO:tasks.workunit.client.0.smithi136.stdout:remove #1:eeae2a94:::obj11:head# 2024-09-15T21:42:44.772 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:690: TEST_repair_stats: for i in $(seq 1 $REPAIRS) 2024-09-15T21:42:44.773 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:693: TEST_repair_stats: expr 12 % 2 2024-09-15T21:42:44.775 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:693: TEST_repair_stats: OSD=0 2024-09-15T21:42:44.775 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:694: TEST_repair_stats: _objectstore_tool_nodown td/osd-scrub-repair 0 obj12 remove 2024-09-15T21:42:44.775 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-15T21:42:44.775 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-15T21:42:44.775 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=0 2024-09-15T21:42:44.775 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-15T21:42:44.775 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-09-15T21:42:44.775 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 obj12 remove 2024-09-15T21:42:45.710 INFO:tasks.workunit.client.0.smithi136.stdout:remove #1:af5f95cb:::obj12:head# 2024-09-15T21:42:46.478 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:690: TEST_repair_stats: for i in $(seq 1 $REPAIRS) 2024-09-15T21:42:46.478 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:693: TEST_repair_stats: expr 13 % 2 2024-09-15T21:42:46.480 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:693: TEST_repair_stats: OSD=1 2024-09-15T21:42:46.480 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:694: TEST_repair_stats: _objectstore_tool_nodown td/osd-scrub-repair 1 obj13 remove 2024-09-15T21:42:46.480 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-15T21:42:46.481 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-15T21:42:46.481 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=1 2024-09-15T21:42:46.481 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-15T21:42:46.481 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-09-15T21:42:46.481 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 obj13 remove 2024-09-15T21:42:47.489 INFO:tasks.workunit.client.0.smithi136.stdout:remove #1:a61ad63e:::obj13:head# 2024-09-15T21:42:48.003 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:690: TEST_repair_stats: for i in $(seq 1 $REPAIRS) 2024-09-15T21:42:48.004 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:693: TEST_repair_stats: expr 14 % 2 2024-09-15T21:42:48.006 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:693: TEST_repair_stats: OSD=0 2024-09-15T21:42:48.006 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:694: TEST_repair_stats: _objectstore_tool_nodown td/osd-scrub-repair 0 obj14 remove 2024-09-15T21:42:48.006 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-15T21:42:48.007 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-15T21:42:48.007 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=0 2024-09-15T21:42:48.007 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-15T21:42:48.007 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-09-15T21:42:48.007 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 obj14 remove 2024-09-15T21:42:48.898 INFO:tasks.workunit.client.0.smithi136.stdout:remove #1:e0a44829:::obj14:head# 2024-09-15T21:42:49.423 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:690: TEST_repair_stats: for i in $(seq 1 $REPAIRS) 2024-09-15T21:42:49.423 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:693: TEST_repair_stats: expr 15 % 2 2024-09-15T21:42:49.426 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:693: TEST_repair_stats: OSD=1 2024-09-15T21:42:49.426 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:694: TEST_repair_stats: _objectstore_tool_nodown td/osd-scrub-repair 1 obj15 remove 2024-09-15T21:42:49.426 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-15T21:42:49.426 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-15T21:42:49.426 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=1 2024-09-15T21:42:49.426 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-15T21:42:49.426 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-09-15T21:42:49.426 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 obj15 remove 2024-09-15T21:42:50.333 INFO:tasks.workunit.client.0.smithi136.stdout:remove #1:ab946124:::obj15:head# 2024-09-15T21:42:50.848 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:690: TEST_repair_stats: for i in $(seq 1 $REPAIRS) 2024-09-15T21:42:50.848 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:693: TEST_repair_stats: expr 16 % 2 2024-09-15T21:42:50.851 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:693: TEST_repair_stats: OSD=0 2024-09-15T21:42:50.851 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:694: TEST_repair_stats: _objectstore_tool_nodown td/osd-scrub-repair 0 obj16 remove 2024-09-15T21:42:50.851 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-15T21:42:50.851 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-15T21:42:50.851 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=0 2024-09-15T21:42:50.851 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-15T21:42:50.851 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-09-15T21:42:50.851 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 obj16 remove 2024-09-15T21:42:51.993 INFO:tasks.workunit.client.0.smithi136.stdout:remove #1:461f8b5e:::obj16:head# 2024-09-15T21:42:52.508 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:690: TEST_repair_stats: for i in $(seq 1 $REPAIRS) 2024-09-15T21:42:52.508 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:693: TEST_repair_stats: expr 17 % 2 2024-09-15T21:42:52.510 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:693: TEST_repair_stats: OSD=1 2024-09-15T21:42:52.510 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:694: TEST_repair_stats: _objectstore_tool_nodown td/osd-scrub-repair 1 obj17 remove 2024-09-15T21:42:52.511 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-15T21:42:52.511 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-15T21:42:52.511 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=1 2024-09-15T21:42:52.511 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-15T21:42:52.511 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-09-15T21:42:52.511 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 obj17 remove 2024-09-15T21:42:53.520 INFO:tasks.workunit.client.0.smithi136.stdout:remove #1:24fc6e92:::obj17:head# 2024-09-15T21:42:54.035 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:690: TEST_repair_stats: for i in $(seq 1 $REPAIRS) 2024-09-15T21:42:54.035 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:693: TEST_repair_stats: expr 18 % 2 2024-09-15T21:42:54.037 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:693: TEST_repair_stats: OSD=0 2024-09-15T21:42:54.037 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:694: TEST_repair_stats: _objectstore_tool_nodown td/osd-scrub-repair 0 obj18 remove 2024-09-15T21:42:54.037 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-15T21:42:54.038 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-15T21:42:54.038 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=0 2024-09-15T21:42:54.038 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-15T21:42:54.038 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-09-15T21:42:54.038 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 obj18 remove 2024-09-15T21:42:55.338 INFO:tasks.workunit.client.0.smithi136.stdout:remove #1:b9836a99:::obj18:head# 2024-09-15T21:42:55.856 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:690: TEST_repair_stats: for i in $(seq 1 $REPAIRS) 2024-09-15T21:42:55.856 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:693: TEST_repair_stats: expr 19 % 2 2024-09-15T21:42:55.859 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:693: TEST_repair_stats: OSD=1 2024-09-15T21:42:55.859 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:694: TEST_repair_stats: _objectstore_tool_nodown td/osd-scrub-repair 1 obj19 remove 2024-09-15T21:42:55.859 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-15T21:42:55.859 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-15T21:42:55.859 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=1 2024-09-15T21:42:55.859 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-15T21:42:55.859 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-09-15T21:42:55.859 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 obj19 remove 2024-09-15T21:42:56.773 INFO:tasks.workunit.client.0.smithi136.stdout:remove #1:00b31b6c:::obj19:head# 2024-09-15T21:42:57.292 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:690: TEST_repair_stats: for i in $(seq 1 $REPAIRS) 2024-09-15T21:42:57.292 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:693: TEST_repair_stats: expr 20 % 2 2024-09-15T21:42:57.294 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:693: TEST_repair_stats: OSD=0 2024-09-15T21:42:57.294 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:694: TEST_repair_stats: _objectstore_tool_nodown td/osd-scrub-repair 0 obj20 remove 2024-09-15T21:42:57.294 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-15T21:42:57.294 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-15T21:42:57.294 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=0 2024-09-15T21:42:57.295 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-15T21:42:57.295 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-09-15T21:42:57.295 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 obj20 remove 2024-09-15T21:42:58.200 INFO:tasks.workunit.client.0.smithi136.stdout:remove #1:24b5611b:::obj20:head# 2024-09-15T21:42:58.716 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:696: TEST_repair_stats: activate_osd td/osd-scrub-repair 1 --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-09-15T21:42:58.716 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2024-09-15T21:42:58.716 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-15T21:42:58.717 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=1 2024-09-15T21:42:58.717 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-15T21:42:58.717 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-09-15T21:42:58.717 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=ce4892c7-6c6d-4b4f-8e6f-07a3e11133e5 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-15T21:42:58.717 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T21:42:58.717 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T21:42:58.717 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T21:42:58.717 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-09-15T21:42:58.717 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-09-15T21:42:58.717 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-15T21:42:58.717 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-15T21:42:58.717 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-15T21:42:58.717 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-15T21:42:58.717 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-15T21:42:58.718 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-15T21:42:58.718 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-15T21:42:58.718 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T21:42:58.718 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52602 2024-09-15T21:42:58.718 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.52602/$cluster-$name.asok' 2024-09-15T21:42:58.719 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.52602/$cluster-$name.asok' 2024-09-15T21:42:58.719 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-15T21:42:58.719 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-15T21:42:58.719 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-15T21:42:58.719 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T21:42:58.720 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T21:42:58.720 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T21:42:58.720 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T21:42:58.720 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-15T21:42:58.720 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+='--osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0' 2024-09-15T21:42:58.720 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-09-15T21:42:58.720 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.1 2024-09-15T21:42:58.720 INFO:tasks.workunit.client.0.smithi136.stdout:start osd.1 2024-09-15T21:42:58.721 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 1 --fsid=ce4892c7-6c6d-4b4f-8e6f-07a3e11133e5 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.52602/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-09-15T21:42:58.721 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-09-15T21:42:58.721 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 1 = 1 ']' 2024-09-15T21:42:58.721 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-15T21:42:58.721 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-15T21:42:58.721 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-15T21:42:58.751 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:42:58.748+0000 7f9a5880d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:42:58.753 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:42:58.750+0000 7f9a5880d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:42:58.754 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:42:58.751+0000 7f9a5880d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:42:59.008 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 1 2024-09-15T21:42:59.008 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-15T21:42:59.008 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2024-09-15T21:42:59.008 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-15T21:42:59.008 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-15T21:42:59.008 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:42:59.009 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-15T21:42:59.009 INFO:tasks.workunit.client.0.smithi136.stdout:0 2024-09-15T21:42:59.009 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:42:59.009 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T21:42:59.297 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:43:00.298 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:43:00.298 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:43:00.298 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-15T21:43:00.298 INFO:tasks.workunit.client.0.smithi136.stdout:1 2024-09-15T21:43:00.299 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:43:00.299 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T21:43:00.379 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:43:00.377+0000 7f9a5880d640 -1 Falling back to public interface 2024-09-15T21:43:00.574 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:43:01.576 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:43:01.576 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:43:01.576 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-15T21:43:01.576 INFO:tasks.workunit.client.0.smithi136.stdout:2 2024-09-15T21:43:01.576 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:43:01.577 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T21:43:01.863 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:43:02.865 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:43:02.865 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:43:02.865 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-15T21:43:02.865 INFO:tasks.workunit.client.0.smithi136.stdout:3 2024-09-15T21:43:02.866 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:43:02.866 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T21:43:02.952 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:43:02.950+0000 7f9a5880d640 -1 osd.1 17 log_to_monitors true 2024-09-15T21:43:03.191 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:43:04.193 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:43:04.193 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:43:04.193 INFO:tasks.workunit.client.0.smithi136.stdout:4 2024-09-15T21:43:04.194 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-15T21:43:04.194 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:43:04.194 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T21:43:04.477 INFO:tasks.workunit.client.0.smithi136.stdout:osd.1 up in weight 1 up_from 20 up_thru 20 down_at 18 last_clean_interval [10,17) [v2:127.0.0.1:6802/1391141074,v1:127.0.0.1:6803/1391141074] [v2:127.0.0.1:6804/1391141074,v1:127.0.0.1:6805/1391141074] exists,up 1f119f37-cbbe-4de1-9593-d6efe8897aeb 2024-09-15T21:43:04.480 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-15T21:43:04.480 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-15T21:43:04.480 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-15T21:43:04.480 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:697: TEST_repair_stats: activate_osd td/osd-scrub-repair 0 --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-09-15T21:43:04.481 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2024-09-15T21:43:04.481 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-15T21:43:04.481 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=0 2024-09-15T21:43:04.481 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-15T21:43:04.481 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-09-15T21:43:04.481 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=ce4892c7-6c6d-4b4f-8e6f-07a3e11133e5 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-15T21:43:04.481 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T21:43:04.481 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T21:43:04.481 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T21:43:04.481 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-09-15T21:43:04.482 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-09-15T21:43:04.482 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-15T21:43:04.482 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-15T21:43:04.482 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-15T21:43:04.482 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-15T21:43:04.482 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-15T21:43:04.482 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-15T21:43:04.482 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-15T21:43:04.482 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T21:43:04.482 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52602 2024-09-15T21:43:04.482 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.52602/$cluster-$name.asok' 2024-09-15T21:43:04.482 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.52602/$cluster-$name.asok' 2024-09-15T21:43:04.482 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-15T21:43:04.483 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-15T21:43:04.483 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-15T21:43:04.483 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T21:43:04.483 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T21:43:04.483 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T21:43:04.483 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T21:43:04.483 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-15T21:43:04.483 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+='--osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0' 2024-09-15T21:43:04.483 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-09-15T21:43:04.483 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.0 2024-09-15T21:43:04.484 INFO:tasks.workunit.client.0.smithi136.stdout:start osd.0 2024-09-15T21:43:04.485 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 0 --fsid=ce4892c7-6c6d-4b4f-8e6f-07a3e11133e5 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.52602/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-09-15T21:43:04.485 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-09-15T21:43:04.485 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 0 = 0 ']' 2024-09-15T21:43:04.485 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-15T21:43:04.485 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-15T21:43:04.485 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-15T21:43:04.508 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:43:04.506+0000 7f1000a1b640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:43:04.510 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:43:04.508+0000 7f1000a1b640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:43:04.511 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:43:04.509+0000 7f1000a1b640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:43:04.757 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 0 2024-09-15T21:43:04.758 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-15T21:43:04.758 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2024-09-15T21:43:04.758 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-15T21:43:04.758 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-15T21:43:04.758 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:43:04.758 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-15T21:43:04.758 INFO:tasks.workunit.client.0.smithi136.stdout:0 2024-09-15T21:43:04.758 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:43:04.759 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:43:05.122 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:43:05.605 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:43:05.603+0000 7f1000a1b640 -1 Falling back to public interface 2024-09-15T21:43:06.123 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:43:06.123 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:43:06.123 INFO:tasks.workunit.client.0.smithi136.stdout:1 2024-09-15T21:43:06.124 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-15T21:43:06.124 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:43:06.124 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:43:06.408 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:43:07.410 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:43:07.410 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:43:07.410 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-15T21:43:07.410 INFO:tasks.workunit.client.0.smithi136.stdout:2 2024-09-15T21:43:07.411 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:43:07.411 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:43:07.692 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:43:08.445 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:43:08.442+0000 7f1000a1b640 -1 osd.0 16 log_to_monitors true 2024-09-15T21:43:08.693 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:43:08.693 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:43:08.694 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-15T21:43:08.694 INFO:tasks.workunit.client.0.smithi136.stdout:3 2024-09-15T21:43:08.694 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:43:08.694 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:43:09.045 INFO:tasks.workunit.client.0.smithi136.stdout:osd.0 up in weight 1 up_from 23 up_thru 0 down_at 17 last_clean_interval [5,16) [v2:127.0.0.1:6810/1821708676,v1:127.0.0.1:6811/1821708676] [v2:127.0.0.1:6812/1821708676,v1:127.0.0.1:6813/1821708676] exists,up 26207316-4540-4f5c-9936-c8113d190460 2024-09-15T21:43:09.046 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-15T21:43:09.046 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-15T21:43:09.046 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-15T21:43:09.046 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:698: TEST_repair_stats: wait_for_clean 2024-09-15T21:43:09.046 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-15T21:43:09.046 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-15T21:43:09.047 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-15T21:43:09.047 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-15T21:43:09.047 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-15T21:43:09.047 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-15T21:43:09.047 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-15T21:43:09.047 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-15T21:43:09.047 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-15T21:43:09.143 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-15T21:43:09.143 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-15T21:43:09.143 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-15T21:43:09.143 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-15T21:43:09.143 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-15T21:43:09.143 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-15T21:43:09.503 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-15T21:43:09.504 INFO:tasks.workunit.client.0.smithi136.stderr:1' 2024-09-15T21:43:09.504 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-15T21:43:09.504 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T21:43:09.504 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-15T21:43:09.607 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=98784247810 2024-09-15T21:43:09.607 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 98784247810 2024-09-15T21:43:09.607 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-98784247810' 2024-09-15T21:43:09.608 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T21:43:09.608 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-15T21:43:09.712 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=85899345923 2024-09-15T21:43:09.712 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 85899345923 2024-09-15T21:43:09.712 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-98784247810 1-85899345923' 2024-09-15T21:43:09.712 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T21:43:09.713 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-98784247810 2024-09-15T21:43:09.713 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T21:43:09.713 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-15T21:43:09.714 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-98784247810 2024-09-15T21:43:09.714 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T21:43:09.715 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=98784247810 2024-09-15T21:43:09.715 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 98784247810' 2024-09-15T21:43:09.715 INFO:tasks.workunit.client.0.smithi136.stdout:waiting osd.0 seq 98784247810 2024-09-15T21:43:09.715 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T21:43:09.991 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 1 -lt 98784247810 2024-09-15T21:43:09.991 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-15T21:43:10.992 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-15T21:43:10.993 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T21:43:11.291 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 98784247810 -lt 98784247810 2024-09-15T21:43:11.292 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T21:43:11.292 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-85899345923 2024-09-15T21:43:11.292 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T21:43:11.293 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-15T21:43:11.294 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-85899345923 2024-09-15T21:43:11.294 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T21:43:11.295 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=85899345923 2024-09-15T21:43:11.295 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 85899345923' 2024-09-15T21:43:11.295 INFO:tasks.workunit.client.0.smithi136.stdout:waiting osd.1 seq 85899345923 2024-09-15T21:43:11.295 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-15T21:43:11.570 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 85899345923 -lt 85899345923 2024-09-15T21:43:11.571 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-15T21:43:11.571 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T21:43:11.571 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T21:43:11.912 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 1 == 0 2024-09-15T21:43:11.912 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-15T21:43:11.912 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-15T21:43:11.912 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-15T21:43:11.912 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T21:43:11.912 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T21:43:11.912 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T21:43:11.912 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T21:43:12.188 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=1 2024-09-15T21:43:12.188 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-15T21:43:12.188 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T21:43:12.189 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T21:43:12.531 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 = 1 2024-09-15T21:43:12.531 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-15T21:43:12.531 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-15T21:43:12.531 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:700: TEST_repair_stats: repair 1.0 2024-09-15T21:43:12.531 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1846: repair: local pgid=1.0 2024-09-15T21:43:12.532 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1847: repair: get_last_scrub_stamp 1.0 2024-09-15T21:43:12.532 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-15T21:43:12.532 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-15T21:43:12.532 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-15T21:43:12.532 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-15T21:43:12.805 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1847: repair: local last_scrub=2024-09-15T21:42:11.941170+0000 2024-09-15T21:43:12.805 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1848: repair: ceph pg repair 1.0 2024-09-15T21:43:13.068 INFO:tasks.workunit.client.0.smithi136.stderr:instructing pg 1.0 on osd.1 to repair 2024-09-15T21:43:13.080 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1849: repair: wait_for_scrub 1.0 2024-09-15T21:42:11.941170+0000 2024-09-15T21:43:13.080 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: local pgid=1.0 2024-09-15T21:43:13.080 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2018: wait_for_scrub: local last_scrub=2024-09-15T21:42:11.941170+0000 2024-09-15T21:43:13.080 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2019: wait_for_scrub: local sname=last_scrub_stamp 2024-09-15T21:43:13.080 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i=0 )) 2024-09-15T21:43:13.080 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-15T21:43:13.080 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-15T21:43:13.080 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-15T21:43:13.080 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-15T21:43:13.081 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-15T21:43:13.081 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-15T21:43:13.351 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-15T21:42:11.941170+0000 '>' 2024-09-15T21:42:11.941170+0000 2024-09-15T21:43:13.351 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-15T21:43:14.352 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-15T21:43:14.353 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-15T21:43:14.353 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-15T21:43:14.353 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-15T21:43:14.353 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-15T21:43:14.354 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-15T21:43:14.354 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-15T21:43:14.643 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-15T21:42:11.941170+0000 '>' 2024-09-15T21:42:11.941170+0000 2024-09-15T21:43:14.643 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-15T21:43:15.644 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-15T21:43:15.644 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-15T21:43:15.645 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-15T21:43:15.645 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-15T21:43:15.645 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-15T21:43:15.645 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-15T21:43:15.646 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-15T21:43:15.931 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-15T21:42:11.941170+0000 '>' 2024-09-15T21:42:11.941170+0000 2024-09-15T21:43:15.931 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-15T21:43:16.932 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-15T21:43:16.932 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-15T21:43:16.933 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-15T21:43:16.933 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-15T21:43:16.933 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-15T21:43:16.933 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-15T21:43:16.933 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-15T21:43:17.219 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-15T21:42:11.941170+0000 '>' 2024-09-15T21:42:11.941170+0000 2024-09-15T21:43:17.219 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-15T21:43:18.220 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-15T21:43:18.220 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-15T21:43:18.221 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-15T21:43:18.221 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-15T21:43:18.221 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-15T21:43:18.221 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-15T21:43:18.221 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-15T21:43:18.504 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-15T21:42:11.941170+0000 '>' 2024-09-15T21:42:11.941170+0000 2024-09-15T21:43:18.504 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-15T21:43:19.505 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-15T21:43:19.505 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-15T21:43:19.506 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-15T21:43:19.506 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-15T21:43:19.507 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-15T21:43:19.507 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-15T21:43:19.507 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-15T21:43:19.791 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-15T21:43:14.035933+0000 '>' 2024-09-15T21:42:11.941170+0000 2024-09-15T21:43:19.791 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2023: wait_for_scrub: return 0 2024-09-15T21:43:19.791 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1850: repair: sleep 2 2024-09-15T21:43:21.793 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:701: TEST_repair_stats: wait_for_clean 2024-09-15T21:43:21.793 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-15T21:43:21.793 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-15T21:43:21.793 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-15T21:43:21.794 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-15T21:43:21.794 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-15T21:43:21.794 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-15T21:43:21.794 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-15T21:43:21.794 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-15T21:43:21.794 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-15T21:43:21.901 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-15T21:43:21.902 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-15T21:43:21.902 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-15T21:43:21.902 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-15T21:43:21.902 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-15T21:43:21.902 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-15T21:43:22.181 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-15T21:43:22.181 INFO:tasks.workunit.client.0.smithi136.stderr:1' 2024-09-15T21:43:22.181 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-15T21:43:22.181 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T21:43:22.181 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-15T21:43:22.285 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=98784247813 2024-09-15T21:43:22.285 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 98784247813 2024-09-15T21:43:22.285 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-98784247813' 2024-09-15T21:43:22.285 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T21:43:22.285 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-15T21:43:22.388 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=85899345926 2024-09-15T21:43:22.388 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 85899345926 2024-09-15T21:43:22.388 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-98784247813 1-85899345926' 2024-09-15T21:43:22.389 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T21:43:22.389 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-98784247813 2024-09-15T21:43:22.389 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T21:43:22.389 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-15T21:43:22.390 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-98784247813 2024-09-15T21:43:22.390 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T21:43:22.391 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=98784247813 2024-09-15T21:43:22.392 INFO:tasks.workunit.client.0.smithi136.stdout:waiting osd.0 seq 98784247813 2024-09-15T21:43:22.392 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 98784247813' 2024-09-15T21:43:22.392 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T21:43:22.668 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 98784247812 -lt 98784247813 2024-09-15T21:43:22.668 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-15T21:43:23.669 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-15T21:43:23.670 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T21:43:23.957 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 98784247813 -lt 98784247813 2024-09-15T21:43:23.957 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T21:43:23.957 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-85899345926 2024-09-15T21:43:23.957 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T21:43:23.958 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-15T21:43:23.959 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-85899345926 2024-09-15T21:43:23.959 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T21:43:23.960 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=85899345926 2024-09-15T21:43:23.960 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 85899345926' 2024-09-15T21:43:23.960 INFO:tasks.workunit.client.0.smithi136.stdout:waiting osd.1 seq 85899345926 2024-09-15T21:43:23.961 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-15T21:43:24.237 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 85899345926 -lt 85899345926 2024-09-15T21:43:24.237 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-15T21:43:24.237 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T21:43:24.238 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T21:43:24.582 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 1 == 0 2024-09-15T21:43:24.582 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-15T21:43:24.583 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-15T21:43:24.583 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-15T21:43:24.583 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T21:43:24.583 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T21:43:24.583 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T21:43:24.583 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T21:43:24.856 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=1 2024-09-15T21:43:24.856 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-15T21:43:24.856 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T21:43:24.857 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T21:43:25.229 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 = 1 2024-09-15T21:43:25.230 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-15T21:43:25.230 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-15T21:43:25.230 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:702: TEST_repair_stats: ceph pg dump pgs 2024-09-15T21:43:25.491 INFO:tasks.workunit.client.0.smithi136.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-15T21:43:25.491 INFO:tasks.workunit.client.0.smithi136.stdout:1.0 30 0 0 0 0 210 0 0 30 0 30 active+clean 2024-09-15T21:43:14.916246+0000 16'30 24:122 [1,0] 1 [1,0] 1 16'30 2024-09-15T21:43:14.035933+0000 16'30 2024-09-15T21:43:14.035933+0000 0 1 periodic scrub scheduled @ 2024-09-16T21:43:14.035933+0000 20 0 2024-09-15T21:43:25.491 INFO:tasks.workunit.client.0.smithi136.stdout: 2024-09-15T21:43:25.491 INFO:tasks.workunit.client.0.smithi136.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-15T21:43:25.491 INFO:tasks.workunit.client.0.smithi136.stderr:dumped pgs 2024-09-15T21:43:25.503 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:703: TEST_repair_stats: flush_pg_stats 2024-09-15T21:43:25.504 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-15T21:43:25.504 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-15T21:43:25.779 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-15T21:43:25.779 INFO:tasks.workunit.client.0.smithi136.stderr:1' 2024-09-15T21:43:25.779 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-15T21:43:25.779 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T21:43:25.780 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-15T21:43:25.883 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=98784247815 2024-09-15T21:43:25.883 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 98784247815 2024-09-15T21:43:25.883 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-98784247815' 2024-09-15T21:43:25.883 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T21:43:25.883 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-15T21:43:25.985 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=85899345928 2024-09-15T21:43:25.985 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 85899345928 2024-09-15T21:43:25.985 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-98784247815 1-85899345928' 2024-09-15T21:43:25.985 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T21:43:25.986 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-98784247815 2024-09-15T21:43:25.986 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T21:43:25.986 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-15T21:43:25.987 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-98784247815 2024-09-15T21:43:25.987 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T21:43:25.988 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=98784247815 2024-09-15T21:43:25.988 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 98784247815' 2024-09-15T21:43:25.989 INFO:tasks.workunit.client.0.smithi136.stdout:waiting osd.0 seq 98784247815 2024-09-15T21:43:25.989 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T21:43:26.268 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 98784247814 -lt 98784247815 2024-09-15T21:43:26.268 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-15T21:43:27.269 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-15T21:43:27.269 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T21:43:27.553 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 98784247815 -lt 98784247815 2024-09-15T21:43:27.554 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T21:43:27.554 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-85899345928 2024-09-15T21:43:27.554 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T21:43:27.555 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-15T21:43:27.556 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-85899345928 2024-09-15T21:43:27.556 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T21:43:27.557 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=85899345928 2024-09-15T21:43:27.557 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 85899345928' 2024-09-15T21:43:27.557 INFO:tasks.workunit.client.0.smithi136.stdout:waiting osd.1 seq 85899345928 2024-09-15T21:43:27.557 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-15T21:43:27.831 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 85899345928 -lt 85899345928 2024-09-15T21:43:27.832 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:706: TEST_repair_stats: ceph pg 1.0 query 2024-09-15T21:43:27.832 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:706: TEST_repair_stats: jq .info.stats.stat_sum 2024-09-15T21:43:27.936 INFO:tasks.workunit.client.0.smithi136.stdout:{ 2024-09-15T21:43:27.936 INFO:tasks.workunit.client.0.smithi136.stdout: "num_bytes": 210, 2024-09-15T21:43:27.936 INFO:tasks.workunit.client.0.smithi136.stdout: "num_objects": 30, 2024-09-15T21:43:27.936 INFO:tasks.workunit.client.0.smithi136.stdout: "num_object_clones": 0, 2024-09-15T21:43:27.937 INFO:tasks.workunit.client.0.smithi136.stdout: "num_object_copies": 60, 2024-09-15T21:43:27.937 INFO:tasks.workunit.client.0.smithi136.stdout: "num_objects_missing_on_primary": 0, 2024-09-15T21:43:27.937 INFO:tasks.workunit.client.0.smithi136.stdout: "num_objects_missing": 0, 2024-09-15T21:43:27.937 INFO:tasks.workunit.client.0.smithi136.stdout: "num_objects_degraded": 0, 2024-09-15T21:43:27.937 INFO:tasks.workunit.client.0.smithi136.stdout: "num_objects_misplaced": 0, 2024-09-15T21:43:27.937 INFO:tasks.workunit.client.0.smithi136.stdout: "num_objects_unfound": 0, 2024-09-15T21:43:27.937 INFO:tasks.workunit.client.0.smithi136.stdout: "num_objects_dirty": 30, 2024-09-15T21:43:27.937 INFO:tasks.workunit.client.0.smithi136.stdout: "num_whiteouts": 0, 2024-09-15T21:43:27.937 INFO:tasks.workunit.client.0.smithi136.stdout: "num_read": 0, 2024-09-15T21:43:27.937 INFO:tasks.workunit.client.0.smithi136.stdout: "num_read_kb": 0, 2024-09-15T21:43:27.937 INFO:tasks.workunit.client.0.smithi136.stdout: "num_write": 30, 2024-09-15T21:43:27.937 INFO:tasks.workunit.client.0.smithi136.stdout: "num_write_kb": 30, 2024-09-15T21:43:27.937 INFO:tasks.workunit.client.0.smithi136.stdout: "num_scrub_errors": 0, 2024-09-15T21:43:27.937 INFO:tasks.workunit.client.0.smithi136.stdout: "num_shallow_scrub_errors": 0, 2024-09-15T21:43:27.937 INFO:tasks.workunit.client.0.smithi136.stdout: "num_deep_scrub_errors": 0, 2024-09-15T21:43:27.938 INFO:tasks.workunit.client.0.smithi136.stdout: "num_objects_recovered": 20, 2024-09-15T21:43:27.938 INFO:tasks.workunit.client.0.smithi136.stdout: "num_bytes_recovered": 140, 2024-09-15T21:43:27.938 INFO:tasks.workunit.client.0.smithi136.stdout: "num_keys_recovered": 0, 2024-09-15T21:43:27.938 INFO:tasks.workunit.client.0.smithi136.stdout: "num_objects_omap": 0, 2024-09-15T21:43:27.938 INFO:tasks.workunit.client.0.smithi136.stdout: "num_objects_hit_set_archive": 0, 2024-09-15T21:43:27.938 INFO:tasks.workunit.client.0.smithi136.stdout: "num_bytes_hit_set_archive": 0, 2024-09-15T21:43:27.938 INFO:tasks.workunit.client.0.smithi136.stdout: "num_flush": 0, 2024-09-15T21:43:27.938 INFO:tasks.workunit.client.0.smithi136.stdout: "num_flush_kb": 0, 2024-09-15T21:43:27.938 INFO:tasks.workunit.client.0.smithi136.stdout: "num_evict": 0, 2024-09-15T21:43:27.939 INFO:tasks.workunit.client.0.smithi136.stdout: "num_evict_kb": 0, 2024-09-15T21:43:27.939 INFO:tasks.workunit.client.0.smithi136.stdout: "num_promote": 0, 2024-09-15T21:43:27.940 INFO:tasks.workunit.client.0.smithi136.stdout: "num_flush_mode_high": 0, 2024-09-15T21:43:27.940 INFO:tasks.workunit.client.0.smithi136.stdout: "num_flush_mode_low": 0, 2024-09-15T21:43:27.940 INFO:tasks.workunit.client.0.smithi136.stdout: "num_evict_mode_some": 0, 2024-09-15T21:43:27.940 INFO:tasks.workunit.client.0.smithi136.stdout: "num_evict_mode_full": 0, 2024-09-15T21:43:27.940 INFO:tasks.workunit.client.0.smithi136.stdout: "num_objects_pinned": 0, 2024-09-15T21:43:27.940 INFO:tasks.workunit.client.0.smithi136.stdout: "num_legacy_snapsets": 0, 2024-09-15T21:43:27.940 INFO:tasks.workunit.client.0.smithi136.stdout: "num_large_omap_objects": 0, 2024-09-15T21:43:27.940 INFO:tasks.workunit.client.0.smithi136.stdout: "num_objects_manifest": 0, 2024-09-15T21:43:27.940 INFO:tasks.workunit.client.0.smithi136.stdout: "num_omap_bytes": 0, 2024-09-15T21:43:27.940 INFO:tasks.workunit.client.0.smithi136.stdout: "num_omap_keys": 0, 2024-09-15T21:43:27.940 INFO:tasks.workunit.client.0.smithi136.stdout: "num_objects_repaired": 20 2024-09-15T21:43:27.940 INFO:tasks.workunit.client.0.smithi136.stdout:} 2024-09-15T21:43:27.941 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:707: TEST_repair_stats: ceph pg 1.0 query 2024-09-15T21:43:27.941 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:707: TEST_repair_stats: jq .info.stats.stat_sum.num_objects_repaired 2024-09-15T21:43:28.042 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:707: TEST_repair_stats: COUNT=20 2024-09-15T21:43:28.042 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:708: TEST_repair_stats: test 20 = 20 2024-09-15T21:43:28.042 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:710: TEST_repair_stats: ceph pg dump --format=json-pretty 2024-09-15T21:43:28.042 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:710: TEST_repair_stats: jq '.pg_map.osd_stats[] | select(.osd == 1 )' 2024-09-15T21:43:28.300 INFO:tasks.workunit.client.0.smithi136.stderr:dumped all 2024-09-15T21:43:28.311 INFO:tasks.workunit.client.0.smithi136.stdout:{ 2024-09-15T21:43:28.311 INFO:tasks.workunit.client.0.smithi136.stdout: "osd": 1, 2024-09-15T21:43:28.311 INFO:tasks.workunit.client.0.smithi136.stdout: "up_from": 20, 2024-09-15T21:43:28.311 INFO:tasks.workunit.client.0.smithi136.stdout: "seq": 85899345928, 2024-09-15T21:43:28.311 INFO:tasks.workunit.client.0.smithi136.stdout: "num_pgs": 1, 2024-09-15T21:43:28.311 INFO:tasks.workunit.client.0.smithi136.stdout: "num_osds": 1, 2024-09-15T21:43:28.311 INFO:tasks.workunit.client.0.smithi136.stdout: "num_per_pool_osds": 1, 2024-09-15T21:43:28.311 INFO:tasks.workunit.client.0.smithi136.stdout: "num_per_pool_omap_osds": 1, 2024-09-15T21:43:28.311 INFO:tasks.workunit.client.0.smithi136.stdout: "kb": 104857600, 2024-09-15T21:43:28.311 INFO:tasks.workunit.client.0.smithi136.stdout: "kb_used": 28560, 2024-09-15T21:43:28.311 INFO:tasks.workunit.client.0.smithi136.stdout: "kb_used_data": 320, 2024-09-15T21:43:28.312 INFO:tasks.workunit.client.0.smithi136.stdout: "kb_used_omap": 4, 2024-09-15T21:43:28.312 INFO:tasks.workunit.client.0.smithi136.stdout: "kb_used_meta": 28219, 2024-09-15T21:43:28.312 INFO:tasks.workunit.client.0.smithi136.stdout: "kb_avail": 104829040, 2024-09-15T21:43:28.312 INFO:tasks.workunit.client.0.smithi136.stdout: "statfs": { 2024-09-15T21:43:28.312 INFO:tasks.workunit.client.0.smithi136.stdout: "total": 107374182400, 2024-09-15T21:43:28.312 INFO:tasks.workunit.client.0.smithi136.stdout: "available": 107344936960, 2024-09-15T21:43:28.312 INFO:tasks.workunit.client.0.smithi136.stdout: "internally_reserved": 0, 2024-09-15T21:43:28.312 INFO:tasks.workunit.client.0.smithi136.stdout: "allocated": 327680, 2024-09-15T21:43:28.312 INFO:tasks.workunit.client.0.smithi136.stdout: "data_stored": 53958, 2024-09-15T21:43:28.312 INFO:tasks.workunit.client.0.smithi136.stdout: "data_compressed": 0, 2024-09-15T21:43:28.312 INFO:tasks.workunit.client.0.smithi136.stdout: "data_compressed_allocated": 0, 2024-09-15T21:43:28.312 INFO:tasks.workunit.client.0.smithi136.stdout: "data_compressed_original": 0, 2024-09-15T21:43:28.312 INFO:tasks.workunit.client.0.smithi136.stdout: "omap_allocated": 4398, 2024-09-15T21:43:28.312 INFO:tasks.workunit.client.0.smithi136.stdout: "internal_metadata": 28896978 2024-09-15T21:43:28.313 INFO:tasks.workunit.client.0.smithi136.stdout: }, 2024-09-15T21:43:28.313 INFO:tasks.workunit.client.0.smithi136.stdout: "hb_peers": [ 2024-09-15T21:43:28.313 INFO:tasks.workunit.client.0.smithi136.stdout: 0 2024-09-15T21:43:28.313 INFO:tasks.workunit.client.0.smithi136.stdout: ], 2024-09-15T21:43:28.313 INFO:tasks.workunit.client.0.smithi136.stdout: "snap_trim_queue_len": 0, 2024-09-15T21:43:28.313 INFO:tasks.workunit.client.0.smithi136.stdout: "num_snap_trimming": 0, 2024-09-15T21:43:28.313 INFO:tasks.workunit.client.0.smithi136.stdout: "num_shards_repaired": 10, 2024-09-15T21:43:28.313 INFO:tasks.workunit.client.0.smithi136.stdout: "op_queue_age_hist": { 2024-09-15T21:43:28.313 INFO:tasks.workunit.client.0.smithi136.stdout: "histogram": [], 2024-09-15T21:43:28.313 INFO:tasks.workunit.client.0.smithi136.stdout: "upper_bound": 1 2024-09-15T21:43:28.313 INFO:tasks.workunit.client.0.smithi136.stdout: }, 2024-09-15T21:43:28.313 INFO:tasks.workunit.client.0.smithi136.stdout: "perf_stat": { 2024-09-15T21:43:28.313 INFO:tasks.workunit.client.0.smithi136.stdout: "commit_latency_ms": 0, 2024-09-15T21:43:28.313 INFO:tasks.workunit.client.0.smithi136.stdout: "apply_latency_ms": 0, 2024-09-15T21:43:28.313 INFO:tasks.workunit.client.0.smithi136.stdout: "commit_latency_ns": 0, 2024-09-15T21:43:28.314 INFO:tasks.workunit.client.0.smithi136.stdout: "apply_latency_ns": 0 2024-09-15T21:43:28.314 INFO:tasks.workunit.client.0.smithi136.stdout: }, 2024-09-15T21:43:28.314 INFO:tasks.workunit.client.0.smithi136.stdout: "alerts": [] 2024-09-15T21:43:28.314 INFO:tasks.workunit.client.0.smithi136.stdout:} 2024-09-15T21:43:28.314 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:711: TEST_repair_stats: ceph pg dump --format=json-pretty 2024-09-15T21:43:28.314 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:711: TEST_repair_stats: jq '.pg_map.osd_stats[] | select(.osd == 1 ).num_shards_repaired' 2024-09-15T21:43:28.568 INFO:tasks.workunit.client.0.smithi136.stderr:dumped all 2024-09-15T21:43:28.579 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:711: TEST_repair_stats: COUNT=10 2024-09-15T21:43:28.580 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:712: TEST_repair_stats: expr 20 / 2 2024-09-15T21:43:28.581 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:712: TEST_repair_stats: test 10 = 10 2024-09-15T21:43:28.581 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:714: TEST_repair_stats: ceph pg dump --format=json-pretty 2024-09-15T21:43:28.581 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:714: TEST_repair_stats: jq '.pg_map.osd_stats[] | select(.osd == 0 )' 2024-09-15T21:43:28.841 INFO:tasks.workunit.client.0.smithi136.stderr:dumped all 2024-09-15T21:43:28.852 INFO:tasks.workunit.client.0.smithi136.stdout:{ 2024-09-15T21:43:28.852 INFO:tasks.workunit.client.0.smithi136.stdout: "osd": 0, 2024-09-15T21:43:28.852 INFO:tasks.workunit.client.0.smithi136.stdout: "up_from": 23, 2024-09-15T21:43:28.853 INFO:tasks.workunit.client.0.smithi136.stdout: "seq": 98784247815, 2024-09-15T21:43:28.853 INFO:tasks.workunit.client.0.smithi136.stdout: "num_pgs": 1, 2024-09-15T21:43:28.853 INFO:tasks.workunit.client.0.smithi136.stdout: "num_osds": 1, 2024-09-15T21:43:28.853 INFO:tasks.workunit.client.0.smithi136.stdout: "num_per_pool_osds": 1, 2024-09-15T21:43:28.853 INFO:tasks.workunit.client.0.smithi136.stdout: "num_per_pool_omap_osds": 1, 2024-09-15T21:43:28.853 INFO:tasks.workunit.client.0.smithi136.stdout: "kb": 104857600, 2024-09-15T21:43:28.853 INFO:tasks.workunit.client.0.smithi136.stdout: "kb_used": 28560, 2024-09-15T21:43:28.853 INFO:tasks.workunit.client.0.smithi136.stdout: "kb_used_data": 320, 2024-09-15T21:43:28.853 INFO:tasks.workunit.client.0.smithi136.stdout: "kb_used_omap": 4, 2024-09-15T21:43:28.853 INFO:tasks.workunit.client.0.smithi136.stdout: "kb_used_meta": 28219, 2024-09-15T21:43:28.853 INFO:tasks.workunit.client.0.smithi136.stdout: "kb_avail": 104829040, 2024-09-15T21:43:28.853 INFO:tasks.workunit.client.0.smithi136.stdout: "statfs": { 2024-09-15T21:43:28.853 INFO:tasks.workunit.client.0.smithi136.stdout: "total": 107374182400, 2024-09-15T21:43:28.853 INFO:tasks.workunit.client.0.smithi136.stdout: "available": 107344936960, 2024-09-15T21:43:28.854 INFO:tasks.workunit.client.0.smithi136.stdout: "internally_reserved": 0, 2024-09-15T21:43:28.854 INFO:tasks.workunit.client.0.smithi136.stdout: "allocated": 327680, 2024-09-15T21:43:28.854 INFO:tasks.workunit.client.0.smithi136.stdout: "data_stored": 53958, 2024-09-15T21:43:28.854 INFO:tasks.workunit.client.0.smithi136.stdout: "data_compressed": 0, 2024-09-15T21:43:28.854 INFO:tasks.workunit.client.0.smithi136.stdout: "data_compressed_allocated": 0, 2024-09-15T21:43:28.854 INFO:tasks.workunit.client.0.smithi136.stdout: "data_compressed_original": 0, 2024-09-15T21:43:28.854 INFO:tasks.workunit.client.0.smithi136.stdout: "omap_allocated": 4401, 2024-09-15T21:43:28.854 INFO:tasks.workunit.client.0.smithi136.stdout: "internal_metadata": 28896975 2024-09-15T21:43:28.854 INFO:tasks.workunit.client.0.smithi136.stdout: }, 2024-09-15T21:43:28.854 INFO:tasks.workunit.client.0.smithi136.stdout: "hb_peers": [ 2024-09-15T21:43:28.854 INFO:tasks.workunit.client.0.smithi136.stdout: 1 2024-09-15T21:43:28.854 INFO:tasks.workunit.client.0.smithi136.stdout: ], 2024-09-15T21:43:28.854 INFO:tasks.workunit.client.0.smithi136.stdout: "snap_trim_queue_len": 0, 2024-09-15T21:43:28.854 INFO:tasks.workunit.client.0.smithi136.stdout: "num_snap_trimming": 0, 2024-09-15T21:43:28.854 INFO:tasks.workunit.client.0.smithi136.stdout: "num_shards_repaired": 10, 2024-09-15T21:43:28.855 INFO:tasks.workunit.client.0.smithi136.stdout: "op_queue_age_hist": { 2024-09-15T21:43:28.855 INFO:tasks.workunit.client.0.smithi136.stdout: "histogram": [], 2024-09-15T21:43:28.855 INFO:tasks.workunit.client.0.smithi136.stdout: "upper_bound": 1 2024-09-15T21:43:28.855 INFO:tasks.workunit.client.0.smithi136.stdout: }, 2024-09-15T21:43:28.855 INFO:tasks.workunit.client.0.smithi136.stdout: "perf_stat": { 2024-09-15T21:43:28.855 INFO:tasks.workunit.client.0.smithi136.stdout: "commit_latency_ms": 0, 2024-09-15T21:43:28.855 INFO:tasks.workunit.client.0.smithi136.stdout: "apply_latency_ms": 0, 2024-09-15T21:43:28.855 INFO:tasks.workunit.client.0.smithi136.stdout: "commit_latency_ns": 0, 2024-09-15T21:43:28.855 INFO:tasks.workunit.client.0.smithi136.stdout: "apply_latency_ns": 0 2024-09-15T21:43:28.855 INFO:tasks.workunit.client.0.smithi136.stdout: }, 2024-09-15T21:43:28.855 INFO:tasks.workunit.client.0.smithi136.stdout: "alerts": [] 2024-09-15T21:43:28.855 INFO:tasks.workunit.client.0.smithi136.stdout:} 2024-09-15T21:43:28.856 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:715: TEST_repair_stats: ceph pg dump --format=json-pretty 2024-09-15T21:43:28.856 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:715: TEST_repair_stats: jq '.pg_map.osd_stats[] | select(.osd == 0 ).num_shards_repaired' 2024-09-15T21:43:29.112 INFO:tasks.workunit.client.0.smithi136.stderr:dumped all 2024-09-15T21:43:29.124 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:715: TEST_repair_stats: COUNT=10 2024-09-15T21:43:29.125 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:716: TEST_repair_stats: expr 20 / 2 2024-09-15T21:43:29.126 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:716: TEST_repair_stats: test 10 = 10 2024-09-15T21:43:29.126 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:718: TEST_repair_stats: ceph pg dump --format=json-pretty 2024-09-15T21:43:29.126 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:718: TEST_repair_stats: jq .pg_map.osd_stats_sum 2024-09-15T21:43:29.397 INFO:tasks.workunit.client.0.smithi136.stderr:dumped all 2024-09-15T21:43:29.408 INFO:tasks.workunit.client.0.smithi136.stdout:{ 2024-09-15T21:43:29.408 INFO:tasks.workunit.client.0.smithi136.stdout: "up_from": 0, 2024-09-15T21:43:29.409 INFO:tasks.workunit.client.0.smithi136.stdout: "seq": 0, 2024-09-15T21:43:29.409 INFO:tasks.workunit.client.0.smithi136.stdout: "num_pgs": 2, 2024-09-15T21:43:29.409 INFO:tasks.workunit.client.0.smithi136.stdout: "num_osds": 2, 2024-09-15T21:43:29.409 INFO:tasks.workunit.client.0.smithi136.stdout: "num_per_pool_osds": 2, 2024-09-15T21:43:29.409 INFO:tasks.workunit.client.0.smithi136.stdout: "num_per_pool_omap_osds": 2, 2024-09-15T21:43:29.409 INFO:tasks.workunit.client.0.smithi136.stdout: "kb": 209715200, 2024-09-15T21:43:29.409 INFO:tasks.workunit.client.0.smithi136.stdout: "kb_used": 57120, 2024-09-15T21:43:29.409 INFO:tasks.workunit.client.0.smithi136.stdout: "kb_used_data": 640, 2024-09-15T21:43:29.409 INFO:tasks.workunit.client.0.smithi136.stdout: "kb_used_omap": 8, 2024-09-15T21:43:29.409 INFO:tasks.workunit.client.0.smithi136.stdout: "kb_used_meta": 56439, 2024-09-15T21:43:29.409 INFO:tasks.workunit.client.0.smithi136.stdout: "kb_avail": 209658080, 2024-09-15T21:43:29.409 INFO:tasks.workunit.client.0.smithi136.stdout: "statfs": { 2024-09-15T21:43:29.410 INFO:tasks.workunit.client.0.smithi136.stdout: "total": 214748364800, 2024-09-15T21:43:29.410 INFO:tasks.workunit.client.0.smithi136.stdout: "available": 214689873920, 2024-09-15T21:43:29.410 INFO:tasks.workunit.client.0.smithi136.stdout: "internally_reserved": 0, 2024-09-15T21:43:29.410 INFO:tasks.workunit.client.0.smithi136.stdout: "allocated": 655360, 2024-09-15T21:43:29.410 INFO:tasks.workunit.client.0.smithi136.stdout: "data_stored": 107916, 2024-09-15T21:43:29.410 INFO:tasks.workunit.client.0.smithi136.stdout: "data_compressed": 0, 2024-09-15T21:43:29.410 INFO:tasks.workunit.client.0.smithi136.stdout: "data_compressed_allocated": 0, 2024-09-15T21:43:29.410 INFO:tasks.workunit.client.0.smithi136.stdout: "data_compressed_original": 0, 2024-09-15T21:43:29.410 INFO:tasks.workunit.client.0.smithi136.stdout: "omap_allocated": 8799, 2024-09-15T21:43:29.410 INFO:tasks.workunit.client.0.smithi136.stdout: "internal_metadata": 57793953 2024-09-15T21:43:29.410 INFO:tasks.workunit.client.0.smithi136.stdout: }, 2024-09-15T21:43:29.410 INFO:tasks.workunit.client.0.smithi136.stdout: "hb_peers": [], 2024-09-15T21:43:29.410 INFO:tasks.workunit.client.0.smithi136.stdout: "snap_trim_queue_len": 0, 2024-09-15T21:43:29.411 INFO:tasks.workunit.client.0.smithi136.stdout: "num_snap_trimming": 0, 2024-09-15T21:43:29.411 INFO:tasks.workunit.client.0.smithi136.stdout: "num_shards_repaired": 20, 2024-09-15T21:43:29.411 INFO:tasks.workunit.client.0.smithi136.stdout: "op_queue_age_hist": { 2024-09-15T21:43:29.411 INFO:tasks.workunit.client.0.smithi136.stdout: "histogram": [], 2024-09-15T21:43:29.411 INFO:tasks.workunit.client.0.smithi136.stdout: "upper_bound": 1 2024-09-15T21:43:29.411 INFO:tasks.workunit.client.0.smithi136.stdout: }, 2024-09-15T21:43:29.411 INFO:tasks.workunit.client.0.smithi136.stdout: "perf_stat": { 2024-09-15T21:43:29.411 INFO:tasks.workunit.client.0.smithi136.stdout: "commit_latency_ms": 0, 2024-09-15T21:43:29.411 INFO:tasks.workunit.client.0.smithi136.stdout: "apply_latency_ms": 0, 2024-09-15T21:43:29.411 INFO:tasks.workunit.client.0.smithi136.stdout: "commit_latency_ns": 0, 2024-09-15T21:43:29.411 INFO:tasks.workunit.client.0.smithi136.stdout: "apply_latency_ns": 0 2024-09-15T21:43:29.411 INFO:tasks.workunit.client.0.smithi136.stdout: }, 2024-09-15T21:43:29.411 INFO:tasks.workunit.client.0.smithi136.stdout: "alerts": [], 2024-09-15T21:43:29.412 INFO:tasks.workunit.client.0.smithi136.stdout: "network_ping_times": [] 2024-09-15T21:43:29.412 INFO:tasks.workunit.client.0.smithi136.stdout:} 2024-09-15T21:43:29.413 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:719: TEST_repair_stats: ceph pg dump --format=json-pretty 2024-09-15T21:43:29.413 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:719: TEST_repair_stats: jq .pg_map.osd_stats_sum.num_shards_repaired 2024-09-15T21:43:29.668 INFO:tasks.workunit.client.0.smithi136.stderr:dumped all 2024-09-15T21:43:29.681 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:719: TEST_repair_stats: COUNT=20 2024-09-15T21:43:29.681 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:720: TEST_repair_stats: test 20 = 20 2024-09-15T21:43:29.681 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:66: run: teardown td/osd-scrub-repair 2024-09-15T21:43:29.681 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/osd-scrub-repair 2024-09-15T21:43:29.681 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-15T21:43:29.681 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/osd-scrub-repair KILL 2024-09-15T21:43:29.681 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-15T21:43:29.681 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-15T21:43:29.682 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-15T21:43:29.682 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-15T21:43:29.682 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-15T21:43:29.789 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-15T21:43:29.789 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-15T21:43:29.790 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-15T21:43:29.790 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-15T21:43:29.791 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-15T21:43:29.791 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-15T21:43:29.792 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-15T21:43:29.793 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T21:43:29.793 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-15T21:43:29.793 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-15T21:43:29.793 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T21:43:29.794 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-15T21:43:29.795 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-15T21:43:29.795 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/osd-scrub-repair 2024-09-15T21:43:29.809 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-15T21:43:29.809 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T21:43:29.809 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52602 2024-09-15T21:43:29.809 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.52602 2024-09-15T21:43:29.810 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-15T21:43:29.811 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-15T21:43:29.811 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:63: run: for func in $funcs 2024-09-15T21:43:29.811 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:64: run: setup td/osd-scrub-repair 2024-09-15T21:43:29.811 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/osd-scrub-repair 2024-09-15T21:43:29.811 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/osd-scrub-repair 2024-09-15T21:43:29.811 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/osd-scrub-repair 2024-09-15T21:43:29.811 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-15T21:43:29.811 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/osd-scrub-repair KILL 2024-09-15T21:43:29.811 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-15T21:43:29.811 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-15T21:43:29.811 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-15T21:43:29.811 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-15T21:43:29.811 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-15T21:43:29.812 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-15T21:43:29.813 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-15T21:43:29.814 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-15T21:43:29.814 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-15T21:43:29.815 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-15T21:43:29.815 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-15T21:43:29.815 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-15T21:43:29.816 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T21:43:29.816 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-15T21:43:29.817 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-15T21:43:29.817 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T21:43:29.817 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-15T21:43:29.819 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-15T21:43:29.819 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/osd-scrub-repair 2024-09-15T21:43:29.820 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-15T21:43:29.820 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T21:43:29.820 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52602 2024-09-15T21:43:29.820 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.52602 2024-09-15T21:43:29.821 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-15T21:43:29.821 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-15T21:43:29.821 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/osd-scrub-repair 2024-09-15T21:43:29.822 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2024-09-15T21:43:29.822 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T21:43:29.823 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52602 2024-09-15T21:43:29.823 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.52602 2024-09-15T21:43:29.825 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2024-09-15T21:43:29.825 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 4096 -le 1024 ']' 2024-09-15T21:43:29.825 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2024-09-15T21:43:29.826 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/osd-scrub-repair 1' TERM HUP INT 2024-09-15T21:43:29.826 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:65: run: TEST_repair_stats_ec td/osd-scrub-repair 2024-09-15T21:43:29.826 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:724: TEST_repair_stats_ec: local dir=td/osd-scrub-repair 2024-09-15T21:43:29.826 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:725: TEST_repair_stats_ec: local poolname=testpool 2024-09-15T21:43:29.826 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:726: TEST_repair_stats_ec: local OSDS=3 2024-09-15T21:43:29.826 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:727: TEST_repair_stats_ec: local OBJS=30 2024-09-15T21:43:29.826 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:729: TEST_repair_stats_ec: local REPAIRS=26 2024-09-15T21:43:29.826 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:730: TEST_repair_stats_ec: local allow_overwrites=false 2024-09-15T21:43:29.826 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:733: TEST_repair_stats_ec: run_mon td/osd-scrub-repair a 2024-09-15T21:43:29.826 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/osd-scrub-repair 2024-09-15T21:43:29.826 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2024-09-15T21:43:29.826 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2024-09-15T21:43:29.826 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2024-09-15T21:43:29.826 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/osd-scrub-repair/a 2024-09-15T21:43:29.827 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-scrub-repair/a --run-dir=td/osd-scrub-repair 2024-09-15T21:43:30.287 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2024-09-15T21:43:30.287 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-15T21:43:30.287 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-15T21:43:30.287 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-15T21:43:30.287 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T21:43:30.287 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52602 2024-09-15T21:43:30.287 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.52602/$cluster-$name.asok' 2024-09-15T21:43:30.288 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-scrub-repair/a '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.52602/$cluster-$name.asok' --mon-cluster-log-file=td/osd-scrub-repair/log --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-09-15T21:43:30.553 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2024-09-15T21:43:30.555 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2024-09-15T21:43:30.555 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-15T21:43:30.555 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-15T21:43:30.555 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2024-09-15T21:43:30.555 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2024-09-15T21:43:30.555 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-15T21:43:30.555 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-15T21:43:30.555 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-15T21:43:30.556 INFO:tasks.workunit.client.0.smithi136.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-15T21:43:30.556 INFO:tasks.workunit.client.0.smithi136.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T21:43:30.556 INFO:tasks.workunit.client.0.smithi136.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52602 2024-09-15T21:43:30.556 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.52602/ceph-mon.a.asok 2024-09-15T21:43:30.556 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-15T21:43:30.557 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.52602/ceph-mon.a.asok config get fsid 2024-09-15T21:43:30.636 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2024-09-15T21:43:30.636 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-15T21:43:30.636 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-15T21:43:30.636 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2024-09-15T21:43:30.636 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2024-09-15T21:43:30.636 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-15T21:43:30.637 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-15T21:43:30.637 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-15T21:43:30.637 INFO:tasks.workunit.client.0.smithi136.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-15T21:43:30.638 INFO:tasks.workunit.client.0.smithi136.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T21:43:30.638 INFO:tasks.workunit.client.0.smithi136.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52602 2024-09-15T21:43:30.638 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.52602/ceph-mon.a.asok 2024-09-15T21:43:30.638 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-15T21:43:30.638 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.52602/ceph-mon.a.asok config get mon_host 2024-09-15T21:43:30.715 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:734: TEST_repair_stats_ec: run_mgr td/osd-scrub-repair x 2024-09-15T21:43:30.716 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: run_mgr: local dir=td/osd-scrub-repair 2024-09-15T21:43:30.716 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: run_mgr: shift 2024-09-15T21:43:30.716 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: run_mgr: local id=x 2024-09-15T21:43:30.716 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: run_mgr: shift 2024-09-15T21:43:30.716 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:548: run_mgr: local data=td/osd-scrub-repair/x 2024-09-15T21:43:30.716 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:550: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-15T21:43:31.530 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: get_asok_path 2024-09-15T21:43:31.530 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-15T21:43:31.530 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-15T21:43:31.530 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-15T21:43:31.530 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T21:43:31.531 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52602 2024-09-15T21:43:31.531 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.52602/$cluster-$name.asok' 2024-09-15T21:43:31.531 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-15T21:43:31.532 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-scrub-repair/x '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.52602/$cluster-$name.asok' --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-15T21:43:31.559 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:736: TEST_repair_stats_ec: local 'ceph_osd_args=--osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0' 2024-09-15T21:43:31.560 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:737: TEST_repair_stats_ec: expr 3 - 1 2024-09-15T21:43:31.561 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:737: TEST_repair_stats_ec: seq 0 2 2024-09-15T21:43:31.566 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:737: TEST_repair_stats_ec: for id in $(seq 0 $(expr $OSDS - 1)) 2024-09-15T21:43:31.566 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:738: TEST_repair_stats_ec: run_osd td/osd-scrub-repair 0 --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-09-15T21:43:31.566 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-scrub-repair 2024-09-15T21:43:31.566 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-15T21:43:31.566 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=0 2024-09-15T21:43:31.566 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-15T21:43:31.566 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-scrub-repair/0 2024-09-15T21:43:31.566 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=ce4892c7-6c6d-4b4f-8e6f-07a3e11133e5 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-15T21:43:31.566 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T21:43:31.566 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T21:43:31.566 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T21:43:31.566 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-09-15T21:43:31.566 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-09-15T21:43:31.567 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-15T21:43:31.567 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-15T21:43:31.567 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-15T21:43:31.567 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-15T21:43:31.567 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-15T21:43:31.567 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-15T21:43:31.567 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-15T21:43:31.567 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T21:43:31.567 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52602 2024-09-15T21:43:31.567 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.52602/$cluster-$name.asok' 2024-09-15T21:43:31.567 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.52602/$cluster-$name.asok' 2024-09-15T21:43:31.567 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-15T21:43:31.568 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-15T21:43:31.568 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-15T21:43:31.568 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-15T21:43:31.568 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-15T21:43:31.568 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T21:43:31.568 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T21:43:31.568 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T21:43:31.568 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T21:43:31.568 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-15T21:43:31.568 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+='--osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0' 2024-09-15T21:43:31.568 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-scrub-repair/0 2024-09-15T21:43:31.568 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-15T21:43:31.569 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=df860f68-ddd2-4fa6-ac8f-53f9bc8e3435 2024-09-15T21:43:31.569 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd0 df860f68-ddd2-4fa6-ac8f-53f9bc8e3435' 2024-09-15T21:43:31.569 INFO:tasks.workunit.client.0.smithi136.stdout:add osd0 df860f68-ddd2-4fa6-ac8f-53f9bc8e3435 2024-09-15T21:43:31.569 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-15T21:43:31.586 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQADVedmW8nKIhAAofy+XXcbIZeyBKcqPNpMSQ== 2024-09-15T21:43:31.586 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQADVedmW8nKIhAAofy+XXcbIZeyBKcqPNpMSQ=="}' 2024-09-15T21:43:31.586 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new df860f68-ddd2-4fa6-ac8f-53f9bc8e3435 -i td/osd-scrub-repair/0/new.json 2024-09-15T21:43:31.794 INFO:tasks.workunit.client.0.smithi136.stdout:0 2024-09-15T21:43:31.804 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-scrub-repair/0/new.json 2024-09-15T21:43:31.805 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 0 --fsid=ce4892c7-6c6d-4b4f-8e6f-07a3e11133e5 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.52602/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --mkfs --key AQADVedmW8nKIhAAofy+XXcbIZeyBKcqPNpMSQ== --osd-uuid df860f68-ddd2-4fa6-ac8f-53f9bc8e3435 2024-09-15T21:43:31.836 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:43:31.834+0000 7f840400c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:43:31.845 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:43:31.843+0000 7f840400c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:43:31.846 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:43:31.844+0000 7f840400c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:43:31.846 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:43:31.844+0000 7f840400c640 -1 bdev(0x556e6cf4fc00 td/osd-scrub-repair/0/block) open stat got: (1) Operation not permitted 2024-09-15T21:43:31.846 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:43:31.845+0000 7f840400c640 -1 bluestore(td/osd-scrub-repair/0) _read_fsid unparsable uuid 2024-09-15T21:43:37.359 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-scrub-repair/0/keyring 2024-09-15T21:43:37.359 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-15T21:43:37.360 INFO:tasks.workunit.client.0.smithi136.stdout:adding osd0 key to auth repository 2024-09-15T21:43:37.360 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd0 key to auth repository 2024-09-15T21:43:37.360 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-scrub-repair/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-15T21:43:37.725 INFO:tasks.workunit.client.0.smithi136.stdout:start osd.0 2024-09-15T21:43:37.726 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.0 2024-09-15T21:43:37.726 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 0 --fsid=ce4892c7-6c6d-4b4f-8e6f-07a3e11133e5 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.52602/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-09-15T21:43:37.726 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-15T21:43:37.726 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-15T21:43:37.726 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-15T21:43:37.754 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:43:37.752+0000 7f5f05e14640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:43:37.766 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:43:37.764+0000 7f5f05e14640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:43:37.768 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:43:37.765+0000 7f5f05e14640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:43:38.003 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 0 2024-09-15T21:43:38.003 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-15T21:43:38.003 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2024-09-15T21:43:38.003 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-15T21:43:38.003 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-15T21:43:38.003 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:43:38.003 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-15T21:43:38.004 INFO:tasks.workunit.client.0.smithi136.stdout:0 2024-09-15T21:43:38.004 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:43:38.004 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:43:38.277 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:43:39.279 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:43:39.279 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:43:39.279 INFO:tasks.workunit.client.0.smithi136.stdout:1 2024-09-15T21:43:39.279 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-15T21:43:39.280 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:43:39.280 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:43:39.371 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:43:39.370+0000 7f5f05e14640 -1 Falling back to public interface 2024-09-15T21:43:39.571 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:43:40.572 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:43:40.572 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:43:40.572 INFO:tasks.workunit.client.0.smithi136.stdout:2 2024-09-15T21:43:40.573 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-15T21:43:40.573 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:43:40.573 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:43:40.858 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:43:41.860 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:43:41.861 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:43:41.861 INFO:tasks.workunit.client.0.smithi136.stdout:3 2024-09-15T21:43:41.862 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-15T21:43:41.862 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:43:41.862 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:43:42.140 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:43:42.348 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:43:42.345+0000 7f5f05e14640 -1 osd.0 0 log_to_monitors true 2024-09-15T21:43:43.141 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:43:43.141 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:43:43.141 INFO:tasks.workunit.client.0.smithi136.stdout:4 2024-09-15T21:43:43.142 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-15T21:43:43.142 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:43:43.142 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:43:43.549 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:43:44.550 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:43:44.550 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:43:44.550 INFO:tasks.workunit.client.0.smithi136.stdout:5 2024-09-15T21:43:44.551 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-15T21:43:44.551 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:43:44.551 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:43:44.835 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:43:45.836 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:43:45.837 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:43:45.837 INFO:tasks.workunit.client.0.smithi136.stdout:6 2024-09-15T21:43:45.837 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-15T21:43:45.837 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:43:45.838 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:43:46.229 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:43:47.230 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:43:47.230 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:43:47.230 INFO:tasks.workunit.client.0.smithi136.stdout:7 2024-09-15T21:43:47.231 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-15T21:43:47.231 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:43:47.231 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:43:47.520 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:43:48.522 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:43:48.522 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:43:48.522 INFO:tasks.workunit.client.0.smithi136.stdout:8 2024-09-15T21:43:48.522 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-15T21:43:48.523 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:43:48.523 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:43:48.813 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:43:49.814 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:43:49.814 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:43:49.814 INFO:tasks.workunit.client.0.smithi136.stdout:9 2024-09-15T21:43:49.815 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-15T21:43:49.815 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:43:49.815 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:43:50.168 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:43:51.170 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:43:51.170 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:43:51.171 INFO:tasks.workunit.client.0.smithi136.stdout:10 2024-09-15T21:43:51.171 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-15T21:43:51.172 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:43:51.172 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:43:51.457 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:43:52.459 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:43:52.459 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:43:52.459 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-15T21:43:52.459 INFO:tasks.workunit.client.0.smithi136.stdout:11 2024-09-15T21:43:52.459 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:43:52.460 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:43:52.748 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:43:53.749 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:43:53.749 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:43:53.750 INFO:tasks.workunit.client.0.smithi136.stdout:12 2024-09-15T21:43:53.750 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-15T21:43:53.750 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:43:53.750 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:43:54.082 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:43:55.084 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:43:55.084 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:43:55.084 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-15T21:43:55.084 INFO:tasks.workunit.client.0.smithi136.stdout:13 2024-09-15T21:43:55.084 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:43:55.085 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:43:55.643 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:43:56.645 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:43:56.645 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:43:56.645 INFO:tasks.workunit.client.0.smithi136.stdout:14 2024-09-15T21:43:56.645 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-15T21:43:56.646 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:43:56.646 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:43:57.337 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:43:58.338 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:43:58.338 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:43:58.339 INFO:tasks.workunit.client.0.smithi136.stdout:15 2024-09-15T21:43:58.339 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-15T21:43:58.339 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:43:58.339 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:43:59.040 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:44:00.041 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:44:00.041 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:44:00.041 INFO:tasks.workunit.client.0.smithi136.stdout:16 2024-09-15T21:44:00.042 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-15T21:44:00.042 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:44:00.042 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:44:00.459 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:44:01.460 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:44:01.460 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:44:01.461 INFO:tasks.workunit.client.0.smithi136.stdout:17 2024-09-15T21:44:01.461 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-15T21:44:01.461 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:44:01.461 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:44:02.036 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:44:03.037 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:44:03.038 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:44:03.038 INFO:tasks.workunit.client.0.smithi136.stdout:18 2024-09-15T21:44:03.038 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 18 2024-09-15T21:44:03.038 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:44:03.039 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:44:03.847 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:44:04.849 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:44:04.849 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:44:04.849 INFO:tasks.workunit.client.0.smithi136.stdout:19 2024-09-15T21:44:04.849 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 19 2024-09-15T21:44:04.850 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:44:04.850 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:44:05.153 INFO:tasks.workunit.client.0.smithi136.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/1037498718,v1:127.0.0.1:6803/1037498718] [v2:127.0.0.1:6804/1037498718,v1:127.0.0.1:6805/1037498718] exists,up df860f68-ddd2-4fa6-ac8f-53f9bc8e3435 2024-09-15T21:44:05.157 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-15T21:44:05.157 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-15T21:44:05.157 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-15T21:44:05.157 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:737: TEST_repair_stats_ec: for id in $(seq 0 $(expr $OSDS - 1)) 2024-09-15T21:44:05.157 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:738: TEST_repair_stats_ec: run_osd td/osd-scrub-repair 1 --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-09-15T21:44:05.157 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-scrub-repair 2024-09-15T21:44:05.158 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-15T21:44:05.158 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=1 2024-09-15T21:44:05.158 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-15T21:44:05.158 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-scrub-repair/1 2024-09-15T21:44:05.158 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=ce4892c7-6c6d-4b4f-8e6f-07a3e11133e5 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-15T21:44:05.158 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T21:44:05.158 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T21:44:05.158 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T21:44:05.158 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-09-15T21:44:05.158 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-09-15T21:44:05.158 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-15T21:44:05.158 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-15T21:44:05.158 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-15T21:44:05.158 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-15T21:44:05.159 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-15T21:44:05.159 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-15T21:44:05.159 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-15T21:44:05.159 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T21:44:05.159 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52602 2024-09-15T21:44:05.159 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.52602/$cluster-$name.asok' 2024-09-15T21:44:05.159 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.52602/$cluster-$name.asok' 2024-09-15T21:44:05.159 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-15T21:44:05.159 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-15T21:44:05.159 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-15T21:44:05.159 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-15T21:44:05.159 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-15T21:44:05.159 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T21:44:05.159 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T21:44:05.160 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T21:44:05.160 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T21:44:05.160 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-15T21:44:05.160 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+='--osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0' 2024-09-15T21:44:05.160 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-scrub-repair/1 2024-09-15T21:44:05.160 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-15T21:44:05.160 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=bfde3e12-1aa2-4789-a6a6-efd7f99566bf 2024-09-15T21:44:05.161 INFO:tasks.workunit.client.0.smithi136.stdout:add osd1 bfde3e12-1aa2-4789-a6a6-efd7f99566bf 2024-09-15T21:44:05.161 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd1 bfde3e12-1aa2-4789-a6a6-efd7f99566bf' 2024-09-15T21:44:05.161 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-15T21:44:05.177 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQAlVedmUxJyChAAV68SFYcSAP7wnTTqrzTLsg== 2024-09-15T21:44:05.177 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQAlVedmUxJyChAAV68SFYcSAP7wnTTqrzTLsg=="}' 2024-09-15T21:44:05.177 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new bfde3e12-1aa2-4789-a6a6-efd7f99566bf -i td/osd-scrub-repair/1/new.json 2024-09-15T21:44:05.550 INFO:tasks.workunit.client.0.smithi136.stdout:1 2024-09-15T21:44:05.561 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-scrub-repair/1/new.json 2024-09-15T21:44:05.562 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 1 --fsid=ce4892c7-6c6d-4b4f-8e6f-07a3e11133e5 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.52602/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --mkfs --key AQAlVedmUxJyChAAV68SFYcSAP7wnTTqrzTLsg== --osd-uuid bfde3e12-1aa2-4789-a6a6-efd7f99566bf 2024-09-15T21:44:05.593 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:44:05.591+0000 7f0ee5018640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:44:05.643 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:44:05.641+0000 7f0ee5018640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:44:05.644 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:44:05.642+0000 7f0ee5018640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:44:05.644 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:44:05.642+0000 7f0ee5018640 -1 bdev(0x55af04213c00 td/osd-scrub-repair/1/block) open stat got: (1) Operation not permitted 2024-09-15T21:44:05.644 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:44:05.642+0000 7f0ee5018640 -1 bluestore(td/osd-scrub-repair/1) _read_fsid unparsable uuid 2024-09-15T21:44:12.709 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-scrub-repair/1/keyring 2024-09-15T21:44:12.709 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-15T21:44:12.710 INFO:tasks.workunit.client.0.smithi136.stdout:adding osd1 key to auth repository 2024-09-15T21:44:12.711 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd1 key to auth repository 2024-09-15T21:44:12.711 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-scrub-repair/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-15T21:44:13.070 INFO:tasks.workunit.client.0.smithi136.stdout:start osd.1 2024-09-15T21:44:13.071 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.1 2024-09-15T21:44:13.071 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 1 --fsid=ce4892c7-6c6d-4b4f-8e6f-07a3e11133e5 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.52602/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-09-15T21:44:13.071 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-15T21:44:13.071 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-15T21:44:13.072 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-15T21:44:13.099 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:44:13.097+0000 7f0b99c2f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:44:13.101 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:44:13.099+0000 7f0b99c2f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:44:13.102 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:44:13.100+0000 7f0b99c2f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:44:13.348 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 1 2024-09-15T21:44:13.348 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-15T21:44:13.348 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2024-09-15T21:44:13.349 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-15T21:44:13.349 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-15T21:44:13.349 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:44:13.349 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-15T21:44:13.349 INFO:tasks.workunit.client.0.smithi136.stdout:0 2024-09-15T21:44:13.349 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:44:13.349 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T21:44:13.626 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:44:14.474 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:44:14.472+0000 7f0b99c2f640 -1 Falling back to public interface 2024-09-15T21:44:14.627 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:44:14.627 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:44:14.627 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-15T21:44:14.628 INFO:tasks.workunit.client.0.smithi136.stdout:1 2024-09-15T21:44:14.628 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:44:14.628 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T21:44:14.914 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:44:15.915 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:44:15.915 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:44:15.915 INFO:tasks.workunit.client.0.smithi136.stdout:2 2024-09-15T21:44:15.916 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-15T21:44:15.916 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:44:15.916 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T21:44:16.200 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:44:17.201 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:44:17.201 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:44:17.201 INFO:tasks.workunit.client.0.smithi136.stdout:3 2024-09-15T21:44:17.202 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-15T21:44:17.202 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:44:17.202 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T21:44:17.349 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:44:17.347+0000 7f0b99c2f640 -1 osd.1 0 log_to_monitors true 2024-09-15T21:44:17.549 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:44:18.550 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:44:18.551 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:44:18.552 INFO:tasks.workunit.client.0.smithi136.stdout:4 2024-09-15T21:44:18.552 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-15T21:44:18.552 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:44:18.552 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T21:44:18.837 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:44:19.838 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:44:19.838 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:44:19.839 INFO:tasks.workunit.client.0.smithi136.stdout:5 2024-09-15T21:44:19.839 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-15T21:44:19.839 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:44:19.839 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T21:44:20.246 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:44:21.247 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:44:21.248 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:44:21.248 INFO:tasks.workunit.client.0.smithi136.stdout:6 2024-09-15T21:44:21.248 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-15T21:44:21.249 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:44:21.249 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T21:44:21.539 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:44:22.540 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:44:22.540 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:44:22.541 INFO:tasks.workunit.client.0.smithi136.stdout:7 2024-09-15T21:44:22.541 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-15T21:44:22.541 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:44:22.541 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T21:44:22.827 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:44:23.829 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:44:23.829 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:44:23.829 INFO:tasks.workunit.client.0.smithi136.stdout:8 2024-09-15T21:44:23.830 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-15T21:44:23.830 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:44:23.830 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T21:44:24.200 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:44:25.201 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:44:25.202 INFO:tasks.workunit.client.0.smithi136.stdout:9 2024-09-15T21:44:25.202 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:44:25.202 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-15T21:44:25.202 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:44:25.203 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T21:44:25.486 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:44:26.487 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:44:26.488 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:44:26.488 INFO:tasks.workunit.client.0.smithi136.stdout:10 2024-09-15T21:44:26.488 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-15T21:44:26.488 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:44:26.489 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T21:44:26.777 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:44:27.778 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:44:27.778 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:44:27.778 INFO:tasks.workunit.client.0.smithi136.stdout:11 2024-09-15T21:44:27.779 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-15T21:44:27.779 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:44:27.779 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T21:44:28.061 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:44:29.063 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:44:29.063 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:44:29.063 INFO:tasks.workunit.client.0.smithi136.stdout:12 2024-09-15T21:44:29.064 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-15T21:44:29.064 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:44:29.064 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T21:44:29.351 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:44:30.352 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:44:30.352 INFO:tasks.workunit.client.0.smithi136.stdout:13 2024-09-15T21:44:30.353 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:44:30.353 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-15T21:44:30.353 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:44:30.353 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T21:44:30.967 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:44:31.969 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:44:31.969 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:44:31.969 INFO:tasks.workunit.client.0.smithi136.stdout:14 2024-09-15T21:44:31.969 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-15T21:44:31.970 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:44:31.970 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T21:44:32.365 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:44:33.366 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:44:33.366 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:44:33.366 INFO:tasks.workunit.client.0.smithi136.stdout:15 2024-09-15T21:44:33.366 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-15T21:44:33.367 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:44:33.367 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T21:44:33.897 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:44:34.899 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:44:34.899 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:44:34.899 INFO:tasks.workunit.client.0.smithi136.stdout:16 2024-09-15T21:44:34.900 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-15T21:44:34.900 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:44:34.900 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T21:44:35.456 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:44:36.457 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:44:36.458 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:44:36.458 INFO:tasks.workunit.client.0.smithi136.stdout:17 2024-09-15T21:44:36.459 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-15T21:44:36.459 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:44:36.459 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T21:44:37.093 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:44:38.094 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:44:38.094 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:44:38.094 INFO:tasks.workunit.client.0.smithi136.stdout:18 2024-09-15T21:44:38.095 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 18 2024-09-15T21:44:38.095 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:44:38.095 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T21:44:38.409 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:44:39.411 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:44:39.411 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:44:39.411 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 19 2024-09-15T21:44:39.411 INFO:tasks.workunit.client.0.smithi136.stdout:19 2024-09-15T21:44:39.412 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:44:39.412 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T21:44:39.862 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:44:40.863 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:44:40.863 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:44:40.864 INFO:tasks.workunit.client.0.smithi136.stdout:20 2024-09-15T21:44:40.864 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 20 2024-09-15T21:44:40.864 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:44:40.864 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T21:44:41.145 INFO:tasks.workunit.client.0.smithi136.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/1792937982,v1:127.0.0.1:6811/1792937982] [v2:127.0.0.1:6812/1792937982,v1:127.0.0.1:6813/1792937982] exists,up bfde3e12-1aa2-4789-a6a6-efd7f99566bf 2024-09-15T21:44:41.148 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-15T21:44:41.148 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-15T21:44:41.148 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-15T21:44:41.148 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:737: TEST_repair_stats_ec: for id in $(seq 0 $(expr $OSDS - 1)) 2024-09-15T21:44:41.148 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:738: TEST_repair_stats_ec: run_osd td/osd-scrub-repair 2 --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-09-15T21:44:41.149 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-scrub-repair 2024-09-15T21:44:41.149 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-15T21:44:41.149 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=2 2024-09-15T21:44:41.149 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-15T21:44:41.149 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-scrub-repair/2 2024-09-15T21:44:41.149 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=ce4892c7-6c6d-4b4f-8e6f-07a3e11133e5 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-15T21:44:41.149 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T21:44:41.149 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T21:44:41.149 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T21:44:41.149 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/2' 2024-09-15T21:44:41.149 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/2/journal' 2024-09-15T21:44:41.149 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-15T21:44:41.149 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-15T21:44:41.149 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-15T21:44:41.150 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-15T21:44:41.150 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-15T21:44:41.150 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-15T21:44:41.150 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-15T21:44:41.150 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T21:44:41.150 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52602 2024-09-15T21:44:41.150 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.52602/$cluster-$name.asok' 2024-09-15T21:44:41.150 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.52602/$cluster-$name.asok' 2024-09-15T21:44:41.150 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-15T21:44:41.150 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-15T21:44:41.150 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-15T21:44:41.150 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-15T21:44:41.150 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-15T21:44:41.150 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T21:44:41.151 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T21:44:41.151 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T21:44:41.151 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T21:44:41.151 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-15T21:44:41.151 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+='--osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0' 2024-09-15T21:44:41.151 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-scrub-repair/2 2024-09-15T21:44:41.151 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-15T21:44:41.151 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=230daabb-2e56-4efe-9d2c-b93105a92c2e 2024-09-15T21:44:41.152 INFO:tasks.workunit.client.0.smithi136.stdout:add osd2 230daabb-2e56-4efe-9d2c-b93105a92c2e 2024-09-15T21:44:41.152 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd2 230daabb-2e56-4efe-9d2c-b93105a92c2e' 2024-09-15T21:44:41.152 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-15T21:44:41.168 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQBJVedmEMnlCRAAgKDXo0gF5iCNI6Gsdv2Xmw== 2024-09-15T21:44:41.168 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQBJVedmEMnlCRAAgKDXo0gF5iCNI6Gsdv2Xmw=="}' 2024-09-15T21:44:41.168 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 230daabb-2e56-4efe-9d2c-b93105a92c2e -i td/osd-scrub-repair/2/new.json 2024-09-15T21:44:41.565 INFO:tasks.workunit.client.0.smithi136.stdout:2 2024-09-15T21:44:41.576 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-scrub-repair/2/new.json 2024-09-15T21:44:41.577 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 2 --fsid=ce4892c7-6c6d-4b4f-8e6f-07a3e11133e5 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.52602/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --mkfs --key AQBJVedmEMnlCRAAgKDXo0gF5iCNI6Gsdv2Xmw== --osd-uuid 230daabb-2e56-4efe-9d2c-b93105a92c2e 2024-09-15T21:44:41.608 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:44:41.606+0000 7f20e8e0d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:44:41.641 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:44:41.639+0000 7f20e8e0d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:44:41.642 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:44:41.640+0000 7f20e8e0d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:44:41.642 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:44:41.640+0000 7f20e8e0d640 -1 bdev(0x560998f34800 td/osd-scrub-repair/2/block) open stat got: (1) Operation not permitted 2024-09-15T21:44:41.642 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:44:41.640+0000 7f20e8e0d640 -1 bluestore(td/osd-scrub-repair/2) _read_fsid unparsable uuid 2024-09-15T21:44:47.565 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-scrub-repair/2/keyring 2024-09-15T21:44:47.565 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-15T21:44:47.566 INFO:tasks.workunit.client.0.smithi136.stdout:adding osd2 key to auth repository 2024-09-15T21:44:47.566 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd2 key to auth repository 2024-09-15T21:44:47.566 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-scrub-repair/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-15T21:44:47.928 INFO:tasks.workunit.client.0.smithi136.stdout:start osd.2 2024-09-15T21:44:47.929 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.2 2024-09-15T21:44:47.929 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-15T21:44:47.929 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 2 --fsid=ce4892c7-6c6d-4b4f-8e6f-07a3e11133e5 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.52602/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-09-15T21:44:47.929 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-15T21:44:47.929 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-15T21:44:47.956 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:44:47.955+0000 7f31da1c5640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:44:47.958 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:44:47.957+0000 7f31da1c5640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:44:47.959 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:44:47.958+0000 7f31da1c5640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:44:48.205 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 2 2024-09-15T21:44:48.205 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-15T21:44:48.205 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=2 2024-09-15T21:44:48.206 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-15T21:44:48.206 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-15T21:44:48.206 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:44:48.206 INFO:tasks.workunit.client.0.smithi136.stdout:0 2024-09-15T21:44:48.207 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-15T21:44:48.207 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:44:48.207 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T21:44:48.484 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:44:49.485 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:44:49.485 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:44:49.485 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-15T21:44:49.485 INFO:tasks.workunit.client.0.smithi136.stdout:1 2024-09-15T21:44:49.486 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:44:49.486 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T21:44:49.591 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:44:49.590+0000 7f31da1c5640 -1 Falling back to public interface 2024-09-15T21:44:49.771 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:44:50.773 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:44:50.773 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:44:50.773 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-15T21:44:50.773 INFO:tasks.workunit.client.0.smithi136.stdout:2 2024-09-15T21:44:50.774 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:44:50.774 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T21:44:51.055 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:44:52.056 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:44:52.057 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:44:52.057 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-15T21:44:52.057 INFO:tasks.workunit.client.0.smithi136.stdout:3 2024-09-15T21:44:52.057 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:44:52.058 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T21:44:52.340 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:44:52.617 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:44:52.615+0000 7f31da1c5640 -1 osd.2 0 log_to_monitors true 2024-09-15T21:44:53.342 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:44:53.342 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:44:53.342 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-15T21:44:53.342 INFO:tasks.workunit.client.0.smithi136.stdout:4 2024-09-15T21:44:53.343 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:44:53.343 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T21:44:53.623 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:44:54.624 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:44:54.625 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:44:54.625 INFO:tasks.workunit.client.0.smithi136.stdout:5 2024-09-15T21:44:54.625 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-15T21:44:54.625 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:44:54.626 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T21:44:54.915 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:44:55.916 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:44:55.916 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:44:55.917 INFO:tasks.workunit.client.0.smithi136.stdout:6 2024-09-15T21:44:55.917 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-15T21:44:55.917 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:44:55.917 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T21:44:56.282 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:44:57.283 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:44:57.284 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:44:57.284 INFO:tasks.workunit.client.0.smithi136.stdout:7 2024-09-15T21:44:57.284 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-15T21:44:57.284 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:44:57.285 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T21:44:57.638 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:44:58.639 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:44:58.639 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:44:58.639 INFO:tasks.workunit.client.0.smithi136.stdout:8 2024-09-15T21:44:58.640 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-15T21:44:58.640 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:44:58.640 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T21:44:58.925 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:44:59.926 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:44:59.927 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:44:59.927 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-15T21:44:59.927 INFO:tasks.workunit.client.0.smithi136.stdout:9 2024-09-15T21:44:59.927 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:44:59.927 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T21:45:00.287 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:45:01.289 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:45:01.289 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:45:01.289 INFO:tasks.workunit.client.0.smithi136.stdout:10 2024-09-15T21:45:01.289 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-15T21:45:01.290 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:45:01.290 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T21:45:01.682 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:45:02.683 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:45:02.684 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:45:02.684 INFO:tasks.workunit.client.0.smithi136.stdout:11 2024-09-15T21:45:02.684 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-15T21:45:02.684 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:45:02.685 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T21:45:02.968 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:45:03.970 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:45:03.970 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:45:03.970 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-15T21:45:03.970 INFO:tasks.workunit.client.0.smithi136.stdout:12 2024-09-15T21:45:03.971 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:45:03.971 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T21:45:04.367 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:45:05.368 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:45:05.368 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:45:05.369 INFO:tasks.workunit.client.0.smithi136.stdout:13 2024-09-15T21:45:05.369 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-15T21:45:05.369 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:45:05.369 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T21:45:06.081 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:45:07.082 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:45:07.082 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:45:07.083 INFO:tasks.workunit.client.0.smithi136.stdout:14 2024-09-15T21:45:07.083 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-15T21:45:07.083 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:45:07.083 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T21:45:07.412 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:45:08.413 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:45:08.413 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:45:08.414 INFO:tasks.workunit.client.0.smithi136.stdout:15 2024-09-15T21:45:08.414 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-15T21:45:08.414 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:45:08.414 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T21:45:09.126 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:45:10.128 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:45:10.128 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:45:10.128 INFO:tasks.workunit.client.0.smithi136.stdout:16 2024-09-15T21:45:10.129 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-15T21:45:10.129 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:45:10.129 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T21:45:10.929 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:45:11.930 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:45:11.930 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:45:11.930 INFO:tasks.workunit.client.0.smithi136.stdout:17 2024-09-15T21:45:11.931 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-15T21:45:11.931 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:45:11.931 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T21:45:12.723 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:45:13.724 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:45:13.725 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:45:13.725 INFO:tasks.workunit.client.0.smithi136.stdout:18 2024-09-15T21:45:13.725 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 18 2024-09-15T21:45:13.725 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:45:13.726 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T21:45:14.492 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:45:15.493 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:45:15.494 INFO:tasks.workunit.client.0.smithi136.stdout:19 2024-09-15T21:45:15.494 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:45:15.494 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 19 2024-09-15T21:45:15.494 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:45:15.495 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T21:45:15.970 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:45:16.971 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:45:16.971 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:45:16.972 INFO:tasks.workunit.client.0.smithi136.stdout:20 2024-09-15T21:45:16.972 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 20 2024-09-15T21:45:16.972 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:45:16.972 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T21:45:17.252 INFO:tasks.workunit.client.0.smithi136.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/3627214880,v1:127.0.0.1:6819/3627214880] [v2:127.0.0.1:6820/3627214880,v1:127.0.0.1:6821/3627214880] exists,up 230daabb-2e56-4efe-9d2c-b93105a92c2e 2024-09-15T21:45:17.253 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-15T21:45:17.253 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-15T21:45:17.253 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-15T21:45:17.253 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:742: TEST_repair_stats_ec: create_ec_pool testpool false k=2 m=1 2024-09-15T21:45:17.253 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2443: create_ec_pool: local pool_name=testpool 2024-09-15T21:45:17.253 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2444: create_ec_pool: shift 2024-09-15T21:45:17.253 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2445: create_ec_pool: local allow_overwrites=false 2024-09-15T21:45:17.253 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2446: create_ec_pool: shift 2024-09-15T21:45:17.254 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2448: create_ec_pool: ceph osd erasure-code-profile set myprofile crush-failure-domain=osd k=2 m=1 2024-09-15T21:45:17.785 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2450: create_ec_pool: create_pool testpool 1 1 erasure myprofile 2024-09-15T21:45:17.785 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create testpool 1 1 erasure myprofile 2024-09-15T21:45:18.385 INFO:tasks.workunit.client.0.smithi136.stderr:pool 'testpool' created 2024-09-15T21:45:18.397 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2024-09-15T21:45:19.399 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2452: create_ec_pool: '[' false = true ']' 2024-09-15T21:45:19.399 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2456: create_ec_pool: wait_for_clean 2024-09-15T21:45:19.399 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-15T21:45:19.399 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-15T21:45:19.399 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-15T21:45:19.400 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-15T21:45:19.400 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-15T21:45:19.400 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-15T21:45:19.400 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-15T21:45:19.400 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-15T21:45:19.401 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-15T21:45:19.506 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-15T21:45:19.506 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-15T21:45:19.506 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-15T21:45:19.506 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-15T21:45:19.507 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-15T21:45:19.507 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-15T21:45:19.868 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-15T21:45:19.868 INFO:tasks.workunit.client.0.smithi136.stderr:1 2024-09-15T21:45:19.868 INFO:tasks.workunit.client.0.smithi136.stderr:2' 2024-09-15T21:45:19.868 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-15T21:45:19.868 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T21:45:19.868 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-15T21:45:20.078 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836501 2024-09-15T21:45:20.078 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836501 2024-09-15T21:45:20.079 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836501' 2024-09-15T21:45:20.079 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T21:45:20.079 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-15T21:45:20.186 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=42949672974 2024-09-15T21:45:20.187 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 42949672974 2024-09-15T21:45:20.187 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836501 1-42949672974' 2024-09-15T21:45:20.187 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T21:45:20.187 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-15T21:45:20.291 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=64424509445 2024-09-15T21:45:20.291 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 64424509445 2024-09-15T21:45:20.291 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836501 1-42949672974 2-64424509445' 2024-09-15T21:45:20.291 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T21:45:20.292 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836501 2024-09-15T21:45:20.292 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T21:45:20.293 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-15T21:45:20.293 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836501 2024-09-15T21:45:20.293 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T21:45:20.294 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836501 2024-09-15T21:45:20.295 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836501' 2024-09-15T21:45:20.295 INFO:tasks.workunit.client.0.smithi136.stdout:waiting osd.0 seq 21474836501 2024-09-15T21:45:20.295 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T21:45:20.617 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836500 -lt 21474836501 2024-09-15T21:45:20.617 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-15T21:45:21.618 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-15T21:45:21.619 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T21:45:21.931 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836502 -lt 21474836501 2024-09-15T21:45:21.932 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T21:45:21.932 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-42949672974 2024-09-15T21:45:21.932 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T21:45:21.933 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-15T21:45:21.934 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-42949672974 2024-09-15T21:45:21.934 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T21:45:21.935 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=42949672974 2024-09-15T21:45:21.935 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 42949672974' 2024-09-15T21:45:21.936 INFO:tasks.workunit.client.0.smithi136.stdout:waiting osd.1 seq 42949672974 2024-09-15T21:45:21.936 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-15T21:45:22.211 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 42949672975 -lt 42949672974 2024-09-15T21:45:22.212 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T21:45:22.212 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-64424509445 2024-09-15T21:45:22.212 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T21:45:22.213 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-15T21:45:22.214 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-64424509445 2024-09-15T21:45:22.214 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T21:45:22.215 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=64424509445 2024-09-15T21:45:22.215 INFO:tasks.workunit.client.0.smithi136.stdout:waiting osd.2 seq 64424509445 2024-09-15T21:45:22.215 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 64424509445' 2024-09-15T21:45:22.215 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-15T21:45:22.493 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 64424509445 -lt 64424509445 2024-09-15T21:45:22.493 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-15T21:45:22.494 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T21:45:22.494 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T21:45:22.842 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 1 == 0 2024-09-15T21:45:22.843 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-15T21:45:22.843 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-15T21:45:22.843 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-15T21:45:22.843 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T21:45:22.843 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T21:45:22.843 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T21:45:22.843 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T21:45:23.116 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=1 2024-09-15T21:45:23.117 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-15T21:45:23.117 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T21:45:23.117 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T21:45:23.461 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 = 1 2024-09-15T21:45:23.461 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-15T21:45:23.461 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-15T21:45:23.462 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2457: create_ec_pool: return 0 2024-09-15T21:45:23.462 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:745: TEST_repair_stats_ec: local payload=ABCDEF 2024-09-15T21:45:23.462 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:746: TEST_repair_stats_ec: echo ABCDEF 2024-09-15T21:45:23.462 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:747: TEST_repair_stats_ec: seq 1 30 2024-09-15T21:45:23.462 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:747: TEST_repair_stats_ec: for i in $(seq 1 $OBJS) 2024-09-15T21:45:23.462 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:749: TEST_repair_stats_ec: rados --pool testpool put obj1 td/osd-scrub-repair/ORIGINAL 2024-09-15T21:45:23.621 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:747: TEST_repair_stats_ec: for i in $(seq 1 $OBJS) 2024-09-15T21:45:23.621 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:749: TEST_repair_stats_ec: rados --pool testpool put obj2 td/osd-scrub-repair/ORIGINAL 2024-09-15T21:45:23.906 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:747: TEST_repair_stats_ec: for i in $(seq 1 $OBJS) 2024-09-15T21:45:23.906 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:749: TEST_repair_stats_ec: rados --pool testpool put obj3 td/osd-scrub-repair/ORIGINAL 2024-09-15T21:45:24.124 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:747: TEST_repair_stats_ec: for i in $(seq 1 $OBJS) 2024-09-15T21:45:24.124 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:749: TEST_repair_stats_ec: rados --pool testpool put obj4 td/osd-scrub-repair/ORIGINAL 2024-09-15T21:45:24.350 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:747: TEST_repair_stats_ec: for i in $(seq 1 $OBJS) 2024-09-15T21:45:24.351 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:749: TEST_repair_stats_ec: rados --pool testpool put obj5 td/osd-scrub-repair/ORIGINAL 2024-09-15T21:45:24.577 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:747: TEST_repair_stats_ec: for i in $(seq 1 $OBJS) 2024-09-15T21:45:24.577 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:749: TEST_repair_stats_ec: rados --pool testpool put obj6 td/osd-scrub-repair/ORIGINAL 2024-09-15T21:45:24.804 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:747: TEST_repair_stats_ec: for i in $(seq 1 $OBJS) 2024-09-15T21:45:24.804 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:749: TEST_repair_stats_ec: rados --pool testpool put obj7 td/osd-scrub-repair/ORIGINAL 2024-09-15T21:45:25.013 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:747: TEST_repair_stats_ec: for i in $(seq 1 $OBJS) 2024-09-15T21:45:25.013 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:749: TEST_repair_stats_ec: rados --pool testpool put obj8 td/osd-scrub-repair/ORIGINAL 2024-09-15T21:45:25.231 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:747: TEST_repair_stats_ec: for i in $(seq 1 $OBJS) 2024-09-15T21:45:25.231 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:749: TEST_repair_stats_ec: rados --pool testpool put obj9 td/osd-scrub-repair/ORIGINAL 2024-09-15T21:45:25.449 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:747: TEST_repair_stats_ec: for i in $(seq 1 $OBJS) 2024-09-15T21:45:25.449 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:749: TEST_repair_stats_ec: rados --pool testpool put obj10 td/osd-scrub-repair/ORIGINAL 2024-09-15T21:45:25.678 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:747: TEST_repair_stats_ec: for i in $(seq 1 $OBJS) 2024-09-15T21:45:25.679 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:749: TEST_repair_stats_ec: rados --pool testpool put obj11 td/osd-scrub-repair/ORIGINAL 2024-09-15T21:45:25.986 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:747: TEST_repair_stats_ec: for i in $(seq 1 $OBJS) 2024-09-15T21:45:25.986 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:749: TEST_repair_stats_ec: rados --pool testpool put obj12 td/osd-scrub-repair/ORIGINAL 2024-09-15T21:45:26.221 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:747: TEST_repair_stats_ec: for i in $(seq 1 $OBJS) 2024-09-15T21:45:26.221 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:749: TEST_repair_stats_ec: rados --pool testpool put obj13 td/osd-scrub-repair/ORIGINAL 2024-09-15T21:45:26.447 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:747: TEST_repair_stats_ec: for i in $(seq 1 $OBJS) 2024-09-15T21:45:26.447 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:749: TEST_repair_stats_ec: rados --pool testpool put obj14 td/osd-scrub-repair/ORIGINAL 2024-09-15T21:45:26.690 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:747: TEST_repair_stats_ec: for i in $(seq 1 $OBJS) 2024-09-15T21:45:26.690 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:749: TEST_repair_stats_ec: rados --pool testpool put obj15 td/osd-scrub-repair/ORIGINAL 2024-09-15T21:45:26.875 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:747: TEST_repair_stats_ec: for i in $(seq 1 $OBJS) 2024-09-15T21:45:26.875 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:749: TEST_repair_stats_ec: rados --pool testpool put obj16 td/osd-scrub-repair/ORIGINAL 2024-09-15T21:45:27.102 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:747: TEST_repair_stats_ec: for i in $(seq 1 $OBJS) 2024-09-15T21:45:27.144 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:749: TEST_repair_stats_ec: rados --pool testpool put obj17 td/osd-scrub-repair/ORIGINAL 2024-09-15T21:45:27.303 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:747: TEST_repair_stats_ec: for i in $(seq 1 $OBJS) 2024-09-15T21:45:27.303 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:749: TEST_repair_stats_ec: rados --pool testpool put obj18 td/osd-scrub-repair/ORIGINAL 2024-09-15T21:45:27.538 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:747: TEST_repair_stats_ec: for i in $(seq 1 $OBJS) 2024-09-15T21:45:27.538 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:749: TEST_repair_stats_ec: rados --pool testpool put obj19 td/osd-scrub-repair/ORIGINAL 2024-09-15T21:45:27.772 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:747: TEST_repair_stats_ec: for i in $(seq 1 $OBJS) 2024-09-15T21:45:27.773 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:749: TEST_repair_stats_ec: rados --pool testpool put obj20 td/osd-scrub-repair/ORIGINAL 2024-09-15T21:45:28.008 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:747: TEST_repair_stats_ec: for i in $(seq 1 $OBJS) 2024-09-15T21:45:28.008 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:749: TEST_repair_stats_ec: rados --pool testpool put obj21 td/osd-scrub-repair/ORIGINAL 2024-09-15T21:45:28.234 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:747: TEST_repair_stats_ec: for i in $(seq 1 $OBJS) 2024-09-15T21:45:28.234 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:749: TEST_repair_stats_ec: rados --pool testpool put obj22 td/osd-scrub-repair/ORIGINAL 2024-09-15T21:45:28.443 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:747: TEST_repair_stats_ec: for i in $(seq 1 $OBJS) 2024-09-15T21:45:28.444 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:749: TEST_repair_stats_ec: rados --pool testpool put obj23 td/osd-scrub-repair/ORIGINAL 2024-09-15T21:45:28.737 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:747: TEST_repair_stats_ec: for i in $(seq 1 $OBJS) 2024-09-15T21:45:28.737 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:749: TEST_repair_stats_ec: rados --pool testpool put obj24 td/osd-scrub-repair/ORIGINAL 2024-09-15T21:45:28.946 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:747: TEST_repair_stats_ec: for i in $(seq 1 $OBJS) 2024-09-15T21:45:28.947 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:749: TEST_repair_stats_ec: rados --pool testpool put obj25 td/osd-scrub-repair/ORIGINAL 2024-09-15T21:45:29.156 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:747: TEST_repair_stats_ec: for i in $(seq 1 $OBJS) 2024-09-15T21:45:29.157 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:749: TEST_repair_stats_ec: rados --pool testpool put obj26 td/osd-scrub-repair/ORIGINAL 2024-09-15T21:45:29.374 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:747: TEST_repair_stats_ec: for i in $(seq 1 $OBJS) 2024-09-15T21:45:29.374 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:749: TEST_repair_stats_ec: rados --pool testpool put obj27 td/osd-scrub-repair/ORIGINAL 2024-09-15T21:45:29.576 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:747: TEST_repair_stats_ec: for i in $(seq 1 $OBJS) 2024-09-15T21:45:29.576 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:749: TEST_repair_stats_ec: rados --pool testpool put obj28 td/osd-scrub-repair/ORIGINAL 2024-09-15T21:45:29.827 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:747: TEST_repair_stats_ec: for i in $(seq 1 $OBJS) 2024-09-15T21:45:29.827 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:749: TEST_repair_stats_ec: rados --pool testpool put obj29 td/osd-scrub-repair/ORIGINAL 2024-09-15T21:45:30.054 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:747: TEST_repair_stats_ec: for i in $(seq 1 $OBJS) 2024-09-15T21:45:30.054 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:749: TEST_repair_stats_ec: rados --pool testpool put obj30 td/osd-scrub-repair/ORIGINAL 2024-09-15T21:45:30.281 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:754: TEST_repair_stats_ec: get_not_primary testpool obj1 2024-09-15T21:45:30.281 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1220: get_not_primary: local poolname=testpool 2024-09-15T21:45:30.281 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1221: get_not_primary: local objectname=obj1 2024-09-15T21:45:30.281 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1223: get_not_primary: get_primary testpool obj1 2024-09-15T21:45:30.281 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1187: get_primary: local poolname=testpool 2024-09-15T21:45:30.282 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1188: get_primary: local objectname=obj1 2024-09-15T21:45:30.282 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1190: get_primary: ceph --format json osd map testpool obj1 2024-09-15T21:45:30.282 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1191: get_primary: jq .acting_primary 2024-09-15T21:45:30.567 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1223: get_not_primary: local primary=1 2024-09-15T21:45:30.567 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1224: get_not_primary: ceph --format json osd map testpool obj1 2024-09-15T21:45:30.567 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1225: get_not_primary: jq '.acting | map(select (. != 1)) | .[0]' 2024-09-15T21:45:30.845 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:754: TEST_repair_stats_ec: local other=0 2024-09-15T21:45:30.846 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:755: TEST_repair_stats_ec: get_pg testpool obj1 2024-09-15T21:45:30.846 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1084: get_pg: local poolname=testpool 2024-09-15T21:45:30.846 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1085: get_pg: local objectname=obj1 2024-09-15T21:45:30.846 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1087: get_pg: ceph --format json osd map testpool obj1 2024-09-15T21:45:30.846 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1087: get_pg: jq -r .pgid 2024-09-15T21:45:31.126 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:755: TEST_repair_stats_ec: local pgid=1.0 2024-09-15T21:45:31.126 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:756: TEST_repair_stats_ec: get_primary testpool obj1 2024-09-15T21:45:31.126 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1187: get_primary: local poolname=testpool 2024-09-15T21:45:31.126 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1188: get_primary: local objectname=obj1 2024-09-15T21:45:31.126 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1190: get_primary: ceph --format json osd map testpool obj1 2024-09-15T21:45:31.126 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1191: get_primary: jq .acting_primary 2024-09-15T21:45:31.403 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:756: TEST_repair_stats_ec: local primary=1 2024-09-15T21:45:31.403 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:758: TEST_repair_stats_ec: kill_daemons td/osd-scrub-repair TERM osd.0 2024-09-15T21:45:31.403 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-15T21:45:31.403 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-15T21:45:31.403 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-15T21:45:31.403 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-15T21:45:31.403 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-15T21:45:31.710 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-15T21:45:31.710 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:759: TEST_repair_stats_ec: kill_daemons td/osd-scrub-repair TERM osd.1 2024-09-15T21:45:31.710 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-15T21:45:31.710 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-15T21:45:31.711 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-15T21:45:31.711 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-15T21:45:31.711 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-15T21:45:32.018 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-15T21:45:32.018 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:760: TEST_repair_stats_ec: seq 1 26 2024-09-15T21:45:32.019 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:760: TEST_repair_stats_ec: for i in $(seq 1 $REPAIRS) 2024-09-15T21:45:32.019 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: expr 1 % 2 2024-09-15T21:45:32.021 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: OSD=1 2024-09-15T21:45:32.021 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:764: TEST_repair_stats_ec: _objectstore_tool_nodown td/osd-scrub-repair 1 obj1 remove 2024-09-15T21:45:32.021 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-15T21:45:32.021 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-15T21:45:32.022 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=1 2024-09-15T21:45:32.022 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-15T21:45:32.022 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-09-15T21:45:32.022 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 obj1 remove 2024-09-15T21:45:34.042 INFO:tasks.workunit.client.0.smithi136.stdout:remove 0#1:ff7b1f36:::obj1:head# 2024-09-15T21:45:34.557 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:760: TEST_repair_stats_ec: for i in $(seq 1 $REPAIRS) 2024-09-15T21:45:34.558 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: expr 2 % 2 2024-09-15T21:45:34.560 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: OSD=0 2024-09-15T21:45:34.560 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:764: TEST_repair_stats_ec: _objectstore_tool_nodown td/osd-scrub-repair 0 obj2 remove 2024-09-15T21:45:34.560 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-15T21:45:34.560 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-15T21:45:34.560 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=0 2024-09-15T21:45:34.560 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-15T21:45:34.560 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-09-15T21:45:34.560 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 obj2 remove 2024-09-15T21:45:36.081 INFO:tasks.workunit.client.0.smithi136.stdout:remove 1#1:104778fc:::obj2:head# 2024-09-15T21:45:36.595 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:760: TEST_repair_stats_ec: for i in $(seq 1 $REPAIRS) 2024-09-15T21:45:36.595 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: expr 3 % 2 2024-09-15T21:45:36.597 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: OSD=1 2024-09-15T21:45:36.597 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:764: TEST_repair_stats_ec: _objectstore_tool_nodown td/osd-scrub-repair 1 obj3 remove 2024-09-15T21:45:36.597 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-15T21:45:36.597 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-15T21:45:36.597 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=1 2024-09-15T21:45:36.597 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-15T21:45:36.597 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-09-15T21:45:36.598 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 obj3 remove 2024-09-15T21:45:37.677 INFO:tasks.workunit.client.0.smithi136.stdout:remove 0#1:8dd16f86:::obj3:head# 2024-09-15T21:45:38.188 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:760: TEST_repair_stats_ec: for i in $(seq 1 $REPAIRS) 2024-09-15T21:45:38.188 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: expr 4 % 2 2024-09-15T21:45:38.190 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: OSD=0 2024-09-15T21:45:38.190 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:764: TEST_repair_stats_ec: _objectstore_tool_nodown td/osd-scrub-repair 0 obj4 remove 2024-09-15T21:45:38.190 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-15T21:45:38.191 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-15T21:45:38.191 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=0 2024-09-15T21:45:38.191 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-15T21:45:38.191 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-09-15T21:45:38.191 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 obj4 remove 2024-09-15T21:45:39.286 INFO:tasks.workunit.client.0.smithi136.stdout:remove 1#1:0ee9ae15:::obj4:head# 2024-09-15T21:45:39.804 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:760: TEST_repair_stats_ec: for i in $(seq 1 $REPAIRS) 2024-09-15T21:45:39.804 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: expr 5 % 2 2024-09-15T21:45:39.806 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: OSD=1 2024-09-15T21:45:39.806 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:764: TEST_repair_stats_ec: _objectstore_tool_nodown td/osd-scrub-repair 1 obj5 remove 2024-09-15T21:45:39.806 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-15T21:45:39.806 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-15T21:45:39.807 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=1 2024-09-15T21:45:39.807 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-15T21:45:39.807 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-09-15T21:45:39.807 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 obj5 remove 2024-09-15T21:45:40.782 INFO:tasks.workunit.client.0.smithi136.stdout:remove 0#1:c52c9666:::obj5:head# 2024-09-15T21:45:41.294 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:760: TEST_repair_stats_ec: for i in $(seq 1 $REPAIRS) 2024-09-15T21:45:41.294 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: expr 6 % 2 2024-09-15T21:45:41.297 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: OSD=0 2024-09-15T21:45:41.297 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:764: TEST_repair_stats_ec: _objectstore_tool_nodown td/osd-scrub-repair 0 obj6 remove 2024-09-15T21:45:41.297 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-15T21:45:41.297 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-15T21:45:41.297 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=0 2024-09-15T21:45:41.298 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-15T21:45:41.298 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-09-15T21:45:41.298 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 obj6 remove 2024-09-15T21:45:42.241 INFO:tasks.workunit.client.0.smithi136.stdout:remove 1#1:233a42c1:::obj6:head# 2024-09-15T21:45:42.755 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:760: TEST_repair_stats_ec: for i in $(seq 1 $REPAIRS) 2024-09-15T21:45:42.756 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: expr 7 % 2 2024-09-15T21:45:42.758 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: OSD=1 2024-09-15T21:45:42.758 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:764: TEST_repair_stats_ec: _objectstore_tool_nodown td/osd-scrub-repair 1 obj7 remove 2024-09-15T21:45:42.758 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-15T21:45:42.758 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-15T21:45:42.758 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=1 2024-09-15T21:45:42.759 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-15T21:45:42.759 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-09-15T21:45:42.759 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 obj7 remove 2024-09-15T21:45:43.817 INFO:tasks.workunit.client.0.smithi136.stdout:remove 0#1:9a09113a:::obj7:head# 2024-09-15T21:45:44.332 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:760: TEST_repair_stats_ec: for i in $(seq 1 $REPAIRS) 2024-09-15T21:45:44.332 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: expr 8 % 2 2024-09-15T21:45:44.334 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: OSD=0 2024-09-15T21:45:44.334 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:764: TEST_repair_stats_ec: _objectstore_tool_nodown td/osd-scrub-repair 0 obj8 remove 2024-09-15T21:45:44.334 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-15T21:45:44.335 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-15T21:45:44.335 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=0 2024-09-15T21:45:44.335 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-15T21:45:44.335 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-09-15T21:45:44.335 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 obj8 remove 2024-09-15T21:45:45.253 INFO:tasks.workunit.client.0.smithi136.stdout:remove 1#1:a557efb1:::obj8:head# 2024-09-15T21:45:45.769 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:760: TEST_repair_stats_ec: for i in $(seq 1 $REPAIRS) 2024-09-15T21:45:45.769 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: expr 9 % 2 2024-09-15T21:45:45.771 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: OSD=1 2024-09-15T21:45:45.772 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:764: TEST_repair_stats_ec: _objectstore_tool_nodown td/osd-scrub-repair 1 obj9 remove 2024-09-15T21:45:45.772 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-15T21:45:45.772 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-15T21:45:45.772 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=1 2024-09-15T21:45:45.772 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-15T21:45:45.772 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-09-15T21:45:45.772 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 obj9 remove 2024-09-15T21:45:46.697 INFO:tasks.workunit.client.0.smithi136.stdout:remove 0#1:fe351e27:::obj9:head# 2024-09-15T21:45:47.214 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:760: TEST_repair_stats_ec: for i in $(seq 1 $REPAIRS) 2024-09-15T21:45:47.214 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: expr 10 % 2 2024-09-15T21:45:47.216 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: OSD=0 2024-09-15T21:45:47.216 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:764: TEST_repair_stats_ec: _objectstore_tool_nodown td/osd-scrub-repair 0 obj10 remove 2024-09-15T21:45:47.217 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-15T21:45:47.217 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-15T21:45:47.217 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=0 2024-09-15T21:45:47.217 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-15T21:45:47.217 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-09-15T21:45:47.217 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 obj10 remove 2024-09-15T21:45:48.174 INFO:tasks.workunit.client.0.smithi136.stdout:remove 1#1:d1337354:::obj10:head# 2024-09-15T21:45:48.688 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:760: TEST_repair_stats_ec: for i in $(seq 1 $REPAIRS) 2024-09-15T21:45:48.689 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: expr 11 % 2 2024-09-15T21:45:48.691 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: OSD=1 2024-09-15T21:45:48.691 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:764: TEST_repair_stats_ec: _objectstore_tool_nodown td/osd-scrub-repair 1 obj11 remove 2024-09-15T21:45:48.691 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-15T21:45:48.691 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-15T21:45:48.691 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=1 2024-09-15T21:45:48.691 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-15T21:45:48.691 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-09-15T21:45:48.691 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 obj11 remove 2024-09-15T21:45:49.684 INFO:tasks.workunit.client.0.smithi136.stdout:remove 0#1:eeae2a94:::obj11:head# 2024-09-15T21:45:50.450 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:760: TEST_repair_stats_ec: for i in $(seq 1 $REPAIRS) 2024-09-15T21:45:50.450 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: expr 12 % 2 2024-09-15T21:45:50.452 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: OSD=0 2024-09-15T21:45:50.452 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:764: TEST_repair_stats_ec: _objectstore_tool_nodown td/osd-scrub-repair 0 obj12 remove 2024-09-15T21:45:50.453 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-15T21:45:50.453 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-15T21:45:50.453 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=0 2024-09-15T21:45:50.453 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-15T21:45:50.453 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-09-15T21:45:50.453 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 obj12 remove 2024-09-15T21:45:51.388 INFO:tasks.workunit.client.0.smithi136.stdout:remove 1#1:af5f95cb:::obj12:head# 2024-09-15T21:45:52.153 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:760: TEST_repair_stats_ec: for i in $(seq 1 $REPAIRS) 2024-09-15T21:45:52.154 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: expr 13 % 2 2024-09-15T21:45:52.155 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: OSD=1 2024-09-15T21:45:52.156 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:764: TEST_repair_stats_ec: _objectstore_tool_nodown td/osd-scrub-repair 1 obj13 remove 2024-09-15T21:45:52.156 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-15T21:45:52.156 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-15T21:45:52.156 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=1 2024-09-15T21:45:52.156 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-15T21:45:52.156 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-09-15T21:45:52.156 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 obj13 remove 2024-09-15T21:45:53.051 INFO:tasks.workunit.client.0.smithi136.stdout:remove 0#1:a61ad63e:::obj13:head# 2024-09-15T21:45:53.565 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:760: TEST_repair_stats_ec: for i in $(seq 1 $REPAIRS) 2024-09-15T21:45:53.565 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: expr 14 % 2 2024-09-15T21:45:53.567 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: OSD=0 2024-09-15T21:45:53.568 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:764: TEST_repair_stats_ec: _objectstore_tool_nodown td/osd-scrub-repair 0 obj14 remove 2024-09-15T21:45:53.568 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-15T21:45:53.568 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-15T21:45:53.568 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=0 2024-09-15T21:45:53.568 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-15T21:45:53.568 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-09-15T21:45:53.568 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 obj14 remove 2024-09-15T21:45:54.711 INFO:tasks.workunit.client.0.smithi136.stdout:remove 1#1:e0a44829:::obj14:head# 2024-09-15T21:45:55.227 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:760: TEST_repair_stats_ec: for i in $(seq 1 $REPAIRS) 2024-09-15T21:45:55.227 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: expr 15 % 2 2024-09-15T21:45:55.229 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: OSD=1 2024-09-15T21:45:55.229 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:764: TEST_repair_stats_ec: _objectstore_tool_nodown td/osd-scrub-repair 1 obj15 remove 2024-09-15T21:45:55.229 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-15T21:45:55.229 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-15T21:45:55.229 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=1 2024-09-15T21:45:55.230 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-15T21:45:55.230 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-09-15T21:45:55.230 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 obj15 remove 2024-09-15T21:45:56.371 INFO:tasks.workunit.client.0.smithi136.stdout:remove 0#1:ab946124:::obj15:head# 2024-09-15T21:45:56.886 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:760: TEST_repair_stats_ec: for i in $(seq 1 $REPAIRS) 2024-09-15T21:45:56.886 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: expr 16 % 2 2024-09-15T21:45:56.888 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: OSD=0 2024-09-15T21:45:56.888 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:764: TEST_repair_stats_ec: _objectstore_tool_nodown td/osd-scrub-repair 0 obj16 remove 2024-09-15T21:45:56.889 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-15T21:45:56.889 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-15T21:45:56.889 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=0 2024-09-15T21:45:56.889 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-15T21:45:56.889 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-09-15T21:45:56.889 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 obj16 remove 2024-09-15T21:45:58.056 INFO:tasks.workunit.client.0.smithi136.stdout:remove 1#1:461f8b5e:::obj16:head# 2024-09-15T21:45:58.575 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:760: TEST_repair_stats_ec: for i in $(seq 1 $REPAIRS) 2024-09-15T21:45:58.575 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: expr 17 % 2 2024-09-15T21:45:58.577 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: OSD=1 2024-09-15T21:45:58.577 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:764: TEST_repair_stats_ec: _objectstore_tool_nodown td/osd-scrub-repair 1 obj17 remove 2024-09-15T21:45:58.577 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-15T21:45:58.577 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-15T21:45:58.577 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=1 2024-09-15T21:45:58.577 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-15T21:45:58.578 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-09-15T21:45:58.578 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 obj17 remove 2024-09-15T21:45:59.866 INFO:tasks.workunit.client.0.smithi136.stdout:remove 0#1:24fc6e92:::obj17:head# 2024-09-15T21:46:00.382 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:760: TEST_repair_stats_ec: for i in $(seq 1 $REPAIRS) 2024-09-15T21:46:00.382 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: expr 18 % 2 2024-09-15T21:46:00.384 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: OSD=0 2024-09-15T21:46:00.384 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:764: TEST_repair_stats_ec: _objectstore_tool_nodown td/osd-scrub-repair 0 obj18 remove 2024-09-15T21:46:00.385 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-15T21:46:00.385 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-15T21:46:00.385 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=0 2024-09-15T21:46:00.385 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-15T21:46:00.385 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-09-15T21:46:00.385 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 obj18 remove 2024-09-15T21:46:01.335 INFO:tasks.workunit.client.0.smithi136.stdout:remove 1#1:b9836a99:::obj18:head# 2024-09-15T21:46:01.850 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:760: TEST_repair_stats_ec: for i in $(seq 1 $REPAIRS) 2024-09-15T21:46:01.850 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: expr 19 % 2 2024-09-15T21:46:01.852 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: OSD=1 2024-09-15T21:46:01.853 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:764: TEST_repair_stats_ec: _objectstore_tool_nodown td/osd-scrub-repair 1 obj19 remove 2024-09-15T21:46:01.853 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-15T21:46:01.853 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-15T21:46:01.853 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=1 2024-09-15T21:46:01.853 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-15T21:46:01.853 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-09-15T21:46:01.853 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 obj19 remove 2024-09-15T21:46:02.763 INFO:tasks.workunit.client.0.smithi136.stdout:remove 0#1:00b31b6c:::obj19:head# 2024-09-15T21:46:03.281 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:760: TEST_repair_stats_ec: for i in $(seq 1 $REPAIRS) 2024-09-15T21:46:03.281 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: expr 20 % 2 2024-09-15T21:46:03.284 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: OSD=0 2024-09-15T21:46:03.284 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:764: TEST_repair_stats_ec: _objectstore_tool_nodown td/osd-scrub-repair 0 obj20 remove 2024-09-15T21:46:03.284 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-15T21:46:03.284 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-15T21:46:03.285 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=0 2024-09-15T21:46:03.285 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-15T21:46:03.285 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-09-15T21:46:03.285 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 obj20 remove 2024-09-15T21:46:04.231 INFO:tasks.workunit.client.0.smithi136.stdout:remove 1#1:24b5611b:::obj20:head# 2024-09-15T21:46:04.745 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:760: TEST_repair_stats_ec: for i in $(seq 1 $REPAIRS) 2024-09-15T21:46:04.745 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: expr 21 % 2 2024-09-15T21:46:04.747 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: OSD=1 2024-09-15T21:46:04.747 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:764: TEST_repair_stats_ec: _objectstore_tool_nodown td/osd-scrub-repair 1 obj21 remove 2024-09-15T21:46:04.747 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-15T21:46:04.747 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-15T21:46:04.747 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=1 2024-09-15T21:46:04.748 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-15T21:46:04.748 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-09-15T21:46:04.748 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 obj21 remove 2024-09-15T21:46:05.833 INFO:tasks.workunit.client.0.smithi136.stdout:remove 0#1:1243fe1a:::obj21:head# 2024-09-15T21:46:06.351 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:760: TEST_repair_stats_ec: for i in $(seq 1 $REPAIRS) 2024-09-15T21:46:06.351 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: expr 22 % 2 2024-09-15T21:46:06.353 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: OSD=0 2024-09-15T21:46:06.353 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:764: TEST_repair_stats_ec: _objectstore_tool_nodown td/osd-scrub-repair 0 obj22 remove 2024-09-15T21:46:06.353 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-15T21:46:06.353 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-15T21:46:06.353 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=0 2024-09-15T21:46:06.353 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-15T21:46:06.353 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-09-15T21:46:06.353 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 obj22 remove 2024-09-15T21:46:07.303 INFO:tasks.workunit.client.0.smithi136.stdout:remove 1#1:c0c49def:::obj22:head# 2024-09-15T21:46:07.821 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:760: TEST_repair_stats_ec: for i in $(seq 1 $REPAIRS) 2024-09-15T21:46:07.822 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: expr 23 % 2 2024-09-15T21:46:07.824 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: OSD=1 2024-09-15T21:46:07.824 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:764: TEST_repair_stats_ec: _objectstore_tool_nodown td/osd-scrub-repair 1 obj23 remove 2024-09-15T21:46:07.824 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-15T21:46:07.824 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-15T21:46:07.824 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=1 2024-09-15T21:46:07.824 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-15T21:46:07.824 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-09-15T21:46:07.824 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 obj23 remove 2024-09-15T21:46:08.814 INFO:tasks.workunit.client.0.smithi136.stdout:remove 0#1:242c40e0:::obj23:head# 2024-09-15T21:46:09.329 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:760: TEST_repair_stats_ec: for i in $(seq 1 $REPAIRS) 2024-09-15T21:46:09.329 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: expr 24 % 2 2024-09-15T21:46:09.331 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: OSD=0 2024-09-15T21:46:09.331 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:764: TEST_repair_stats_ec: _objectstore_tool_nodown td/osd-scrub-repair 0 obj24 remove 2024-09-15T21:46:09.331 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-15T21:46:09.331 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-15T21:46:09.331 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=0 2024-09-15T21:46:09.331 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-15T21:46:09.332 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-09-15T21:46:09.332 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 obj24 remove 2024-09-15T21:46:10.283 INFO:tasks.workunit.client.0.smithi136.stdout:remove 1#1:7eacede5:::obj24:head# 2024-09-15T21:46:10.798 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:760: TEST_repair_stats_ec: for i in $(seq 1 $REPAIRS) 2024-09-15T21:46:10.798 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: expr 25 % 2 2024-09-15T21:46:10.800 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: OSD=1 2024-09-15T21:46:10.800 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:764: TEST_repair_stats_ec: _objectstore_tool_nodown td/osd-scrub-repair 1 obj25 remove 2024-09-15T21:46:10.800 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-15T21:46:10.800 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-15T21:46:10.801 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=1 2024-09-15T21:46:10.801 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-15T21:46:10.801 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-09-15T21:46:10.801 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 obj25 remove 2024-09-15T21:46:11.859 INFO:tasks.workunit.client.0.smithi136.stdout:remove 0#1:7109478f:::obj25:head# 2024-09-15T21:46:12.374 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:760: TEST_repair_stats_ec: for i in $(seq 1 $REPAIRS) 2024-09-15T21:46:12.375 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: expr 26 % 2 2024-09-15T21:46:12.376 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: OSD=0 2024-09-15T21:46:12.377 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:764: TEST_repair_stats_ec: _objectstore_tool_nodown td/osd-scrub-repair 0 obj26 remove 2024-09-15T21:46:12.377 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-15T21:46:12.377 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-15T21:46:12.377 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=0 2024-09-15T21:46:12.377 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-15T21:46:12.377 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-09-15T21:46:12.377 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 obj26 remove 2024-09-15T21:46:13.295 INFO:tasks.workunit.client.0.smithi136.stdout:remove 1#1:f3c2593a:::obj26:head# 2024-09-15T21:46:13.811 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:766: TEST_repair_stats_ec: activate_osd td/osd-scrub-repair 1 --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-09-15T21:46:13.812 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2024-09-15T21:46:13.812 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-15T21:46:13.812 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=1 2024-09-15T21:46:13.812 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-15T21:46:13.812 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-09-15T21:46:13.812 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=ce4892c7-6c6d-4b4f-8e6f-07a3e11133e5 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-15T21:46:13.812 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T21:46:13.812 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T21:46:13.812 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T21:46:13.812 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-09-15T21:46:13.812 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-09-15T21:46:13.813 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-15T21:46:13.813 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-15T21:46:13.813 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-15T21:46:13.813 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-15T21:46:13.813 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-15T21:46:13.813 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-15T21:46:13.813 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-15T21:46:13.813 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T21:46:13.813 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52602 2024-09-15T21:46:13.813 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.52602/$cluster-$name.asok' 2024-09-15T21:46:13.814 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.52602/$cluster-$name.asok' 2024-09-15T21:46:13.814 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-15T21:46:13.815 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-15T21:46:13.815 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-15T21:46:13.815 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T21:46:13.815 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T21:46:13.815 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T21:46:13.815 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T21:46:13.815 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-15T21:46:13.815 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+='--osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0' 2024-09-15T21:46:13.815 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-09-15T21:46:13.815 INFO:tasks.workunit.client.0.smithi136.stdout:start osd.1 2024-09-15T21:46:13.816 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.1 2024-09-15T21:46:13.816 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 1 --fsid=ce4892c7-6c6d-4b4f-8e6f-07a3e11133e5 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.52602/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-09-15T21:46:13.816 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-09-15T21:46:13.817 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 1 = 1 ']' 2024-09-15T21:46:13.817 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-15T21:46:13.817 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-15T21:46:13.817 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-15T21:46:13.846 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:46:13.845+0000 7fe86ee28640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:46:13.849 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:46:13.847+0000 7fe86ee28640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:46:13.850 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:46:13.848+0000 7fe86ee28640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:46:14.099 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 1 2024-09-15T21:46:14.099 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-15T21:46:14.099 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2024-09-15T21:46:14.099 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-15T21:46:14.099 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-15T21:46:14.099 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:46:14.099 INFO:tasks.workunit.client.0.smithi136.stdout:0 2024-09-15T21:46:14.100 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-15T21:46:14.100 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:46:14.100 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T21:46:14.374 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:46:14.967 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:46:14.965+0000 7fe86ee28640 -1 Falling back to public interface 2024-09-15T21:46:15.376 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:46:15.377 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:46:15.377 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-15T21:46:15.377 INFO:tasks.workunit.client.0.smithi136.stdout:1 2024-09-15T21:46:15.378 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:46:15.378 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T21:46:15.663 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:46:16.664 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:46:16.665 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:46:16.665 INFO:tasks.workunit.client.0.smithi136.stdout:2 2024-09-15T21:46:16.665 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-15T21:46:16.665 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:46:16.666 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T21:46:16.963 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:46:17.393 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:46:17.392+0000 7fe86ee28640 -1 osd.1 22 log_to_monitors true 2024-09-15T21:46:17.964 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:46:17.964 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:46:17.965 INFO:tasks.workunit.client.0.smithi136.stdout:3 2024-09-15T21:46:17.965 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-15T21:46:17.965 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:46:17.965 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T21:46:18.327 INFO:tasks.workunit.client.0.smithi136.stdout:osd.1 up in weight 1 up_from 26 up_thru 22 down_at 23 last_clean_interval [10,22) [v2:127.0.0.1:6802/1765513494,v1:127.0.0.1:6803/1765513494] [v2:127.0.0.1:6804/1765513494,v1:127.0.0.1:6805/1765513494] exists,up bfde3e12-1aa2-4789-a6a6-efd7f99566bf 2024-09-15T21:46:18.331 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-15T21:46:18.331 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-15T21:46:18.331 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-15T21:46:18.331 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:767: TEST_repair_stats_ec: activate_osd td/osd-scrub-repair 0 --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-09-15T21:46:18.331 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2024-09-15T21:46:18.331 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-15T21:46:18.331 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=0 2024-09-15T21:46:18.331 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-15T21:46:18.331 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-09-15T21:46:18.331 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=ce4892c7-6c6d-4b4f-8e6f-07a3e11133e5 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-15T21:46:18.331 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T21:46:18.332 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T21:46:18.332 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T21:46:18.332 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-09-15T21:46:18.332 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-09-15T21:46:18.332 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-15T21:46:18.332 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-15T21:46:18.332 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-15T21:46:18.332 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-15T21:46:18.332 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-15T21:46:18.332 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-15T21:46:18.332 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-15T21:46:18.332 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T21:46:18.332 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52602 2024-09-15T21:46:18.333 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.52602/$cluster-$name.asok' 2024-09-15T21:46:18.333 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.52602/$cluster-$name.asok' 2024-09-15T21:46:18.333 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-15T21:46:18.333 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-15T21:46:18.333 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-15T21:46:18.333 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T21:46:18.333 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T21:46:18.333 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T21:46:18.333 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T21:46:18.333 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-15T21:46:18.333 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+='--osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0' 2024-09-15T21:46:18.333 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-09-15T21:46:18.334 INFO:tasks.workunit.client.0.smithi136.stdout:start osd.0 2024-09-15T21:46:18.335 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.0 2024-09-15T21:46:18.335 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 0 --fsid=ce4892c7-6c6d-4b4f-8e6f-07a3e11133e5 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.52602/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-09-15T21:46:18.335 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-09-15T21:46:18.335 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 0 = 0 ']' 2024-09-15T21:46:18.335 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-15T21:46:18.335 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-15T21:46:18.335 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-15T21:46:18.359 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:46:18.358+0000 7f07e7754640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:46:18.372 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:46:18.370+0000 7f07e7754640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:46:18.373 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:46:18.372+0000 7f07e7754640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:46:18.693 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 0 2024-09-15T21:46:18.693 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-15T21:46:18.693 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2024-09-15T21:46:18.693 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-15T21:46:18.693 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-15T21:46:18.693 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:46:18.693 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-15T21:46:18.693 INFO:tasks.workunit.client.0.smithi136.stdout:0 2024-09-15T21:46:18.694 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:46:18.694 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:46:19.009 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:46:19.713 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:46:19.711+0000 7f07e7754640 -1 Falling back to public interface 2024-09-15T21:46:20.010 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:46:20.010 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:46:20.010 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-15T21:46:20.010 INFO:tasks.workunit.client.0.smithi136.stdout:1 2024-09-15T21:46:20.010 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:46:20.011 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:46:20.295 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:46:21.296 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:46:21.297 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:46:21.297 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-15T21:46:21.297 INFO:tasks.workunit.client.0.smithi136.stdout:2 2024-09-15T21:46:21.297 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:46:21.297 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:46:21.597 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:46:22.135 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:46:22.134+0000 7f07e7754640 -1 osd.0 21 log_to_monitors true 2024-09-15T21:46:22.598 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:46:22.598 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:46:22.598 INFO:tasks.workunit.client.0.smithi136.stdout:3 2024-09-15T21:46:22.599 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-15T21:46:22.599 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:46:22.599 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:46:23.031 INFO:tasks.workunit.client.0.smithi136.stdout:osd.0 up in weight 1 up_from 29 up_thru 0 down_at 22 last_clean_interval [5,21) [v2:127.0.0.1:6810/3593213286,v1:127.0.0.1:6811/3593213286] [v2:127.0.0.1:6812/3593213286,v1:127.0.0.1:6813/3593213286] exists,up df860f68-ddd2-4fa6-ac8f-53f9bc8e3435 2024-09-15T21:46:23.032 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-15T21:46:23.033 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-15T21:46:23.033 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-15T21:46:23.033 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:768: TEST_repair_stats_ec: wait_for_clean 2024-09-15T21:46:23.033 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-15T21:46:23.033 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-15T21:46:23.033 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-15T21:46:23.033 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-15T21:46:23.033 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-15T21:46:23.033 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-15T21:46:23.033 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-15T21:46:23.033 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-15T21:46:23.033 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-15T21:46:23.129 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-15T21:46:23.129 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-15T21:46:23.129 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-15T21:46:23.129 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-15T21:46:23.129 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-15T21:46:23.129 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-15T21:46:23.423 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-15T21:46:23.423 INFO:tasks.workunit.client.0.smithi136.stderr:1 2024-09-15T21:46:23.424 INFO:tasks.workunit.client.0.smithi136.stderr:2' 2024-09-15T21:46:23.424 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-15T21:46:23.424 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T21:46:23.424 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-15T21:46:23.599 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=124554051586 2024-09-15T21:46:23.599 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 124554051586 2024-09-15T21:46:23.599 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-124554051586' 2024-09-15T21:46:23.600 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T21:46:23.600 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-15T21:46:23.703 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=111669149699 2024-09-15T21:46:23.703 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 111669149699 2024-09-15T21:46:23.703 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-124554051586 1-111669149699' 2024-09-15T21:46:23.703 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T21:46:23.703 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-15T21:46:23.834 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=64424509459 2024-09-15T21:46:23.834 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 64424509459 2024-09-15T21:46:23.834 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-124554051586 1-111669149699 2-64424509459' 2024-09-15T21:46:23.835 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T21:46:23.835 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-124554051586 2024-09-15T21:46:23.835 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T21:46:23.835 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-15T21:46:23.836 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-124554051586 2024-09-15T21:46:23.836 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T21:46:23.837 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=124554051586 2024-09-15T21:46:23.837 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 124554051586' 2024-09-15T21:46:23.837 INFO:tasks.workunit.client.0.smithi136.stdout:waiting osd.0 seq 124554051586 2024-09-15T21:46:23.837 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T21:46:24.155 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 1 -lt 124554051586 2024-09-15T21:46:24.156 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-15T21:46:25.157 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-15T21:46:25.157 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T21:46:25.446 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 1 -lt 124554051586 2024-09-15T21:46:25.447 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-15T21:46:26.448 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 299 -eq 0 ']' 2024-09-15T21:46:26.448 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T21:46:26.735 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 124554051586 -lt 124554051586 2024-09-15T21:46:26.736 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T21:46:26.736 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-111669149699 2024-09-15T21:46:26.737 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T21:46:26.737 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-15T21:46:26.738 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-111669149699 2024-09-15T21:46:26.738 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T21:46:26.739 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=111669149699 2024-09-15T21:46:26.739 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 111669149699' 2024-09-15T21:46:26.740 INFO:tasks.workunit.client.0.smithi136.stdout:waiting osd.1 seq 111669149699 2024-09-15T21:46:26.740 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-15T21:46:27.056 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 111669149699 -lt 111669149699 2024-09-15T21:46:27.056 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T21:46:27.056 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-64424509459 2024-09-15T21:46:27.056 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T21:46:27.057 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-15T21:46:27.058 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-64424509459 2024-09-15T21:46:27.058 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T21:46:27.059 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=64424509459 2024-09-15T21:46:27.059 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 64424509459' 2024-09-15T21:46:27.059 INFO:tasks.workunit.client.0.smithi136.stdout:waiting osd.2 seq 64424509459 2024-09-15T21:46:27.060 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-15T21:46:27.336 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 64424509459 -lt 64424509459 2024-09-15T21:46:27.337 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-15T21:46:27.337 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T21:46:27.337 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T21:46:27.684 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 1 == 0 2024-09-15T21:46:27.684 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-15T21:46:27.685 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-15T21:46:27.685 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-15T21:46:27.685 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T21:46:27.685 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T21:46:27.685 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T21:46:27.685 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T21:46:27.956 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=1 2024-09-15T21:46:27.956 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-15T21:46:27.957 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T21:46:27.957 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T21:46:28.301 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 = 1 2024-09-15T21:46:28.301 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-15T21:46:28.301 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-15T21:46:28.301 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:770: TEST_repair_stats_ec: repair 1.0 2024-09-15T21:46:28.302 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1846: repair: local pgid=1.0 2024-09-15T21:46:28.302 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1847: repair: get_last_scrub_stamp 1.0 2024-09-15T21:46:28.302 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-15T21:46:28.302 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-15T21:46:28.302 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-15T21:46:28.302 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-15T21:46:28.573 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1847: repair: local last_scrub=2024-09-15T21:45:18.265796+0000 2024-09-15T21:46:28.573 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1848: repair: ceph pg repair 1.0 2024-09-15T21:46:28.849 INFO:tasks.workunit.client.0.smithi136.stderr:instructing pg 1.0s0 on osd.1 to repair 2024-09-15T21:46:28.861 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1849: repair: wait_for_scrub 1.0 2024-09-15T21:45:18.265796+0000 2024-09-15T21:46:28.861 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: local pgid=1.0 2024-09-15T21:46:28.861 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2018: wait_for_scrub: local last_scrub=2024-09-15T21:45:18.265796+0000 2024-09-15T21:46:28.861 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2019: wait_for_scrub: local sname=last_scrub_stamp 2024-09-15T21:46:28.861 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i=0 )) 2024-09-15T21:46:28.861 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-15T21:46:28.861 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-15T21:46:28.861 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-15T21:46:28.862 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-15T21:46:28.862 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-15T21:46:28.862 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-15T21:46:29.129 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-15T21:45:18.265796+0000 '>' 2024-09-15T21:45:18.265796+0000 2024-09-15T21:46:29.130 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-15T21:46:30.131 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-15T21:46:30.131 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-15T21:46:30.132 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-15T21:46:30.132 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-15T21:46:30.132 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-15T21:46:30.132 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-15T21:46:30.132 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-15T21:46:30.410 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-15T21:45:18.265796+0000 '>' 2024-09-15T21:45:18.265796+0000 2024-09-15T21:46:30.410 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-15T21:46:31.412 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-15T21:46:31.412 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-15T21:46:31.412 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-15T21:46:31.413 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-15T21:46:31.413 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-15T21:46:31.413 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-15T21:46:31.413 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-15T21:46:31.691 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-15T21:45:18.265796+0000 '>' 2024-09-15T21:45:18.265796+0000 2024-09-15T21:46:31.691 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-15T21:46:32.693 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-15T21:46:32.693 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-15T21:46:32.693 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-15T21:46:32.693 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-15T21:46:32.694 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-15T21:46:32.694 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-15T21:46:32.694 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-15T21:46:32.975 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-15T21:45:18.265796+0000 '>' 2024-09-15T21:45:18.265796+0000 2024-09-15T21:46:32.975 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-15T21:46:33.976 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-15T21:46:33.977 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-15T21:46:33.978 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-15T21:46:33.978 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-15T21:46:33.978 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-15T21:46:33.978 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-15T21:46:33.978 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-15T21:46:34.256 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-15T21:46:29.367635+0000 '>' 2024-09-15T21:45:18.265796+0000 2024-09-15T21:46:34.256 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2023: wait_for_scrub: return 0 2024-09-15T21:46:34.256 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1850: repair: sleep 2 2024-09-15T21:46:36.257 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:771: TEST_repair_stats_ec: wait_for_clean 2024-09-15T21:46:36.257 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-15T21:46:36.257 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-15T21:46:36.258 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-15T21:46:36.258 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-15T21:46:36.258 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-15T21:46:36.258 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-15T21:46:36.259 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-15T21:46:36.259 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-15T21:46:36.259 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-15T21:46:36.366 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-15T21:46:36.367 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-15T21:46:36.367 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-15T21:46:36.367 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-15T21:46:36.367 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-15T21:46:36.367 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-15T21:46:36.643 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-15T21:46:36.643 INFO:tasks.workunit.client.0.smithi136.stderr:1 2024-09-15T21:46:36.643 INFO:tasks.workunit.client.0.smithi136.stderr:2' 2024-09-15T21:46:36.643 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-15T21:46:36.643 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T21:46:36.643 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-15T21:46:36.746 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=124554051589 2024-09-15T21:46:36.746 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 124554051589 2024-09-15T21:46:36.746 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-124554051589' 2024-09-15T21:46:36.746 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T21:46:36.747 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-15T21:46:36.849 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=111669149702 2024-09-15T21:46:36.849 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 111669149702 2024-09-15T21:46:36.849 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-124554051589 1-111669149702' 2024-09-15T21:46:36.849 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T21:46:36.849 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-15T21:46:36.952 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=64424509462 2024-09-15T21:46:36.952 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 64424509462 2024-09-15T21:46:36.952 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-124554051589 1-111669149702 2-64424509462' 2024-09-15T21:46:36.952 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T21:46:36.953 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-124554051589 2024-09-15T21:46:36.953 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T21:46:36.954 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-15T21:46:36.954 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-124554051589 2024-09-15T21:46:36.954 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T21:46:36.955 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=124554051589 2024-09-15T21:46:36.955 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 124554051589' 2024-09-15T21:46:36.956 INFO:tasks.workunit.client.0.smithi136.stdout:waiting osd.0 seq 124554051589 2024-09-15T21:46:36.956 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T21:46:37.231 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 124554051588 -lt 124554051589 2024-09-15T21:46:37.231 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-15T21:46:38.232 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-15T21:46:38.233 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T21:46:38.520 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 124554051590 -lt 124554051589 2024-09-15T21:46:38.521 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T21:46:38.521 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-111669149702 2024-09-15T21:46:38.521 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T21:46:38.522 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-15T21:46:38.523 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-111669149702 2024-09-15T21:46:38.523 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T21:46:38.524 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=111669149702 2024-09-15T21:46:38.524 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 111669149702' 2024-09-15T21:46:38.524 INFO:tasks.workunit.client.0.smithi136.stdout:waiting osd.1 seq 111669149702 2024-09-15T21:46:38.525 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-15T21:46:38.801 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 111669149702 -lt 111669149702 2024-09-15T21:46:38.802 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T21:46:38.802 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-64424509462 2024-09-15T21:46:38.802 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T21:46:38.803 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-15T21:46:38.804 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-64424509462 2024-09-15T21:46:38.804 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T21:46:38.805 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=64424509462 2024-09-15T21:46:38.805 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 64424509462' 2024-09-15T21:46:38.805 INFO:tasks.workunit.client.0.smithi136.stdout:waiting osd.2 seq 64424509462 2024-09-15T21:46:38.805 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-15T21:46:39.082 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 64424509463 -lt 64424509462 2024-09-15T21:46:39.082 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-15T21:46:39.083 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T21:46:39.083 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T21:46:39.428 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 1 == 0 2024-09-15T21:46:39.428 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-15T21:46:39.429 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-15T21:46:39.429 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-15T21:46:39.429 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T21:46:39.429 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T21:46:39.429 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T21:46:39.429 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T21:46:39.703 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=1 2024-09-15T21:46:39.704 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-15T21:46:39.704 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T21:46:39.704 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T21:46:40.046 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 = 1 2024-09-15T21:46:40.046 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-15T21:46:40.046 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-15T21:46:40.046 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:772: TEST_repair_stats_ec: ceph pg dump pgs 2024-09-15T21:46:40.309 INFO:tasks.workunit.client.0.smithi136.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-15T21:46:40.309 INFO:tasks.workunit.client.0.smithi136.stdout:1.0 30 0 0 0 0 210 0 0 30 0 30 active+clean 2024-09-15T21:46:30.586105+0000 21'30 30:133 [1,0,2] 1 [1,0,2] 1 21'30 2024-09-15T21:46:29.367635+0000 21'30 2024-09-15T21:46:29.367635+0000 0 1 periodic scrub scheduled @ 2024-09-16T21:46:29.367635+0000 17 0 2024-09-15T21:46:40.309 INFO:tasks.workunit.client.0.smithi136.stdout: 2024-09-15T21:46:40.309 INFO:tasks.workunit.client.0.smithi136.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-15T21:46:40.310 INFO:tasks.workunit.client.0.smithi136.stderr:dumped pgs 2024-09-15T21:46:40.321 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:773: TEST_repair_stats_ec: flush_pg_stats 2024-09-15T21:46:40.321 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-15T21:46:40.321 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-15T21:46:40.595 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-15T21:46:40.595 INFO:tasks.workunit.client.0.smithi136.stderr:1 2024-09-15T21:46:40.595 INFO:tasks.workunit.client.0.smithi136.stderr:2' 2024-09-15T21:46:40.595 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-15T21:46:40.595 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T21:46:40.595 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-15T21:46:40.697 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=124554051591 2024-09-15T21:46:40.697 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 124554051591 2024-09-15T21:46:40.697 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-124554051591' 2024-09-15T21:46:40.697 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T21:46:40.697 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-15T21:46:40.800 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=111669149704 2024-09-15T21:46:40.800 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 111669149704 2024-09-15T21:46:40.800 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-124554051591 1-111669149704' 2024-09-15T21:46:40.800 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T21:46:40.801 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-15T21:46:40.903 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=64424509464 2024-09-15T21:46:40.903 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 64424509464 2024-09-15T21:46:40.904 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-124554051591 1-111669149704 2-64424509464' 2024-09-15T21:46:40.904 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T21:46:40.904 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-124554051591 2024-09-15T21:46:40.904 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T21:46:40.905 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-15T21:46:40.906 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-124554051591 2024-09-15T21:46:40.906 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T21:46:40.907 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=124554051591 2024-09-15T21:46:40.907 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 124554051591' 2024-09-15T21:46:40.907 INFO:tasks.workunit.client.0.smithi136.stdout:waiting osd.0 seq 124554051591 2024-09-15T21:46:40.907 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T21:46:41.180 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 124554051590 -lt 124554051591 2024-09-15T21:46:41.181 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-15T21:46:42.181 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-15T21:46:42.182 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T21:46:42.469 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 124554051591 -lt 124554051591 2024-09-15T21:46:42.469 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T21:46:42.470 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-111669149704 2024-09-15T21:46:42.470 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T21:46:42.471 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-15T21:46:42.472 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-111669149704 2024-09-15T21:46:42.472 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T21:46:42.473 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=111669149704 2024-09-15T21:46:42.473 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 111669149704' 2024-09-15T21:46:42.473 INFO:tasks.workunit.client.0.smithi136.stdout:waiting osd.1 seq 111669149704 2024-09-15T21:46:42.473 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-15T21:46:42.746 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 111669149704 -lt 111669149704 2024-09-15T21:46:42.747 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T21:46:42.747 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-64424509464 2024-09-15T21:46:42.747 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T21:46:42.748 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-15T21:46:42.749 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-64424509464 2024-09-15T21:46:42.749 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T21:46:42.750 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=64424509464 2024-09-15T21:46:42.750 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 64424509464' 2024-09-15T21:46:42.750 INFO:tasks.workunit.client.0.smithi136.stdout:waiting osd.2 seq 64424509464 2024-09-15T21:46:42.750 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-15T21:46:43.029 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 64424509464 -lt 64424509464 2024-09-15T21:46:43.030 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:776: TEST_repair_stats_ec: ceph pg 1.0 query 2024-09-15T21:46:43.030 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:776: TEST_repair_stats_ec: jq .info.stats.stat_sum 2024-09-15T21:46:43.135 INFO:tasks.workunit.client.0.smithi136.stdout:{ 2024-09-15T21:46:43.135 INFO:tasks.workunit.client.0.smithi136.stdout: "num_bytes": 210, 2024-09-15T21:46:43.135 INFO:tasks.workunit.client.0.smithi136.stdout: "num_objects": 30, 2024-09-15T21:46:43.135 INFO:tasks.workunit.client.0.smithi136.stdout: "num_object_clones": 0, 2024-09-15T21:46:43.135 INFO:tasks.workunit.client.0.smithi136.stdout: "num_object_copies": 90, 2024-09-15T21:46:43.135 INFO:tasks.workunit.client.0.smithi136.stdout: "num_objects_missing_on_primary": 0, 2024-09-15T21:46:43.135 INFO:tasks.workunit.client.0.smithi136.stdout: "num_objects_missing": 0, 2024-09-15T21:46:43.135 INFO:tasks.workunit.client.0.smithi136.stdout: "num_objects_degraded": 0, 2024-09-15T21:46:43.135 INFO:tasks.workunit.client.0.smithi136.stdout: "num_objects_misplaced": 0, 2024-09-15T21:46:43.136 INFO:tasks.workunit.client.0.smithi136.stdout: "num_objects_unfound": 0, 2024-09-15T21:46:43.136 INFO:tasks.workunit.client.0.smithi136.stdout: "num_objects_dirty": 30, 2024-09-15T21:46:43.136 INFO:tasks.workunit.client.0.smithi136.stdout: "num_whiteouts": 0, 2024-09-15T21:46:43.136 INFO:tasks.workunit.client.0.smithi136.stdout: "num_read": 0, 2024-09-15T21:46:43.136 INFO:tasks.workunit.client.0.smithi136.stdout: "num_read_kb": 0, 2024-09-15T21:46:43.136 INFO:tasks.workunit.client.0.smithi136.stdout: "num_write": 30, 2024-09-15T21:46:43.136 INFO:tasks.workunit.client.0.smithi136.stdout: "num_write_kb": 30, 2024-09-15T21:46:43.136 INFO:tasks.workunit.client.0.smithi136.stdout: "num_scrub_errors": 0, 2024-09-15T21:46:43.136 INFO:tasks.workunit.client.0.smithi136.stdout: "num_shallow_scrub_errors": 0, 2024-09-15T21:46:43.136 INFO:tasks.workunit.client.0.smithi136.stdout: "num_deep_scrub_errors": 0, 2024-09-15T21:46:43.136 INFO:tasks.workunit.client.0.smithi136.stdout: "num_objects_recovered": 26, 2024-09-15T21:46:43.136 INFO:tasks.workunit.client.0.smithi136.stdout: "num_bytes_recovered": 182, 2024-09-15T21:46:43.136 INFO:tasks.workunit.client.0.smithi136.stdout: "num_keys_recovered": 0, 2024-09-15T21:46:43.136 INFO:tasks.workunit.client.0.smithi136.stdout: "num_objects_omap": 0, 2024-09-15T21:46:43.137 INFO:tasks.workunit.client.0.smithi136.stdout: "num_objects_hit_set_archive": 0, 2024-09-15T21:46:43.137 INFO:tasks.workunit.client.0.smithi136.stdout: "num_bytes_hit_set_archive": 0, 2024-09-15T21:46:43.137 INFO:tasks.workunit.client.0.smithi136.stdout: "num_flush": 0, 2024-09-15T21:46:43.137 INFO:tasks.workunit.client.0.smithi136.stdout: "num_flush_kb": 0, 2024-09-15T21:46:43.137 INFO:tasks.workunit.client.0.smithi136.stdout: "num_evict": 0, 2024-09-15T21:46:43.137 INFO:tasks.workunit.client.0.smithi136.stdout: "num_evict_kb": 0, 2024-09-15T21:46:43.137 INFO:tasks.workunit.client.0.smithi136.stdout: "num_promote": 0, 2024-09-15T21:46:43.137 INFO:tasks.workunit.client.0.smithi136.stdout: "num_flush_mode_high": 0, 2024-09-15T21:46:43.137 INFO:tasks.workunit.client.0.smithi136.stdout: "num_flush_mode_low": 0, 2024-09-15T21:46:43.137 INFO:tasks.workunit.client.0.smithi136.stdout: "num_evict_mode_some": 0, 2024-09-15T21:46:43.137 INFO:tasks.workunit.client.0.smithi136.stdout: "num_evict_mode_full": 0, 2024-09-15T21:46:43.137 INFO:tasks.workunit.client.0.smithi136.stdout: "num_objects_pinned": 0, 2024-09-15T21:46:43.137 INFO:tasks.workunit.client.0.smithi136.stdout: "num_legacy_snapsets": 0, 2024-09-15T21:46:43.137 INFO:tasks.workunit.client.0.smithi136.stdout: "num_large_omap_objects": 0, 2024-09-15T21:46:43.137 INFO:tasks.workunit.client.0.smithi136.stdout: "num_objects_manifest": 0, 2024-09-15T21:46:43.138 INFO:tasks.workunit.client.0.smithi136.stdout: "num_omap_bytes": 0, 2024-09-15T21:46:43.138 INFO:tasks.workunit.client.0.smithi136.stdout: "num_omap_keys": 0, 2024-09-15T21:46:43.138 INFO:tasks.workunit.client.0.smithi136.stdout: "num_objects_repaired": 26 2024-09-15T21:46:43.138 INFO:tasks.workunit.client.0.smithi136.stdout:} 2024-09-15T21:46:43.138 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:777: TEST_repair_stats_ec: ceph pg 1.0 query 2024-09-15T21:46:43.138 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:777: TEST_repair_stats_ec: jq .info.stats.stat_sum.num_objects_repaired 2024-09-15T21:46:43.242 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:777: TEST_repair_stats_ec: COUNT=26 2024-09-15T21:46:43.242 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:778: TEST_repair_stats_ec: test 26 = 26 2024-09-15T21:46:43.242 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:780: TEST_repair_stats_ec: expr 3 - 1 2024-09-15T21:46:43.244 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:780: TEST_repair_stats_ec: seq 0 2 2024-09-15T21:46:43.245 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:780: TEST_repair_stats_ec: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-09-15T21:46:43.245 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:781: TEST_repair_stats_ec: '[' 0 = 0 -o 0 = 1 ']' 2024-09-15T21:46:43.245 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:782: TEST_repair_stats_ec: expr 26 / 2 2024-09-15T21:46:43.246 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:782: TEST_repair_stats_ec: repair=13 2024-09-15T21:46:43.246 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:787: TEST_repair_stats_ec: ceph pg dump --format=json-pretty 2024-09-15T21:46:43.246 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:787: TEST_repair_stats_ec: jq '.pg_map.osd_stats[] | select(.osd == 0 )' 2024-09-15T21:46:43.507 INFO:tasks.workunit.client.0.smithi136.stderr:dumped all 2024-09-15T21:46:43.518 INFO:tasks.workunit.client.0.smithi136.stdout:{ 2024-09-15T21:46:43.518 INFO:tasks.workunit.client.0.smithi136.stdout: "osd": 0, 2024-09-15T21:46:43.518 INFO:tasks.workunit.client.0.smithi136.stdout: "up_from": 29, 2024-09-15T21:46:43.518 INFO:tasks.workunit.client.0.smithi136.stdout: "seq": 124554051591, 2024-09-15T21:46:43.518 INFO:tasks.workunit.client.0.smithi136.stdout: "num_pgs": 1, 2024-09-15T21:46:43.518 INFO:tasks.workunit.client.0.smithi136.stdout: "num_osds": 1, 2024-09-15T21:46:43.519 INFO:tasks.workunit.client.0.smithi136.stdout: "num_per_pool_osds": 1, 2024-09-15T21:46:43.519 INFO:tasks.workunit.client.0.smithi136.stdout: "num_per_pool_omap_osds": 1, 2024-09-15T21:46:43.519 INFO:tasks.workunit.client.0.smithi136.stdout: "kb": 104857600, 2024-09-15T21:46:43.519 INFO:tasks.workunit.client.0.smithi136.stdout: "kb_used": 28480, 2024-09-15T21:46:43.519 INFO:tasks.workunit.client.0.smithi136.stdout: "kb_used_data": 368, 2024-09-15T21:46:43.519 INFO:tasks.workunit.client.0.smithi136.stdout: "kb_used_omap": 4, 2024-09-15T21:46:43.519 INFO:tasks.workunit.client.0.smithi136.stdout: "kb_used_meta": 28091, 2024-09-15T21:46:43.519 INFO:tasks.workunit.client.0.smithi136.stdout: "kb_avail": 104829120, 2024-09-15T21:46:43.519 INFO:tasks.workunit.client.0.smithi136.stdout: "statfs": { 2024-09-15T21:46:43.519 INFO:tasks.workunit.client.0.smithi136.stdout: "total": 107374182400, 2024-09-15T21:46:43.520 INFO:tasks.workunit.client.0.smithi136.stdout: "available": 107345018880, 2024-09-15T21:46:43.520 INFO:tasks.workunit.client.0.smithi136.stdout: "internally_reserved": 0, 2024-09-15T21:46:43.520 INFO:tasks.workunit.client.0.smithi136.stdout: "allocated": 376832, 2024-09-15T21:46:43.520 INFO:tasks.workunit.client.0.smithi136.stdout: "data_stored": 203086, 2024-09-15T21:46:43.520 INFO:tasks.workunit.client.0.smithi136.stdout: "data_compressed": 0, 2024-09-15T21:46:43.520 INFO:tasks.workunit.client.0.smithi136.stdout: "data_compressed_allocated": 0, 2024-09-15T21:46:43.520 INFO:tasks.workunit.client.0.smithi136.stdout: "data_compressed_original": 0, 2024-09-15T21:46:43.520 INFO:tasks.workunit.client.0.smithi136.stdout: "omap_allocated": 4402, 2024-09-15T21:46:43.520 INFO:tasks.workunit.client.0.smithi136.stdout: "internal_metadata": 28765902 2024-09-15T21:46:43.520 INFO:tasks.workunit.client.0.smithi136.stdout: }, 2024-09-15T21:46:43.520 INFO:tasks.workunit.client.0.smithi136.stdout: "hb_peers": [ 2024-09-15T21:46:43.520 INFO:tasks.workunit.client.0.smithi136.stdout: 1, 2024-09-15T21:46:43.521 INFO:tasks.workunit.client.0.smithi136.stdout: 2 2024-09-15T21:46:43.521 INFO:tasks.workunit.client.0.smithi136.stdout: ], 2024-09-15T21:46:43.521 INFO:tasks.workunit.client.0.smithi136.stdout: "snap_trim_queue_len": 0, 2024-09-15T21:46:43.521 INFO:tasks.workunit.client.0.smithi136.stdout: "num_snap_trimming": 0, 2024-09-15T21:46:43.521 INFO:tasks.workunit.client.0.smithi136.stdout: "num_shards_repaired": 13, 2024-09-15T21:46:43.521 INFO:tasks.workunit.client.0.smithi136.stdout: "op_queue_age_hist": { 2024-09-15T21:46:43.521 INFO:tasks.workunit.client.0.smithi136.stdout: "histogram": [], 2024-09-15T21:46:43.521 INFO:tasks.workunit.client.0.smithi136.stdout: "upper_bound": 1 2024-09-15T21:46:43.521 INFO:tasks.workunit.client.0.smithi136.stdout: }, 2024-09-15T21:46:43.521 INFO:tasks.workunit.client.0.smithi136.stdout: "perf_stat": { 2024-09-15T21:46:43.521 INFO:tasks.workunit.client.0.smithi136.stdout: "commit_latency_ms": 0, 2024-09-15T21:46:43.521 INFO:tasks.workunit.client.0.smithi136.stdout: "apply_latency_ms": 0, 2024-09-15T21:46:43.521 INFO:tasks.workunit.client.0.smithi136.stdout: "commit_latency_ns": 0, 2024-09-15T21:46:43.521 INFO:tasks.workunit.client.0.smithi136.stdout: "apply_latency_ns": 0 2024-09-15T21:46:43.521 INFO:tasks.workunit.client.0.smithi136.stdout: }, 2024-09-15T21:46:43.522 INFO:tasks.workunit.client.0.smithi136.stdout: "alerts": [] 2024-09-15T21:46:43.522 INFO:tasks.workunit.client.0.smithi136.stdout:} 2024-09-15T21:46:43.522 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:788: TEST_repair_stats_ec: ceph pg dump --format=json-pretty 2024-09-15T21:46:43.522 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:788: TEST_repair_stats_ec: jq '.pg_map.osd_stats[] | select(.osd == 0 ).num_shards_repaired' 2024-09-15T21:46:43.778 INFO:tasks.workunit.client.0.smithi136.stderr:dumped all 2024-09-15T21:46:43.790 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:788: TEST_repair_stats_ec: COUNT=13 2024-09-15T21:46:43.790 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:789: TEST_repair_stats_ec: test 13 = 13 2024-09-15T21:46:43.790 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:780: TEST_repair_stats_ec: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-09-15T21:46:43.790 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:781: TEST_repair_stats_ec: '[' 1 = 0 -o 1 = 1 ']' 2024-09-15T21:46:43.791 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:782: TEST_repair_stats_ec: expr 26 / 2 2024-09-15T21:46:43.791 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:782: TEST_repair_stats_ec: repair=13 2024-09-15T21:46:43.792 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:787: TEST_repair_stats_ec: ceph pg dump --format=json-pretty 2024-09-15T21:46:43.792 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:787: TEST_repair_stats_ec: jq '.pg_map.osd_stats[] | select(.osd == 1 )' 2024-09-15T21:46:44.051 INFO:tasks.workunit.client.0.smithi136.stderr:dumped all 2024-09-15T21:46:44.062 INFO:tasks.workunit.client.0.smithi136.stdout:{ 2024-09-15T21:46:44.062 INFO:tasks.workunit.client.0.smithi136.stdout: "osd": 1, 2024-09-15T21:46:44.063 INFO:tasks.workunit.client.0.smithi136.stdout: "up_from": 26, 2024-09-15T21:46:44.063 INFO:tasks.workunit.client.0.smithi136.stdout: "seq": 111669149705, 2024-09-15T21:46:44.063 INFO:tasks.workunit.client.0.smithi136.stdout: "num_pgs": 1, 2024-09-15T21:46:44.063 INFO:tasks.workunit.client.0.smithi136.stdout: "num_osds": 1, 2024-09-15T21:46:44.063 INFO:tasks.workunit.client.0.smithi136.stdout: "num_per_pool_osds": 1, 2024-09-15T21:46:44.063 INFO:tasks.workunit.client.0.smithi136.stdout: "num_per_pool_omap_osds": 1, 2024-09-15T21:46:44.064 INFO:tasks.workunit.client.0.smithi136.stdout: "kb": 104857600, 2024-09-15T21:46:44.064 INFO:tasks.workunit.client.0.smithi136.stdout: "kb_used": 28480, 2024-09-15T21:46:44.064 INFO:tasks.workunit.client.0.smithi136.stdout: "kb_used_data": 368, 2024-09-15T21:46:44.064 INFO:tasks.workunit.client.0.smithi136.stdout: "kb_used_omap": 4, 2024-09-15T21:46:44.064 INFO:tasks.workunit.client.0.smithi136.stdout: "kb_used_meta": 28091, 2024-09-15T21:46:44.064 INFO:tasks.workunit.client.0.smithi136.stdout: "kb_avail": 104829120, 2024-09-15T21:46:44.065 INFO:tasks.workunit.client.0.smithi136.stdout: "statfs": { 2024-09-15T21:46:44.065 INFO:tasks.workunit.client.0.smithi136.stdout: "total": 107374182400, 2024-09-15T21:46:44.065 INFO:tasks.workunit.client.0.smithi136.stdout: "available": 107345018880, 2024-09-15T21:46:44.065 INFO:tasks.workunit.client.0.smithi136.stdout: "internally_reserved": 0, 2024-09-15T21:46:44.065 INFO:tasks.workunit.client.0.smithi136.stdout: "allocated": 376832, 2024-09-15T21:46:44.066 INFO:tasks.workunit.client.0.smithi136.stdout: "data_stored": 203086, 2024-09-15T21:46:44.066 INFO:tasks.workunit.client.0.smithi136.stdout: "data_compressed": 0, 2024-09-15T21:46:44.066 INFO:tasks.workunit.client.0.smithi136.stdout: "data_compressed_allocated": 0, 2024-09-15T21:46:44.066 INFO:tasks.workunit.client.0.smithi136.stdout: "data_compressed_original": 0, 2024-09-15T21:46:44.066 INFO:tasks.workunit.client.0.smithi136.stdout: "omap_allocated": 4398, 2024-09-15T21:46:44.066 INFO:tasks.workunit.client.0.smithi136.stdout: "internal_metadata": 28765906 2024-09-15T21:46:44.067 INFO:tasks.workunit.client.0.smithi136.stdout: }, 2024-09-15T21:46:44.067 INFO:tasks.workunit.client.0.smithi136.stdout: "hb_peers": [ 2024-09-15T21:46:44.067 INFO:tasks.workunit.client.0.smithi136.stdout: 0, 2024-09-15T21:46:44.067 INFO:tasks.workunit.client.0.smithi136.stdout: 2 2024-09-15T21:46:44.067 INFO:tasks.workunit.client.0.smithi136.stdout: ], 2024-09-15T21:46:44.068 INFO:tasks.workunit.client.0.smithi136.stdout: "snap_trim_queue_len": 0, 2024-09-15T21:46:44.068 INFO:tasks.workunit.client.0.smithi136.stdout: "num_snap_trimming": 0, 2024-09-15T21:46:44.068 INFO:tasks.workunit.client.0.smithi136.stdout: "num_shards_repaired": 13, 2024-09-15T21:46:44.068 INFO:tasks.workunit.client.0.smithi136.stdout: "op_queue_age_hist": { 2024-09-15T21:46:44.068 INFO:tasks.workunit.client.0.smithi136.stdout: "histogram": [], 2024-09-15T21:46:44.068 INFO:tasks.workunit.client.0.smithi136.stdout: "upper_bound": 1 2024-09-15T21:46:44.069 INFO:tasks.workunit.client.0.smithi136.stdout: }, 2024-09-15T21:46:44.069 INFO:tasks.workunit.client.0.smithi136.stdout: "perf_stat": { 2024-09-15T21:46:44.069 INFO:tasks.workunit.client.0.smithi136.stdout: "commit_latency_ms": 0, 2024-09-15T21:46:44.069 INFO:tasks.workunit.client.0.smithi136.stdout: "apply_latency_ms": 0, 2024-09-15T21:46:44.069 INFO:tasks.workunit.client.0.smithi136.stdout: "commit_latency_ns": 0, 2024-09-15T21:46:44.070 INFO:tasks.workunit.client.0.smithi136.stdout: "apply_latency_ns": 0 2024-09-15T21:46:44.070 INFO:tasks.workunit.client.0.smithi136.stdout: }, 2024-09-15T21:46:44.070 INFO:tasks.workunit.client.0.smithi136.stdout: "alerts": [] 2024-09-15T21:46:44.070 INFO:tasks.workunit.client.0.smithi136.stdout:} 2024-09-15T21:46:44.071 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:788: TEST_repair_stats_ec: ceph pg dump --format=json-pretty 2024-09-15T21:46:44.071 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:788: TEST_repair_stats_ec: jq '.pg_map.osd_stats[] | select(.osd == 1 ).num_shards_repaired' 2024-09-15T21:46:44.321 INFO:tasks.workunit.client.0.smithi136.stderr:dumped all 2024-09-15T21:46:44.334 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:788: TEST_repair_stats_ec: COUNT=13 2024-09-15T21:46:44.334 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:789: TEST_repair_stats_ec: test 13 = 13 2024-09-15T21:46:44.334 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:780: TEST_repair_stats_ec: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-09-15T21:46:44.334 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:781: TEST_repair_stats_ec: '[' 2 = 0 -o 2 = 1 ']' 2024-09-15T21:46:44.334 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:784: TEST_repair_stats_ec: repair=0 2024-09-15T21:46:44.334 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:787: TEST_repair_stats_ec: ceph pg dump --format=json-pretty 2024-09-15T21:46:44.334 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:787: TEST_repair_stats_ec: jq '.pg_map.osd_stats[] | select(.osd == 2 )' 2024-09-15T21:46:44.595 INFO:tasks.workunit.client.0.smithi136.stderr:dumped all 2024-09-15T21:46:44.607 INFO:tasks.workunit.client.0.smithi136.stdout:{ 2024-09-15T21:46:44.607 INFO:tasks.workunit.client.0.smithi136.stdout: "osd": 2, 2024-09-15T21:46:44.607 INFO:tasks.workunit.client.0.smithi136.stdout: "up_from": 15, 2024-09-15T21:46:44.607 INFO:tasks.workunit.client.0.smithi136.stdout: "seq": 64424509465, 2024-09-15T21:46:44.607 INFO:tasks.workunit.client.0.smithi136.stdout: "num_pgs": 1, 2024-09-15T21:46:44.607 INFO:tasks.workunit.client.0.smithi136.stdout: "num_osds": 1, 2024-09-15T21:46:44.607 INFO:tasks.workunit.client.0.smithi136.stdout: "num_per_pool_osds": 1, 2024-09-15T21:46:44.607 INFO:tasks.workunit.client.0.smithi136.stdout: "num_per_pool_omap_osds": 1, 2024-09-15T21:46:44.607 INFO:tasks.workunit.client.0.smithi136.stdout: "kb": 104857600, 2024-09-15T21:46:44.607 INFO:tasks.workunit.client.0.smithi136.stdout: "kb_used": 27208, 2024-09-15T21:46:44.607 INFO:tasks.workunit.client.0.smithi136.stdout: "kb_used_data": 368, 2024-09-15T21:46:44.608 INFO:tasks.workunit.client.0.smithi136.stdout: "kb_used_omap": 1, 2024-09-15T21:46:44.608 INFO:tasks.workunit.client.0.smithi136.stdout: "kb_used_meta": 26814, 2024-09-15T21:46:44.608 INFO:tasks.workunit.client.0.smithi136.stdout: "kb_avail": 104830392, 2024-09-15T21:46:44.608 INFO:tasks.workunit.client.0.smithi136.stdout: "statfs": { 2024-09-15T21:46:44.608 INFO:tasks.workunit.client.0.smithi136.stdout: "total": 107374182400, 2024-09-15T21:46:44.608 INFO:tasks.workunit.client.0.smithi136.stdout: "available": 107346321408, 2024-09-15T21:46:44.608 INFO:tasks.workunit.client.0.smithi136.stdout: "internally_reserved": 0, 2024-09-15T21:46:44.608 INFO:tasks.workunit.client.0.smithi136.stdout: "allocated": 376832, 2024-09-15T21:46:44.608 INFO:tasks.workunit.client.0.smithi136.stdout: "data_stored": 203086, 2024-09-15T21:46:44.608 INFO:tasks.workunit.client.0.smithi136.stdout: "data_compressed": 0, 2024-09-15T21:46:44.608 INFO:tasks.workunit.client.0.smithi136.stdout: "data_compressed_allocated": 0, 2024-09-15T21:46:44.608 INFO:tasks.workunit.client.0.smithi136.stdout: "data_compressed_original": 0, 2024-09-15T21:46:44.608 INFO:tasks.workunit.client.0.smithi136.stdout: "omap_allocated": 1595, 2024-09-15T21:46:44.608 INFO:tasks.workunit.client.0.smithi136.stdout: "internal_metadata": 27457989 2024-09-15T21:46:44.609 INFO:tasks.workunit.client.0.smithi136.stdout: }, 2024-09-15T21:46:44.609 INFO:tasks.workunit.client.0.smithi136.stdout: "hb_peers": [ 2024-09-15T21:46:44.609 INFO:tasks.workunit.client.0.smithi136.stdout: 0, 2024-09-15T21:46:44.609 INFO:tasks.workunit.client.0.smithi136.stdout: 1 2024-09-15T21:46:44.609 INFO:tasks.workunit.client.0.smithi136.stdout: ], 2024-09-15T21:46:44.609 INFO:tasks.workunit.client.0.smithi136.stdout: "snap_trim_queue_len": 0, 2024-09-15T21:46:44.609 INFO:tasks.workunit.client.0.smithi136.stdout: "num_snap_trimming": 0, 2024-09-15T21:46:44.609 INFO:tasks.workunit.client.0.smithi136.stdout: "num_shards_repaired": 0, 2024-09-15T21:46:44.609 INFO:tasks.workunit.client.0.smithi136.stdout: "op_queue_age_hist": { 2024-09-15T21:46:44.609 INFO:tasks.workunit.client.0.smithi136.stdout: "histogram": [], 2024-09-15T21:46:44.609 INFO:tasks.workunit.client.0.smithi136.stdout: "upper_bound": 1 2024-09-15T21:46:44.609 INFO:tasks.workunit.client.0.smithi136.stdout: }, 2024-09-15T21:46:44.609 INFO:tasks.workunit.client.0.smithi136.stdout: "perf_stat": { 2024-09-15T21:46:44.609 INFO:tasks.workunit.client.0.smithi136.stdout: "commit_latency_ms": 0, 2024-09-15T21:46:44.609 INFO:tasks.workunit.client.0.smithi136.stdout: "apply_latency_ms": 0, 2024-09-15T21:46:44.610 INFO:tasks.workunit.client.0.smithi136.stdout: "commit_latency_ns": 0, 2024-09-15T21:46:44.610 INFO:tasks.workunit.client.0.smithi136.stdout: "apply_latency_ns": 0 2024-09-15T21:46:44.610 INFO:tasks.workunit.client.0.smithi136.stdout: }, 2024-09-15T21:46:44.610 INFO:tasks.workunit.client.0.smithi136.stdout: "alerts": [] 2024-09-15T21:46:44.610 INFO:tasks.workunit.client.0.smithi136.stdout:} 2024-09-15T21:46:44.610 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:788: TEST_repair_stats_ec: ceph pg dump --format=json-pretty 2024-09-15T21:46:44.610 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:788: TEST_repair_stats_ec: jq '.pg_map.osd_stats[] | select(.osd == 2 ).num_shards_repaired' 2024-09-15T21:46:44.866 INFO:tasks.workunit.client.0.smithi136.stderr:dumped all 2024-09-15T21:46:44.879 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:788: TEST_repair_stats_ec: COUNT=0 2024-09-15T21:46:44.879 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:789: TEST_repair_stats_ec: test 0 = 0 2024-09-15T21:46:44.879 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:792: TEST_repair_stats_ec: ceph pg dump --format=json-pretty 2024-09-15T21:46:44.879 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:792: TEST_repair_stats_ec: jq .pg_map.osd_stats_sum 2024-09-15T21:46:45.136 INFO:tasks.workunit.client.0.smithi136.stderr:dumped all 2024-09-15T21:46:45.147 INFO:tasks.workunit.client.0.smithi136.stdout:{ 2024-09-15T21:46:45.147 INFO:tasks.workunit.client.0.smithi136.stdout: "up_from": 0, 2024-09-15T21:46:45.147 INFO:tasks.workunit.client.0.smithi136.stdout: "seq": 0, 2024-09-15T21:46:45.147 INFO:tasks.workunit.client.0.smithi136.stdout: "num_pgs": 3, 2024-09-15T21:46:45.147 INFO:tasks.workunit.client.0.smithi136.stdout: "num_osds": 3, 2024-09-15T21:46:45.147 INFO:tasks.workunit.client.0.smithi136.stdout: "num_per_pool_osds": 3, 2024-09-15T21:46:45.147 INFO:tasks.workunit.client.0.smithi136.stdout: "num_per_pool_omap_osds": 3, 2024-09-15T21:46:45.148 INFO:tasks.workunit.client.0.smithi136.stdout: "kb": 314572800, 2024-09-15T21:46:45.148 INFO:tasks.workunit.client.0.smithi136.stdout: "kb_used": 84168, 2024-09-15T21:46:45.148 INFO:tasks.workunit.client.0.smithi136.stdout: "kb_used_data": 1104, 2024-09-15T21:46:45.148 INFO:tasks.workunit.client.0.smithi136.stdout: "kb_used_omap": 10, 2024-09-15T21:46:45.148 INFO:tasks.workunit.client.0.smithi136.stdout: "kb_used_meta": 82997, 2024-09-15T21:46:45.148 INFO:tasks.workunit.client.0.smithi136.stdout: "kb_avail": 314488632, 2024-09-15T21:46:45.148 INFO:tasks.workunit.client.0.smithi136.stdout: "statfs": { 2024-09-15T21:46:45.148 INFO:tasks.workunit.client.0.smithi136.stdout: "total": 322122547200, 2024-09-15T21:46:45.148 INFO:tasks.workunit.client.0.smithi136.stdout: "available": 322036359168, 2024-09-15T21:46:45.148 INFO:tasks.workunit.client.0.smithi136.stdout: "internally_reserved": 0, 2024-09-15T21:46:45.148 INFO:tasks.workunit.client.0.smithi136.stdout: "allocated": 1130496, 2024-09-15T21:46:45.148 INFO:tasks.workunit.client.0.smithi136.stdout: "data_stored": 609258, 2024-09-15T21:46:45.148 INFO:tasks.workunit.client.0.smithi136.stdout: "data_compressed": 0, 2024-09-15T21:46:45.148 INFO:tasks.workunit.client.0.smithi136.stdout: "data_compressed_allocated": 0, 2024-09-15T21:46:45.149 INFO:tasks.workunit.client.0.smithi136.stdout: "data_compressed_original": 0, 2024-09-15T21:46:45.149 INFO:tasks.workunit.client.0.smithi136.stdout: "omap_allocated": 10395, 2024-09-15T21:46:45.149 INFO:tasks.workunit.client.0.smithi136.stdout: "internal_metadata": 84989797 2024-09-15T21:46:45.149 INFO:tasks.workunit.client.0.smithi136.stdout: }, 2024-09-15T21:46:45.149 INFO:tasks.workunit.client.0.smithi136.stdout: "hb_peers": [], 2024-09-15T21:46:45.149 INFO:tasks.workunit.client.0.smithi136.stdout: "snap_trim_queue_len": 0, 2024-09-15T21:46:45.149 INFO:tasks.workunit.client.0.smithi136.stdout: "num_snap_trimming": 0, 2024-09-15T21:46:45.149 INFO:tasks.workunit.client.0.smithi136.stdout: "num_shards_repaired": 26, 2024-09-15T21:46:45.149 INFO:tasks.workunit.client.0.smithi136.stdout: "op_queue_age_hist": { 2024-09-15T21:46:45.149 INFO:tasks.workunit.client.0.smithi136.stdout: "histogram": [], 2024-09-15T21:46:45.149 INFO:tasks.workunit.client.0.smithi136.stdout: "upper_bound": 1 2024-09-15T21:46:45.149 INFO:tasks.workunit.client.0.smithi136.stdout: }, 2024-09-15T21:46:45.149 INFO:tasks.workunit.client.0.smithi136.stdout: "perf_stat": { 2024-09-15T21:46:45.149 INFO:tasks.workunit.client.0.smithi136.stdout: "commit_latency_ms": 0, 2024-09-15T21:46:45.149 INFO:tasks.workunit.client.0.smithi136.stdout: "apply_latency_ms": 0, 2024-09-15T21:46:45.150 INFO:tasks.workunit.client.0.smithi136.stdout: "commit_latency_ns": 0, 2024-09-15T21:46:45.150 INFO:tasks.workunit.client.0.smithi136.stdout: "apply_latency_ns": 0 2024-09-15T21:46:45.150 INFO:tasks.workunit.client.0.smithi136.stdout: }, 2024-09-15T21:46:45.150 INFO:tasks.workunit.client.0.smithi136.stdout: "alerts": [], 2024-09-15T21:46:45.150 INFO:tasks.workunit.client.0.smithi136.stdout: "network_ping_times": [] 2024-09-15T21:46:45.150 INFO:tasks.workunit.client.0.smithi136.stdout:} 2024-09-15T21:46:45.150 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:793: TEST_repair_stats_ec: ceph pg dump --format=json-pretty 2024-09-15T21:46:45.150 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:793: TEST_repair_stats_ec: jq .pg_map.osd_stats_sum.num_shards_repaired 2024-09-15T21:46:45.406 INFO:tasks.workunit.client.0.smithi136.stderr:dumped all 2024-09-15T21:46:45.419 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:793: TEST_repair_stats_ec: COUNT=26 2024-09-15T21:46:45.419 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:794: TEST_repair_stats_ec: test 26 = 26 2024-09-15T21:46:45.419 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:66: run: teardown td/osd-scrub-repair 2024-09-15T21:46:45.419 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/osd-scrub-repair 2024-09-15T21:46:45.419 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-15T21:46:45.419 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/osd-scrub-repair KILL 2024-09-15T21:46:45.419 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-15T21:46:45.419 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-15T21:46:45.420 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-15T21:46:45.420 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-15T21:46:45.420 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-15T21:46:45.527 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-15T21:46:45.527 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-15T21:46:45.528 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-15T21:46:45.529 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-15T21:46:45.530 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-15T21:46:45.530 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-15T21:46:45.530 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-15T21:46:45.531 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T21:46:45.531 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-15T21:46:45.532 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-15T21:46:45.532 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T21:46:45.533 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-15T21:46:45.534 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-15T21:46:45.534 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/osd-scrub-repair 2024-09-15T21:46:45.552 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-15T21:46:45.552 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T21:46:45.552 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52602 2024-09-15T21:46:45.552 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.52602 2024-09-15T21:46:45.554 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-15T21:46:45.554 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-15T21:46:45.554 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:63: run: for func in $funcs 2024-09-15T21:46:45.554 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:64: run: setup td/osd-scrub-repair 2024-09-15T21:46:45.554 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/osd-scrub-repair 2024-09-15T21:46:45.554 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/osd-scrub-repair 2024-09-15T21:46:45.554 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/osd-scrub-repair 2024-09-15T21:46:45.554 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-15T21:46:45.554 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/osd-scrub-repair KILL 2024-09-15T21:46:45.554 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-15T21:46:45.555 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-15T21:46:45.555 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-15T21:46:45.555 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-15T21:46:45.555 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-15T21:46:45.556 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-15T21:46:45.556 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-15T21:46:45.557 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-15T21:46:45.557 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-15T21:46:45.558 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-15T21:46:45.558 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-15T21:46:45.559 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-15T21:46:45.560 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T21:46:45.560 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-15T21:46:45.560 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-15T21:46:45.560 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T21:46:45.561 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-15T21:46:45.562 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-15T21:46:45.562 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/osd-scrub-repair 2024-09-15T21:46:45.563 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-15T21:46:45.563 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T21:46:45.564 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52602 2024-09-15T21:46:45.564 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.52602 2024-09-15T21:46:45.564 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-15T21:46:45.564 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-15T21:46:45.565 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/osd-scrub-repair 2024-09-15T21:46:45.566 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2024-09-15T21:46:45.566 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T21:46:45.566 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52602 2024-09-15T21:46:45.566 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.52602 2024-09-15T21:46:45.569 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2024-09-15T21:46:45.569 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 4096 -le 1024 ']' 2024-09-15T21:46:45.569 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2024-09-15T21:46:45.569 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/osd-scrub-repair 1' TERM HUP INT 2024-09-15T21:46:45.569 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:65: run: TEST_request_scrub_priority td/osd-scrub-repair 2024-09-15T21:46:45.569 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6200: TEST_request_scrub_priority: local dir=td/osd-scrub-repair 2024-09-15T21:46:45.570 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6201: TEST_request_scrub_priority: local poolname=psr_pool 2024-09-15T21:46:45.570 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6202: TEST_request_scrub_priority: local objname=POBJ 2024-09-15T21:46:45.570 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6203: TEST_request_scrub_priority: local OBJECTS=64 2024-09-15T21:46:45.570 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6204: TEST_request_scrub_priority: local PGS=8 2024-09-15T21:46:45.570 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6206: TEST_request_scrub_priority: run_mon td/osd-scrub-repair a --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-09-15T21:46:45.570 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/osd-scrub-repair 2024-09-15T21:46:45.570 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2024-09-15T21:46:45.570 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2024-09-15T21:46:45.570 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2024-09-15T21:46:45.570 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/osd-scrub-repair/a 2024-09-15T21:46:45.570 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-scrub-repair/a --run-dir=td/osd-scrub-repair --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-09-15T21:46:46.042 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2024-09-15T21:46:46.042 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-15T21:46:46.042 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-15T21:46:46.042 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-15T21:46:46.043 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T21:46:46.043 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52602 2024-09-15T21:46:46.043 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.52602/$cluster-$name.asok' 2024-09-15T21:46:46.043 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-scrub-repair/a '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.52602/$cluster-$name.asok' --mon-cluster-log-file=td/osd-scrub-repair/log --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-09-15T21:46:46.234 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2024-09-15T21:46:46.235 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2024-09-15T21:46:46.235 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-15T21:46:46.235 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-15T21:46:46.235 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2024-09-15T21:46:46.237 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2024-09-15T21:46:46.237 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-15T21:46:46.237 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-15T21:46:46.237 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-15T21:46:46.237 INFO:tasks.workunit.client.0.smithi136.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-15T21:46:46.237 INFO:tasks.workunit.client.0.smithi136.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T21:46:46.237 INFO:tasks.workunit.client.0.smithi136.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52602 2024-09-15T21:46:46.238 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.52602/ceph-mon.a.asok 2024-09-15T21:46:46.238 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-15T21:46:46.238 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.52602/ceph-mon.a.asok config get fsid 2024-09-15T21:46:46.318 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2024-09-15T21:46:46.319 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-15T21:46:46.319 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-15T21:46:46.319 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2024-09-15T21:46:46.319 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2024-09-15T21:46:46.319 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-15T21:46:46.319 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-15T21:46:46.319 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-15T21:46:46.319 INFO:tasks.workunit.client.0.smithi136.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-15T21:46:46.319 INFO:tasks.workunit.client.0.smithi136.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T21:46:46.319 INFO:tasks.workunit.client.0.smithi136.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52602 2024-09-15T21:46:46.319 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.52602/ceph-mon.a.asok 2024-09-15T21:46:46.320 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-15T21:46:46.320 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.52602/ceph-mon.a.asok config get mon_host 2024-09-15T21:46:46.397 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6207: TEST_request_scrub_priority: run_mgr td/osd-scrub-repair x 2024-09-15T21:46:46.397 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: run_mgr: local dir=td/osd-scrub-repair 2024-09-15T21:46:46.397 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: run_mgr: shift 2024-09-15T21:46:46.397 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: run_mgr: local id=x 2024-09-15T21:46:46.397 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: run_mgr: shift 2024-09-15T21:46:46.397 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:548: run_mgr: local data=td/osd-scrub-repair/x 2024-09-15T21:46:46.398 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:550: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-15T21:46:47.203 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: get_asok_path 2024-09-15T21:46:47.203 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-15T21:46:47.203 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-15T21:46:47.203 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-15T21:46:47.203 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T21:46:47.203 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52602 2024-09-15T21:46:47.203 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.52602/$cluster-$name.asok' 2024-09-15T21:46:47.204 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-15T21:46:47.205 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-scrub-repair/x '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.52602/$cluster-$name.asok' --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-15T21:46:47.237 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6208: TEST_request_scrub_priority: local 'ceph_osd_args=--osd-scrub-interval-randomize-ratio=0 --osd-deep-scrub-randomize-ratio=0 ' 2024-09-15T21:46:47.237 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6209: TEST_request_scrub_priority: ceph_osd_args+=--osd_scrub_backoff_ratio=0 2024-09-15T21:46:47.237 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6210: TEST_request_scrub_priority: run_osd td/osd-scrub-repair 0 --osd-scrub-interval-randomize-ratio=0 --osd-deep-scrub-randomize-ratio=0 --osd_scrub_backoff_ratio=0 2024-09-15T21:46:47.237 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-scrub-repair 2024-09-15T21:46:47.237 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-15T21:46:47.238 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=0 2024-09-15T21:46:47.238 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-15T21:46:47.238 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-scrub-repair/0 2024-09-15T21:46:47.238 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=ce4892c7-6c6d-4b4f-8e6f-07a3e11133e5 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-15T21:46:47.238 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T21:46:47.238 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T21:46:47.238 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T21:46:47.238 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-09-15T21:46:47.238 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-09-15T21:46:47.238 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-15T21:46:47.238 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-15T21:46:47.238 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-15T21:46:47.238 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-15T21:46:47.239 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-15T21:46:47.239 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-15T21:46:47.239 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-15T21:46:47.239 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T21:46:47.239 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52602 2024-09-15T21:46:47.240 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.52602/$cluster-$name.asok' 2024-09-15T21:46:47.240 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.52602/$cluster-$name.asok' 2024-09-15T21:46:47.240 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-15T21:46:47.240 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-15T21:46:47.240 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-15T21:46:47.240 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-15T21:46:47.240 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-15T21:46:47.240 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T21:46:47.240 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T21:46:47.240 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T21:46:47.240 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T21:46:47.240 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-15T21:46:47.240 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+='--osd-scrub-interval-randomize-ratio=0 --osd-deep-scrub-randomize-ratio=0 --osd_scrub_backoff_ratio=0' 2024-09-15T21:46:47.240 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-scrub-repair/0 2024-09-15T21:46:47.241 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-15T21:46:47.241 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=ba9f3310-c023-41b3-a0ed-fbdb274788ac 2024-09-15T21:46:47.241 INFO:tasks.workunit.client.0.smithi136.stdout:add osd0 ba9f3310-c023-41b3-a0ed-fbdb274788ac 2024-09-15T21:46:47.242 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd0 ba9f3310-c023-41b3-a0ed-fbdb274788ac' 2024-09-15T21:46:47.242 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-15T21:46:47.257 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQDHVedmYIc6DxAAiFkljDxLEU2dTClBL0UIrg== 2024-09-15T21:46:47.258 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQDHVedmYIc6DxAAiFkljDxLEU2dTClBL0UIrg=="}' 2024-09-15T21:46:47.258 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new ba9f3310-c023-41b3-a0ed-fbdb274788ac -i td/osd-scrub-repair/0/new.json 2024-09-15T21:46:47.459 INFO:tasks.workunit.client.0.smithi136.stdout:0 2024-09-15T21:46:47.468 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-scrub-repair/0/new.json 2024-09-15T21:46:47.469 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 0 --fsid=ce4892c7-6c6d-4b4f-8e6f-07a3e11133e5 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.52602/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-interval-randomize-ratio=0 --osd-deep-scrub-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --mkfs --key AQDHVedmYIc6DxAAiFkljDxLEU2dTClBL0UIrg== --osd-uuid ba9f3310-c023-41b3-a0ed-fbdb274788ac 2024-09-15T21:46:47.501 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:46:47.499+0000 7fe430f23640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:46:47.510 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:46:47.508+0000 7fe430f23640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:46:47.511 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:46:47.510+0000 7fe430f23640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:46:47.512 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:46:47.510+0000 7fe430f23640 -1 bdev(0x56524339dc00 td/osd-scrub-repair/0/block) open stat got: (1) Operation not permitted 2024-09-15T21:46:47.512 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:46:47.510+0000 7fe430f23640 -1 bluestore(td/osd-scrub-repair/0) _read_fsid unparsable uuid 2024-09-15T21:46:53.160 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-scrub-repair/0/keyring 2024-09-15T21:46:53.160 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-15T21:46:53.161 INFO:tasks.workunit.client.0.smithi136.stdout:adding osd0 key to auth repository 2024-09-15T21:46:53.161 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd0 key to auth repository 2024-09-15T21:46:53.161 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-scrub-repair/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-15T21:46:53.523 INFO:tasks.workunit.client.0.smithi136.stdout:start osd.0 2024-09-15T21:46:53.523 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.0 2024-09-15T21:46:53.524 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 0 --fsid=ce4892c7-6c6d-4b4f-8e6f-07a3e11133e5 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.52602/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-interval-randomize-ratio=0 --osd-deep-scrub-randomize-ratio=0 --osd_scrub_backoff_ratio=0 2024-09-15T21:46:53.524 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-15T21:46:53.524 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-15T21:46:53.524 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-15T21:46:53.553 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:46:53.550+0000 7f20b14e4640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:46:53.554 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:46:53.552+0000 7f20b14e4640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:46:53.555 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:46:53.553+0000 7f20b14e4640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:46:53.799 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 0 2024-09-15T21:46:53.800 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-15T21:46:53.800 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2024-09-15T21:46:53.800 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-15T21:46:53.800 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-15T21:46:53.800 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:46:53.800 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-15T21:46:53.800 INFO:tasks.workunit.client.0.smithi136.stdout:0 2024-09-15T21:46:53.801 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:46:53.801 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:46:54.073 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:46:54.916 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:46:54.913+0000 7f20b14e4640 -1 Falling back to public interface 2024-09-15T21:46:55.074 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:46:55.075 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:46:55.075 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-15T21:46:55.075 INFO:tasks.workunit.client.0.smithi136.stdout:1 2024-09-15T21:46:55.075 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:46:55.075 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:46:55.354 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:46:56.355 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:46:56.355 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:46:56.356 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-15T21:46:56.356 INFO:tasks.workunit.client.0.smithi136.stdout:2 2024-09-15T21:46:56.356 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:46:56.356 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:46:56.643 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:46:57.645 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:46:57.645 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:46:57.645 INFO:tasks.workunit.client.0.smithi136.stdout:3 2024-09-15T21:46:57.646 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-15T21:46:57.646 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:46:57.646 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:46:57.792 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:46:57.790+0000 7f20b14e4640 -1 osd.0 0 log_to_monitors true 2024-09-15T21:46:57.966 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:46:58.968 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:46:58.968 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:46:58.968 INFO:tasks.workunit.client.0.smithi136.stdout:4 2024-09-15T21:46:58.969 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-15T21:46:58.969 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:46:58.969 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:46:59.257 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:47:00.258 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:47:00.258 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:47:00.259 INFO:tasks.workunit.client.0.smithi136.stdout:5 2024-09-15T21:47:00.260 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-15T21:47:00.260 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:47:00.260 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:47:00.551 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:47:01.552 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:47:01.552 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:47:01.553 INFO:tasks.workunit.client.0.smithi136.stdout:6 2024-09-15T21:47:01.553 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-15T21:47:01.553 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:47:01.553 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:47:01.839 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:47:02.841 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:47:02.841 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:47:02.841 INFO:tasks.workunit.client.0.smithi136.stdout:7 2024-09-15T21:47:02.842 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-15T21:47:02.842 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:47:02.842 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:47:03.128 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:47:04.129 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:47:04.129 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:47:04.129 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-15T21:47:04.129 INFO:tasks.workunit.client.0.smithi136.stdout:8 2024-09-15T21:47:04.130 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:47:04.130 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:47:04.420 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:47:05.422 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:47:05.422 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:47:05.422 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-15T21:47:05.422 INFO:tasks.workunit.client.0.smithi136.stdout:9 2024-09-15T21:47:05.422 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:47:05.423 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:47:05.751 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:47:06.752 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:47:06.752 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:47:06.753 INFO:tasks.workunit.client.0.smithi136.stdout:10 2024-09-15T21:47:06.753 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-15T21:47:06.753 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:47:06.753 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:47:07.042 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:47:08.039 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:47:08.040 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:47:08.040 INFO:tasks.workunit.client.0.smithi136.stdout:11 2024-09-15T21:47:08.040 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-15T21:47:08.040 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:47:08.040 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:47:08.359 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:47:09.361 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:47:09.361 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:47:09.361 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-15T21:47:09.361 INFO:tasks.workunit.client.0.smithi136.stdout:12 2024-09-15T21:47:09.362 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:47:09.362 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:47:09.952 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:47:10.953 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:47:10.954 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:47:10.954 INFO:tasks.workunit.client.0.smithi136.stdout:13 2024-09-15T21:47:10.955 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-15T21:47:10.955 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:47:10.955 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:47:11.932 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:47:12.933 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:47:12.934 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:47:12.934 INFO:tasks.workunit.client.0.smithi136.stdout:14 2024-09-15T21:47:12.935 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-15T21:47:12.935 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:47:12.935 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:47:13.359 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:47:14.360 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:47:14.360 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:47:14.360 INFO:tasks.workunit.client.0.smithi136.stdout:15 2024-09-15T21:47:14.361 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-15T21:47:14.361 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:47:14.361 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:47:14.997 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:47:15.999 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:47:15.999 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:47:15.999 INFO:tasks.workunit.client.0.smithi136.stdout:16 2024-09-15T21:47:16.000 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-15T21:47:16.000 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:47:16.000 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:47:16.748 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:47:17.749 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:47:17.750 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:47:17.750 INFO:tasks.workunit.client.0.smithi136.stdout:17 2024-09-15T21:47:17.750 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-15T21:47:17.750 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:47:17.751 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:47:18.156 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:47:19.158 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:47:19.158 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:47:19.158 INFO:tasks.workunit.client.0.smithi136.stdout:18 2024-09-15T21:47:19.159 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 18 2024-09-15T21:47:19.159 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:47:19.159 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:47:19.492 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:47:20.493 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:47:20.493 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:47:20.493 INFO:tasks.workunit.client.0.smithi136.stdout:19 2024-09-15T21:47:20.494 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 19 2024-09-15T21:47:20.494 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:47:20.494 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:47:20.797 INFO:tasks.workunit.client.0.smithi136.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/1362505626,v1:127.0.0.1:6803/1362505626] [v2:127.0.0.1:6804/1362505626,v1:127.0.0.1:6805/1362505626] exists,up ba9f3310-c023-41b3-a0ed-fbdb274788ac 2024-09-15T21:47:20.798 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-15T21:47:20.798 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-15T21:47:20.798 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-15T21:47:20.798 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6212: TEST_request_scrub_priority: create_pool psr_pool 8 8 2024-09-15T21:47:20.798 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create psr_pool 8 8 2024-09-15T21:47:21.177 INFO:tasks.workunit.client.0.smithi136.stderr:pool 'psr_pool' created 2024-09-15T21:47:21.189 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2024-09-15T21:47:22.191 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6213: TEST_request_scrub_priority: wait_for_clean 2024-09-15T21:47:22.191 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-15T21:47:22.191 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-15T21:47:22.191 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-15T21:47:22.192 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-15T21:47:22.192 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-15T21:47:22.192 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-15T21:47:22.192 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-15T21:47:22.192 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-15T21:47:22.192 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-15T21:47:22.297 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-15T21:47:22.298 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-15T21:47:22.298 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-15T21:47:22.298 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-15T21:47:22.298 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-15T21:47:22.298 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-15T21:47:22.613 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids=0 2024-09-15T21:47:22.614 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-15T21:47:22.614 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T21:47:22.614 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-15T21:47:22.737 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836483 2024-09-15T21:47:22.737 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836483 2024-09-15T21:47:22.737 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836483' 2024-09-15T21:47:22.738 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T21:47:22.738 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836483 2024-09-15T21:47:22.738 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T21:47:22.739 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-15T21:47:22.739 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836483 2024-09-15T21:47:22.739 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T21:47:22.740 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836483 2024-09-15T21:47:22.741 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836483' 2024-09-15T21:47:22.741 INFO:tasks.workunit.client.0.smithi136.stdout:waiting osd.0 seq 21474836483 2024-09-15T21:47:22.741 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T21:47:23.015 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 0 -lt 21474836483 2024-09-15T21:47:23.015 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-15T21:47:24.016 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-15T21:47:24.016 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T21:47:24.304 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836483 -lt 21474836483 2024-09-15T21:47:24.305 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-15T21:47:24.305 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T21:47:24.305 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T21:47:24.649 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 8 == 0 2024-09-15T21:47:24.649 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-15T21:47:24.649 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-15T21:47:24.649 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-15T21:47:24.649 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T21:47:24.649 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T21:47:24.649 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T21:47:24.649 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T21:47:24.922 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=8 2024-09-15T21:47:24.923 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-15T21:47:24.923 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T21:47:24.923 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T21:47:25.271 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 8 = 8 2024-09-15T21:47:25.271 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-15T21:47:25.272 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-15T21:47:25.272 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6215: TEST_request_scrub_priority: local osd=0 2024-09-15T21:47:25.272 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6216: TEST_request_scrub_priority: add_something td/osd-scrub-repair psr_pool POBJ noscrub 2024-09-15T21:47:25.272 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-09-15T21:47:25.272 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=psr_pool 2024-09-15T21:47:25.272 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=POBJ 2024-09-15T21:47:25.272 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-09-15T21:47:25.272 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-09-15T21:47:25.272 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-09-15T21:47:25.670 INFO:tasks.workunit.client.0.smithi136.stderr:noscrub is set 2024-09-15T21:47:25.682 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-09-15T21:47:26.181 INFO:tasks.workunit.client.0.smithi136.stderr:nodeep-scrub is set 2024-09-15T21:47:26.193 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-09-15T21:47:26.193 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-09-15T21:47:26.193 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool psr_pool put POBJ td/osd-scrub-repair/ORIGINAL 2024-09-15T21:47:26.343 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6217: TEST_request_scrub_priority: get_primary psr_pool POBJ 2024-09-15T21:47:26.343 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1187: get_primary: local poolname=psr_pool 2024-09-15T21:47:26.343 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1188: get_primary: local objectname=POBJ 2024-09-15T21:47:26.343 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1190: get_primary: ceph --format json osd map psr_pool POBJ 2024-09-15T21:47:26.343 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1191: get_primary: jq .acting_primary 2024-09-15T21:47:26.621 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6217: TEST_request_scrub_priority: local primary=0 2024-09-15T21:47:26.622 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6218: TEST_request_scrub_priority: get_pg psr_pool POBJ 2024-09-15T21:47:26.622 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1084: get_pg: local poolname=psr_pool 2024-09-15T21:47:26.622 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1085: get_pg: local objectname=POBJ 2024-09-15T21:47:26.622 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1087: get_pg: ceph --format json osd map psr_pool POBJ 2024-09-15T21:47:26.622 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1087: get_pg: jq -r .pgid 2024-09-15T21:47:26.902 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6218: TEST_request_scrub_priority: local pg=1.1 2024-09-15T21:47:26.902 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6219: TEST_request_scrub_priority: ceph osd dump 2024-09-15T21:47:26.902 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6219: TEST_request_scrub_priority: grep '^pool.*['\'']psr_pool['\'']' 2024-09-15T21:47:26.903 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6219: TEST_request_scrub_priority: awk '{ print $2 }' 2024-09-15T21:47:27.179 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6219: TEST_request_scrub_priority: poolid=1 2024-09-15T21:47:27.179 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6221: TEST_request_scrub_priority: local otherpgs 2024-09-15T21:47:27.179 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6222: TEST_request_scrub_priority: expr 8 - 1 2024-09-15T21:47:27.180 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6222: TEST_request_scrub_priority: seq 0 7 2024-09-15T21:47:27.182 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6222: TEST_request_scrub_priority: for i in $(seq 0 $(expr $PGS - 1)) 2024-09-15T21:47:27.182 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6224: TEST_request_scrub_priority: opg=1.0 2024-09-15T21:47:27.182 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6225: TEST_request_scrub_priority: '[' 1.0 = 1.1 ']' 2024-09-15T21:47:27.182 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6228: TEST_request_scrub_priority: otherpgs='1.0 ' 2024-09-15T21:47:27.183 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6229: TEST_request_scrub_priority: get_last_scrub_stamp 1.1 2024-09-15T21:47:27.183 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.1 2024-09-15T21:47:27.183 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-15T21:47:27.183 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-15T21:47:27.183 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.1") | .last_scrub_stamp' 2024-09-15T21:47:27.456 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6229: TEST_request_scrub_priority: local other_last_scrub=2024-09-15T21:47:21.125504+0000 2024-09-15T21:47:27.456 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6231: TEST_request_scrub_priority: ceph tell 1.0 schedule-scrub 1.0 2024-09-15T21:47:27.555 INFO:tasks.workunit.client.0.smithi136.stdout:{ 2024-09-15T21:47:27.556 INFO:tasks.workunit.client.0.smithi136.stdout: "deep": false, 2024-09-15T21:47:27.556 INFO:tasks.workunit.client.0.smithi136.stdout: "must": false, 2024-09-15T21:47:27.556 INFO:tasks.workunit.client.0.smithi136.stdout: "stamp": "2024-09-08T21:45:47.554770+0000" 2024-09-15T21:47:27.556 INFO:tasks.workunit.client.0.smithi136.stdout:} 2024-09-15T21:47:27.565 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6222: TEST_request_scrub_priority: for i in $(seq 0 $(expr $PGS - 1)) 2024-09-15T21:47:27.565 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6224: TEST_request_scrub_priority: opg=1.1 2024-09-15T21:47:27.565 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6225: TEST_request_scrub_priority: '[' 1.1 = 1.1 ']' 2024-09-15T21:47:27.566 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6226: TEST_request_scrub_priority: continue 2024-09-15T21:47:27.566 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6222: TEST_request_scrub_priority: for i in $(seq 0 $(expr $PGS - 1)) 2024-09-15T21:47:27.566 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6224: TEST_request_scrub_priority: opg=1.2 2024-09-15T21:47:27.566 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6225: TEST_request_scrub_priority: '[' 1.2 = 1.1 ']' 2024-09-15T21:47:27.566 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6228: TEST_request_scrub_priority: otherpgs='1.0 1.2 ' 2024-09-15T21:47:27.566 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6229: TEST_request_scrub_priority: get_last_scrub_stamp 1.1 2024-09-15T21:47:27.566 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.1 2024-09-15T21:47:27.566 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-15T21:47:27.566 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-15T21:47:27.566 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.1") | .last_scrub_stamp' 2024-09-15T21:47:27.837 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6229: TEST_request_scrub_priority: local other_last_scrub=2024-09-15T21:47:21.125504+0000 2024-09-15T21:47:27.838 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6231: TEST_request_scrub_priority: ceph tell 1.2 schedule-scrub 1.2 2024-09-15T21:47:27.935 INFO:tasks.workunit.client.0.smithi136.stdout:{ 2024-09-15T21:47:27.935 INFO:tasks.workunit.client.0.smithi136.stdout: "deep": false, 2024-09-15T21:47:27.935 INFO:tasks.workunit.client.0.smithi136.stdout: "must": false, 2024-09-15T21:47:27.935 INFO:tasks.workunit.client.0.smithi136.stdout: "stamp": "2024-09-08T21:45:47.933953+0000" 2024-09-15T21:47:27.935 INFO:tasks.workunit.client.0.smithi136.stdout:} 2024-09-15T21:47:27.944 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6222: TEST_request_scrub_priority: for i in $(seq 0 $(expr $PGS - 1)) 2024-09-15T21:47:27.944 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6224: TEST_request_scrub_priority: opg=1.3 2024-09-15T21:47:27.944 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6225: TEST_request_scrub_priority: '[' 1.3 = 1.1 ']' 2024-09-15T21:47:27.944 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6228: TEST_request_scrub_priority: otherpgs='1.0 1.2 1.3 ' 2024-09-15T21:47:27.945 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6229: TEST_request_scrub_priority: get_last_scrub_stamp 1.1 2024-09-15T21:47:27.945 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.1 2024-09-15T21:47:27.945 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-15T21:47:27.945 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-15T21:47:27.945 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.1") | .last_scrub_stamp' 2024-09-15T21:47:28.215 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6229: TEST_request_scrub_priority: local other_last_scrub=2024-09-15T21:47:21.125504+0000 2024-09-15T21:47:28.215 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6231: TEST_request_scrub_priority: ceph tell 1.3 schedule-scrub 1.3 2024-09-15T21:47:28.338 INFO:tasks.workunit.client.0.smithi136.stdout:{ 2024-09-15T21:47:28.338 INFO:tasks.workunit.client.0.smithi136.stdout: "deep": false, 2024-09-15T21:47:28.338 INFO:tasks.workunit.client.0.smithi136.stdout: "must": false, 2024-09-15T21:47:28.338 INFO:tasks.workunit.client.0.smithi136.stdout: "stamp": "2024-09-08T21:45:48.337176+0000" 2024-09-15T21:47:28.338 INFO:tasks.workunit.client.0.smithi136.stdout:} 2024-09-15T21:47:28.347 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6222: TEST_request_scrub_priority: for i in $(seq 0 $(expr $PGS - 1)) 2024-09-15T21:47:28.347 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6224: TEST_request_scrub_priority: opg=1.4 2024-09-15T21:47:28.347 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6225: TEST_request_scrub_priority: '[' 1.4 = 1.1 ']' 2024-09-15T21:47:28.347 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6228: TEST_request_scrub_priority: otherpgs='1.0 1.2 1.3 1.4 ' 2024-09-15T21:47:28.348 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6229: TEST_request_scrub_priority: get_last_scrub_stamp 1.1 2024-09-15T21:47:28.348 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.1 2024-09-15T21:47:28.348 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-15T21:47:28.348 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-15T21:47:28.348 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.1") | .last_scrub_stamp' 2024-09-15T21:47:28.617 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6229: TEST_request_scrub_priority: local other_last_scrub=2024-09-15T21:47:21.125504+0000 2024-09-15T21:47:28.617 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6231: TEST_request_scrub_priority: ceph tell 1.4 schedule-scrub 1.4 2024-09-15T21:47:28.714 INFO:tasks.workunit.client.0.smithi136.stdout:{ 2024-09-15T21:47:28.714 INFO:tasks.workunit.client.0.smithi136.stdout: "deep": false, 2024-09-15T21:47:28.714 INFO:tasks.workunit.client.0.smithi136.stdout: "must": false, 2024-09-15T21:47:28.714 INFO:tasks.workunit.client.0.smithi136.stdout: "stamp": "2024-09-08T21:45:48.713204+0000" 2024-09-15T21:47:28.714 INFO:tasks.workunit.client.0.smithi136.stdout:} 2024-09-15T21:47:28.723 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6222: TEST_request_scrub_priority: for i in $(seq 0 $(expr $PGS - 1)) 2024-09-15T21:47:28.723 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6224: TEST_request_scrub_priority: opg=1.5 2024-09-15T21:47:28.723 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6225: TEST_request_scrub_priority: '[' 1.5 = 1.1 ']' 2024-09-15T21:47:28.723 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6228: TEST_request_scrub_priority: otherpgs='1.0 1.2 1.3 1.4 1.5 ' 2024-09-15T21:47:28.724 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6229: TEST_request_scrub_priority: get_last_scrub_stamp 1.1 2024-09-15T21:47:28.724 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.1 2024-09-15T21:47:28.724 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-15T21:47:28.724 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-15T21:47:28.724 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.1") | .last_scrub_stamp' 2024-09-15T21:47:28.992 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6229: TEST_request_scrub_priority: local other_last_scrub=2024-09-15T21:47:21.125504+0000 2024-09-15T21:47:28.993 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6231: TEST_request_scrub_priority: ceph tell 1.5 schedule-scrub 1.5 2024-09-15T21:47:29.089 INFO:tasks.workunit.client.0.smithi136.stdout:{ 2024-09-15T21:47:29.090 INFO:tasks.workunit.client.0.smithi136.stdout: "deep": false, 2024-09-15T21:47:29.090 INFO:tasks.workunit.client.0.smithi136.stdout: "must": false, 2024-09-15T21:47:29.090 INFO:tasks.workunit.client.0.smithi136.stdout: "stamp": "2024-09-08T21:45:49.088716+0000" 2024-09-15T21:47:29.090 INFO:tasks.workunit.client.0.smithi136.stdout:} 2024-09-15T21:47:29.098 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6222: TEST_request_scrub_priority: for i in $(seq 0 $(expr $PGS - 1)) 2024-09-15T21:47:29.099 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6224: TEST_request_scrub_priority: opg=1.6 2024-09-15T21:47:29.099 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6225: TEST_request_scrub_priority: '[' 1.6 = 1.1 ']' 2024-09-15T21:47:29.099 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6228: TEST_request_scrub_priority: otherpgs='1.0 1.2 1.3 1.4 1.5 1.6 ' 2024-09-15T21:47:29.099 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6229: TEST_request_scrub_priority: get_last_scrub_stamp 1.1 2024-09-15T21:47:29.099 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.1 2024-09-15T21:47:29.100 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-15T21:47:29.100 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-15T21:47:29.100 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.1") | .last_scrub_stamp' 2024-09-15T21:47:29.370 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6229: TEST_request_scrub_priority: local other_last_scrub=2024-09-15T21:47:21.125504+0000 2024-09-15T21:47:29.370 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6231: TEST_request_scrub_priority: ceph tell 1.6 schedule-scrub 1.6 2024-09-15T21:47:29.468 INFO:tasks.workunit.client.0.smithi136.stdout:{ 2024-09-15T21:47:29.468 INFO:tasks.workunit.client.0.smithi136.stdout: "deep": false, 2024-09-15T21:47:29.468 INFO:tasks.workunit.client.0.smithi136.stdout: "must": false, 2024-09-15T21:47:29.468 INFO:tasks.workunit.client.0.smithi136.stdout: "stamp": "2024-09-08T21:45:49.467379+0000" 2024-09-15T21:47:29.468 INFO:tasks.workunit.client.0.smithi136.stdout:} 2024-09-15T21:47:29.477 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6222: TEST_request_scrub_priority: for i in $(seq 0 $(expr $PGS - 1)) 2024-09-15T21:47:29.477 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6224: TEST_request_scrub_priority: opg=1.7 2024-09-15T21:47:29.477 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6225: TEST_request_scrub_priority: '[' 1.7 = 1.1 ']' 2024-09-15T21:47:29.477 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6228: TEST_request_scrub_priority: otherpgs='1.0 1.2 1.3 1.4 1.5 1.6 1.7 ' 2024-09-15T21:47:29.478 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6229: TEST_request_scrub_priority: get_last_scrub_stamp 1.1 2024-09-15T21:47:29.478 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.1 2024-09-15T21:47:29.478 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-15T21:47:29.478 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-15T21:47:29.478 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.1") | .last_scrub_stamp' 2024-09-15T21:47:29.746 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6229: TEST_request_scrub_priority: local other_last_scrub=2024-09-15T21:47:21.125504+0000 2024-09-15T21:47:29.746 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6231: TEST_request_scrub_priority: ceph tell 1.7 schedule-scrub 1.7 2024-09-15T21:47:29.843 INFO:tasks.workunit.client.0.smithi136.stdout:{ 2024-09-15T21:47:29.843 INFO:tasks.workunit.client.0.smithi136.stdout: "deep": false, 2024-09-15T21:47:29.843 INFO:tasks.workunit.client.0.smithi136.stdout: "must": false, 2024-09-15T21:47:29.843 INFO:tasks.workunit.client.0.smithi136.stdout: "stamp": "2024-09-08T21:45:49.841824+0000" 2024-09-15T21:47:29.843 INFO:tasks.workunit.client.0.smithi136.stdout:} 2024-09-15T21:47:29.851 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6234: TEST_request_scrub_priority: sleep 15 2024-09-15T21:47:44.853 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6235: TEST_request_scrub_priority: flush_pg_stats 2024-09-15T21:47:44.853 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-15T21:47:44.853 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-15T21:47:45.140 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids=0 2024-09-15T21:47:45.140 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-15T21:47:45.140 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T21:47:45.141 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-15T21:47:45.245 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836488 2024-09-15T21:47:45.245 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836488 2024-09-15T21:47:45.245 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836488' 2024-09-15T21:47:45.246 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T21:47:45.246 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836488 2024-09-15T21:47:45.246 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T21:47:45.247 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-15T21:47:45.248 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836488 2024-09-15T21:47:45.248 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T21:47:45.248 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836488 2024-09-15T21:47:45.249 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836488' 2024-09-15T21:47:45.249 INFO:tasks.workunit.client.0.smithi136.stdout:waiting osd.0 seq 21474836488 2024-09-15T21:47:45.249 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T21:47:45.524 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836487 -lt 21474836488 2024-09-15T21:47:45.524 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-15T21:47:46.526 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-15T21:47:46.526 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T21:47:46.815 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836487 -lt 21474836488 2024-09-15T21:47:46.815 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-15T21:47:47.816 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 299 -eq 0 ']' 2024-09-15T21:47:47.816 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T21:47:48.099 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836489 -lt 21474836488 2024-09-15T21:47:48.100 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6238: TEST_request_scrub_priority: get_last_scrub_stamp 1.1 2024-09-15T21:47:48.100 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.1 2024-09-15T21:47:48.100 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-15T21:47:48.100 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-15T21:47:48.100 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.1") | .last_scrub_stamp' 2024-09-15T21:47:48.371 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6238: TEST_request_scrub_priority: local last_scrub=2024-09-15T21:47:21.125504+0000 2024-09-15T21:47:48.371 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6239: TEST_request_scrub_priority: ceph tell 1.1 scrub 2024-09-15T21:47:48.467 INFO:tasks.workunit.client.0.smithi136.stdout:{ 2024-09-15T21:47:48.467 INFO:tasks.workunit.client.0.smithi136.stdout: "deep": false, 2024-09-15T21:47:48.467 INFO:tasks.workunit.client.0.smithi136.stdout: "must": true, 2024-09-15T21:47:48.467 INFO:tasks.workunit.client.0.smithi136.stdout: "stamp": "0.000000" 2024-09-15T21:47:48.467 INFO:tasks.workunit.client.0.smithi136.stdout:} 2024-09-15T21:47:48.476 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6241: TEST_request_scrub_priority: ceph osd unset noscrub 2024-09-15T21:47:48.841 INFO:tasks.workunit.client.0.smithi136.stderr:noscrub is unset 2024-09-15T21:47:48.854 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6242: TEST_request_scrub_priority: ceph osd unset nodeep-scrub 2024-09-15T21:47:49.360 INFO:tasks.workunit.client.0.smithi136.stderr:nodeep-scrub is unset 2024-09-15T21:47:49.372 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6244: TEST_request_scrub_priority: wait_for_scrub 1.1 2024-09-15T21:47:21.125504+0000 2024-09-15T21:47:49.372 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: local pgid=1.1 2024-09-15T21:47:49.372 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2018: wait_for_scrub: local last_scrub=2024-09-15T21:47:21.125504+0000 2024-09-15T21:47:49.373 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2019: wait_for_scrub: local sname=last_scrub_stamp 2024-09-15T21:47:49.373 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i=0 )) 2024-09-15T21:47:49.373 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-15T21:47:49.373 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.1 last_scrub_stamp 2024-09-15T21:47:49.373 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.1 2024-09-15T21:47:49.374 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-15T21:47:49.374 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-15T21:47:49.374 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.1") | .last_scrub_stamp' 2024-09-15T21:47:49.650 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-15T21:47:21.125504+0000 '>' 2024-09-15T21:47:21.125504+0000 2024-09-15T21:47:49.650 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-15T21:47:50.651 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-15T21:47:50.651 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-15T21:47:50.652 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.1 last_scrub_stamp 2024-09-15T21:47:50.652 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.1 2024-09-15T21:47:50.652 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-15T21:47:50.652 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-15T21:47:50.652 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.1") | .last_scrub_stamp' 2024-09-15T21:47:50.930 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-15T21:47:21.125504+0000 '>' 2024-09-15T21:47:21.125504+0000 2024-09-15T21:47:50.930 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-15T21:47:51.931 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-15T21:47:51.932 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-15T21:47:51.933 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.1 last_scrub_stamp 2024-09-15T21:47:51.933 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.1 2024-09-15T21:47:51.933 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-15T21:47:51.933 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-15T21:47:51.934 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.1") | .last_scrub_stamp' 2024-09-15T21:47:52.216 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-15T21:47:21.125504+0000 '>' 2024-09-15T21:47:21.125504+0000 2024-09-15T21:47:52.216 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-15T21:47:53.217 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-15T21:47:53.217 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-15T21:47:53.218 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.1 last_scrub_stamp 2024-09-15T21:47:53.218 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.1 2024-09-15T21:47:53.218 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-15T21:47:53.218 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-15T21:47:53.218 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.1") | .last_scrub_stamp' 2024-09-15T21:47:53.511 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-15T21:47:48.501739+0000 '>' 2024-09-15T21:47:21.125504+0000 2024-09-15T21:47:53.511 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2023: wait_for_scrub: return 0 2024-09-15T21:47:53.511 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6246: TEST_request_scrub_priority: for opg in $otherpgs $pg 2024-09-15T21:47:53.511 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6248: TEST_request_scrub_priority: wait_for_scrub 1.0 2024-09-15T21:47:21.125504+0000 2024-09-15T21:47:53.511 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: local pgid=1.0 2024-09-15T21:47:53.511 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2018: wait_for_scrub: local last_scrub=2024-09-15T21:47:21.125504+0000 2024-09-15T21:47:53.511 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2019: wait_for_scrub: local sname=last_scrub_stamp 2024-09-15T21:47:53.512 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i=0 )) 2024-09-15T21:47:53.512 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-15T21:47:53.512 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-15T21:47:53.512 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-15T21:47:53.512 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-15T21:47:53.512 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-15T21:47:53.512 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-15T21:47:53.782 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-08T21:45:47.554770+0000 '>' 2024-09-15T21:47:21.125504+0000 2024-09-15T21:47:53.782 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-15T21:47:54.784 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-15T21:47:54.784 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-15T21:47:54.785 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-15T21:47:54.785 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-15T21:47:54.785 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-15T21:47:54.785 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-15T21:47:54.785 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-15T21:47:55.069 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-08T21:45:47.554770+0000 '>' 2024-09-15T21:47:21.125504+0000 2024-09-15T21:47:55.069 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-15T21:47:56.073 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-15T21:47:56.073 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-15T21:47:56.073 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-15T21:47:56.073 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-15T21:47:56.073 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-15T21:47:56.073 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-15T21:47:56.073 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-15T21:47:56.353 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-08T21:45:47.554770+0000 '>' 2024-09-15T21:47:21.125504+0000 2024-09-15T21:47:56.353 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-15T21:47:57.354 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-15T21:47:57.354 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-15T21:47:57.355 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-15T21:47:57.355 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-15T21:47:57.355 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-15T21:47:57.355 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-15T21:47:57.355 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-15T21:47:57.635 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-08T21:45:47.554770+0000 '>' 2024-09-15T21:47:21.125504+0000 2024-09-15T21:47:57.636 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-15T21:47:58.637 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-15T21:47:58.637 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-15T21:47:58.638 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-15T21:47:58.638 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-15T21:47:58.638 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-15T21:47:58.638 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-15T21:47:58.638 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-15T21:47:58.919 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-08T21:45:47.554770+0000 '>' 2024-09-15T21:47:21.125504+0000 2024-09-15T21:47:58.919 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-15T21:47:59.920 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-15T21:47:59.920 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-15T21:47:59.921 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-15T21:47:59.921 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-15T21:47:59.921 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-15T21:47:59.921 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-15T21:47:59.921 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-15T21:48:00.208 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-08T21:45:47.554770+0000 '>' 2024-09-15T21:47:21.125504+0000 2024-09-15T21:48:00.208 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-15T21:48:01.209 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-15T21:48:01.209 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-15T21:48:01.210 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-15T21:48:01.210 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-15T21:48:01.210 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-15T21:48:01.210 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-15T21:48:01.210 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-15T21:48:01.513 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-08T21:45:47.554770+0000 '>' 2024-09-15T21:47:21.125504+0000 2024-09-15T21:48:01.513 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-15T21:48:02.515 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-15T21:48:02.515 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-15T21:48:02.516 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-15T21:48:02.516 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-15T21:48:02.516 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-15T21:48:02.516 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-15T21:48:02.516 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-15T21:48:02.800 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-08T21:45:47.554770+0000 '>' 2024-09-15T21:47:21.125504+0000 2024-09-15T21:48:02.800 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-15T21:48:03.801 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-15T21:48:03.801 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-15T21:48:03.802 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-15T21:48:03.802 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-15T21:48:03.802 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-15T21:48:03.802 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-15T21:48:03.802 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-15T21:48:04.086 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-08T21:45:47.554770+0000 '>' 2024-09-15T21:47:21.125504+0000 2024-09-15T21:48:04.086 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-15T21:48:05.087 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-15T21:48:05.087 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-15T21:48:05.088 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-15T21:48:05.088 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-15T21:48:05.088 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-15T21:48:05.088 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-15T21:48:05.088 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-15T21:48:05.370 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-08T21:45:47.554770+0000 '>' 2024-09-15T21:47:21.125504+0000 2024-09-15T21:48:05.370 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-15T21:48:06.371 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-15T21:48:06.371 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-15T21:48:06.372 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-15T21:48:06.372 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-15T21:48:06.372 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-15T21:48:06.372 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-15T21:48:06.372 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-15T21:48:06.652 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-08T21:45:47.554770+0000 '>' 2024-09-15T21:47:21.125504+0000 2024-09-15T21:48:06.652 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-15T21:48:07.653 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-15T21:48:07.654 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-15T21:48:07.654 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-15T21:48:07.654 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-15T21:48:07.654 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-15T21:48:07.655 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-15T21:48:07.655 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-15T21:48:07.934 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-08T21:45:47.554770+0000 '>' 2024-09-15T21:47:21.125504+0000 2024-09-15T21:48:07.935 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-15T21:48:08.936 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-15T21:48:08.936 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-15T21:48:08.937 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-15T21:48:08.937 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-15T21:48:08.937 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-15T21:48:08.937 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-15T21:48:08.937 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-15T21:48:09.218 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-08T21:45:47.554770+0000 '>' 2024-09-15T21:47:21.125504+0000 2024-09-15T21:48:09.218 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-15T21:48:10.220 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-15T21:48:10.220 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-15T21:48:10.221 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-15T21:48:10.221 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-15T21:48:10.222 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-15T21:48:10.222 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-15T21:48:10.222 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-15T21:48:10.503 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-08T21:45:47.554770+0000 '>' 2024-09-15T21:47:21.125504+0000 2024-09-15T21:48:10.503 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-15T21:48:11.504 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-15T21:48:11.504 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-15T21:48:11.505 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-15T21:48:11.505 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-15T21:48:11.505 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-15T21:48:11.505 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-15T21:48:11.505 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-15T21:48:11.780 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-08T21:45:47.554770+0000 '>' 2024-09-15T21:47:21.125504+0000 2024-09-15T21:48:11.781 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-15T21:48:12.781 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-15T21:48:12.782 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-15T21:48:12.782 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-15T21:48:12.782 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-15T21:48:12.782 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-15T21:48:12.783 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-15T21:48:12.783 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-15T21:48:13.064 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-08T21:45:47.554770+0000 '>' 2024-09-15T21:47:21.125504+0000 2024-09-15T21:48:13.064 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-15T21:48:14.066 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-15T21:48:14.066 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-15T21:48:14.067 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-15T21:48:14.067 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-15T21:48:14.067 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-15T21:48:14.067 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-15T21:48:14.067 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-15T21:48:14.341 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-08T21:45:47.554770+0000 '>' 2024-09-15T21:47:21.125504+0000 2024-09-15T21:48:14.341 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-15T21:48:15.343 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-15T21:48:15.343 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-15T21:48:15.343 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-15T21:48:15.344 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-15T21:48:15.344 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-15T21:48:15.344 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-15T21:48:15.344 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-15T21:48:15.625 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-08T21:45:47.554770+0000 '>' 2024-09-15T21:47:21.125504+0000 2024-09-15T21:48:15.625 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-15T21:48:16.626 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-15T21:48:16.626 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-15T21:48:16.628 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-15T21:48:16.628 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-15T21:48:16.628 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-15T21:48:16.628 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-15T21:48:16.629 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-15T21:48:16.908 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-08T21:45:47.554770+0000 '>' 2024-09-15T21:47:21.125504+0000 2024-09-15T21:48:16.908 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-15T21:48:17.909 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-15T21:48:17.909 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-15T21:48:17.909 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-15T21:48:17.910 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-15T21:48:17.910 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-15T21:48:17.910 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-15T21:48:17.910 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-15T21:48:18.188 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-08T21:45:47.554770+0000 '>' 2024-09-15T21:47:21.125504+0000 2024-09-15T21:48:18.188 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-15T21:48:19.189 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-15T21:48:19.189 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-15T21:48:19.189 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-15T21:48:19.190 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-15T21:48:19.190 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-15T21:48:19.190 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-15T21:48:19.190 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-15T21:48:19.473 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-08T21:45:47.554770+0000 '>' 2024-09-15T21:47:21.125504+0000 2024-09-15T21:48:19.473 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-15T21:48:20.474 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-15T21:48:20.474 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-15T21:48:20.475 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-15T21:48:20.475 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-15T21:48:20.475 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-15T21:48:20.475 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-15T21:48:20.475 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-15T21:48:20.759 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-08T21:45:47.554770+0000 '>' 2024-09-15T21:47:21.125504+0000 2024-09-15T21:48:20.759 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-15T21:48:21.760 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-15T21:48:21.760 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-15T21:48:21.761 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-15T21:48:21.761 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-15T21:48:21.761 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-15T21:48:21.761 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-15T21:48:21.761 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-15T21:48:22.040 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-08T21:45:47.554770+0000 '>' 2024-09-15T21:47:21.125504+0000 2024-09-15T21:48:22.040 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-15T21:48:23.041 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-15T21:48:23.041 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-15T21:48:23.042 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-15T21:48:23.042 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-15T21:48:23.042 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-15T21:48:23.042 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-15T21:48:23.043 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-15T21:48:23.323 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-08T21:45:47.554770+0000 '>' 2024-09-15T21:47:21.125504+0000 2024-09-15T21:48:23.323 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-15T21:48:24.324 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-15T21:48:24.324 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-15T21:48:24.326 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-15T21:48:24.326 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-15T21:48:24.326 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-15T21:48:24.326 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-15T21:48:24.326 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-15T21:48:24.613 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-08T21:45:47.554770+0000 '>' 2024-09-15T21:47:21.125504+0000 2024-09-15T21:48:24.613 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-15T21:48:25.614 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-15T21:48:25.614 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-15T21:48:25.615 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-15T21:48:25.615 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-15T21:48:25.615 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-15T21:48:25.615 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-15T21:48:25.616 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-15T21:48:25.894 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-08T21:45:47.554770+0000 '>' 2024-09-15T21:47:21.125504+0000 2024-09-15T21:48:25.895 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-15T21:48:26.896 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-15T21:48:26.896 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-15T21:48:26.896 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-15T21:48:26.897 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-15T21:48:26.897 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-15T21:48:26.897 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-15T21:48:26.897 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-15T21:48:27.170 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-08T21:45:47.554770+0000 '>' 2024-09-15T21:47:21.125504+0000 2024-09-15T21:48:27.170 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-15T21:48:28.171 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-15T21:48:28.171 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-15T21:48:28.172 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-15T21:48:28.172 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-15T21:48:28.172 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-15T21:48:28.172 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-15T21:48:28.172 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-15T21:48:28.456 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-08T21:45:47.554770+0000 '>' 2024-09-15T21:47:21.125504+0000 2024-09-15T21:48:28.456 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-15T21:48:29.458 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-15T21:48:29.458 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-15T21:48:29.458 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-15T21:48:29.459 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-15T21:48:29.459 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-15T21:48:29.459 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-15T21:48:29.459 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-15T21:48:29.740 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-08T21:45:47.554770+0000 '>' 2024-09-15T21:47:21.125504+0000 2024-09-15T21:48:29.740 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-15T21:48:30.741 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-15T21:48:30.741 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-15T21:48:30.741 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-15T21:48:30.741 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-15T21:48:30.742 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-15T21:48:30.742 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-15T21:48:30.742 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-15T21:48:31.026 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-08T21:45:47.554770+0000 '>' 2024-09-15T21:47:21.125504+0000 2024-09-15T21:48:31.026 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-15T21:48:32.027 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-15T21:48:32.027 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-15T21:48:32.028 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-15T21:48:32.028 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-15T21:48:32.028 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-15T21:48:32.028 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-15T21:48:32.028 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-15T21:48:32.306 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-08T21:45:47.554770+0000 '>' 2024-09-15T21:47:21.125504+0000 2024-09-15T21:48:32.306 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-15T21:48:33.307 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-15T21:48:33.308 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-15T21:48:33.308 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-15T21:48:33.308 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-15T21:48:33.308 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-15T21:48:33.309 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-15T21:48:33.309 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-15T21:48:33.599 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-15T21:48:27.869647+0000 '>' 2024-09-15T21:47:21.125504+0000 2024-09-15T21:48:33.599 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2023: wait_for_scrub: return 0 2024-09-15T21:48:33.599 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6246: TEST_request_scrub_priority: for opg in $otherpgs $pg 2024-09-15T21:48:33.599 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6248: TEST_request_scrub_priority: wait_for_scrub 1.2 2024-09-15T21:47:21.125504+0000 2024-09-15T21:48:33.599 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: local pgid=1.2 2024-09-15T21:48:33.599 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2018: wait_for_scrub: local last_scrub=2024-09-15T21:47:21.125504+0000 2024-09-15T21:48:33.599 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2019: wait_for_scrub: local sname=last_scrub_stamp 2024-09-15T21:48:33.599 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i=0 )) 2024-09-15T21:48:33.599 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-15T21:48:33.599 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.2 last_scrub_stamp 2024-09-15T21:48:33.600 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.2 2024-09-15T21:48:33.600 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-15T21:48:33.600 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-15T21:48:33.600 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.2") | .last_scrub_stamp' 2024-09-15T21:48:33.873 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-15T21:48:28.877740+0000 '>' 2024-09-15T21:47:21.125504+0000 2024-09-15T21:48:33.873 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2023: wait_for_scrub: return 0 2024-09-15T21:48:33.873 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6246: TEST_request_scrub_priority: for opg in $otherpgs $pg 2024-09-15T21:48:33.873 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6248: TEST_request_scrub_priority: wait_for_scrub 1.3 2024-09-15T21:47:21.125504+0000 2024-09-15T21:48:33.873 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: local pgid=1.3 2024-09-15T21:48:33.873 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2018: wait_for_scrub: local last_scrub=2024-09-15T21:47:21.125504+0000 2024-09-15T21:48:33.873 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2019: wait_for_scrub: local sname=last_scrub_stamp 2024-09-15T21:48:33.874 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i=0 )) 2024-09-15T21:48:33.874 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-15T21:48:33.874 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.3 last_scrub_stamp 2024-09-15T21:48:33.874 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.3 2024-09-15T21:48:33.874 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-15T21:48:33.874 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-15T21:48:33.874 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.3") | .last_scrub_stamp' 2024-09-15T21:48:34.173 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-15T21:48:29.905603+0000 '>' 2024-09-15T21:47:21.125504+0000 2024-09-15T21:48:34.173 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2023: wait_for_scrub: return 0 2024-09-15T21:48:34.173 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6246: TEST_request_scrub_priority: for opg in $otherpgs $pg 2024-09-15T21:48:34.173 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6248: TEST_request_scrub_priority: wait_for_scrub 1.4 2024-09-15T21:47:21.125504+0000 2024-09-15T21:48:34.173 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: local pgid=1.4 2024-09-15T21:48:34.173 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2018: wait_for_scrub: local last_scrub=2024-09-15T21:47:21.125504+0000 2024-09-15T21:48:34.173 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2019: wait_for_scrub: local sname=last_scrub_stamp 2024-09-15T21:48:34.173 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i=0 )) 2024-09-15T21:48:34.173 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-15T21:48:34.174 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.4 last_scrub_stamp 2024-09-15T21:48:34.174 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.4 2024-09-15T21:48:34.174 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-15T21:48:34.174 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-15T21:48:34.174 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.4") | .last_scrub_stamp' 2024-09-15T21:48:34.447 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-15T21:48:30.896071+0000 '>' 2024-09-15T21:47:21.125504+0000 2024-09-15T21:48:34.448 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2023: wait_for_scrub: return 0 2024-09-15T21:48:34.448 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6246: TEST_request_scrub_priority: for opg in $otherpgs $pg 2024-09-15T21:48:34.448 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6248: TEST_request_scrub_priority: wait_for_scrub 1.5 2024-09-15T21:47:21.125504+0000 2024-09-15T21:48:34.448 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: local pgid=1.5 2024-09-15T21:48:34.448 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2018: wait_for_scrub: local last_scrub=2024-09-15T21:47:21.125504+0000 2024-09-15T21:48:34.448 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2019: wait_for_scrub: local sname=last_scrub_stamp 2024-09-15T21:48:34.448 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i=0 )) 2024-09-15T21:48:34.448 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-15T21:48:34.448 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.5 last_scrub_stamp 2024-09-15T21:48:34.448 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.5 2024-09-15T21:48:34.448 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-15T21:48:34.448 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-15T21:48:34.448 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.5") | .last_scrub_stamp' 2024-09-15T21:48:34.721 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-08T21:45:49.088716+0000 '>' 2024-09-15T21:47:21.125504+0000 2024-09-15T21:48:34.721 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-15T21:48:35.722 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-15T21:48:35.722 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-15T21:48:35.723 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.5 last_scrub_stamp 2024-09-15T21:48:35.723 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.5 2024-09-15T21:48:35.723 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-15T21:48:35.723 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-15T21:48:35.723 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.5") | .last_scrub_stamp' 2024-09-15T21:48:36.008 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-08T21:45:49.088716+0000 '>' 2024-09-15T21:47:21.125504+0000 2024-09-15T21:48:36.008 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-15T21:48:37.009 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-15T21:48:37.009 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-15T21:48:37.010 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.5 last_scrub_stamp 2024-09-15T21:48:37.010 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.5 2024-09-15T21:48:37.010 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-15T21:48:37.010 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-15T21:48:37.010 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.5") | .last_scrub_stamp' 2024-09-15T21:48:37.293 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-15T21:48:31.867565+0000 '>' 2024-09-15T21:47:21.125504+0000 2024-09-15T21:48:37.293 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2023: wait_for_scrub: return 0 2024-09-15T21:48:37.293 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6246: TEST_request_scrub_priority: for opg in $otherpgs $pg 2024-09-15T21:48:37.293 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6248: TEST_request_scrub_priority: wait_for_scrub 1.6 2024-09-15T21:47:21.125504+0000 2024-09-15T21:48:37.293 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: local pgid=1.6 2024-09-15T21:48:37.293 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2018: wait_for_scrub: local last_scrub=2024-09-15T21:47:21.125504+0000 2024-09-15T21:48:37.294 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2019: wait_for_scrub: local sname=last_scrub_stamp 2024-09-15T21:48:37.294 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i=0 )) 2024-09-15T21:48:37.294 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-15T21:48:37.294 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.6 last_scrub_stamp 2024-09-15T21:48:37.294 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.6 2024-09-15T21:48:37.294 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-15T21:48:37.294 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-15T21:48:37.294 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.6") | .last_scrub_stamp' 2024-09-15T21:48:37.601 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-15T21:48:32.866866+0000 '>' 2024-09-15T21:47:21.125504+0000 2024-09-15T21:48:37.601 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2023: wait_for_scrub: return 0 2024-09-15T21:48:37.601 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6246: TEST_request_scrub_priority: for opg in $otherpgs $pg 2024-09-15T21:48:37.601 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6248: TEST_request_scrub_priority: wait_for_scrub 1.7 2024-09-15T21:47:21.125504+0000 2024-09-15T21:48:37.601 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: local pgid=1.7 2024-09-15T21:48:37.601 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2018: wait_for_scrub: local last_scrub=2024-09-15T21:47:21.125504+0000 2024-09-15T21:48:37.601 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2019: wait_for_scrub: local sname=last_scrub_stamp 2024-09-15T21:48:37.601 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i=0 )) 2024-09-15T21:48:37.601 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-15T21:48:37.601 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.7 last_scrub_stamp 2024-09-15T21:48:37.601 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.7 2024-09-15T21:48:37.601 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-15T21:48:37.601 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-15T21:48:37.602 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.7") | .last_scrub_stamp' 2024-09-15T21:48:37.874 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-15T21:48:33.916597+0000 '>' 2024-09-15T21:47:21.125504+0000 2024-09-15T21:48:37.875 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2023: wait_for_scrub: return 0 2024-09-15T21:48:37.875 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6246: TEST_request_scrub_priority: for opg in $otherpgs $pg 2024-09-15T21:48:37.875 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6248: TEST_request_scrub_priority: wait_for_scrub 1.1 2024-09-15T21:47:21.125504+0000 2024-09-15T21:48:37.875 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: local pgid=1.1 2024-09-15T21:48:37.875 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2018: wait_for_scrub: local last_scrub=2024-09-15T21:47:21.125504+0000 2024-09-15T21:48:37.875 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2019: wait_for_scrub: local sname=last_scrub_stamp 2024-09-15T21:48:37.875 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i=0 )) 2024-09-15T21:48:37.875 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-15T21:48:37.875 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.1 last_scrub_stamp 2024-09-15T21:48:37.875 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.1 2024-09-15T21:48:37.875 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-15T21:48:37.875 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-15T21:48:37.875 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.1") | .last_scrub_stamp' 2024-09-15T21:48:38.143 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-15T21:47:48.501739+0000 '>' 2024-09-15T21:47:21.125504+0000 2024-09-15T21:48:38.143 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2023: wait_for_scrub: return 0 2024-09-15T21:48:38.143 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6252: TEST_request_scrub_priority: grep 'log_channel.*scrub ok' td/osd-scrub-repair/osd.0.log 2024-09-15T21:48:38.143 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6252: TEST_request_scrub_priority: grep -v purged_snaps 2024-09-15T21:48:38.143 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6252: TEST_request_scrub_priority: head -1 2024-09-15T21:48:38.143 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6252: TEST_request_scrub_priority: sed 's/.*[[]DBG[]]//' 2024-09-15T21:48:38.143 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6252: TEST_request_scrub_priority: grep -q 1.1 2024-09-15T21:48:38.146 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:66: run: teardown td/osd-scrub-repair 2024-09-15T21:48:38.146 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/osd-scrub-repair 2024-09-15T21:48:38.146 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-15T21:48:38.146 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/osd-scrub-repair KILL 2024-09-15T21:48:38.146 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-15T21:48:38.146 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-15T21:48:38.147 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-15T21:48:38.147 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-15T21:48:38.147 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-15T21:48:38.253 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-15T21:48:38.253 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-15T21:48:38.254 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-15T21:48:38.255 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-15T21:48:38.256 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-15T21:48:38.256 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-15T21:48:38.256 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-15T21:48:38.257 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T21:48:38.257 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-15T21:48:38.257 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-15T21:48:38.258 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T21:48:38.259 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-15T21:48:38.259 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-15T21:48:38.259 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/osd-scrub-repair 2024-09-15T21:48:38.268 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-15T21:48:38.268 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T21:48:38.268 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52602 2024-09-15T21:48:38.268 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.52602 2024-09-15T21:48:38.270 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-15T21:48:38.270 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-15T21:48:38.270 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:63: run: for func in $funcs 2024-09-15T21:48:38.270 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:64: run: setup td/osd-scrub-repair 2024-09-15T21:48:38.270 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/osd-scrub-repair 2024-09-15T21:48:38.270 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/osd-scrub-repair 2024-09-15T21:48:38.270 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/osd-scrub-repair 2024-09-15T21:48:38.270 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-15T21:48:38.270 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/osd-scrub-repair KILL 2024-09-15T21:48:38.270 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-15T21:48:38.271 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-15T21:48:38.271 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-15T21:48:38.271 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-15T21:48:38.271 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-15T21:48:38.272 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-15T21:48:38.272 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-15T21:48:38.273 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-15T21:48:38.273 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-15T21:48:38.274 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-15T21:48:38.275 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-15T21:48:38.275 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-15T21:48:38.276 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T21:48:38.276 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-15T21:48:38.276 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-15T21:48:38.276 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T21:48:38.277 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-15T21:48:38.280 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-15T21:48:38.280 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/osd-scrub-repair 2024-09-15T21:48:38.280 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-15T21:48:38.280 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T21:48:38.280 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52602 2024-09-15T21:48:38.280 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.52602 2024-09-15T21:48:38.281 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-15T21:48:38.281 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-15T21:48:38.281 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/osd-scrub-repair 2024-09-15T21:48:38.282 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2024-09-15T21:48:38.283 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T21:48:38.283 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52602 2024-09-15T21:48:38.283 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.52602 2024-09-15T21:48:38.284 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2024-09-15T21:48:38.284 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 4096 -le 1024 ']' 2024-09-15T21:48:38.284 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2024-09-15T21:48:38.284 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/osd-scrub-repair 1' TERM HUP INT 2024-09-15T21:48:38.284 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:65: run: TEST_scrub_warning td/osd-scrub-repair 2024-09-15T21:48:38.285 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5845: TEST_scrub_warning: local dir=td/osd-scrub-repair 2024-09-15T21:48:38.285 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5846: TEST_scrub_warning: local poolname=psr_pool 2024-09-15T21:48:38.285 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5847: TEST_scrub_warning: local objname=POBJ 2024-09-15T21:48:38.285 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5848: TEST_scrub_warning: local scrubs=5 2024-09-15T21:48:38.285 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5849: TEST_scrub_warning: local deep_scrubs=5 2024-09-15T21:48:38.285 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5850: TEST_scrub_warning: local i1_day=86400 2024-09-15T21:48:38.285 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5851: TEST_scrub_warning: calc 86400 '*' 7 2024-09-15T21:48:38.285 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1566: calc: awk 'BEGIN{print 86400 * 7}' 2024-09-15T21:48:38.286 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5851: TEST_scrub_warning: local i7_days=604800 2024-09-15T21:48:38.286 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5852: TEST_scrub_warning: calc 86400 '*' 14 2024-09-15T21:48:38.286 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1566: calc: awk 'BEGIN{print 86400 * 14}' 2024-09-15T21:48:38.288 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5852: TEST_scrub_warning: local i14_days=1209600 2024-09-15T21:48:38.288 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5853: TEST_scrub_warning: local overdue=0.5 2024-09-15T21:48:38.288 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5854: TEST_scrub_warning: calc 604800 + 86400 + '(' 604800 '*' 0.5 ')' 2024-09-15T21:48:38.288 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1566: calc: awk 'BEGIN{print 604800 + 86400 + ( 604800 * 0.5 )}' 2024-09-15T21:48:38.289 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5854: TEST_scrub_warning: local conf_overdue_seconds=993600 2024-09-15T21:48:38.290 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5855: TEST_scrub_warning: calc 1209600 + 86400 + '(' 1209600 '*' 0.5 ')' 2024-09-15T21:48:38.290 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1566: calc: awk 'BEGIN{print 1209600 + 86400 + ( 1209600 * 0.5 )}' 2024-09-15T21:48:38.292 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5855: TEST_scrub_warning: local pool_overdue_seconds=1900800 2024-09-15T21:48:38.292 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5857: TEST_scrub_warning: run_mon td/osd-scrub-repair a --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-09-15T21:48:38.292 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/osd-scrub-repair 2024-09-15T21:48:38.292 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2024-09-15T21:48:38.292 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2024-09-15T21:48:38.293 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2024-09-15T21:48:38.293 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/osd-scrub-repair/a 2024-09-15T21:48:38.293 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-scrub-repair/a --run-dir=td/osd-scrub-repair --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-09-15T21:48:38.711 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2024-09-15T21:48:38.711 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-15T21:48:38.711 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-15T21:48:38.711 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-15T21:48:38.711 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T21:48:38.712 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52602 2024-09-15T21:48:38.712 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.52602/$cluster-$name.asok' 2024-09-15T21:48:38.712 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-scrub-repair/a '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.52602/$cluster-$name.asok' --mon-cluster-log-file=td/osd-scrub-repair/log --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-09-15T21:48:38.911 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2024-09-15T21:48:38.912 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2024-09-15T21:48:38.912 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-15T21:48:38.912 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-15T21:48:38.912 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2024-09-15T21:48:38.913 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2024-09-15T21:48:38.914 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-15T21:48:38.914 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-15T21:48:38.914 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-15T21:48:38.914 INFO:tasks.workunit.client.0.smithi136.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-15T21:48:38.914 INFO:tasks.workunit.client.0.smithi136.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T21:48:38.914 INFO:tasks.workunit.client.0.smithi136.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52602 2024-09-15T21:48:38.914 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.52602/ceph-mon.a.asok 2024-09-15T21:48:38.914 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-15T21:48:38.914 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.52602/ceph-mon.a.asok config get fsid 2024-09-15T21:48:38.994 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2024-09-15T21:48:38.994 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-15T21:48:38.994 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-15T21:48:38.994 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2024-09-15T21:48:38.994 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2024-09-15T21:48:38.994 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-15T21:48:38.994 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-15T21:48:38.994 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-15T21:48:38.995 INFO:tasks.workunit.client.0.smithi136.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-15T21:48:38.995 INFO:tasks.workunit.client.0.smithi136.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T21:48:38.996 INFO:tasks.workunit.client.0.smithi136.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52602 2024-09-15T21:48:38.996 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.52602/ceph-mon.a.asok 2024-09-15T21:48:38.996 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-15T21:48:38.996 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.52602/ceph-mon.a.asok config get mon_host 2024-09-15T21:48:39.075 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5858: TEST_scrub_warning: run_mgr td/osd-scrub-repair x --mon_warn_pg_not_scrubbed_ratio=0.5 --mon_warn_pg_not_deep_scrubbed_ratio=0.5 2024-09-15T21:48:39.075 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: run_mgr: local dir=td/osd-scrub-repair 2024-09-15T21:48:39.075 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: run_mgr: shift 2024-09-15T21:48:39.075 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: run_mgr: local id=x 2024-09-15T21:48:39.075 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: run_mgr: shift 2024-09-15T21:48:39.075 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:548: run_mgr: local data=td/osd-scrub-repair/x 2024-09-15T21:48:39.075 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:550: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-15T21:48:39.888 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: get_asok_path 2024-09-15T21:48:39.888 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-15T21:48:39.888 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-15T21:48:39.888 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-15T21:48:39.888 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T21:48:39.888 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52602 2024-09-15T21:48:39.889 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.52602/$cluster-$name.asok' 2024-09-15T21:48:39.889 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-15T21:48:39.889 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-scrub-repair/x '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.52602/$cluster-$name.asok' --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr --mon_warn_pg_not_scrubbed_ratio=0.5 --mon_warn_pg_not_deep_scrubbed_ratio=0.5 2024-09-15T21:48:39.921 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5859: TEST_scrub_warning: run_osd td/osd-scrub-repair 0 --osd_scrub_backoff_ratio=0 2024-09-15T21:48:39.921 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-scrub-repair 2024-09-15T21:48:39.921 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-15T21:48:39.921 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=0 2024-09-15T21:48:39.921 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-15T21:48:39.921 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-scrub-repair/0 2024-09-15T21:48:39.921 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=ce4892c7-6c6d-4b4f-8e6f-07a3e11133e5 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-15T21:48:39.921 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T21:48:39.921 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T21:48:39.921 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T21:48:39.921 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-09-15T21:48:39.921 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-09-15T21:48:39.922 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-15T21:48:39.922 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-15T21:48:39.922 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-15T21:48:39.922 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-15T21:48:39.922 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-15T21:48:39.922 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-15T21:48:39.922 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-15T21:48:39.922 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T21:48:39.922 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52602 2024-09-15T21:48:39.922 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.52602/$cluster-$name.asok' 2024-09-15T21:48:39.922 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.52602/$cluster-$name.asok' 2024-09-15T21:48:39.922 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-15T21:48:39.922 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-15T21:48:39.922 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-15T21:48:39.923 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-15T21:48:39.923 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-15T21:48:39.923 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T21:48:39.923 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T21:48:39.923 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T21:48:39.923 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T21:48:39.923 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-15T21:48:39.923 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=--osd_scrub_backoff_ratio=0 2024-09-15T21:48:39.923 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-scrub-repair/0 2024-09-15T21:48:39.924 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-15T21:48:39.924 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=1c0070db-67d1-466d-8bab-e13d155ab34f 2024-09-15T21:48:39.924 INFO:tasks.workunit.client.0.smithi136.stdout:add osd0 1c0070db-67d1-466d-8bab-e13d155ab34f 2024-09-15T21:48:39.924 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd0 1c0070db-67d1-466d-8bab-e13d155ab34f' 2024-09-15T21:48:39.924 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-15T21:48:39.942 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQA3VudmJA0AOBAAove8r3VIl5iOsnofoaukCg== 2024-09-15T21:48:39.942 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQA3VudmJA0AOBAAove8r3VIl5iOsnofoaukCg=="}' 2024-09-15T21:48:39.942 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 1c0070db-67d1-466d-8bab-e13d155ab34f -i td/osd-scrub-repair/0/new.json 2024-09-15T21:48:40.143 INFO:tasks.workunit.client.0.smithi136.stdout:0 2024-09-15T21:48:40.153 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-scrub-repair/0/new.json 2024-09-15T21:48:40.155 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 0 --fsid=ce4892c7-6c6d-4b4f-8e6f-07a3e11133e5 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.52602/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_scrub_backoff_ratio=0 --mkfs --key AQA3VudmJA0AOBAAove8r3VIl5iOsnofoaukCg== --osd-uuid 1c0070db-67d1-466d-8bab-e13d155ab34f 2024-09-15T21:48:40.185 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:48:40.184+0000 7f30afd54640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:48:40.194 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:48:40.192+0000 7f30afd54640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:48:40.195 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:48:40.193+0000 7f30afd54640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:48:40.195 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:48:40.194+0000 7f30afd54640 -1 bdev(0x55d4e5bd5c00 td/osd-scrub-repair/0/block) open stat got: (1) Operation not permitted 2024-09-15T21:48:40.196 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:48:40.194+0000 7f30afd54640 -1 bluestore(td/osd-scrub-repair/0) _read_fsid unparsable uuid 2024-09-15T21:48:46.241 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-scrub-repair/0/keyring 2024-09-15T21:48:46.241 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-15T21:48:46.242 INFO:tasks.workunit.client.0.smithi136.stdout:adding osd0 key to auth repository 2024-09-15T21:48:46.242 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd0 key to auth repository 2024-09-15T21:48:46.242 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-scrub-repair/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-15T21:48:46.599 INFO:tasks.workunit.client.0.smithi136.stdout:start osd.0 2024-09-15T21:48:46.600 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.0 2024-09-15T21:48:46.600 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 0 --fsid=ce4892c7-6c6d-4b4f-8e6f-07a3e11133e5 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.52602/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_scrub_backoff_ratio=0 2024-09-15T21:48:46.600 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-15T21:48:46.600 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-15T21:48:46.600 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-15T21:48:46.627 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:48:46.626+0000 7fdbe1169640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:48:46.629 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:48:46.628+0000 7fdbe1169640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:48:46.630 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:48:46.629+0000 7fdbe1169640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:48:46.880 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 0 2024-09-15T21:48:46.880 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-15T21:48:46.880 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2024-09-15T21:48:46.880 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-15T21:48:46.880 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-15T21:48:46.880 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:48:46.880 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-15T21:48:46.881 INFO:tasks.workunit.client.0.smithi136.stdout:0 2024-09-15T21:48:46.881 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:48:46.881 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:48:47.158 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:48:48.158 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:48:48.158 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:48:48.159 INFO:tasks.workunit.client.0.smithi136.stdout:1 2024-09-15T21:48:48.159 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-15T21:48:48.159 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:48:48.159 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:48:48.242 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:48:48.241+0000 7fdbe1169640 -1 Falling back to public interface 2024-09-15T21:48:48.445 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:48:49.446 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:48:49.447 INFO:tasks.workunit.client.0.smithi136.stdout:2 2024-09-15T21:48:49.447 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:48:49.447 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-15T21:48:49.447 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:48:49.448 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:48:49.734 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:48:50.735 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:48:50.736 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:48:50.736 INFO:tasks.workunit.client.0.smithi136.stdout:3 2024-09-15T21:48:50.736 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-15T21:48:50.736 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:48:50.737 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:48:51.017 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:48:51.147 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:48:51.146+0000 7fdbe1169640 -1 osd.0 0 log_to_monitors true 2024-09-15T21:48:52.018 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:48:52.018 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:48:52.018 INFO:tasks.workunit.client.0.smithi136.stdout:4 2024-09-15T21:48:52.019 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-15T21:48:52.019 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:48:52.019 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:48:52.340 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:48:53.342 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:48:53.342 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:48:53.342 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-15T21:48:53.342 INFO:tasks.workunit.client.0.smithi136.stdout:5 2024-09-15T21:48:53.342 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:48:53.343 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:48:53.630 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:48:54.631 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:48:54.631 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:48:54.632 INFO:tasks.workunit.client.0.smithi136.stdout:6 2024-09-15T21:48:54.632 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-15T21:48:54.632 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:48:54.632 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:48:55.030 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:48:56.032 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:48:56.032 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:48:56.032 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-15T21:48:56.032 INFO:tasks.workunit.client.0.smithi136.stdout:7 2024-09-15T21:48:56.033 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:48:56.033 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:48:56.318 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:48:57.319 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:48:57.319 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:48:57.319 INFO:tasks.workunit.client.0.smithi136.stdout:8 2024-09-15T21:48:57.320 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-15T21:48:57.320 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:48:57.320 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:48:57.605 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:48:58.606 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:48:58.606 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:48:58.607 INFO:tasks.workunit.client.0.smithi136.stdout:9 2024-09-15T21:48:58.607 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-15T21:48:58.607 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:48:58.607 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:48:58.986 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:48:59.987 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:48:59.988 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:48:59.988 INFO:tasks.workunit.client.0.smithi136.stdout:10 2024-09-15T21:48:59.988 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-15T21:48:59.989 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:48:59.989 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:49:00.272 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:49:01.273 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:49:01.273 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:49:01.274 INFO:tasks.workunit.client.0.smithi136.stdout:11 2024-09-15T21:49:01.274 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-15T21:49:01.274 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:49:01.274 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:49:01.561 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:49:02.562 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:49:02.562 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:49:02.562 INFO:tasks.workunit.client.0.smithi136.stdout:12 2024-09-15T21:49:02.563 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-15T21:49:02.563 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:49:02.563 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:49:02.930 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:49:03.932 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:49:03.932 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:49:03.932 INFO:tasks.workunit.client.0.smithi136.stdout:13 2024-09-15T21:49:03.933 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-15T21:49:03.933 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:49:03.933 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:49:04.461 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:49:05.463 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:49:05.463 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:49:05.463 INFO:tasks.workunit.client.0.smithi136.stdout:14 2024-09-15T21:49:05.464 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-15T21:49:05.464 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:49:05.464 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:49:06.153 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:49:07.155 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:49:07.155 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:49:07.155 INFO:tasks.workunit.client.0.smithi136.stdout:15 2024-09-15T21:49:07.156 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-15T21:49:07.156 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:49:07.156 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:49:07.754 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:49:08.755 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:49:08.756 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:49:08.756 INFO:tasks.workunit.client.0.smithi136.stdout:16 2024-09-15T21:49:08.756 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-15T21:49:08.756 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:49:08.756 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:49:09.154 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:49:10.155 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:49:10.155 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:49:10.156 INFO:tasks.workunit.client.0.smithi136.stdout:17 2024-09-15T21:49:10.156 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-15T21:49:10.156 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:49:10.156 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:49:10.552 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:49:11.553 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:49:11.554 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:49:11.554 INFO:tasks.workunit.client.0.smithi136.stdout:18 2024-09-15T21:49:11.554 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 18 2024-09-15T21:49:11.554 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:49:11.555 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:49:12.034 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:49:13.036 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:49:13.036 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:49:13.036 INFO:tasks.workunit.client.0.smithi136.stdout:19 2024-09-15T21:49:13.036 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 19 2024-09-15T21:49:13.037 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:49:13.037 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:49:13.402 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:49:14.404 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:49:14.404 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:49:14.404 INFO:tasks.workunit.client.0.smithi136.stdout:20 2024-09-15T21:49:14.405 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 20 2024-09-15T21:49:14.405 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:49:14.405 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:49:14.693 INFO:tasks.workunit.client.0.smithi136.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/3381174584,v1:127.0.0.1:6803/3381174584] [v2:127.0.0.1:6804/3381174584,v1:127.0.0.1:6805/3381174584] exists,up 1c0070db-67d1-466d-8bab-e13d155ab34f 2024-09-15T21:49:14.693 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-15T21:49:14.693 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-15T21:49:14.693 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-15T21:49:14.693 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5861: TEST_scrub_warning: expr 5 + 5 2024-09-15T21:49:14.694 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5861: TEST_scrub_warning: seq 1 10 2024-09-15T21:49:14.695 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5861: TEST_scrub_warning: for i in $(seq 1 $(expr $scrubs + $deep_scrubs)) 2024-09-15T21:49:14.696 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5863: TEST_scrub_warning: create_pool psr_pool-1 1 1 2024-09-15T21:49:14.696 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create psr_pool-1 1 1 2024-09-15T21:49:15.103 INFO:tasks.workunit.client.0.smithi136.stderr:pool 'psr_pool-1' created 2024-09-15T21:49:15.115 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2024-09-15T21:49:16.117 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5864: TEST_scrub_warning: wait_for_clean 2024-09-15T21:49:16.118 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-15T21:49:16.118 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-15T21:49:16.118 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-15T21:49:16.118 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-15T21:49:16.119 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-15T21:49:16.119 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-15T21:49:16.119 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-15T21:49:16.119 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-15T21:49:16.119 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-15T21:49:16.219 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-15T21:49:16.219 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-15T21:49:16.219 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-15T21:49:16.219 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-15T21:49:16.219 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-15T21:49:16.219 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-15T21:49:16.495 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids=0 2024-09-15T21:49:16.495 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-15T21:49:16.495 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T21:49:16.495 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-15T21:49:16.597 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836487 2024-09-15T21:49:16.598 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836487 2024-09-15T21:49:16.598 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836487' 2024-09-15T21:49:16.598 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T21:49:16.598 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836487 2024-09-15T21:49:16.598 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T21:49:16.599 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-15T21:49:16.600 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836487 2024-09-15T21:49:16.600 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T21:49:16.601 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836487 2024-09-15T21:49:16.601 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836487' 2024-09-15T21:49:16.601 INFO:tasks.workunit.client.0.smithi136.stdout:waiting osd.0 seq 21474836487 2024-09-15T21:49:16.601 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T21:49:16.879 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836486 -lt 21474836487 2024-09-15T21:49:16.880 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-15T21:49:17.881 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-15T21:49:17.881 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T21:49:18.173 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836487 -lt 21474836487 2024-09-15T21:49:18.173 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-15T21:49:18.173 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T21:49:18.174 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T21:49:18.519 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 1 == 0 2024-09-15T21:49:18.520 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-15T21:49:18.520 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-15T21:49:18.521 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-15T21:49:18.521 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T21:49:18.521 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T21:49:18.521 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T21:49:18.521 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T21:49:18.790 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=1 2024-09-15T21:49:18.791 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-15T21:49:18.791 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T21:49:18.791 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T21:49:19.136 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 = 1 2024-09-15T21:49:19.136 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-15T21:49:19.136 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-15T21:49:19.136 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5865: TEST_scrub_warning: '[' 1 = 1 ']' 2024-09-15T21:49:19.136 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5867: TEST_scrub_warning: ceph osd pool set psr_pool-1 scrub_max_interval 1209600 2024-09-15T21:49:19.511 INFO:tasks.workunit.client.0.smithi136.stderr:set pool 1 scrub_max_interval to 1209600 2024-09-15T21:49:19.526 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5869: TEST_scrub_warning: expr 5 + 1 2024-09-15T21:49:19.526 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5869: TEST_scrub_warning: '[' 1 = 6 ']' 2024-09-15T21:49:19.526 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5861: TEST_scrub_warning: for i in $(seq 1 $(expr $scrubs + $deep_scrubs)) 2024-09-15T21:49:19.526 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5863: TEST_scrub_warning: create_pool psr_pool-2 1 1 2024-09-15T21:49:19.526 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create psr_pool-2 1 1 2024-09-15T21:49:20.013 INFO:tasks.workunit.client.0.smithi136.stderr:pool 'psr_pool-2' created 2024-09-15T21:49:20.025 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2024-09-15T21:49:21.027 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5864: TEST_scrub_warning: wait_for_clean 2024-09-15T21:49:21.027 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-15T21:49:21.027 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-15T21:49:21.027 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-15T21:49:21.027 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-15T21:49:21.027 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-15T21:49:21.027 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-15T21:49:21.028 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-15T21:49:21.028 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-15T21:49:21.028 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-15T21:49:21.136 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-15T21:49:21.136 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-15T21:49:21.136 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-15T21:49:21.136 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-15T21:49:21.136 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-15T21:49:21.136 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-15T21:49:21.409 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids=0 2024-09-15T21:49:21.410 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-15T21:49:21.410 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T21:49:21.410 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-15T21:49:21.513 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836489 2024-09-15T21:49:21.513 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836489 2024-09-15T21:49:21.513 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836489' 2024-09-15T21:49:21.513 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T21:49:21.514 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836489 2024-09-15T21:49:21.514 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T21:49:21.514 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-15T21:49:21.516 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836489 2024-09-15T21:49:21.516 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T21:49:21.516 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836489 2024-09-15T21:49:21.517 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836489' 2024-09-15T21:49:21.517 INFO:tasks.workunit.client.0.smithi136.stdout:waiting osd.0 seq 21474836489 2024-09-15T21:49:21.517 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T21:49:21.788 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836488 -lt 21474836489 2024-09-15T21:49:21.788 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-15T21:49:22.789 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-15T21:49:22.790 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T21:49:23.076 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836489 -lt 21474836489 2024-09-15T21:49:23.077 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-15T21:49:23.077 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T21:49:23.077 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T21:49:23.419 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 2 == 0 2024-09-15T21:49:23.419 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-15T21:49:23.420 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-15T21:49:23.421 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-15T21:49:23.421 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T21:49:23.421 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T21:49:23.421 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T21:49:23.421 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T21:49:23.692 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=2 2024-09-15T21:49:23.692 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-15T21:49:23.693 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T21:49:23.693 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T21:49:24.038 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 2 = 2 2024-09-15T21:49:24.039 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-15T21:49:24.039 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-15T21:49:24.039 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5865: TEST_scrub_warning: '[' 2 = 1 ']' 2024-09-15T21:49:24.039 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5869: TEST_scrub_warning: expr 5 + 1 2024-09-15T21:49:24.040 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5869: TEST_scrub_warning: '[' 2 = 6 ']' 2024-09-15T21:49:24.041 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5861: TEST_scrub_warning: for i in $(seq 1 $(expr $scrubs + $deep_scrubs)) 2024-09-15T21:49:24.041 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5863: TEST_scrub_warning: create_pool psr_pool-3 1 1 2024-09-15T21:49:24.041 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create psr_pool-3 1 1 2024-09-15T21:49:24.422 INFO:tasks.workunit.client.0.smithi136.stderr:pool 'psr_pool-3' created 2024-09-15T21:49:24.433 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2024-09-15T21:49:25.435 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5864: TEST_scrub_warning: wait_for_clean 2024-09-15T21:49:25.435 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-15T21:49:25.435 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-15T21:49:25.436 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-15T21:49:25.436 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-15T21:49:25.437 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-15T21:49:25.437 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-15T21:49:25.437 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-15T21:49:25.437 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-15T21:49:25.438 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-15T21:49:25.541 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-15T21:49:25.541 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-15T21:49:25.541 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-15T21:49:25.541 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-15T21:49:25.541 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-15T21:49:25.541 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-15T21:49:25.816 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids=0 2024-09-15T21:49:25.817 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-15T21:49:25.817 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T21:49:25.817 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-15T21:49:25.921 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836491 2024-09-15T21:49:25.921 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836491 2024-09-15T21:49:25.921 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836491' 2024-09-15T21:49:25.921 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T21:49:25.921 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836491 2024-09-15T21:49:25.921 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T21:49:25.921 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-15T21:49:25.922 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836491 2024-09-15T21:49:25.922 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T21:49:25.923 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836491 2024-09-15T21:49:25.923 INFO:tasks.workunit.client.0.smithi136.stdout:waiting osd.0 seq 21474836491 2024-09-15T21:49:25.923 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836491' 2024-09-15T21:49:25.923 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T21:49:26.201 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836490 -lt 21474836491 2024-09-15T21:49:26.201 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-15T21:49:27.202 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-15T21:49:27.203 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T21:49:27.491 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836490 -lt 21474836491 2024-09-15T21:49:27.492 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-15T21:49:28.493 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 299 -eq 0 ']' 2024-09-15T21:49:28.493 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T21:49:28.773 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836491 -lt 21474836491 2024-09-15T21:49:28.774 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-15T21:49:28.774 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T21:49:28.774 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T21:49:29.121 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 3 == 0 2024-09-15T21:49:29.121 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-15T21:49:29.122 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-15T21:49:29.122 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-15T21:49:29.122 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T21:49:29.123 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T21:49:29.123 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T21:49:29.123 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T21:49:29.392 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=3 2024-09-15T21:49:29.393 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-15T21:49:29.393 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T21:49:29.393 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T21:49:29.736 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 3 = 3 2024-09-15T21:49:29.736 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-15T21:49:29.736 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-15T21:49:29.736 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5865: TEST_scrub_warning: '[' 3 = 1 ']' 2024-09-15T21:49:29.736 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5869: TEST_scrub_warning: expr 5 + 1 2024-09-15T21:49:29.737 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5869: TEST_scrub_warning: '[' 3 = 6 ']' 2024-09-15T21:49:29.737 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5861: TEST_scrub_warning: for i in $(seq 1 $(expr $scrubs + $deep_scrubs)) 2024-09-15T21:49:29.737 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5863: TEST_scrub_warning: create_pool psr_pool-4 1 1 2024-09-15T21:49:29.737 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create psr_pool-4 1 1 2024-09-15T21:49:30.139 INFO:tasks.workunit.client.0.smithi136.stderr:pool 'psr_pool-4' created 2024-09-15T21:49:30.150 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2024-09-15T21:49:31.152 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5864: TEST_scrub_warning: wait_for_clean 2024-09-15T21:49:31.152 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-15T21:49:31.152 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-15T21:49:31.152 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-15T21:49:31.153 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-15T21:49:31.153 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-15T21:49:31.153 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-15T21:49:31.153 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-15T21:49:31.154 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-15T21:49:31.154 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-15T21:49:31.260 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-15T21:49:31.260 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-15T21:49:31.261 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-15T21:49:31.261 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-15T21:49:31.261 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-15T21:49:31.261 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-15T21:49:31.537 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids=0 2024-09-15T21:49:31.537 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-15T21:49:31.537 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T21:49:31.538 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-15T21:49:31.641 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836493 2024-09-15T21:49:31.641 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836493 2024-09-15T21:49:31.641 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836493' 2024-09-15T21:49:31.641 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T21:49:31.642 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836493 2024-09-15T21:49:31.642 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T21:49:31.642 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-15T21:49:31.644 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836493 2024-09-15T21:49:31.644 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T21:49:31.644 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836493 2024-09-15T21:49:31.644 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836493' 2024-09-15T21:49:31.645 INFO:tasks.workunit.client.0.smithi136.stdout:waiting osd.0 seq 21474836493 2024-09-15T21:49:31.645 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T21:49:31.939 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836492 -lt 21474836493 2024-09-15T21:49:31.939 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-15T21:49:32.940 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-15T21:49:32.940 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T21:49:33.228 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836493 -lt 21474836493 2024-09-15T21:49:33.229 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-15T21:49:33.229 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T21:49:33.229 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T21:49:33.573 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 4 == 0 2024-09-15T21:49:33.573 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-15T21:49:33.573 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-15T21:49:33.573 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-15T21:49:33.573 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T21:49:33.573 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T21:49:33.573 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T21:49:33.574 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T21:49:33.846 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=4 2024-09-15T21:49:33.846 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-15T21:49:33.847 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T21:49:33.847 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T21:49:34.200 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 = 4 2024-09-15T21:49:34.200 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-15T21:49:34.200 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-15T21:49:34.200 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5865: TEST_scrub_warning: '[' 4 = 1 ']' 2024-09-15T21:49:34.200 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5869: TEST_scrub_warning: expr 5 + 1 2024-09-15T21:49:34.201 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5869: TEST_scrub_warning: '[' 4 = 6 ']' 2024-09-15T21:49:34.201 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5861: TEST_scrub_warning: for i in $(seq 1 $(expr $scrubs + $deep_scrubs)) 2024-09-15T21:49:34.201 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5863: TEST_scrub_warning: create_pool psr_pool-5 1 1 2024-09-15T21:49:34.201 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create psr_pool-5 1 1 2024-09-15T21:49:34.571 INFO:tasks.workunit.client.0.smithi136.stderr:pool 'psr_pool-5' created 2024-09-15T21:49:34.583 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2024-09-15T21:49:35.585 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5864: TEST_scrub_warning: wait_for_clean 2024-09-15T21:49:35.585 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-15T21:49:35.585 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-15T21:49:35.585 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-15T21:49:35.586 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-15T21:49:35.586 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-15T21:49:35.586 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-15T21:49:35.586 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-15T21:49:35.586 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-15T21:49:35.586 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-15T21:49:35.693 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-15T21:49:35.693 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-15T21:49:35.693 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-15T21:49:35.693 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-15T21:49:35.693 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-15T21:49:35.694 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-15T21:49:35.972 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids=0 2024-09-15T21:49:35.973 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-15T21:49:35.973 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T21:49:35.973 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-15T21:49:36.112 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836495 2024-09-15T21:49:36.112 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836495 2024-09-15T21:49:36.112 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836495' 2024-09-15T21:49:36.112 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T21:49:36.113 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836495 2024-09-15T21:49:36.113 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T21:49:36.114 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-15T21:49:36.114 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836495 2024-09-15T21:49:36.114 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T21:49:36.115 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836495 2024-09-15T21:49:36.115 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836495' 2024-09-15T21:49:36.116 INFO:tasks.workunit.client.0.smithi136.stdout:waiting osd.0 seq 21474836495 2024-09-15T21:49:36.116 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T21:49:36.389 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836494 -lt 21474836495 2024-09-15T21:49:36.390 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-15T21:49:37.390 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-15T21:49:37.391 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T21:49:37.677 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836494 -lt 21474836495 2024-09-15T21:49:37.677 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-15T21:49:38.678 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 299 -eq 0 ']' 2024-09-15T21:49:38.679 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T21:49:38.966 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836495 -lt 21474836495 2024-09-15T21:49:38.967 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-15T21:49:38.967 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T21:49:38.968 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T21:49:39.343 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 5 == 0 2024-09-15T21:49:39.343 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-15T21:49:39.343 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-15T21:49:39.343 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-15T21:49:39.343 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T21:49:39.343 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T21:49:39.343 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T21:49:39.344 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T21:49:39.613 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=5 2024-09-15T21:49:39.614 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-15T21:49:39.614 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T21:49:39.614 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T21:49:39.959 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 = 5 2024-09-15T21:49:39.960 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-15T21:49:39.960 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-15T21:49:39.960 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5865: TEST_scrub_warning: '[' 5 = 1 ']' 2024-09-15T21:49:39.960 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5869: TEST_scrub_warning: expr 5 + 1 2024-09-15T21:49:39.961 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5869: TEST_scrub_warning: '[' 5 = 6 ']' 2024-09-15T21:49:39.961 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5861: TEST_scrub_warning: for i in $(seq 1 $(expr $scrubs + $deep_scrubs)) 2024-09-15T21:49:39.961 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5863: TEST_scrub_warning: create_pool psr_pool-6 1 1 2024-09-15T21:49:39.961 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create psr_pool-6 1 1 2024-09-15T21:49:40.496 INFO:tasks.workunit.client.0.smithi136.stderr:pool 'psr_pool-6' created 2024-09-15T21:49:40.508 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2024-09-15T21:49:41.509 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5864: TEST_scrub_warning: wait_for_clean 2024-09-15T21:49:41.510 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-15T21:49:41.510 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-15T21:49:41.510 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-15T21:49:41.511 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-15T21:49:41.511 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-15T21:49:41.511 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-15T21:49:41.511 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-15T21:49:41.511 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-15T21:49:41.511 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-15T21:49:41.614 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-15T21:49:41.614 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-15T21:49:41.614 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-15T21:49:41.614 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-15T21:49:41.614 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-15T21:49:41.614 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-15T21:49:41.892 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids=0 2024-09-15T21:49:41.892 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-15T21:49:41.892 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T21:49:41.892 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-15T21:49:42.010 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836497 2024-09-15T21:49:42.011 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836497 2024-09-15T21:49:42.011 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836497' 2024-09-15T21:49:42.011 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T21:49:42.011 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836497 2024-09-15T21:49:42.011 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T21:49:42.012 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-15T21:49:42.013 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836497 2024-09-15T21:49:42.013 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T21:49:42.014 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836497 2024-09-15T21:49:42.014 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836497' 2024-09-15T21:49:42.014 INFO:tasks.workunit.client.0.smithi136.stdout:waiting osd.0 seq 21474836497 2024-09-15T21:49:42.014 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T21:49:42.290 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836496 -lt 21474836497 2024-09-15T21:49:42.290 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-15T21:49:43.290 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-15T21:49:43.290 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T21:49:43.570 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836496 -lt 21474836497 2024-09-15T21:49:43.571 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-15T21:49:44.572 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 299 -eq 0 ']' 2024-09-15T21:49:44.572 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T21:49:44.857 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836497 -lt 21474836497 2024-09-15T21:49:44.857 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-15T21:49:44.857 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T21:49:44.857 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T21:49:45.201 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 6 == 0 2024-09-15T21:49:45.201 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-15T21:49:45.201 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-15T21:49:45.202 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-15T21:49:45.202 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T21:49:45.202 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T21:49:45.202 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T21:49:45.202 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T21:49:45.474 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=6 2024-09-15T21:49:45.475 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-15T21:49:45.475 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T21:49:45.475 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T21:49:45.817 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 = 6 2024-09-15T21:49:45.817 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-15T21:49:45.818 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-15T21:49:45.818 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5865: TEST_scrub_warning: '[' 6 = 1 ']' 2024-09-15T21:49:45.818 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5869: TEST_scrub_warning: expr 5 + 1 2024-09-15T21:49:45.819 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5869: TEST_scrub_warning: '[' 6 = 6 ']' 2024-09-15T21:49:45.819 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5871: TEST_scrub_warning: ceph osd pool set psr_pool-6 deep_scrub_interval 1209600 2024-09-15T21:49:46.308 INFO:tasks.workunit.client.0.smithi136.stderr:set pool 6 deep_scrub_interval to 1209600 2024-09-15T21:49:46.320 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5861: TEST_scrub_warning: for i in $(seq 1 $(expr $scrubs + $deep_scrubs)) 2024-09-15T21:49:46.320 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5863: TEST_scrub_warning: create_pool psr_pool-7 1 1 2024-09-15T21:49:46.321 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create psr_pool-7 1 1 2024-09-15T21:49:46.692 INFO:tasks.workunit.client.0.smithi136.stderr:pool 'psr_pool-7' created 2024-09-15T21:49:46.704 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2024-09-15T21:49:47.706 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5864: TEST_scrub_warning: wait_for_clean 2024-09-15T21:49:47.706 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-15T21:49:47.706 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-15T21:49:47.706 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-15T21:49:47.707 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-15T21:49:47.707 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-15T21:49:47.707 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-15T21:49:47.707 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-15T21:49:47.707 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-15T21:49:47.707 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-15T21:49:47.814 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-15T21:49:47.814 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-15T21:49:47.814 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-15T21:49:47.815 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-15T21:49:47.815 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-15T21:49:47.815 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-15T21:49:48.094 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids=0 2024-09-15T21:49:48.094 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-15T21:49:48.094 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T21:49:48.094 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-15T21:49:48.199 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836499 2024-09-15T21:49:48.199 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836499 2024-09-15T21:49:48.200 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836499' 2024-09-15T21:49:48.200 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T21:49:48.200 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836499 2024-09-15T21:49:48.200 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T21:49:48.201 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-15T21:49:48.202 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836499 2024-09-15T21:49:48.202 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T21:49:48.203 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836499 2024-09-15T21:49:48.203 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836499' 2024-09-15T21:49:48.203 INFO:tasks.workunit.client.0.smithi136.stdout:waiting osd.0 seq 21474836499 2024-09-15T21:49:48.203 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T21:49:48.478 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836498 -lt 21474836499 2024-09-15T21:49:48.478 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-15T21:49:49.480 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-15T21:49:49.481 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T21:49:49.763 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836498 -lt 21474836499 2024-09-15T21:49:49.763 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-15T21:49:50.764 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 299 -eq 0 ']' 2024-09-15T21:49:50.765 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T21:49:51.050 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836500 -lt 21474836499 2024-09-15T21:49:51.050 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-15T21:49:51.051 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T21:49:51.051 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T21:49:51.398 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 7 == 0 2024-09-15T21:49:51.399 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-15T21:49:51.399 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-15T21:49:51.399 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-15T21:49:51.399 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T21:49:51.399 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T21:49:51.399 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T21:49:51.399 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T21:49:51.671 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=7 2024-09-15T21:49:51.672 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-15T21:49:51.672 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T21:49:51.672 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T21:49:52.015 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 7 = 7 2024-09-15T21:49:52.015 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-15T21:49:52.015 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-15T21:49:52.015 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5865: TEST_scrub_warning: '[' 7 = 1 ']' 2024-09-15T21:49:52.015 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5869: TEST_scrub_warning: expr 5 + 1 2024-09-15T21:49:52.016 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5869: TEST_scrub_warning: '[' 7 = 6 ']' 2024-09-15T21:49:52.016 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5861: TEST_scrub_warning: for i in $(seq 1 $(expr $scrubs + $deep_scrubs)) 2024-09-15T21:49:52.016 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5863: TEST_scrub_warning: create_pool psr_pool-8 1 1 2024-09-15T21:49:52.016 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create psr_pool-8 1 1 2024-09-15T21:49:52.386 INFO:tasks.workunit.client.0.smithi136.stderr:pool 'psr_pool-8' created 2024-09-15T21:49:52.397 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2024-09-15T21:49:53.399 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5864: TEST_scrub_warning: wait_for_clean 2024-09-15T21:49:53.400 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-15T21:49:53.400 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-15T21:49:53.400 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-15T21:49:53.401 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-15T21:49:53.401 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-15T21:49:53.401 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-15T21:49:53.401 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-15T21:49:53.401 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-15T21:49:53.401 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-15T21:49:53.504 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-15T21:49:53.504 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-15T21:49:53.504 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-15T21:49:53.504 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-15T21:49:53.505 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-15T21:49:53.505 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-15T21:49:53.780 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids=0 2024-09-15T21:49:53.781 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-15T21:49:53.781 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T21:49:53.781 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-15T21:49:53.884 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836501 2024-09-15T21:49:53.884 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836501 2024-09-15T21:49:53.884 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836501' 2024-09-15T21:49:53.884 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T21:49:53.884 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836501 2024-09-15T21:49:53.884 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T21:49:53.885 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-15T21:49:53.886 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836501 2024-09-15T21:49:53.886 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T21:49:53.887 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836501 2024-09-15T21:49:53.887 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836501' 2024-09-15T21:49:53.887 INFO:tasks.workunit.client.0.smithi136.stdout:waiting osd.0 seq 21474836501 2024-09-15T21:49:53.887 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T21:49:54.165 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836500 -lt 21474836501 2024-09-15T21:49:54.165 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-15T21:49:55.166 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-15T21:49:55.168 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T21:49:55.453 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836500 -lt 21474836501 2024-09-15T21:49:55.453 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-15T21:49:56.455 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 299 -eq 0 ']' 2024-09-15T21:49:56.456 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T21:49:56.739 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836502 -lt 21474836501 2024-09-15T21:49:56.739 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-15T21:49:56.740 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T21:49:56.740 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T21:49:57.083 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 8 == 0 2024-09-15T21:49:57.083 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-15T21:49:57.084 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-15T21:49:57.084 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-15T21:49:57.084 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T21:49:57.084 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T21:49:57.084 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T21:49:57.084 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T21:49:57.357 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=8 2024-09-15T21:49:57.357 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-15T21:49:57.358 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T21:49:57.358 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T21:49:57.699 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 8 = 8 2024-09-15T21:49:57.699 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-15T21:49:57.699 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-15T21:49:57.699 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5865: TEST_scrub_warning: '[' 8 = 1 ']' 2024-09-15T21:49:57.699 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5869: TEST_scrub_warning: expr 5 + 1 2024-09-15T21:49:57.700 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5869: TEST_scrub_warning: '[' 8 = 6 ']' 2024-09-15T21:49:57.700 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5861: TEST_scrub_warning: for i in $(seq 1 $(expr $scrubs + $deep_scrubs)) 2024-09-15T21:49:57.701 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5863: TEST_scrub_warning: create_pool psr_pool-9 1 1 2024-09-15T21:49:57.701 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create psr_pool-9 1 1 2024-09-15T21:49:58.145 INFO:tasks.workunit.client.0.smithi136.stderr:pool 'psr_pool-9' created 2024-09-15T21:49:58.156 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2024-09-15T21:49:59.158 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5864: TEST_scrub_warning: wait_for_clean 2024-09-15T21:49:59.158 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-15T21:49:59.158 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-15T21:49:59.158 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-15T21:49:59.159 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-15T21:49:59.159 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-15T21:49:59.159 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-15T21:49:59.159 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-15T21:49:59.159 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-15T21:49:59.160 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-15T21:49:59.260 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-15T21:49:59.260 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-15T21:49:59.260 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-15T21:49:59.260 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-15T21:49:59.260 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-15T21:49:59.261 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-15T21:49:59.536 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids=0 2024-09-15T21:49:59.536 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-15T21:49:59.536 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T21:49:59.536 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-15T21:49:59.639 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836503 2024-09-15T21:49:59.639 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836503 2024-09-15T21:49:59.639 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836503' 2024-09-15T21:49:59.639 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T21:49:59.639 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836503 2024-09-15T21:49:59.639 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T21:49:59.640 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-15T21:49:59.641 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836503 2024-09-15T21:49:59.641 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T21:49:59.642 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836503 2024-09-15T21:49:59.642 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836503' 2024-09-15T21:49:59.642 INFO:tasks.workunit.client.0.smithi136.stdout:waiting osd.0 seq 21474836503 2024-09-15T21:49:59.642 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T21:49:59.938 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836502 -lt 21474836503 2024-09-15T21:49:59.938 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-15T21:50:00.939 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-15T21:50:00.939 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T21:50:01.222 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836504 -lt 21474836503 2024-09-15T21:50:01.223 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-15T21:50:01.223 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T21:50:01.223 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T21:50:01.572 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 9 == 0 2024-09-15T21:50:01.572 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-15T21:50:01.572 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-15T21:50:01.572 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-15T21:50:01.573 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T21:50:01.573 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T21:50:01.573 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T21:50:01.573 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T21:50:01.844 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=9 2024-09-15T21:50:01.845 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-15T21:50:01.845 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T21:50:01.845 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T21:50:02.198 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 9 = 9 2024-09-15T21:50:02.198 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-15T21:50:02.198 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-15T21:50:02.198 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5865: TEST_scrub_warning: '[' 9 = 1 ']' 2024-09-15T21:50:02.198 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5869: TEST_scrub_warning: expr 5 + 1 2024-09-15T21:50:02.199 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5869: TEST_scrub_warning: '[' 9 = 6 ']' 2024-09-15T21:50:02.199 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5861: TEST_scrub_warning: for i in $(seq 1 $(expr $scrubs + $deep_scrubs)) 2024-09-15T21:50:02.199 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5863: TEST_scrub_warning: create_pool psr_pool-10 1 1 2024-09-15T21:50:02.200 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create psr_pool-10 1 1 2024-09-15T21:50:02.570 INFO:tasks.workunit.client.0.smithi136.stderr:pool 'psr_pool-10' created 2024-09-15T21:50:02.582 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2024-09-15T21:50:03.583 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5864: TEST_scrub_warning: wait_for_clean 2024-09-15T21:50:03.583 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-15T21:50:03.584 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-15T21:50:03.584 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-15T21:50:03.585 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-15T21:50:03.585 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-15T21:50:03.585 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-15T21:50:03.585 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-15T21:50:03.585 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-15T21:50:03.585 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-15T21:50:03.690 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-15T21:50:03.690 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-15T21:50:03.690 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-15T21:50:03.690 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-15T21:50:03.690 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-15T21:50:03.690 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-15T21:50:03.967 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids=0 2024-09-15T21:50:03.968 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-15T21:50:03.968 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T21:50:03.968 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-15T21:50:04.112 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836505 2024-09-15T21:50:04.112 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836505 2024-09-15T21:50:04.112 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836505' 2024-09-15T21:50:04.112 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T21:50:04.112 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836505 2024-09-15T21:50:04.113 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T21:50:04.113 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-15T21:50:04.114 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836505 2024-09-15T21:50:04.114 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T21:50:04.115 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836505 2024-09-15T21:50:04.115 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836505' 2024-09-15T21:50:04.115 INFO:tasks.workunit.client.0.smithi136.stdout:waiting osd.0 seq 21474836505 2024-09-15T21:50:04.116 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T21:50:04.393 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836504 -lt 21474836505 2024-09-15T21:50:04.393 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-15T21:50:05.394 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-15T21:50:05.394 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T21:50:05.678 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836504 -lt 21474836505 2024-09-15T21:50:05.679 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-15T21:50:06.680 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 299 -eq 0 ']' 2024-09-15T21:50:06.680 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T21:50:06.969 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836506 -lt 21474836505 2024-09-15T21:50:06.969 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-15T21:50:06.970 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T21:50:06.970 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T21:50:07.314 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 10 == 0 2024-09-15T21:50:07.314 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-15T21:50:07.315 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-15T21:50:07.315 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-15T21:50:07.315 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T21:50:07.315 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T21:50:07.315 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T21:50:07.316 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T21:50:07.589 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=10 2024-09-15T21:50:07.590 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-15T21:50:07.590 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T21:50:07.590 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T21:50:07.941 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 10 = 10 2024-09-15T21:50:07.941 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-15T21:50:07.941 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-15T21:50:07.941 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5865: TEST_scrub_warning: '[' 10 = 1 ']' 2024-09-15T21:50:07.941 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5869: TEST_scrub_warning: expr 5 + 1 2024-09-15T21:50:07.942 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5869: TEST_scrub_warning: '[' 10 = 6 ']' 2024-09-15T21:50:07.942 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5876: TEST_scrub_warning: local primary=0 2024-09-15T21:50:07.942 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5878: TEST_scrub_warning: ceph osd set noscrub 2024-09-15T21:50:08.312 INFO:tasks.workunit.client.0.smithi136.stderr:noscrub is set 2024-09-15T21:50:08.324 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5879: TEST_scrub_warning: ceph osd set nodeep-scrub 2024-09-15T21:50:08.823 INFO:tasks.workunit.client.0.smithi136.stderr:nodeep-scrub is set 2024-09-15T21:50:08.835 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5880: TEST_scrub_warning: ceph config set global osd_scrub_interval_randomize_ratio 0 2024-09-15T21:50:09.196 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5881: TEST_scrub_warning: ceph config set global osd_deep_scrub_randomize_ratio 0 2024-09-15T21:50:09.530 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5882: TEST_scrub_warning: ceph config set global osd_scrub_max_interval 604800 2024-09-15T21:50:09.865 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5883: TEST_scrub_warning: ceph config set global osd_deep_scrub_interval 604800 2024-09-15T21:50:10.199 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5886: TEST_scrub_warning: seq 1 5 2024-09-15T21:50:10.200 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5886: TEST_scrub_warning: for i in $(seq 1 $scrubs) 2024-09-15T21:50:10.200 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5888: TEST_scrub_warning: '[' 1 = 1 ']' 2024-09-15T21:50:10.200 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5890: TEST_scrub_warning: overdue_seconds=1900800 2024-09-15T21:50:10.200 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5894: TEST_scrub_warning: expr 1900800 + 100 2024-09-15T21:50:10.201 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5894: TEST_scrub_warning: ceph tell 1.0 schedule-scrub 1900900 2024-09-15T21:50:10.300 INFO:tasks.workunit.client.0.smithi136.stdout:{ 2024-09-15T21:50:10.300 INFO:tasks.workunit.client.0.smithi136.stdout: "deep": false, 2024-09-15T21:50:10.300 INFO:tasks.workunit.client.0.smithi136.stdout: "must": false, 2024-09-15T21:50:10.300 INFO:tasks.workunit.client.0.smithi136.stdout: "stamp": "2024-08-24T21:46:50.298961+0000" 2024-09-15T21:50:10.300 INFO:tasks.workunit.client.0.smithi136.stdout:} 2024-09-15T21:50:10.309 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5886: TEST_scrub_warning: for i in $(seq 1 $scrubs) 2024-09-15T21:50:10.309 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5888: TEST_scrub_warning: '[' 2 = 1 ']' 2024-09-15T21:50:10.309 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5892: TEST_scrub_warning: overdue_seconds=993600 2024-09-15T21:50:10.310 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5894: TEST_scrub_warning: expr 993600 + 200 2024-09-15T21:50:10.310 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5894: TEST_scrub_warning: ceph tell 2.0 schedule-scrub 993800 2024-09-15T21:50:10.409 INFO:tasks.workunit.client.0.smithi136.stdout:{ 2024-09-15T21:50:10.409 INFO:tasks.workunit.client.0.smithi136.stdout: "deep": false, 2024-09-15T21:50:10.409 INFO:tasks.workunit.client.0.smithi136.stdout: "must": false, 2024-09-15T21:50:10.409 INFO:tasks.workunit.client.0.smithi136.stdout: "stamp": "2024-09-04T09:45:10.407805+0000" 2024-09-15T21:50:10.409 INFO:tasks.workunit.client.0.smithi136.stdout:} 2024-09-15T21:50:10.418 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5886: TEST_scrub_warning: for i in $(seq 1 $scrubs) 2024-09-15T21:50:10.418 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5888: TEST_scrub_warning: '[' 3 = 1 ']' 2024-09-15T21:50:10.418 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5892: TEST_scrub_warning: overdue_seconds=993600 2024-09-15T21:50:10.418 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5894: TEST_scrub_warning: expr 993600 + 300 2024-09-15T21:50:10.419 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5894: TEST_scrub_warning: ceph tell 3.0 schedule-scrub 993900 2024-09-15T21:50:10.518 INFO:tasks.workunit.client.0.smithi136.stdout:{ 2024-09-15T21:50:10.518 INFO:tasks.workunit.client.0.smithi136.stdout: "deep": false, 2024-09-15T21:50:10.518 INFO:tasks.workunit.client.0.smithi136.stdout: "must": false, 2024-09-15T21:50:10.518 INFO:tasks.workunit.client.0.smithi136.stdout: "stamp": "2024-09-04T09:43:30.517517+0000" 2024-09-15T21:50:10.518 INFO:tasks.workunit.client.0.smithi136.stdout:} 2024-09-15T21:50:10.528 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5886: TEST_scrub_warning: for i in $(seq 1 $scrubs) 2024-09-15T21:50:10.528 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5888: TEST_scrub_warning: '[' 4 = 1 ']' 2024-09-15T21:50:10.528 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5892: TEST_scrub_warning: overdue_seconds=993600 2024-09-15T21:50:10.528 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5894: TEST_scrub_warning: expr 993600 + 400 2024-09-15T21:50:10.528 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5894: TEST_scrub_warning: ceph tell 4.0 schedule-scrub 994000 2024-09-15T21:50:10.626 INFO:tasks.workunit.client.0.smithi136.stdout:{ 2024-09-15T21:50:10.626 INFO:tasks.workunit.client.0.smithi136.stdout: "deep": false, 2024-09-15T21:50:10.626 INFO:tasks.workunit.client.0.smithi136.stdout: "must": false, 2024-09-15T21:50:10.626 INFO:tasks.workunit.client.0.smithi136.stdout: "stamp": "2024-09-04T09:41:50.625229+0000" 2024-09-15T21:50:10.626 INFO:tasks.workunit.client.0.smithi136.stdout:} 2024-09-15T21:50:10.635 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5886: TEST_scrub_warning: for i in $(seq 1 $scrubs) 2024-09-15T21:50:10.635 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5888: TEST_scrub_warning: '[' 5 = 1 ']' 2024-09-15T21:50:10.635 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5892: TEST_scrub_warning: overdue_seconds=993600 2024-09-15T21:50:10.636 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5894: TEST_scrub_warning: expr 993600 + 500 2024-09-15T21:50:10.637 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5894: TEST_scrub_warning: ceph tell 5.0 schedule-scrub 994100 2024-09-15T21:50:10.734 INFO:tasks.workunit.client.0.smithi136.stdout:{ 2024-09-15T21:50:10.734 INFO:tasks.workunit.client.0.smithi136.stdout: "deep": false, 2024-09-15T21:50:10.734 INFO:tasks.workunit.client.0.smithi136.stdout: "must": false, 2024-09-15T21:50:10.734 INFO:tasks.workunit.client.0.smithi136.stdout: "stamp": "2024-09-04T09:40:10.733024+0000" 2024-09-15T21:50:10.734 INFO:tasks.workunit.client.0.smithi136.stdout:} 2024-09-15T21:50:10.744 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5897: TEST_scrub_warning: expr 5 + 1 2024-09-15T21:50:10.745 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5897: TEST_scrub_warning: expr 5 + 5 2024-09-15T21:50:10.746 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5897: TEST_scrub_warning: seq 6 10 2024-09-15T21:50:10.747 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5897: TEST_scrub_warning: for i in $(seq $(expr $scrubs + 1) $(expr $scrubs + $deep_scrubs)) 2024-09-15T21:50:10.748 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5899: TEST_scrub_warning: expr 5 + 1 2024-09-15T21:50:10.748 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5899: TEST_scrub_warning: '[' 6 = 6 ']' 2024-09-15T21:50:10.748 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5901: TEST_scrub_warning: overdue_seconds=1900800 2024-09-15T21:50:10.749 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5905: TEST_scrub_warning: expr 1900800 + 600 2024-09-15T21:50:10.750 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5905: TEST_scrub_warning: ceph tell 6.0 schedule-deep-scrub 1901400 2024-09-15T21:50:10.846 INFO:tasks.workunit.client.0.smithi136.stdout:{ 2024-09-15T21:50:10.846 INFO:tasks.workunit.client.0.smithi136.stdout: "deep": true, 2024-09-15T21:50:10.846 INFO:tasks.workunit.client.0.smithi136.stdout: "must": false, 2024-09-15T21:50:10.847 INFO:tasks.workunit.client.0.smithi136.stdout: "stamp": "2024-08-24T21:38:30.845704+0000" 2024-09-15T21:50:10.847 INFO:tasks.workunit.client.0.smithi136.stdout:} 2024-09-15T21:50:10.855 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5897: TEST_scrub_warning: for i in $(seq $(expr $scrubs + 1) $(expr $scrubs + $deep_scrubs)) 2024-09-15T21:50:10.856 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5899: TEST_scrub_warning: expr 5 + 1 2024-09-15T21:50:10.857 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5899: TEST_scrub_warning: '[' 7 = 6 ']' 2024-09-15T21:50:10.857 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5903: TEST_scrub_warning: overdue_seconds=993600 2024-09-15T21:50:10.857 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5905: TEST_scrub_warning: expr 993600 + 700 2024-09-15T21:50:10.858 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5905: TEST_scrub_warning: ceph tell 7.0 schedule-deep-scrub 994300 2024-09-15T21:50:10.956 INFO:tasks.workunit.client.0.smithi136.stdout:{ 2024-09-15T21:50:10.956 INFO:tasks.workunit.client.0.smithi136.stdout: "deep": true, 2024-09-15T21:50:10.956 INFO:tasks.workunit.client.0.smithi136.stdout: "must": false, 2024-09-15T21:50:10.956 INFO:tasks.workunit.client.0.smithi136.stdout: "stamp": "2024-09-04T09:36:50.955398+0000" 2024-09-15T21:50:10.956 INFO:tasks.workunit.client.0.smithi136.stdout:} 2024-09-15T21:50:10.965 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5897: TEST_scrub_warning: for i in $(seq $(expr $scrubs + 1) $(expr $scrubs + $deep_scrubs)) 2024-09-15T21:50:10.965 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5899: TEST_scrub_warning: expr 5 + 1 2024-09-15T21:50:10.966 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5899: TEST_scrub_warning: '[' 8 = 6 ']' 2024-09-15T21:50:10.966 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5903: TEST_scrub_warning: overdue_seconds=993600 2024-09-15T21:50:10.967 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5905: TEST_scrub_warning: expr 993600 + 800 2024-09-15T21:50:10.968 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5905: TEST_scrub_warning: ceph tell 8.0 schedule-deep-scrub 994400 2024-09-15T21:50:11.066 INFO:tasks.workunit.client.0.smithi136.stdout:{ 2024-09-15T21:50:11.066 INFO:tasks.workunit.client.0.smithi136.stdout: "deep": true, 2024-09-15T21:50:11.066 INFO:tasks.workunit.client.0.smithi136.stdout: "must": false, 2024-09-15T21:50:11.066 INFO:tasks.workunit.client.0.smithi136.stdout: "stamp": "2024-09-04T09:35:11.065088+0000" 2024-09-15T21:50:11.066 INFO:tasks.workunit.client.0.smithi136.stdout:} 2024-09-15T21:50:11.075 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5897: TEST_scrub_warning: for i in $(seq $(expr $scrubs + 1) $(expr $scrubs + $deep_scrubs)) 2024-09-15T21:50:11.075 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5899: TEST_scrub_warning: expr 5 + 1 2024-09-15T21:50:11.076 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5899: TEST_scrub_warning: '[' 9 = 6 ']' 2024-09-15T21:50:11.076 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5903: TEST_scrub_warning: overdue_seconds=993600 2024-09-15T21:50:11.077 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5905: TEST_scrub_warning: expr 993600 + 900 2024-09-15T21:50:11.078 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5905: TEST_scrub_warning: ceph tell 9.0 schedule-deep-scrub 994500 2024-09-15T21:50:11.176 INFO:tasks.workunit.client.0.smithi136.stdout:{ 2024-09-15T21:50:11.176 INFO:tasks.workunit.client.0.smithi136.stdout: "deep": true, 2024-09-15T21:50:11.177 INFO:tasks.workunit.client.0.smithi136.stdout: "must": false, 2024-09-15T21:50:11.177 INFO:tasks.workunit.client.0.smithi136.stdout: "stamp": "2024-09-04T09:33:31.175711+0000" 2024-09-15T21:50:11.177 INFO:tasks.workunit.client.0.smithi136.stdout:} 2024-09-15T21:50:11.185 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5897: TEST_scrub_warning: for i in $(seq $(expr $scrubs + 1) $(expr $scrubs + $deep_scrubs)) 2024-09-15T21:50:11.186 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5899: TEST_scrub_warning: expr 5 + 1 2024-09-15T21:50:11.187 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5899: TEST_scrub_warning: '[' 10 = 6 ']' 2024-09-15T21:50:11.187 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5903: TEST_scrub_warning: overdue_seconds=993600 2024-09-15T21:50:11.187 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5905: TEST_scrub_warning: expr 993600 + 1000 2024-09-15T21:50:11.188 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5905: TEST_scrub_warning: ceph tell 10.0 schedule-deep-scrub 994600 2024-09-15T21:50:11.286 INFO:tasks.workunit.client.0.smithi136.stdout:{ 2024-09-15T21:50:11.286 INFO:tasks.workunit.client.0.smithi136.stdout: "deep": true, 2024-09-15T21:50:11.286 INFO:tasks.workunit.client.0.smithi136.stdout: "must": false, 2024-09-15T21:50:11.286 INFO:tasks.workunit.client.0.smithi136.stdout: "stamp": "2024-09-04T09:31:51.285093+0000" 2024-09-15T21:50:11.286 INFO:tasks.workunit.client.0.smithi136.stdout:} 2024-09-15T21:50:11.297 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5907: TEST_scrub_warning: flush_pg_stats 2024-09-15T21:50:11.297 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-15T21:50:11.297 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-15T21:50:11.568 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids=0 2024-09-15T21:50:11.568 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-15T21:50:11.568 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T21:50:11.568 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-15T21:50:11.671 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836508 2024-09-15T21:50:11.671 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836508 2024-09-15T21:50:11.671 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836508' 2024-09-15T21:50:11.671 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T21:50:11.672 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836508 2024-09-15T21:50:11.672 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T21:50:11.672 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-15T21:50:11.673 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836508 2024-09-15T21:50:11.673 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T21:50:11.674 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836508 2024-09-15T21:50:11.675 INFO:tasks.workunit.client.0.smithi136.stdout:waiting osd.0 seq 21474836508 2024-09-15T21:50:11.675 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836508' 2024-09-15T21:50:11.675 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T21:50:11.952 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836507 -lt 21474836508 2024-09-15T21:50:11.953 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-15T21:50:12.954 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-15T21:50:12.954 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T21:50:13.241 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836508 -lt 21474836508 2024-09-15T21:50:13.241 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5909: TEST_scrub_warning: ceph health 2024-09-15T21:50:13.544 INFO:tasks.workunit.client.0.smithi136.stdout:HEALTH_WARN noscrub,nodeep-scrub flag(s) set; 5 pgs not deep-scrubbed in time; 10 pgs not scrubbed in time; 10 pool(s) do not have an application enabled; 10 pool(s) have no replicas configured; OSD count 1 < osd_pool_default_size 3 2024-09-15T21:50:13.555 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5910: TEST_scrub_warning: ceph health detail 2024-09-15T21:50:13.889 INFO:tasks.workunit.client.0.smithi136.stdout:HEALTH_WARN noscrub,nodeep-scrub flag(s) set; 5 pgs not deep-scrubbed in time; 10 pgs not scrubbed in time; 10 pool(s) do not have an application enabled; 10 pool(s) have no replicas configured; OSD count 1 < osd_pool_default_size 3 2024-09-15T21:50:13.889 INFO:tasks.workunit.client.0.smithi136.stdout:[WRN] OSDMAP_FLAGS: noscrub,nodeep-scrub flag(s) set 2024-09-15T21:50:13.889 INFO:tasks.workunit.client.0.smithi136.stdout:[WRN] PG_NOT_DEEP_SCRUBBED: 5 pgs not deep-scrubbed in time 2024-09-15T21:50:13.889 INFO:tasks.workunit.client.0.smithi136.stdout: pg 10.0 not deep-scrubbed since 2024-09-04T09:31:51.285093+0000 2024-09-15T21:50:13.890 INFO:tasks.workunit.client.0.smithi136.stdout: pg 9.0 not deep-scrubbed since 2024-09-04T09:33:31.175711+0000 2024-09-15T21:50:13.890 INFO:tasks.workunit.client.0.smithi136.stdout: pg 8.0 not deep-scrubbed since 2024-09-04T09:35:11.065088+0000 2024-09-15T21:50:13.890 INFO:tasks.workunit.client.0.smithi136.stdout: pg 7.0 not deep-scrubbed since 2024-09-04T09:36:50.955398+0000 2024-09-15T21:50:13.890 INFO:tasks.workunit.client.0.smithi136.stdout: pg 6.0 not deep-scrubbed since 2024-08-24T21:38:30.845704+0000 2024-09-15T21:50:13.890 INFO:tasks.workunit.client.0.smithi136.stdout:[WRN] PG_NOT_SCRUBBED: 10 pgs not scrubbed in time 2024-09-15T21:50:13.890 INFO:tasks.workunit.client.0.smithi136.stdout: pg 10.0 not scrubbed since 2024-09-04T09:31:51.285093+0000 2024-09-15T21:50:13.890 INFO:tasks.workunit.client.0.smithi136.stdout: pg 9.0 not scrubbed since 2024-09-04T09:33:31.175711+0000 2024-09-15T21:50:13.890 INFO:tasks.workunit.client.0.smithi136.stdout: pg 8.0 not scrubbed since 2024-09-04T09:35:11.065088+0000 2024-09-15T21:50:13.890 INFO:tasks.workunit.client.0.smithi136.stdout: pg 7.0 not scrubbed since 2024-09-04T09:36:50.955398+0000 2024-09-15T21:50:13.890 INFO:tasks.workunit.client.0.smithi136.stdout: pg 6.0 not scrubbed since 2024-08-24T21:38:30.845704+0000 2024-09-15T21:50:13.890 INFO:tasks.workunit.client.0.smithi136.stdout: pg 5.0 not scrubbed since 2024-09-04T09:40:10.733024+0000 2024-09-15T21:50:13.890 INFO:tasks.workunit.client.0.smithi136.stdout: pg 4.0 not scrubbed since 2024-09-04T09:41:50.625229+0000 2024-09-15T21:50:13.890 INFO:tasks.workunit.client.0.smithi136.stdout: pg 3.0 not scrubbed since 2024-09-04T09:43:30.517517+0000 2024-09-15T21:50:13.891 INFO:tasks.workunit.client.0.smithi136.stdout: pg 2.0 not scrubbed since 2024-09-04T09:45:10.407805+0000 2024-09-15T21:50:13.891 INFO:tasks.workunit.client.0.smithi136.stdout: pg 1.0 not scrubbed since 2024-08-24T21:46:50.298961+0000 2024-09-15T21:50:13.891 INFO:tasks.workunit.client.0.smithi136.stdout:[WRN] POOL_APP_NOT_ENABLED: 10 pool(s) do not have an application enabled 2024-09-15T21:50:13.891 INFO:tasks.workunit.client.0.smithi136.stdout: application not enabled on pool 'psr_pool-1' 2024-09-15T21:50:13.891 INFO:tasks.workunit.client.0.smithi136.stdout: application not enabled on pool 'psr_pool-2' 2024-09-15T21:50:13.891 INFO:tasks.workunit.client.0.smithi136.stdout: application not enabled on pool 'psr_pool-3' 2024-09-15T21:50:13.891 INFO:tasks.workunit.client.0.smithi136.stdout: application not enabled on pool 'psr_pool-4' 2024-09-15T21:50:13.891 INFO:tasks.workunit.client.0.smithi136.stdout: application not enabled on pool 'psr_pool-5' 2024-09-15T21:50:13.891 INFO:tasks.workunit.client.0.smithi136.stdout: application not enabled on pool 'psr_pool-6' 2024-09-15T21:50:13.891 INFO:tasks.workunit.client.0.smithi136.stdout: application not enabled on pool 'psr_pool-7' 2024-09-15T21:50:13.891 INFO:tasks.workunit.client.0.smithi136.stdout: application not enabled on pool 'psr_pool-8' 2024-09-15T21:50:13.891 INFO:tasks.workunit.client.0.smithi136.stdout: application not enabled on pool 'psr_pool-9' 2024-09-15T21:50:13.891 INFO:tasks.workunit.client.0.smithi136.stdout: application not enabled on pool 'psr_pool-10' 2024-09-15T21:50:13.891 INFO:tasks.workunit.client.0.smithi136.stdout: use 'ceph osd pool application enable ', where is 'cephfs', 'rbd', 'rgw', or freeform for custom applications. 2024-09-15T21:50:13.892 INFO:tasks.workunit.client.0.smithi136.stdout:[WRN] POOL_NO_REDUNDANCY: 10 pool(s) have no replicas configured 2024-09-15T21:50:13.892 INFO:tasks.workunit.client.0.smithi136.stdout: pool 'psr_pool-1' has no replicas configured 2024-09-15T21:50:13.892 INFO:tasks.workunit.client.0.smithi136.stdout: pool 'psr_pool-2' has no replicas configured 2024-09-15T21:50:13.892 INFO:tasks.workunit.client.0.smithi136.stdout: pool 'psr_pool-3' has no replicas configured 2024-09-15T21:50:13.892 INFO:tasks.workunit.client.0.smithi136.stdout: pool 'psr_pool-4' has no replicas configured 2024-09-15T21:50:13.892 INFO:tasks.workunit.client.0.smithi136.stdout: pool 'psr_pool-5' has no replicas configured 2024-09-15T21:50:13.892 INFO:tasks.workunit.client.0.smithi136.stdout: pool 'psr_pool-6' has no replicas configured 2024-09-15T21:50:13.892 INFO:tasks.workunit.client.0.smithi136.stdout: pool 'psr_pool-7' has no replicas configured 2024-09-15T21:50:13.892 INFO:tasks.workunit.client.0.smithi136.stdout: pool 'psr_pool-8' has no replicas configured 2024-09-15T21:50:13.892 INFO:tasks.workunit.client.0.smithi136.stdout: pool 'psr_pool-9' has no replicas configured 2024-09-15T21:50:13.893 INFO:tasks.workunit.client.0.smithi136.stdout: pool 'psr_pool-10' has no replicas configured 2024-09-15T21:50:13.893 INFO:tasks.workunit.client.0.smithi136.stdout:[WRN] TOO_FEW_OSDS: OSD count 1 < osd_pool_default_size 3 2024-09-15T21:50:13.901 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5911: TEST_scrub_warning: ceph health 2024-09-15T21:50:13.901 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5911: TEST_scrub_warning: grep -q ' pgs not deep-scrubbed in time' 2024-09-15T21:50:14.219 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5912: TEST_scrub_warning: ceph health 2024-09-15T21:50:14.219 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5912: TEST_scrub_warning: grep -q ' pgs not scrubbed in time' 2024-09-15T21:50:14.536 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5917: TEST_scrub_warning: ceph health detail 2024-09-15T21:50:14.536 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5917: TEST_scrub_warning: grep 'not scrubbed since' 2024-09-15T21:50:14.536 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5917: TEST_scrub_warning: wc -l 2024-09-15T21:50:14.882 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5917: TEST_scrub_warning: COUNT=10 2024-09-15T21:50:14.883 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5919: TEST_scrub_warning: expr 5+5 2024-09-15T21:50:14.884 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5919: TEST_scrub_warning: (( 10 != 5 && 10 != 5+5 )) 2024-09-15T21:50:14.885 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5923: TEST_scrub_warning: ceph health detail 2024-09-15T21:50:14.885 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5923: TEST_scrub_warning: grep 'not deep-scrubbed since' 2024-09-15T21:50:14.885 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5923: TEST_scrub_warning: wc -l 2024-09-15T21:50:15.248 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5923: TEST_scrub_warning: COUNT=5 2024-09-15T21:50:15.248 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5924: TEST_scrub_warning: '[' 5 '!=' 5 ']' 2024-09-15T21:50:15.248 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:66: run: teardown td/osd-scrub-repair 2024-09-15T21:50:15.248 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/osd-scrub-repair 2024-09-15T21:50:15.248 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-15T21:50:15.248 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/osd-scrub-repair KILL 2024-09-15T21:50:15.249 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-15T21:50:15.249 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-15T21:50:15.249 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-15T21:50:15.249 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-15T21:50:15.249 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-15T21:50:15.355 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-15T21:50:15.356 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-15T21:50:15.357 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-15T21:50:15.357 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-15T21:50:15.358 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-15T21:50:15.358 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-15T21:50:15.358 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-15T21:50:15.360 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T21:50:15.360 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-15T21:50:15.360 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-15T21:50:15.360 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T21:50:15.361 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-15T21:50:15.362 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-15T21:50:15.362 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/osd-scrub-repair 2024-09-15T21:50:15.372 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-15T21:50:15.372 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T21:50:15.372 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52602 2024-09-15T21:50:15.372 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.52602 2024-09-15T21:50:15.374 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-15T21:50:15.374 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-15T21:50:15.374 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:63: run: for func in $funcs 2024-09-15T21:50:15.374 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:64: run: setup td/osd-scrub-repair 2024-09-15T21:50:15.374 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/osd-scrub-repair 2024-09-15T21:50:15.374 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/osd-scrub-repair 2024-09-15T21:50:15.374 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/osd-scrub-repair 2024-09-15T21:50:15.374 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-15T21:50:15.374 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/osd-scrub-repair KILL 2024-09-15T21:50:15.374 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-15T21:50:15.374 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-15T21:50:15.375 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-15T21:50:15.375 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-15T21:50:15.375 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-15T21:50:15.376 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-15T21:50:15.376 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-15T21:50:15.377 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-15T21:50:15.378 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-15T21:50:15.378 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-15T21:50:15.378 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-15T21:50:15.379 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-15T21:50:15.380 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T21:50:15.380 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-15T21:50:15.380 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-15T21:50:15.380 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T21:50:15.381 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-15T21:50:15.382 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-15T21:50:15.382 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/osd-scrub-repair 2024-09-15T21:50:15.383 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-15T21:50:15.384 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T21:50:15.384 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52602 2024-09-15T21:50:15.384 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.52602 2024-09-15T21:50:15.384 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-15T21:50:15.384 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-15T21:50:15.385 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/osd-scrub-repair 2024-09-15T21:50:15.386 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2024-09-15T21:50:15.386 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T21:50:15.386 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52602 2024-09-15T21:50:15.386 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.52602 2024-09-15T21:50:15.389 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2024-09-15T21:50:15.389 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 4096 -le 1024 ']' 2024-09-15T21:50:15.389 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2024-09-15T21:50:15.389 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/osd-scrub-repair 1' TERM HUP INT 2024-09-15T21:50:15.389 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:65: run: TEST_skip_non_repair_during_recovery td/osd-scrub-repair 2024-09-15T21:50:15.389 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:136: TEST_skip_non_repair_during_recovery: local dir=td/osd-scrub-repair 2024-09-15T21:50:15.389 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:137: TEST_skip_non_repair_during_recovery: local poolname=rbd 2024-09-15T21:50:15.389 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:139: TEST_skip_non_repair_during_recovery: run_mon td/osd-scrub-repair a --osd_pool_default_size=2 2024-09-15T21:50:15.389 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/osd-scrub-repair 2024-09-15T21:50:15.390 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2024-09-15T21:50:15.390 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2024-09-15T21:50:15.390 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2024-09-15T21:50:15.390 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/osd-scrub-repair/a 2024-09-15T21:50:15.390 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-scrub-repair/a --run-dir=td/osd-scrub-repair --osd_pool_default_size=2 2024-09-15T21:50:15.847 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2024-09-15T21:50:15.847 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-15T21:50:15.848 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-15T21:50:15.848 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-15T21:50:15.848 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T21:50:15.848 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52602 2024-09-15T21:50:15.848 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.52602/$cluster-$name.asok' 2024-09-15T21:50:15.848 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-scrub-repair/a '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.52602/$cluster-$name.asok' --mon-cluster-log-file=td/osd-scrub-repair/log --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=2 2024-09-15T21:50:16.056 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2024-09-15T21:50:16.057 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2024-09-15T21:50:16.057 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-15T21:50:16.057 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-15T21:50:16.057 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2024-09-15T21:50:16.057 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2024-09-15T21:50:16.058 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-15T21:50:16.058 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-15T21:50:16.058 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-15T21:50:16.059 INFO:tasks.workunit.client.0.smithi136.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-15T21:50:16.059 INFO:tasks.workunit.client.0.smithi136.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T21:50:16.059 INFO:tasks.workunit.client.0.smithi136.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52602 2024-09-15T21:50:16.059 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.52602/ceph-mon.a.asok 2024-09-15T21:50:16.059 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-15T21:50:16.059 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.52602/ceph-mon.a.asok config get fsid 2024-09-15T21:50:16.140 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2024-09-15T21:50:16.140 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-15T21:50:16.140 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-15T21:50:16.140 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2024-09-15T21:50:16.140 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2024-09-15T21:50:16.141 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-15T21:50:16.141 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-15T21:50:16.141 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-15T21:50:16.142 INFO:tasks.workunit.client.0.smithi136.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-15T21:50:16.142 INFO:tasks.workunit.client.0.smithi136.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T21:50:16.142 INFO:tasks.workunit.client.0.smithi136.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52602 2024-09-15T21:50:16.142 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.52602/ceph-mon.a.asok 2024-09-15T21:50:16.142 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-15T21:50:16.142 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.52602/ceph-mon.a.asok config get mon_host 2024-09-15T21:50:16.220 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:140: TEST_skip_non_repair_during_recovery: run_mgr td/osd-scrub-repair x 2024-09-15T21:50:16.220 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: run_mgr: local dir=td/osd-scrub-repair 2024-09-15T21:50:16.220 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: run_mgr: shift 2024-09-15T21:50:16.220 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: run_mgr: local id=x 2024-09-15T21:50:16.220 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: run_mgr: shift 2024-09-15T21:50:16.220 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:548: run_mgr: local data=td/osd-scrub-repair/x 2024-09-15T21:50:16.220 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:550: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-15T21:50:17.033 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: get_asok_path 2024-09-15T21:50:17.033 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-15T21:50:17.033 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-15T21:50:17.033 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-15T21:50:17.033 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T21:50:17.033 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52602 2024-09-15T21:50:17.034 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.52602/$cluster-$name.asok' 2024-09-15T21:50:17.034 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-15T21:50:17.035 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-scrub-repair/x '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.52602/$cluster-$name.asok' --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-15T21:50:17.065 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:141: TEST_skip_non_repair_during_recovery: run_osd td/osd-scrub-repair 0 --osd_scrub_during_recovery=false --osd_repair_during_recovery=true --osd_debug_pretend_recovery_active=true 2024-09-15T21:50:17.065 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-scrub-repair 2024-09-15T21:50:17.066 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-15T21:50:17.066 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=0 2024-09-15T21:50:17.066 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-15T21:50:17.066 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-scrub-repair/0 2024-09-15T21:50:17.066 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=ce4892c7-6c6d-4b4f-8e6f-07a3e11133e5 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-15T21:50:17.066 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T21:50:17.066 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T21:50:17.066 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T21:50:17.066 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-09-15T21:50:17.066 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-09-15T21:50:17.066 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-15T21:50:17.066 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-15T21:50:17.066 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-15T21:50:17.066 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-15T21:50:17.067 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-15T21:50:17.067 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-15T21:50:17.067 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-15T21:50:17.067 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T21:50:17.067 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52602 2024-09-15T21:50:17.067 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.52602/$cluster-$name.asok' 2024-09-15T21:50:17.067 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.52602/$cluster-$name.asok' 2024-09-15T21:50:17.067 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-15T21:50:17.067 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-15T21:50:17.067 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-15T21:50:17.067 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-15T21:50:17.067 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-15T21:50:17.067 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T21:50:17.067 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T21:50:17.067 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T21:50:17.068 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T21:50:17.068 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-15T21:50:17.068 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+='--osd_scrub_during_recovery=false --osd_repair_during_recovery=true --osd_debug_pretend_recovery_active=true' 2024-09-15T21:50:17.068 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-scrub-repair/0 2024-09-15T21:50:17.068 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-15T21:50:17.068 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=78e6362a-cc7e-43d3-a140-b4f9ac206c2c 2024-09-15T21:50:17.069 INFO:tasks.workunit.client.0.smithi136.stdout:add osd0 78e6362a-cc7e-43d3-a140-b4f9ac206c2c 2024-09-15T21:50:17.069 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd0 78e6362a-cc7e-43d3-a140-b4f9ac206c2c' 2024-09-15T21:50:17.069 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-15T21:50:17.086 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQCZVudmpboJBRAA+LjFKfPGkVIc0Nus/HCzSg== 2024-09-15T21:50:17.086 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQCZVudmpboJBRAA+LjFKfPGkVIc0Nus/HCzSg=="}' 2024-09-15T21:50:17.086 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 78e6362a-cc7e-43d3-a140-b4f9ac206c2c -i td/osd-scrub-repair/0/new.json 2024-09-15T21:50:17.289 INFO:tasks.workunit.client.0.smithi136.stdout:0 2024-09-15T21:50:17.299 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-scrub-repair/0/new.json 2024-09-15T21:50:17.300 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 0 --fsid=ce4892c7-6c6d-4b4f-8e6f-07a3e11133e5 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.52602/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_scrub_during_recovery=false --osd_repair_during_recovery=true --osd_debug_pretend_recovery_active=true --mkfs --key AQCZVudmpboJBRAA+LjFKfPGkVIc0Nus/HCzSg== --osd-uuid 78e6362a-cc7e-43d3-a140-b4f9ac206c2c 2024-09-15T21:50:17.331 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:50:17.329+0000 7f2758b51640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:50:17.340 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:50:17.338+0000 7f2758b51640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:50:17.341 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:50:17.339+0000 7f2758b51640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:50:17.341 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:50:17.340+0000 7f2758b51640 -1 bdev(0x560bfbc0dc00 td/osd-scrub-repair/0/block) open stat got: (1) Operation not permitted 2024-09-15T21:50:17.341 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:50:17.340+0000 7f2758b51640 -1 bluestore(td/osd-scrub-repair/0) _read_fsid unparsable uuid 2024-09-15T21:50:23.314 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-scrub-repair/0/keyring 2024-09-15T21:50:23.315 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-15T21:50:23.315 INFO:tasks.workunit.client.0.smithi136.stdout:adding osd0 key to auth repository 2024-09-15T21:50:23.316 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd0 key to auth repository 2024-09-15T21:50:23.316 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-scrub-repair/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-15T21:50:23.678 INFO:tasks.workunit.client.0.smithi136.stdout:start osd.0 2024-09-15T21:50:23.679 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.0 2024-09-15T21:50:23.679 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-15T21:50:23.679 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 0 --fsid=ce4892c7-6c6d-4b4f-8e6f-07a3e11133e5 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.52602/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_scrub_during_recovery=false --osd_repair_during_recovery=true --osd_debug_pretend_recovery_active=true 2024-09-15T21:50:23.679 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-15T21:50:23.679 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-15T21:50:23.708 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:50:23.706+0000 7fd8f44d8640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:50:23.710 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:50:23.708+0000 7fd8f44d8640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:50:23.711 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:50:23.709+0000 7fd8f44d8640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:50:23.960 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 0 2024-09-15T21:50:23.960 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-15T21:50:23.960 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2024-09-15T21:50:23.960 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-15T21:50:23.960 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-15T21:50:23.960 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:50:23.961 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-15T21:50:23.961 INFO:tasks.workunit.client.0.smithi136.stdout:0 2024-09-15T21:50:23.961 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:50:23.961 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:50:24.236 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:50:25.238 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:50:25.238 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:50:25.238 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-15T21:50:25.238 INFO:tasks.workunit.client.0.smithi136.stdout:1 2024-09-15T21:50:25.239 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:50:25.239 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:50:25.326 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:50:25.324+0000 7fd8f44d8640 -1 Falling back to public interface 2024-09-15T21:50:25.525 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:50:26.525 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:50:26.525 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:50:26.525 INFO:tasks.workunit.client.0.smithi136.stdout:2 2024-09-15T21:50:26.525 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-15T21:50:26.526 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:50:26.526 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:50:26.810 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:50:27.811 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:50:27.811 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:50:27.812 INFO:tasks.workunit.client.0.smithi136.stdout:3 2024-09-15T21:50:27.812 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-15T21:50:27.812 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:50:27.812 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:50:28.094 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:50:28.287 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:50:28.285+0000 7fd8f44d8640 -1 osd.0 0 log_to_monitors true 2024-09-15T21:50:29.096 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:50:29.096 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:50:29.096 INFO:tasks.workunit.client.0.smithi136.stdout:4 2024-09-15T21:50:29.097 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-15T21:50:29.097 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:50:29.097 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:50:29.491 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:50:30.493 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:50:30.493 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:50:30.493 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-15T21:50:30.493 INFO:tasks.workunit.client.0.smithi136.stdout:5 2024-09-15T21:50:30.494 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:50:30.494 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:50:30.776 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:50:31.778 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:50:31.778 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:50:31.778 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-15T21:50:31.778 INFO:tasks.workunit.client.0.smithi136.stdout:6 2024-09-15T21:50:31.779 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:50:31.779 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:50:32.124 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:50:33.126 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:50:33.126 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:50:33.126 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-15T21:50:33.126 INFO:tasks.workunit.client.0.smithi136.stdout:7 2024-09-15T21:50:33.127 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:50:33.127 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:50:33.411 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:50:34.414 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:50:34.414 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:50:34.414 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-15T21:50:34.415 INFO:tasks.workunit.client.0.smithi136.stdout:8 2024-09-15T21:50:34.415 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:50:34.415 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:50:34.704 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:50:35.705 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:50:35.706 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:50:35.706 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-15T21:50:35.706 INFO:tasks.workunit.client.0.smithi136.stdout:9 2024-09-15T21:50:35.706 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:50:35.707 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:50:35.992 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:50:36.994 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:50:36.994 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:50:36.994 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-15T21:50:36.994 INFO:tasks.workunit.client.0.smithi136.stdout:10 2024-09-15T21:50:36.995 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:50:36.995 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:50:37.312 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:50:38.314 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:50:38.314 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:50:38.314 INFO:tasks.workunit.client.0.smithi136.stdout:11 2024-09-15T21:50:38.315 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-15T21:50:38.315 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:50:38.315 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:50:38.596 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:50:39.598 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:50:39.598 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:50:39.598 INFO:tasks.workunit.client.0.smithi136.stdout:12 2024-09-15T21:50:39.599 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-15T21:50:39.599 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:50:39.599 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:50:40.105 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:50:41.107 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:50:41.107 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:50:41.107 INFO:tasks.workunit.client.0.smithi136.stdout:13 2024-09-15T21:50:41.108 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-15T21:50:41.108 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:50:41.108 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:50:41.858 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:50:42.859 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:50:42.859 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:50:42.859 INFO:tasks.workunit.client.0.smithi136.stdout:14 2024-09-15T21:50:42.860 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-15T21:50:42.860 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:50:42.860 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:50:43.268 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:50:44.269 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:50:44.269 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:50:44.270 INFO:tasks.workunit.client.0.smithi136.stdout:15 2024-09-15T21:50:44.270 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-15T21:50:44.270 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:50:44.270 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:50:44.826 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:50:45.827 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:50:45.827 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:50:45.827 INFO:tasks.workunit.client.0.smithi136.stdout:16 2024-09-15T21:50:45.828 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-15T21:50:45.828 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:50:45.828 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:50:46.408 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:50:47.409 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:50:47.409 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:50:47.410 INFO:tasks.workunit.client.0.smithi136.stdout:17 2024-09-15T21:50:47.410 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-15T21:50:47.410 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:50:47.410 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:50:48.018 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:50:49.019 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:50:49.019 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:50:49.019 INFO:tasks.workunit.client.0.smithi136.stdout:18 2024-09-15T21:50:49.020 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 18 2024-09-15T21:50:49.020 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:50:49.020 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:50:49.555 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:50:50.556 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:50:50.557 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:50:50.557 INFO:tasks.workunit.client.0.smithi136.stdout:19 2024-09-15T21:50:50.557 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 19 2024-09-15T21:50:50.558 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:50:50.558 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:50:50.845 INFO:tasks.workunit.client.0.smithi136.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/3148993520,v1:127.0.0.1:6803/3148993520] [v2:127.0.0.1:6804/3148993520,v1:127.0.0.1:6805/3148993520] exists,up 78e6362a-cc7e-43d3-a140-b4f9ac206c2c 2024-09-15T21:50:50.849 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-15T21:50:50.849 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-15T21:50:50.849 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-15T21:50:50.849 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:144: TEST_skip_non_repair_during_recovery: run_osd td/osd-scrub-repair 1 --osd_scrub_during_recovery=false --osd_repair_during_recovery=true --osd_debug_pretend_recovery_active=true 2024-09-15T21:50:50.849 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-scrub-repair 2024-09-15T21:50:50.849 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-15T21:50:50.849 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=1 2024-09-15T21:50:50.849 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-15T21:50:50.849 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-scrub-repair/1 2024-09-15T21:50:50.850 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=ce4892c7-6c6d-4b4f-8e6f-07a3e11133e5 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-15T21:50:50.850 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T21:50:50.850 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T21:50:50.850 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T21:50:50.850 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-09-15T21:50:50.850 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-09-15T21:50:50.850 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-15T21:50:50.850 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-15T21:50:50.850 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-15T21:50:50.850 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-15T21:50:50.850 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-15T21:50:50.850 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-15T21:50:50.850 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-15T21:50:50.850 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T21:50:50.851 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52602 2024-09-15T21:50:50.851 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.52602/$cluster-$name.asok' 2024-09-15T21:50:50.851 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.52602/$cluster-$name.asok' 2024-09-15T21:50:50.851 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-15T21:50:50.851 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-15T21:50:50.851 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-15T21:50:50.851 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-15T21:50:50.851 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-15T21:50:50.851 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T21:50:50.851 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T21:50:50.851 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T21:50:50.851 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T21:50:50.851 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-15T21:50:50.851 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+='--osd_scrub_during_recovery=false --osd_repair_during_recovery=true --osd_debug_pretend_recovery_active=true' 2024-09-15T21:50:50.852 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-scrub-repair/1 2024-09-15T21:50:50.852 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-15T21:50:50.852 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=77089584-713a-4af5-a641-d78887d8b875 2024-09-15T21:50:50.852 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd1 77089584-713a-4af5-a641-d78887d8b875' 2024-09-15T21:50:50.852 INFO:tasks.workunit.client.0.smithi136.stdout:add osd1 77089584-713a-4af5-a641-d78887d8b875 2024-09-15T21:50:50.853 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-15T21:50:50.869 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQC6VudmO/asMxAAOszNpmsvzed+IA4Axv42mA== 2024-09-15T21:50:50.869 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQC6VudmO/asMxAAOszNpmsvzed+IA4Axv42mA=="}' 2024-09-15T21:50:50.869 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 77089584-713a-4af5-a641-d78887d8b875 -i td/osd-scrub-repair/1/new.json 2024-09-15T21:50:51.263 INFO:tasks.workunit.client.0.smithi136.stdout:1 2024-09-15T21:50:51.275 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-scrub-repair/1/new.json 2024-09-15T21:50:51.276 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 1 --fsid=ce4892c7-6c6d-4b4f-8e6f-07a3e11133e5 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.52602/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_scrub_during_recovery=false --osd_repair_during_recovery=true --osd_debug_pretend_recovery_active=true --mkfs --key AQC6VudmO/asMxAAOszNpmsvzed+IA4Axv42mA== --osd-uuid 77089584-713a-4af5-a641-d78887d8b875 2024-09-15T21:50:51.306 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:50:51.305+0000 7fdba7119640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:50:51.373 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:50:51.371+0000 7fdba7119640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:50:51.374 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:50:51.372+0000 7fdba7119640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:50:51.374 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:50:51.373+0000 7fdba7119640 -1 bdev(0x560d3d309c00 td/osd-scrub-repair/1/block) open stat got: (1) Operation not permitted 2024-09-15T21:50:51.374 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:50:51.373+0000 7fdba7119640 -1 bluestore(td/osd-scrub-repair/1) _read_fsid unparsable uuid 2024-09-15T21:50:57.324 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-scrub-repair/1/keyring 2024-09-15T21:50:57.324 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-15T21:50:57.325 INFO:tasks.workunit.client.0.smithi136.stdout:adding osd1 key to auth repository 2024-09-15T21:50:57.326 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd1 key to auth repository 2024-09-15T21:50:57.326 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-scrub-repair/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-15T21:50:57.687 INFO:tasks.workunit.client.0.smithi136.stdout:start osd.1 2024-09-15T21:50:57.687 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.1 2024-09-15T21:50:57.687 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 1 --fsid=ce4892c7-6c6d-4b4f-8e6f-07a3e11133e5 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.52602/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_scrub_during_recovery=false --osd_repair_during_recovery=true --osd_debug_pretend_recovery_active=true 2024-09-15T21:50:57.688 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-15T21:50:57.688 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-15T21:50:57.688 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-15T21:50:57.716 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:50:57.714+0000 7f5332502640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:50:57.718 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:50:57.717+0000 7f5332502640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:50:57.719 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:50:57.718+0000 7f5332502640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:50:57.962 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 1 2024-09-15T21:50:57.962 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-15T21:50:57.962 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2024-09-15T21:50:57.962 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-15T21:50:57.962 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-15T21:50:57.962 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:50:57.962 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-15T21:50:57.963 INFO:tasks.workunit.client.0.smithi136.stdout:0 2024-09-15T21:50:57.963 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:50:57.963 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T21:50:58.238 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:50:59.240 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:50:59.240 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:50:59.240 INFO:tasks.workunit.client.0.smithi136.stdout:1 2024-09-15T21:50:59.241 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-15T21:50:59.241 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:50:59.241 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T21:50:59.342 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:50:59.341+0000 7f5332502640 -1 Falling back to public interface 2024-09-15T21:50:59.525 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:51:00.526 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:51:00.526 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:51:00.527 INFO:tasks.workunit.client.0.smithi136.stdout:2 2024-09-15T21:51:00.527 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-15T21:51:00.527 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:51:00.527 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T21:51:00.813 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:51:01.815 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:51:01.815 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:51:01.815 INFO:tasks.workunit.client.0.smithi136.stdout:3 2024-09-15T21:51:01.815 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-15T21:51:01.816 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:51:01.816 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T21:51:01.982 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:51:01.980+0000 7f5332502640 -1 osd.1 0 log_to_monitors true 2024-09-15T21:51:02.098 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:51:03.099 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:51:03.099 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:51:03.100 INFO:tasks.workunit.client.0.smithi136.stdout:4 2024-09-15T21:51:03.100 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-15T21:51:03.100 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:51:03.100 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T21:51:03.530 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:51:04.532 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:51:04.532 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:51:04.532 INFO:tasks.workunit.client.0.smithi136.stdout:5 2024-09-15T21:51:04.533 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-15T21:51:04.533 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:51:04.533 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T21:51:04.844 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:51:05.845 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:51:05.845 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:51:05.846 INFO:tasks.workunit.client.0.smithi136.stdout:6 2024-09-15T21:51:05.846 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-15T21:51:05.846 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:51:05.846 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T21:51:06.216 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:51:07.217 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:51:07.217 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:51:07.217 INFO:tasks.workunit.client.0.smithi136.stdout:7 2024-09-15T21:51:07.218 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-15T21:51:07.218 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:51:07.218 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T21:51:07.655 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:51:08.657 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:51:08.657 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:51:08.657 INFO:tasks.workunit.client.0.smithi136.stdout:8 2024-09-15T21:51:08.657 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-15T21:51:08.658 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:51:08.658 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T21:51:09.135 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:51:10.136 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:51:10.137 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:51:10.137 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-15T21:51:10.137 INFO:tasks.workunit.client.0.smithi136.stdout:9 2024-09-15T21:51:10.137 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:51:10.138 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T21:51:10.423 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:51:11.424 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:51:11.424 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:51:11.425 INFO:tasks.workunit.client.0.smithi136.stdout:10 2024-09-15T21:51:11.425 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-15T21:51:11.425 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:51:11.425 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T21:51:11.805 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:51:12.806 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:51:12.806 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:51:12.807 INFO:tasks.workunit.client.0.smithi136.stdout:11 2024-09-15T21:51:12.807 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-15T21:51:12.807 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:51:12.807 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T21:51:13.121 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:51:14.122 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:51:14.122 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:51:14.123 INFO:tasks.workunit.client.0.smithi136.stdout:12 2024-09-15T21:51:14.123 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-15T21:51:14.123 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:51:14.123 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T21:51:14.407 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:51:15.408 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:51:15.408 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:51:15.408 INFO:tasks.workunit.client.0.smithi136.stdout:13 2024-09-15T21:51:15.409 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-15T21:51:15.409 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:51:15.409 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T21:51:16.233 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:51:17.234 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:51:17.234 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:51:17.235 INFO:tasks.workunit.client.0.smithi136.stdout:14 2024-09-15T21:51:17.235 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-15T21:51:17.235 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:51:17.235 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T21:51:18.143 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:51:19.145 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:51:19.145 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:51:19.146 INFO:tasks.workunit.client.0.smithi136.stdout:15 2024-09-15T21:51:19.147 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-15T21:51:19.147 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:51:19.147 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T21:51:19.903 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:51:20.904 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:51:20.904 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:51:20.905 INFO:tasks.workunit.client.0.smithi136.stdout:16 2024-09-15T21:51:20.905 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-15T21:51:20.905 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:51:20.905 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T21:51:21.372 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:51:22.373 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:51:22.374 INFO:tasks.workunit.client.0.smithi136.stdout:17 2024-09-15T21:51:22.374 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:51:22.374 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-15T21:51:22.374 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:51:22.375 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T21:51:23.097 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:51:24.098 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:51:24.098 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:51:24.098 INFO:tasks.workunit.client.0.smithi136.stdout:18 2024-09-15T21:51:24.099 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 18 2024-09-15T21:51:24.099 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:51:24.099 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T21:51:24.383 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:51:25.385 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:51:25.385 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:51:25.385 INFO:tasks.workunit.client.0.smithi136.stdout:19 2024-09-15T21:51:25.386 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 19 2024-09-15T21:51:25.386 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:51:25.386 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T21:51:25.685 INFO:tasks.workunit.client.0.smithi136.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/1397493379,v1:127.0.0.1:6811/1397493379] [v2:127.0.0.1:6812/1397493379,v1:127.0.0.1:6813/1397493379] exists,up 77089584-713a-4af5-a641-d78887d8b875 2024-09-15T21:51:25.686 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-15T21:51:25.686 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-15T21:51:25.686 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-15T21:51:25.686 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:147: TEST_skip_non_repair_during_recovery: create_rbd_pool 2024-09-15T21:51:25.687 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:526: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2024-09-15T21:51:25.954 INFO:tasks.workunit.client.0.smithi136.stderr:pool 'rbd' does not exist 2024-09-15T21:51:25.965 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:527: create_rbd_pool: create_pool rbd 4 2024-09-15T21:51:25.966 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create rbd 4 2024-09-15T21:51:26.387 INFO:tasks.workunit.client.0.smithi136.stderr:pool 'rbd' created 2024-09-15T21:51:26.399 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2024-09-15T21:51:27.401 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:528: create_rbd_pool: rbd pool init rbd 2024-09-15T21:51:29.037 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:148: TEST_skip_non_repair_during_recovery: wait_for_clean 2024-09-15T21:51:29.037 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-15T21:51:29.038 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-15T21:51:29.038 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-15T21:51:29.038 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-15T21:51:29.039 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-15T21:51:29.039 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-15T21:51:29.039 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-15T21:51:29.039 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-15T21:51:29.039 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-15T21:51:29.141 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-15T21:51:29.142 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-15T21:51:29.142 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-15T21:51:29.142 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-15T21:51:29.142 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-15T21:51:29.142 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-15T21:51:29.421 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-15T21:51:29.421 INFO:tasks.workunit.client.0.smithi136.stderr:1' 2024-09-15T21:51:29.421 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-15T21:51:29.421 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T21:51:29.421 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-15T21:51:29.524 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836494 2024-09-15T21:51:29.524 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836494 2024-09-15T21:51:29.524 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836494' 2024-09-15T21:51:29.524 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T21:51:29.524 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-15T21:51:29.629 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=42949672965 2024-09-15T21:51:29.629 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 42949672965 2024-09-15T21:51:29.629 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836494 1-42949672965' 2024-09-15T21:51:29.629 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T21:51:29.629 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836494 2024-09-15T21:51:29.629 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T21:51:29.630 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-15T21:51:29.631 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836494 2024-09-15T21:51:29.631 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T21:51:29.632 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836494 2024-09-15T21:51:29.632 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836494' 2024-09-15T21:51:29.632 INFO:tasks.workunit.client.0.smithi136.stdout:waiting osd.0 seq 21474836494 2024-09-15T21:51:29.632 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T21:51:29.904 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836493 -lt 21474836494 2024-09-15T21:51:29.904 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-15T21:51:30.906 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-15T21:51:30.906 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T21:51:31.232 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836495 -lt 21474836494 2024-09-15T21:51:31.232 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T21:51:31.232 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-42949672965 2024-09-15T21:51:31.233 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T21:51:31.233 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-15T21:51:31.234 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-42949672965 2024-09-15T21:51:31.234 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T21:51:31.235 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=42949672965 2024-09-15T21:51:31.235 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 42949672965' 2024-09-15T21:51:31.236 INFO:tasks.workunit.client.0.smithi136.stdout:waiting osd.1 seq 42949672965 2024-09-15T21:51:31.236 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-15T21:51:31.546 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 42949672966 -lt 42949672965 2024-09-15T21:51:31.546 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-15T21:51:31.547 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T21:51:31.547 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T21:51:31.891 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 4 == 0 2024-09-15T21:51:31.891 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-15T21:51:31.892 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-15T21:51:31.892 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-15T21:51:31.892 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T21:51:31.892 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T21:51:31.892 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T21:51:31.892 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T21:51:32.165 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=4 2024-09-15T21:51:32.165 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-15T21:51:32.166 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T21:51:32.166 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T21:51:32.515 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 = 4 2024-09-15T21:51:32.515 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-15T21:51:32.515 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-15T21:51:32.515 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:150: TEST_skip_non_repair_during_recovery: add_something td/osd-scrub-repair rbd 2024-09-15T21:51:32.515 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-09-15T21:51:32.515 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=rbd 2024-09-15T21:51:32.516 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=SOMETHING 2024-09-15T21:51:32.516 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-09-15T21:51:32.516 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-09-15T21:51:32.516 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-09-15T21:51:32.889 INFO:tasks.workunit.client.0.smithi136.stderr:noscrub is set 2024-09-15T21:51:32.901 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-09-15T21:51:33.408 INFO:tasks.workunit.client.0.smithi136.stderr:nodeep-scrub is set 2024-09-15T21:51:33.420 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-09-15T21:51:33.420 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-09-15T21:51:33.420 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool rbd put SOMETHING td/osd-scrub-repair/ORIGINAL 2024-09-15T21:51:33.672 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:151: TEST_skip_non_repair_during_recovery: get_not_primary rbd SOMETHING 2024-09-15T21:51:33.672 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1220: get_not_primary: local poolname=rbd 2024-09-15T21:51:33.672 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1221: get_not_primary: local objectname=SOMETHING 2024-09-15T21:51:33.672 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1223: get_not_primary: get_primary rbd SOMETHING 2024-09-15T21:51:33.672 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1187: get_primary: local poolname=rbd 2024-09-15T21:51:33.672 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1188: get_primary: local objectname=SOMETHING 2024-09-15T21:51:33.672 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1190: get_primary: ceph --format json osd map rbd SOMETHING 2024-09-15T21:51:33.673 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1191: get_primary: jq .acting_primary 2024-09-15T21:51:33.947 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1223: get_not_primary: local primary=1 2024-09-15T21:51:33.947 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1224: get_not_primary: ceph --format json osd map rbd SOMETHING 2024-09-15T21:51:33.947 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1225: get_not_primary: jq '.acting | map(select (. != 1)) | .[0]' 2024-09-15T21:51:34.225 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:151: TEST_skip_non_repair_during_recovery: scrub_and_not_schedule td/osd-scrub-repair rbd 0 2024-09-15T21:51:34.226 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:155: scrub_and_not_schedule: local dir=td/osd-scrub-repair 2024-09-15T21:51:34.226 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:156: scrub_and_not_schedule: local poolname=rbd 2024-09-15T21:51:34.226 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:157: scrub_and_not_schedule: local osd=0 2024-09-15T21:51:34.226 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:162: scrub_and_not_schedule: get_pg rbd SOMETHING 2024-09-15T21:51:34.226 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1084: get_pg: local poolname=rbd 2024-09-15T21:51:34.226 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1085: get_pg: local objectname=SOMETHING 2024-09-15T21:51:34.226 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1087: get_pg: ceph --format json osd map rbd SOMETHING 2024-09-15T21:51:34.226 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1087: get_pg: jq -r .pgid 2024-09-15T21:51:34.502 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:162: scrub_and_not_schedule: local pg=1.3 2024-09-15T21:51:34.502 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:163: scrub_and_not_schedule: get_last_scrub_stamp 1.3 2024-09-15T21:51:34.502 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.3 2024-09-15T21:51:34.503 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-15T21:51:34.503 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-15T21:51:34.503 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.3") | .last_scrub_stamp' 2024-09-15T21:51:34.775 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:163: scrub_and_not_schedule: local last_scrub=2024-09-15T21:51:26.269551+0000 2024-09-15T21:51:34.775 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:164: scrub_and_not_schedule: ceph tell 1.3 schedule-scrub 2024-09-15T21:51:34.871 INFO:tasks.workunit.client.0.smithi136.stdout:{ 2024-09-15T21:51:34.872 INFO:tasks.workunit.client.0.smithi136.stdout: "deep": false, 2024-09-15T21:51:34.872 INFO:tasks.workunit.client.0.smithi136.stdout: "must": false, 2024-09-15T21:51:34.872 INFO:tasks.workunit.client.0.smithi136.stdout: "stamp": "2024-09-08T21:49:54.870885+0000" 2024-09-15T21:51:34.872 INFO:tasks.workunit.client.0.smithi136.stdout:} 2024-09-15T21:51:34.880 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:169: scrub_and_not_schedule: (( i=0 )) 2024-09-15T21:51:34.880 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:169: scrub_and_not_schedule: (( i < 3 )) 2024-09-15T21:51:34.881 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:170: scrub_and_not_schedule: get_last_scrub_stamp 1.3 2024-09-15T21:51:34.881 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.3 2024-09-15T21:51:34.881 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-15T21:51:34.881 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-15T21:51:34.881 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.3") | .last_scrub_stamp' 2024-09-15T21:51:35.155 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:170: scrub_and_not_schedule: test 2024-09-15T21:51:26.269551+0000 '>' 2024-09-15T21:51:26.269551+0000 2024-09-15T21:51:35.155 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:173: scrub_and_not_schedule: sleep 1 2024-09-15T21:51:36.156 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:169: scrub_and_not_schedule: (( i++ )) 2024-09-15T21:51:36.157 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:169: scrub_and_not_schedule: (( i < 3 )) 2024-09-15T21:51:36.157 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:170: scrub_and_not_schedule: get_last_scrub_stamp 1.3 2024-09-15T21:51:36.158 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.3 2024-09-15T21:51:36.158 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-15T21:51:36.158 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-15T21:51:36.158 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.3") | .last_scrub_stamp' 2024-09-15T21:51:36.442 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:170: scrub_and_not_schedule: test 2024-09-15T21:51:26.269551+0000 '>' 2024-09-15T21:51:26.269551+0000 2024-09-15T21:51:36.442 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:173: scrub_and_not_schedule: sleep 1 2024-09-15T21:51:37.443 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:169: scrub_and_not_schedule: (( i++ )) 2024-09-15T21:51:37.443 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:169: scrub_and_not_schedule: (( i < 3 )) 2024-09-15T21:51:37.444 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:170: scrub_and_not_schedule: get_last_scrub_stamp 1.3 2024-09-15T21:51:37.444 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.3 2024-09-15T21:51:37.444 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-15T21:51:37.445 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-15T21:51:37.445 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.3") | .last_scrub_stamp' 2024-09-15T21:51:37.725 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:170: scrub_and_not_schedule: test 2024-09-08T21:49:54.870885+0000 '>' 2024-09-15T21:51:26.269551+0000 2024-09-15T21:51:37.725 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:173: scrub_and_not_schedule: sleep 1 2024-09-15T21:51:38.728 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:169: scrub_and_not_schedule: (( i++ )) 2024-09-15T21:51:38.728 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:169: scrub_and_not_schedule: (( i < 3 )) 2024-09-15T21:51:38.728 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:179: scrub_and_not_schedule: objectstore_tool td/osd-scrub-repair 0 SOMETHING list-attrs 2024-09-15T21:51:38.728 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2024-09-15T21:51:38.729 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2024-09-15T21:51:38.729 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=0 2024-09-15T21:51:38.729 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2024-09-15T21:51:38.729 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 SOMETHING list-attrs 2024-09-15T21:51:38.729 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-09-15T21:51:38.729 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2024-09-15T21:51:38.729 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=0 2024-09-15T21:51:38.729 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2024-09-15T21:51:38.729 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2024-09-15T21:51:38.729 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-15T21:51:38.729 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-15T21:51:38.730 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-15T21:51:38.730 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-15T21:51:38.730 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-15T21:51:39.037 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-15T21:51:39.037 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 SOMETHING list-attrs 2024-09-15T21:51:39.037 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-15T21:51:39.037 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-15T21:51:39.037 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=0 2024-09-15T21:51:39.037 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-15T21:51:39.037 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-09-15T21:51:39.037 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 SOMETHING list-attrs 2024-09-15T21:51:39.520 INFO:tasks.workunit.client.0.smithi136.stdout:_ 2024-09-15T21:51:39.520 INFO:tasks.workunit.client.0.smithi136.stdout:snapset 2024-09-15T21:51:39.788 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 2024-09-15T21:51:39.788 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2024-09-15T21:51:39.789 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-15T21:51:39.789 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=0 2024-09-15T21:51:39.789 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-15T21:51:39.789 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-09-15T21:51:39.789 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=ce4892c7-6c6d-4b4f-8e6f-07a3e11133e5 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-15T21:51:39.789 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T21:51:39.789 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T21:51:39.789 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T21:51:39.789 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-09-15T21:51:39.789 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-09-15T21:51:39.789 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-15T21:51:39.790 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-15T21:51:39.790 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-15T21:51:39.790 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-15T21:51:39.790 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-15T21:51:39.790 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-15T21:51:39.790 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-15T21:51:39.790 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T21:51:39.790 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52602 2024-09-15T21:51:39.790 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.52602/$cluster-$name.asok' 2024-09-15T21:51:39.790 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.52602/$cluster-$name.asok' 2024-09-15T21:51:39.790 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-15T21:51:39.790 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-15T21:51:39.790 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-15T21:51:39.790 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T21:51:39.791 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T21:51:39.791 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T21:51:39.791 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T21:51:39.791 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-15T21:51:39.791 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-15T21:51:39.791 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-09-15T21:51:39.792 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.0 2024-09-15T21:51:39.792 INFO:tasks.workunit.client.0.smithi136.stderr:start osd.0 2024-09-15T21:51:39.792 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 0 --fsid=ce4892c7-6c6d-4b4f-8e6f-07a3e11133e5 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.52602/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T21:51:39.792 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-09-15T21:51:39.793 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 0 = 0 ']' 2024-09-15T21:51:39.793 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-15T21:51:39.793 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-15T21:51:39.793 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-15T21:51:39.820 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:51:39.818+0000 7f162f5a8640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:51:39.822 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:51:39.821+0000 7f162f5a8640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:51:39.823 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:51:39.822+0000 7f162f5a8640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:51:40.078 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 0 2024-09-15T21:51:40.078 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-15T21:51:40.078 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2024-09-15T21:51:40.078 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-15T21:51:40.078 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-15T21:51:40.079 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:51:40.079 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-15T21:51:40.079 INFO:tasks.workunit.client.0.smithi136.stderr:0 2024-09-15T21:51:40.079 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:51:40.079 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:51:40.353 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:51:41.181 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:51:41.179+0000 7f162f5a8640 -1 Falling back to public interface 2024-09-15T21:51:41.355 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:51:41.355 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:51:41.355 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-15T21:51:41.355 INFO:tasks.workunit.client.0.smithi136.stderr:1 2024-09-15T21:51:41.355 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:51:41.355 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:51:41.638 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:51:42.640 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:51:42.640 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:51:42.640 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-15T21:51:42.641 INFO:tasks.workunit.client.0.smithi136.stderr:2 2024-09-15T21:51:42.641 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:51:42.641 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:51:42.924 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:51:43.925 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:51:43.926 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:51:43.926 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-15T21:51:43.926 INFO:tasks.workunit.client.0.smithi136.stderr:3 2024-09-15T21:51:43.926 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:51:43.926 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:51:44.205 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:51:44.343 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:51:44.341+0000 7f162f5a8640 -1 osd.0 22 log_to_monitors true 2024-09-15T21:51:45.207 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:51:45.207 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:51:45.207 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-15T21:51:45.207 INFO:tasks.workunit.client.0.smithi136.stderr:4 2024-09-15T21:51:45.207 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:51:45.207 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:51:45.642 INFO:tasks.workunit.client.0.smithi136.stderr:osd.0 up in weight 1 up_from 26 up_thru 26 down_at 23 last_clean_interval [5,22) [v2:127.0.0.1:6802/10754238,v1:127.0.0.1:6803/10754238] [v2:127.0.0.1:6804/10754238,v1:127.0.0.1:6805/10754238] exists,up 78e6362a-cc7e-43d3-a140-b4f9ac206c2c 2024-09-15T21:51:45.642 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-15T21:51:45.642 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-15T21:51:45.642 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-15T21:51:45.642 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2024-09-15T21:51:45.642 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-15T21:51:45.642 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-15T21:51:45.642 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-15T21:51:45.642 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-15T21:51:45.643 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-15T21:51:45.643 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-15T21:51:45.643 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-15T21:51:45.643 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-15T21:51:45.644 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-15T21:51:45.738 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-15T21:51:45.738 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-15T21:51:45.739 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-15T21:51:45.739 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-15T21:51:45.739 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-15T21:51:45.739 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-15T21:51:46.028 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-15T21:51:46.028 INFO:tasks.workunit.client.0.smithi136.stderr:1' 2024-09-15T21:51:46.028 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-15T21:51:46.028 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T21:51:46.028 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-15T21:51:46.135 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=111669149698 2024-09-15T21:51:46.135 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 111669149698 2024-09-15T21:51:46.135 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-111669149698' 2024-09-15T21:51:46.135 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T21:51:46.135 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-15T21:51:46.238 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=42949672970 2024-09-15T21:51:46.239 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 42949672970 2024-09-15T21:51:46.239 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-111669149698 1-42949672970' 2024-09-15T21:51:46.239 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T21:51:46.239 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-111669149698 2024-09-15T21:51:46.239 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T21:51:46.240 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-15T21:51:46.241 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-111669149698 2024-09-15T21:51:46.241 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T21:51:46.242 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=111669149698 2024-09-15T21:51:46.242 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 111669149698' 2024-09-15T21:51:46.242 INFO:tasks.workunit.client.0.smithi136.stderr:waiting osd.0 seq 111669149698 2024-09-15T21:51:46.242 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T21:51:46.518 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 1 -lt 111669149698 2024-09-15T21:51:46.518 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-15T21:51:47.519 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-15T21:51:47.520 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T21:51:47.807 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 111669149698 -lt 111669149698 2024-09-15T21:51:47.808 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T21:51:47.808 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-42949672970 2024-09-15T21:51:47.808 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T21:51:47.809 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-15T21:51:47.810 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-42949672970 2024-09-15T21:51:47.810 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T21:51:47.811 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=42949672970 2024-09-15T21:51:47.811 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 42949672970' 2024-09-15T21:51:47.811 INFO:tasks.workunit.client.0.smithi136.stderr:waiting osd.1 seq 42949672970 2024-09-15T21:51:47.811 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-15T21:51:48.089 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 42949672970 -lt 42949672970 2024-09-15T21:51:48.090 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-15T21:51:48.090 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T21:51:48.090 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T21:51:48.436 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 4 == 0 2024-09-15T21:51:48.436 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-15T21:51:48.437 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-15T21:51:48.437 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-15T21:51:48.437 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T21:51:48.437 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T21:51:48.437 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T21:51:48.437 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T21:51:48.707 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=4 2024-09-15T21:51:48.708 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-15T21:51:48.708 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T21:51:48.708 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T21:51:49.058 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 = 4 2024-09-15T21:51:49.058 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-15T21:51:49.058 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-15T21:51:49.058 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:180: scrub_and_not_schedule: rados --pool rbd get SOMETHING td/osd-scrub-repair/COPY 2024-09-15T21:51:49.087 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:181: scrub_and_not_schedule: diff td/osd-scrub-repair/ORIGINAL td/osd-scrub-repair/COPY 2024-09-15T21:51:49.089 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:66: run: teardown td/osd-scrub-repair 2024-09-15T21:51:49.089 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/osd-scrub-repair 2024-09-15T21:51:49.089 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-15T21:51:49.089 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/osd-scrub-repair KILL 2024-09-15T21:51:49.089 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-15T21:51:49.089 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-15T21:51:49.089 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-15T21:51:49.090 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-15T21:51:49.090 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-15T21:51:49.197 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-15T21:51:49.197 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-15T21:51:49.198 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-15T21:51:49.198 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-15T21:51:49.200 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-15T21:51:49.200 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-15T21:51:49.200 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-15T21:51:49.203 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T21:51:49.203 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-15T21:51:49.203 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-15T21:51:49.203 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T21:51:49.203 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-15T21:51:49.204 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-15T21:51:49.204 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/osd-scrub-repair 2024-09-15T21:51:49.218 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-15T21:51:49.218 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T21:51:49.218 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52602 2024-09-15T21:51:49.219 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.52602 2024-09-15T21:51:49.219 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-15T21:51:49.219 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-15T21:51:49.219 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:63: run: for func in $funcs 2024-09-15T21:51:49.219 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:64: run: setup td/osd-scrub-repair 2024-09-15T21:51:49.219 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/osd-scrub-repair 2024-09-15T21:51:49.220 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/osd-scrub-repair 2024-09-15T21:51:49.220 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/osd-scrub-repair 2024-09-15T21:51:49.220 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-15T21:51:49.220 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/osd-scrub-repair KILL 2024-09-15T21:51:49.220 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-15T21:51:49.220 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-15T21:51:49.221 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-15T21:51:49.221 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-15T21:51:49.221 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-15T21:51:49.224 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-15T21:51:49.225 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-15T21:51:49.225 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-15T21:51:49.225 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-15T21:51:49.225 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-15T21:51:49.225 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-15T21:51:49.226 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-15T21:51:49.226 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T21:51:49.226 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-15T21:51:49.226 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-15T21:51:49.226 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T21:51:49.226 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-15T21:51:49.227 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-15T21:51:49.227 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/osd-scrub-repair 2024-09-15T21:51:49.235 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-15T21:51:49.235 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T21:51:49.236 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52602 2024-09-15T21:51:49.236 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.52602 2024-09-15T21:51:49.236 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-15T21:51:49.236 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-15T21:51:49.236 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/osd-scrub-repair 2024-09-15T21:51:49.236 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2024-09-15T21:51:49.237 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T21:51:49.237 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52602 2024-09-15T21:51:49.237 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.52602 2024-09-15T21:51:49.237 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2024-09-15T21:51:49.237 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 4096 -le 1024 ']' 2024-09-15T21:51:49.237 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2024-09-15T21:51:49.238 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/osd-scrub-repair 1' TERM HUP INT 2024-09-15T21:51:49.238 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:65: run: TEST_unfound_erasure_coded_appends td/osd-scrub-repair 2024-09-15T21:51:49.238 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:903: TEST_unfound_erasure_coded_appends: unfound_erasure_coded td/osd-scrub-repair false 2024-09-15T21:51:49.238 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:852: unfound_erasure_coded: local dir=td/osd-scrub-repair 2024-09-15T21:51:49.238 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:853: unfound_erasure_coded: local allow_overwrites=false 2024-09-15T21:51:49.238 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:854: unfound_erasure_coded: local poolname=ecpool 2024-09-15T21:51:49.239 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:855: unfound_erasure_coded: local payload=ABCDEF 2024-09-15T21:51:49.239 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:857: unfound_erasure_coded: run_mon td/osd-scrub-repair a 2024-09-15T21:51:49.239 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/osd-scrub-repair 2024-09-15T21:51:49.239 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2024-09-15T21:51:49.239 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2024-09-15T21:51:49.240 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2024-09-15T21:51:49.240 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/osd-scrub-repair/a 2024-09-15T21:51:49.240 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-scrub-repair/a --run-dir=td/osd-scrub-repair 2024-09-15T21:51:49.669 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2024-09-15T21:51:49.669 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-15T21:51:49.669 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-15T21:51:49.669 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-15T21:51:49.669 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T21:51:49.669 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52602 2024-09-15T21:51:49.669 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.52602/$cluster-$name.asok' 2024-09-15T21:51:49.670 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-scrub-repair/a '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.52602/$cluster-$name.asok' --mon-cluster-log-file=td/osd-scrub-repair/log --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-09-15T21:51:49.869 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2024-09-15T21:51:49.870 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2024-09-15T21:51:49.870 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-15T21:51:49.870 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-15T21:51:49.870 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2024-09-15T21:51:49.871 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2024-09-15T21:51:49.871 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-15T21:51:49.871 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-15T21:51:49.871 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-15T21:51:49.871 INFO:tasks.workunit.client.0.smithi136.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-15T21:51:49.871 INFO:tasks.workunit.client.0.smithi136.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T21:51:49.872 INFO:tasks.workunit.client.0.smithi136.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52602 2024-09-15T21:51:49.872 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.52602/ceph-mon.a.asok 2024-09-15T21:51:49.872 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-15T21:51:49.872 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.52602/ceph-mon.a.asok config get fsid 2024-09-15T21:51:49.951 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2024-09-15T21:51:49.952 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-15T21:51:49.952 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-15T21:51:49.952 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2024-09-15T21:51:49.952 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2024-09-15T21:51:49.952 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-15T21:51:49.952 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-15T21:51:49.952 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-15T21:51:49.953 INFO:tasks.workunit.client.0.smithi136.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-15T21:51:49.953 INFO:tasks.workunit.client.0.smithi136.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T21:51:49.953 INFO:tasks.workunit.client.0.smithi136.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52602 2024-09-15T21:51:49.953 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.52602/ceph-mon.a.asok 2024-09-15T21:51:49.953 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-15T21:51:49.953 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.52602/ceph-mon.a.asok config get mon_host 2024-09-15T21:51:50.031 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:858: unfound_erasure_coded: run_mgr td/osd-scrub-repair x 2024-09-15T21:51:50.031 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: run_mgr: local dir=td/osd-scrub-repair 2024-09-15T21:51:50.031 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: run_mgr: shift 2024-09-15T21:51:50.031 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: run_mgr: local id=x 2024-09-15T21:51:50.032 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: run_mgr: shift 2024-09-15T21:51:50.032 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:548: run_mgr: local data=td/osd-scrub-repair/x 2024-09-15T21:51:50.032 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:550: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-15T21:51:50.837 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: get_asok_path 2024-09-15T21:51:50.837 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-15T21:51:50.837 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-15T21:51:50.837 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-15T21:51:50.837 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T21:51:50.837 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52602 2024-09-15T21:51:50.837 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.52602/$cluster-$name.asok' 2024-09-15T21:51:50.838 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-15T21:51:50.838 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-scrub-repair/x '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.52602/$cluster-$name.asok' --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-15T21:51:50.868 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:859: unfound_erasure_coded: seq 0 3 2024-09-15T21:51:50.872 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:859: unfound_erasure_coded: for id in $(seq 0 3) 2024-09-15T21:51:50.872 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:860: unfound_erasure_coded: run_osd td/osd-scrub-repair 0 2024-09-15T21:51:50.872 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-scrub-repair 2024-09-15T21:51:50.872 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-15T21:51:50.872 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=0 2024-09-15T21:51:50.872 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-15T21:51:50.872 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-scrub-repair/0 2024-09-15T21:51:50.872 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=ce4892c7-6c6d-4b4f-8e6f-07a3e11133e5 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-15T21:51:50.872 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T21:51:50.872 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T21:51:50.872 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T21:51:50.873 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-09-15T21:51:50.873 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-09-15T21:51:50.873 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-15T21:51:50.873 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-15T21:51:50.873 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-15T21:51:50.873 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-15T21:51:50.873 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-15T21:51:50.873 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-15T21:51:50.873 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-15T21:51:50.873 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T21:51:50.873 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52602 2024-09-15T21:51:50.873 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.52602/$cluster-$name.asok' 2024-09-15T21:51:50.873 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.52602/$cluster-$name.asok' 2024-09-15T21:51:50.874 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-15T21:51:50.874 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-15T21:51:50.874 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-15T21:51:50.874 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-15T21:51:50.874 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-15T21:51:50.874 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T21:51:50.874 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T21:51:50.874 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T21:51:50.874 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T21:51:50.874 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-15T21:51:50.874 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-15T21:51:50.874 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-scrub-repair/0 2024-09-15T21:51:50.875 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-15T21:51:50.875 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=bbf3cffd-17f5-4af6-95f1-e75ca630ad35 2024-09-15T21:51:50.875 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd0 bbf3cffd-17f5-4af6-95f1-e75ca630ad35' 2024-09-15T21:51:50.875 INFO:tasks.workunit.client.0.smithi136.stdout:add osd0 bbf3cffd-17f5-4af6-95f1-e75ca630ad35 2024-09-15T21:51:50.876 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-15T21:51:50.892 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQD2VudmWtcPNRAAVjhb4fk2euxyZ8JCzz/weA== 2024-09-15T21:51:50.892 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQD2VudmWtcPNRAAVjhb4fk2euxyZ8JCzz/weA=="}' 2024-09-15T21:51:50.892 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new bbf3cffd-17f5-4af6-95f1-e75ca630ad35 -i td/osd-scrub-repair/0/new.json 2024-09-15T21:51:51.093 INFO:tasks.workunit.client.0.smithi136.stdout:0 2024-09-15T21:51:51.103 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-scrub-repair/0/new.json 2024-09-15T21:51:51.104 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 0 --fsid=ce4892c7-6c6d-4b4f-8e6f-07a3e11133e5 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.52602/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQD2VudmWtcPNRAAVjhb4fk2euxyZ8JCzz/weA== --osd-uuid bbf3cffd-17f5-4af6-95f1-e75ca630ad35 2024-09-15T21:51:51.133 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:51:51.132+0000 7fc1a8d45640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:51:51.144 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:51:51.142+0000 7fc1a8d45640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:51:51.145 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:51:51.143+0000 7fc1a8d45640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:51:51.145 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:51:51.144+0000 7fc1a8d45640 -1 bdev(0x55b31e26fc00 td/osd-scrub-repair/0/block) open stat got: (1) Operation not permitted 2024-09-15T21:51:51.145 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:51:51.144+0000 7fc1a8d45640 -1 bluestore(td/osd-scrub-repair/0) _read_fsid unparsable uuid 2024-09-15T21:51:56.638 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-scrub-repair/0/keyring 2024-09-15T21:51:56.638 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-15T21:51:56.639 INFO:tasks.workunit.client.0.smithi136.stdout:adding osd0 key to auth repository 2024-09-15T21:51:56.640 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd0 key to auth repository 2024-09-15T21:51:56.640 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-scrub-repair/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-15T21:51:57.016 INFO:tasks.workunit.client.0.smithi136.stdout:start osd.0 2024-09-15T21:51:57.017 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.0 2024-09-15T21:51:57.017 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 0 --fsid=ce4892c7-6c6d-4b4f-8e6f-07a3e11133e5 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.52602/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T21:51:57.017 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-15T21:51:57.017 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-15T21:51:57.017 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-15T21:51:57.043 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:51:57.042+0000 7fae7940f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:51:57.073 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:51:57.072+0000 7fae7940f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:51:57.074 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:51:57.073+0000 7fae7940f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:51:57.296 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 0 2024-09-15T21:51:57.296 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-15T21:51:57.296 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2024-09-15T21:51:57.296 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-15T21:51:57.296 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-15T21:51:57.296 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:51:57.296 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-15T21:51:57.297 INFO:tasks.workunit.client.0.smithi136.stdout:0 2024-09-15T21:51:57.297 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:51:57.297 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:51:57.572 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:51:58.434 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:51:58.432+0000 7fae7940f640 -1 Falling back to public interface 2024-09-15T21:51:58.573 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:51:58.573 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:51:58.574 INFO:tasks.workunit.client.0.smithi136.stdout:1 2024-09-15T21:51:58.574 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-15T21:51:58.574 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:51:58.574 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:51:58.862 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:51:59.863 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:51:59.863 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:51:59.863 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-15T21:51:59.863 INFO:tasks.workunit.client.0.smithi136.stdout:2 2024-09-15T21:51:59.864 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:51:59.864 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:52:00.151 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:52:01.047 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:52:01.045+0000 7fae7940f640 -1 osd.0 0 log_to_monitors true 2024-09-15T21:52:01.153 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:52:01.153 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:52:01.153 INFO:tasks.workunit.client.0.smithi136.stdout:3 2024-09-15T21:52:01.154 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-15T21:52:01.154 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:52:01.154 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:52:01.489 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:52:02.493 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:52:02.493 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:52:02.493 INFO:tasks.workunit.client.0.smithi136.stdout:4 2024-09-15T21:52:02.494 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-15T21:52:02.494 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:52:02.494 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:52:02.803 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:52:03.804 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:52:03.804 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:52:03.805 INFO:tasks.workunit.client.0.smithi136.stdout:5 2024-09-15T21:52:03.805 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-15T21:52:03.805 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:52:03.805 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:52:04.092 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:52:05.094 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:52:05.094 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:52:05.094 INFO:tasks.workunit.client.0.smithi136.stdout:6 2024-09-15T21:52:05.095 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-15T21:52:05.095 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:52:05.095 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:52:05.383 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:52:06.384 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:52:06.384 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:52:06.384 INFO:tasks.workunit.client.0.smithi136.stdout:7 2024-09-15T21:52:06.385 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-15T21:52:06.385 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:52:06.385 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:52:06.673 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:52:07.675 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:52:07.675 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:52:07.675 INFO:tasks.workunit.client.0.smithi136.stdout:8 2024-09-15T21:52:07.675 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-15T21:52:07.676 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:52:07.676 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:52:07.996 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:52:08.998 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:52:08.998 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:52:08.998 INFO:tasks.workunit.client.0.smithi136.stdout:9 2024-09-15T21:52:08.999 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-15T21:52:08.999 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:52:08.999 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:52:09.288 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:52:10.289 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:52:10.289 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:52:10.290 INFO:tasks.workunit.client.0.smithi136.stdout:10 2024-09-15T21:52:10.290 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-15T21:52:10.290 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:52:10.290 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:52:10.596 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:52:11.597 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:52:11.597 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:52:11.598 INFO:tasks.workunit.client.0.smithi136.stdout:11 2024-09-15T21:52:11.598 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-15T21:52:11.598 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:52:11.598 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:52:11.991 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:52:12.992 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:52:12.993 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:52:12.993 INFO:tasks.workunit.client.0.smithi136.stdout:12 2024-09-15T21:52:12.993 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-15T21:52:12.994 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:52:12.994 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:52:13.528 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:52:14.530 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:52:14.530 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:52:14.530 INFO:tasks.workunit.client.0.smithi136.stdout:13 2024-09-15T21:52:14.530 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-15T21:52:14.531 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:52:14.531 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:52:15.126 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:52:16.128 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:52:16.128 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:52:16.128 INFO:tasks.workunit.client.0.smithi136.stdout:14 2024-09-15T21:52:16.129 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-15T21:52:16.129 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:52:16.129 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:52:16.419 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:52:17.421 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:52:17.421 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:52:17.421 INFO:tasks.workunit.client.0.smithi136.stdout:15 2024-09-15T21:52:17.421 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-15T21:52:17.422 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:52:17.422 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:52:18.002 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:52:19.003 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:52:19.004 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:52:19.004 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-15T21:52:19.004 INFO:tasks.workunit.client.0.smithi136.stdout:16 2024-09-15T21:52:19.004 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:52:19.004 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:52:20.056 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:52:21.057 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:52:21.058 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:52:21.058 INFO:tasks.workunit.client.0.smithi136.stdout:17 2024-09-15T21:52:21.058 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-15T21:52:21.058 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:52:21.058 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:52:21.816 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:52:22.818 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:52:22.818 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:52:22.818 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 18 2024-09-15T21:52:22.819 INFO:tasks.workunit.client.0.smithi136.stdout:18 2024-09-15T21:52:22.819 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:52:22.820 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:52:23.263 INFO:tasks.workunit.client.0.smithi136.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/2648329848,v1:127.0.0.1:6803/2648329848] [v2:127.0.0.1:6804/2648329848,v1:127.0.0.1:6805/2648329848] exists,up bbf3cffd-17f5-4af6-95f1-e75ca630ad35 2024-09-15T21:52:23.267 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-15T21:52:23.267 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-15T21:52:23.267 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-15T21:52:23.267 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:859: unfound_erasure_coded: for id in $(seq 0 3) 2024-09-15T21:52:23.267 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:860: unfound_erasure_coded: run_osd td/osd-scrub-repair 1 2024-09-15T21:52:23.267 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-scrub-repair 2024-09-15T21:52:23.268 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-15T21:52:23.268 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=1 2024-09-15T21:52:23.268 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-15T21:52:23.268 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-scrub-repair/1 2024-09-15T21:52:23.268 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=ce4892c7-6c6d-4b4f-8e6f-07a3e11133e5 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-15T21:52:23.268 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T21:52:23.268 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T21:52:23.268 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T21:52:23.268 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-09-15T21:52:23.268 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-09-15T21:52:23.268 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-15T21:52:23.268 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-15T21:52:23.268 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-15T21:52:23.269 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-15T21:52:23.269 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-15T21:52:23.269 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-15T21:52:23.269 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-15T21:52:23.269 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T21:52:23.269 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52602 2024-09-15T21:52:23.269 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.52602/$cluster-$name.asok' 2024-09-15T21:52:23.269 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.52602/$cluster-$name.asok' 2024-09-15T21:52:23.269 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-15T21:52:23.269 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-15T21:52:23.269 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-15T21:52:23.269 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-15T21:52:23.269 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-15T21:52:23.269 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T21:52:23.269 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T21:52:23.270 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T21:52:23.270 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T21:52:23.270 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-15T21:52:23.270 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-15T21:52:23.270 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-scrub-repair/1 2024-09-15T21:52:23.270 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-15T21:52:23.270 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=1908b9ce-60a2-4d6e-bf9b-aab52455d6b2 2024-09-15T21:52:23.271 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd1 1908b9ce-60a2-4d6e-bf9b-aab52455d6b2' 2024-09-15T21:52:23.271 INFO:tasks.workunit.client.0.smithi136.stdout:add osd1 1908b9ce-60a2-4d6e-bf9b-aab52455d6b2 2024-09-15T21:52:23.271 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-15T21:52:23.286 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQAXV+dmjHn6EBAAK2NWSFBy3fkHJljW3nseMQ== 2024-09-15T21:52:23.287 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQAXV+dmjHn6EBAAK2NWSFBy3fkHJljW3nseMQ=="}' 2024-09-15T21:52:23.287 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 1908b9ce-60a2-4d6e-bf9b-aab52455d6b2 -i td/osd-scrub-repair/1/new.json 2024-09-15T21:52:23.611 INFO:tasks.workunit.client.0.smithi136.stdout:1 2024-09-15T21:52:23.622 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-scrub-repair/1/new.json 2024-09-15T21:52:23.623 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 1 --fsid=ce4892c7-6c6d-4b4f-8e6f-07a3e11133e5 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.52602/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAXV+dmjHn6EBAAK2NWSFBy3fkHJljW3nseMQ== --osd-uuid 1908b9ce-60a2-4d6e-bf9b-aab52455d6b2 2024-09-15T21:52:23.653 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:52:23.651+0000 7f20695cc640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:52:23.695 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:52:23.694+0000 7f20695cc640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:52:23.696 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:52:23.695+0000 7f20695cc640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:52:23.697 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:52:23.695+0000 7f20695cc640 -1 bdev(0x555a110dfc00 td/osd-scrub-repair/1/block) open stat got: (1) Operation not permitted 2024-09-15T21:52:23.697 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:52:23.695+0000 7f20695cc640 -1 bluestore(td/osd-scrub-repair/1) _read_fsid unparsable uuid 2024-09-15T21:52:30.261 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-scrub-repair/1/keyring 2024-09-15T21:52:30.261 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-15T21:52:30.261 INFO:tasks.workunit.client.0.smithi136.stdout:adding osd1 key to auth repository 2024-09-15T21:52:30.262 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd1 key to auth repository 2024-09-15T21:52:30.262 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-scrub-repair/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-15T21:52:30.621 INFO:tasks.workunit.client.0.smithi136.stdout:start osd.1 2024-09-15T21:52:30.621 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.1 2024-09-15T21:52:30.621 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-15T21:52:30.621 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 1 --fsid=ce4892c7-6c6d-4b4f-8e6f-07a3e11133e5 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.52602/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T21:52:30.622 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-15T21:52:30.622 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-15T21:52:30.648 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:52:30.646+0000 7f65643dd640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:52:30.650 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:52:30.648+0000 7f65643dd640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:52:30.651 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:52:30.649+0000 7f65643dd640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:52:30.899 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 1 2024-09-15T21:52:30.899 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-15T21:52:30.899 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2024-09-15T21:52:30.899 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-15T21:52:30.899 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-15T21:52:30.899 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:52:30.899 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-15T21:52:30.899 INFO:tasks.workunit.client.0.smithi136.stdout:0 2024-09-15T21:52:30.900 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:52:30.900 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T21:52:31.174 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:52:32.003 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:52:32.001+0000 7f65643dd640 -1 Falling back to public interface 2024-09-15T21:52:32.176 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:52:32.176 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:52:32.176 INFO:tasks.workunit.client.0.smithi136.stdout:1 2024-09-15T21:52:32.176 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-15T21:52:32.177 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:52:32.177 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T21:52:32.467 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:52:33.468 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:52:33.468 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:52:33.468 INFO:tasks.workunit.client.0.smithi136.stdout:2 2024-09-15T21:52:33.469 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-15T21:52:33.469 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:52:33.469 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T21:52:33.754 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:52:34.677 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:52:34.676+0000 7f65643dd640 -1 osd.1 0 log_to_monitors true 2024-09-15T21:52:34.756 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:52:34.756 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:52:34.756 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-15T21:52:34.756 INFO:tasks.workunit.client.0.smithi136.stdout:3 2024-09-15T21:52:34.757 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:52:34.757 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T21:52:35.187 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:52:36.188 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:52:36.189 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:52:36.189 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-15T21:52:36.189 INFO:tasks.workunit.client.0.smithi136.stdout:4 2024-09-15T21:52:36.189 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:52:36.189 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T21:52:36.552 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:52:37.554 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:52:37.554 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:52:37.554 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-15T21:52:37.555 INFO:tasks.workunit.client.0.smithi136.stdout:5 2024-09-15T21:52:37.555 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:52:37.555 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T21:52:37.841 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:52:38.842 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:52:38.842 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:52:38.843 INFO:tasks.workunit.client.0.smithi136.stdout:6 2024-09-15T21:52:38.843 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-15T21:52:38.843 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:52:38.843 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T21:52:39.156 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:52:40.157 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:52:40.158 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:52:40.158 INFO:tasks.workunit.client.0.smithi136.stdout:7 2024-09-15T21:52:40.159 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-15T21:52:40.159 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:52:40.159 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T21:52:40.546 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:52:41.547 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:52:41.548 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:52:41.548 INFO:tasks.workunit.client.0.smithi136.stdout:8 2024-09-15T21:52:41.548 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-15T21:52:41.548 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:52:41.549 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T21:52:41.927 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:52:42.929 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:52:42.929 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:52:42.929 INFO:tasks.workunit.client.0.smithi136.stdout:9 2024-09-15T21:52:42.929 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-15T21:52:42.930 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:52:42.930 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T21:52:43.270 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:52:44.271 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:52:44.272 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:52:44.272 INFO:tasks.workunit.client.0.smithi136.stdout:10 2024-09-15T21:52:44.272 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-15T21:52:44.273 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:52:44.273 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T21:52:44.727 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:52:45.729 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:52:45.729 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:52:45.729 INFO:tasks.workunit.client.0.smithi136.stdout:11 2024-09-15T21:52:45.730 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-15T21:52:45.730 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:52:45.730 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T21:52:46.015 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:52:47.017 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:52:47.017 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:52:47.017 INFO:tasks.workunit.client.0.smithi136.stdout:12 2024-09-15T21:52:47.018 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-15T21:52:47.018 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:52:47.018 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T21:52:47.344 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:52:48.345 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:52:48.345 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:52:48.346 INFO:tasks.workunit.client.0.smithi136.stdout:13 2024-09-15T21:52:48.346 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-15T21:52:48.346 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:52:48.346 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T21:52:48.981 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:52:49.983 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:52:49.983 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:52:49.983 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-15T21:52:49.983 INFO:tasks.workunit.client.0.smithi136.stdout:14 2024-09-15T21:52:49.983 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:52:49.983 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T21:52:50.655 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:52:51.657 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:52:51.657 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:52:51.657 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-15T21:52:51.657 INFO:tasks.workunit.client.0.smithi136.stdout:15 2024-09-15T21:52:51.658 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:52:51.658 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T21:52:52.237 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:52:53.239 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:52:53.239 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:52:53.239 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-15T21:52:53.239 INFO:tasks.workunit.client.0.smithi136.stdout:16 2024-09-15T21:52:53.240 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:52:53.240 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T21:52:53.596 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:52:54.597 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:52:54.598 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:52:54.598 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-15T21:52:54.598 INFO:tasks.workunit.client.0.smithi136.stdout:17 2024-09-15T21:52:54.598 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:52:54.598 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T21:52:55.066 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:52:56.068 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:52:56.068 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:52:56.068 INFO:tasks.workunit.client.0.smithi136.stdout:18 2024-09-15T21:52:56.069 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 18 2024-09-15T21:52:56.069 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:52:56.069 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T21:52:56.408 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:52:57.410 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:52:57.410 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:52:57.410 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 19 2024-09-15T21:52:57.411 INFO:tasks.workunit.client.0.smithi136.stdout:19 2024-09-15T21:52:57.411 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:52:57.411 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T21:52:58.154 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:52:59.155 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:52:59.155 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:52:59.156 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 20 2024-09-15T21:52:59.156 INFO:tasks.workunit.client.0.smithi136.stdout:20 2024-09-15T21:52:59.156 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:52:59.156 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T21:52:59.432 INFO:tasks.workunit.client.0.smithi136.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/2166261312,v1:127.0.0.1:6811/2166261312] [v2:127.0.0.1:6812/2166261312,v1:127.0.0.1:6813/2166261312] exists,up 1908b9ce-60a2-4d6e-bf9b-aab52455d6b2 2024-09-15T21:52:59.436 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-15T21:52:59.436 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-15T21:52:59.436 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-15T21:52:59.436 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:859: unfound_erasure_coded: for id in $(seq 0 3) 2024-09-15T21:52:59.436 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:860: unfound_erasure_coded: run_osd td/osd-scrub-repair 2 2024-09-15T21:52:59.436 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-scrub-repair 2024-09-15T21:52:59.437 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-15T21:52:59.437 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=2 2024-09-15T21:52:59.437 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-15T21:52:59.437 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-scrub-repair/2 2024-09-15T21:52:59.437 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=ce4892c7-6c6d-4b4f-8e6f-07a3e11133e5 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-15T21:52:59.437 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T21:52:59.437 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T21:52:59.437 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T21:52:59.437 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/2' 2024-09-15T21:52:59.437 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/2/journal' 2024-09-15T21:52:59.437 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-15T21:52:59.437 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-15T21:52:59.438 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-15T21:52:59.438 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-15T21:52:59.438 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-15T21:52:59.438 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-15T21:52:59.438 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-15T21:52:59.438 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T21:52:59.438 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52602 2024-09-15T21:52:59.438 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.52602/$cluster-$name.asok' 2024-09-15T21:52:59.438 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.52602/$cluster-$name.asok' 2024-09-15T21:52:59.438 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-15T21:52:59.438 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-15T21:52:59.438 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-15T21:52:59.438 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-15T21:52:59.438 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-15T21:52:59.439 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T21:52:59.439 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T21:52:59.439 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T21:52:59.439 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T21:52:59.439 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-15T21:52:59.439 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-15T21:52:59.439 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-scrub-repair/2 2024-09-15T21:52:59.439 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-15T21:52:59.440 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=5e86a859-3353-49e8-b6c4-5fb38b8b4f05 2024-09-15T21:52:59.440 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd2 5e86a859-3353-49e8-b6c4-5fb38b8b4f05' 2024-09-15T21:52:59.440 INFO:tasks.workunit.client.0.smithi136.stdout:add osd2 5e86a859-3353-49e8-b6c4-5fb38b8b4f05 2024-09-15T21:52:59.440 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-15T21:52:59.457 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQA7V+dmaPcfGxAAD/Ef1ZD3W6F0iFVs+QJQdQ== 2024-09-15T21:52:59.457 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQA7V+dmaPcfGxAAD/Ef1ZD3W6F0iFVs+QJQdQ=="}' 2024-09-15T21:52:59.457 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 5e86a859-3353-49e8-b6c4-5fb38b8b4f05 -i td/osd-scrub-repair/2/new.json 2024-09-15T21:52:59.806 INFO:tasks.workunit.client.0.smithi136.stdout:2 2024-09-15T21:52:59.817 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-scrub-repair/2/new.json 2024-09-15T21:52:59.818 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 2 --fsid=ce4892c7-6c6d-4b4f-8e6f-07a3e11133e5 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.52602/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQA7V+dmaPcfGxAAD/Ef1ZD3W6F0iFVs+QJQdQ== --osd-uuid 5e86a859-3353-49e8-b6c4-5fb38b8b4f05 2024-09-15T21:52:59.848 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:52:59.847+0000 7fbbd860f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:52:59.890 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:52:59.889+0000 7fbbd860f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:52:59.891 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:52:59.890+0000 7fbbd860f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:52:59.892 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:52:59.890+0000 7fbbd860f640 -1 bdev(0x55b526b6bc00 td/osd-scrub-repair/2/block) open stat got: (1) Operation not permitted 2024-09-15T21:52:59.892 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:52:59.890+0000 7fbbd860f640 -1 bluestore(td/osd-scrub-repair/2) _read_fsid unparsable uuid 2024-09-15T21:53:06.217 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-scrub-repair/2/keyring 2024-09-15T21:53:06.217 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-15T21:53:06.218 INFO:tasks.workunit.client.0.smithi136.stdout:adding osd2 key to auth repository 2024-09-15T21:53:06.219 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd2 key to auth repository 2024-09-15T21:53:06.219 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-scrub-repair/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-15T21:53:06.580 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.2 2024-09-15T21:53:06.580 INFO:tasks.workunit.client.0.smithi136.stdout:start osd.2 2024-09-15T21:53:06.581 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 2 --fsid=ce4892c7-6c6d-4b4f-8e6f-07a3e11133e5 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.52602/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T21:53:06.581 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-15T21:53:06.581 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-15T21:53:06.581 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-15T21:53:06.607 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:53:06.604+0000 7fb741eda640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:53:06.608 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:53:06.606+0000 7fb741eda640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:53:06.610 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:53:06.607+0000 7fb741eda640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:53:06.858 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 2 2024-09-15T21:53:06.858 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-15T21:53:06.858 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=2 2024-09-15T21:53:06.858 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-15T21:53:06.858 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-15T21:53:06.858 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:53:06.859 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-15T21:53:06.859 INFO:tasks.workunit.client.0.smithi136.stdout:0 2024-09-15T21:53:06.859 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:53:06.859 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T21:53:07.138 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:53:08.139 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:53:08.139 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:53:08.140 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-15T21:53:08.140 INFO:tasks.workunit.client.0.smithi136.stdout:1 2024-09-15T21:53:08.140 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:53:08.140 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T21:53:08.229 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:53:08.226+0000 7fb741eda640 -1 Falling back to public interface 2024-09-15T21:53:08.424 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:53:09.425 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:53:09.426 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:53:09.426 INFO:tasks.workunit.client.0.smithi136.stdout:2 2024-09-15T21:53:09.426 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-15T21:53:09.426 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:53:09.427 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T21:53:09.714 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:53:10.716 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:53:10.716 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:53:10.716 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-15T21:53:10.716 INFO:tasks.workunit.client.0.smithi136.stdout:3 2024-09-15T21:53:10.717 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:53:10.717 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T21:53:11.004 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:53:11.282 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:53:11.281+0000 7fb741eda640 -1 osd.2 0 log_to_monitors true 2024-09-15T21:53:12.005 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:53:12.005 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:53:12.005 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-15T21:53:12.006 INFO:tasks.workunit.client.0.smithi136.stdout:4 2024-09-15T21:53:12.006 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:53:12.006 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T21:53:12.378 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:53:13.380 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:53:13.380 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:53:13.380 INFO:tasks.workunit.client.0.smithi136.stdout:5 2024-09-15T21:53:13.381 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-15T21:53:13.381 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:53:13.381 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T21:53:13.669 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:53:14.671 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:53:14.671 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:53:14.671 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-15T21:53:14.671 INFO:tasks.workunit.client.0.smithi136.stdout:6 2024-09-15T21:53:14.672 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:53:14.672 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T21:53:14.998 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:53:15.999 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:53:16.000 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:53:16.000 INFO:tasks.workunit.client.0.smithi136.stdout:7 2024-09-15T21:53:16.000 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-15T21:53:16.000 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:53:16.000 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T21:53:16.346 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:53:17.347 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:53:17.348 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:53:17.348 INFO:tasks.workunit.client.0.smithi136.stdout:8 2024-09-15T21:53:17.348 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-15T21:53:17.348 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:53:17.349 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T21:53:17.638 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:53:18.640 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:53:18.640 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:53:18.640 INFO:tasks.workunit.client.0.smithi136.stdout:9 2024-09-15T21:53:18.641 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-15T21:53:18.641 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:53:18.641 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T21:53:18.989 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:53:19.990 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:53:19.990 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:53:19.990 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-15T21:53:19.991 INFO:tasks.workunit.client.0.smithi136.stdout:10 2024-09-15T21:53:19.991 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:53:19.991 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T21:53:20.363 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:53:21.364 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:53:21.364 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:53:21.365 INFO:tasks.workunit.client.0.smithi136.stdout:11 2024-09-15T21:53:21.365 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-15T21:53:21.365 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:53:21.365 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T21:53:21.652 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:53:22.653 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:53:22.653 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:53:22.654 INFO:tasks.workunit.client.0.smithi136.stdout:12 2024-09-15T21:53:22.655 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-15T21:53:22.655 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:53:22.655 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T21:53:22.972 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:53:23.973 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:53:23.974 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:53:23.974 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-15T21:53:23.974 INFO:tasks.workunit.client.0.smithi136.stdout:13 2024-09-15T21:53:23.974 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:53:23.974 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T21:53:24.481 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:53:25.482 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:53:25.482 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:53:25.482 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-15T21:53:25.482 INFO:tasks.workunit.client.0.smithi136.stdout:14 2024-09-15T21:53:25.483 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:53:25.483 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T21:53:25.836 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:53:26.837 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:53:26.838 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:53:26.838 INFO:tasks.workunit.client.0.smithi136.stdout:15 2024-09-15T21:53:26.838 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-15T21:53:26.838 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:53:26.838 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T21:53:27.323 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:53:28.324 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:53:28.324 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:53:28.324 INFO:tasks.workunit.client.0.smithi136.stdout:16 2024-09-15T21:53:28.325 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-15T21:53:28.325 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:53:28.325 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T21:53:28.912 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:53:29.914 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:53:29.914 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:53:29.914 INFO:tasks.workunit.client.0.smithi136.stdout:17 2024-09-15T21:53:29.914 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-15T21:53:29.915 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:53:29.915 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T21:53:30.529 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:53:31.531 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:53:31.531 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:53:31.531 INFO:tasks.workunit.client.0.smithi136.stdout:18 2024-09-15T21:53:31.532 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 18 2024-09-15T21:53:31.532 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:53:31.533 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T21:53:31.845 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:53:32.846 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:53:32.847 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:53:32.847 INFO:tasks.workunit.client.0.smithi136.stdout:19 2024-09-15T21:53:32.848 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 19 2024-09-15T21:53:32.848 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:53:32.848 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T21:53:33.239 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:53:34.241 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:53:34.241 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:53:34.241 INFO:tasks.workunit.client.0.smithi136.stdout:20 2024-09-15T21:53:34.242 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 20 2024-09-15T21:53:34.242 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:53:34.242 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T21:53:34.527 INFO:tasks.workunit.client.0.smithi136.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/3240590072,v1:127.0.0.1:6819/3240590072] [v2:127.0.0.1:6820/3240590072,v1:127.0.0.1:6821/3240590072] exists,up 5e86a859-3353-49e8-b6c4-5fb38b8b4f05 2024-09-15T21:53:34.532 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-15T21:53:34.532 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-15T21:53:34.532 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-15T21:53:34.532 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:859: unfound_erasure_coded: for id in $(seq 0 3) 2024-09-15T21:53:34.532 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:860: unfound_erasure_coded: run_osd td/osd-scrub-repair 3 2024-09-15T21:53:34.532 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-scrub-repair 2024-09-15T21:53:34.532 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-15T21:53:34.533 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=3 2024-09-15T21:53:34.533 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-15T21:53:34.533 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-scrub-repair/3 2024-09-15T21:53:34.533 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=ce4892c7-6c6d-4b4f-8e6f-07a3e11133e5 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-15T21:53:34.533 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T21:53:34.533 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T21:53:34.533 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T21:53:34.533 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/3' 2024-09-15T21:53:34.533 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/3/journal' 2024-09-15T21:53:34.533 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-15T21:53:34.533 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-15T21:53:34.533 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-15T21:53:34.534 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-15T21:53:34.534 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-15T21:53:34.534 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-15T21:53:34.534 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-15T21:53:34.534 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T21:53:34.534 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52602 2024-09-15T21:53:34.534 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.52602/$cluster-$name.asok' 2024-09-15T21:53:34.534 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.52602/$cluster-$name.asok' 2024-09-15T21:53:34.534 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-15T21:53:34.534 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-15T21:53:34.534 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-15T21:53:34.534 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-15T21:53:34.534 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-15T21:53:34.534 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T21:53:34.535 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T21:53:34.535 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T21:53:34.535 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T21:53:34.535 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-15T21:53:34.535 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-15T21:53:34.535 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-scrub-repair/3 2024-09-15T21:53:34.535 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-15T21:53:34.535 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=f5c50658-f1c3-4781-8e8c-8267a810ab3e 2024-09-15T21:53:34.535 INFO:tasks.workunit.client.0.smithi136.stdout:add osd3 f5c50658-f1c3-4781-8e8c-8267a810ab3e 2024-09-15T21:53:34.536 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd3 f5c50658-f1c3-4781-8e8c-8267a810ab3e' 2024-09-15T21:53:34.536 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-15T21:53:34.550 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQBeV+dmqde0IBAA6JjFNdn4XtZ7p55IWuiIUg== 2024-09-15T21:53:34.550 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQBeV+dmqde0IBAA6JjFNdn4XtZ7p55IWuiIUg=="}' 2024-09-15T21:53:34.550 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new f5c50658-f1c3-4781-8e8c-8267a810ab3e -i td/osd-scrub-repair/3/new.json 2024-09-15T21:53:34.875 INFO:tasks.workunit.client.0.smithi136.stdout:3 2024-09-15T21:53:34.886 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-scrub-repair/3/new.json 2024-09-15T21:53:34.887 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 3 --fsid=ce4892c7-6c6d-4b4f-8e6f-07a3e11133e5 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/3 --osd-journal=td/osd-scrub-repair/3/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.52602/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBeV+dmqde0IBAA6JjFNdn4XtZ7p55IWuiIUg== --osd-uuid f5c50658-f1c3-4781-8e8c-8267a810ab3e 2024-09-15T21:53:34.917 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:53:34.915+0000 7f6b318b4640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:53:34.959 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:53:34.958+0000 7f6b318b4640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:53:34.960 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:53:34.959+0000 7f6b318b4640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:53:34.961 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:53:34.959+0000 7f6b318b4640 -1 bdev(0x5632f498dc00 td/osd-scrub-repair/3/block) open stat got: (1) Operation not permitted 2024-09-15T21:53:34.961 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:53:34.959+0000 7f6b318b4640 -1 bluestore(td/osd-scrub-repair/3) _read_fsid unparsable uuid 2024-09-15T21:53:41.425 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-scrub-repair/3/keyring 2024-09-15T21:53:41.425 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-15T21:53:41.426 INFO:tasks.workunit.client.0.smithi136.stdout:adding osd3 key to auth repository 2024-09-15T21:53:41.426 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd3 key to auth repository 2024-09-15T21:53:41.426 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-scrub-repair/3/keyring auth add osd.3 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-15T21:53:41.789 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.3 2024-09-15T21:53:41.790 INFO:tasks.workunit.client.0.smithi136.stdout:start osd.3 2024-09-15T21:53:41.790 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 3 --fsid=ce4892c7-6c6d-4b4f-8e6f-07a3e11133e5 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/3 --osd-journal=td/osd-scrub-repair/3/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.52602/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T21:53:41.790 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-15T21:53:41.791 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-15T21:53:41.791 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-15T21:53:41.817 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:53:41.815+0000 7f3cefe17640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:53:41.818 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:53:41.817+0000 7f3cefe17640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:53:41.819 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:53:41.818+0000 7f3cefe17640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:53:42.067 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 3 2024-09-15T21:53:42.067 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-15T21:53:42.067 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=3 2024-09-15T21:53:42.067 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-15T21:53:42.067 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-15T21:53:42.067 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:53:42.067 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-15T21:53:42.068 INFO:tasks.workunit.client.0.smithi136.stdout:0 2024-09-15T21:53:42.068 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:53:42.068 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-15T21:53:42.346 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:53:43.347 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:53:43.347 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:53:43.348 INFO:tasks.workunit.client.0.smithi136.stdout:1 2024-09-15T21:53:43.348 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-15T21:53:43.348 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:53:43.348 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-15T21:53:43.424 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:53:43.423+0000 7f3cefe17640 -1 Falling back to public interface 2024-09-15T21:53:43.626 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:53:44.628 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:53:44.628 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:53:44.628 INFO:tasks.workunit.client.0.smithi136.stdout:2 2024-09-15T21:53:44.629 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-15T21:53:44.629 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:53:44.629 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-15T21:53:44.907 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:53:45.909 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:53:45.909 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:53:45.909 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-15T21:53:45.909 INFO:tasks.workunit.client.0.smithi136.stdout:3 2024-09-15T21:53:45.909 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:53:45.910 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-15T21:53:46.068 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:53:46.067+0000 7f3cefe17640 -1 osd.3 0 log_to_monitors true 2024-09-15T21:53:46.276 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:53:47.278 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:53:47.278 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:53:47.278 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-15T21:53:47.278 INFO:tasks.workunit.client.0.smithi136.stdout:4 2024-09-15T21:53:47.278 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:53:47.279 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-15T21:53:47.563 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:53:48.564 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:53:48.565 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:53:48.565 INFO:tasks.workunit.client.0.smithi136.stdout:5 2024-09-15T21:53:48.565 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-15T21:53:48.565 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:53:48.565 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-15T21:53:48.974 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:53:49.975 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:53:49.975 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:53:49.976 INFO:tasks.workunit.client.0.smithi136.stdout:6 2024-09-15T21:53:49.976 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-15T21:53:49.976 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:53:49.976 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-15T21:53:50.265 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:53:51.266 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:53:51.266 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:53:51.266 INFO:tasks.workunit.client.0.smithi136.stdout:7 2024-09-15T21:53:51.267 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-15T21:53:51.267 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:53:51.267 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-15T21:53:51.612 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:53:52.613 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:53:52.613 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:53:52.614 INFO:tasks.workunit.client.0.smithi136.stdout:8 2024-09-15T21:53:52.614 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-15T21:53:52.614 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:53:52.614 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-15T21:53:52.984 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:53:53.985 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:53:53.985 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:53:53.985 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-15T21:53:53.986 INFO:tasks.workunit.client.0.smithi136.stdout:9 2024-09-15T21:53:53.986 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:53:53.986 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-15T21:53:54.270 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:53:55.272 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:53:55.272 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:53:55.272 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-15T21:53:55.272 INFO:tasks.workunit.client.0.smithi136.stdout:10 2024-09-15T21:53:55.272 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:53:55.272 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-15T21:53:55.614 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:53:56.615 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:53:56.616 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:53:56.616 INFO:tasks.workunit.client.0.smithi136.stdout:11 2024-09-15T21:53:56.616 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-15T21:53:56.616 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:53:56.617 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-15T21:53:56.993 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:53:57.995 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:53:57.995 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:53:57.995 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-15T21:53:57.995 INFO:tasks.workunit.client.0.smithi136.stdout:12 2024-09-15T21:53:57.996 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:53:57.996 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-15T21:53:58.284 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:53:59.285 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:53:59.285 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:53:59.285 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-15T21:53:59.286 INFO:tasks.workunit.client.0.smithi136.stdout:13 2024-09-15T21:53:59.286 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:53:59.286 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-15T21:54:00.204 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:54:01.206 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:54:01.206 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:54:01.206 INFO:tasks.workunit.client.0.smithi136.stdout:14 2024-09-15T21:54:01.207 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-15T21:54:01.207 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:54:01.207 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-15T21:54:01.491 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:54:02.492 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:54:02.493 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:54:02.493 INFO:tasks.workunit.client.0.smithi136.stdout:15 2024-09-15T21:54:02.493 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-15T21:54:02.494 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:54:02.494 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-15T21:54:02.953 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:54:03.954 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:54:03.955 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:54:03.955 INFO:tasks.workunit.client.0.smithi136.stdout:16 2024-09-15T21:54:03.955 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-15T21:54:03.955 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:54:03.955 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-15T21:54:04.496 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:54:05.498 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:54:05.498 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:54:05.498 INFO:tasks.workunit.client.0.smithi136.stdout:17 2024-09-15T21:54:05.499 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-15T21:54:05.499 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:54:05.499 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-15T21:54:06.040 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:54:07.041 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:54:07.042 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:54:07.042 INFO:tasks.workunit.client.0.smithi136.stdout:18 2024-09-15T21:54:07.042 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 18 2024-09-15T21:54:07.042 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:54:07.043 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-15T21:54:07.723 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:54:08.724 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:54:08.725 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:54:08.725 INFO:tasks.workunit.client.0.smithi136.stdout:19 2024-09-15T21:54:08.725 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 19 2024-09-15T21:54:08.725 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:54:08.725 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-15T21:54:09.063 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:54:10.064 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:54:10.065 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:54:10.065 INFO:tasks.workunit.client.0.smithi136.stdout:20 2024-09-15T21:54:10.065 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 20 2024-09-15T21:54:10.065 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:54:10.066 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-15T21:54:10.347 INFO:tasks.workunit.client.0.smithi136.stdout:osd.3 up in weight 1 up_from 20 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6826/1685291137,v1:127.0.0.1:6827/1685291137] [v2:127.0.0.1:6828/1685291137,v1:127.0.0.1:6829/1685291137] exists,up f5c50658-f1c3-4781-8e8c-8267a810ab3e 2024-09-15T21:54:10.348 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-15T21:54:10.349 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-15T21:54:10.349 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-15T21:54:10.349 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:863: unfound_erasure_coded: create_ec_pool ecpool false k=2 m=2 2024-09-15T21:54:10.349 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2443: create_ec_pool: local pool_name=ecpool 2024-09-15T21:54:10.349 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2444: create_ec_pool: shift 2024-09-15T21:54:10.349 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2445: create_ec_pool: local allow_overwrites=false 2024-09-15T21:54:10.349 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2446: create_ec_pool: shift 2024-09-15T21:54:10.349 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2448: create_ec_pool: ceph osd erasure-code-profile set myprofile crush-failure-domain=osd k=2 m=2 2024-09-15T21:54:10.743 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2450: create_ec_pool: create_pool ecpool 1 1 erasure myprofile 2024-09-15T21:54:10.743 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create ecpool 1 1 erasure myprofile 2024-09-15T21:54:11.418 INFO:tasks.workunit.client.0.smithi136.stderr:pool 'ecpool' created 2024-09-15T21:54:11.430 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2024-09-15T21:54:12.432 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2452: create_ec_pool: '[' false = true ']' 2024-09-15T21:54:12.432 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2456: create_ec_pool: wait_for_clean 2024-09-15T21:54:12.432 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-15T21:54:12.433 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-15T21:54:12.433 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-15T21:54:12.433 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-15T21:54:12.433 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-15T21:54:12.434 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-15T21:54:12.434 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-15T21:54:12.434 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-15T21:54:12.434 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-15T21:54:12.540 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-15T21:54:12.540 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-15T21:54:12.541 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-15T21:54:12.541 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-15T21:54:12.541 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-15T21:54:12.541 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-15T21:54:12.844 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-15T21:54:12.844 INFO:tasks.workunit.client.0.smithi136.stderr:1 2024-09-15T21:54:12.844 INFO:tasks.workunit.client.0.smithi136.stderr:2 2024-09-15T21:54:12.844 INFO:tasks.workunit.client.0.smithi136.stderr:3' 2024-09-15T21:54:12.844 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-15T21:54:12.844 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T21:54:12.844 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-15T21:54:12.949 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836507 2024-09-15T21:54:12.950 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836507 2024-09-15T21:54:12.950 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836507' 2024-09-15T21:54:12.950 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T21:54:12.950 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-15T21:54:13.053 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=42949672982 2024-09-15T21:54:13.053 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 42949672982 2024-09-15T21:54:13.053 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836507 1-42949672982' 2024-09-15T21:54:13.053 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T21:54:13.053 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-15T21:54:13.160 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=64424509452 2024-09-15T21:54:13.161 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 64424509452 2024-09-15T21:54:13.161 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836507 1-42949672982 2-64424509452' 2024-09-15T21:54:13.161 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T21:54:13.161 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-15T21:54:13.263 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=85899345928 2024-09-15T21:54:13.263 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 85899345928 2024-09-15T21:54:13.263 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836507 1-42949672982 2-64424509452 3-85899345928' 2024-09-15T21:54:13.263 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T21:54:13.264 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836507 2024-09-15T21:54:13.264 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T21:54:13.265 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-15T21:54:13.265 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836507 2024-09-15T21:54:13.265 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T21:54:13.266 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836507 2024-09-15T21:54:13.266 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836507' 2024-09-15T21:54:13.267 INFO:tasks.workunit.client.0.smithi136.stdout:waiting osd.0 seq 21474836507 2024-09-15T21:54:13.267 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T21:54:13.543 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836506 -lt 21474836507 2024-09-15T21:54:13.544 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-15T21:54:14.545 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-15T21:54:14.545 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T21:54:14.828 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836506 -lt 21474836507 2024-09-15T21:54:14.829 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-15T21:54:15.829 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 299 -eq 0 ']' 2024-09-15T21:54:15.830 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T21:54:16.115 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836508 -lt 21474836507 2024-09-15T21:54:16.115 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T21:54:16.115 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-42949672982 2024-09-15T21:54:16.116 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T21:54:16.117 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-15T21:54:16.117 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-42949672982 2024-09-15T21:54:16.117 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T21:54:16.118 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=42949672982 2024-09-15T21:54:16.118 INFO:tasks.workunit.client.0.smithi136.stdout:waiting osd.1 seq 42949672982 2024-09-15T21:54:16.119 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 42949672982' 2024-09-15T21:54:16.119 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-15T21:54:16.445 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 42949672983 -lt 42949672982 2024-09-15T21:54:16.446 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T21:54:16.446 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-64424509452 2024-09-15T21:54:16.446 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T21:54:16.447 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-15T21:54:16.448 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-64424509452 2024-09-15T21:54:16.448 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T21:54:16.449 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=64424509452 2024-09-15T21:54:16.449 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 64424509452' 2024-09-15T21:54:16.449 INFO:tasks.workunit.client.0.smithi136.stdout:waiting osd.2 seq 64424509452 2024-09-15T21:54:16.449 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-15T21:54:16.725 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 64424509452 -lt 64424509452 2024-09-15T21:54:16.725 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T21:54:16.725 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 3-85899345928 2024-09-15T21:54:16.725 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T21:54:16.726 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=3 2024-09-15T21:54:16.727 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 3-85899345928 2024-09-15T21:54:16.727 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T21:54:16.728 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=85899345928 2024-09-15T21:54:16.728 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.3 seq 85899345928' 2024-09-15T21:54:16.728 INFO:tasks.workunit.client.0.smithi136.stdout:waiting osd.3 seq 85899345928 2024-09-15T21:54:16.728 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-15T21:54:17.020 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 85899345929 -lt 85899345928 2024-09-15T21:54:17.020 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-15T21:54:17.020 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T21:54:17.021 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T21:54:17.413 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 1 == 0 2024-09-15T21:54:17.413 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-15T21:54:17.413 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-15T21:54:17.413 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-15T21:54:17.413 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T21:54:17.413 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T21:54:17.413 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T21:54:17.413 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T21:54:17.683 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=1 2024-09-15T21:54:17.684 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-15T21:54:17.684 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T21:54:17.684 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T21:54:18.050 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 = 1 2024-09-15T21:54:18.050 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-15T21:54:18.050 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-15T21:54:18.050 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2457: create_ec_pool: return 0 2024-09-15T21:54:18.050 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:865: unfound_erasure_coded: add_something td/osd-scrub-repair ecpool 2024-09-15T21:54:18.051 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-09-15T21:54:18.051 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=ecpool 2024-09-15T21:54:18.051 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=SOMETHING 2024-09-15T21:54:18.051 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-09-15T21:54:18.051 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-09-15T21:54:18.051 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-09-15T21:54:18.420 INFO:tasks.workunit.client.0.smithi136.stderr:noscrub is set 2024-09-15T21:54:18.431 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-09-15T21:54:18.997 INFO:tasks.workunit.client.0.smithi136.stderr:nodeep-scrub is set 2024-09-15T21:54:19.009 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-09-15T21:54:19.010 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-09-15T21:54:19.010 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool ecpool put SOMETHING td/osd-scrub-repair/ORIGINAL 2024-09-15T21:54:19.244 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:867: unfound_erasure_coded: get_primary ecpool SOMETHING 2024-09-15T21:54:19.244 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1187: get_primary: local poolname=ecpool 2024-09-15T21:54:19.244 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1188: get_primary: local objectname=SOMETHING 2024-09-15T21:54:19.244 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1190: get_primary: ceph --format json osd map ecpool SOMETHING 2024-09-15T21:54:19.245 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1191: get_primary: jq .acting_primary 2024-09-15T21:54:19.603 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:867: unfound_erasure_coded: local primary=1 2024-09-15T21:54:19.603 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:868: unfound_erasure_coded: get_osds ecpool SOMETHING 2024-09-15T21:54:19.604 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:868: unfound_erasure_coded: sed -e s/1// 2024-09-15T21:54:19.604 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1012: get_osds: local poolname=ecpool 2024-09-15T21:54:19.604 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1013: get_osds: local objectname=SOMETHING 2024-09-15T21:54:19.604 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: ceph --format json osd map ecpool SOMETHING 2024-09-15T21:54:19.604 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: jq '.acting | .[]' 2024-09-15T21:54:19.893 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: local 'osds=1 2024-09-15T21:54:19.893 INFO:tasks.workunit.client.0.smithi136.stderr:0 2024-09-15T21:54:19.893 INFO:tasks.workunit.client.0.smithi136.stderr:2 2024-09-15T21:54:19.894 INFO:tasks.workunit.client.0.smithi136.stderr:3' 2024-09-15T21:54:19.894 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1018: get_osds: echo 1 0 2 3 2024-09-15T21:54:19.897 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:868: unfound_erasure_coded: osds=('0' '2' '3') 2024-09-15T21:54:19.897 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:868: unfound_erasure_coded: local -a osds 2024-09-15T21:54:19.897 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:869: unfound_erasure_coded: local not_primary_first=0 2024-09-15T21:54:19.897 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:870: unfound_erasure_coded: local not_primary_second=2 2024-09-15T21:54:19.897 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:871: unfound_erasure_coded: local not_primary_third=3 2024-09-15T21:54:19.897 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:876: unfound_erasure_coded: pids= 2024-09-15T21:54:19.897 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:877: unfound_erasure_coded: run_in_background pids objectstore_tool td/osd-scrub-repair 0 SOMETHING remove 2024-09-15T21:54:19.897 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2135: run_in_background: local pid_variable=pids 2024-09-15T21:54:19.898 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2136: run_in_background: shift 2024-09-15T21:54:19.898 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2140: run_in_background: eval 'pids+=" 446127"' 2024-09-15T21:54:19.898 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2140: run_in_background: pids+=' 446127' 2024-09-15T21:54:19.898 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:878: unfound_erasure_coded: run_in_background pids objectstore_tool td/osd-scrub-repair 2 SOMETHING remove 2024-09-15T21:54:19.898 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2135: run_in_background: local pid_variable=pids 2024-09-15T21:54:19.898 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2136: run_in_background: shift 2024-09-15T21:54:19.898 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2140: run_in_background: eval 'pids+=" 446128"' 2024-09-15T21:54:19.898 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2140: run_in_background: pids+=' 446128' 2024-09-15T21:54:19.898 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:879: unfound_erasure_coded: run_in_background pids objectstore_tool td/osd-scrub-repair 3 SOMETHING remove 2024-09-15T21:54:19.898 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: run_in_background: objectstore_tool td/osd-scrub-repair 0 SOMETHING remove 2024-09-15T21:54:19.898 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2135: run_in_background: local pid_variable=pids 2024-09-15T21:54:19.898 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2136: run_in_background: shift 2024-09-15T21:54:19.898 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: run_in_background: sed 's/^/446130: /' 2024-09-15T21:54:19.899 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2140: run_in_background: eval 'pids+=" 446132"' 2024-09-15T21:54:19.899 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: run_in_background: objectstore_tool td/osd-scrub-repair 2 SOMETHING remove 2024-09-15T21:54:19.899 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2140: run_in_background: pids+=' 446132' 2024-09-15T21:54:19.899 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:880: unfound_erasure_coded: wait_background pids 2024-09-15T21:54:19.899 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: wait_background: pids=' 446127 446128 446132' 2024-09-15T21:54:19.899 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: wait_background: return_code=0 2024-09-15T21:54:19.899 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: wait_background: for pid in $pids 2024-09-15T21:54:19.899 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2175: wait_background: wait 446127 2024-09-15T21:54:19.899 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: run_in_background: sed 's/^/446133: /' 2024-09-15T21:54:19.899 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: run_in_background: objectstore_tool td/osd-scrub-repair 3 SOMETHING remove 2024-09-15T21:54:19.899 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: run_in_background: sed 's/^/446136: /' 2024-09-15T21:54:23.189 INFO:tasks.workunit.client.0.smithi136.stderr:446130: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2024-09-15T21:54:23.190 INFO:tasks.workunit.client.0.smithi136.stderr:446130: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2024-09-15T21:54:23.190 INFO:tasks.workunit.client.0.smithi136.stderr:446130: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=0 2024-09-15T21:54:23.190 INFO:tasks.workunit.client.0.smithi136.stderr:446130: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2024-09-15T21:54:23.190 INFO:tasks.workunit.client.0.smithi136.stderr:446130: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 SOMETHING remove 2024-09-15T21:54:23.190 INFO:tasks.workunit.client.0.smithi136.stderr:446130: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-09-15T21:54:23.190 INFO:tasks.workunit.client.0.smithi136.stderr:446130: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2024-09-15T21:54:23.190 INFO:tasks.workunit.client.0.smithi136.stderr:446130: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=0 2024-09-15T21:54:23.190 INFO:tasks.workunit.client.0.smithi136.stderr:446130: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2024-09-15T21:54:23.190 INFO:tasks.workunit.client.0.smithi136.stderr:446130: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2024-09-15T21:54:23.190 INFO:tasks.workunit.client.0.smithi136.stderr:446130: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-15T21:54:23.190 INFO:tasks.workunit.client.0.smithi136.stderr:446130: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-15T21:54:23.190 INFO:tasks.workunit.client.0.smithi136.stderr:446130: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-15T21:54:23.191 INFO:tasks.workunit.client.0.smithi136.stderr:446130: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-15T21:54:23.191 INFO:tasks.workunit.client.0.smithi136.stderr:446130: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-15T21:54:23.191 INFO:tasks.workunit.client.0.smithi136.stderr:446130: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-15T21:54:23.191 INFO:tasks.workunit.client.0.smithi136.stderr:446130: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 SOMETHING remove 2024-09-15T21:54:23.191 INFO:tasks.workunit.client.0.smithi136.stderr:446130: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-15T21:54:23.191 INFO:tasks.workunit.client.0.smithi136.stderr:446130: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-15T21:54:23.191 INFO:tasks.workunit.client.0.smithi136.stderr:446130: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=0 2024-09-15T21:54:23.191 INFO:tasks.workunit.client.0.smithi136.stderr:446130: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-15T21:54:23.191 INFO:tasks.workunit.client.0.smithi136.stderr:446130: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-09-15T21:54:23.191 INFO:tasks.workunit.client.0.smithi136.stderr:446130: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 SOMETHING remove 2024-09-15T21:54:23.191 INFO:tasks.workunit.client.0.smithi136.stderr:446130: remove 1#1:eb822e21:::SOMETHING:head# 2024-09-15T21:54:23.191 INFO:tasks.workunit.client.0.smithi136.stderr:446130: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 2024-09-15T21:54:23.191 INFO:tasks.workunit.client.0.smithi136.stderr:446130: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2024-09-15T21:54:23.192 INFO:tasks.workunit.client.0.smithi136.stderr:446130: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-15T21:54:23.192 INFO:tasks.workunit.client.0.smithi136.stderr:446130: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=0 2024-09-15T21:54:23.192 INFO:tasks.workunit.client.0.smithi136.stderr:446130: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-15T21:54:23.192 INFO:tasks.workunit.client.0.smithi136.stderr:446130: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-09-15T21:54:23.192 INFO:tasks.workunit.client.0.smithi136.stderr:446130: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=ce4892c7-6c6d-4b4f-8e6f-07a3e11133e5 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-15T21:54:23.192 INFO:tasks.workunit.client.0.smithi136.stderr:446130: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T21:54:23.193 INFO:tasks.workunit.client.0.smithi136.stderr:446130: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T21:54:23.194 INFO:tasks.workunit.client.0.smithi136.stderr:446130: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T21:54:23.194 INFO:tasks.workunit.client.0.smithi136.stderr:446130: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-09-15T21:54:23.194 INFO:tasks.workunit.client.0.smithi136.stderr:446130: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-09-15T21:54:23.194 INFO:tasks.workunit.client.0.smithi136.stderr:446130: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-15T21:54:23.194 INFO:tasks.workunit.client.0.smithi136.stderr:446130: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-15T21:54:23.194 INFO:tasks.workunit.client.0.smithi136.stderr:446130: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-15T21:54:23.194 INFO:tasks.workunit.client.0.smithi136.stderr:446130: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-15T21:54:23.194 INFO:tasks.workunit.client.0.smithi136.stderr:446130: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-15T21:54:23.194 INFO:tasks.workunit.client.0.smithi136.stderr:446130: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-15T21:54:23.194 INFO:tasks.workunit.client.0.smithi136.stderr:446130: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-15T21:54:23.194 INFO:tasks.workunit.client.0.smithi136.stderr:446130: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T21:54:23.194 INFO:tasks.workunit.client.0.smithi136.stderr:446130: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52602 2024-09-15T21:54:23.194 INFO:tasks.workunit.client.0.smithi136.stderr:446130: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.52602/$cluster-$name.asok' 2024-09-15T21:54:23.195 INFO:tasks.workunit.client.0.smithi136.stderr:446130: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.52602/$cluster-$name.asok' 2024-09-15T21:54:23.195 INFO:tasks.workunit.client.0.smithi136.stderr:446130: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-15T21:54:23.195 INFO:tasks.workunit.client.0.smithi136.stderr:446130: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-15T21:54:23.195 INFO:tasks.workunit.client.0.smithi136.stderr:446130: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-15T21:54:23.195 INFO:tasks.workunit.client.0.smithi136.stderr:446130: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T21:54:23.195 INFO:tasks.workunit.client.0.smithi136.stderr:446130: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T21:54:23.195 INFO:tasks.workunit.client.0.smithi136.stderr:446130: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T21:54:23.195 INFO:tasks.workunit.client.0.smithi136.stderr:446130: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T21:54:23.195 INFO:tasks.workunit.client.0.smithi136.stderr:446130: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-15T21:54:23.195 INFO:tasks.workunit.client.0.smithi136.stderr:446130: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-15T21:54:23.195 INFO:tasks.workunit.client.0.smithi136.stderr:446130: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-09-15T21:54:23.195 INFO:tasks.workunit.client.0.smithi136.stderr:446130: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.0 2024-09-15T21:54:23.196 INFO:tasks.workunit.client.0.smithi136.stderr:446130: start osd.0 2024-09-15T21:54:23.196 INFO:tasks.workunit.client.0.smithi136.stderr:446130: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 0 --fsid=ce4892c7-6c6d-4b4f-8e6f-07a3e11133e5 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.52602/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T21:54:23.494 INFO:tasks.workunit.client.0.smithi136.stderr:4446136: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2024-09-15T21:54:23.494 INFO:tasks.workunit.client.0.smithi136.stderr:446136: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2024-09-15T21:54:23.494 INFO:tasks.workunit.client.0.smithi136.stderr:446136: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=3 2024-09-15T21:54:23.494 INFO:tasks.workunit.client.0.smithi136.stderr:446136: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2024-09-15T21:54:23.494 INFO:tasks.workunit.client.0.smithi136.stderr:446136: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 3 SOMETHING remove 2024-09-15T21:54:23.494 INFO:tasks.workunit.client.0.smithi136.stderr:446136: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-09-15T21:54:23.494 INFO:tasks.workunit.client.0.smithi136.stderr:446136: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2024-09-15T21:54:23.494 INFO:tasks.workunit.client.0.smithi136.stderr:446136: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=3 2024-09-15T21:54:23.494 INFO:tasks.workunit.client.0.smithi136.stderr:446136: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2024-09-15T21:54:23.495 INFO:tasks.workunit.client.0.smithi136.stderr:446136: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.3 2024-09-15T21:54:23.495 INFO:tasks.workunit.client.0.smithi136.stderr:446136: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-15T21:54:23.495 INFO:tasks.workunit.client.0.smithi136.stderr:446136: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-15T21:54:23.495 INFO:tasks.workunit.client.0.smithi136.stderr:446136: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-15T21:54:23.495 INFO:tasks.workunit.client.0.smithi136.stderr:446136: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-15T21:54:23.495 INFO:tasks.workunit.client.0.smithi136.stderr:446136: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-15T21:54:23.495 INFO:tasks.workunit.client.0.smithi136.stderr:446136: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-15T21:54:23.495 INFO:tasks.workunit.client.0.smithi136.stderr:446136: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 3 SOMETHING remove 2024-09-15T21:54:23.495 INFO:tasks.workunit.client.0.smithi136.stderr:446136: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-15T21:54:23.495 INFO:tasks.workunit.client.0.smithi136.stderr:446136: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-15T21:54:23.495 INFO:tasks.workunit.client.0.smithi136.stderr:446136: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=3 2024-09-15T21:54:23.496 INFO:tasks.workunit.client.0.smithi136.stderr:446136: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-15T21:54:23.496 INFO:tasks.workunit.client.0.smithi136.stderr:446136: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/3 2024-09-15T21:54:23.496 INFO:tasks.workunit.client.0.smithi136.stderr:446136: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/3 SOMETHING remove 2024-09-15T21:54:23.496 INFO:tasks.workunit.client.0.smithi136.stderr:446136: remove 3#1:eb822e21:::SOMETHING:head# 2024-09-15T21:54:23.496 INFO:tasks.workunit.client.0.smithi136.stderr:446136: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 3 2024-09-15T21:54:23.496 INFO:tasks.workunit.client.0.smithi136.stderr:446136: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2024-09-15T21:54:23.496 INFO:tasks.workunit.client.0.smithi136.stderr:446136: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-15T21:54:23.496 INFO:tasks.workunit.client.0.smithi136.stderr:446136: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=3 2024-09-15T21:54:23.496 INFO:tasks.workunit.client.0.smithi136.stderr:446136: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-15T21:54:23.496 INFO:tasks.workunit.client.0.smithi136.stderr:446136: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/3 2024-09-15T21:54:23.496 INFO:tasks.workunit.client.0.smithi136.stderr:446136: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=ce4892c7-6c6d-4b4f-8e6f-07a3e11133e5 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-15T21:54:23.496 INFO:tasks.workunit.client.0.smithi136.stderr:446136: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T21:54:23.497 INFO:tasks.workunit.client.0.smithi136.stderr:446136: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: 446133: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2024-09-15T21:54:23.497 INFO:tasks.workunit.client.0.smithi136.stderr:446133: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2024-09-15T21:54:23.497 INFO:tasks.workunit.client.0.smithi136.stderr:446133: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=2 2024-09-15T21:54:23.497 INFO:tasks.workunit.client.0.smithi136.stderr:446133: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2024-09-15T21:54:23.497 INFO:tasks.workunit.client.0.smithi136.stderr:446133: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 2 SOMETHING remove 2024-09-15T21:54:23.497 INFO:tasks.workunit.client.0.smithi136.stderr:446133: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-09-15T21:54:23.498 INFO:tasks.workunit.client.0.smithi136.stderr:446133: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2024-09-15T21:54:23.498 INFO:tasks.workunit.client.0.smithi136.stderr:446133: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=2 2024-09-15T21:54:23.498 INFO:tasks.workunit.client.0.smithi136.stderr:446133: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2024-09-15T21:54:23.498 INFO:tasks.workunit.client.0.smithi136.stderr:446133: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.2 2024-09-15T21:54:23.498 INFO:tasks.workunit.client.0.smithi136.stderr:446133: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-15T21:54:23.498 INFO:tasks.workunit.client.0.smithi136.stderr:446133: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-15T21:54:23.498 INFO:tasks.workunit.client.0.smithi136.stderr:446133: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-15T21:54:23.498 INFO:tasks.workunit.client.0.smithi136.stderr:446133: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-15T21:54:23.498 INFO:tasks.workunit.client.0.smithi136.stderr:446133: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-15T21:54:23.498 INFO:tasks.workunit.client.0.smithi136.stderr:446133: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-15T21:54:23.498 INFO:tasks.workunit.client.0.smithi136.stderr:446133: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 2 SOMETHING remove 2024-09-15T21:54:23.498 INFO:tasks.workunit.client.0.smithi136.stderr:446133: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-15T21:54:23.498 INFO:tasks.workunit.client.0.smithi136.stderr:446133: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-15T21:54:23.498 INFO:tasks.workunit.client.0.smithi136.stderr:446133: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=2 2024-09-15T21:54:23.498 INFO:tasks.workunit.client.0.smithi136.stderr:446133: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-15T21:54:23.499 INFO:tasks.workunit.client.0.smithi136.stderr:446133: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/2 2024-09-15T21:54:23.499 INFO:tasks.workunit.client.0.smithi136.stderr:446133: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/2 SOMETHING remove 2024-09-15T21:54:23.499 INFO:tasks.workunit.client.0.smithi136.stderr:446133: remove 2#1:eb822e21:::SOMETHING:head# 2024-09-15T21:54:23.499 INFO:tasks.workunit.client.0.smithi136.stderr:446133: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 2 2024-09-15T21:54:23.499 INFO:tasks.workunit.client.0.smithi136.stderr:446133: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2024-09-15T21:54:23.499 INFO:tasks.workunit.client.0.smithi136.stderr:446133: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-15T21:54:23.499 INFO:tasks.workunit.client.0.smithi136.stderr:446133: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=2 2024-09-15T21:54:23.499 INFO:tasks.workunit.client.0.smithi136.stderr:446133: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-15T21:54:23.499 INFO:tasks.workunit.client.0.smithi136.stderr:446133: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/2 2024-09-15T21:54:23.499 INFO:tasks.workunit.client.0.smithi136.stderr:446133: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=ce4892c7-6c6d-4b4f-8e6f-07a3e11133e5 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-15T21:54:23.499 INFO:tasks.workunit.client.0.smithi136.stderr:446133: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T21:54:23.500 INFO:tasks.workunit.client.0.smithi136.stderr:446133: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T21:54:23.500 INFO:tasks.workunit.client.0.smithi136.stderr:446136: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T21:54:23.500 INFO:tasks.workunit.client.0.smithi136.stderr:446136: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/3' 2024-09-15T21:54:23.500 INFO:tasks.workunit.client.0.smithi136.stderr:446136: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/3/journal' 2024-09-15T21:54:23.500 INFO:tasks.workunit.client.0.smithi136.stderr:446136: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-15T21:54:23.500 INFO:tasks.workunit.client.0.smithi136.stderr:446136: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-15T21:54:23.500 INFO:tasks.workunit.client.0.smithi136.stderr:446136: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-15T21:54:23.500 INFO:tasks.workunit.client.0.smithi136.stderr:446136: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-15T21:54:23.500 INFO:tasks.workunit.client.0.smithi136.stderr:446136: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-15T21:54:23.500 INFO:tasks.workunit.client.0.smithi136.stderr:446136: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-15T21:54:23.500 INFO:tasks.workunit.client.0.smithi136.stderr:446136: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-15T21:54:23.500 INFO:tasks.workunit.client.0.smithi136.stderr:446136: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T21:54:23.500 INFO:tasks.workunit.client.0.smithi136.stderr:446136: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52602 2024-09-15T21:54:23.500 INFO:tasks.workunit.client.0.smithi136.stderr:446136: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.52602/$cluster-$name.asok' 2024-09-15T21:54:23.501 INFO:tasks.workunit.client.0.smithi136.stderr:446136: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.52602/$cluster-$name.asok' 2024-09-15T21:54:23.501 INFO:tasks.workunit.client.0.smithi136.stderr:446136: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-15T21:54:23.501 INFO:tasks.workunit.client.0.smithi136.stderr:446136: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-15T21:54:23.501 INFO:tasks.workunit.client.0.smithi136.stderr:446136: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-15T21:54:23.501 INFO:tasks.workunit.client.0.smithi136.stderr:446136: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T21:54:23.501 INFO:tasks.workunit.client.0.smithi136.stderr:446136: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T21:54:23.501 INFO:tasks.workunit.client.0.smithi136.stderr:446136: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T21:54:23.501 INFO:tasks.workunit.client.0.smithi136.stderr:446136: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T21:54:23.501 INFO:tasks.workunit.client.0.smithi136.stderr:446136: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-15T21:54:23.501 INFO:tasks.workunit.client.0.smithi136.stderr:446136: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-15T21:54:23.501 INFO:tasks.workunit.client.0.smithi136.stderr:446136: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/3 2024-09-15T21:54:23.501 INFO:tasks.workunit.client.0.smithi136.stderr:446136: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.3 2024-09-15T21:54:23.501 INFO:tasks.workunit.client.0.smithi136.stderr:446136: start osd.3 2024-09-15T21:54:23.502 INFO:tasks.workunit.client.0.smithi136.stderr:446136: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 3 --fsid=ce4892c7-6c6d-4b4f-8e6f-07a3e11133e5 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/3 --osd-journal=td/osd-scrub-repair/3/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.52602/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T21:54:23.502 INFO:tasks.workunit.client.0.smithi136.stderr:4activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T21:54:23.502 INFO:tasks.workunit.client.0.smithi136.stderr:446133: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T21:54:23.502 INFO:tasks.workunit.client.0.smithi136.stderr:446133: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/2' 2024-09-15T21:54:23.502 INFO:tasks.workunit.client.0.smithi136.stderr:446133: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/2/journal' 2024-09-15T21:54:23.502 INFO:tasks.workunit.client.0.smithi136.stderr:446133: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-15T21:54:23.502 INFO:tasks.workunit.client.0.smithi136.stderr:446133: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-15T21:54:23.502 INFO:tasks.workunit.client.0.smithi136.stderr:446133: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-15T21:54:23.502 INFO:tasks.workunit.client.0.smithi136.stderr:446133: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-15T21:54:23.502 INFO:tasks.workunit.client.0.smithi136.stderr:446133: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-15T21:54:23.503 INFO:tasks.workunit.client.0.smithi136.stderr:446133: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-15T21:54:23.503 INFO:tasks.workunit.client.0.smithi136.stderr:446133: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-15T21:54:23.503 INFO:tasks.workunit.client.0.smithi136.stderr:446133: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T21:54:23.503 INFO:tasks.workunit.client.0.smithi136.stderr:446133: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52602 2024-09-15T21:54:23.503 INFO:tasks.workunit.client.0.smithi136.stderr:446133: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.52602/$cluster-$name.asok' 2024-09-15T21:54:23.503 INFO:tasks.workunit.client.0.smithi136.stderr:446133: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.52602/$cluster-$name.asok' 2024-09-15T21:54:23.503 INFO:tasks.workunit.client.0.smithi136.stderr:446133: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-15T21:54:23.503 INFO:tasks.workunit.client.0.smithi136.stderr:446133: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-15T21:54:23.503 INFO:tasks.workunit.client.0.smithi136.stderr:446133: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-15T21:54:23.503 INFO:tasks.workunit.client.0.smithi136.stderr:446133: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T21:54:23.503 INFO:tasks.workunit.client.0.smithi136.stderr:446133: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T21:54:23.503 INFO:tasks.workunit.client.0.smithi136.stderr:446133: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T21:54:23.503 INFO:tasks.workunit.client.0.smithi136.stderr:446133: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T21:54:23.503 INFO:tasks.workunit.client.0.smithi136.stderr:446133: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-15T21:54:23.504 INFO:tasks.workunit.client.0.smithi136.stderr:446133: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-15T21:54:23.504 INFO:tasks.workunit.client.0.smithi136.stderr:446133: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/2 2024-09-15T21:54:23.504 INFO:tasks.workunit.client.0.smithi136.stderr:446133: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.2 2024-09-15T21:54:23.504 INFO:tasks.workunit.client.0.smithi136.stderr:446133: start osd.2 2024-09-15T21:54:23.504 INFO:tasks.workunit.client.0.smithi136.stderr:446133: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 2 --fsid=ce4892c7-6c6d-4b4f-8e6f-07a3e11133e5 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.52602/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T21:54:27.896 INFO:tasks.workunit.client.0.smithi136.stderr:446130: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-09-15T21:54:27.896 INFO:tasks.workunit.client.0.smithi136.stderr:446130: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 0 = 0 ']' 2024-09-15T21:54:27.896 INFO:tasks.workunit.client.0.smithi136.stderr:446130: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-15T21:54:27.896 INFO:tasks.workunit.client.0.smithi136.stderr:446130: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-15T21:54:27.896 INFO:tasks.workunit.client.0.smithi136.stderr:446130: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-15T21:54:27.896 INFO:tasks.workunit.client.0.smithi136.stderr:446130: 2024-09-15T21:54:23.223+0000 7fbf12018640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:54:27.896 INFO:tasks.workunit.client.0.smithi136.stderr:446130: 2024-09-15T21:54:23.226+0000 7fbf12018640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:54:27.896 INFO:tasks.workunit.client.0.smithi136.stderr:446130: 2024-09-15T21:54:23.227+0000 7fbf12018640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:54:27.896 INFO:tasks.workunit.client.0.smithi136.stderr:446130: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 0 2024-09-15T21:54:27.896 INFO:tasks.workunit.client.0.smithi136.stderr:446130: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-15T21:54:27.896 INFO:tasks.workunit.client.0.smithi136.stderr:446130: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2024-09-15T21:54:27.897 INFO:tasks.workunit.client.0.smithi136.stderr:446130: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-15T21:54:27.897 INFO:tasks.workunit.client.0.smithi136.stderr:446130: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-15T21:54:27.897 INFO:tasks.workunit.client.0.smithi136.stderr:446130: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:54:27.897 INFO:tasks.workunit.client.0.smithi136.stderr:446130: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-15T21:54:27.897 INFO:tasks.workunit.client.0.smithi136.stderr:446130: 0 2024-09-15T21:54:27.897 INFO:tasks.workunit.client.0.smithi136.stderr:446130: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:54:27.897 INFO:tasks.workunit.client.0.smithi136.stderr:446130: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:54:27.897 INFO:tasks.workunit.client.0.smithi136.stderr:446130: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:54:27.897 INFO:tasks.workunit.client.0.smithi136.stderr:446130: 2024-09-15T21:54:24.579+0000 7fbf12018640 -1 Falling back to public interface 2024-09-15T21:54:27.897 INFO:tasks.workunit.client.0.smithi136.stderr:446130: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:54:27.897 INFO:tasks.workunit.client.0.smithi136.stderr:446130: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:54:27.897 INFO:tasks.workunit.client.0.smithi136.stderr:446130: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-15T21:54:27.897 INFO:tasks.workunit.client.0.smithi136.stderr:446130: 1 2024-09-15T21:54:27.897 INFO:tasks.workunit.client.0.smithi136.stderr:446130: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:54:27.897 INFO:tasks.workunit.client.0.smithi136.stderr:446130: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:54:27.898 INFO:tasks.workunit.client.0.smithi136.stderr:446130: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:54:27.898 INFO:tasks.workunit.client.0.smithi136.stderr:446130: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:54:27.898 INFO:tasks.workunit.client.0.smithi136.stderr:446130: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:54:27.898 INFO:tasks.workunit.client.0.smithi136.stderr:446130: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-15T21:54:27.898 INFO:tasks.workunit.client.0.smithi136.stderr:446130: 2 2024-09-15T21:54:27.898 INFO:tasks.workunit.client.0.smithi136.stderr:446130: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:54:27.898 INFO:tasks.workunit.client.0.smithi136.stderr:446130: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:54:27.898 INFO:tasks.workunit.client.0.smithi136.stderr:446130: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:54:27.898 INFO:tasks.workunit.client.0.smithi136.stderr:446130: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:54:27.898 INFO:tasks.workunit.client.0.smithi136.stderr:446130: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:54:27.898 INFO:tasks.workunit.client.0.smithi136.stderr:446130: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-15T21:54:27.898 INFO:tasks.workunit.client.0.smithi136.stderr:446130: 3 2024-09-15T21:54:27.898 INFO:tasks.workunit.client.0.smithi136.stderr:446130: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:54:27.898 INFO:tasks.workunit.client.0.smithi136.stderr:446130: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:54:27.898 INFO:tasks.workunit.client.0.smithi136.stderr:446130: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:54:28.540 INFO:tasks.workunit.client.0.smithi136.stderr:446130: 2024-09-15T21:54:27.894+0000 7fbf12018640 -46133: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/2/whoami 2024-09-15T21:54:28.540 INFO:tasks.workunit.client.0.smithi136.stderr:446133: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 2 = 2 ']' 2024-09-15T21:54:28.540 INFO:tasks.workunit.client.0.smithi136.stderr:446133: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-15T21:54:28.540 INFO:tasks.workunit.client.0.smithi136.stderr:446133: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-15T21:54:28.540 INFO:tasks.workunit.client.0.smithi136.stderr:446133: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-15T21:54:28.540 INFO:tasks.workunit.client.0.smithi136.stderr:446133: 2024-09-15T21:54:23.545+0000 7fbdecd45640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:54:28.540 INFO:tasks.workunit.client.0.smithi136.stderr:446133: 2024-09-15T21:54:23.550+0000 7fbdecd45640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:54:28.540 INFO:tasks.workunit.client.0.smithi136.stderr:446133: 2024-09-15T21:54:23.551+0000 7fbdecd45640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:54:28.540 INFO:tasks.workunit.client.0.smithi136.stderr:446133: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 2 2024-09-15T21:54:28.540 INFO:tasks.workunit.client.0.smithi136.stderr:446133: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-15T21:54:28.540 INFO:tasks.workunit.client.0.smithi136.stderr:446133: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=2 2024-09-15T21:54:28.540 INFO:tasks.workunit.client.0.smithi136.stderr:446133: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-15T21:54:28.541 INFO:tasks.workunit.client.0.smithi136.stderr:446133: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-15T21:54:28.541 INFO:tasks.workunit.client.0.smithi136.stderr:446133: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:54:28.541 INFO:tasks.workunit.client.0.smithi136.stderr:446133: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-15T21:54:28.541 INFO:tasks.workunit.client.0.smithi136.stderr:446133: 0 2024-09-15T21:54:28.541 INFO:tasks.workunit.client.0.smithi136.stderr:446133: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:54:28.541 INFO:tasks.workunit.client.0.smithi136.stderr:446133: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T21:54:28.541 INFO:tasks.workunit.client.0.smithi136.stderr:446133: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:54:28.541 INFO:tasks.workunit.client.0.smithi136.stderr:446133: 2024-09-15T21:54:25.168+0000 7fbdecd45640 -1 Falling back to public interface 2024-09-15T21:54:28.541 INFO:tasks.workunit.client.0.smithi136.stderr:446133: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:54:28.541 INFO:tasks.workunit.client.0.smithi136.stderr:446133: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:54:28.541 INFO:tasks.workunit.client.0.smithi136.stderr:446133: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-15T21:54:28.541 INFO:tasks.workunit.client.0.smithi136.stderr:446133: 1 2024-09-15T21:54:28.541 INFO:tasks.workunit.client.0.smithi136.stderr:446133: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:54:28.541 INFO:tasks.workunit.client.0.smithi136.stderr:446133: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T21:54:28.542 INFO:tasks.workunit.client.0.smithi136.stderr:446133: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:54:28.542 INFO:tasks.workunit.client.0.smithi136.stderr:446133: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:54:28.542 INFO:tasks.workunit.client.0.smithi136.stderr:446133: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:54:28.542 INFO:tasks.workunit.client.0.smithi136.stderr:446133: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-15T21:54:28.542 INFO:tasks.workunit.client.0.smithi136.stderr:446133: 2 2024-09-15T21:54:28.542 INFO:tasks.workunit.client.0.smithi136.stderr:446133: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:54:28.542 INFO:tasks.workunit.client.0.smithi136.stderr:446133: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T21:54:28.542 INFO:tasks.workunit.client.0.smithi136.stderr:446133: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:54:28.542 INFO:tasks.workunit.client.0.smithi136.stderr:446133: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:54:28.542 INFO:tasks.workunit.client.0.smithi136.stderr:446133: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:54:28.542 INFO:tasks.workunit.client.0.smithi136.stderr:446133: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-15T21:54:28.542 INFO:tasks.workunit.client.0.smithi136.stderr:446133: 3 2024-09-15T21:54:28.542 INFO:tasks.workunit.client.0.smithi136.stderr:446133: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:54:28.542 INFO:tasks.workunit.client.0.smithi136.stderr:446133: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T21:54:28.542 INFO:tasks.workunit.client.0.smithi136.stderr:446133: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:54:28.895 INFO:tasks.workunit.client.0.smithi136.stderr:446133: 2024-09-15T21:54:28.538+0000 7fbdecd45640 -46136: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/3/whoami 2024-09-15T21:54:28.895 INFO:tasks.workunit.client.0.smithi136.stderr:446136: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 3 = 3 ']' 2024-09-15T21:54:28.895 INFO:tasks.workunit.client.0.smithi136.stderr:446136: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-15T21:54:28.895 INFO:tasks.workunit.client.0.smithi136.stderr:446136: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-15T21:54:28.895 INFO:tasks.workunit.client.0.smithi136.stderr:446136: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-15T21:54:28.895 INFO:tasks.workunit.client.0.smithi136.stderr:446136: 2024-09-15T21:54:23.524+0000 7efd81529640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:54:28.895 INFO:tasks.workunit.client.0.smithi136.stderr:446136: 2024-09-15T21:54:23.550+0000 7efd81529640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:54:28.895 INFO:tasks.workunit.client.0.smithi136.stderr:446136: 2024-09-15T21:54:23.552+0000 7efd81529640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:54:28.895 INFO:tasks.workunit.client.0.smithi136.stderr:446136: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 3 2024-09-15T21:54:28.895 INFO:tasks.workunit.client.0.smithi136.stderr:446136: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-15T21:54:28.895 INFO:tasks.workunit.client.0.smithi136.stderr:446136: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=3 2024-09-15T21:54:28.895 INFO:tasks.workunit.client.0.smithi136.stderr:446136: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-15T21:54:28.896 INFO:tasks.workunit.client.0.smithi136.stderr:446136: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-15T21:54:28.896 INFO:tasks.workunit.client.0.smithi136.stderr:446136: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:54:28.896 INFO:tasks.workunit.client.0.smithi136.stderr:446136: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-15T21:54:28.896 INFO:tasks.workunit.client.0.smithi136.stderr:446136: 0 2024-09-15T21:54:28.896 INFO:tasks.workunit.client.0.smithi136.stderr:446136: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:54:28.896 INFO:tasks.workunit.client.0.smithi136.stderr:446136: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-15T21:54:28.896 INFO:tasks.workunit.client.0.smithi136.stderr:446136: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:54:28.896 INFO:tasks.workunit.client.0.smithi136.stderr:446136: 2024-09-15T21:54:25.167+0000 7efd81529640 -1 Falling back to public interface 2024-09-15T21:54:28.896 INFO:tasks.workunit.client.0.smithi136.stderr:446136: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:54:28.896 INFO:tasks.workunit.client.0.smithi136.stderr:446136: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:54:28.896 INFO:tasks.workunit.client.0.smithi136.stderr:446136: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-15T21:54:28.896 INFO:tasks.workunit.client.0.smithi136.stderr:446136: 1 2024-09-15T21:54:28.896 INFO:tasks.workunit.client.0.smithi136.stderr:446136: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:54:28.897 INFO:tasks.workunit.client.0.smithi136.stderr:446136: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-15T21:54:28.897 INFO:tasks.workunit.client.0.smithi136.stderr:446136: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:54:28.897 INFO:tasks.workunit.client.0.smithi136.stderr:446136: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:54:28.897 INFO:tasks.workunit.client.0.smithi136.stderr:446136: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:54:28.897 INFO:tasks.workunit.client.0.smithi136.stderr:446136: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-15T21:54:28.897 INFO:tasks.workunit.client.0.smithi136.stderr:446136: 2 2024-09-15T21:54:28.897 INFO:tasks.workunit.client.0.smithi136.stderr:446136: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:54:28.897 INFO:tasks.workunit.client.0.smithi136.stderr:446136: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-15T21:54:28.897 INFO:tasks.workunit.client.0.smithi136.stderr:446136: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:54:28.897 INFO:tasks.workunit.client.0.smithi136.stderr:446136: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:54:28.897 INFO:tasks.workunit.client.0.smithi136.stderr:446136: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:54:28.897 INFO:tasks.workunit.client.0.smithi136.stderr:446136: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-15T21:54:28.897 INFO:tasks.workunit.client.0.smithi136.stderr:446136: 3 2024-09-15T21:54:28.898 INFO:tasks.workunit.client.0.smithi136.stderr:446136: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:54:28.898 INFO:tasks.workunit.client.0.smithi136.stderr:446136: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-15T21:54:28.898 INFO:tasks.workunit.client.0.smithi136.stderr:446136: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:54:29.568 INFO:tasks.workunit.client.0.smithi136.stderr:446136: 2024-09-15T21:54:28.893+0000 7efd81529640 -1 osd.0 28 log_to_monitors true 2024-09-15T21:54:29.568 INFO:tasks.workunit.client.0.smithi136.stderr:446130: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:54:29.568 INFO:tasks.workunit.client.0.smithi136.stderr:446130: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:54:29.568 INFO:tasks.workunit.client.0.smithi136.stderr:446130: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-15T21:54:29.568 INFO:tasks.workunit.client.0.smithi136.stderr:446130: 4 2024-09-15T21:54:29.568 INFO:tasks.workunit.client.0.smithi136.stderr:446130: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:54:29.568 INFO:tasks.workunit.client.0.smithi136.stderr:446130: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:54:29.568 INFO:tasks.workunit.client.0.smithi136.stderr:446130: osd.0 up in weight 1 up_from 32 up_thru 0 down_at 29 last_clean_interval [5,28) [v2:127.0.0.1:6802/1430276841,v1:127.0.0.1:6803/1430276841] [v2:127.0.0.1:6804/1430276841,v1:127.0.0.1:6805/1430276841] exists,up bbf3cffd-17f5-4af6-95f1-e75ca630ad35 2024-09-15T21:54:29.568 INFO:tasks.workunit.client.0.smithi136.stderr:446130: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-15T21:54:29.568 INFO:tasks.workunit.client.0.smithi136.stderr:446130: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-15T21:54:29.568 INFO:tasks.workunit.client.0.smithi136.stderr:446130: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-15T21:54:29.569 INFO:tasks.workunit.client.0.smithi136.stderr:446130: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2024-09-15T21:54:29.569 INFO:tasks.workunit.client.0.smithi136.stderr:446130: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-15T21:54:29.569 INFO:tasks.workunit.client.0.smithi136.stderr:446130: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-15T21:54:29.569 INFO:tasks.workunit.client.0.smithi136.stderr:446130: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-15T21:54:29.569 INFO:tasks.workunit.client.0.smithi136.stderr:446130: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-15T21:54:29.569 INFO:tasks.workunit.client.0.smithi136.stderr:446130: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-15T21:54:29.569 INFO:tasks.workunit.client.0.smithi136.stderr:446130: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-15T21:54:29.569 INFO:tasks.workunit.client.0.smithi136.stderr:446130: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-15T21:54:29.569 INFO:tasks.workunit.client.0.smithi136.stderr:446130: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-15T21:54:29.569 INFO:tasks.workunit.client.0.smithi136.stderr:446130: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-15T21:54:29.569 INFO:tasks.workunit.client.0.smithi136.stderr:446130: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-15T21:54:29.569 INFO:tasks.workunit.client.0.smithi136.stderr:446130: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-15T21:54:29.569 INFO:tasks.workunit.client.0.smithi136.stderr:446130: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-15T21:54:29.570 INFO:tasks.workunit.client.0.smithi136.stderr:446130: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-15T21:54:29.570 INFO:tasks.workunit.client.0.smithi136.stderr:446130: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-15T21:54:29.570 INFO:tasks.workunit.client.0.smithi136.stderr:446130: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-15T21:54:29.570 INFO:tasks.workunit.client.0.smithi136.stderr:446130: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-15T21:54:29.570 INFO:tasks.workunit.client.0.smithi136.stderr:446130: 1 2024-09-15T21:54:29.570 INFO:tasks.workunit.client.0.smithi136.stderr:446130: 2 2024-09-15T21:54:29.570 INFO:tasks.workunit.client.0.smithi136.stderr:446130: 3' 2024-09-15T21:54:29.570 INFO:tasks.workunit.client.0.smithi136.stderr:446130: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-15T21:54:29.570 INFO:tasks.workunit.client.0.smithi136.stderr:446130: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T21:54:29.570 INFO:tasks.workunit.client.0.smithi136.stderr:446130: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-15T21:54:29.570 INFO:tasks.workunit.client.0.smithi136.stderr:446130: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=137438953474 2024-09-15T21:54:29.570 INFO:tasks.workunit.client.0.smithi136.stderr:446130: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 137438953474 2024-09-15T21:54:29.570 INFO:tasks.workunit.client.0.smithi136.stderr:446130: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-137438953474' 2024-09-15T21:54:29.571 INFO:tasks.workunit.client.0.smithi136.stderr:446130: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T21:54:30.253 INFO:tasks.workunit.client.0.smithi136.stderr:446130: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-hel1 osd.2 28 log_to_monitors true 2024-09-15T21:54:30.253 INFO:tasks.workunit.client.0.smithi136.stderr:446133: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:54:30.253 INFO:tasks.workunit.client.0.smithi136.stderr:446133: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:54:30.253 INFO:tasks.workunit.client.0.smithi136.stderr:446133: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-15T21:54:30.254 INFO:tasks.workunit.client.0.smithi136.stderr:446133: 4 2024-09-15T21:54:30.254 INFO:tasks.workunit.client.0.smithi136.stderr:446133: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T21:54:30.254 INFO:tasks.workunit.client.0.smithi136.stderr:446133: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:54:30.254 INFO:tasks.workunit.client.0.smithi136.stderr:446133: osd.2 up in weight 1 up_from 35 up_thru 0 down_at 29 last_clean_interval [15,28) [v2:127.0.0.1:6822/4075618628,v1:127.0.0.1:6824/4075618628] [v2:127.0.0.1:6826/4075618628,v1:127.0.0.1:6828/4075618628] exists,up 5e86a859-3353-49e8-b6c4-5fb38b8b4f05 2024-09-15T21:54:30.254 INFO:tasks.workunit.client.0.smithi136.stderr:446133: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-15T21:54:30.254 INFO:tasks.workunit.client.0.smithi136.stderr:446133: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-15T21:54:30.254 INFO:tasks.workunit.client.0.smithi136.stderr:446133: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-15T21:54:30.254 INFO:tasks.workunit.client.0.smithi136.stderr:446133: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2024-09-15T21:54:30.254 INFO:tasks.workunit.client.0.smithi136.stderr:446133: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-15T21:54:30.254 INFO:tasks.workunit.client.0.smithi136.stderr:446133: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-15T21:54:30.254 INFO:tasks.workunit.client.0.smithi136.stderr:446133: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-15T21:54:30.254 INFO:tasks.workunit.client.0.smithi136.stderr:446133: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-15T21:54:30.254 INFO:tasks.workunit.client.0.smithi136.stderr:446133: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-15T21:54:30.254 INFO:tasks.workunit.client.0.smithi136.stderr:446133: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-15T21:54:30.255 INFO:tasks.workunit.client.0.smithi136.stderr:446133: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-15T21:54:30.255 INFO:tasks.workunit.client.0.smithi136.stderr:446133: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-15T21:54:30.255 INFO:tasks.workunit.client.0.smithi136.stderr:446133: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-15T21:54:30.255 INFO:tasks.workunit.client.0.smithi136.stderr:446133: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-15T21:54:30.255 INFO:tasks.workunit.client.0.smithi136.stderr:446133: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-15T21:54:30.255 INFO:tasks.workunit.client.0.smithi136.stderr:446133: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-15T21:54:30.255 INFO:tasks.workunit.client.0.smithi136.stderr:446133: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-15T21:54:30.255 INFO:tasks.workunit.client.0.smithi136.stderr:446133: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-15T21:54:30.255 INFO:tasks.workunit.client.0.smithi136.stderr:446133: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-15T21:54:30.255 INFO:tasks.workunit.client.0.smithi136.stderr:446133: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-15T21:54:30.255 INFO:tasks.workunit.client.0.smithi136.stderr:446133: 1 2024-09-15T21:54:30.255 INFO:tasks.workunit.client.0.smithi136.stderr:446133: 2 2024-09-15T21:54:30.255 INFO:tasks.workunit.client.0.smithi136.stderr:446133: 3' 2024-09-15T21:54:30.255 INFO:tasks.workunit.client.0.smithi136.stderr:446133: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-15T21:54:30.255 INFO:tasks.workunit.client.0.smithi136.stderr:446133: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T21:54:30.256 INFO:tasks.workunit.client.0.smithi136.stderr:446133: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-15T21:54:30.256 INFO:tasks.workunit.client.0.smithi136.stderr:446133: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=137438953475 2024-09-15T21:54:30.256 INFO:tasks.workunit.client.0.smithi136.stderr:446133: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 137438953475 2024-09-15T21:54:30.256 INFO:tasks.workunit.client.0.smithi136.stderr:446133: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-137438953475' 2024-09-15T21:54:30.256 INFO:tasks.workunit.client.0.smithi136.stderr:446133: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T21:54:31.268 INFO:tasks.workunit.client.0.smithi136.stderr:446133: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-he1 osd.3 28 log_to_monitors true 2024-09-15T21:54:31.268 INFO:tasks.workunit.client.0.smithi136.stderr:446136: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:54:31.269 INFO:tasks.workunit.client.0.smithi136.stderr:446136: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:54:31.269 INFO:tasks.workunit.client.0.smithi136.stderr:446136: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-15T21:54:31.269 INFO:tasks.workunit.client.0.smithi136.stderr:446136: 4 2024-09-15T21:54:31.269 INFO:tasks.workunit.client.0.smithi136.stderr:446136: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:54:31.269 INFO:tasks.workunit.client.0.smithi136.stderr:446136: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-15T21:54:31.269 INFO:tasks.workunit.client.0.smithi136.stderr:446136: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:54:31.269 INFO:tasks.workunit.client.0.smithi136.stderr:446136: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:54:31.269 INFO:tasks.workunit.client.0.smithi136.stderr:446136: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:54:31.269 INFO:tasks.workunit.client.0.smithi136.stderr:446136: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-15T21:54:31.269 INFO:tasks.workunit.client.0.smithi136.stderr:446136: 5 2024-09-15T21:54:31.269 INFO:tasks.workunit.client.0.smithi136.stderr:446136: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:54:31.269 INFO:tasks.workunit.client.0.smithi136.stderr:446136: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-15T21:54:31.269 INFO:tasks.workunit.client.0.smithi136.stderr:446136: osd.3 up in weight 1 up_from 35 up_thru 0 down_at 29 last_clean_interval [20,28) [v2:127.0.0.1:6818/1920510679,v1:127.0.0.1:6819/1920510679] [v2:127.0.0.1:6820/1920510679,v1:127.0.0.1:6821/1920510679] exists,up f5c50658-f1c3-4781-8e8c-8267a810ab3e 2024-09-15T21:54:31.269 INFO:tasks.workunit.client.0.smithi136.stderr:446136: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-15T21:54:31.270 INFO:tasks.workunit.client.0.smithi136.stderr:446136: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-15T21:54:31.270 INFO:tasks.workunit.client.0.smithi136.stderr:446136: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-15T21:54:31.270 INFO:tasks.workunit.client.0.smithi136.stderr:446136: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2024-09-15T21:54:31.270 INFO:tasks.workunit.client.0.smithi136.stderr:446136: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-15T21:54:31.270 INFO:tasks.workunit.client.0.smithi136.stderr:446136: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-15T21:54:31.270 INFO:tasks.workunit.client.0.smithi136.stderr:446136: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-15T21:54:31.270 INFO:tasks.workunit.client.0.smithi136.stderr:446136: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-15T21:54:31.270 INFO:tasks.workunit.client.0.smithi136.stderr:446136: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-15T21:54:31.270 INFO:tasks.workunit.client.0.smithi136.stderr:446136: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-15T21:54:31.270 INFO:tasks.workunit.client.0.smithi136.stderr:446136: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-15T21:54:31.270 INFO:tasks.workunit.client.0.smithi136.stderr:446136: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-15T21:54:31.270 INFO:tasks.workunit.client.0.smithi136.stderr:446136: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-15T21:54:31.270 INFO:tasks.workunit.client.0.smithi136.stderr:446136: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-15T21:54:31.270 INFO:tasks.workunit.client.0.smithi136.stderr:446136: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-15T21:54:31.271 INFO:tasks.workunit.client.0.smithi136.stderr:446136: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-15T21:54:31.271 INFO:tasks.workunit.client.0.smithi136.stderr:446136: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-15T21:54:31.271 INFO:tasks.workunit.client.0.smithi136.stderr:446136: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-15T21:54:31.271 INFO:tasks.workunit.client.0.smithi136.stderr:446136: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-15T21:54:31.271 INFO:tasks.workunit.client.0.smithi136.stderr:446136: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-15T21:54:31.271 INFO:tasks.workunit.client.0.smithi136.stderr:446136: 1 2024-09-15T21:54:31.271 INFO:tasks.workunit.client.0.smithi136.stderr:446136: 2 2024-09-15T21:54:31.271 INFO:tasks.workunit.client.0.smithi136.stderr:446136: 3' 2024-09-15T21:54:31.271 INFO:tasks.workunit.client.0.smithi136.stderr:446136: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-15T21:54:31.271 INFO:tasks.workunit.client.0.smithi136.stderr:446136: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T21:54:31.302 INFO:tasks.workunit.client.0.smithi136.stderr:446136: //home/ublpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-15T21:54:31.302 INFO:tasks.workunit.client.0.smithi136.stderr:446133: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=42949672988 2024-09-15T21:54:31.303 INFO:tasks.workunit.client.0.smithi136.stderr:446133: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 42949672988 2024-09-15T21:54:31.303 INFO:tasks.workunit.client.0.smithi136.stderr:446133: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-137438953475 1-42949672988' 2024-09-15T21:54:31.303 INFO:tasks.workunit.client.0.smithi136.stderr:446133: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T21:54:31.303 INFO:tasks.workunit.client.0.smithi136.stderr:446133: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-15T21:54:31.303 INFO:tasks.workunit.client.0.smithi136.stderr:446133: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=150323855363 2024-09-15T21:54:31.303 INFO:tasks.workunit.client.0.smithi136.stderr:446133: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 150323855363 2024-09-15T21:54:31.303 INFO:tasks.workunit.client.0.smithi136.stderr:446133: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-137438953475 1-42949672988 2-150323855363' 2024-09-15T21:54:31.303 INFO:tasks.workunit.client.0.smithi136.stderr:446133: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T21:54:31.303 INFO:tasks.workunit.client.0.smithi136.stderr:446133: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-15T21:54:31.303 INFO:tasks.workunit.client.0.smithi136.stderr:446133: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=150323855363 2024-09-15T21:54:31.303 INFO:tasks.workunit.client.0.smithi136.stderr:446133: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 150323855363 2024-09-15T21:54:31.303 INFO:tasks.workunit.client.0.smithi136.stderr:446133: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-137438953475 1-42949672988 2-150323855363 3-150323855363' 2024-09-15T21:54:31.303 INFO:tasks.workunit.client.0.smithi136.stderr:446133: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T21:54:31.304 INFO:tasks.workunit.client.0.smithi136.stderr:446133: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-137438953475 2024-09-15T21:54:31.304 INFO:tasks.workunit.client.0.smithi136.stderr:446133: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T21:54:31.304 INFO:tasks.workunit.client.0.smithi136.stderr:446133: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-15T21:54:31.304 INFO:tasks.workunit.client.0.smithi136.stderr:446133: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-137438953475 2024-09-15T21:54:31.304 INFO:tasks.workunit.client.0.smithi136.stderr:446133: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T21:54:31.304 INFO:tasks.workunit.client.0.smithi136.stderr:446133: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=137438953475 2024-09-15T21:54:31.304 INFO:tasks.workunit.client.0.smithi136.stderr:446133: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 137438953475' 2024-09-15T21:54:31.304 INFO:tasks.workunit.client.0.smithi136.stderr:446133: waiting osd.0 seq 137438953475 2024-09-15T21:54:31.304 INFO:tasks.workunit.client.0.smithi136.stderr:446133: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T21:54:31.304 INFO:tasks.workunit.client.0.smithi136.stderr:446133: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 137438953475 -lt 137438953475 2024-09-15T21:54:31.304 INFO:tasks.workunit.client.0.smithi136.stderr:446133: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T21:54:31.304 INFO:tasks.workunit.client.0.smithi136.stderr:446133: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-42949672988 2024-09-15T21:54:31.304 INFO:tasks.workunit.client.0.smithi136.stderr:446133: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T21:54:31.305 INFO:tasks.workunit.client.0.smithi136.stderr:446133: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-15T21:54:31.305 INFO:tasks.workunit.client.0.smithi136.stderr:446133: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-42949672988 2024-09-15T21:54:31.305 INFO:tasks.workunit.client.0.smithi136.stderr:446133: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T21:54:31.305 INFO:tasks.workunit.client.0.smithi136.stderr:446133: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=42949672988 2024-09-15T21:54:31.305 INFO:tasks.workunit.client.0.smithi136.stderr:446133: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 42949672988' 2024-09-15T21:54:31.305 INFO:tasks.workunit.client.0.smithi136.stderr:446133: waiting osd.1 seq 42949672988 2024-09-15T21:54:31.305 INFO:tasks.workunit.client.0.smithi136.stderr:446133: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-15T21:54:31.770 INFO:tasks.workunit.client.0.smithi136.stderr:446133: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:22pers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-15T21:54:31.770 INFO:tasks.workunit.client.0.smithi136.stderr:446130: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=42949672987 2024-09-15T21:54:31.770 INFO:tasks.workunit.client.0.smithi136.stderr:446130: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 42949672987 2024-09-15T21:54:31.770 INFO:tasks.workunit.client.0.smithi136.stderr:446130: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-137438953474 1-42949672987' 2024-09-15T21:54:31.770 INFO:tasks.workunit.client.0.smithi136.stderr:446130: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T21:54:31.770 INFO:tasks.workunit.client.0.smithi136.stderr:446130: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-15T21:54:31.770 INFO:tasks.workunit.client.0.smithi136.stderr:446130: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=150323855362 2024-09-15T21:54:31.770 INFO:tasks.workunit.client.0.smithi136.stderr:446130: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 150323855362 2024-09-15T21:54:31.770 INFO:tasks.workunit.client.0.smithi136.stderr:446130: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-137438953474 1-42949672987 2-150323855362' 2024-09-15T21:54:31.770 INFO:tasks.workunit.client.0.smithi136.stderr:446130: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T21:54:31.771 INFO:tasks.workunit.client.0.smithi136.stderr:446130: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-15T21:54:31.771 INFO:tasks.workunit.client.0.smithi136.stderr:446130: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=150323855362 2024-09-15T21:54:31.771 INFO:tasks.workunit.client.0.smithi136.stderr:446130: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 150323855362 2024-09-15T21:54:31.771 INFO:tasks.workunit.client.0.smithi136.stderr:446130: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-137438953474 1-42949672987 2-150323855362 3-150323855362' 2024-09-15T21:54:31.771 INFO:tasks.workunit.client.0.smithi136.stderr:446130: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T21:54:31.771 INFO:tasks.workunit.client.0.smithi136.stderr:446130: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-137438953474 2024-09-15T21:54:31.771 INFO:tasks.workunit.client.0.smithi136.stderr:446130: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T21:54:31.771 INFO:tasks.workunit.client.0.smithi136.stderr:446130: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-15T21:54:31.771 INFO:tasks.workunit.client.0.smithi136.stderr:446130: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-137438953474 2024-09-15T21:54:31.771 INFO:tasks.workunit.client.0.smithi136.stderr:446130: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T21:54:31.771 INFO:tasks.workunit.client.0.smithi136.stderr:446130: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=137438953474 2024-09-15T21:54:31.771 INFO:tasks.workunit.client.0.smithi136.stderr:446130: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 137438953474' 2024-09-15T21:54:31.771 INFO:tasks.workunit.client.0.smithi136.stderr:446130: waiting osd.0 seq 137438953474 2024-09-15T21:54:31.771 INFO:tasks.workunit.client.0.smithi136.stderr:446130: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T21:54:31.772 INFO:tasks.workunit.client.0.smithi136.stderr:446130: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 1 -lt 137438953474 2024-09-15T21:54:31.772 INFO:tasks.workunit.client.0.smithi136.stderr:446130: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-15T21:54:31.772 INFO:tasks.workunit.client.0.smithi136.stderr:446130: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-15T21:54:31.772 INFO:tasks.workunit.client.0.smithi136.stderr:446130: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T21:54:31.772 INFO:tasks.workunit.client.0.smithi136.stderr:446130: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 137438953475 -lt 137438953474 2024-09-15T21:54:31.772 INFO:tasks.workunit.client.0.smithi136.stderr:446130: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T21:54:31.772 INFO:tasks.workunit.client.0.smithi136.stderr:446130: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-42949672987 2024-09-15T21:54:31.772 INFO:tasks.workunit.client.0.smithi136.stderr:446130: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T21:54:31.772 INFO:tasks.workunit.client.0.smithi136.stderr:446130: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-15T21:54:31.772 INFO:tasks.workunit.client.0.smithi136.stderr:446130: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-42949672987 2024-09-15T21:54:31.772 INFO:tasks.workunit.client.0.smithi136.stderr:446130: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T21:54:32.614 INFO:tasks.workunit.client.0.smithi136.stderr:446130: /home/ubuntu/22: flush_pg_stats: test 42949672988 -lt 42949672988 2024-09-15T21:54:32.615 INFO:tasks.workunit.client.0.smithi136.stderr:446133: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T21:54:32.615 INFO:tasks.workunit.client.0.smithi136.stderr:446133: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-150323855363 2024-09-15T21:54:32.615 INFO:tasks.workunit.client.0.smithi136.stderr:446133: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T21:54:32.615 INFO:tasks.workunit.client.0.smithi136.stderr:446133: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-15T21:54:32.615 INFO:tasks.workunit.client.0.smithi136.stderr:446133: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-150323855363 2024-09-15T21:54:32.615 INFO:tasks.workunit.client.0.smithi136.stderr:446133: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T21:54:32.615 INFO:tasks.workunit.client.0.smithi136.stderr:446133: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=150323855363 2024-09-15T21:54:32.615 INFO:tasks.workunit.client.0.smithi136.stderr:446133: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 150323855363' 2024-09-15T21:54:32.615 INFO:tasks.workunit.client.0.smithi136.stderr:446133: waiting osd.2 seq 150323855363 2024-09-15T21:54:32.615 INFO:tasks.workunit.client.0.smithi136.stderr:446133: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-15T21:54:32.615 INFO:tasks.workunit.client.0.smithi136.stderr:446133: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 150323855363 -lt 150323855363 2024-09-15T21:54:32.615 INFO:tasks.workunit.client.0.smithi136.stderr:446133: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T21:54:32.616 INFO:tasks.workunit.client.0.smithi136.stderr:446133: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 3-150323855363 2024-09-15T21:54:32.616 INFO:tasks.workunit.client.0.smithi136.stderr:446133: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T21:54:32.616 INFO:tasks.workunit.client.0.smithi136.stderr:446133: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=3 2024-09-15T21:54:32.616 INFO:tasks.workunit.client.0.smithi136.stderr:446133: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 3-150323855363 2024-09-15T21:54:32.616 INFO:tasks.workunit.client.0.smithi136.stderr:446133: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T21:54:32.616 INFO:tasks.workunit.client.0.smithi136.stderr:446133: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=150323855363 2024-09-15T21:54:32.616 INFO:tasks.workunit.client.0.smithi136.stderr:446133: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.3 seq 150323855363' 2024-09-15T21:54:32.616 INFO:tasks.workunit.client.0.smithi136.stderr:446133: waiting osd.3 seq 150323855363 2024-09-15T21:54:32.616 INFO:tasks.workunit.client.0.smithi136.stderr:446133: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-15T21:54:32.616 INFO:tasks.workunit.client.0.smithi136.stderr:446133: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 150323855363 -lt 150323855363 2024-09-15T21:54:32.616 INFO:tasks.workunit.client.0.smithi136.stderr:446133: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-15T21:54:32.616 INFO:tasks.workunit.client.0.smithi136.stderr:446133: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T21:54:32.616 INFO:tasks.workunit.client.0.smithi136.stderr:446133: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T21:54:32.617 INFO:tasks.workunit.client.0.smithi136.stderr:446133: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 1 == 0 2024-09-15T21:54:32.617 INFO:tasks.workunit.client.0.smithi136.stderr:446133: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-15T21:54:32.617 INFO:tasks.workunit.client.0.smithi136.stderr:446133: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-15T21:54:32.617 INFO:tasks.workunit.client.0.smithi136.stderr:446133: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-15T21:54:32.617 INFO:tasks.workunit.client.0.smithi136.stderr:446133: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T21:54:32.617 INFO:tasks.workunit.client.0.smithi136.stderr:446133: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T21:54:32.617 INFO:tasks.workunit.client.0.smithi136.stderr:446133: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T21:54:32.617 INFO:tasks.workunit.client.0.smithi136.stderr:446133: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T21:54:33.065 INFO:tasks.workunit.client.0.smithi136.stderr:446133: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_cleancephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=42949672987 2024-09-15T21:54:33.065 INFO:tasks.workunit.client.0.smithi136.stderr:446130: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 42949672987' 2024-09-15T21:54:33.065 INFO:tasks.workunit.client.0.smithi136.stderr:446130: waiting osd.1 seq 42949672987 2024-09-15T21:54:33.065 INFO:tasks.workunit.client.0.smithi136.stderr:446130: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-15T21:54:33.065 INFO:tasks.workunit.client.0.smithi136.stderr:446130: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 42949672988 -lt 42949672987 2024-09-15T21:54:33.066 INFO:tasks.workunit.client.0.smithi136.stderr:446130: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T21:54:33.066 INFO:tasks.workunit.client.0.smithi136.stderr:446130: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-150323855362 2024-09-15T21:54:33.066 INFO:tasks.workunit.client.0.smithi136.stderr:446130: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T21:54:33.066 INFO:tasks.workunit.client.0.smithi136.stderr:446130: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-15T21:54:33.066 INFO:tasks.workunit.client.0.smithi136.stderr:446130: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-150323855362 2024-09-15T21:54:33.066 INFO:tasks.workunit.client.0.smithi136.stderr:446130: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T21:54:33.066 INFO:tasks.workunit.client.0.smithi136.stderr:446130: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=150323855362 2024-09-15T21:54:33.066 INFO:tasks.workunit.client.0.smithi136.stderr:446130: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 150323855362' 2024-09-15T21:54:33.066 INFO:tasks.workunit.client.0.smithi136.stderr:446130: waiting osd.2 seq 150323855362 2024-09-15T21:54:33.066 INFO:tasks.workunit.client.0.smithi136.stderr:446130: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-15T21:54:33.066 INFO:tasks.workunit.client.0.smithi136.stderr:446130: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 150323855363 -lt 150323855362 2024-09-15T21:54:33.066 INFO:tasks.workunit.client.0.smithi136.stderr:446130: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T21:54:33.067 INFO:tasks.workunit.client.0.smithi136.stderr:446130: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 3-150323855362 2024-09-15T21:54:33.067 INFO:tasks.workunit.client.0.smithi136.stderr:446130: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T21:54:33.067 INFO:tasks.workunit.client.0.smithi136.stderr:446130: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=3 2024-09-15T21:54:33.067 INFO:tasks.workunit.client.0.smithi136.stderr:446130: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 3-150323855362 2024-09-15T21:54:33.067 INFO:tasks.workunit.client.0.smithi136.stderr:446130: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T21:54:33.067 INFO:tasks.workunit.client.0.smithi136.stderr:446130: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=150323855362 2024-09-15T21:54:33.067 INFO:tasks.workunit.client.0.smithi136.stderr:446130: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.3 seq 150323855362' 2024-09-15T21:54:33.067 INFO:tasks.workunit.client.0.smithi136.stderr:446130: waiting osd.3 seq 150323855362 2024-09-15T21:54:33.067 INFO:tasks.workunit.client.0.smithi136.stderr:446130: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-15T21:54:33.067 INFO:tasks.workunit.client.0.smithi136.stderr:446130: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 150323855363 -lt 150323855362 2024-09-15T21:54:33.067 INFO:tasks.workunit.client.0.smithi136.stderr:446130: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-15T21:54:33.067 INFO:tasks.workunit.client.0.smithi136.stderr:446130: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T21:54:33.067 INFO:tasks.workunit.client.0.smithi136.stderr:446130: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T21:54:33.068 INFO:tasks.workunit.client.0.smithi136.stderr:446130: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 1 == 0 2024-09-15T21:54:33.068 INFO:tasks.workunit.client.0.smithi136.stderr:446130: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-15T21:54:33.068 INFO:tasks.workunit.client.0.smithi136.stderr:446130: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-15T21:54:33.068 INFO:tasks.workunit.client.0.smithi136.stderr:446130: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-15T21:54:33.068 INFO:tasks.workunit.client.0.smithi136.stderr:446130: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T21:54:33.409 INFO:tasks.workunit.client.0.smithi136.stderr:446130: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | nuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-15T21:54:33.409 INFO:tasks.workunit.client.0.smithi136.stderr:446136: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=137438953476 2024-09-15T21:54:33.409 INFO:tasks.workunit.client.0.smithi136.stderr:446136: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 137438953476 2024-09-15T21:54:33.409 INFO:tasks.workunit.client.0.smithi136.stderr:446136: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-137438953476' 2024-09-15T21:54:33.409 INFO:tasks.workunit.client.0.smithi136.stderr:446136: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T21:54:33.409 INFO:tasks.workunit.client.0.smithi136.stderr:446136: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-15T21:54:33.409 INFO:tasks.workunit.client.0.smithi136.stderr:446136: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=42949672989 2024-09-15T21:54:33.410 INFO:tasks.workunit.client.0.smithi136.stderr:446136: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 42949672989 2024-09-15T21:54:33.410 INFO:tasks.workunit.client.0.smithi136.stderr:446136: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-137438953476 1-42949672989' 2024-09-15T21:54:33.410 INFO:tasks.workunit.client.0.smithi136.stderr:446136: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T21:54:33.410 INFO:tasks.workunit.client.0.smithi136.stderr:446136: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-15T21:54:33.410 INFO:tasks.workunit.client.0.smithi136.stderr:446136: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=150323855364 2024-09-15T21:54:33.410 INFO:tasks.workunit.client.0.smithi136.stderr:446136: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 150323855364 2024-09-15T21:54:33.410 INFO:tasks.workunit.client.0.smithi136.stderr:446136: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-137438953476 1-42949672989 2-150323855364' 2024-09-15T21:54:33.410 INFO:tasks.workunit.client.0.smithi136.stderr:446136: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T21:54:33.410 INFO:tasks.workunit.client.0.smithi136.stderr:446136: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-15T21:54:33.410 INFO:tasks.workunit.client.0.smithi136.stderr:446136: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=150323855364 2024-09-15T21:54:33.410 INFO:tasks.workunit.client.0.smithi136.stderr:446136: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 150323855364 2024-09-15T21:54:33.410 INFO:tasks.workunit.client.0.smithi136.stderr:446136: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-137438953476 1-42949672989 2-150323855364 3-150323855364' 2024-09-15T21:54:33.410 INFO:tasks.workunit.client.0.smithi136.stderr:446136: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T21:54:33.410 INFO:tasks.workunit.client.0.smithi136.stderr:446136: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-137438953476 2024-09-15T21:54:33.411 INFO:tasks.workunit.client.0.smithi136.stderr:446136: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T21:54:33.411 INFO:tasks.workunit.client.0.smithi136.stderr:446136: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-15T21:54:33.411 INFO:tasks.workunit.client.0.smithi136.stderr:446136: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-137438953476 2024-09-15T21:54:33.411 INFO:tasks.workunit.client.0.smithi136.stderr:446136: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T21:54:33.411 INFO:tasks.workunit.client.0.smithi136.stderr:446136: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=137438953476 2024-09-15T21:54:33.411 INFO:tasks.workunit.client.0.smithi136.stderr:446136: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 137438953476' 2024-09-15T21:54:33.411 INFO:tasks.workunit.client.0.smithi136.stderr:446136: waiting osd.0 seq 137438953476 2024-09-15T21:54:33.411 INFO:tasks.workunit.client.0.smithi136.stderr:446136: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T21:54:33.411 INFO:tasks.workunit.client.0.smithi136.stderr:446136: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 137438953475 -lt 137438953476 2024-09-15T21:54:33.411 INFO:tasks.workunit.client.0.smithi136.stderr:446136: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-15T21:54:33.411 INFO:tasks.workunit.client.0.smithi136.stderr:446136: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-15T21:54:33.411 INFO:tasks.workunit.client.0.smithi136.stderr:446136: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T21:54:33.411 INFO:tasks.workunit.client.0.smithi136.stderr:446136: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 137438953476 -lt 137438953476 2024-09-15T21:54:33.771 INFO:tasks.workunit.client.0.smithi136.stderr:446136: /home/ubuntu/cephtest/clone.cli: cur_active_clean=0 2024-09-15T21:54:33.771 INFO:tasks.workunit.client.0.smithi136.stderr:446133: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-15T21:54:33.771 INFO:tasks.workunit.client.0.smithi136.stderr:446133: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T21:54:33.771 INFO:tasks.workunit.client.0.smithi136.stderr:446133: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T21:54:33.771 INFO:tasks.workunit.client.0.smithi136.stderr:446133: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 0 = 1 2024-09-15T21:54:33.771 INFO:tasks.workunit.client.0.smithi136.stderr:446133: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 0 '!=' -1 2024-09-15T21:54:33.772 INFO:tasks.workunit.client.0.smithi136.stderr:446133: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: loop=0 2024-09-15T21:54:33.772 INFO:tasks.workunit.client.0.smithi136.stderr:446133: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1652: wait_for_clean: num_active_clean=0 2024-09-15T21:54:33.772 INFO:tasks.workunit.client.0.smithi136.stderr:446133: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-15T21:54:33.772 INFO:tasks.workunit.client.0.smithi136.stderr:446133: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 0.1 2024-09-15T21:54:33.772 INFO:tasks.workunit.client.0.smithi136.stderr:446133: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-15T21:54:33.772 INFO:tasks.workunit.client.0.smithi136.stderr:446133: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-15T21:54:33.772 INFO:tasks.workunit.client.0.smithi136.stderr:446133: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-15T21:54:33.772 INFO:tasks.workunit.client.0.smithi136.stderr:446133: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-15T21:54:33.772 INFO:tasks.workunit.client.0.smithi136.stderr:446133: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T21:54:33.772 INFO:tasks.workunit.client.0.smithi136.stderr:446133: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T21:54:33.772 INFO:tasks.workunit.client.0.smithi136.stderr:446133: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T21:54:33.772 INFO:tasks.workunit.client.0.smithi136.stderr:446133: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T21:54:33.772 INFO:tasks.workunit.client.0.smithi136.stderr:446133: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=1 2024-09-15T21:54:33.772 INFO:tasks.workunit.client.0.smithi136.stderr:446133: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-15T21:54:33.773 INFO:tasks.workunit.client.0.smithi136.stderr:446133: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T21:54:33.773 INFO:tasks.workunit.client.0.smithi136.stderr:446133: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T21:54:33.773 INFO:tasks.workunit.client.0.smithi136.stderr:446133: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 = 1 2024-09-15T21:54:33.773 INFO:tasks.workunit.client.0.smithi136.stderr:446133: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-15T21:54:33.773 INFO:tasks.workunit.client.0.smithi136.stderr:446133: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-15T21:54:33.773 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: run_in_background: return 0 2024-09-15T21:54:33.782 INFO:tasks.workunit.client.0.smithi136.stderr:ot)' 2024-09-15T21:54:33.782 INFO:tasks.workunit.client.0.smithi136.stderr:446130: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T21:54:33.782 INFO:tasks.workunit.client.0.smithi136.stderr:446130: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T21:54:33.782 INFO:tasks.workunit.client.0.smithi136.stderr:446130: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=1 2024-09-15T21:54:33.782 INFO:tasks.workunit.client.0.smithi136.stderr:446130: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-15T21:54:33.783 INFO:tasks.workunit.client.0.smithi136.stderr:446130: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T21:54:33.783 INFO:tasks.workunit.client.0.smithi136.stderr:446130: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T21:54:33.783 INFO:tasks.workunit.client.0.smithi136.stderr:446130: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 = 1 2024-09-15T21:54:33.783 INFO:tasks.workunit.client.0.smithi136.stderr:446130: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-15T21:54:33.783 INFO:tasks.workunit.client.0.smithi136.stderr:446130: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-15T21:54:33.783 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: run_in_background: return 0 2024-09-15T21:54:33.783 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: wait_background: for pid in $pids 2024-09-15T21:54:33.783 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2175: wait_background: wait 446128 2024-09-15T21:54:33.783 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: wait_background: for pid in $pids 2024-09-15T21:54:33.783 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2175: wait_background: wait 446132 2024-09-15T21:54:34.705 INFO:tasks.workunit.client.0.smithi136.stderr:ent.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T21:54:34.705 INFO:tasks.workunit.client.0.smithi136.stderr:446136: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-42949672989 2024-09-15T21:54:34.706 INFO:tasks.workunit.client.0.smithi136.stderr:446136: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T21:54:34.706 INFO:tasks.workunit.client.0.smithi136.stderr:446136: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-15T21:54:34.706 INFO:tasks.workunit.client.0.smithi136.stderr:446136: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-42949672989 2024-09-15T21:54:34.706 INFO:tasks.workunit.client.0.smithi136.stderr:446136: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T21:54:34.706 INFO:tasks.workunit.client.0.smithi136.stderr:446136: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=42949672989 2024-09-15T21:54:34.706 INFO:tasks.workunit.client.0.smithi136.stderr:446136: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 42949672989' 2024-09-15T21:54:34.706 INFO:tasks.workunit.client.0.smithi136.stderr:446136: waiting osd.1 seq 42949672989 2024-09-15T21:54:34.706 INFO:tasks.workunit.client.0.smithi136.stderr:446136: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-15T21:54:34.706 INFO:tasks.workunit.client.0.smithi136.stderr:446136: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 42949672989 -lt 42949672989 2024-09-15T21:54:34.706 INFO:tasks.workunit.client.0.smithi136.stderr:446136: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T21:54:34.706 INFO:tasks.workunit.client.0.smithi136.stderr:446136: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-150323855364 2024-09-15T21:54:34.706 INFO:tasks.workunit.client.0.smithi136.stderr:446136: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T21:54:34.706 INFO:tasks.workunit.client.0.smithi136.stderr:446136: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-15T21:54:34.707 INFO:tasks.workunit.client.0.smithi136.stderr:446136: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-150323855364 2024-09-15T21:54:34.707 INFO:tasks.workunit.client.0.smithi136.stderr:446136: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T21:54:34.707 INFO:tasks.workunit.client.0.smithi136.stderr:446136: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=150323855364 2024-09-15T21:54:34.707 INFO:tasks.workunit.client.0.smithi136.stderr:446136: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 150323855364' 2024-09-15T21:54:34.707 INFO:tasks.workunit.client.0.smithi136.stderr:446136: waiting osd.2 seq 150323855364 2024-09-15T21:54:34.707 INFO:tasks.workunit.client.0.smithi136.stderr:446136: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-15T21:54:34.707 INFO:tasks.workunit.client.0.smithi136.stderr:446136: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 150323855364 -lt 150323855364 2024-09-15T21:54:34.707 INFO:tasks.workunit.client.0.smithi136.stderr:446136: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T21:54:34.707 INFO:tasks.workunit.client.0.smithi136.stderr:446136: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 3-150323855364 2024-09-15T21:54:34.707 INFO:tasks.workunit.client.0.smithi136.stderr:446136: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T21:54:34.707 INFO:tasks.workunit.client.0.smithi136.stderr:446136: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=3 2024-09-15T21:54:34.707 INFO:tasks.workunit.client.0.smithi136.stderr:446136: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 3-150323855364 2024-09-15T21:54:34.707 INFO:tasks.workunit.client.0.smithi136.stderr:446136: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T21:54:34.707 INFO:tasks.workunit.client.0.smithi136.stderr:446136: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=150323855364 2024-09-15T21:54:34.708 INFO:tasks.workunit.client.0.smithi136.stderr:446136: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.3 seq 150323855364' 2024-09-15T21:54:34.708 INFO:tasks.workunit.client.0.smithi136.stderr:446136: waiting osd.3 seq 150323855364 2024-09-15T21:54:34.708 INFO:tasks.workunit.client.0.smithi136.stderr:446136: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-15T21:54:34.708 INFO:tasks.workunit.client.0.smithi136.stderr:446136: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 150323855364 -lt 150323855364 2024-09-15T21:54:34.708 INFO:tasks.workunit.client.0.smithi136.stderr:446136: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-15T21:54:34.708 INFO:tasks.workunit.client.0.smithi136.stderr:446136: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T21:54:34.708 INFO:tasks.workunit.client.0.smithi136.stderr:446136: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T21:54:34.708 INFO:tasks.workunit.client.0.smithi136.stderr:446136: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 1 == 0 2024-09-15T21:54:35.327 INFO:tasks.workunit.client.0.smithi136.stderr:446136: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-15T21:54:35.327 INFO:tasks.workunit.client.0.smithi136.stderr:446136: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-15T21:54:35.327 INFO:tasks.workunit.client.0.smithi136.stderr:446136: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-15T21:54:35.327 INFO:tasks.workunit.client.0.smithi136.stderr:446136: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T21:54:35.328 INFO:tasks.workunit.client.0.smithi136.stderr:446136: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T21:54:35.328 INFO:tasks.workunit.client.0.smithi136.stderr:446136: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T21:54:35.328 INFO:tasks.workunit.client.0.smithi136.stderr:446136: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T21:54:35.328 INFO:tasks.workunit.client.0.smithi136.stderr:446136: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=1 2024-09-15T21:54:35.328 INFO:tasks.workunit.client.0.smithi136.stderr:446136: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-15T21:54:35.328 INFO:tasks.workunit.client.0.smithi136.stderr:446136: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T21:54:35.328 INFO:tasks.workunit.client.0.smithi136.stderr:446136: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T21:54:35.328 INFO:tasks.workunit.client.0.smithi136.stderr:446136: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 = 1 2024-09-15T21:54:35.328 INFO:tasks.workunit.client.0.smithi136.stderr:446136: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-15T21:54:35.328 INFO:tasks.workunit.client.0.smithi136.stderr:446136: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-15T21:54:35.328 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: run_in_background: return 0 2024-09-15T21:54:35.328 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: wait_background: eval 'pids='\'''\''' 2024-09-15T21:54:35.328 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: wait_background: pids= 2024-09-15T21:54:35.329 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: wait_background: return 0 2024-09-15T21:54:35.329 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:881: unfound_erasure_coded: return_code=0 2024-09-15T21:54:35.329 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:882: unfound_erasure_coded: '[' 0 -ne 0 ']' 2024-09-15T21:54:35.329 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:887: unfound_erasure_coded: get_pg ecpool SOMETHING 2024-09-15T21:54:35.329 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1084: get_pg: local poolname=ecpool 2024-09-15T21:54:35.330 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1085: get_pg: local objectname=SOMETHING 2024-09-15T21:54:35.330 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1087: get_pg: ceph --format json osd map ecpool SOMETHING 2024-09-15T21:54:35.330 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1087: get_pg: jq -r .pgid 2024-09-15T21:54:35.608 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:887: unfound_erasure_coded: local pg=1.0 2024-09-15T21:54:35.608 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:888: unfound_erasure_coded: repair 1.0 2024-09-15T21:54:35.609 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1846: repair: local pgid=1.0 2024-09-15T21:54:35.609 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1847: repair: get_last_scrub_stamp 1.0 2024-09-15T21:54:35.609 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-15T21:54:35.609 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-15T21:54:35.609 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-15T21:54:35.609 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-15T21:54:35.881 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1847: repair: local last_scrub=2024-09-15T21:54:11.291318+0000 2024-09-15T21:54:35.881 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1848: repair: ceph pg repair 1.0 2024-09-15T21:54:36.144 INFO:tasks.workunit.client.0.smithi136.stderr:instructing pg 1.0s0 on osd.1 to repair 2024-09-15T21:54:36.156 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1849: repair: wait_for_scrub 1.0 2024-09-15T21:54:11.291318+0000 2024-09-15T21:54:36.157 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: local pgid=1.0 2024-09-15T21:54:36.157 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2018: wait_for_scrub: local last_scrub=2024-09-15T21:54:11.291318+0000 2024-09-15T21:54:36.157 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2019: wait_for_scrub: local sname=last_scrub_stamp 2024-09-15T21:54:36.157 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i=0 )) 2024-09-15T21:54:36.157 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-15T21:54:36.157 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-15T21:54:36.157 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-15T21:54:36.157 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-15T21:54:36.157 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-15T21:54:36.158 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-15T21:54:36.428 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-15T21:54:11.291318+0000 '>' 2024-09-15T21:54:11.291318+0000 2024-09-15T21:54:36.429 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-15T21:54:37.430 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-15T21:54:37.430 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-15T21:54:37.430 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-15T21:54:37.431 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-15T21:54:37.431 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-15T21:54:37.431 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-15T21:54:37.431 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-15T21:54:37.710 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-15T21:54:11.291318+0000 '>' 2024-09-15T21:54:11.291318+0000 2024-09-15T21:54:37.710 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-15T21:54:38.711 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-15T21:54:38.711 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-15T21:54:38.712 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-15T21:54:38.712 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-15T21:54:38.712 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-15T21:54:38.712 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-15T21:54:38.713 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-15T21:54:38.995 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-15T21:54:11.291318+0000 '>' 2024-09-15T21:54:11.291318+0000 2024-09-15T21:54:38.995 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-15T21:54:39.995 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-15T21:54:39.996 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-15T21:54:39.996 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-15T21:54:39.996 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-15T21:54:39.996 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-15T21:54:39.997 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-15T21:54:39.997 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-15T21:54:40.279 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-15T21:54:11.291318+0000 '>' 2024-09-15T21:54:11.291318+0000 2024-09-15T21:54:40.280 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-15T21:54:41.281 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-15T21:54:41.281 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-15T21:54:41.282 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-15T21:54:41.282 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-15T21:54:41.282 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-15T21:54:41.282 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-15T21:54:41.283 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-15T21:54:41.568 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-15T21:54:36.188090+0000 '>' 2024-09-15T21:54:11.291318+0000 2024-09-15T21:54:41.568 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2023: wait_for_scrub: return 0 2024-09-15T21:54:41.568 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1850: repair: sleep 2 2024-09-15T21:54:43.570 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:893: unfound_erasure_coded: seq 1 60 2024-09-15T21:54:43.571 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:893: unfound_erasure_coded: for f in `seq 1 60` 2024-09-15T21:54:43.571 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:894: unfound_erasure_coded: ceph -s 2024-09-15T21:54:43.572 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:894: unfound_erasure_coded: grep '1/1 objects unfound' 2024-09-15T21:54:43.923 INFO:tasks.workunit.client.0.smithi136.stdout: 1/1 objects unfound (100.000%) 2024-09-15T21:54:43.923 INFO:tasks.workunit.client.0.smithi136.stdout: 1/1 objects unfound (100.000%) 2024-09-15T21:54:43.923 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:894: unfound_erasure_coded: break 2024-09-15T21:54:43.923 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:897: unfound_erasure_coded: ceph -s 2024-09-15T21:54:43.924 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:897: unfound_erasure_coded: grep '4 up' 2024-09-15T21:54:44.272 INFO:tasks.workunit.client.0.smithi136.stdout: osd: 4 osds: 4 up (since 14s), 4 in (since 69s) 2024-09-15T21:54:44.273 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:898: unfound_erasure_coded: ceph -s 2024-09-15T21:54:44.273 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:898: unfound_erasure_coded: grep '4 in' 2024-09-15T21:54:44.618 INFO:tasks.workunit.client.0.smithi136.stdout: osd: 4 osds: 4 up (since 15s), 4 in (since 69s) 2024-09-15T21:54:44.618 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:899: unfound_erasure_coded: ceph -s 2024-09-15T21:54:44.618 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:899: unfound_erasure_coded: grep '1/1 objects unfound' 2024-09-15T21:54:44.964 INFO:tasks.workunit.client.0.smithi136.stdout: 1/1 objects unfound (100.000%) 2024-09-15T21:54:44.964 INFO:tasks.workunit.client.0.smithi136.stdout: 1/1 objects unfound (100.000%) 2024-09-15T21:54:44.965 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:66: run: teardown td/osd-scrub-repair 2024-09-15T21:54:44.965 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/osd-scrub-repair 2024-09-15T21:54:44.965 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-15T21:54:44.966 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/osd-scrub-repair KILL 2024-09-15T21:54:44.966 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-15T21:54:44.966 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-15T21:54:44.966 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-15T21:54:44.966 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-15T21:54:44.966 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-15T21:54:45.074 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-15T21:54:45.075 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-15T21:54:45.075 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-15T21:54:45.076 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-15T21:54:45.077 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-15T21:54:45.077 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-15T21:54:45.077 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-15T21:54:45.078 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T21:54:45.079 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-15T21:54:45.079 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-15T21:54:45.079 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T21:54:45.080 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-15T21:54:45.081 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-15T21:54:45.081 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/osd-scrub-repair 2024-09-15T21:54:45.101 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-15T21:54:45.101 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T21:54:45.101 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52602 2024-09-15T21:54:45.101 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.52602 2024-09-15T21:54:45.103 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-15T21:54:45.104 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-15T21:54:45.104 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:63: run: for func in $funcs 2024-09-15T21:54:45.104 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:64: run: setup td/osd-scrub-repair 2024-09-15T21:54:45.104 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/osd-scrub-repair 2024-09-15T21:54:45.104 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/osd-scrub-repair 2024-09-15T21:54:45.104 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/osd-scrub-repair 2024-09-15T21:54:45.104 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-15T21:54:45.104 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/osd-scrub-repair KILL 2024-09-15T21:54:45.104 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-15T21:54:45.104 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-15T21:54:45.104 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-15T21:54:45.104 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-15T21:54:45.105 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-15T21:54:45.105 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-15T21:54:45.105 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-15T21:54:45.106 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-15T21:54:45.107 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-15T21:54:45.108 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-15T21:54:45.108 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-15T21:54:45.108 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-15T21:54:45.109 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T21:54:45.109 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-15T21:54:45.110 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-15T21:54:45.110 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T21:54:45.110 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-15T21:54:45.111 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-15T21:54:45.112 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/osd-scrub-repair 2024-09-15T21:54:45.113 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-15T21:54:45.113 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T21:54:45.113 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52602 2024-09-15T21:54:45.113 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.52602 2024-09-15T21:54:45.114 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-15T21:54:45.114 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-15T21:54:45.114 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/osd-scrub-repair 2024-09-15T21:54:45.115 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2024-09-15T21:54:45.115 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T21:54:45.116 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52602 2024-09-15T21:54:45.116 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.52602 2024-09-15T21:54:45.118 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2024-09-15T21:54:45.118 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 4096 -le 1024 ']' 2024-09-15T21:54:45.118 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2024-09-15T21:54:45.118 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/osd-scrub-repair 1' TERM HUP INT 2024-09-15T21:54:45.118 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:65: run: TEST_unfound_erasure_coded_overwrites td/osd-scrub-repair 2024-09-15T21:54:45.118 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:907: TEST_unfound_erasure_coded_overwrites: '[' true = true ']' 2024-09-15T21:54:45.119 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:908: TEST_unfound_erasure_coded_overwrites: unfound_erasure_coded td/osd-scrub-repair true 2024-09-15T21:54:45.119 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:852: unfound_erasure_coded: local dir=td/osd-scrub-repair 2024-09-15T21:54:45.119 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:853: unfound_erasure_coded: local allow_overwrites=true 2024-09-15T21:54:45.119 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:854: unfound_erasure_coded: local poolname=ecpool 2024-09-15T21:54:45.119 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:855: unfound_erasure_coded: local payload=ABCDEF 2024-09-15T21:54:45.119 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:857: unfound_erasure_coded: run_mon td/osd-scrub-repair a 2024-09-15T21:54:45.119 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/osd-scrub-repair 2024-09-15T21:54:45.119 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2024-09-15T21:54:45.119 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2024-09-15T21:54:45.119 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2024-09-15T21:54:45.119 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/osd-scrub-repair/a 2024-09-15T21:54:45.119 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-scrub-repair/a --run-dir=td/osd-scrub-repair 2024-09-15T21:54:45.583 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2024-09-15T21:54:45.583 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-15T21:54:45.583 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-15T21:54:45.583 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-15T21:54:45.583 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T21:54:45.584 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52602 2024-09-15T21:54:45.584 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.52602/$cluster-$name.asok' 2024-09-15T21:54:45.584 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-scrub-repair/a '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.52602/$cluster-$name.asok' --mon-cluster-log-file=td/osd-scrub-repair/log --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-09-15T21:54:45.767 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2024-09-15T21:54:45.770 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2024-09-15T21:54:45.770 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-15T21:54:45.771 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-15T21:54:45.771 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2024-09-15T21:54:45.771 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2024-09-15T21:54:45.771 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-15T21:54:45.771 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-15T21:54:45.771 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-15T21:54:45.772 INFO:tasks.workunit.client.0.smithi136.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-15T21:54:45.772 INFO:tasks.workunit.client.0.smithi136.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T21:54:45.772 INFO:tasks.workunit.client.0.smithi136.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52602 2024-09-15T21:54:45.772 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.52602/ceph-mon.a.asok 2024-09-15T21:54:45.772 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-15T21:54:45.773 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.52602/ceph-mon.a.asok config get fsid 2024-09-15T21:54:45.849 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2024-09-15T21:54:45.850 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-15T21:54:45.850 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-15T21:54:45.850 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2024-09-15T21:54:45.850 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2024-09-15T21:54:45.850 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-15T21:54:45.850 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-15T21:54:45.851 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-15T21:54:45.851 INFO:tasks.workunit.client.0.smithi136.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-15T21:54:45.851 INFO:tasks.workunit.client.0.smithi136.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T21:54:45.851 INFO:tasks.workunit.client.0.smithi136.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52602 2024-09-15T21:54:45.851 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.52602/ceph-mon.a.asok 2024-09-15T21:54:45.851 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-15T21:54:45.851 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.52602/ceph-mon.a.asok config get mon_host 2024-09-15T21:54:45.928 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:858: unfound_erasure_coded: run_mgr td/osd-scrub-repair x 2024-09-15T21:54:45.928 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: run_mgr: local dir=td/osd-scrub-repair 2024-09-15T21:54:45.928 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: run_mgr: shift 2024-09-15T21:54:45.928 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: run_mgr: local id=x 2024-09-15T21:54:45.928 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: run_mgr: shift 2024-09-15T21:54:45.928 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:548: run_mgr: local data=td/osd-scrub-repair/x 2024-09-15T21:54:45.929 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:550: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-15T21:54:46.742 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: get_asok_path 2024-09-15T21:54:46.743 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-15T21:54:46.743 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-15T21:54:46.743 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-15T21:54:46.743 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T21:54:46.743 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52602 2024-09-15T21:54:46.743 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.52602/$cluster-$name.asok' 2024-09-15T21:54:46.743 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-15T21:54:46.744 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-scrub-repair/x '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.52602/$cluster-$name.asok' --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-15T21:54:46.772 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:859: unfound_erasure_coded: seq 0 3 2024-09-15T21:54:46.776 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:859: unfound_erasure_coded: for id in $(seq 0 3) 2024-09-15T21:54:46.776 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:860: unfound_erasure_coded: run_osd td/osd-scrub-repair 0 2024-09-15T21:54:46.776 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-scrub-repair 2024-09-15T21:54:46.776 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-15T21:54:46.776 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=0 2024-09-15T21:54:46.776 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-15T21:54:46.776 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-scrub-repair/0 2024-09-15T21:54:46.776 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=ce4892c7-6c6d-4b4f-8e6f-07a3e11133e5 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-15T21:54:46.777 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T21:54:46.777 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T21:54:46.777 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T21:54:46.777 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-09-15T21:54:46.777 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-09-15T21:54:46.777 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-15T21:54:46.777 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-15T21:54:46.777 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-15T21:54:46.777 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-15T21:54:46.777 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-15T21:54:46.777 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-15T21:54:46.777 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-15T21:54:46.777 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T21:54:46.777 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52602 2024-09-15T21:54:46.778 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.52602/$cluster-$name.asok' 2024-09-15T21:54:46.778 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.52602/$cluster-$name.asok' 2024-09-15T21:54:46.778 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-15T21:54:46.778 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-15T21:54:46.778 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-15T21:54:46.778 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-15T21:54:46.778 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-15T21:54:46.778 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T21:54:46.778 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T21:54:46.778 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T21:54:46.778 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T21:54:46.778 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-15T21:54:46.778 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-15T21:54:46.778 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-scrub-repair/0 2024-09-15T21:54:46.779 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-15T21:54:46.779 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=5e15525f-cf1a-4270-8160-c8c91fa3959b 2024-09-15T21:54:46.779 INFO:tasks.workunit.client.0.smithi136.stdout:add osd0 5e15525f-cf1a-4270-8160-c8c91fa3959b 2024-09-15T21:54:46.780 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd0 5e15525f-cf1a-4270-8160-c8c91fa3959b' 2024-09-15T21:54:46.780 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-15T21:54:46.796 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQCmV+dmaahcLxAA2jB+La7pxuTIoAARd+eVQg== 2024-09-15T21:54:46.796 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQCmV+dmaahcLxAA2jB+La7pxuTIoAARd+eVQg=="}' 2024-09-15T21:54:46.796 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 5e15525f-cf1a-4270-8160-c8c91fa3959b -i td/osd-scrub-repair/0/new.json 2024-09-15T21:54:46.999 INFO:tasks.workunit.client.0.smithi136.stdout:0 2024-09-15T21:54:47.009 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-scrub-repair/0/new.json 2024-09-15T21:54:47.010 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 0 --fsid=ce4892c7-6c6d-4b4f-8e6f-07a3e11133e5 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.52602/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCmV+dmaahcLxAA2jB+La7pxuTIoAARd+eVQg== --osd-uuid 5e15525f-cf1a-4270-8160-c8c91fa3959b 2024-09-15T21:54:47.039 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:54:47.038+0000 7f064f019640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:54:47.049 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:54:47.048+0000 7f064f019640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:54:47.051 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:54:47.049+0000 7f064f019640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:54:47.051 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:54:47.049+0000 7f064f019640 -1 bdev(0x5651ee16fc00 td/osd-scrub-repair/0/block) open stat got: (1) Operation not permitted 2024-09-15T21:54:47.051 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:54:47.050+0000 7f064f019640 -1 bluestore(td/osd-scrub-repair/0) _read_fsid unparsable uuid 2024-09-15T21:54:52.865 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-scrub-repair/0/keyring 2024-09-15T21:54:52.866 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-15T21:54:52.866 INFO:tasks.workunit.client.0.smithi136.stdout:adding osd0 key to auth repository 2024-09-15T21:54:52.867 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd0 key to auth repository 2024-09-15T21:54:52.867 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-scrub-repair/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-15T21:54:53.224 INFO:tasks.workunit.client.0.smithi136.stdout:start osd.0 2024-09-15T21:54:53.225 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.0 2024-09-15T21:54:53.225 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 0 --fsid=ce4892c7-6c6d-4b4f-8e6f-07a3e11133e5 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.52602/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T21:54:53.225 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-15T21:54:53.225 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-15T21:54:53.226 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-15T21:54:53.252 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:54:53.250+0000 7fcb9c54b640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:54:53.253 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:54:53.252+0000 7fcb9c54b640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:54:53.254 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:54:53.253+0000 7fcb9c54b640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:54:53.503 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 0 2024-09-15T21:54:53.503 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-15T21:54:53.503 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2024-09-15T21:54:53.504 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-15T21:54:53.504 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-15T21:54:53.504 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:54:53.504 INFO:tasks.workunit.client.0.smithi136.stdout:0 2024-09-15T21:54:53.504 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-15T21:54:53.505 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:54:53.505 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:54:53.782 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:54:54.369 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:54:54.367+0000 7fcb9c54b640 -1 Falling back to public interface 2024-09-15T21:54:54.784 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:54:54.784 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:54:54.784 INFO:tasks.workunit.client.0.smithi136.stdout:1 2024-09-15T21:54:54.784 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-15T21:54:54.784 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:54:54.785 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:54:55.071 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:54:56.072 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:54:56.073 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:54:56.073 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-15T21:54:56.073 INFO:tasks.workunit.client.0.smithi136.stdout:2 2024-09-15T21:54:56.073 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:54:56.073 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:54:56.363 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:54:56.837 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:54:56.835+0000 7fcb9c54b640 -1 osd.0 0 log_to_monitors true 2024-09-15T21:54:57.364 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:54:57.364 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:54:57.364 INFO:tasks.workunit.client.0.smithi136.stdout:3 2024-09-15T21:54:57.365 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-15T21:54:57.365 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:54:57.365 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:54:57.658 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:54:58.660 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:54:58.660 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:54:58.660 INFO:tasks.workunit.client.0.smithi136.stdout:4 2024-09-15T21:54:58.661 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-15T21:54:58.661 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:54:58.661 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:54:58.949 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:54:59.950 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:54:59.951 INFO:tasks.workunit.client.0.smithi136.stdout:5 2024-09-15T21:54:59.951 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:54:59.951 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-15T21:54:59.951 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:54:59.952 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:55:00.240 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:55:01.242 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:55:01.242 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:55:01.242 INFO:tasks.workunit.client.0.smithi136.stdout:6 2024-09-15T21:55:01.243 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-15T21:55:01.243 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:55:01.243 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:55:01.610 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:55:02.612 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:55:02.612 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:55:02.612 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-15T21:55:02.612 INFO:tasks.workunit.client.0.smithi136.stdout:7 2024-09-15T21:55:02.613 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:55:02.613 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:55:02.898 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:55:03.900 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:55:03.900 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:55:03.900 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-15T21:55:03.900 INFO:tasks.workunit.client.0.smithi136.stdout:8 2024-09-15T21:55:03.901 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:55:03.901 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:55:04.178 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:55:05.180 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:55:05.180 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:55:05.180 INFO:tasks.workunit.client.0.smithi136.stdout:9 2024-09-15T21:55:05.181 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-15T21:55:05.181 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:55:05.181 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:55:05.469 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:55:06.470 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:55:06.471 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:55:06.471 INFO:tasks.workunit.client.0.smithi136.stdout:10 2024-09-15T21:55:06.471 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-15T21:55:06.471 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:55:06.471 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:55:06.804 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:55:07.805 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:55:07.805 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:55:07.805 INFO:tasks.workunit.client.0.smithi136.stdout:11 2024-09-15T21:55:07.806 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-15T21:55:07.806 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:55:07.806 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:55:08.094 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:55:09.096 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:55:09.096 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:55:09.096 INFO:tasks.workunit.client.0.smithi136.stdout:12 2024-09-15T21:55:09.097 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-15T21:55:09.097 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:55:09.097 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:55:09.495 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:55:10.496 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:55:10.496 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:55:10.496 INFO:tasks.workunit.client.0.smithi136.stdout:13 2024-09-15T21:55:10.497 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-15T21:55:10.497 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:55:10.497 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:55:11.238 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:55:12.239 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:55:12.240 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:55:12.240 INFO:tasks.workunit.client.0.smithi136.stdout:14 2024-09-15T21:55:12.240 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-15T21:55:12.240 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:55:12.241 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:55:12.770 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:55:13.772 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:55:13.772 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:55:13.772 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-15T21:55:13.772 INFO:tasks.workunit.client.0.smithi136.stdout:15 2024-09-15T21:55:13.773 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:55:13.773 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:55:14.317 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:55:15.319 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:55:15.319 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:55:15.319 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-15T21:55:15.319 INFO:tasks.workunit.client.0.smithi136.stdout:16 2024-09-15T21:55:15.320 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:55:15.320 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:55:16.092 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:55:17.094 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:55:17.094 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:55:17.094 INFO:tasks.workunit.client.0.smithi136.stdout:17 2024-09-15T21:55:17.094 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-15T21:55:17.095 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:55:17.095 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:55:17.862 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:55:18.864 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:55:18.864 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:55:18.864 INFO:tasks.workunit.client.0.smithi136.stdout:18 2024-09-15T21:55:18.865 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 18 2024-09-15T21:55:18.865 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:55:18.865 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:55:19.151 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:55:20.153 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:55:20.153 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:55:20.153 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 19 2024-09-15T21:55:20.153 INFO:tasks.workunit.client.0.smithi136.stdout:19 2024-09-15T21:55:20.154 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:55:20.154 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:55:20.480 INFO:tasks.workunit.client.0.smithi136.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/4260459695,v1:127.0.0.1:6803/4260459695] [v2:127.0.0.1:6804/4260459695,v1:127.0.0.1:6805/4260459695] exists,up 5e15525f-cf1a-4270-8160-c8c91fa3959b 2024-09-15T21:55:20.483 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-15T21:55:20.483 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-15T21:55:20.483 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-15T21:55:20.483 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:859: unfound_erasure_coded: for id in $(seq 0 3) 2024-09-15T21:55:20.484 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:860: unfound_erasure_coded: run_osd td/osd-scrub-repair 1 2024-09-15T21:55:20.484 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-scrub-repair 2024-09-15T21:55:20.484 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-15T21:55:20.484 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=1 2024-09-15T21:55:20.484 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-15T21:55:20.484 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-scrub-repair/1 2024-09-15T21:55:20.484 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=ce4892c7-6c6d-4b4f-8e6f-07a3e11133e5 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-15T21:55:20.484 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T21:55:20.484 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T21:55:20.484 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T21:55:20.484 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-09-15T21:55:20.484 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-09-15T21:55:20.484 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-15T21:55:20.485 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-15T21:55:20.485 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-15T21:55:20.485 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-15T21:55:20.485 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-15T21:55:20.485 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-15T21:55:20.485 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-15T21:55:20.485 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T21:55:20.485 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52602 2024-09-15T21:55:20.485 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.52602/$cluster-$name.asok' 2024-09-15T21:55:20.485 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.52602/$cluster-$name.asok' 2024-09-15T21:55:20.485 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-15T21:55:20.485 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-15T21:55:20.485 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-15T21:55:20.485 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-15T21:55:20.486 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-15T21:55:20.486 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T21:55:20.486 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T21:55:20.486 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T21:55:20.486 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T21:55:20.486 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-15T21:55:20.486 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-15T21:55:20.486 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-scrub-repair/1 2024-09-15T21:55:20.486 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-15T21:55:20.486 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=c38ee691-ade5-4386-920b-c7487d771627 2024-09-15T21:55:20.487 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd1 c38ee691-ade5-4386-920b-c7487d771627' 2024-09-15T21:55:20.487 INFO:tasks.workunit.client.0.smithi136.stdout:add osd1 c38ee691-ade5-4386-920b-c7487d771627 2024-09-15T21:55:20.487 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-15T21:55:20.503 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQDIV+dmBAjjHRAAJbVG14IfNjqSo7JXqBkxcg== 2024-09-15T21:55:20.503 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQDIV+dmBAjjHRAAJbVG14IfNjqSo7JXqBkxcg=="}' 2024-09-15T21:55:20.503 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new c38ee691-ade5-4386-920b-c7487d771627 -i td/osd-scrub-repair/1/new.json 2024-09-15T21:55:20.857 INFO:tasks.workunit.client.0.smithi136.stdout:1 2024-09-15T21:55:20.869 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-scrub-repair/1/new.json 2024-09-15T21:55:20.870 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 1 --fsid=ce4892c7-6c6d-4b4f-8e6f-07a3e11133e5 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.52602/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDIV+dmBAjjHRAAJbVG14IfNjqSo7JXqBkxcg== --osd-uuid c38ee691-ade5-4386-920b-c7487d771627 2024-09-15T21:55:20.899 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:55:20.897+0000 7f63d1444640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:55:20.942 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:55:20.940+0000 7f63d1444640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:55:20.943 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:55:20.941+0000 7f63d1444640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:55:20.943 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:55:20.942+0000 7f63d1444640 -1 bdev(0x5622444a3c00 td/osd-scrub-repair/1/block) open stat got: (1) Operation not permitted 2024-09-15T21:55:20.943 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:55:20.942+0000 7f63d1444640 -1 bluestore(td/osd-scrub-repair/1) _read_fsid unparsable uuid 2024-09-15T21:55:26.576 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-scrub-repair/1/keyring 2024-09-15T21:55:26.576 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-15T21:55:26.577 INFO:tasks.workunit.client.0.smithi136.stdout:adding osd1 key to auth repository 2024-09-15T21:55:26.577 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd1 key to auth repository 2024-09-15T21:55:26.578 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-scrub-repair/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-15T21:55:26.980 INFO:tasks.workunit.client.0.smithi136.stdout:start osd.1 2024-09-15T21:55:26.981 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.1 2024-09-15T21:55:26.981 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-15T21:55:26.981 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 1 --fsid=ce4892c7-6c6d-4b4f-8e6f-07a3e11133e5 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.52602/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T21:55:26.981 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-15T21:55:26.981 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-15T21:55:27.008 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:55:27.005+0000 7f4a14c18640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:55:27.010 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:55:27.007+0000 7f4a14c18640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:55:27.011 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:55:27.008+0000 7f4a14c18640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:55:27.260 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 1 2024-09-15T21:55:27.260 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-15T21:55:27.260 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2024-09-15T21:55:27.260 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-15T21:55:27.260 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-15T21:55:27.261 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:55:27.261 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-15T21:55:27.261 INFO:tasks.workunit.client.0.smithi136.stdout:0 2024-09-15T21:55:27.261 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:55:27.261 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T21:55:27.539 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:55:28.361 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:55:28.359+0000 7f4a14c18640 -1 Falling back to public interface 2024-09-15T21:55:28.540 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:55:28.541 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:55:28.541 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-15T21:55:28.541 INFO:tasks.workunit.client.0.smithi136.stdout:1 2024-09-15T21:55:28.541 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:55:28.541 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T21:55:28.827 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:55:29.829 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:55:29.829 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:55:29.829 INFO:tasks.workunit.client.0.smithi136.stdout:2 2024-09-15T21:55:29.829 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-15T21:55:29.829 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:55:29.830 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T21:55:30.107 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:55:31.109 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:55:31.109 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:55:31.109 INFO:tasks.workunit.client.0.smithi136.stdout:3 2024-09-15T21:55:31.110 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-15T21:55:31.110 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:55:31.110 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T21:55:31.352 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:55:31.350+0000 7f4a14c18640 -1 osd.1 0 log_to_monitors true 2024-09-15T21:55:31.400 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:55:32.401 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:55:32.401 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:55:32.402 INFO:tasks.workunit.client.0.smithi136.stdout:4 2024-09-15T21:55:32.402 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-15T21:55:32.402 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:55:32.402 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T21:55:32.687 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:55:33.689 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:55:33.689 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:55:33.689 INFO:tasks.workunit.client.0.smithi136.stdout:5 2024-09-15T21:55:33.690 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-15T21:55:33.690 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:55:33.690 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T21:55:34.002 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:55:35.004 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:55:35.004 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:55:35.004 INFO:tasks.workunit.client.0.smithi136.stdout:6 2024-09-15T21:55:35.005 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-15T21:55:35.005 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:55:35.005 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T21:55:35.351 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:55:36.352 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:55:36.352 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:55:36.353 INFO:tasks.workunit.client.0.smithi136.stdout:7 2024-09-15T21:55:36.357 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-15T21:55:36.357 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:55:36.357 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T21:55:36.638 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:55:37.640 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:55:37.640 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:55:37.640 INFO:tasks.workunit.client.0.smithi136.stdout:8 2024-09-15T21:55:37.641 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-15T21:55:37.641 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:55:37.641 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T21:55:38.002 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:55:39.003 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:55:39.004 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:55:39.004 INFO:tasks.workunit.client.0.smithi136.stdout:9 2024-09-15T21:55:39.004 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-15T21:55:39.004 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:55:39.004 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T21:55:39.368 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:55:40.369 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:55:40.369 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:55:40.369 INFO:tasks.workunit.client.0.smithi136.stdout:10 2024-09-15T21:55:40.370 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-15T21:55:40.370 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:55:40.370 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T21:55:40.657 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:55:41.658 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:55:41.659 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:55:41.659 INFO:tasks.workunit.client.0.smithi136.stdout:11 2024-09-15T21:55:41.659 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-15T21:55:41.659 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:55:41.660 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T21:55:41.991 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:55:42.993 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:55:42.993 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:55:42.993 INFO:tasks.workunit.client.0.smithi136.stdout:12 2024-09-15T21:55:42.994 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-15T21:55:42.994 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:55:42.994 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T21:55:43.781 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:55:44.782 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:55:44.783 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:55:44.783 INFO:tasks.workunit.client.0.smithi136.stdout:13 2024-09-15T21:55:44.783 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-15T21:55:44.783 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:55:44.783 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T21:55:45.540 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:55:46.541 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:55:46.541 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:55:46.541 INFO:tasks.workunit.client.0.smithi136.stdout:14 2024-09-15T21:55:46.542 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-15T21:55:46.542 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:55:46.542 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T21:55:46.941 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:55:47.943 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:55:47.943 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:55:47.943 INFO:tasks.workunit.client.0.smithi136.stdout:15 2024-09-15T21:55:47.944 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-15T21:55:47.944 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:55:47.944 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T21:55:48.587 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:55:49.588 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:55:49.588 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:55:49.589 INFO:tasks.workunit.client.0.smithi136.stdout:16 2024-09-15T21:55:49.589 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-15T21:55:49.589 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:55:49.589 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T21:55:50.230 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:55:51.232 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:55:51.232 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:55:51.232 INFO:tasks.workunit.client.0.smithi136.stdout:17 2024-09-15T21:55:51.233 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-15T21:55:51.233 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:55:51.233 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T21:55:51.689 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:55:52.690 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:55:52.690 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:55:52.691 INFO:tasks.workunit.client.0.smithi136.stdout:18 2024-09-15T21:55:52.691 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 18 2024-09-15T21:55:52.691 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:55:52.691 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T21:55:53.117 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:55:54.119 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:55:54.119 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:55:54.119 INFO:tasks.workunit.client.0.smithi136.stdout:19 2024-09-15T21:55:54.120 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 19 2024-09-15T21:55:54.120 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:55:54.120 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T21:55:54.407 INFO:tasks.workunit.client.0.smithi136.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/333825686,v1:127.0.0.1:6811/333825686] [v2:127.0.0.1:6812/333825686,v1:127.0.0.1:6813/333825686] exists,up c38ee691-ade5-4386-920b-c7487d771627 2024-09-15T21:55:54.411 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-15T21:55:54.411 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-15T21:55:54.411 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-15T21:55:54.411 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:859: unfound_erasure_coded: for id in $(seq 0 3) 2024-09-15T21:55:54.411 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:860: unfound_erasure_coded: run_osd td/osd-scrub-repair 2 2024-09-15T21:55:54.411 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-scrub-repair 2024-09-15T21:55:54.411 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-15T21:55:54.411 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=2 2024-09-15T21:55:54.411 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-15T21:55:54.411 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-scrub-repair/2 2024-09-15T21:55:54.411 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=ce4892c7-6c6d-4b4f-8e6f-07a3e11133e5 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-15T21:55:54.411 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T21:55:54.411 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T21:55:54.412 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T21:55:54.412 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/2' 2024-09-15T21:55:54.412 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/2/journal' 2024-09-15T21:55:54.412 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-15T21:55:54.412 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-15T21:55:54.412 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-15T21:55:54.412 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-15T21:55:54.412 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-15T21:55:54.412 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-15T21:55:54.412 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-15T21:55:54.412 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T21:55:54.412 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52602 2024-09-15T21:55:54.412 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.52602/$cluster-$name.asok' 2024-09-15T21:55:54.413 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.52602/$cluster-$name.asok' 2024-09-15T21:55:54.413 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-15T21:55:54.413 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-15T21:55:54.413 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-15T21:55:54.413 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-15T21:55:54.413 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-15T21:55:54.413 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T21:55:54.413 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T21:55:54.413 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T21:55:54.413 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T21:55:54.413 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-15T21:55:54.413 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-15T21:55:54.413 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-scrub-repair/2 2024-09-15T21:55:54.413 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-15T21:55:54.414 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=1a14e877-c54e-41ad-9f45-41e1f568ebc8 2024-09-15T21:55:54.414 INFO:tasks.workunit.client.0.smithi136.stdout:add osd2 1a14e877-c54e-41ad-9f45-41e1f568ebc8 2024-09-15T21:55:54.414 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd2 1a14e877-c54e-41ad-9f45-41e1f568ebc8' 2024-09-15T21:55:54.415 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-15T21:55:54.430 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQDqV+dmBXOKGRAAZQb+tFYURhLwgN7wmPXWkw== 2024-09-15T21:55:54.430 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQDqV+dmBXOKGRAAZQb+tFYURhLwgN7wmPXWkw=="}' 2024-09-15T21:55:54.430 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 1a14e877-c54e-41ad-9f45-41e1f568ebc8 -i td/osd-scrub-repair/2/new.json 2024-09-15T21:55:54.893 INFO:tasks.workunit.client.0.smithi136.stdout:2 2024-09-15T21:55:54.905 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-scrub-repair/2/new.json 2024-09-15T21:55:54.906 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 2 --fsid=ce4892c7-6c6d-4b4f-8e6f-07a3e11133e5 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.52602/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDqV+dmBXOKGRAAZQb+tFYURhLwgN7wmPXWkw== --osd-uuid 1a14e877-c54e-41ad-9f45-41e1f568ebc8 2024-09-15T21:55:54.935 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:55:54.933+0000 7f4c51813640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:55:54.986 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:55:54.984+0000 7f4c51813640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:55:54.987 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:55:54.985+0000 7f4c51813640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:55:54.988 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:55:54.986+0000 7f4c51813640 -1 bdev(0x5625d38adc00 td/osd-scrub-repair/2/block) open stat got: (1) Operation not permitted 2024-09-15T21:55:54.988 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:55:54.986+0000 7f4c51813640 -1 bluestore(td/osd-scrub-repair/2) _read_fsid unparsable uuid 2024-09-15T21:56:00.654 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-scrub-repair/2/keyring 2024-09-15T21:56:00.654 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-15T21:56:00.655 INFO:tasks.workunit.client.0.smithi136.stdout:adding osd2 key to auth repository 2024-09-15T21:56:00.655 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd2 key to auth repository 2024-09-15T21:56:00.655 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-scrub-repair/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-15T21:56:01.022 INFO:tasks.workunit.client.0.smithi136.stdout:start osd.2 2024-09-15T21:56:01.022 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.2 2024-09-15T21:56:01.022 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-15T21:56:01.023 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 2 --fsid=ce4892c7-6c6d-4b4f-8e6f-07a3e11133e5 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.52602/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T21:56:01.023 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-15T21:56:01.023 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-15T21:56:01.048 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:56:01.047+0000 7fe91840b640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:56:01.063 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:56:01.062+0000 7fe91840b640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:56:01.064 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:56:01.063+0000 7fe91840b640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:56:01.329 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 2 2024-09-15T21:56:01.329 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-15T21:56:01.329 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=2 2024-09-15T21:56:01.330 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-15T21:56:01.330 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-15T21:56:01.330 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:56:01.330 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-15T21:56:01.330 INFO:tasks.workunit.client.0.smithi136.stdout:0 2024-09-15T21:56:01.330 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:56:01.331 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T21:56:01.605 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:56:02.430 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:56:02.428+0000 7fe91840b640 -1 Falling back to public interface 2024-09-15T21:56:02.606 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:56:02.606 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:56:02.607 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-15T21:56:02.607 INFO:tasks.workunit.client.0.smithi136.stdout:1 2024-09-15T21:56:02.607 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:56:02.607 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T21:56:02.918 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:56:03.919 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:56:03.919 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:56:03.919 INFO:tasks.workunit.client.0.smithi136.stdout:2 2024-09-15T21:56:03.920 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-15T21:56:03.920 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:56:03.920 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T21:56:04.201 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:56:05.203 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:56:05.203 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:56:05.203 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-15T21:56:05.203 INFO:tasks.workunit.client.0.smithi136.stdout:3 2024-09-15T21:56:05.204 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:56:05.204 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T21:56:05.481 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:56:05.606 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:56:05.605+0000 7fe91840b640 -1 osd.2 0 log_to_monitors true 2024-09-15T21:56:06.482 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:56:06.482 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:56:06.482 INFO:tasks.workunit.client.0.smithi136.stdout:4 2024-09-15T21:56:06.483 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-15T21:56:06.483 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:56:06.483 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T21:56:06.859 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:56:07.860 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:56:07.861 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:56:07.861 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-15T21:56:07.861 INFO:tasks.workunit.client.0.smithi136.stdout:5 2024-09-15T21:56:07.861 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:56:07.861 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T21:56:08.150 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:56:09.152 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:56:09.152 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:56:09.152 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-15T21:56:09.152 INFO:tasks.workunit.client.0.smithi136.stdout:6 2024-09-15T21:56:09.153 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:56:09.153 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T21:56:09.437 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:56:10.438 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:56:10.438 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:56:10.439 INFO:tasks.workunit.client.0.smithi136.stdout:7 2024-09-15T21:56:10.439 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-15T21:56:10.439 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:56:10.439 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T21:56:10.726 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:56:11.728 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:56:11.728 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:56:11.728 INFO:tasks.workunit.client.0.smithi136.stdout:8 2024-09-15T21:56:11.729 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-15T21:56:11.729 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:56:11.729 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T21:56:12.015 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:56:13.016 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:56:13.016 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:56:13.017 INFO:tasks.workunit.client.0.smithi136.stdout:9 2024-09-15T21:56:13.017 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-15T21:56:13.017 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:56:13.017 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T21:56:13.369 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:56:14.371 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:56:14.371 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:56:14.371 INFO:tasks.workunit.client.0.smithi136.stdout:10 2024-09-15T21:56:14.372 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-15T21:56:14.372 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:56:14.372 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T21:56:14.667 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:56:15.668 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:56:15.668 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:56:15.669 INFO:tasks.workunit.client.0.smithi136.stdout:11 2024-09-15T21:56:15.669 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-15T21:56:15.669 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:56:15.669 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T21:56:15.993 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:56:16.994 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:56:16.995 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:56:16.995 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-15T21:56:16.995 INFO:tasks.workunit.client.0.smithi136.stdout:12 2024-09-15T21:56:16.995 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:56:16.995 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T21:56:17.509 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:56:18.511 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:56:18.511 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:56:18.511 INFO:tasks.workunit.client.0.smithi136.stdout:13 2024-09-15T21:56:18.512 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-15T21:56:18.512 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:56:18.512 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T21:56:19.144 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:56:20.145 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:56:20.146 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:56:20.146 INFO:tasks.workunit.client.0.smithi136.stdout:14 2024-09-15T21:56:20.146 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-15T21:56:20.146 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:56:20.146 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T21:56:20.861 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:56:21.862 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:56:21.862 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:56:21.863 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-15T21:56:21.863 INFO:tasks.workunit.client.0.smithi136.stdout:15 2024-09-15T21:56:21.863 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:56:21.863 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T21:56:22.559 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:56:23.560 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:56:23.561 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:56:23.561 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-15T21:56:23.561 INFO:tasks.workunit.client.0.smithi136.stdout:16 2024-09-15T21:56:23.561 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:56:23.561 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T21:56:24.162 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:56:25.163 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:56:25.164 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:56:25.164 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-15T21:56:25.164 INFO:tasks.workunit.client.0.smithi136.stdout:17 2024-09-15T21:56:25.164 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:56:25.164 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T21:56:25.579 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:56:26.581 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:56:26.581 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:56:26.581 INFO:tasks.workunit.client.0.smithi136.stdout:18 2024-09-15T21:56:26.582 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 18 2024-09-15T21:56:26.582 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:56:26.582 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T21:56:27.048 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:56:28.049 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:56:28.049 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:56:28.050 INFO:tasks.workunit.client.0.smithi136.stdout:19 2024-09-15T21:56:28.050 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 19 2024-09-15T21:56:28.050 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:56:28.050 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T21:56:28.623 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:56:29.624 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:56:29.624 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:56:29.625 INFO:tasks.workunit.client.0.smithi136.stdout:20 2024-09-15T21:56:29.625 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 20 2024-09-15T21:56:29.625 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:56:29.625 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T21:56:29.909 INFO:tasks.workunit.client.0.smithi136.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/2292515345,v1:127.0.0.1:6819/2292515345] [v2:127.0.0.1:6820/2292515345,v1:127.0.0.1:6821/2292515345] exists,up 1a14e877-c54e-41ad-9f45-41e1f568ebc8 2024-09-15T21:56:29.913 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-15T21:56:29.913 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-15T21:56:29.913 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-15T21:56:29.913 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:859: unfound_erasure_coded: for id in $(seq 0 3) 2024-09-15T21:56:29.913 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:860: unfound_erasure_coded: run_osd td/osd-scrub-repair 3 2024-09-15T21:56:29.913 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-scrub-repair 2024-09-15T21:56:29.914 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-15T21:56:29.914 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=3 2024-09-15T21:56:29.914 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-15T21:56:29.914 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-scrub-repair/3 2024-09-15T21:56:29.914 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=ce4892c7-6c6d-4b4f-8e6f-07a3e11133e5 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-15T21:56:29.914 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T21:56:29.914 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T21:56:29.914 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T21:56:29.914 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/3' 2024-09-15T21:56:29.915 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/3/journal' 2024-09-15T21:56:29.915 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-15T21:56:29.915 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-15T21:56:29.915 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-15T21:56:29.915 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-15T21:56:29.915 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-15T21:56:29.915 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-15T21:56:29.915 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-15T21:56:29.915 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T21:56:29.915 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52602 2024-09-15T21:56:29.915 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.52602/$cluster-$name.asok' 2024-09-15T21:56:29.915 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.52602/$cluster-$name.asok' 2024-09-15T21:56:29.915 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-15T21:56:29.916 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-15T21:56:29.916 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-15T21:56:29.916 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-15T21:56:29.916 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-15T21:56:29.916 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T21:56:29.916 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T21:56:29.916 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T21:56:29.916 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T21:56:29.916 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-15T21:56:29.916 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-15T21:56:29.916 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-scrub-repair/3 2024-09-15T21:56:29.916 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-15T21:56:29.916 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=e738d0f4-c28b-4426-bb0d-1cb17652a4d3 2024-09-15T21:56:29.916 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd3 e738d0f4-c28b-4426-bb0d-1cb17652a4d3' 2024-09-15T21:56:29.917 INFO:tasks.workunit.client.0.smithi136.stdout:add osd3 e738d0f4-c28b-4426-bb0d-1cb17652a4d3 2024-09-15T21:56:29.917 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-15T21:56:29.932 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQANWOdm3HZ3NxAAUTrbR4sVsAfB3cqEG3VNSQ== 2024-09-15T21:56:29.933 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQANWOdm3HZ3NxAAUTrbR4sVsAfB3cqEG3VNSQ=="}' 2024-09-15T21:56:29.933 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new e738d0f4-c28b-4426-bb0d-1cb17652a4d3 -i td/osd-scrub-repair/3/new.json 2024-09-15T21:56:30.287 INFO:tasks.workunit.client.0.smithi136.stdout:3 2024-09-15T21:56:30.298 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-scrub-repair/3/new.json 2024-09-15T21:56:30.299 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 3 --fsid=ce4892c7-6c6d-4b4f-8e6f-07a3e11133e5 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/3 --osd-journal=td/osd-scrub-repair/3/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.52602/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQANWOdm3HZ3NxAAUTrbR4sVsAfB3cqEG3VNSQ== --osd-uuid e738d0f4-c28b-4426-bb0d-1cb17652a4d3 2024-09-15T21:56:30.328 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:56:30.327+0000 7f5eb45c1640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:56:30.372 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:56:30.370+0000 7f5eb45c1640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:56:30.372 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:56:30.371+0000 7f5eb45c1640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:56:30.373 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:56:30.371+0000 7f5eb45c1640 -1 bdev(0x560303b91c00 td/osd-scrub-repair/3/block) open stat got: (1) Operation not permitted 2024-09-15T21:56:30.373 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:56:30.371+0000 7f5eb45c1640 -1 bluestore(td/osd-scrub-repair/3) _read_fsid unparsable uuid 2024-09-15T21:56:36.821 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-scrub-repair/3/keyring 2024-09-15T21:56:36.821 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-15T21:56:36.822 INFO:tasks.workunit.client.0.smithi136.stdout:adding osd3 key to auth repository 2024-09-15T21:56:36.822 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd3 key to auth repository 2024-09-15T21:56:36.822 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-scrub-repair/3/keyring auth add osd.3 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-15T21:56:37.197 INFO:tasks.workunit.client.0.smithi136.stdout:start osd.3 2024-09-15T21:56:37.198 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.3 2024-09-15T21:56:37.198 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 3 --fsid=ce4892c7-6c6d-4b4f-8e6f-07a3e11133e5 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/3 --osd-journal=td/osd-scrub-repair/3/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.52602/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T21:56:37.198 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-15T21:56:37.198 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-15T21:56:37.198 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-15T21:56:37.224 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:56:37.223+0000 7f170a81a640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:56:37.226 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:56:37.225+0000 7f170a81a640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:56:37.227 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:56:37.226+0000 7f170a81a640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:56:37.472 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 3 2024-09-15T21:56:37.472 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-15T21:56:37.472 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=3 2024-09-15T21:56:37.472 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-15T21:56:37.473 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-15T21:56:37.473 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:56:37.473 INFO:tasks.workunit.client.0.smithi136.stdout:0 2024-09-15T21:56:37.473 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-15T21:56:37.474 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:56:37.474 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-15T21:56:37.749 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:56:38.578 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:56:38.576+0000 7f170a81a640 -1 Falling back to public interface 2024-09-15T21:56:38.751 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:56:38.751 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:56:38.751 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-15T21:56:38.751 INFO:tasks.workunit.client.0.smithi136.stdout:1 2024-09-15T21:56:38.752 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:56:38.752 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-15T21:56:39.061 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:56:40.062 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:56:40.062 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:56:40.063 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-15T21:56:40.063 INFO:tasks.workunit.client.0.smithi136.stdout:2 2024-09-15T21:56:40.064 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:56:40.064 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-15T21:56:40.352 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:56:41.350 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:56:41.348+0000 7f170a81a640 -1 osd.3 0 log_to_monitors true 2024-09-15T21:56:41.354 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:56:41.354 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:56:41.354 INFO:tasks.workunit.client.0.smithi136.stdout:3 2024-09-15T21:56:41.354 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-15T21:56:41.355 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:56:41.355 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-15T21:56:41.665 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:56:42.667 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:56:42.667 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:56:42.667 INFO:tasks.workunit.client.0.smithi136.stdout:4 2024-09-15T21:56:42.668 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-15T21:56:42.668 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:56:42.668 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-15T21:56:43.012 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:56:44.013 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:56:44.013 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:56:44.014 INFO:tasks.workunit.client.0.smithi136.stdout:5 2024-09-15T21:56:44.014 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-15T21:56:44.014 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:56:44.014 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-15T21:56:44.372 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:56:45.373 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:56:45.373 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:56:45.374 INFO:tasks.workunit.client.0.smithi136.stdout:6 2024-09-15T21:56:45.374 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-15T21:56:45.374 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:56:45.374 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-15T21:56:45.659 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:56:46.661 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:56:46.661 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:56:46.661 INFO:tasks.workunit.client.0.smithi136.stdout:7 2024-09-15T21:56:46.662 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-15T21:56:46.662 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:56:46.662 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-15T21:56:46.998 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:56:47.999 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:56:47.999 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:56:48.000 INFO:tasks.workunit.client.0.smithi136.stdout:8 2024-09-15T21:56:48.000 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-15T21:56:48.000 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:56:48.000 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-15T21:56:48.373 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:56:49.375 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:56:49.375 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:56:49.375 INFO:tasks.workunit.client.0.smithi136.stdout:9 2024-09-15T21:56:49.376 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-15T21:56:49.376 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:56:49.376 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-15T21:56:49.660 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:56:50.662 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:56:50.662 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:56:50.662 INFO:tasks.workunit.client.0.smithi136.stdout:10 2024-09-15T21:56:50.662 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-15T21:56:50.663 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:56:50.663 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-15T21:56:51.024 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:56:52.025 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:56:52.026 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:56:52.026 INFO:tasks.workunit.client.0.smithi136.stdout:11 2024-09-15T21:56:52.026 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-15T21:56:52.026 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:56:52.026 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-15T21:56:52.364 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:56:53.365 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:56:53.365 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:56:53.365 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-15T21:56:53.366 INFO:tasks.workunit.client.0.smithi136.stdout:12 2024-09-15T21:56:53.366 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:56:53.366 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-15T21:56:53.650 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:56:54.651 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:56:54.651 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:56:54.651 INFO:tasks.workunit.client.0.smithi136.stdout:13 2024-09-15T21:56:54.652 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-15T21:56:54.652 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:56:54.652 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-15T21:56:55.410 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:56:56.412 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:56:56.412 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:56:56.412 INFO:tasks.workunit.client.0.smithi136.stdout:14 2024-09-15T21:56:56.412 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-15T21:56:56.413 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:56:56.413 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-15T21:56:57.047 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:56:58.048 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:56:58.049 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:56:58.049 INFO:tasks.workunit.client.0.smithi136.stdout:15 2024-09-15T21:56:58.049 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-15T21:56:58.049 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:56:58.050 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-15T21:56:58.786 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:56:59.787 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:56:59.787 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:56:59.787 INFO:tasks.workunit.client.0.smithi136.stdout:16 2024-09-15T21:56:59.788 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-15T21:56:59.788 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:56:59.788 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-15T21:57:00.092 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:57:01.093 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:57:01.093 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:57:01.094 INFO:tasks.workunit.client.0.smithi136.stdout:17 2024-09-15T21:57:01.094 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-15T21:57:01.094 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:57:01.094 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-15T21:57:01.428 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:57:02.429 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:57:02.429 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:57:02.430 INFO:tasks.workunit.client.0.smithi136.stdout:18 2024-09-15T21:57:02.430 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 18 2024-09-15T21:57:02.430 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:57:02.430 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-15T21:57:03.071 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:57:04.073 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:57:04.073 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:57:04.073 INFO:tasks.workunit.client.0.smithi136.stdout:19 2024-09-15T21:57:04.074 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 19 2024-09-15T21:57:04.074 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:57:04.074 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-15T21:57:04.412 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:57:05.413 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:57:05.413 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:57:05.414 INFO:tasks.workunit.client.0.smithi136.stdout:20 2024-09-15T21:57:05.414 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 20 2024-09-15T21:57:05.414 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:57:05.414 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-15T21:57:05.727 INFO:tasks.workunit.client.0.smithi136.stdout:osd.3 up in weight 1 up_from 20 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6826/2286875162,v1:127.0.0.1:6827/2286875162] [v2:127.0.0.1:6828/2286875162,v1:127.0.0.1:6829/2286875162] exists,up e738d0f4-c28b-4426-bb0d-1cb17652a4d3 2024-09-15T21:57:05.728 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-15T21:57:05.728 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-15T21:57:05.728 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-15T21:57:05.728 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:863: unfound_erasure_coded: create_ec_pool ecpool true k=2 m=2 2024-09-15T21:57:05.728 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2443: create_ec_pool: local pool_name=ecpool 2024-09-15T21:57:05.728 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2444: create_ec_pool: shift 2024-09-15T21:57:05.728 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2445: create_ec_pool: local allow_overwrites=true 2024-09-15T21:57:05.728 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2446: create_ec_pool: shift 2024-09-15T21:57:05.728 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2448: create_ec_pool: ceph osd erasure-code-profile set myprofile crush-failure-domain=osd k=2 m=2 2024-09-15T21:57:06.113 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2450: create_ec_pool: create_pool ecpool 1 1 erasure myprofile 2024-09-15T21:57:06.114 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create ecpool 1 1 erasure myprofile 2024-09-15T21:57:06.739 INFO:tasks.workunit.client.0.smithi136.stderr:pool 'ecpool' created 2024-09-15T21:57:06.751 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2024-09-15T21:57:07.752 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2452: create_ec_pool: '[' true = true ']' 2024-09-15T21:57:07.753 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2453: create_ec_pool: ceph osd pool set ecpool allow_ec_overwrites true 2024-09-15T21:57:08.204 INFO:tasks.workunit.client.0.smithi136.stderr:set pool 1 allow_ec_overwrites to true 2024-09-15T21:57:08.216 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2456: create_ec_pool: wait_for_clean 2024-09-15T21:57:08.216 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-15T21:57:08.216 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-15T21:57:08.216 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-15T21:57:08.217 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-15T21:57:08.217 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-15T21:57:08.217 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-15T21:57:08.217 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-15T21:57:08.217 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-15T21:57:08.217 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-15T21:57:08.311 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-15T21:57:08.311 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-15T21:57:08.311 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-15T21:57:08.311 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-15T21:57:08.312 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-15T21:57:08.312 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-15T21:57:08.635 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-15T21:57:08.635 INFO:tasks.workunit.client.0.smithi136.stderr:1 2024-09-15T21:57:08.635 INFO:tasks.workunit.client.0.smithi136.stderr:2 2024-09-15T21:57:08.635 INFO:tasks.workunit.client.0.smithi136.stderr:3' 2024-09-15T21:57:08.635 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-15T21:57:08.635 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T21:57:08.636 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-15T21:57:08.740 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836510 2024-09-15T21:57:08.740 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836510 2024-09-15T21:57:08.740 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836510' 2024-09-15T21:57:08.740 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T21:57:08.740 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-15T21:57:08.867 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=42949672983 2024-09-15T21:57:08.867 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 42949672983 2024-09-15T21:57:08.867 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836510 1-42949672983' 2024-09-15T21:57:08.867 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T21:57:08.868 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-15T21:57:08.970 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=64424509453 2024-09-15T21:57:08.971 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 64424509453 2024-09-15T21:57:08.971 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836510 1-42949672983 2-64424509453' 2024-09-15T21:57:08.971 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T21:57:08.971 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-15T21:57:09.087 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=85899345931 2024-09-15T21:57:09.087 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 85899345931 2024-09-15T21:57:09.087 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836510 1-42949672983 2-64424509453 3-85899345931' 2024-09-15T21:57:09.087 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T21:57:09.087 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836510 2024-09-15T21:57:09.087 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T21:57:09.088 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-15T21:57:09.089 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836510 2024-09-15T21:57:09.089 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T21:57:09.090 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836510 2024-09-15T21:57:09.090 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836510' 2024-09-15T21:57:09.090 INFO:tasks.workunit.client.0.smithi136.stdout:waiting osd.0 seq 21474836510 2024-09-15T21:57:09.090 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T21:57:09.367 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836509 -lt 21474836510 2024-09-15T21:57:09.367 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-15T21:57:10.368 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-15T21:57:10.369 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T21:57:10.653 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836509 -lt 21474836510 2024-09-15T21:57:10.653 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-15T21:57:11.654 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 299 -eq 0 ']' 2024-09-15T21:57:11.655 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T21:57:11.938 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836511 -lt 21474836510 2024-09-15T21:57:11.938 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T21:57:11.939 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-42949672983 2024-09-15T21:57:11.939 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T21:57:11.940 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-15T21:57:11.940 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-42949672983 2024-09-15T21:57:11.941 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T21:57:11.941 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=42949672983 2024-09-15T21:57:11.942 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 42949672983' 2024-09-15T21:57:11.942 INFO:tasks.workunit.client.0.smithi136.stdout:waiting osd.1 seq 42949672983 2024-09-15T21:57:11.942 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-15T21:57:12.219 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 42949672984 -lt 42949672983 2024-09-15T21:57:12.219 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T21:57:12.219 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-64424509453 2024-09-15T21:57:12.219 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T21:57:12.220 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-15T21:57:12.221 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-64424509453 2024-09-15T21:57:12.221 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T21:57:12.222 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=64424509453 2024-09-15T21:57:12.222 INFO:tasks.workunit.client.0.smithi136.stdout:waiting osd.2 seq 64424509453 2024-09-15T21:57:12.223 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 64424509453' 2024-09-15T21:57:12.223 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-15T21:57:12.499 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 64424509454 -lt 64424509453 2024-09-15T21:57:12.499 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T21:57:12.499 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 3-85899345931 2024-09-15T21:57:12.500 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T21:57:12.501 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=3 2024-09-15T21:57:12.501 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 3-85899345931 2024-09-15T21:57:12.501 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T21:57:12.502 INFO:tasks.workunit.client.0.smithi136.stdout:waiting osd.3 seq 85899345931 2024-09-15T21:57:12.503 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=85899345931 2024-09-15T21:57:12.503 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.3 seq 85899345931' 2024-09-15T21:57:12.503 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-15T21:57:12.781 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 85899345932 -lt 85899345931 2024-09-15T21:57:12.782 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-15T21:57:12.782 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T21:57:12.782 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T21:57:13.135 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 1 == 0 2024-09-15T21:57:13.135 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-15T21:57:13.135 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-15T21:57:13.135 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-15T21:57:13.135 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T21:57:13.135 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T21:57:13.135 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T21:57:13.135 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T21:57:13.407 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=1 2024-09-15T21:57:13.408 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-15T21:57:13.408 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T21:57:13.408 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T21:57:13.755 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 = 1 2024-09-15T21:57:13.755 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-15T21:57:13.755 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-15T21:57:13.755 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2457: create_ec_pool: return 0 2024-09-15T21:57:13.756 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:865: unfound_erasure_coded: add_something td/osd-scrub-repair ecpool 2024-09-15T21:57:13.756 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-09-15T21:57:13.756 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=ecpool 2024-09-15T21:57:13.756 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=SOMETHING 2024-09-15T21:57:13.756 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-09-15T21:57:13.756 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-09-15T21:57:13.756 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-09-15T21:57:14.284 INFO:tasks.workunit.client.0.smithi136.stderr:noscrub is set 2024-09-15T21:57:14.296 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-09-15T21:57:14.762 INFO:tasks.workunit.client.0.smithi136.stderr:nodeep-scrub is set 2024-09-15T21:57:14.773 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-09-15T21:57:14.773 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-09-15T21:57:14.774 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool ecpool put SOMETHING td/osd-scrub-repair/ORIGINAL 2024-09-15T21:57:15.157 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:867: unfound_erasure_coded: get_primary ecpool SOMETHING 2024-09-15T21:57:15.157 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1187: get_primary: local poolname=ecpool 2024-09-15T21:57:15.157 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1188: get_primary: local objectname=SOMETHING 2024-09-15T21:57:15.157 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1190: get_primary: ceph --format json osd map ecpool SOMETHING 2024-09-15T21:57:15.157 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1191: get_primary: jq .acting_primary 2024-09-15T21:57:15.438 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:867: unfound_erasure_coded: local primary=1 2024-09-15T21:57:15.439 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:868: unfound_erasure_coded: get_osds ecpool SOMETHING 2024-09-15T21:57:15.439 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:868: unfound_erasure_coded: sed -e s/1// 2024-09-15T21:57:15.439 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1012: get_osds: local poolname=ecpool 2024-09-15T21:57:15.439 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1013: get_osds: local objectname=SOMETHING 2024-09-15T21:57:15.439 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: ceph --format json osd map ecpool SOMETHING 2024-09-15T21:57:15.439 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: jq '.acting | .[]' 2024-09-15T21:57:15.717 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: local 'osds=1 2024-09-15T21:57:15.717 INFO:tasks.workunit.client.0.smithi136.stderr:0 2024-09-15T21:57:15.717 INFO:tasks.workunit.client.0.smithi136.stderr:2 2024-09-15T21:57:15.717 INFO:tasks.workunit.client.0.smithi136.stderr:3' 2024-09-15T21:57:15.717 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1018: get_osds: echo 1 0 2 3 2024-09-15T21:57:15.720 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:868: unfound_erasure_coded: osds=('0' '2' '3') 2024-09-15T21:57:15.720 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:868: unfound_erasure_coded: local -a osds 2024-09-15T21:57:15.720 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:869: unfound_erasure_coded: local not_primary_first=0 2024-09-15T21:57:15.720 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:870: unfound_erasure_coded: local not_primary_second=2 2024-09-15T21:57:15.720 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:871: unfound_erasure_coded: local not_primary_third=3 2024-09-15T21:57:15.720 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:876: unfound_erasure_coded: pids= 2024-09-15T21:57:15.721 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:877: unfound_erasure_coded: run_in_background pids objectstore_tool td/osd-scrub-repair 0 SOMETHING remove 2024-09-15T21:57:15.721 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2135: run_in_background: local pid_variable=pids 2024-09-15T21:57:15.721 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2136: run_in_background: shift 2024-09-15T21:57:15.721 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2140: run_in_background: eval 'pids+=" 458568"' 2024-09-15T21:57:15.721 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2140: run_in_background: pids+=' 458568' 2024-09-15T21:57:15.721 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:878: unfound_erasure_coded: run_in_background pids objectstore_tool td/osd-scrub-repair 2 SOMETHING remove 2024-09-15T21:57:15.721 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2135: run_in_background: local pid_variable=pids 2024-09-15T21:57:15.721 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2136: run_in_background: shift 2024-09-15T21:57:15.721 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2140: run_in_background: eval 'pids+=" 458569"' 2024-09-15T21:57:15.721 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2140: run_in_background: pids+=' 458569' 2024-09-15T21:57:15.721 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:879: unfound_erasure_coded: run_in_background pids objectstore_tool td/osd-scrub-repair 3 SOMETHING remove 2024-09-15T21:57:15.721 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: run_in_background: objectstore_tool td/osd-scrub-repair 0 SOMETHING remove 2024-09-15T21:57:15.721 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2135: run_in_background: local pid_variable=pids 2024-09-15T21:57:15.722 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2136: run_in_background: shift 2024-09-15T21:57:15.722 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: run_in_background: sed 's/^/458571: /' 2024-09-15T21:57:15.722 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2140: run_in_background: eval 'pids+=" 458572"' 2024-09-15T21:57:15.722 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2140: run_in_background: pids+=' 458572' 2024-09-15T21:57:15.722 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:880: unfound_erasure_coded: wait_background pids 2024-09-15T21:57:15.722 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: run_in_background: objectstore_tool td/osd-scrub-repair 2 SOMETHING remove 2024-09-15T21:57:15.722 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: wait_background: pids=' 458568 458569 458572' 2024-09-15T21:57:15.722 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: wait_background: return_code=0 2024-09-15T21:57:15.722 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: wait_background: for pid in $pids 2024-09-15T21:57:15.722 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2175: wait_background: wait 458568 2024-09-15T21:57:15.722 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: run_in_background: sed 's/^/458574: /' 2024-09-15T21:57:15.722 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: run_in_background: objectstore_tool td/osd-scrub-repair 3 SOMETHING remove 2024-09-15T21:57:15.722 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: run_in_background: sed 's/^/458577: /' 2024-09-15T21:57:18.992 INFO:tasks.workunit.client.0.smithi136.stderr:458577: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2024-09-15T21:57:18.992 INFO:tasks.workunit.client.0.smithi136.stderr:458577: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2024-09-15T21:57:18.992 INFO:tasks.workunit.client.0.smithi136.stderr:458577: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=3 2024-09-15T21:57:18.992 INFO:tasks.workunit.client.0.smithi136.stderr:458577: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2024-09-15T21:57:18.993 INFO:tasks.workunit.client.0.smithi136.stderr:458577: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 3 SOMETHING remove 2024-09-15T21:57:18.993 INFO:tasks.workunit.client.0.smithi136.stderr:458577: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-09-15T21:57:18.993 INFO:tasks.workunit.client.0.smithi136.stderr:458577: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2024-09-15T21:57:18.993 INFO:tasks.workunit.client.0.smithi136.stderr:458577: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=3 2024-09-15T21:57:18.993 INFO:tasks.workunit.client.0.smithi136.stderr:458577: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2024-09-15T21:57:18.993 INFO:tasks.workunit.client.0.smithi136.stderr:458577: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.3 2024-09-15T21:57:18.993 INFO:tasks.workunit.client.0.smithi136.stderr:458577: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-15T21:57:18.993 INFO:tasks.workunit.client.0.smithi136.stderr:458577: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-15T21:57:18.993 INFO:tasks.workunit.client.0.smithi136.stderr:458577: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-15T21:57:18.994 INFO:tasks.workunit.client.0.smithi136.stderr:458577: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-15T21:57:18.994 INFO:tasks.workunit.client.0.smithi136.stderr:458577: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-15T21:57:18.994 INFO:tasks.workunit.client.0.smithi136.stderr:458577: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-15T21:57:18.994 INFO:tasks.workunit.client.0.smithi136.stderr:458577: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 3 SOMETHING remove 2024-09-15T21:57:18.994 INFO:tasks.workunit.client.0.smithi136.stderr:458577: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-15T21:57:18.994 INFO:tasks.workunit.client.0.smithi136.stderr:458577: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-15T21:57:18.994 INFO:tasks.workunit.client.0.smithi136.stderr:458577: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=3 2024-09-15T21:57:18.994 INFO:tasks.workunit.client.0.smithi136.stderr:458577: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-15T21:57:18.994 INFO:tasks.workunit.client.0.smithi136.stderr:458577: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/3 2024-09-15T21:57:18.994 INFO:tasks.workunit.client.0.smithi136.stderr:458577: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/3 SOMETHING remove 2024-09-15T21:57:18.994 INFO:tasks.workunit.client.0.smithi136.stderr:458577: remove 3#1:eb822e21:::SOMETHING:head# 2024-09-15T21:57:18.994 INFO:tasks.workunit.client.0.smithi136.stderr:458577: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 3 2024-09-15T21:57:18.994 INFO:tasks.workunit.client.0.smithi136.stderr:458577: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2024-09-15T21:57:18.994 INFO:tasks.workunit.client.0.smithi136.stderr:458577: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-15T21:57:18.994 INFO:tasks.workunit.client.0.smithi136.stderr:458577: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=3 2024-09-15T21:57:18.995 INFO:tasks.workunit.client.0.smithi136.stderr:458577: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-15T21:57:18.995 INFO:tasks.workunit.client.0.smithi136.stderr:458577: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/3 2024-09-15T21:57:18.995 INFO:tasks.workunit.client.0.smithi136.stderr:458577: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=ce4892c7-6c6d-4b4f-8e6f-07a3e11133e5 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-15T21:57:18.995 INFO:tasks.workunit.client.0.smithi136.stderr:458577: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T21:57:18.995 INFO:tasks.workunit.client.0.smithi136.stderr:458577: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: 458574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2024-09-15T21:57:18.995 INFO:tasks.workunit.client.0.smithi136.stderr:458574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2024-09-15T21:57:18.995 INFO:tasks.workunit.client.0.smithi136.stderr:458574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=2 2024-09-15T21:57:18.995 INFO:tasks.workunit.client.0.smithi136.stderr:458574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2024-09-15T21:57:18.995 INFO:tasks.workunit.client.0.smithi136.stderr:458574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 2 SOMETHING remove 2024-09-15T21:57:18.995 INFO:tasks.workunit.client.0.smithi136.stderr:458574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-09-15T21:57:18.996 INFO:tasks.workunit.client.0.smithi136.stderr:458574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2024-09-15T21:57:18.996 INFO:tasks.workunit.client.0.smithi136.stderr:458574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=2 2024-09-15T21:57:18.996 INFO:tasks.workunit.client.0.smithi136.stderr:458574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2024-09-15T21:57:18.996 INFO:tasks.workunit.client.0.smithi136.stderr:458574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.2 2024-09-15T21:57:18.996 INFO:tasks.workunit.client.0.smithi136.stderr:458574: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-15T21:57:18.996 INFO:tasks.workunit.client.0.smithi136.stderr:458574: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-15T21:57:18.996 INFO:tasks.workunit.client.0.smithi136.stderr:458574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-15T21:57:18.996 INFO:tasks.workunit.client.0.smithi136.stderr:458574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-15T21:57:18.996 INFO:tasks.workunit.client.0.smithi136.stderr:458574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-15T21:57:18.996 INFO:tasks.workunit.client.0.smithi136.stderr:458574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-15T21:57:18.996 INFO:tasks.workunit.client.0.smithi136.stderr:458574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 2 SOMETHING remove 2024-09-15T21:57:18.996 INFO:tasks.workunit.client.0.smithi136.stderr:458574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-15T21:57:18.996 INFO:tasks.workunit.client.0.smithi136.stderr:458574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-15T21:57:18.997 INFO:tasks.workunit.client.0.smithi136.stderr:458574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=2 2024-09-15T21:57:18.997 INFO:tasks.workunit.client.0.smithi136.stderr:458574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-15T21:57:18.997 INFO:tasks.workunit.client.0.smithi136.stderr:458574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/2 2024-09-15T21:57:18.997 INFO:tasks.workunit.client.0.smithi136.stderr:458574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/2 SOMETHING remove 2024-09-15T21:57:18.997 INFO:tasks.workunit.client.0.smithi136.stderr:458574: remove 2#1:eb822e21:::SOMETHING:head# 2024-09-15T21:57:18.997 INFO:tasks.workunit.client.0.smithi136.stderr:458574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 2 2024-09-15T21:57:18.997 INFO:tasks.workunit.client.0.smithi136.stderr:458574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2024-09-15T21:57:18.997 INFO:tasks.workunit.client.0.smithi136.stderr:458574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-15T21:57:18.997 INFO:tasks.workunit.client.0.smithi136.stderr:458574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=2 2024-09-15T21:57:18.998 INFO:tasks.workunit.client.0.smithi136.stderr:458574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-15T21:57:18.998 INFO:tasks.workunit.client.0.smithi136.stderr:458574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/2 2024-09-15T21:57:18.998 INFO:tasks.workunit.client.0.smithi136.stderr:458574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=ce4892c7-6c6d-4b4f-8e6f-07a3e11133e5 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-15T21:57:18.998 INFO:tasks.workunit.client.0.smithi136.stderr:458574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T21:57:18.998 INFO:tasks.workunit.client.0.smithi136.stderr:458574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T21:57:18.998 INFO:tasks.workunit.client.0.smithi136.stderr:458577: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T21:57:18.998 INFO:tasks.workunit.client.0.smithi136.stderr:458577: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/3' 2024-09-15T21:57:18.999 INFO:tasks.workunit.client.0.smithi136.stderr:458577: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/3/journal' 2024-09-15T21:57:18.999 INFO:tasks.workunit.client.0.smithi136.stderr:458577: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-15T21:57:18.999 INFO:tasks.workunit.client.0.smithi136.stderr:458577: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-15T21:57:18.999 INFO:tasks.workunit.client.0.smithi136.stderr:458577: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-15T21:57:18.999 INFO:tasks.workunit.client.0.smithi136.stderr:458577: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-15T21:57:18.999 INFO:tasks.workunit.client.0.smithi136.stderr:458577: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-15T21:57:18.999 INFO:tasks.workunit.client.0.smithi136.stderr:458577: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-15T21:57:18.999 INFO:tasks.workunit.client.0.smithi136.stderr:458577: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-15T21:57:18.999 INFO:tasks.workunit.client.0.smithi136.stderr:458577: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T21:57:18.999 INFO:tasks.workunit.client.0.smithi136.stderr:458577: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52602 2024-09-15T21:57:18.999 INFO:tasks.workunit.client.0.smithi136.stderr:458577: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.52602/$cluster-$name.asok' 2024-09-15T21:57:18.999 INFO:tasks.workunit.client.0.smithi136.stderr:458577: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.52602/$cluster-$name.asok' 2024-09-15T21:57:18.999 INFO:tasks.workunit.client.0.smithi136.stderr:458577: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-15T21:57:18.999 INFO:tasks.workunit.client.0.smithi136.stderr:458577: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-15T21:57:18.999 INFO:tasks.workunit.client.0.smithi136.stderr:458577: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-15T21:57:18.999 INFO:tasks.workunit.client.0.smithi136.stderr:458577: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T21:57:19.000 INFO:tasks.workunit.client.0.smithi136.stderr:458577: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T21:57:19.000 INFO:tasks.workunit.client.0.smithi136.stderr:458577: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T21:57:19.000 INFO:tasks.workunit.client.0.smithi136.stderr:458577: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T21:57:19.000 INFO:tasks.workunit.client.0.smithi136.stderr:458577: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-15T21:57:19.000 INFO:tasks.workunit.client.0.smithi136.stderr:458577: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-15T21:57:19.000 INFO:tasks.workunit.client.0.smithi136.stderr:458577: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/3 2024-09-15T21:57:19.000 INFO:tasks.workunit.client.0.smithi136.stderr:458577: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.3 2024-09-15T21:57:19.000 INFO:tasks.workunit.client.0.smithi136.stderr:458577: start osd.3 2024-09-15T21:57:19.000 INFO:tasks.workunit.client.0.smithi136.stderr:458577: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 3 --fsid=ce4892c7-6c6d-4b4f-8e6f-07a3e11133e5 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/3 --osd-journal=td/osd-scrub-repair/3/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.52602/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T21:57:19.000 INFO:tasks.workunit.client.0.smithi136.stderr:4activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T21:57:19.000 INFO:tasks.workunit.client.0.smithi136.stderr:458574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T21:57:19.000 INFO:tasks.workunit.client.0.smithi136.stderr:458574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/2' 2024-09-15T21:57:19.000 INFO:tasks.workunit.client.0.smithi136.stderr:458574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/2/journal' 2024-09-15T21:57:19.000 INFO:tasks.workunit.client.0.smithi136.stderr:458574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-15T21:57:19.001 INFO:tasks.workunit.client.0.smithi136.stderr:458574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-15T21:57:19.001 INFO:tasks.workunit.client.0.smithi136.stderr:458574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-15T21:57:19.001 INFO:tasks.workunit.client.0.smithi136.stderr:458574: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-15T21:57:19.001 INFO:tasks.workunit.client.0.smithi136.stderr:458574: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-15T21:57:19.001 INFO:tasks.workunit.client.0.smithi136.stderr:458574: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-15T21:57:19.001 INFO:tasks.workunit.client.0.smithi136.stderr:458574: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-15T21:57:19.001 INFO:tasks.workunit.client.0.smithi136.stderr:458574: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T21:57:19.001 INFO:tasks.workunit.client.0.smithi136.stderr:458574: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52602 2024-09-15T21:57:19.001 INFO:tasks.workunit.client.0.smithi136.stderr:458574: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.52602/$cluster-$name.asok' 2024-09-15T21:57:19.001 INFO:tasks.workunit.client.0.smithi136.stderr:458574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.52602/$cluster-$name.asok' 2024-09-15T21:57:19.002 INFO:tasks.workunit.client.0.smithi136.stderr:458574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-15T21:57:19.002 INFO:tasks.workunit.client.0.smithi136.stderr:458574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-15T21:57:19.002 INFO:tasks.workunit.client.0.smithi136.stderr:458574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-15T21:57:19.002 INFO:tasks.workunit.client.0.smithi136.stderr:458574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T21:57:19.002 INFO:tasks.workunit.client.0.smithi136.stderr:458574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T21:57:19.002 INFO:tasks.workunit.client.0.smithi136.stderr:458574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T21:57:19.002 INFO:tasks.workunit.client.0.smithi136.stderr:458574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T21:57:19.002 INFO:tasks.workunit.client.0.smithi136.stderr:458574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-15T21:57:19.002 INFO:tasks.workunit.client.0.smithi136.stderr:458574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-15T21:57:19.003 INFO:tasks.workunit.client.0.smithi136.stderr:458574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/2 2024-09-15T21:57:19.003 INFO:tasks.workunit.client.0.smithi136.stderr:458574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.2 2024-09-15T21:57:19.003 INFO:tasks.workunit.client.0.smithi136.stderr:458574: start osd.2 2024-09-15T21:57:19.003 INFO:tasks.workunit.client.0.smithi136.stderr:458574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 2 --fsid=ce4892c7-6c6d-4b4f-8e6f-07a3e11133e5 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.52602/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T21:57:19.131 INFO:tasks.workunit.client.0.smithi136.stderr:4458571: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2024-09-15T21:57:19.131 INFO:tasks.workunit.client.0.smithi136.stderr:458571: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2024-09-15T21:57:19.131 INFO:tasks.workunit.client.0.smithi136.stderr:458571: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=0 2024-09-15T21:57:19.131 INFO:tasks.workunit.client.0.smithi136.stderr:458571: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2024-09-15T21:57:19.132 INFO:tasks.workunit.client.0.smithi136.stderr:458571: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 SOMETHING remove 2024-09-15T21:57:19.132 INFO:tasks.workunit.client.0.smithi136.stderr:458571: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-09-15T21:57:19.132 INFO:tasks.workunit.client.0.smithi136.stderr:458571: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2024-09-15T21:57:19.132 INFO:tasks.workunit.client.0.smithi136.stderr:458571: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=0 2024-09-15T21:57:19.132 INFO:tasks.workunit.client.0.smithi136.stderr:458571: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2024-09-15T21:57:19.132 INFO:tasks.workunit.client.0.smithi136.stderr:458571: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2024-09-15T21:57:19.132 INFO:tasks.workunit.client.0.smithi136.stderr:458571: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-15T21:57:19.132 INFO:tasks.workunit.client.0.smithi136.stderr:458571: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-15T21:57:19.132 INFO:tasks.workunit.client.0.smithi136.stderr:458571: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-15T21:57:19.133 INFO:tasks.workunit.client.0.smithi136.stderr:458571: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-15T21:57:19.133 INFO:tasks.workunit.client.0.smithi136.stderr:458571: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-15T21:57:19.133 INFO:tasks.workunit.client.0.smithi136.stderr:458571: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-15T21:57:19.133 INFO:tasks.workunit.client.0.smithi136.stderr:458571: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 SOMETHING remove 2024-09-15T21:57:19.133 INFO:tasks.workunit.client.0.smithi136.stderr:458571: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-15T21:57:19.134 INFO:tasks.workunit.client.0.smithi136.stderr:458571: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-15T21:57:19.134 INFO:tasks.workunit.client.0.smithi136.stderr:458571: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=0 2024-09-15T21:57:19.134 INFO:tasks.workunit.client.0.smithi136.stderr:458571: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-15T21:57:19.134 INFO:tasks.workunit.client.0.smithi136.stderr:458571: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-09-15T21:57:19.134 INFO:tasks.workunit.client.0.smithi136.stderr:458571: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 SOMETHING remove 2024-09-15T21:57:19.134 INFO:tasks.workunit.client.0.smithi136.stderr:458571: remove 1#1:eb822e21:::SOMETHING:head# 2024-09-15T21:57:19.134 INFO:tasks.workunit.client.0.smithi136.stderr:458571: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 2024-09-15T21:57:19.134 INFO:tasks.workunit.client.0.smithi136.stderr:458571: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2024-09-15T21:57:19.134 INFO:tasks.workunit.client.0.smithi136.stderr:458571: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-15T21:57:19.134 INFO:tasks.workunit.client.0.smithi136.stderr:458571: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=0 2024-09-15T21:57:19.134 INFO:tasks.workunit.client.0.smithi136.stderr:458571: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-15T21:57:19.135 INFO:tasks.workunit.client.0.smithi136.stderr:458571: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-09-15T21:57:19.135 INFO:tasks.workunit.client.0.smithi136.stderr:458571: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=ce4892c7-6c6d-4b4f-8e6f-07a3e11133e5 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-15T21:57:19.135 INFO:tasks.workunit.client.0.smithi136.stderr:458571: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T21:57:19.135 INFO:tasks.workunit.client.0.smithi136.stderr:458571: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T21:57:19.135 INFO:tasks.workunit.client.0.smithi136.stderr:458571: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T21:57:19.135 INFO:tasks.workunit.client.0.smithi136.stderr:458571: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-09-15T21:57:19.135 INFO:tasks.workunit.client.0.smithi136.stderr:458571: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-09-15T21:57:19.135 INFO:tasks.workunit.client.0.smithi136.stderr:458571: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-15T21:57:19.135 INFO:tasks.workunit.client.0.smithi136.stderr:458571: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-15T21:57:19.135 INFO:tasks.workunit.client.0.smithi136.stderr:458571: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-15T21:57:19.136 INFO:tasks.workunit.client.0.smithi136.stderr:458571: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-15T21:57:19.136 INFO:tasks.workunit.client.0.smithi136.stderr:458571: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-15T21:57:19.136 INFO:tasks.workunit.client.0.smithi136.stderr:458571: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-15T21:57:19.136 INFO:tasks.workunit.client.0.smithi136.stderr:458571: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-15T21:57:19.136 INFO:tasks.workunit.client.0.smithi136.stderr:458571: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T21:57:19.136 INFO:tasks.workunit.client.0.smithi136.stderr:458571: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52602 2024-09-15T21:57:19.136 INFO:tasks.workunit.client.0.smithi136.stderr:458571: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.52602/$cluster-$name.asok' 2024-09-15T21:57:19.136 INFO:tasks.workunit.client.0.smithi136.stderr:458571: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.52602/$cluster-$name.asok' 2024-09-15T21:57:19.136 INFO:tasks.workunit.client.0.smithi136.stderr:458571: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-15T21:57:19.137 INFO:tasks.workunit.client.0.smithi136.stderr:458571: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-15T21:57:19.137 INFO:tasks.workunit.client.0.smithi136.stderr:458571: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-15T21:57:19.137 INFO:tasks.workunit.client.0.smithi136.stderr:458571: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T21:57:19.137 INFO:tasks.workunit.client.0.smithi136.stderr:458571: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T21:57:19.137 INFO:tasks.workunit.client.0.smithi136.stderr:458571: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T21:57:19.137 INFO:tasks.workunit.client.0.smithi136.stderr:458571: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T21:57:19.138 INFO:tasks.workunit.client.0.smithi136.stderr:458571: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-15T21:57:19.138 INFO:tasks.workunit.client.0.smithi136.stderr:458571: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-15T21:57:19.138 INFO:tasks.workunit.client.0.smithi136.stderr:458571: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-09-15T21:57:19.138 INFO:tasks.workunit.client.0.smithi136.stderr:458571: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.0 2024-09-15T21:57:19.138 INFO:tasks.workunit.client.0.smithi136.stderr:458571: start osd.0 2024-09-15T21:57:19.138 INFO:tasks.workunit.client.0.smithi136.stderr:458571: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 0 --fsid=ce4892c7-6c6d-4b4f-8e6f-07a3e11133e5 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.52602/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T21:57:23.757 INFO:tasks.workunit.client.0.smithi136.stderr:458574: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/2/whoami 2024-09-15T21:57:23.757 INFO:tasks.workunit.client.0.smithi136.stderr:458574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 2 = 2 ']' 2024-09-15T21:57:23.757 INFO:tasks.workunit.client.0.smithi136.stderr:458574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-15T21:57:23.757 INFO:tasks.workunit.client.0.smithi136.stderr:458574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-15T21:57:23.757 INFO:tasks.workunit.client.0.smithi136.stderr:458574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-15T21:57:23.757 INFO:tasks.workunit.client.0.smithi136.stderr:458574: 2024-09-15T21:57:19.025+0000 7f2077011640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:57:23.758 INFO:tasks.workunit.client.0.smithi136.stderr:458574: 2024-09-15T21:57:19.028+0000 7f2077011640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:57:23.758 INFO:tasks.workunit.client.0.smithi136.stderr:458574: 2024-09-15T21:57:19.029+0000 7f2077011640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:57:23.758 INFO:tasks.workunit.client.0.smithi136.stderr:458574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 2 2024-09-15T21:57:23.758 INFO:tasks.workunit.client.0.smithi136.stderr:458574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-15T21:57:23.758 INFO:tasks.workunit.client.0.smithi136.stderr:458574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=2 2024-09-15T21:57:23.758 INFO:tasks.workunit.client.0.smithi136.stderr:458574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-15T21:57:23.758 INFO:tasks.workunit.client.0.smithi136.stderr:458574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-15T21:57:23.758 INFO:tasks.workunit.client.0.smithi136.stderr:458574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:57:23.758 INFO:tasks.workunit.client.0.smithi136.stderr:458574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-15T21:57:23.758 INFO:tasks.workunit.client.0.smithi136.stderr:458574: 0 2024-09-15T21:57:23.758 INFO:tasks.workunit.client.0.smithi136.stderr:458574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:57:23.758 INFO:tasks.workunit.client.0.smithi136.stderr:458574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T21:57:23.758 INFO:tasks.workunit.client.0.smithi136.stderr:458574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:57:23.758 INFO:tasks.workunit.client.0.smithi136.stderr:458574: 2024-09-15T21:57:20.154+0000 7f2077011640 -1 Falling back to public interface 2024-09-15T21:57:23.758 INFO:tasks.workunit.client.0.smithi136.stderr:458574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:57:23.759 INFO:tasks.workunit.client.0.smithi136.stderr:458574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:57:23.759 INFO:tasks.workunit.client.0.smithi136.stderr:458574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-15T21:57:23.759 INFO:tasks.workunit.client.0.smithi136.stderr:458574: 1 2024-09-15T21:57:23.759 INFO:tasks.workunit.client.0.smithi136.stderr:458574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:57:23.759 INFO:tasks.workunit.client.0.smithi136.stderr:458574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T21:57:23.759 INFO:tasks.workunit.client.0.smithi136.stderr:458574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:57:23.759 INFO:tasks.workunit.client.0.smithi136.stderr:458574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:57:23.759 INFO:tasks.workunit.client.0.smithi136.stderr:458574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:57:23.759 INFO:tasks.workunit.client.0.smithi136.stderr:458574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-15T21:57:23.759 INFO:tasks.workunit.client.0.smithi136.stderr:458574: 2 2024-09-15T21:57:23.759 INFO:tasks.workunit.client.0.smithi136.stderr:458574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:57:23.759 INFO:tasks.workunit.client.0.smithi136.stderr:458574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T21:57:23.759 INFO:tasks.workunit.client.0.smithi136.stderr:458574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:57:23.759 INFO:tasks.workunit.client.0.smithi136.stderr:458574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:57:23.759 INFO:tasks.workunit.client.0.smithi136.stderr:458574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:57:23.760 INFO:tasks.workunit.client.0.smithi136.stderr:458574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-15T21:57:23.760 INFO:tasks.workunit.client.0.smithi136.stderr:458574: 3 2024-09-15T21:57:23.760 INFO:tasks.workunit.client.0.smithi136.stderr:458574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:57:23.760 INFO:tasks.workunit.client.0.smithi136.stderr:458574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T21:57:23.760 INFO:tasks.workunit.client.0.smithi136.stderr:458574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:57:23.824 INFO:tasks.workunit.client.0.smithi136.stderr:458574: 2024-09-15T21:57:23.755+0000 7f2077011640 -58577: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/3/whoami 2024-09-15T21:57:23.824 INFO:tasks.workunit.client.0.smithi136.stderr:458577: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 3 = 3 ']' 2024-09-15T21:57:23.824 INFO:tasks.workunit.client.0.smithi136.stderr:458577: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-15T21:57:23.824 INFO:tasks.workunit.client.0.smithi136.stderr:458577: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-15T21:57:23.824 INFO:tasks.workunit.client.0.smithi136.stderr:458577: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-15T21:57:23.825 INFO:tasks.workunit.client.0.smithi136.stderr:458577: 2024-09-15T21:57:19.025+0000 7f2636e1d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:57:23.825 INFO:tasks.workunit.client.0.smithi136.stderr:458577: 2024-09-15T21:57:19.028+0000 7f2636e1d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:57:23.825 INFO:tasks.workunit.client.0.smithi136.stderr:458577: 2024-09-15T21:57:19.030+0000 7f2636e1d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:57:23.825 INFO:tasks.workunit.client.0.smithi136.stderr:458577: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 3 2024-09-15T21:57:23.825 INFO:tasks.workunit.client.0.smithi136.stderr:458577: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-15T21:57:23.825 INFO:tasks.workunit.client.0.smithi136.stderr:458577: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=3 2024-09-15T21:57:23.825 INFO:tasks.workunit.client.0.smithi136.stderr:458577: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-15T21:57:23.825 INFO:tasks.workunit.client.0.smithi136.stderr:458577: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-15T21:57:23.825 INFO:tasks.workunit.client.0.smithi136.stderr:458577: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:57:23.825 INFO:tasks.workunit.client.0.smithi136.stderr:458577: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-15T21:57:23.825 INFO:tasks.workunit.client.0.smithi136.stderr:458577: 0 2024-09-15T21:57:23.825 INFO:tasks.workunit.client.0.smithi136.stderr:458577: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:57:23.825 INFO:tasks.workunit.client.0.smithi136.stderr:458577: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-15T21:57:23.826 INFO:tasks.workunit.client.0.smithi136.stderr:458577: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:57:23.826 INFO:tasks.workunit.client.0.smithi136.stderr:458577: 2024-09-15T21:57:20.375+0000 7f2636e1d640 -1 Falling back to public interface 2024-09-15T21:57:23.826 INFO:tasks.workunit.client.0.smithi136.stderr:458577: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:57:23.826 INFO:tasks.workunit.client.0.smithi136.stderr:458577: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:57:23.826 INFO:tasks.workunit.client.0.smithi136.stderr:458577: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-15T21:57:23.826 INFO:tasks.workunit.client.0.smithi136.stderr:458577: 1 2024-09-15T21:57:23.826 INFO:tasks.workunit.client.0.smithi136.stderr:458577: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:57:23.826 INFO:tasks.workunit.client.0.smithi136.stderr:458577: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-15T21:57:23.826 INFO:tasks.workunit.client.0.smithi136.stderr:458577: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:57:23.826 INFO:tasks.workunit.client.0.smithi136.stderr:458577: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:57:23.826 INFO:tasks.workunit.client.0.smithi136.stderr:458577: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:57:23.826 INFO:tasks.workunit.client.0.smithi136.stderr:458577: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-15T21:57:23.826 INFO:tasks.workunit.client.0.smithi136.stderr:458577: 2 2024-09-15T21:57:23.826 INFO:tasks.workunit.client.0.smithi136.stderr:458577: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:57:23.826 INFO:tasks.workunit.client.0.smithi136.stderr:458577: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-15T21:57:23.827 INFO:tasks.workunit.client.0.smithi136.stderr:458577: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:57:23.827 INFO:tasks.workunit.client.0.smithi136.stderr:458577: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:57:23.827 INFO:tasks.workunit.client.0.smithi136.stderr:458577: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:57:23.827 INFO:tasks.workunit.client.0.smithi136.stderr:458577: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-15T21:57:23.827 INFO:tasks.workunit.client.0.smithi136.stderr:458577: 3 2024-09-15T21:57:23.827 INFO:tasks.workunit.client.0.smithi136.stderr:458577: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:57:23.827 INFO:tasks.workunit.client.0.smithi136.stderr:458577: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-15T21:57:23.827 INFO:tasks.workunit.client.0.smithi136.stderr:458577: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:57:24.314 INFO:tasks.workunit.client.0.smithi136.stderr:458577: 2024-09-15T21:57:23.822+0000 7f2636e1d640 -58571: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-09-15T21:57:24.314 INFO:tasks.workunit.client.0.smithi136.stderr:458571: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 0 = 0 ']' 2024-09-15T21:57:24.314 INFO:tasks.workunit.client.0.smithi136.stderr:458571: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-15T21:57:24.314 INFO:tasks.workunit.client.0.smithi136.stderr:458571: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-15T21:57:24.314 INFO:tasks.workunit.client.0.smithi136.stderr:458571: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-15T21:57:24.315 INFO:tasks.workunit.client.0.smithi136.stderr:458571: 2024-09-15T21:57:19.160+0000 7f6b67c1c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:57:24.315 INFO:tasks.workunit.client.0.smithi136.stderr:458571: 2024-09-15T21:57:19.163+0000 7f6b67c1c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:57:24.315 INFO:tasks.workunit.client.0.smithi136.stderr:458571: 2024-09-15T21:57:19.164+0000 7f6b67c1c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:57:24.315 INFO:tasks.workunit.client.0.smithi136.stderr:458571: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 0 2024-09-15T21:57:24.315 INFO:tasks.workunit.client.0.smithi136.stderr:458571: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-15T21:57:24.315 INFO:tasks.workunit.client.0.smithi136.stderr:458571: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2024-09-15T21:57:24.315 INFO:tasks.workunit.client.0.smithi136.stderr:458571: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-15T21:57:24.315 INFO:tasks.workunit.client.0.smithi136.stderr:458571: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-15T21:57:24.315 INFO:tasks.workunit.client.0.smithi136.stderr:458571: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:57:24.315 INFO:tasks.workunit.client.0.smithi136.stderr:458571: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-15T21:57:24.315 INFO:tasks.workunit.client.0.smithi136.stderr:458571: 0 2024-09-15T21:57:24.315 INFO:tasks.workunit.client.0.smithi136.stderr:458571: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:57:24.315 INFO:tasks.workunit.client.0.smithi136.stderr:458571: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:57:24.315 INFO:tasks.workunit.client.0.smithi136.stderr:458571: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:57:24.316 INFO:tasks.workunit.client.0.smithi136.stderr:458571: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:57:24.316 INFO:tasks.workunit.client.0.smithi136.stderr:458571: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:57:24.316 INFO:tasks.workunit.client.0.smithi136.stderr:458571: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-15T21:57:24.316 INFO:tasks.workunit.client.0.smithi136.stderr:458571: 1 2024-09-15T21:57:24.316 INFO:tasks.workunit.client.0.smithi136.stderr:458571: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:57:24.316 INFO:tasks.workunit.client.0.smithi136.stderr:458571: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:57:24.316 INFO:tasks.workunit.client.0.smithi136.stderr:458571: 2024-09-15T21:57:20.765+0000 7f6b67c1c640 -1 Falling back to public interface 2024-09-15T21:57:24.316 INFO:tasks.workunit.client.0.smithi136.stderr:458571: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:57:24.316 INFO:tasks.workunit.client.0.smithi136.stderr:458571: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:57:24.316 INFO:tasks.workunit.client.0.smithi136.stderr:458571: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:57:24.316 INFO:tasks.workunit.client.0.smithi136.stderr:458571: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-15T21:57:24.316 INFO:tasks.workunit.client.0.smithi136.stderr:458571: 2 2024-09-15T21:57:24.316 INFO:tasks.workunit.client.0.smithi136.stderr:458571: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:57:24.317 INFO:tasks.workunit.client.0.smithi136.stderr:458571: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:57:24.317 INFO:tasks.workunit.client.0.smithi136.stderr:458571: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:57:24.317 INFO:tasks.workunit.client.0.smithi136.stderr:458571: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:57:24.317 INFO:tasks.workunit.client.0.smithi136.stderr:458571: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:57:24.317 INFO:tasks.workunit.client.0.smithi136.stderr:458571: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-15T21:57:24.317 INFO:tasks.workunit.client.0.smithi136.stderr:458571: 3 2024-09-15T21:57:24.317 INFO:tasks.workunit.client.0.smithi136.stderr:458571: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:57:24.317 INFO:tasks.workunit.client.0.smithi136.stderr:458571: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:57:24.317 INFO:tasks.workunit.client.0.smithi136.stderr:458571: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:57:25.493 INFO:tasks.workunit.client.0.smithi136.stderr:458571: 2024-09-15T21:57:24.312+0000 7f6b67c1c640 -1 osd.3 29 log_to_monitors true 2024-09-15T21:57:25.493 INFO:tasks.workunit.client.0.smithi136.stderr:458577: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:57:25.493 INFO:tasks.workunit.client.0.smithi136.stderr:458577: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:57:25.493 INFO:tasks.workunit.client.0.smithi136.stderr:458577: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-15T21:57:25.493 INFO:tasks.workunit.client.0.smithi136.stderr:458577: 4 2024-09-15T21:57:25.493 INFO:tasks.workunit.client.0.smithi136.stderr:458577: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:57:25.494 INFO:tasks.workunit.client.0.smithi136.stderr:458577: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-15T21:57:25.494 INFO:tasks.workunit.client.0.smithi136.stderr:458577: osd.3 up in weight 1 up_from 33 up_thru 0 down_at 30 last_clean_interval [20,29) [v2:127.0.0.1:6818/233894002,v1:127.0.0.1:6819/233894002] [v2:127.0.0.1:6820/233894002,v1:127.0.0.1:6821/233894002] exists,up e738d0f4-c28b-4426-bb0d-1cb17652a4d3 2024-09-15T21:57:25.494 INFO:tasks.workunit.client.0.smithi136.stderr:458577: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-15T21:57:25.494 INFO:tasks.workunit.client.0.smithi136.stderr:458577: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-15T21:57:25.494 INFO:tasks.workunit.client.0.smithi136.stderr:458577: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-15T21:57:25.494 INFO:tasks.workunit.client.0.smithi136.stderr:458577: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2024-09-15T21:57:25.494 INFO:tasks.workunit.client.0.smithi136.stderr:458577: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-15T21:57:25.494 INFO:tasks.workunit.client.0.smithi136.stderr:458577: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-15T21:57:25.494 INFO:tasks.workunit.client.0.smithi136.stderr:458577: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-15T21:57:25.494 INFO:tasks.workunit.client.0.smithi136.stderr:458577: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-15T21:57:25.494 INFO:tasks.workunit.client.0.smithi136.stderr:458577: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-15T21:57:25.494 INFO:tasks.workunit.client.0.smithi136.stderr:458577: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-15T21:57:25.494 INFO:tasks.workunit.client.0.smithi136.stderr:458577: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-15T21:57:25.495 INFO:tasks.workunit.client.0.smithi136.stderr:458577: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-15T21:57:25.495 INFO:tasks.workunit.client.0.smithi136.stderr:458577: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-15T21:57:25.495 INFO:tasks.workunit.client.0.smithi136.stderr:458577: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-15T21:57:25.495 INFO:tasks.workunit.client.0.smithi136.stderr:458577: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-15T21:57:25.495 INFO:tasks.workunit.client.0.smithi136.stderr:458577: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-15T21:57:25.495 INFO:tasks.workunit.client.0.smithi136.stderr:458577: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-15T21:57:25.495 INFO:tasks.workunit.client.0.smithi136.stderr:458577: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-15T21:57:25.495 INFO:tasks.workunit.client.0.smithi136.stderr:458577: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-15T21:57:25.495 INFO:tasks.workunit.client.0.smithi136.stderr:458577: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-15T21:57:25.495 INFO:tasks.workunit.client.0.smithi136.stderr:458577: 1 2024-09-15T21:57:25.495 INFO:tasks.workunit.client.0.smithi136.stderr:458577: 2 2024-09-15T21:57:25.495 INFO:tasks.workunit.client.0.smithi136.stderr:458577: 3' 2024-09-15T21:57:25.495 INFO:tasks.workunit.client.0.smithi136.stderr:458577: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-15T21:57:25.496 INFO:tasks.workunit.client.0.smithi136.stderr:458577: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T21:57:25.496 INFO:tasks.workunit.client.0.smithi136.stderr:458577: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-15T21:57:25.496 INFO:tasks.workunit.client.0.smithi136.stderr:458577: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=146028888066 2024-09-15T21:57:25.496 INFO:tasks.workunit.client.0.smithi136.stderr:458577: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 146028888066 2024-09-15T21:57:25.496 INFO:tasks.workunit.client.0.smithi136.stderr:458577: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-146028888066' 2024-09-15T21:57:25.496 INFO:tasks.workunit.client.0.smithi136.stderr:458577: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T21:57:25.604 INFO:tasks.workunit.client.0.smithi136.stderr:458577: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helper1 osd.0 29 log_to_monitors true 2024-09-15T21:57:25.604 INFO:tasks.workunit.client.0.smithi136.stderr:458571: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:57:25.604 INFO:tasks.workunit.client.0.smithi136.stderr:458571: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:57:25.604 INFO:tasks.workunit.client.0.smithi136.stderr:458571: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-15T21:57:25.604 INFO:tasks.workunit.client.0.smithi136.stderr:458571: 4 2024-09-15T21:57:25.604 INFO:tasks.workunit.client.0.smithi136.stderr:458571: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:57:25.604 INFO:tasks.workunit.client.0.smithi136.stderr:458571: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:57:25.604 INFO:tasks.workunit.client.0.smithi136.stderr:458571: osd.0 up in weight 1 up_from 34 up_thru 0 down_at 30 last_clean_interval [5,29) [v2:127.0.0.1:6826/4243863325,v1:127.0.0.1:6827/4243863325] [v2:127.0.0.1:6828/4243863325,v1:127.0.0.1:6829/4243863325] exists,up 5e15525f-cf1a-4270-8160-c8c91fa3959b 2024-09-15T21:57:25.604 INFO:tasks.workunit.client.0.smithi136.stderr:458571: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-15T21:57:25.604 INFO:tasks.workunit.client.0.smithi136.stderr:458571: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-15T21:57:25.605 INFO:tasks.workunit.client.0.smithi136.stderr:458571: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-15T21:57:25.605 INFO:tasks.workunit.client.0.smithi136.stderr:458571: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2024-09-15T21:57:25.605 INFO:tasks.workunit.client.0.smithi136.stderr:458571: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-15T21:57:25.605 INFO:tasks.workunit.client.0.smithi136.stderr:458571: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-15T21:57:25.605 INFO:tasks.workunit.client.0.smithi136.stderr:458571: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-15T21:57:25.605 INFO:tasks.workunit.client.0.smithi136.stderr:458571: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-15T21:57:25.605 INFO:tasks.workunit.client.0.smithi136.stderr:458571: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-15T21:57:25.605 INFO:tasks.workunit.client.0.smithi136.stderr:458571: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-15T21:57:25.605 INFO:tasks.workunit.client.0.smithi136.stderr:458571: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-15T21:57:25.605 INFO:tasks.workunit.client.0.smithi136.stderr:458571: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-15T21:57:25.605 INFO:tasks.workunit.client.0.smithi136.stderr:458571: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-15T21:57:25.605 INFO:tasks.workunit.client.0.smithi136.stderr:458571: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-15T21:57:25.605 INFO:tasks.workunit.client.0.smithi136.stderr:458571: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-15T21:57:25.605 INFO:tasks.workunit.client.0.smithi136.stderr:458571: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-15T21:57:25.606 INFO:tasks.workunit.client.0.smithi136.stderr:458571: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-15T21:57:25.606 INFO:tasks.workunit.client.0.smithi136.stderr:458571: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-15T21:57:25.606 INFO:tasks.workunit.client.0.smithi136.stderr:458571: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-15T21:57:25.606 INFO:tasks.workunit.client.0.smithi136.stderr:458571: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-15T21:57:25.606 INFO:tasks.workunit.client.0.smithi136.stderr:458571: 1 2024-09-15T21:57:25.606 INFO:tasks.workunit.client.0.smithi136.stderr:458571: 2 2024-09-15T21:57:25.606 INFO:tasks.workunit.client.0.smithi136.stderr:458571: 3' 2024-09-15T21:57:25.606 INFO:tasks.workunit.client.0.smithi136.stderr:458571: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-15T21:57:25.606 INFO:tasks.workunit.client.0.smithi136.stderr:458571: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T21:57:25.606 INFO:tasks.workunit.client.0.smithi136.stderr:458571: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-15T21:57:25.606 INFO:tasks.workunit.client.0.smithi136.stderr:458571: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=146028888067 2024-09-15T21:57:25.606 INFO:tasks.workunit.client.0.smithi136.stderr:458571: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 146028888067 2024-09-15T21:57:25.606 INFO:tasks.workunit.client.0.smithi136.stderr:458571: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-146028888067' 2024-09-15T21:57:25.606 INFO:tasks.workunit.client.0.smithi136.stderr:458571: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T21:57:25.759 INFO:tasks.workunit.client.0.smithi136.stderr:458571: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-hel1 osd.2 29 log_to_monitors true 2024-09-15T21:57:25.759 INFO:tasks.workunit.client.0.smithi136.stderr:458574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:57:25.760 INFO:tasks.workunit.client.0.smithi136.stderr:458574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:57:25.760 INFO:tasks.workunit.client.0.smithi136.stderr:458574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-15T21:57:25.760 INFO:tasks.workunit.client.0.smithi136.stderr:458574: 4 2024-09-15T21:57:25.760 INFO:tasks.workunit.client.0.smithi136.stderr:458574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:57:25.760 INFO:tasks.workunit.client.0.smithi136.stderr:458574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T21:57:25.760 INFO:tasks.workunit.client.0.smithi136.stderr:458574: osd.2 up in weight 1 up_from 33 up_thru 0 down_at 30 last_clean_interval [15,29) [v2:127.0.0.1:6802/4043303003,v1:127.0.0.1:6803/4043303003] [v2:127.0.0.1:6804/4043303003,v1:127.0.0.1:6805/4043303003] exists,up 1a14e877-c54e-41ad-9f45-41e1f568ebc8 2024-09-15T21:57:25.760 INFO:tasks.workunit.client.0.smithi136.stderr:458574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-15T21:57:25.760 INFO:tasks.workunit.client.0.smithi136.stderr:458574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-15T21:57:25.760 INFO:tasks.workunit.client.0.smithi136.stderr:458574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-15T21:57:25.760 INFO:tasks.workunit.client.0.smithi136.stderr:458574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2024-09-15T21:57:25.760 INFO:tasks.workunit.client.0.smithi136.stderr:458574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-15T21:57:25.760 INFO:tasks.workunit.client.0.smithi136.stderr:458574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-15T21:57:25.760 INFO:tasks.workunit.client.0.smithi136.stderr:458574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-15T21:57:25.761 INFO:tasks.workunit.client.0.smithi136.stderr:458574: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-15T21:57:25.761 INFO:tasks.workunit.client.0.smithi136.stderr:458574: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-15T21:57:25.761 INFO:tasks.workunit.client.0.smithi136.stderr:458574: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-15T21:57:25.761 INFO:tasks.workunit.client.0.smithi136.stderr:458574: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-15T21:57:25.761 INFO:tasks.workunit.client.0.smithi136.stderr:458574: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-15T21:57:25.761 INFO:tasks.workunit.client.0.smithi136.stderr:458574: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-15T21:57:25.761 INFO:tasks.workunit.client.0.smithi136.stderr:458574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-15T21:57:25.761 INFO:tasks.workunit.client.0.smithi136.stderr:458574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-15T21:57:25.761 INFO:tasks.workunit.client.0.smithi136.stderr:458574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-15T21:57:25.761 INFO:tasks.workunit.client.0.smithi136.stderr:458574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-15T21:57:25.761 INFO:tasks.workunit.client.0.smithi136.stderr:458574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-15T21:57:25.761 INFO:tasks.workunit.client.0.smithi136.stderr:458574: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-15T21:57:25.761 INFO:tasks.workunit.client.0.smithi136.stderr:458574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-15T21:57:25.761 INFO:tasks.workunit.client.0.smithi136.stderr:458574: 1 2024-09-15T21:57:25.761 INFO:tasks.workunit.client.0.smithi136.stderr:458574: 2 2024-09-15T21:57:25.762 INFO:tasks.workunit.client.0.smithi136.stderr:458574: 3' 2024-09-15T21:57:25.762 INFO:tasks.workunit.client.0.smithi136.stderr:458574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-15T21:57:25.762 INFO:tasks.workunit.client.0.smithi136.stderr:458574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T21:57:25.762 INFO:tasks.workunit.client.0.smithi136.stderr:458574: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-15T21:57:25.762 INFO:tasks.workunit.client.0.smithi136.stderr:458574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=146028888068 2024-09-15T21:57:25.762 INFO:tasks.workunit.client.0.smithi136.stderr:458574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 146028888068 2024-09-15T21:57:25.762 INFO:tasks.workunit.client.0.smithi136.stderr:458574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-146028888068' 2024-09-15T21:57:25.762 INFO:tasks.workunit.client.0.smithi136.stderr:458574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T21:57:27.452 INFO:tasks.workunit.client.0.smithi136.stderr:458574: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-hes.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-15T21:57:27.452 INFO:tasks.workunit.client.0.smithi136.stderr:458577: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=42949672988 2024-09-15T21:57:27.452 INFO:tasks.workunit.client.0.smithi136.stderr:458577: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 42949672988 2024-09-15T21:57:27.452 INFO:tasks.workunit.client.0.smithi136.stderr:458577: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-146028888066 1-42949672988' 2024-09-15T21:57:27.452 INFO:tasks.workunit.client.0.smithi136.stderr:458577: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T21:57:27.453 INFO:tasks.workunit.client.0.smithi136.stderr:458577: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-15T21:57:27.453 INFO:tasks.workunit.client.0.smithi136.stderr:458577: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=141733920770 2024-09-15T21:57:27.453 INFO:tasks.workunit.client.0.smithi136.stderr:458577: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 141733920770 2024-09-15T21:57:27.453 INFO:tasks.workunit.client.0.smithi136.stderr:458577: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-146028888066 1-42949672988 2-141733920770' 2024-09-15T21:57:27.453 INFO:tasks.workunit.client.0.smithi136.stderr:458577: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T21:57:27.453 INFO:tasks.workunit.client.0.smithi136.stderr:458577: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-15T21:57:27.453 INFO:tasks.workunit.client.0.smithi136.stderr:458577: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=141733920770 2024-09-15T21:57:27.453 INFO:tasks.workunit.client.0.smithi136.stderr:458577: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 141733920770 2024-09-15T21:57:27.453 INFO:tasks.workunit.client.0.smithi136.stderr:458577: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-146028888066 1-42949672988 2-141733920770 3-141733920770' 2024-09-15T21:57:27.453 INFO:tasks.workunit.client.0.smithi136.stderr:458577: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T21:57:27.453 INFO:tasks.workunit.client.0.smithi136.stderr:458577: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-146028888066 2024-09-15T21:57:27.453 INFO:tasks.workunit.client.0.smithi136.stderr:458577: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T21:57:27.453 INFO:tasks.workunit.client.0.smithi136.stderr:458577: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-15T21:57:27.453 INFO:tasks.workunit.client.0.smithi136.stderr:458577: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-146028888066 2024-09-15T21:57:27.454 INFO:tasks.workunit.client.0.smithi136.stderr:458577: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T21:57:27.454 INFO:tasks.workunit.client.0.smithi136.stderr:458577: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=146028888066 2024-09-15T21:57:27.454 INFO:tasks.workunit.client.0.smithi136.stderr:458577: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 146028888066' 2024-09-15T21:57:27.454 INFO:tasks.workunit.client.0.smithi136.stderr:458577: waiting osd.0 seq 146028888066 2024-09-15T21:57:27.454 INFO:tasks.workunit.client.0.smithi136.stderr:458577: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T21:57:27.454 INFO:tasks.workunit.client.0.smithi136.stderr:458577: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 1 -lt 146028888066 2024-09-15T21:57:27.454 INFO:tasks.workunit.client.0.smithi136.stderr:458577: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-15T21:57:27.454 INFO:tasks.workunit.client.0.smithi136.stderr:458577: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-15T21:57:27.454 INFO:tasks.workunit.client.0.smithi136.stderr:458577: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T21:57:27.454 INFO:tasks.workunit.client.0.smithi136.stderr:458577: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 146028888068 -lt 146028888066 2024-09-15T21:57:27.454 INFO:tasks.workunit.client.0.smithi136.stderr:458577: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T21:57:27.454 INFO:tasks.workunit.client.0.smithi136.stderr:458577: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-42949672988 2024-09-15T21:57:27.454 INFO:tasks.workunit.client.0.smithi136.stderr:458577: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T21:57:27.454 INFO:tasks.workunit.client.0.smithi136.stderr:458577: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-15T21:57:27.454 INFO:tasks.workunit.client.0.smithi136.stderr:458577: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-42949672988 2024-09-15T21:57:27.455 INFO:tasks.workunit.client.0.smithi136.stderr:458577: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T21:57:27.604 INFO:tasks.workunit.client.0.smithi136.stderr:458577: /home/ubuntu/ceppers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-15T21:57:27.605 INFO:tasks.workunit.client.0.smithi136.stderr:458571: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=42949672989 2024-09-15T21:57:27.605 INFO:tasks.workunit.client.0.smithi136.stderr:458571: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 42949672989 2024-09-15T21:57:27.605 INFO:tasks.workunit.client.0.smithi136.stderr:458571: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-146028888067 1-42949672989' 2024-09-15T21:57:27.605 INFO:tasks.workunit.client.0.smithi136.stderr:458571: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T21:57:27.605 INFO:tasks.workunit.client.0.smithi136.stderr:458571: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-15T21:57:27.605 INFO:tasks.workunit.client.0.smithi136.stderr:458571: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=141733920771 2024-09-15T21:57:27.605 INFO:tasks.workunit.client.0.smithi136.stderr:458571: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 141733920771 2024-09-15T21:57:27.605 INFO:tasks.workunit.client.0.smithi136.stderr:458571: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-146028888067 1-42949672989 2-141733920771' 2024-09-15T21:57:27.605 INFO:tasks.workunit.client.0.smithi136.stderr:458571: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T21:57:27.605 INFO:tasks.workunit.client.0.smithi136.stderr:458571: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-15T21:57:27.605 INFO:tasks.workunit.client.0.smithi136.stderr:458571: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=141733920771 2024-09-15T21:57:27.605 INFO:tasks.workunit.client.0.smithi136.stderr:458571: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 141733920771 2024-09-15T21:57:27.606 INFO:tasks.workunit.client.0.smithi136.stderr:458571: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-146028888067 1-42949672989 2-141733920771 3-141733920771' 2024-09-15T21:57:27.606 INFO:tasks.workunit.client.0.smithi136.stderr:458571: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T21:57:27.606 INFO:tasks.workunit.client.0.smithi136.stderr:458571: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-146028888067 2024-09-15T21:57:27.606 INFO:tasks.workunit.client.0.smithi136.stderr:458571: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T21:57:27.606 INFO:tasks.workunit.client.0.smithi136.stderr:458571: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-15T21:57:27.606 INFO:tasks.workunit.client.0.smithi136.stderr:458571: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-146028888067 2024-09-15T21:57:27.606 INFO:tasks.workunit.client.0.smithi136.stderr:458571: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T21:57:27.606 INFO:tasks.workunit.client.0.smithi136.stderr:458571: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=146028888067 2024-09-15T21:57:27.606 INFO:tasks.workunit.client.0.smithi136.stderr:458571: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 146028888067' 2024-09-15T21:57:27.606 INFO:tasks.workunit.client.0.smithi136.stderr:458571: waiting osd.0 seq 146028888067 2024-09-15T21:57:27.606 INFO:tasks.workunit.client.0.smithi136.stderr:458571: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T21:57:27.606 INFO:tasks.workunit.client.0.smithi136.stderr:458571: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 1 -lt 146028888067 2024-09-15T21:57:27.606 INFO:tasks.workunit.client.0.smithi136.stderr:458571: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-15T21:57:27.607 INFO:tasks.workunit.client.0.smithi136.stderr:458571: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-15T21:57:27.607 INFO:tasks.workunit.client.0.smithi136.stderr:458571: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T21:57:27.607 INFO:tasks.workunit.client.0.smithi136.stderr:458571: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 146028888068 -lt 146028888067 2024-09-15T21:57:27.607 INFO:tasks.workunit.client.0.smithi136.stderr:458571: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T21:57:27.607 INFO:tasks.workunit.client.0.smithi136.stderr:458571: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-42949672989 2024-09-15T21:57:27.607 INFO:tasks.workunit.client.0.smithi136.stderr:458571: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T21:57:27.607 INFO:tasks.workunit.client.0.smithi136.stderr:458571: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-15T21:57:27.607 INFO:tasks.workunit.client.0.smithi136.stderr:458571: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-42949672989 2024-09-15T21:57:27.607 INFO:tasks.workunit.client.0.smithi136.stderr:458571: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T21:57:27.700 INFO:tasks.workunit.client.0.smithi136.stderr:458571: /home/ubuntu/lpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-15T21:57:27.700 INFO:tasks.workunit.client.0.smithi136.stderr:458574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=42949672990 2024-09-15T21:57:27.700 INFO:tasks.workunit.client.0.smithi136.stderr:458574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 42949672990 2024-09-15T21:57:27.700 INFO:tasks.workunit.client.0.smithi136.stderr:458574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-146028888068 1-42949672990' 2024-09-15T21:57:27.700 INFO:tasks.workunit.client.0.smithi136.stderr:458574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T21:57:27.700 INFO:tasks.workunit.client.0.smithi136.stderr:458574: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-15T21:57:27.700 INFO:tasks.workunit.client.0.smithi136.stderr:458574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=141733920772 2024-09-15T21:57:27.700 INFO:tasks.workunit.client.0.smithi136.stderr:458574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 141733920772 2024-09-15T21:57:27.701 INFO:tasks.workunit.client.0.smithi136.stderr:458574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-146028888068 1-42949672990 2-141733920772' 2024-09-15T21:57:27.701 INFO:tasks.workunit.client.0.smithi136.stderr:458574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T21:57:27.701 INFO:tasks.workunit.client.0.smithi136.stderr:458574: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-15T21:57:27.701 INFO:tasks.workunit.client.0.smithi136.stderr:458574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=141733920772 2024-09-15T21:57:27.701 INFO:tasks.workunit.client.0.smithi136.stderr:458574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 141733920772 2024-09-15T21:57:27.701 INFO:tasks.workunit.client.0.smithi136.stderr:458574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-146028888068 1-42949672990 2-141733920772 3-141733920772' 2024-09-15T21:57:27.701 INFO:tasks.workunit.client.0.smithi136.stderr:458574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T21:57:27.701 INFO:tasks.workunit.client.0.smithi136.stderr:458574: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-146028888068 2024-09-15T21:57:27.701 INFO:tasks.workunit.client.0.smithi136.stderr:458574: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T21:57:27.701 INFO:tasks.workunit.client.0.smithi136.stderr:458574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-15T21:57:27.701 INFO:tasks.workunit.client.0.smithi136.stderr:458574: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-146028888068 2024-09-15T21:57:27.701 INFO:tasks.workunit.client.0.smithi136.stderr:458574: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T21:57:27.701 INFO:tasks.workunit.client.0.smithi136.stderr:458574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=146028888068 2024-09-15T21:57:27.701 INFO:tasks.workunit.client.0.smithi136.stderr:458574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 146028888068' 2024-09-15T21:57:27.702 INFO:tasks.workunit.client.0.smithi136.stderr:458574: waiting osd.0 seq 146028888068 2024-09-15T21:57:27.702 INFO:tasks.workunit.client.0.smithi136.stderr:458574: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T21:57:27.702 INFO:tasks.workunit.client.0.smithi136.stderr:458574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 1 -lt 146028888068 2024-09-15T21:57:27.702 INFO:tasks.workunit.client.0.smithi136.stderr:458574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-15T21:57:27.702 INFO:tasks.workunit.client.0.smithi136.stderr:458574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-15T21:57:27.702 INFO:tasks.workunit.client.0.smithi136.stderr:458574: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T21:57:27.702 INFO:tasks.workunit.client.0.smithi136.stderr:458574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 146028888068 -lt 146028888068 2024-09-15T21:57:27.702 INFO:tasks.workunit.client.0.smithi136.stderr:458574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T21:57:27.702 INFO:tasks.workunit.client.0.smithi136.stderr:458574: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-42949672990 2024-09-15T21:57:27.702 INFO:tasks.workunit.client.0.smithi136.stderr:458574: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T21:57:27.702 INFO:tasks.workunit.client.0.smithi136.stderr:458574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-15T21:57:27.702 INFO:tasks.workunit.client.0.smithi136.stderr:458574: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-42949672990 2024-09-15T21:57:27.702 INFO:tasks.workunit.client.0.smithi136.stderr:458574: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T21:57:28.906 INFO:tasks.workunit.client.0.smithi136.stderr:458574: /home/ubuntuhtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=42949672988 2024-09-15T21:57:28.906 INFO:tasks.workunit.client.0.smithi136.stderr:458577: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 42949672988' 2024-09-15T21:57:28.906 INFO:tasks.workunit.client.0.smithi136.stderr:458577: waiting osd.1 seq 42949672988 2024-09-15T21:57:28.906 INFO:tasks.workunit.client.0.smithi136.stderr:458577: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-15T21:57:28.906 INFO:tasks.workunit.client.0.smithi136.stderr:458577: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 42949672990 -lt 42949672988 2024-09-15T21:57:28.906 INFO:tasks.workunit.client.0.smithi136.stderr:458577: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T21:57:28.907 INFO:tasks.workunit.client.0.smithi136.stderr:458577: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-141733920770 2024-09-15T21:57:28.907 INFO:tasks.workunit.client.0.smithi136.stderr:458577: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T21:57:28.907 INFO:tasks.workunit.client.0.smithi136.stderr:458577: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-15T21:57:28.907 INFO:tasks.workunit.client.0.smithi136.stderr:458577: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-141733920770 2024-09-15T21:57:28.907 INFO:tasks.workunit.client.0.smithi136.stderr:458577: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T21:57:28.907 INFO:tasks.workunit.client.0.smithi136.stderr:458577: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=141733920770 2024-09-15T21:57:28.907 INFO:tasks.workunit.client.0.smithi136.stderr:458577: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 141733920770' 2024-09-15T21:57:28.907 INFO:tasks.workunit.client.0.smithi136.stderr:458577: waiting osd.2 seq 141733920770 2024-09-15T21:57:28.907 INFO:tasks.workunit.client.0.smithi136.stderr:458577: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-15T21:57:28.907 INFO:tasks.workunit.client.0.smithi136.stderr:458577: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 141733920772 -lt 141733920770 2024-09-15T21:57:28.907 INFO:tasks.workunit.client.0.smithi136.stderr:458577: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T21:57:28.907 INFO:tasks.workunit.client.0.smithi136.stderr:458577: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 3-141733920770 2024-09-15T21:57:28.907 INFO:tasks.workunit.client.0.smithi136.stderr:458577: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T21:57:28.907 INFO:tasks.workunit.client.0.smithi136.stderr:458577: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=3 2024-09-15T21:57:28.908 INFO:tasks.workunit.client.0.smithi136.stderr:458577: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 3-141733920770 2024-09-15T21:57:28.908 INFO:tasks.workunit.client.0.smithi136.stderr:458577: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T21:57:28.908 INFO:tasks.workunit.client.0.smithi136.stderr:458577: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=141733920770 2024-09-15T21:57:28.908 INFO:tasks.workunit.client.0.smithi136.stderr:458577: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.3 seq 141733920770' 2024-09-15T21:57:28.908 INFO:tasks.workunit.client.0.smithi136.stderr:458577: waiting osd.3 seq 141733920770 2024-09-15T21:57:28.908 INFO:tasks.workunit.client.0.smithi136.stderr:458577: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-15T21:57:28.908 INFO:tasks.workunit.client.0.smithi136.stderr:458577: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 141733920772 -lt 141733920770 2024-09-15T21:57:28.908 INFO:tasks.workunit.client.0.smithi136.stderr:458577: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-15T21:57:28.908 INFO:tasks.workunit.client.0.smithi136.stderr:458577: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T21:57:28.908 INFO:tasks.workunit.client.0.smithi136.stderr:458577: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T21:57:28.908 INFO:tasks.workunit.client.0.smithi136.stderr:458577: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 1 == 0 2024-09-15T21:57:28.908 INFO:tasks.workunit.client.0.smithi136.stderr:458577: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-15T21:57:28.908 INFO:tasks.workunit.client.0.smithi136.stderr:458577: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-15T21:57:28.909 INFO:tasks.workunit.client.0.smithi136.stderr:458577: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-15T21:57:28.909 INFO:tasks.workunit.client.0.smithi136.stderr:458577: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T21:57:28.965 INFO:tasks.workunit.client.0.smithi136.stderr:458577: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=42949672990 2024-09-15T21:57:28.965 INFO:tasks.workunit.client.0.smithi136.stderr:458574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 42949672990' 2024-09-15T21:57:28.965 INFO:tasks.workunit.client.0.smithi136.stderr:458574: waiting osd.1 seq 42949672990 2024-09-15T21:57:28.965 INFO:tasks.workunit.client.0.smithi136.stderr:458574: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-15T21:57:28.965 INFO:tasks.workunit.client.0.smithi136.stderr:458574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 42949672990 -lt 42949672990 2024-09-15T21:57:28.965 INFO:tasks.workunit.client.0.smithi136.stderr:458574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T21:57:28.965 INFO:tasks.workunit.client.0.smithi136.stderr:458574: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-141733920772 2024-09-15T21:57:28.965 INFO:tasks.workunit.client.0.smithi136.stderr:458574: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T21:57:28.965 INFO:tasks.workunit.client.0.smithi136.stderr:458574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-15T21:57:28.966 INFO:tasks.workunit.client.0.smithi136.stderr:458574: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-141733920772 2024-09-15T21:57:28.966 INFO:tasks.workunit.client.0.smithi136.stderr:458574: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T21:57:28.966 INFO:tasks.workunit.client.0.smithi136.stderr:458574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=141733920772 2024-09-15T21:57:28.966 INFO:tasks.workunit.client.0.smithi136.stderr:458574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 141733920772' 2024-09-15T21:57:28.966 INFO:tasks.workunit.client.0.smithi136.stderr:458574: waiting osd.2 seq 141733920772 2024-09-15T21:57:28.966 INFO:tasks.workunit.client.0.smithi136.stderr:458574: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-15T21:57:28.966 INFO:tasks.workunit.client.0.smithi136.stderr:458574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 141733920772 -lt 141733920772 2024-09-15T21:57:28.966 INFO:tasks.workunit.client.0.smithi136.stderr:458574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T21:57:28.966 INFO:tasks.workunit.client.0.smithi136.stderr:458574: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 3-141733920772 2024-09-15T21:57:28.966 INFO:tasks.workunit.client.0.smithi136.stderr:458574: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T21:57:28.966 INFO:tasks.workunit.client.0.smithi136.stderr:458574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=3 2024-09-15T21:57:28.966 INFO:tasks.workunit.client.0.smithi136.stderr:458574: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 3-141733920772 2024-09-15T21:57:28.966 INFO:tasks.workunit.client.0.smithi136.stderr:458574: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T21:57:28.967 INFO:tasks.workunit.client.0.smithi136.stderr:458574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=141733920772 2024-09-15T21:57:28.967 INFO:tasks.workunit.client.0.smithi136.stderr:458574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.3 seq 141733920772' 2024-09-15T21:57:28.967 INFO:tasks.workunit.client.0.smithi136.stderr:458574: waiting osd.3 seq 141733920772 2024-09-15T21:57:28.967 INFO:tasks.workunit.client.0.smithi136.stderr:458574: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-15T21:57:28.967 INFO:tasks.workunit.client.0.smithi136.stderr:458574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 141733920772 -lt 141733920772 2024-09-15T21:57:28.967 INFO:tasks.workunit.client.0.smithi136.stderr:458574: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-15T21:57:28.967 INFO:tasks.workunit.client.0.smithi136.stderr:458574: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T21:57:28.967 INFO:tasks.workunit.client.0.smithi136.stderr:458574: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T21:57:28.967 INFO:tasks.workunit.client.0.smithi136.stderr:458574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 1 == 0 2024-09-15T21:57:28.967 INFO:tasks.workunit.client.0.smithi136.stderr:458574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-15T21:57:28.967 INFO:tasks.workunit.client.0.smithi136.stderr:458574: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-15T21:57:28.967 INFO:tasks.workunit.client.0.smithi136.stderr:458574: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-15T21:57:28.967 INFO:tasks.workunit.client.0.smithi136.stderr:458574: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T21:57:29.053 INFO:tasks.workunit.client.0.smithi136.stderr:458574: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=42949672989 2024-09-15T21:57:29.053 INFO:tasks.workunit.client.0.smithi136.stderr:458571: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 42949672989' 2024-09-15T21:57:29.053 INFO:tasks.workunit.client.0.smithi136.stderr:458571: waiting osd.1 seq 42949672989 2024-09-15T21:57:29.053 INFO:tasks.workunit.client.0.smithi136.stderr:458571: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-15T21:57:29.053 INFO:tasks.workunit.client.0.smithi136.stderr:458571: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 42949672990 -lt 42949672989 2024-09-15T21:57:29.053 INFO:tasks.workunit.client.0.smithi136.stderr:458571: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T21:57:29.053 INFO:tasks.workunit.client.0.smithi136.stderr:458571: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-141733920771 2024-09-15T21:57:29.053 INFO:tasks.workunit.client.0.smithi136.stderr:458571: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T21:57:29.053 INFO:tasks.workunit.client.0.smithi136.stderr:458571: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-15T21:57:29.053 INFO:tasks.workunit.client.0.smithi136.stderr:458571: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-141733920771 2024-09-15T21:57:29.053 INFO:tasks.workunit.client.0.smithi136.stderr:458571: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T21:57:29.053 INFO:tasks.workunit.client.0.smithi136.stderr:458571: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=141733920771 2024-09-15T21:57:29.053 INFO:tasks.workunit.client.0.smithi136.stderr:458571: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 141733920771' 2024-09-15T21:57:29.054 INFO:tasks.workunit.client.0.smithi136.stderr:458571: waiting osd.2 seq 141733920771 2024-09-15T21:57:29.054 INFO:tasks.workunit.client.0.smithi136.stderr:458571: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-15T21:57:29.054 INFO:tasks.workunit.client.0.smithi136.stderr:458571: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 141733920772 -lt 141733920771 2024-09-15T21:57:29.054 INFO:tasks.workunit.client.0.smithi136.stderr:458571: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T21:57:29.054 INFO:tasks.workunit.client.0.smithi136.stderr:458571: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 3-141733920771 2024-09-15T21:57:29.054 INFO:tasks.workunit.client.0.smithi136.stderr:458571: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T21:57:29.054 INFO:tasks.workunit.client.0.smithi136.stderr:458571: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=3 2024-09-15T21:57:29.054 INFO:tasks.workunit.client.0.smithi136.stderr:458571: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 3-141733920771 2024-09-15T21:57:29.054 INFO:tasks.workunit.client.0.smithi136.stderr:458571: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T21:57:29.054 INFO:tasks.workunit.client.0.smithi136.stderr:458571: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=141733920771 2024-09-15T21:57:29.054 INFO:tasks.workunit.client.0.smithi136.stderr:458571: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.3 seq 141733920771' 2024-09-15T21:57:29.054 INFO:tasks.workunit.client.0.smithi136.stderr:458571: waiting osd.3 seq 141733920771 2024-09-15T21:57:29.054 INFO:tasks.workunit.client.0.smithi136.stderr:458571: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-15T21:57:29.055 INFO:tasks.workunit.client.0.smithi136.stderr:458571: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 141733920772 -lt 141733920771 2024-09-15T21:57:29.055 INFO:tasks.workunit.client.0.smithi136.stderr:458571: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-15T21:57:29.055 INFO:tasks.workunit.client.0.smithi136.stderr:458571: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T21:57:29.055 INFO:tasks.workunit.client.0.smithi136.stderr:458571: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T21:57:29.055 INFO:tasks.workunit.client.0.smithi136.stderr:458571: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 1 == 0 2024-09-15T21:57:29.055 INFO:tasks.workunit.client.0.smithi136.stderr:458571: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-15T21:57:29.055 INFO:tasks.workunit.client.0.smithi136.stderr:458571: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-15T21:57:29.055 INFO:tasks.workunit.client.0.smithi136.stderr:458571: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-15T21:57:29.055 INFO:tasks.workunit.client.0.smithi136.stderr:458571: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T21:57:29.619 INFO:tasks.workunit.client.0.smithi136.stderr:458571: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | n' 2024-09-15T21:57:29.620 INFO:tasks.workunit.client.0.smithi136.stderr:458577: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T21:57:29.620 INFO:tasks.workunit.client.0.smithi136.stderr:458577: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T21:57:29.620 INFO:tasks.workunit.client.0.smithi136.stderr:458577: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=1 2024-09-15T21:57:29.620 INFO:tasks.workunit.client.0.smithi136.stderr:458577: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-15T21:57:29.620 INFO:tasks.workunit.client.0.smithi136.stderr:458577: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T21:57:29.620 INFO:tasks.workunit.client.0.smithi136.stderr:458577: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T21:57:29.620 INFO:tasks.workunit.client.0.smithi136.stderr:458577: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 = 1 2024-09-15T21:57:29.620 INFO:tasks.workunit.client.0.smithi136.stderr:458577: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-15T21:57:29.620 INFO:tasks.workunit.client.0.smithi136.stderr:458577: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-15T21:57:29.620 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: run_in_background: return 0 2024-09-15T21:57:29.648 INFO:tasks.workunit.client.0.smithi136.stderr:not)' 2024-09-15T21:57:29.648 INFO:tasks.workunit.client.0.smithi136.stderr:458574: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T21:57:29.648 INFO:tasks.workunit.client.0.smithi136.stderr:458574: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T21:57:29.648 INFO:tasks.workunit.client.0.smithi136.stderr:458574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=1 2024-09-15T21:57:29.649 INFO:tasks.workunit.client.0.smithi136.stderr:458574: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-15T21:57:29.649 INFO:tasks.workunit.client.0.smithi136.stderr:458574: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T21:57:29.649 INFO:tasks.workunit.client.0.smithi136.stderr:458574: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T21:57:29.649 INFO:tasks.workunit.client.0.smithi136.stderr:458574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 = 1 2024-09-15T21:57:29.649 INFO:tasks.workunit.client.0.smithi136.stderr:458574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-15T21:57:29.649 INFO:tasks.workunit.client.0.smithi136.stderr:458574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-15T21:57:29.649 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: run_in_background: return 0 2024-09-15T21:57:29.775 INFO:tasks.workunit.client.0.smithi136.stderr:ot)' 2024-09-15T21:57:29.775 INFO:tasks.workunit.client.0.smithi136.stderr:458571: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T21:57:29.775 INFO:tasks.workunit.client.0.smithi136.stderr:458571: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T21:57:29.775 INFO:tasks.workunit.client.0.smithi136.stderr:458571: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=1 2024-09-15T21:57:29.775 INFO:tasks.workunit.client.0.smithi136.stderr:458571: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-15T21:57:29.775 INFO:tasks.workunit.client.0.smithi136.stderr:458571: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T21:57:29.775 INFO:tasks.workunit.client.0.smithi136.stderr:458571: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T21:57:29.775 INFO:tasks.workunit.client.0.smithi136.stderr:458571: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 = 1 2024-09-15T21:57:29.775 INFO:tasks.workunit.client.0.smithi136.stderr:458571: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-15T21:57:29.775 INFO:tasks.workunit.client.0.smithi136.stderr:458571: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-15T21:57:29.776 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: run_in_background: return 0 2024-09-15T21:57:29.776 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: wait_background: for pid in $pids 2024-09-15T21:57:29.776 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2175: wait_background: wait 458569 2024-09-15T21:57:29.776 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: wait_background: for pid in $pids 2024-09-15T21:57:29.776 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2175: wait_background: wait 458572 2024-09-15T21:57:29.776 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: wait_background: eval 'pids='\'''\''' 2024-09-15T21:57:29.776 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: wait_background: pids= 2024-09-15T21:57:29.776 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: wait_background: return 0 2024-09-15T21:57:29.776 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:881: unfound_erasure_coded: return_code=0 2024-09-15T21:57:29.776 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:882: unfound_erasure_coded: '[' 0 -ne 0 ']' 2024-09-15T21:57:29.776 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:887: unfound_erasure_coded: get_pg ecpool SOMETHING 2024-09-15T21:57:29.776 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1084: get_pg: local poolname=ecpool 2024-09-15T21:57:29.776 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1085: get_pg: local objectname=SOMETHING 2024-09-15T21:57:29.776 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1087: get_pg: ceph --format json osd map ecpool SOMETHING 2024-09-15T21:57:29.776 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1087: get_pg: jq -r .pgid 2024-09-15T21:57:30.054 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:887: unfound_erasure_coded: local pg=1.0 2024-09-15T21:57:30.055 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:888: unfound_erasure_coded: repair 1.0 2024-09-15T21:57:30.055 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1846: repair: local pgid=1.0 2024-09-15T21:57:30.055 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1847: repair: get_last_scrub_stamp 1.0 2024-09-15T21:57:30.055 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-15T21:57:30.055 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-15T21:57:30.055 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-15T21:57:30.055 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-15T21:57:30.326 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1847: repair: local last_scrub=2024-09-15T21:57:06.620442+0000 2024-09-15T21:57:30.327 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1848: repair: ceph pg repair 1.0 2024-09-15T21:57:30.589 INFO:tasks.workunit.client.0.smithi136.stderr:instructing pg 1.0s0 on osd.1 to repair 2024-09-15T21:57:30.601 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1849: repair: wait_for_scrub 1.0 2024-09-15T21:57:06.620442+0000 2024-09-15T21:57:30.601 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: local pgid=1.0 2024-09-15T21:57:30.601 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2018: wait_for_scrub: local last_scrub=2024-09-15T21:57:06.620442+0000 2024-09-15T21:57:30.601 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2019: wait_for_scrub: local sname=last_scrub_stamp 2024-09-15T21:57:30.601 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i=0 )) 2024-09-15T21:57:30.601 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-15T21:57:30.602 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-15T21:57:30.602 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-15T21:57:30.602 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-15T21:57:30.602 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-15T21:57:30.602 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-15T21:57:30.877 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-15T21:57:06.620442+0000 '>' 2024-09-15T21:57:06.620442+0000 2024-09-15T21:57:30.877 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-15T21:57:31.878 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-15T21:57:31.878 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-15T21:57:31.879 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-15T21:57:31.879 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-15T21:57:31.879 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-15T21:57:31.880 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-15T21:57:31.880 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-15T21:57:32.160 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-15T21:57:06.620442+0000 '>' 2024-09-15T21:57:06.620442+0000 2024-09-15T21:57:32.160 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-15T21:57:33.161 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-15T21:57:33.161 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-15T21:57:33.162 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-15T21:57:33.162 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-15T21:57:33.162 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-15T21:57:33.162 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-15T21:57:33.162 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-15T21:57:33.447 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-15T21:57:06.620442+0000 '>' 2024-09-15T21:57:06.620442+0000 2024-09-15T21:57:33.448 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-15T21:57:34.448 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-15T21:57:34.449 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-15T21:57:34.449 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-15T21:57:34.449 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-15T21:57:34.449 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-15T21:57:34.450 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-15T21:57:34.450 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-15T21:57:34.734 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-15T21:57:06.620442+0000 '>' 2024-09-15T21:57:06.620442+0000 2024-09-15T21:57:34.734 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-15T21:57:35.735 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-15T21:57:35.735 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-15T21:57:35.736 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-15T21:57:35.736 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-15T21:57:35.736 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-15T21:57:35.737 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-15T21:57:35.737 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-15T21:57:36.018 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-15T21:57:06.620442+0000 '>' 2024-09-15T21:57:06.620442+0000 2024-09-15T21:57:36.019 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-15T21:57:37.020 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-15T21:57:37.020 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-15T21:57:37.021 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-15T21:57:37.021 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-15T21:57:37.021 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-15T21:57:37.021 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-15T21:57:37.021 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-15T21:57:37.297 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-15T21:57:31.344939+0000 '>' 2024-09-15T21:57:06.620442+0000 2024-09-15T21:57:37.298 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2023: wait_for_scrub: return 0 2024-09-15T21:57:37.298 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1850: repair: sleep 2 2024-09-15T21:57:39.300 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:893: unfound_erasure_coded: seq 1 60 2024-09-15T21:57:39.300 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:893: unfound_erasure_coded: for f in `seq 1 60` 2024-09-15T21:57:39.301 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:894: unfound_erasure_coded: ceph -s 2024-09-15T21:57:39.301 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:894: unfound_erasure_coded: grep '1/1 objects unfound' 2024-09-15T21:57:39.652 INFO:tasks.workunit.client.0.smithi136.stdout: 1/1 objects unfound (100.000%) 2024-09-15T21:57:39.652 INFO:tasks.workunit.client.0.smithi136.stdout: 1/1 objects unfound (100.000%) 2024-09-15T21:57:39.652 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:894: unfound_erasure_coded: break 2024-09-15T21:57:39.653 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:897: unfound_erasure_coded: ceph -s 2024-09-15T21:57:39.653 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:897: unfound_erasure_coded: grep '4 up' 2024-09-15T21:57:39.992 INFO:tasks.workunit.client.0.smithi136.stdout: osd: 4 osds: 4 up (since 15s), 4 in (since 69s) 2024-09-15T21:57:39.993 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:898: unfound_erasure_coded: ceph -s 2024-09-15T21:57:39.993 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:898: unfound_erasure_coded: grep '4 in' 2024-09-15T21:57:40.336 INFO:tasks.workunit.client.0.smithi136.stdout: osd: 4 osds: 4 up (since 15s), 4 in (since 70s) 2024-09-15T21:57:40.336 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:899: unfound_erasure_coded: ceph -s 2024-09-15T21:57:40.336 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:899: unfound_erasure_coded: grep '1/1 objects unfound' 2024-09-15T21:57:40.683 INFO:tasks.workunit.client.0.smithi136.stdout: 1/1 objects unfound (100.000%) 2024-09-15T21:57:40.683 INFO:tasks.workunit.client.0.smithi136.stdout: 1/1 objects unfound (100.000%) 2024-09-15T21:57:40.684 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:66: run: teardown td/osd-scrub-repair 2024-09-15T21:57:40.684 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/osd-scrub-repair 2024-09-15T21:57:40.684 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-15T21:57:40.684 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/osd-scrub-repair KILL 2024-09-15T21:57:40.684 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-15T21:57:40.684 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-15T21:57:40.684 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-15T21:57:40.684 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-15T21:57:40.684 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-15T21:57:40.793 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-15T21:57:40.794 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-15T21:57:40.795 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-15T21:57:40.795 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-15T21:57:40.796 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-15T21:57:40.796 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-15T21:57:40.796 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-15T21:57:40.797 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T21:57:40.797 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-15T21:57:40.798 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-15T21:57:40.798 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T21:57:40.799 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-15T21:57:40.800 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-15T21:57:40.800 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/osd-scrub-repair 2024-09-15T21:57:40.821 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-15T21:57:40.821 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T21:57:40.821 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52602 2024-09-15T21:57:40.821 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.52602 2024-09-15T21:57:40.822 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-15T21:57:40.822 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-15T21:57:40.823 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2322: main: code=0 2024-09-15T21:57:40.823 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2326: main: teardown td/osd-scrub-repair 0 2024-09-15T21:57:40.823 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/osd-scrub-repair 2024-09-15T21:57:40.823 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs=0 2024-09-15T21:57:40.823 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/osd-scrub-repair KILL 2024-09-15T21:57:40.823 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-15T21:57:40.823 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-15T21:57:40.823 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-15T21:57:40.823 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-15T21:57:40.823 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-15T21:57:40.824 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-15T21:57:40.825 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-15T21:57:40.826 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-15T21:57:40.826 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-15T21:57:40.827 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-15T21:57:40.827 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-15T21:57:40.828 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-15T21:57:40.829 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T21:57:40.829 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-15T21:57:40.829 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-15T21:57:40.829 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T21:57:40.830 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-15T21:57:40.831 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o 0 = 1 ']' 2024-09-15T21:57:40.831 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/osd-scrub-repair 2024-09-15T21:57:40.832 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-15T21:57:40.832 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T21:57:40.832 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.52602 2024-09-15T21:57:40.832 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.52602 2024-09-15T21:57:40.834 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-15T21:57:40.834 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-15T21:57:40.834 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2327: main: return 0 2024-09-15T21:57:40.834 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-09-15T21:57:40.834 DEBUG:teuthology.orchestra.run.smithi136:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-09-15T21:57:40.861 INFO:tasks.workunit:Running workunit scrub/osd-scrub-snaps.sh... 2024-09-15T21:57:40.861 DEBUG:teuthology.orchestra.run.smithi136:workunit test scrub/osd-scrub-snaps.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=7498493279982dfe87f00616198a5967475f6169 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh 2024-09-15T21:57:40.923 INFO:tasks.workunit.client.0.smithi136.stderr:stty: 'standard input': Inappropriate ioctl for device 2024-09-15T21:57:40.929 INFO:tasks.workunit.client.0.smithi136.stderr:+ PS4='${BASH_SOURCE[0]}:$LINENO: ${FUNCNAME[0]}: ' 2024-09-15T21:57:40.929 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2315: main: export PATH=.:/home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin 2024-09-15T21:57:40.929 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2315: main: PATH=.:/home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin 2024-09-15T21:57:40.929 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2316: main: export PYTHONWARNINGS=ignore 2024-09-15T21:57:40.929 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2316: main: PYTHONWARNINGS=ignore 2024-09-15T21:57:40.929 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2317: main: export CEPH_CONF=/dev/null 2024-09-15T21:57:40.929 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2317: main: CEPH_CONF=/dev/null 2024-09-15T21:57:40.929 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2318: main: unset CEPH_ARGS 2024-09-15T21:57:40.929 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2320: main: local code 2024-09-15T21:57:40.929 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2321: main: run td/osd-scrub-snaps 2024-09-15T21:57:40.930 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:27: run: local dir=td/osd-scrub-snaps 2024-09-15T21:57:40.930 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:28: run: shift 2024-09-15T21:57:40.930 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:30: run: export CEPH_MON=127.0.0.1:7121 2024-09-15T21:57:40.930 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:30: run: CEPH_MON=127.0.0.1:7121 2024-09-15T21:57:40.930 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:31: run: export CEPH_ARGS 2024-09-15T21:57:40.930 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:32: run: uuidgen 2024-09-15T21:57:40.930 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:32: run: CEPH_ARGS+='--fsid=0098e432-2807-49bf-8c16-1e2a347d825d --auth-supported=none ' 2024-09-15T21:57:40.931 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:33: run: CEPH_ARGS+='--mon-host=127.0.0.1:7121 ' 2024-09-15T21:57:40.931 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:35: run: export -n CEPH_CLI_TEST_DUP_COMMAND 2024-09-15T21:57:40.931 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:36: run: set 2024-09-15T21:57:40.931 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:36: run: sed -n -e 's/^\(TEST_[0-9a-z_]*\) .*/\1/p' 2024-09-15T21:57:40.932 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:36: run: local 'funcs=TEST_scrub_snaps 2024-09-15T21:57:40.932 INFO:tasks.workunit.client.0.smithi136.stderr:TEST_scrub_snaps_primary 2024-09-15T21:57:40.932 INFO:tasks.workunit.client.0.smithi136.stderr:TEST_scrub_snaps_replica' 2024-09-15T21:57:40.932 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:37: run: for func in $funcs 2024-09-15T21:57:40.932 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:38: run: setup td/osd-scrub-snaps 2024-09-15T21:57:40.933 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/osd-scrub-snaps 2024-09-15T21:57:40.933 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/osd-scrub-snaps 2024-09-15T21:57:40.933 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/osd-scrub-snaps 2024-09-15T21:57:40.933 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-15T21:57:40.933 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/osd-scrub-snaps KILL 2024-09-15T21:57:40.933 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-15T21:57:40.933 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-15T21:57:40.933 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-15T21:57:40.933 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-15T21:57:40.933 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-15T21:57:40.934 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-15T21:57:40.934 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-15T21:57:40.934 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-15T21:57:40.934 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-15T21:57:40.935 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-15T21:57:40.936 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-15T21:57:40.936 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-15T21:57:40.936 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T21:57:40.936 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-15T21:57:40.937 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-15T21:57:40.937 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T21:57:40.937 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-15T21:57:40.938 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-15T21:57:40.939 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/osd-scrub-snaps 2024-09-15T21:57:40.940 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-15T21:57:40.940 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T21:57:40.940 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.463062 2024-09-15T21:57:40.940 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.463062 2024-09-15T21:57:40.941 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-15T21:57:40.941 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-15T21:57:40.941 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/osd-scrub-snaps 2024-09-15T21:57:40.942 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2024-09-15T21:57:40.942 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T21:57:40.942 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.463062 2024-09-15T21:57:40.942 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.463062 2024-09-15T21:57:40.944 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2024-09-15T21:57:40.944 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 1024 -le 1024 ']' 2024-09-15T21:57:40.945 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:127: setup: ulimit -n 4096 2024-09-15T21:57:40.945 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2024-09-15T21:57:40.945 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/osd-scrub-snaps 1' TERM HUP INT 2024-09-15T21:57:40.945 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:39: run: TEST_scrub_snaps td/osd-scrub-snaps 2024-09-15T21:57:40.945 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:171: TEST_scrub_snaps: local dir=td/osd-scrub-snaps 2024-09-15T21:57:40.945 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:172: TEST_scrub_snaps: local poolname=test 2024-09-15T21:57:40.945 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:173: TEST_scrub_snaps: local OBJS=16 2024-09-15T21:57:40.945 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:174: TEST_scrub_snaps: local OSDS=1 2024-09-15T21:57:40.945 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:176: TEST_scrub_snaps: TESTDATA=testdata.463062 2024-09-15T21:57:40.945 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:178: TEST_scrub_snaps: run_mon td/osd-scrub-snaps a --osd_pool_default_size=1 2024-09-15T21:57:40.945 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/osd-scrub-snaps 2024-09-15T21:57:40.945 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2024-09-15T21:57:40.945 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2024-09-15T21:57:40.945 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2024-09-15T21:57:40.946 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/osd-scrub-snaps/a 2024-09-15T21:57:40.946 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-scrub-snaps/a --run-dir=td/osd-scrub-snaps --osd_pool_default_size=1 2024-09-15T21:57:41.405 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2024-09-15T21:57:41.405 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-15T21:57:41.405 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-15T21:57:41.405 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-15T21:57:41.405 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T21:57:41.405 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.463062 2024-09-15T21:57:41.405 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.463062/$cluster-$name.asok' 2024-09-15T21:57:41.405 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-scrub-snaps/a '--log-file=td/osd-scrub-snaps/$name.log' '--admin-socket=/tmp/ceph-asok.463062/$cluster-$name.asok' --mon-cluster-log-file=td/osd-scrub-snaps/log --run-dir=td/osd-scrub-snaps '--pid-file=td/osd-scrub-snaps/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=1 2024-09-15T21:57:41.597 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2024-09-15T21:57:41.599 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2024-09-15T21:57:41.599 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-15T21:57:41.599 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-15T21:57:41.599 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2024-09-15T21:57:41.599 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2024-09-15T21:57:41.599 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-15T21:57:41.599 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-15T21:57:41.599 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-15T21:57:41.599 INFO:tasks.workunit.client.0.smithi136.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-15T21:57:41.600 INFO:tasks.workunit.client.0.smithi136.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T21:57:41.600 INFO:tasks.workunit.client.0.smithi136.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.463062 2024-09-15T21:57:41.600 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.463062/ceph-mon.a.asok 2024-09-15T21:57:41.600 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-15T21:57:41.600 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.463062/ceph-mon.a.asok config get fsid 2024-09-15T21:57:41.677 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2024-09-15T21:57:41.677 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-15T21:57:41.678 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-15T21:57:41.678 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2024-09-15T21:57:41.678 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2024-09-15T21:57:41.678 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-15T21:57:41.678 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-15T21:57:41.678 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-15T21:57:41.678 INFO:tasks.workunit.client.0.smithi136.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-15T21:57:41.678 INFO:tasks.workunit.client.0.smithi136.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T21:57:41.678 INFO:tasks.workunit.client.0.smithi136.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.463062 2024-09-15T21:57:41.678 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.463062/ceph-mon.a.asok 2024-09-15T21:57:41.678 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-15T21:57:41.678 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.463062/ceph-mon.a.asok config get mon_host 2024-09-15T21:57:41.755 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:179: TEST_scrub_snaps: run_mgr td/osd-scrub-snaps x 2024-09-15T21:57:41.756 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: run_mgr: local dir=td/osd-scrub-snaps 2024-09-15T21:57:41.756 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: run_mgr: shift 2024-09-15T21:57:41.756 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: run_mgr: local id=x 2024-09-15T21:57:41.756 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: run_mgr: shift 2024-09-15T21:57:41.756 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:548: run_mgr: local data=td/osd-scrub-snaps/x 2024-09-15T21:57:41.756 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:550: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-15T21:57:42.558 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: get_asok_path 2024-09-15T21:57:42.558 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-15T21:57:42.558 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-15T21:57:42.558 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-15T21:57:42.558 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T21:57:42.558 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.463062 2024-09-15T21:57:42.558 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.463062/$cluster-$name.asok' 2024-09-15T21:57:42.558 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-15T21:57:42.559 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-scrub-snaps/x '--log-file=td/osd-scrub-snaps/$name.log' '--admin-socket=/tmp/ceph-asok.463062/$cluster-$name.asok' --run-dir=td/osd-scrub-snaps '--pid-file=td/osd-scrub-snaps/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-15T21:57:42.584 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:180: TEST_scrub_snaps: expr 1 - 1 2024-09-15T21:57:42.585 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:180: TEST_scrub_snaps: seq 0 0 2024-09-15T21:57:42.590 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:180: TEST_scrub_snaps: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-09-15T21:57:42.590 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:182: TEST_scrub_snaps: run_osd td/osd-scrub-snaps 0 2024-09-15T21:57:42.590 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-scrub-snaps 2024-09-15T21:57:42.590 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-15T21:57:42.590 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=0 2024-09-15T21:57:42.590 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-15T21:57:42.590 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-scrub-snaps/0 2024-09-15T21:57:42.590 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=0098e432-2807-49bf-8c16-1e2a347d825d --auth-supported=none --mon-host=127.0.0.1:7121 ' 2024-09-15T21:57:42.590 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T21:57:42.591 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T21:57:42.591 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T21:57:42.591 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-scrub-snaps/0' 2024-09-15T21:57:42.591 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-snaps/0/journal' 2024-09-15T21:57:42.591 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-15T21:57:42.591 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-15T21:57:42.591 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-scrub-snaps' 2024-09-15T21:57:42.591 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-15T21:57:42.591 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-15T21:57:42.591 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-15T21:57:42.591 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-15T21:57:42.591 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T21:57:42.591 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.463062 2024-09-15T21:57:42.591 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.463062/$cluster-$name.asok' 2024-09-15T21:57:42.592 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.463062/$cluster-$name.asok' 2024-09-15T21:57:42.592 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-15T21:57:42.592 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-15T21:57:42.592 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-15T21:57:42.592 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-scrub-snaps/$name.log' 2024-09-15T21:57:42.592 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-scrub-snaps/$name.pid' 2024-09-15T21:57:42.592 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T21:57:42.592 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T21:57:42.592 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T21:57:42.592 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T21:57:42.592 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-15T21:57:42.592 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-15T21:57:42.592 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-scrub-snaps/0 2024-09-15T21:57:42.593 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-15T21:57:42.593 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=e7c6e8c2-8578-4b39-b595-2416fda40c44 2024-09-15T21:57:42.593 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd0 e7c6e8c2-8578-4b39-b595-2416fda40c44' 2024-09-15T21:57:42.593 INFO:tasks.workunit.client.0.smithi136.stdout:add osd0 e7c6e8c2-8578-4b39-b595-2416fda40c44 2024-09-15T21:57:42.593 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-15T21:57:42.607 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQBWWOdmwXASJBAA2XUvHx+VFXUXSvNxt5s9sQ== 2024-09-15T21:57:42.607 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQBWWOdmwXASJBAA2XUvHx+VFXUXSvNxt5s9sQ=="}' 2024-09-15T21:57:42.607 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new e7c6e8c2-8578-4b39-b595-2416fda40c44 -i td/osd-scrub-snaps/0/new.json 2024-09-15T21:57:42.805 INFO:tasks.workunit.client.0.smithi136.stdout:0 2024-09-15T21:57:42.815 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-scrub-snaps/0/new.json 2024-09-15T21:57:42.816 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 0 --fsid=0098e432-2807-49bf-8c16-1e2a347d825d --auth-supported=none --mon-host=127.0.0.1:7121 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-snaps/0 --osd-journal=td/osd-scrub-snaps/0/journal --chdir= --run-dir=td/osd-scrub-snaps '--admin-socket=/tmp/ceph-asok.463062/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-snaps/$name.log' '--pid-file=td/osd-scrub-snaps/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBWWOdmwXASJBAA2XUvHx+VFXUXSvNxt5s9sQ== --osd-uuid e7c6e8c2-8578-4b39-b595-2416fda40c44 2024-09-15T21:57:42.842 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:57:42.841+0000 7fe96dba4640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:57:42.864 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:57:42.863+0000 7fe96dba4640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:57:42.865 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:57:42.864+0000 7fe96dba4640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:57:42.866 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:57:42.864+0000 7fe96dba4640 -1 bdev(0x5587a80d5c00 td/osd-scrub-snaps/0/block) open stat got: (1) Operation not permitted 2024-09-15T21:57:42.866 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:57:42.864+0000 7fe96dba4640 -1 bluestore(td/osd-scrub-snaps/0) _read_fsid unparsable uuid 2024-09-15T21:57:48.864 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-scrub-snaps/0/keyring 2024-09-15T21:57:48.864 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-15T21:57:48.865 INFO:tasks.workunit.client.0.smithi136.stdout:adding osd0 key to auth repository 2024-09-15T21:57:48.866 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd0 key to auth repository 2024-09-15T21:57:48.866 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-scrub-snaps/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-15T21:57:49.223 INFO:tasks.workunit.client.0.smithi136.stdout:start osd.0 2024-09-15T21:57:49.224 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.0 2024-09-15T21:57:49.225 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 0 --fsid=0098e432-2807-49bf-8c16-1e2a347d825d --auth-supported=none --mon-host=127.0.0.1:7121 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-snaps/0 --osd-journal=td/osd-scrub-snaps/0/journal --chdir= --run-dir=td/osd-scrub-snaps '--admin-socket=/tmp/ceph-asok.463062/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-snaps/$name.log' '--pid-file=td/osd-scrub-snaps/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T21:57:49.225 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-15T21:57:49.225 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-15T21:57:49.225 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-15T21:57:49.248 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:57:49.246+0000 7faa2f6e4640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:57:49.250 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:57:49.248+0000 7faa2f6e4640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:57:49.251 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:57:49.249+0000 7faa2f6e4640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:57:49.498 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 0 2024-09-15T21:57:49.498 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-15T21:57:49.498 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2024-09-15T21:57:49.498 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-15T21:57:49.498 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-15T21:57:49.498 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:57:49.499 INFO:tasks.workunit.client.0.smithi136.stdout:0 2024-09-15T21:57:49.499 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-15T21:57:49.499 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:57:49.499 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:57:49.773 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:57:50.617 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:57:50.614+0000 7faa2f6e4640 -1 Falling back to public interface 2024-09-15T21:57:50.774 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:57:50.775 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:57:50.775 INFO:tasks.workunit.client.0.smithi136.stdout:1 2024-09-15T21:57:50.775 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-15T21:57:50.775 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:57:50.775 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:57:51.063 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:57:52.064 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:57:52.064 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:57:52.065 INFO:tasks.workunit.client.0.smithi136.stdout:2 2024-09-15T21:57:52.065 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-15T21:57:52.065 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:57:52.065 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:57:52.343 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:57:53.254 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:57:53.253+0000 7faa2f6e4640 -1 osd.0 0 log_to_monitors true 2024-09-15T21:57:53.344 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:57:53.344 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:57:53.344 INFO:tasks.workunit.client.0.smithi136.stdout:3 2024-09-15T21:57:53.345 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-15T21:57:53.345 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:57:53.345 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:57:53.780 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:57:54.782 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:57:54.782 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:57:54.782 INFO:tasks.workunit.client.0.smithi136.stdout:4 2024-09-15T21:57:54.782 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-15T21:57:54.782 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:57:54.783 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:57:55.076 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:57:56.077 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:57:56.078 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:57:56.078 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-15T21:57:56.078 INFO:tasks.workunit.client.0.smithi136.stdout:5 2024-09-15T21:57:56.078 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:57:56.078 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:57:56.362 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:57:57.363 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:57:57.363 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:57:57.364 INFO:tasks.workunit.client.0.smithi136.stdout:6 2024-09-15T21:57:57.364 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-15T21:57:57.364 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:57:57.364 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:57:57.650 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:57:58.651 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:57:58.652 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:57:58.652 INFO:tasks.workunit.client.0.smithi136.stdout:7 2024-09-15T21:57:58.652 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-15T21:57:58.652 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:57:58.653 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:57:58.939 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:57:59.940 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:57:59.940 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:57:59.941 INFO:tasks.workunit.client.0.smithi136.stdout:8 2024-09-15T21:57:59.941 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-15T21:57:59.941 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:57:59.941 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:58:00.288 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:58:01.289 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:58:01.289 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:58:01.289 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-15T21:58:01.290 INFO:tasks.workunit.client.0.smithi136.stdout:9 2024-09-15T21:58:01.290 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:58:01.290 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:58:01.574 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:58:02.576 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:58:02.576 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:58:02.576 INFO:tasks.workunit.client.0.smithi136.stdout:10 2024-09-15T21:58:02.577 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-15T21:58:02.577 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:58:02.577 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:58:02.897 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:58:03.898 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:58:03.899 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:58:03.899 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-15T21:58:03.899 INFO:tasks.workunit.client.0.smithi136.stdout:11 2024-09-15T21:58:03.899 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:58:03.899 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:58:04.262 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:58:05.263 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:58:05.263 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:58:05.263 INFO:tasks.workunit.client.0.smithi136.stdout:12 2024-09-15T21:58:05.264 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-15T21:58:05.264 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:58:05.264 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:58:05.548 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:58:06.549 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:58:06.549 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:58:06.549 INFO:tasks.workunit.client.0.smithi136.stdout:13 2024-09-15T21:58:06.550 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-15T21:58:06.550 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:58:06.550 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:58:07.401 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:58:08.402 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:58:08.402 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:58:08.402 INFO:tasks.workunit.client.0.smithi136.stdout:14 2024-09-15T21:58:08.403 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-15T21:58:08.403 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:58:08.403 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:58:08.871 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:58:09.872 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:58:09.872 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:58:09.873 INFO:tasks.workunit.client.0.smithi136.stdout:15 2024-09-15T21:58:09.873 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-15T21:58:09.873 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:58:09.873 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:58:10.523 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:58:11.524 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:58:11.524 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:58:11.525 INFO:tasks.workunit.client.0.smithi136.stdout:16 2024-09-15T21:58:11.525 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-15T21:58:11.525 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:58:11.525 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:58:11.809 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:58:12.810 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:58:12.811 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:58:12.811 INFO:tasks.workunit.client.0.smithi136.stdout:17 2024-09-15T21:58:12.811 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-15T21:58:12.812 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:58:12.812 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:58:13.300 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:58:14.301 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:58:14.301 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:58:14.301 INFO:tasks.workunit.client.0.smithi136.stdout:18 2024-09-15T21:58:14.302 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 18 2024-09-15T21:58:14.302 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:58:14.302 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:58:14.781 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:58:15.782 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:58:15.783 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:58:15.783 INFO:tasks.workunit.client.0.smithi136.stdout:19 2024-09-15T21:58:15.783 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 19 2024-09-15T21:58:15.783 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:58:15.783 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:58:16.068 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:58:17.069 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:58:17.070 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:58:17.070 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 20 2024-09-15T21:58:17.070 INFO:tasks.workunit.client.0.smithi136.stdout:20 2024-09-15T21:58:17.070 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:58:17.070 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:58:17.351 INFO:tasks.workunit.client.0.smithi136.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/3837349284,v1:127.0.0.1:6803/3837349284] [v2:127.0.0.1:6804/3837349284,v1:127.0.0.1:6805/3837349284] exists,up e7c6e8c2-8578-4b39-b595-2416fda40c44 2024-09-15T21:58:17.351 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-15T21:58:17.352 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-15T21:58:17.352 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-15T21:58:17.352 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:186: TEST_scrub_snaps: ceph osd set noscrub 2024-09-15T21:58:17.722 INFO:tasks.workunit.client.0.smithi136.stderr:noscrub is set 2024-09-15T21:58:17.734 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:187: TEST_scrub_snaps: ceph osd set nodeep-scrub 2024-09-15T21:58:18.141 INFO:tasks.workunit.client.0.smithi136.stderr:nodeep-scrub is set 2024-09-15T21:58:18.152 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:190: TEST_scrub_snaps: create_pool test 1 1 2024-09-15T21:58:18.152 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create test 1 1 2024-09-15T21:58:18.525 INFO:tasks.workunit.client.0.smithi136.stderr:pool 'test' created 2024-09-15T21:58:18.537 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2024-09-15T21:58:19.539 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:191: TEST_scrub_snaps: wait_for_clean 2024-09-15T21:58:19.539 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-15T21:58:19.539 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-15T21:58:19.539 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-15T21:58:19.539 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-15T21:58:19.540 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-15T21:58:19.540 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-15T21:58:19.540 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-15T21:58:19.540 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-15T21:58:19.540 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-15T21:58:19.634 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-15T21:58:19.634 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-15T21:58:19.635 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-15T21:58:19.635 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-15T21:58:19.635 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-15T21:58:19.635 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-15T21:58:19.905 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids=0 2024-09-15T21:58:19.905 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-15T21:58:19.905 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T21:58:19.905 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-15T21:58:20.004 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836484 2024-09-15T21:58:20.005 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836484 2024-09-15T21:58:20.005 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836484' 2024-09-15T21:58:20.005 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T21:58:20.005 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836484 2024-09-15T21:58:20.005 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T21:58:20.005 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-15T21:58:20.006 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836484 2024-09-15T21:58:20.006 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T21:58:20.007 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836484 2024-09-15T21:58:20.007 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836484' 2024-09-15T21:58:20.007 INFO:tasks.workunit.client.0.smithi136.stdout:waiting osd.0 seq 21474836484 2024-09-15T21:58:20.008 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T21:58:20.278 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836483 -lt 21474836484 2024-09-15T21:58:20.279 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-15T21:58:21.280 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-15T21:58:21.280 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T21:58:21.561 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836484 -lt 21474836484 2024-09-15T21:58:21.562 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-15T21:58:21.562 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T21:58:21.562 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T21:58:21.900 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 1 == 0 2024-09-15T21:58:21.900 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-15T21:58:21.900 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-15T21:58:21.900 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-15T21:58:21.900 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T21:58:21.900 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T21:58:21.900 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T21:58:21.901 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T21:58:22.169 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=1 2024-09-15T21:58:22.170 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-15T21:58:22.170 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T21:58:22.170 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T21:58:22.511 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 = 1 2024-09-15T21:58:22.511 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-15T21:58:22.511 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-15T21:58:22.511 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:192: TEST_scrub_snaps: ceph osd dump 2024-09-15T21:58:22.511 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:192: TEST_scrub_snaps: grep '^pool.*['\'']test['\'']' 2024-09-15T21:58:22.512 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:192: TEST_scrub_snaps: awk '{ print $2 }' 2024-09-15T21:58:22.804 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:192: TEST_scrub_snaps: poolid=1 2024-09-15T21:58:22.805 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:194: TEST_scrub_snaps: dd if=/dev/urandom of=testdata.463062 bs=1032 count=1 2024-09-15T21:58:22.805 INFO:tasks.workunit.client.0.smithi136.stderr:1+0 records in 2024-09-15T21:58:22.805 INFO:tasks.workunit.client.0.smithi136.stderr:1+0 records out 2024-09-15T21:58:22.805 INFO:tasks.workunit.client.0.smithi136.stderr:1032 bytes (1.0 kB, 1.0 KiB) copied, 4.7186e-05 s, 21.9 MB/s 2024-09-15T21:58:22.805 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:195: TEST_scrub_snaps: seq 1 16 2024-09-15T21:58:22.806 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:195: TEST_scrub_snaps: for i in `seq 1 $OBJS` 2024-09-15T21:58:22.806 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:197: TEST_scrub_snaps: rados -p test put obj1 testdata.463062 2024-09-15T21:58:22.920 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:195: TEST_scrub_snaps: for i in `seq 1 $OBJS` 2024-09-15T21:58:22.920 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:197: TEST_scrub_snaps: rados -p test put obj2 testdata.463062 2024-09-15T21:58:23.062 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:195: TEST_scrub_snaps: for i in `seq 1 $OBJS` 2024-09-15T21:58:23.062 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:197: TEST_scrub_snaps: rados -p test put obj3 testdata.463062 2024-09-15T21:58:23.138 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:195: TEST_scrub_snaps: for i in `seq 1 $OBJS` 2024-09-15T21:58:23.138 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:197: TEST_scrub_snaps: rados -p test put obj4 testdata.463062 2024-09-15T21:58:23.213 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:195: TEST_scrub_snaps: for i in `seq 1 $OBJS` 2024-09-15T21:58:23.213 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:197: TEST_scrub_snaps: rados -p test put obj5 testdata.463062 2024-09-15T21:58:23.297 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:195: TEST_scrub_snaps: for i in `seq 1 $OBJS` 2024-09-15T21:58:23.297 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:197: TEST_scrub_snaps: rados -p test put obj6 testdata.463062 2024-09-15T21:58:23.372 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:195: TEST_scrub_snaps: for i in `seq 1 $OBJS` 2024-09-15T21:58:23.372 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:197: TEST_scrub_snaps: rados -p test put obj7 testdata.463062 2024-09-15T21:58:23.456 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:195: TEST_scrub_snaps: for i in `seq 1 $OBJS` 2024-09-15T21:58:23.456 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:197: TEST_scrub_snaps: rados -p test put obj8 testdata.463062 2024-09-15T21:58:23.539 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:195: TEST_scrub_snaps: for i in `seq 1 $OBJS` 2024-09-15T21:58:23.540 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:197: TEST_scrub_snaps: rados -p test put obj9 testdata.463062 2024-09-15T21:58:23.631 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:195: TEST_scrub_snaps: for i in `seq 1 $OBJS` 2024-09-15T21:58:23.632 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:197: TEST_scrub_snaps: rados -p test put obj10 testdata.463062 2024-09-15T21:58:23.707 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:195: TEST_scrub_snaps: for i in `seq 1 $OBJS` 2024-09-15T21:58:23.707 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:197: TEST_scrub_snaps: rados -p test put obj11 testdata.463062 2024-09-15T21:58:23.782 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:195: TEST_scrub_snaps: for i in `seq 1 $OBJS` 2024-09-15T21:58:23.782 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:197: TEST_scrub_snaps: rados -p test put obj12 testdata.463062 2024-09-15T21:58:23.865 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:195: TEST_scrub_snaps: for i in `seq 1 $OBJS` 2024-09-15T21:58:23.866 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:197: TEST_scrub_snaps: rados -p test put obj13 testdata.463062 2024-09-15T21:58:23.941 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:195: TEST_scrub_snaps: for i in `seq 1 $OBJS` 2024-09-15T21:58:23.941 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:197: TEST_scrub_snaps: rados -p test put obj14 testdata.463062 2024-09-15T21:58:24.016 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:195: TEST_scrub_snaps: for i in `seq 1 $OBJS` 2024-09-15T21:58:24.016 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:197: TEST_scrub_snaps: rados -p test put obj15 testdata.463062 2024-09-15T21:58:24.091 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:195: TEST_scrub_snaps: for i in `seq 1 $OBJS` 2024-09-15T21:58:24.092 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:197: TEST_scrub_snaps: rados -p test put obj16 testdata.463062 2024-09-15T21:58:24.168 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:200: TEST_scrub_snaps: get_primary test obj1 2024-09-15T21:58:24.168 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1187: get_primary: local poolname=test 2024-09-15T21:58:24.168 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1188: get_primary: local objectname=obj1 2024-09-15T21:58:24.168 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1190: get_primary: ceph --format json osd map test obj1 2024-09-15T21:58:24.168 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1191: get_primary: jq .acting_primary 2024-09-15T21:58:24.447 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:200: TEST_scrub_snaps: local primary=0 2024-09-15T21:58:24.447 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:202: TEST_scrub_snaps: create_scenario td/osd-scrub-snaps test testdata.463062 0 2024-09-15T21:58:24.447 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:45: create_scenario: local dir=td/osd-scrub-snaps 2024-09-15T21:58:24.447 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:46: create_scenario: local poolname=test 2024-09-15T21:58:24.447 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:47: create_scenario: local TESTDATA=testdata.463062 2024-09-15T21:58:24.447 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:48: create_scenario: local osd=0 2024-09-15T21:58:24.447 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:50: create_scenario: SNAP=1 2024-09-15T21:58:24.447 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:51: create_scenario: rados -p test mksnap snap1 2024-09-15T21:58:24.707 INFO:tasks.workunit.client.0.smithi136.stdout:created pool test snap snap1 2024-09-15T21:58:24.709 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:52: create_scenario: dd if=/dev/urandom of=testdata.463062 bs=256 count=1 2024-09-15T21:58:24.710 INFO:tasks.workunit.client.0.smithi136.stderr:1+0 records in 2024-09-15T21:58:24.711 INFO:tasks.workunit.client.0.smithi136.stderr:1+0 records out 2024-09-15T21:58:24.711 INFO:tasks.workunit.client.0.smithi136.stderr:256 bytes copied, 6.3967e-05 s, 4.0 MB/s 2024-09-15T21:58:24.711 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:53: create_scenario: rados -p test put obj1 testdata.463062 2024-09-15T21:58:24.852 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:54: create_scenario: rados -p test put obj5 testdata.463062 2024-09-15T21:58:24.969 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:55: create_scenario: rados -p test put obj3 testdata.463062 2024-09-15T21:58:25.095 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:56: create_scenario: seq 6 14 2024-09-15T21:58:25.096 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:56: create_scenario: for i in `seq 6 14` 2024-09-15T21:58:25.096 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:57: create_scenario: rados -p test put obj6 testdata.463062 2024-09-15T21:58:25.170 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:56: create_scenario: for i in `seq 6 14` 2024-09-15T21:58:25.171 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:57: create_scenario: rados -p test put obj7 testdata.463062 2024-09-15T21:58:25.246 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:56: create_scenario: for i in `seq 6 14` 2024-09-15T21:58:25.246 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:57: create_scenario: rados -p test put obj8 testdata.463062 2024-09-15T21:58:25.321 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:56: create_scenario: for i in `seq 6 14` 2024-09-15T21:58:25.321 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:57: create_scenario: rados -p test put obj9 testdata.463062 2024-09-15T21:58:25.405 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:56: create_scenario: for i in `seq 6 14` 2024-09-15T21:58:25.405 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:57: create_scenario: rados -p test put obj10 testdata.463062 2024-09-15T21:58:25.480 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:56: create_scenario: for i in `seq 6 14` 2024-09-15T21:58:25.480 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:57: create_scenario: rados -p test put obj11 testdata.463062 2024-09-15T21:58:25.555 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:56: create_scenario: for i in `seq 6 14` 2024-09-15T21:58:25.556 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:57: create_scenario: rados -p test put obj12 testdata.463062 2024-09-15T21:58:25.656 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:56: create_scenario: for i in `seq 6 14` 2024-09-15T21:58:25.656 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:57: create_scenario: rados -p test put obj13 testdata.463062 2024-09-15T21:58:25.731 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:56: create_scenario: for i in `seq 6 14` 2024-09-15T21:58:25.731 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:57: create_scenario: rados -p test put obj14 testdata.463062 2024-09-15T21:58:25.806 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:60: create_scenario: SNAP=2 2024-09-15T21:58:25.806 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:61: create_scenario: rados -p test mksnap snap2 2024-09-15T21:58:25.964 INFO:tasks.workunit.client.0.smithi136.stdout:created pool test snap snap2 2024-09-15T21:58:25.966 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:62: create_scenario: dd if=/dev/urandom of=testdata.463062 bs=256 count=2 2024-09-15T21:58:25.967 INFO:tasks.workunit.client.0.smithi136.stderr:2+0 records in 2024-09-15T21:58:25.967 INFO:tasks.workunit.client.0.smithi136.stderr:2+0 records out 2024-09-15T21:58:25.967 INFO:tasks.workunit.client.0.smithi136.stderr:512 bytes copied, 6.2135e-05 s, 8.2 MB/s 2024-09-15T21:58:25.967 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:63: create_scenario: rados -p test put obj5 testdata.463062 2024-09-15T21:58:26.124 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:65: create_scenario: SNAP=3 2024-09-15T21:58:26.125 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:66: create_scenario: rados -p test mksnap snap3 2024-09-15T21:58:26.316 INFO:tasks.workunit.client.0.smithi136.stdout:created pool test snap snap3 2024-09-15T21:58:26.317 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:67: create_scenario: dd if=/dev/urandom of=testdata.463062 bs=256 count=3 2024-09-15T21:58:26.318 INFO:tasks.workunit.client.0.smithi136.stderr:3+0 records in 2024-09-15T21:58:26.318 INFO:tasks.workunit.client.0.smithi136.stderr:3+0 records out 2024-09-15T21:58:26.319 INFO:tasks.workunit.client.0.smithi136.stderr:768 bytes copied, 6.9492e-05 s, 11.1 MB/s 2024-09-15T21:58:26.319 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:68: create_scenario: rados -p test put obj3 testdata.463062 2024-09-15T21:58:26.462 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:70: create_scenario: SNAP=4 2024-09-15T21:58:26.462 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:71: create_scenario: rados -p test mksnap snap4 2024-09-15T21:58:26.758 INFO:tasks.workunit.client.0.smithi136.stdout:created pool test snap snap4 2024-09-15T21:58:26.760 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:72: create_scenario: dd if=/dev/urandom of=testdata.463062 bs=256 count=4 2024-09-15T21:58:26.761 INFO:tasks.workunit.client.0.smithi136.stderr:4+0 records in 2024-09-15T21:58:26.761 INFO:tasks.workunit.client.0.smithi136.stderr:4+0 records out 2024-09-15T21:58:26.761 INFO:tasks.workunit.client.0.smithi136.stderr:1024 bytes (1.0 kB, 1.0 KiB) copied, 7.0066e-05 s, 14.6 MB/s 2024-09-15T21:58:26.761 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:73: create_scenario: rados -p test put obj5 testdata.463062 2024-09-15T21:58:26.877 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:74: create_scenario: rados -p test put obj2 testdata.463062 2024-09-15T21:58:26.986 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:76: create_scenario: SNAP=5 2024-09-15T21:58:26.986 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:77: create_scenario: rados -p test mksnap snap5 2024-09-15T21:58:27.178 INFO:tasks.workunit.client.0.smithi136.stdout:created pool test snap snap5 2024-09-15T21:58:27.180 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:78: create_scenario: SNAP=6 2024-09-15T21:58:27.181 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:79: create_scenario: rados -p test mksnap snap6 2024-09-15T21:58:27.478 INFO:tasks.workunit.client.0.smithi136.stdout:created pool test snap snap6 2024-09-15T21:58:27.480 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:80: create_scenario: dd if=/dev/urandom of=testdata.463062 bs=256 count=6 2024-09-15T21:58:27.481 INFO:tasks.workunit.client.0.smithi136.stderr:6+0 records in 2024-09-15T21:58:27.481 INFO:tasks.workunit.client.0.smithi136.stderr:6+0 records out 2024-09-15T21:58:27.482 INFO:tasks.workunit.client.0.smithi136.stderr:1536 bytes (1.5 kB, 1.5 KiB) copied, 7.4305e-05 s, 20.7 MB/s 2024-09-15T21:58:27.482 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:81: create_scenario: rados -p test put obj5 testdata.463062 2024-09-15T21:58:27.589 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:83: create_scenario: SNAP=7 2024-09-15T21:58:27.590 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:84: create_scenario: rados -p test mksnap snap7 2024-09-15T21:58:27.789 INFO:tasks.workunit.client.0.smithi136.stdout:created pool test snap snap7 2024-09-15T21:58:27.791 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:86: create_scenario: rados -p test rm obj4 2024-09-15T21:58:27.924 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:87: create_scenario: rados -p test rm obj16 2024-09-15T21:58:28.041 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:88: create_scenario: rados -p test rm obj2 2024-09-15T21:58:28.100 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:90: create_scenario: kill_daemons td/osd-scrub-snaps TERM osd 2024-09-15T21:58:28.100 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-15T21:58:28.101 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-15T21:58:28.101 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-15T21:58:28.101 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-15T21:58:28.101 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-15T21:58:28.406 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-15T21:58:28.406 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:94: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 --head --op list obj1 2024-09-15T21:58:29.091 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:94: create_scenario: JSON='["1.0",{"oid":"obj1","key":"","snapid":-2,"hash":1828249343,"max":0,"pool":1,"namespace":"","max":0}]' 2024-09-15T21:58:29.091 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:95: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 '["1.0",{"oid":"obj1","key":"","snapid":-2,"hash":1828249343,"max":0,"pool":1,"namespace":"","max":0}]' --force remove 2024-09-15T21:58:30.456 INFO:tasks.workunit.client.0.smithi136.stdout:WARNING: only removing head with clones present 2024-09-15T21:58:30.456 INFO:tasks.workunit.client.0.smithi136.stdout:remove #1:ff7b1f36:::obj1:head# 2024-09-15T21:58:30.971 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:97: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 --op list obj5 2024-09-15T21:58:30.971 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:97: create_scenario: grep '"snapid":2' 2024-09-15T21:58:31.570 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:97: create_scenario: JSON='["1.0",{"oid":"obj5","key":"","snapid":2,"hash":1718170787,"max":0,"pool":1,"namespace":"","max":0}]' 2024-09-15T21:58:31.570 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:98: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 '["1.0",{"oid":"obj5","key":"","snapid":2,"hash":1718170787,"max":0,"pool":1,"namespace":"","max":0}]' remove 2024-09-15T21:58:32.716 INFO:tasks.workunit.client.0.smithi136.stdout:remove #1:c52c9666:::obj5:2# 2024-09-15T21:58:33.231 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:100: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 --op list obj5 2024-09-15T21:58:33.232 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:100: create_scenario: grep '"snapid":1' 2024-09-15T21:58:33.816 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:100: create_scenario: JSON='["1.0",{"oid":"obj5","key":"","snapid":1,"hash":1718170787,"max":0,"pool":1,"namespace":"","max":0}]' 2024-09-15T21:58:33.816 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:101: create_scenario: OBJ5SAVE='["1.0",{"oid":"obj5","key":"","snapid":1,"hash":1718170787,"max":0,"pool":1,"namespace":"","max":0}]' 2024-09-15T21:58:33.816 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:103: create_scenario: ceph-kvstore-tool bluestore-kv td/osd-scrub-snaps/0 list 2024-09-15T21:58:34.869 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:104: create_scenario: grep SNA_ td/osd-scrub-snaps/drk.log 2024-09-15T21:58:34.870 INFO:tasks.workunit.client.0.smithi136.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.0779578A.7.obj4.. 2024-09-15T21:58:34.870 INFO:tasks.workunit.client.0.smithi136.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.1BB86F16.1.obj3.. 2024-09-15T21:58:34.870 INFO:tasks.workunit.client.0.smithi136.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.268F1DA7.7.obj16.. 2024-09-15T21:58:34.871 INFO:tasks.workunit.client.0.smithi136.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.3A439666.1.obj5.. 2024-09-15T21:58:34.871 INFO:tasks.workunit.client.0.smithi136.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.4CC52438.1.obj6.. 2024-09-15T21:58:34.871 INFO:tasks.workunit.client.0.smithi136.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.5685B6C7.1.obj13.. 2024-09-15T21:58:34.871 INFO:tasks.workunit.client.0.smithi136.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.5AAE7FD8.1.obj8.. 2024-09-15T21:58:34.871 INFO:tasks.workunit.client.0.smithi136.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.5FAF9A3D.1.obj12.. 2024-09-15T21:58:34.871 INFO:tasks.workunit.client.0.smithi136.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.70522149.1.obj14.. 2024-09-15T21:58:34.871 INFO:tasks.workunit.client.0.smithi136.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.77574592.1.obj11.. 2024-09-15T21:58:34.871 INFO:tasks.workunit.client.0.smithi136.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.802EE1F3.4.obj2.. 2024-09-15T21:58:34.871 INFO:tasks.workunit.client.0.smithi136.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.950988C5.1.obj7.. 2024-09-15T21:58:34.871 INFO:tasks.workunit.client.0.smithi136.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.B8CCECA2.1.obj10.. 2024-09-15T21:58:34.872 INFO:tasks.workunit.client.0.smithi136.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.F7CA874E.1.obj9.. 2024-09-15T21:58:34.872 INFO:tasks.workunit.client.0.smithi136.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.FFED8FC6.1.obj1.. 2024-09-15T21:58:34.872 INFO:tasks.workunit.client.0.smithi136.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000002_0000000000000001.0779578A.7.obj4.. 2024-09-15T21:58:34.872 INFO:tasks.workunit.client.0.smithi136.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000002_0000000000000001.1BB86F16.3.obj3.. 2024-09-15T21:58:34.872 INFO:tasks.workunit.client.0.smithi136.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000002_0000000000000001.268F1DA7.7.obj16.. 2024-09-15T21:58:34.872 INFO:tasks.workunit.client.0.smithi136.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000002_0000000000000001.802EE1F3.4.obj2.. 2024-09-15T21:58:34.872 INFO:tasks.workunit.client.0.smithi136.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.0779578A.7.obj4.. 2024-09-15T21:58:34.872 INFO:tasks.workunit.client.0.smithi136.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.1BB86F16.3.obj3.. 2024-09-15T21:58:34.872 INFO:tasks.workunit.client.0.smithi136.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.268F1DA7.7.obj16.. 2024-09-15T21:58:34.872 INFO:tasks.workunit.client.0.smithi136.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.3A439666.4.obj5.. 2024-09-15T21:58:34.872 INFO:tasks.workunit.client.0.smithi136.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.802EE1F3.4.obj2.. 2024-09-15T21:58:34.873 INFO:tasks.workunit.client.0.smithi136.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000004_0000000000000001.0779578A.7.obj4.. 2024-09-15T21:58:34.873 INFO:tasks.workunit.client.0.smithi136.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000004_0000000000000001.268F1DA7.7.obj16.. 2024-09-15T21:58:34.873 INFO:tasks.workunit.client.0.smithi136.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000004_0000000000000001.3A439666.4.obj5.. 2024-09-15T21:58:34.873 INFO:tasks.workunit.client.0.smithi136.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000004_0000000000000001.802EE1F3.4.obj2.. 2024-09-15T21:58:34.873 INFO:tasks.workunit.client.0.smithi136.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000005_0000000000000001.0779578A.7.obj4.. 2024-09-15T21:58:34.873 INFO:tasks.workunit.client.0.smithi136.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000005_0000000000000001.268F1DA7.7.obj16.. 2024-09-15T21:58:34.873 INFO:tasks.workunit.client.0.smithi136.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000005_0000000000000001.3A439666.6.obj5.. 2024-09-15T21:58:34.873 INFO:tasks.workunit.client.0.smithi136.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000005_0000000000000001.802EE1F3.7.obj2.. 2024-09-15T21:58:34.873 INFO:tasks.workunit.client.0.smithi136.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000006_0000000000000001.0779578A.7.obj4.. 2024-09-15T21:58:34.873 INFO:tasks.workunit.client.0.smithi136.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000006_0000000000000001.268F1DA7.7.obj16.. 2024-09-15T21:58:34.873 INFO:tasks.workunit.client.0.smithi136.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000006_0000000000000001.3A439666.6.obj5.. 2024-09-15T21:58:34.873 INFO:tasks.workunit.client.0.smithi136.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000006_0000000000000001.802EE1F3.7.obj2.. 2024-09-15T21:58:34.873 INFO:tasks.workunit.client.0.smithi136.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000007_0000000000000001.0779578A.7.obj4.. 2024-09-15T21:58:34.874 INFO:tasks.workunit.client.0.smithi136.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000007_0000000000000001.268F1DA7.7.obj16.. 2024-09-15T21:58:34.874 INFO:tasks.workunit.client.0.smithi136.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000007_0000000000000001.802EE1F3.7.obj2.. 2024-09-15T21:58:34.874 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:105: create_scenario: grep '^[pm].*SNA_.*[.]1[.]obj5[.][.]$' td/osd-scrub-snaps/drk.log 2024-09-15T21:58:34.874 INFO:tasks.workunit.client.0.smithi136.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.3A439666.1.obj5.. 2024-09-15T21:58:34.874 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:106: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 --rmtype nosnapmap '["1.0",{"oid":"obj5","key":"","snapid":1,"hash":1718170787,"max":0,"pool":1,"namespace":"","max":0}]' remove 2024-09-15T21:58:35.878 INFO:tasks.workunit.client.0.smithi136.stdout:remove #1:c52c9666:::obj5:1# 2024-09-15T21:58:36.393 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:108: create_scenario: ceph-kvstore-tool bluestore-kv td/osd-scrub-snaps/0 list 2024-09-15T21:58:37.448 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:109: create_scenario: grep SNA_ td/osd-scrub-snaps/drk.log 2024-09-15T21:58:37.449 INFO:tasks.workunit.client.0.smithi136.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.0779578A.7.obj4.. 2024-09-15T21:58:37.450 INFO:tasks.workunit.client.0.smithi136.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.1BB86F16.1.obj3.. 2024-09-15T21:58:37.450 INFO:tasks.workunit.client.0.smithi136.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.268F1DA7.7.obj16.. 2024-09-15T21:58:37.450 INFO:tasks.workunit.client.0.smithi136.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.3A439666.1.obj5.. 2024-09-15T21:58:37.450 INFO:tasks.workunit.client.0.smithi136.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.4CC52438.1.obj6.. 2024-09-15T21:58:37.450 INFO:tasks.workunit.client.0.smithi136.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.5685B6C7.1.obj13.. 2024-09-15T21:58:37.450 INFO:tasks.workunit.client.0.smithi136.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.5AAE7FD8.1.obj8.. 2024-09-15T21:58:37.450 INFO:tasks.workunit.client.0.smithi136.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.5FAF9A3D.1.obj12.. 2024-09-15T21:58:37.450 INFO:tasks.workunit.client.0.smithi136.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.70522149.1.obj14.. 2024-09-15T21:58:37.450 INFO:tasks.workunit.client.0.smithi136.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.77574592.1.obj11.. 2024-09-15T21:58:37.450 INFO:tasks.workunit.client.0.smithi136.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.802EE1F3.4.obj2.. 2024-09-15T21:58:37.450 INFO:tasks.workunit.client.0.smithi136.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.950988C5.1.obj7.. 2024-09-15T21:58:37.450 INFO:tasks.workunit.client.0.smithi136.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.B8CCECA2.1.obj10.. 2024-09-15T21:58:37.450 INFO:tasks.workunit.client.0.smithi136.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.F7CA874E.1.obj9.. 2024-09-15T21:58:37.451 INFO:tasks.workunit.client.0.smithi136.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.FFED8FC6.1.obj1.. 2024-09-15T21:58:37.451 INFO:tasks.workunit.client.0.smithi136.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000002_0000000000000001.0779578A.7.obj4.. 2024-09-15T21:58:37.451 INFO:tasks.workunit.client.0.smithi136.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000002_0000000000000001.1BB86F16.3.obj3.. 2024-09-15T21:58:37.451 INFO:tasks.workunit.client.0.smithi136.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000002_0000000000000001.268F1DA7.7.obj16.. 2024-09-15T21:58:37.451 INFO:tasks.workunit.client.0.smithi136.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000002_0000000000000001.802EE1F3.4.obj2.. 2024-09-15T21:58:37.451 INFO:tasks.workunit.client.0.smithi136.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.0779578A.7.obj4.. 2024-09-15T21:58:37.451 INFO:tasks.workunit.client.0.smithi136.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.1BB86F16.3.obj3.. 2024-09-15T21:58:37.451 INFO:tasks.workunit.client.0.smithi136.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.268F1DA7.7.obj16.. 2024-09-15T21:58:37.451 INFO:tasks.workunit.client.0.smithi136.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.3A439666.4.obj5.. 2024-09-15T21:58:37.451 INFO:tasks.workunit.client.0.smithi136.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.802EE1F3.4.obj2.. 2024-09-15T21:58:37.451 INFO:tasks.workunit.client.0.smithi136.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000004_0000000000000001.0779578A.7.obj4.. 2024-09-15T21:58:37.451 INFO:tasks.workunit.client.0.smithi136.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000004_0000000000000001.268F1DA7.7.obj16.. 2024-09-15T21:58:37.451 INFO:tasks.workunit.client.0.smithi136.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000004_0000000000000001.3A439666.4.obj5.. 2024-09-15T21:58:37.451 INFO:tasks.workunit.client.0.smithi136.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000004_0000000000000001.802EE1F3.4.obj2.. 2024-09-15T21:58:37.452 INFO:tasks.workunit.client.0.smithi136.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000005_0000000000000001.0779578A.7.obj4.. 2024-09-15T21:58:37.452 INFO:tasks.workunit.client.0.smithi136.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000005_0000000000000001.268F1DA7.7.obj16.. 2024-09-15T21:58:37.452 INFO:tasks.workunit.client.0.smithi136.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000005_0000000000000001.3A439666.6.obj5.. 2024-09-15T21:58:37.452 INFO:tasks.workunit.client.0.smithi136.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000005_0000000000000001.802EE1F3.7.obj2.. 2024-09-15T21:58:37.452 INFO:tasks.workunit.client.0.smithi136.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000006_0000000000000001.0779578A.7.obj4.. 2024-09-15T21:58:37.452 INFO:tasks.workunit.client.0.smithi136.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000006_0000000000000001.268F1DA7.7.obj16.. 2024-09-15T21:58:37.452 INFO:tasks.workunit.client.0.smithi136.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000006_0000000000000001.3A439666.6.obj5.. 2024-09-15T21:58:37.452 INFO:tasks.workunit.client.0.smithi136.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000006_0000000000000001.802EE1F3.7.obj2.. 2024-09-15T21:58:37.452 INFO:tasks.workunit.client.0.smithi136.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000007_0000000000000001.0779578A.7.obj4.. 2024-09-15T21:58:37.452 INFO:tasks.workunit.client.0.smithi136.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000007_0000000000000001.268F1DA7.7.obj16.. 2024-09-15T21:58:37.452 INFO:tasks.workunit.client.0.smithi136.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000007_0000000000000001.802EE1F3.7.obj2.. 2024-09-15T21:58:37.452 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:110: create_scenario: grep '^[pm].*SNA_.*[.]1[.]obj5[.][.]$' td/osd-scrub-snaps/drk.log 2024-09-15T21:58:37.453 INFO:tasks.workunit.client.0.smithi136.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.3A439666.1.obj5.. 2024-09-15T21:58:37.453 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:111: create_scenario: rm -f td/osd-scrub-snaps/drk.log 2024-09-15T21:58:37.453 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:113: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 --op list obj5 2024-09-15T21:58:37.453 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:113: create_scenario: grep '"snapid":4' 2024-09-15T21:58:38.039 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:113: create_scenario: JSON='["1.0",{"oid":"obj5","key":"","snapid":4,"hash":1718170787,"max":0,"pool":1,"namespace":"","max":0}]' 2024-09-15T21:58:38.039 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:114: create_scenario: dd if=/dev/urandom of=testdata.463062 bs=256 count=18 2024-09-15T21:58:38.040 INFO:tasks.workunit.client.0.smithi136.stderr:18+0 records in 2024-09-15T21:58:38.040 INFO:tasks.workunit.client.0.smithi136.stderr:18+0 records out 2024-09-15T21:58:38.040 INFO:tasks.workunit.client.0.smithi136.stderr:4608 bytes (4.6 kB, 4.5 KiB) copied, 0.000133028 s, 34.6 MB/s 2024-09-15T21:58:38.040 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:115: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 '["1.0",{"oid":"obj5","key":"","snapid":4,"hash":1718170787,"max":0,"pool":1,"namespace":"","max":0}]' set-bytes testdata.463062 2024-09-15T21:58:39.571 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:117: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 --head --op list obj3 2024-09-15T21:58:40.163 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:117: create_scenario: JSON='["1.0",{"oid":"obj3","key":"","snapid":-2,"hash":1643547569,"max":0,"pool":1,"namespace":"","max":0}]' 2024-09-15T21:58:40.163 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:118: create_scenario: dd if=/dev/urandom of=testdata.463062 bs=256 count=15 2024-09-15T21:58:40.164 INFO:tasks.workunit.client.0.smithi136.stderr:15+0 records in 2024-09-15T21:58:40.164 INFO:tasks.workunit.client.0.smithi136.stderr:15+0 records out 2024-09-15T21:58:40.164 INFO:tasks.workunit.client.0.smithi136.stderr:3840 bytes (3.8 kB, 3.8 KiB) copied, 0.000124824 s, 30.8 MB/s 2024-09-15T21:58:40.164 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:119: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 '["1.0",{"oid":"obj3","key":"","snapid":-2,"hash":1643547569,"max":0,"pool":1,"namespace":"","max":0}]' set-bytes testdata.463062 2024-09-15T21:58:41.657 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:121: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 --op list obj4 2024-09-15T21:58:41.657 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:121: create_scenario: grep '"snapid":7' 2024-09-15T21:58:42.248 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:121: create_scenario: JSON='["1.0",{"oid":"obj4","key":"","snapid":7,"hash":2826278768,"max":0,"pool":1,"namespace":"","max":0}]' 2024-09-15T21:58:42.248 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:122: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 '["1.0",{"oid":"obj4","key":"","snapid":7,"hash":2826278768,"max":0,"pool":1,"namespace":"","max":0}]' remove 2024-09-15T21:58:43.470 INFO:tasks.workunit.client.0.smithi136.stdout:remove #1:0ee9ae15:::obj4:7# 2024-09-15T21:58:44.235 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:125: create_scenario: ceph-kvstore-tool bluestore-kv td/osd-scrub-snaps/0 list 2024-09-15T21:58:45.287 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:126: create_scenario: grep SNA_ td/osd-scrub-snaps/drk.log 2024-09-15T21:58:45.289 INFO:tasks.workunit.client.0.smithi136.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.1BB86F16.1.obj3.. 2024-09-15T21:58:45.289 INFO:tasks.workunit.client.0.smithi136.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.268F1DA7.7.obj16.. 2024-09-15T21:58:45.289 INFO:tasks.workunit.client.0.smithi136.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.3A439666.1.obj5.. 2024-09-15T21:58:45.289 INFO:tasks.workunit.client.0.smithi136.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.4CC52438.1.obj6.. 2024-09-15T21:58:45.289 INFO:tasks.workunit.client.0.smithi136.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.5685B6C7.1.obj13.. 2024-09-15T21:58:45.289 INFO:tasks.workunit.client.0.smithi136.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.5AAE7FD8.1.obj8.. 2024-09-15T21:58:45.289 INFO:tasks.workunit.client.0.smithi136.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.5FAF9A3D.1.obj12.. 2024-09-15T21:58:45.289 INFO:tasks.workunit.client.0.smithi136.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.70522149.1.obj14.. 2024-09-15T21:58:45.289 INFO:tasks.workunit.client.0.smithi136.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.77574592.1.obj11.. 2024-09-15T21:58:45.289 INFO:tasks.workunit.client.0.smithi136.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.802EE1F3.4.obj2.. 2024-09-15T21:58:45.289 INFO:tasks.workunit.client.0.smithi136.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.950988C5.1.obj7.. 2024-09-15T21:58:45.289 INFO:tasks.workunit.client.0.smithi136.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.B8CCECA2.1.obj10.. 2024-09-15T21:58:45.290 INFO:tasks.workunit.client.0.smithi136.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.F7CA874E.1.obj9.. 2024-09-15T21:58:45.290 INFO:tasks.workunit.client.0.smithi136.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.FFED8FC6.1.obj1.. 2024-09-15T21:58:45.290 INFO:tasks.workunit.client.0.smithi136.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000002_0000000000000001.1BB86F16.3.obj3.. 2024-09-15T21:58:45.290 INFO:tasks.workunit.client.0.smithi136.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000002_0000000000000001.268F1DA7.7.obj16.. 2024-09-15T21:58:45.290 INFO:tasks.workunit.client.0.smithi136.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000002_0000000000000001.802EE1F3.4.obj2.. 2024-09-15T21:58:45.290 INFO:tasks.workunit.client.0.smithi136.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.1BB86F16.3.obj3.. 2024-09-15T21:58:45.290 INFO:tasks.workunit.client.0.smithi136.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.268F1DA7.7.obj16.. 2024-09-15T21:58:45.290 INFO:tasks.workunit.client.0.smithi136.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.3A439666.4.obj5.. 2024-09-15T21:58:45.290 INFO:tasks.workunit.client.0.smithi136.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.802EE1F3.4.obj2.. 2024-09-15T21:58:45.290 INFO:tasks.workunit.client.0.smithi136.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000004_0000000000000001.268F1DA7.7.obj16.. 2024-09-15T21:58:45.290 INFO:tasks.workunit.client.0.smithi136.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000004_0000000000000001.3A439666.4.obj5.. 2024-09-15T21:58:45.290 INFO:tasks.workunit.client.0.smithi136.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000004_0000000000000001.802EE1F3.4.obj2.. 2024-09-15T21:58:45.290 INFO:tasks.workunit.client.0.smithi136.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000005_0000000000000001.268F1DA7.7.obj16.. 2024-09-15T21:58:45.291 INFO:tasks.workunit.client.0.smithi136.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000005_0000000000000001.3A439666.6.obj5.. 2024-09-15T21:58:45.291 INFO:tasks.workunit.client.0.smithi136.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000005_0000000000000001.802EE1F3.7.obj2.. 2024-09-15T21:58:45.291 INFO:tasks.workunit.client.0.smithi136.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000006_0000000000000001.268F1DA7.7.obj16.. 2024-09-15T21:58:45.291 INFO:tasks.workunit.client.0.smithi136.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000006_0000000000000001.3A439666.6.obj5.. 2024-09-15T21:58:45.291 INFO:tasks.workunit.client.0.smithi136.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000006_0000000000000001.802EE1F3.7.obj2.. 2024-09-15T21:58:45.291 INFO:tasks.workunit.client.0.smithi136.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000007_0000000000000001.268F1DA7.7.obj16.. 2024-09-15T21:58:45.291 INFO:tasks.workunit.client.0.smithi136.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000007_0000000000000001.802EE1F3.7.obj2.. 2024-09-15T21:58:45.291 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:127: create_scenario: grep '^[pm].*SNA_.*[.]7[.]obj16[.][.]$' td/osd-scrub-snaps/drk.log 2024-09-15T21:58:45.291 INFO:tasks.workunit.client.0.smithi136.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.268F1DA7.7.obj16.. 2024-09-15T21:58:45.292 INFO:tasks.workunit.client.0.smithi136.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000002_0000000000000001.268F1DA7.7.obj16.. 2024-09-15T21:58:45.292 INFO:tasks.workunit.client.0.smithi136.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.268F1DA7.7.obj16.. 2024-09-15T21:58:45.292 INFO:tasks.workunit.client.0.smithi136.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000004_0000000000000001.268F1DA7.7.obj16.. 2024-09-15T21:58:45.292 INFO:tasks.workunit.client.0.smithi136.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000005_0000000000000001.268F1DA7.7.obj16.. 2024-09-15T21:58:45.292 INFO:tasks.workunit.client.0.smithi136.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000006_0000000000000001.268F1DA7.7.obj16.. 2024-09-15T21:58:45.292 INFO:tasks.workunit.client.0.smithi136.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000007_0000000000000001.268F1DA7.7.obj16.. 2024-09-15T21:58:45.292 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:128: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 --op list obj16 2024-09-15T21:58:45.292 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:128: create_scenario: grep '"snapid":7' 2024-09-15T21:58:45.871 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:128: create_scenario: JSON='["1.0",{"oid":"obj16","key":"","snapid":7,"hash":2060580962,"max":0,"pool":1,"namespace":"","max":0}]' 2024-09-15T21:58:45.871 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:129: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 --rmtype snapmap '["1.0",{"oid":"obj16","key":"","snapid":7,"hash":2060580962,"max":0,"pool":1,"namespace":"","max":0}]' remove 2024-09-15T21:58:46.957 INFO:tasks.workunit.client.0.smithi136.stdout:remove #1:461f8b5e:::obj16:7# 2024-09-15T21:58:47.471 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:131: create_scenario: ceph-kvstore-tool bluestore-kv td/osd-scrub-snaps/0 list 2024-09-15T21:58:48.527 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:132: create_scenario: grep SNA_ td/osd-scrub-snaps/drk.log 2024-09-15T21:58:48.528 INFO:tasks.workunit.client.0.smithi136.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.1BB86F16.1.obj3.. 2024-09-15T21:58:48.529 INFO:tasks.workunit.client.0.smithi136.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.3A439666.1.obj5.. 2024-09-15T21:58:48.529 INFO:tasks.workunit.client.0.smithi136.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.4CC52438.1.obj6.. 2024-09-15T21:58:48.529 INFO:tasks.workunit.client.0.smithi136.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.5685B6C7.1.obj13.. 2024-09-15T21:58:48.529 INFO:tasks.workunit.client.0.smithi136.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.5AAE7FD8.1.obj8.. 2024-09-15T21:58:48.529 INFO:tasks.workunit.client.0.smithi136.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.5FAF9A3D.1.obj12.. 2024-09-15T21:58:48.529 INFO:tasks.workunit.client.0.smithi136.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.70522149.1.obj14.. 2024-09-15T21:58:48.529 INFO:tasks.workunit.client.0.smithi136.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.77574592.1.obj11.. 2024-09-15T21:58:48.529 INFO:tasks.workunit.client.0.smithi136.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.802EE1F3.4.obj2.. 2024-09-15T21:58:48.529 INFO:tasks.workunit.client.0.smithi136.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.950988C5.1.obj7.. 2024-09-15T21:58:48.529 INFO:tasks.workunit.client.0.smithi136.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.B8CCECA2.1.obj10.. 2024-09-15T21:58:48.529 INFO:tasks.workunit.client.0.smithi136.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.F7CA874E.1.obj9.. 2024-09-15T21:58:48.529 INFO:tasks.workunit.client.0.smithi136.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.FFED8FC6.1.obj1.. 2024-09-15T21:58:48.530 INFO:tasks.workunit.client.0.smithi136.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000002_0000000000000001.1BB86F16.3.obj3.. 2024-09-15T21:58:48.530 INFO:tasks.workunit.client.0.smithi136.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000002_0000000000000001.802EE1F3.4.obj2.. 2024-09-15T21:58:48.530 INFO:tasks.workunit.client.0.smithi136.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.1BB86F16.3.obj3.. 2024-09-15T21:58:48.530 INFO:tasks.workunit.client.0.smithi136.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.3A439666.4.obj5.. 2024-09-15T21:58:48.530 INFO:tasks.workunit.client.0.smithi136.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.802EE1F3.4.obj2.. 2024-09-15T21:58:48.530 INFO:tasks.workunit.client.0.smithi136.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000004_0000000000000001.3A439666.4.obj5.. 2024-09-15T21:58:48.530 INFO:tasks.workunit.client.0.smithi136.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000004_0000000000000001.802EE1F3.4.obj2.. 2024-09-15T21:58:48.530 INFO:tasks.workunit.client.0.smithi136.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000005_0000000000000001.3A439666.6.obj5.. 2024-09-15T21:58:48.530 INFO:tasks.workunit.client.0.smithi136.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000005_0000000000000001.802EE1F3.7.obj2.. 2024-09-15T21:58:48.530 INFO:tasks.workunit.client.0.smithi136.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000006_0000000000000001.3A439666.6.obj5.. 2024-09-15T21:58:48.530 INFO:tasks.workunit.client.0.smithi136.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000006_0000000000000001.802EE1F3.7.obj2.. 2024-09-15T21:58:48.530 INFO:tasks.workunit.client.0.smithi136.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000007_0000000000000001.802EE1F3.7.obj2.. 2024-09-15T21:58:48.531 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:133: create_scenario: grep '^[pm].*SNA_.*[.]7[.]obj16[.][.]$' td/osd-scrub-snaps/drk.log 2024-09-15T21:58:48.531 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:134: create_scenario: rm -f td/osd-scrub-snaps/drk.log 2024-09-15T21:58:48.531 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:136: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 --head --op list obj2 2024-09-15T21:58:49.132 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:136: create_scenario: JSON='["1.0",{"oid":"obj2","key":"","snapid":-2,"hash":1058988552,"max":0,"pool":1,"namespace":"","max":0}]' 2024-09-15T21:58:49.132 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:137: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 '["1.0",{"oid":"obj2","key":"","snapid":-2,"hash":1058988552,"max":0,"pool":1,"namespace":"","max":0}]' rm-attr snapset 2024-09-15T21:58:50.591 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:140: create_scenario: echo '["1.0",{"oid":"obj5","key":"","snapid":1,"hash":1718170787,"max":0,"pool":1,"namespace":"","max":0}]' 2024-09-15T21:58:50.592 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:140: create_scenario: sed 's/snapid":1/snapid":7/' 2024-09-15T21:58:50.593 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:140: create_scenario: JSON='["1.0",{"oid":"obj5","key":"","snapid":7,"hash":1718170787,"max":0,"pool":1,"namespace":"","max":0}]' 2024-09-15T21:58:50.593 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:141: create_scenario: dd if=/dev/urandom of=testdata.463062 bs=256 count=7 2024-09-15T21:58:50.594 INFO:tasks.workunit.client.0.smithi136.stderr:7+0 records in 2024-09-15T21:58:50.594 INFO:tasks.workunit.client.0.smithi136.stderr:7+0 records out 2024-09-15T21:58:50.594 INFO:tasks.workunit.client.0.smithi136.stderr:1792 bytes (1.8 kB, 1.8 KiB) copied, 9.4802e-05 s, 18.9 MB/s 2024-09-15T21:58:50.594 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:142: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 '["1.0",{"oid":"obj5","key":"","snapid":7,"hash":1718170787,"max":0,"pool":1,"namespace":"","max":0}]' set-bytes testdata.463062 2024-09-15T21:58:52.051 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:144: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 --head --op list obj6 2024-09-15T21:58:52.367 INFO:tasks.workunit.client.0.smithi136.stderr:Error getting attr on : 1.0_head,#1:c52c9666:::obj5:7#, (61) No data available 2024-09-15T21:58:52.638 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:144: create_scenario: JSON='["1.0",{"oid":"obj6","key":"","snapid":-2,"hash":2202164420,"max":0,"pool":1,"namespace":"","max":0}]' 2024-09-15T21:58:52.638 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:145: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 '["1.0",{"oid":"obj6","key":"","snapid":-2,"hash":2202164420,"max":0,"pool":1,"namespace":"","max":0}]' clear-snapset 2024-09-15T21:58:54.097 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:146: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 --head --op list obj7 2024-09-15T21:58:54.417 INFO:tasks.workunit.client.0.smithi136.stderr:Error getting attr on : 1.0_head,#1:c52c9666:::obj5:7#, (61) No data available 2024-09-15T21:58:54.682 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:146: create_scenario: JSON='["1.0",{"oid":"obj7","key":"","snapid":-2,"hash":1552453721,"max":0,"pool":1,"namespace":"","max":0}]' 2024-09-15T21:58:54.682 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:147: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 '["1.0",{"oid":"obj7","key":"","snapid":-2,"hash":1552453721,"max":0,"pool":1,"namespace":"","max":0}]' clear-snapset corrupt 2024-09-15T21:58:56.673 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:148: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 --head --op list obj8 2024-09-15T21:58:57.026 INFO:tasks.workunit.client.0.smithi136.stderr:Error getting attr on : 1.0_head,#1:c52c9666:::obj5:7#, (61) No data available 2024-09-15T21:58:57.294 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:148: create_scenario: JSON='["1.0",{"oid":"obj8","key":"","snapid":-2,"hash":2381834917,"max":0,"pool":1,"namespace":"","max":0}]' 2024-09-15T21:58:57.295 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:149: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 '["1.0",{"oid":"obj8","key":"","snapid":-2,"hash":2381834917,"max":0,"pool":1,"namespace":"","max":0}]' clear-snapset seq 2024-09-15T21:58:58.732 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:150: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 --head --op list obj9 2024-09-15T21:58:59.050 INFO:tasks.workunit.client.0.smithi136.stderr:Error getting attr on : 1.0_head,#1:c52c9666:::obj5:7#, (61) No data available 2024-09-15T21:58:59.314 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:150: create_scenario: JSON='["1.0",{"oid":"obj9","key":"","snapid":-2,"hash":3833113727,"max":0,"pool":1,"namespace":"","max":0}]' 2024-09-15T21:58:59.314 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:151: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 '["1.0",{"oid":"obj9","key":"","snapid":-2,"hash":3833113727,"max":0,"pool":1,"namespace":"","max":0}]' clear-snapset clone_size 2024-09-15T21:59:00.703 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:152: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 --head --op list obj10 2024-09-15T21:59:01.025 INFO:tasks.workunit.client.0.smithi136.stderr:Error getting attr on : 1.0_head,#1:c52c9666:::obj5:7#, (61) No data available 2024-09-15T21:59:01.295 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:152: create_scenario: JSON='["1.0",{"oid":"obj10","key":"","snapid":-2,"hash":718195851,"max":0,"pool":1,"namespace":"","max":0}]' 2024-09-15T21:59:01.295 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:153: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 '["1.0",{"oid":"obj10","key":"","snapid":-2,"hash":718195851,"max":0,"pool":1,"namespace":"","max":0}]' clear-snapset clone_overlap 2024-09-15T21:59:03.025 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:154: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 --head --op list obj11 2024-09-15T21:59:03.349 INFO:tasks.workunit.client.0.smithi136.stderr:Error getting attr on : 1.0_head,#1:c52c9666:::obj5:7#, (61) No data available 2024-09-15T21:59:03.614 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:154: create_scenario: JSON='["1.0",{"oid":"obj11","key":"","snapid":-2,"hash":693400951,"max":0,"pool":1,"namespace":"","max":0}]' 2024-09-15T21:59:03.615 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:155: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 '["1.0",{"oid":"obj11","key":"","snapid":-2,"hash":693400951,"max":0,"pool":1,"namespace":"","max":0}]' clear-snapset clones 2024-09-15T21:59:05.017 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:156: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 --head --op list obj12 2024-09-15T21:59:05.340 INFO:tasks.workunit.client.0.smithi136.stderr:Error getting attr on : 1.0_head,#1:c52c9666:::obj5:7#, (61) No data available 2024-09-15T21:59:05.608 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:156: create_scenario: JSON='["1.0",{"oid":"obj12","key":"","snapid":-2,"hash":3551132405,"max":0,"pool":1,"namespace":"","max":0}]' 2024-09-15T21:59:05.609 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:157: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 '["1.0",{"oid":"obj12","key":"","snapid":-2,"hash":3551132405,"max":0,"pool":1,"namespace":"","max":0}]' clear-snapset head 2024-09-15T21:59:07.053 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:158: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 --head --op list obj13 2024-09-15T21:59:07.383 INFO:tasks.workunit.client.0.smithi136.stderr:Error getting attr on : 1.0_head,#1:c52c9666:::obj5:7#, (61) No data available 2024-09-15T21:59:07.647 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:158: create_scenario: JSON='["1.0",{"oid":"obj13","key":"","snapid":-2,"hash":2087409765,"max":0,"pool":1,"namespace":"","max":0}]' 2024-09-15T21:59:07.648 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:159: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 '["1.0",{"oid":"obj13","key":"","snapid":-2,"hash":2087409765,"max":0,"pool":1,"namespace":"","max":0}]' clear-snapset snaps 2024-09-15T21:59:09.050 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:160: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 --head --op list obj14 2024-09-15T21:59:09.374 INFO:tasks.workunit.client.0.smithi136.stderr:Error getting attr on : 1.0_head,#1:c52c9666:::obj5:7#, (61) No data available 2024-09-15T21:59:09.646 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:160: create_scenario: JSON='["1.0",{"oid":"obj14","key":"","snapid":-2,"hash":2484217095,"max":0,"pool":1,"namespace":"","max":0}]' 2024-09-15T21:59:09.646 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:161: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 '["1.0",{"oid":"obj14","key":"","snapid":-2,"hash":2484217095,"max":0,"pool":1,"namespace":"","max":0}]' clear-snapset size 2024-09-15T21:59:11.302 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:163: create_scenario: echo garbage 2024-09-15T21:59:11.303 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:164: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 --head --op list obj15 2024-09-15T21:59:11.624 INFO:tasks.workunit.client.0.smithi136.stderr:Error getting attr on : 1.0_head,#1:c52c9666:::obj5:7#, (61) No data available 2024-09-15T21:59:11.889 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:164: create_scenario: JSON='["1.0",{"oid":"obj15","key":"","snapid":-2,"hash":612772309,"max":0,"pool":1,"namespace":"","max":0}]' 2024-09-15T21:59:11.889 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:165: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 '["1.0",{"oid":"obj15","key":"","snapid":-2,"hash":612772309,"max":0,"pool":1,"namespace":"","max":0}]' set-attr snapset td/osd-scrub-snaps/bad 2024-09-15T21:59:13.478 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:166: create_scenario: rm -f td/osd-scrub-snaps/bad 2024-09-15T21:59:13.479 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:167: create_scenario: return 0 2024-09-15T21:59:13.479 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:204: TEST_scrub_snaps: rm -f testdata.463062 2024-09-15T21:59:13.480 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:206: TEST_scrub_snaps: expr 1 - 1 2024-09-15T21:59:13.482 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:206: TEST_scrub_snaps: seq 0 0 2024-09-15T21:59:13.485 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:206: TEST_scrub_snaps: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-09-15T21:59:13.486 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:208: TEST_scrub_snaps: activate_osd td/osd-scrub-snaps 0 2024-09-15T21:59:13.486 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-snaps 2024-09-15T21:59:13.486 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-15T21:59:13.486 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=0 2024-09-15T21:59:13.486 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-15T21:59:13.486 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-snaps/0 2024-09-15T21:59:13.486 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=0098e432-2807-49bf-8c16-1e2a347d825d --auth-supported=none --mon-host=127.0.0.1:7121 ' 2024-09-15T21:59:13.486 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T21:59:13.486 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T21:59:13.486 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T21:59:13.486 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-snaps/0' 2024-09-15T21:59:13.486 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-snaps/0/journal' 2024-09-15T21:59:13.487 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-15T21:59:13.487 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-15T21:59:13.487 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-snaps' 2024-09-15T21:59:13.487 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-15T21:59:13.487 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-15T21:59:13.487 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-15T21:59:13.487 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-15T21:59:13.487 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T21:59:13.487 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.463062 2024-09-15T21:59:13.487 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.463062/$cluster-$name.asok' 2024-09-15T21:59:13.487 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.463062/$cluster-$name.asok' 2024-09-15T21:59:13.487 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-15T21:59:13.487 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-snaps/$name.log' 2024-09-15T21:59:13.487 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-snaps/$name.pid' 2024-09-15T21:59:13.488 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T21:59:13.488 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T21:59:13.488 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T21:59:13.488 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T21:59:13.488 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-15T21:59:13.488 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-15T21:59:13.488 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-snaps/0 2024-09-15T21:59:13.488 INFO:tasks.workunit.client.0.smithi136.stdout:start osd.0 2024-09-15T21:59:13.489 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.0 2024-09-15T21:59:13.489 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 0 --fsid=0098e432-2807-49bf-8c16-1e2a347d825d --auth-supported=none --mon-host=127.0.0.1:7121 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-snaps/0 --osd-journal=td/osd-scrub-snaps/0/journal --chdir= --run-dir=td/osd-scrub-snaps '--admin-socket=/tmp/ceph-asok.463062/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-snaps/$name.log' '--pid-file=td/osd-scrub-snaps/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T21:59:13.489 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-snaps/0/whoami 2024-09-15T21:59:13.489 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 0 = 0 ']' 2024-09-15T21:59:13.489 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-15T21:59:13.489 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-15T21:59:13.490 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-15T21:59:13.514 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:59:13.512+0000 7f3463b4d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:59:13.516 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:59:13.514+0000 7f3463b4d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:59:13.517 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:59:13.516+0000 7f3463b4d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:59:13.770 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 0 2024-09-15T21:59:13.770 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-15T21:59:13.770 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2024-09-15T21:59:13.770 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-15T21:59:13.770 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-15T21:59:13.770 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:59:13.770 INFO:tasks.workunit.client.0.smithi136.stdout:0 2024-09-15T21:59:13.771 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-15T21:59:13.771 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:59:13.771 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:59:14.044 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:59:15.045 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:59:15.045 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:59:15.045 INFO:tasks.workunit.client.0.smithi136.stdout:1 2024-09-15T21:59:15.046 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-15T21:59:15.046 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:59:15.046 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:59:15.131 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:59:15.130+0000 7f3463b4d640 -1 Falling back to public interface 2024-09-15T21:59:15.322 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:59:16.324 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:59:16.324 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:59:16.324 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-15T21:59:16.324 INFO:tasks.workunit.client.0.smithi136.stdout:2 2024-09-15T21:59:16.325 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:59:16.325 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:59:16.608 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:59:17.610 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:59:17.610 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:59:17.610 INFO:tasks.workunit.client.0.smithi136.stdout:3 2024-09-15T21:59:17.611 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-15T21:59:17.611 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:59:17.611 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:59:17.865 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:59:17.864+0000 7f3463b4d640 -1 osd.0 18 log_to_monitors true 2024-09-15T21:59:17.896 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:59:18.897 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:59:18.897 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:59:18.897 INFO:tasks.workunit.client.0.smithi136.stdout:4 2024-09-15T21:59:18.898 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-15T21:59:18.898 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:59:18.898 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:59:19.178 INFO:tasks.workunit.client.0.smithi136.stdout:osd.0 up in weight 1 up_from 21 up_thru 21 down_at 19 last_clean_interval [5,18) [v2:127.0.0.1:6802/1779783277,v1:127.0.0.1:6803/1779783277] [v2:127.0.0.1:6804/1779783277,v1:127.0.0.1:6805/1779783277] exists,up e7c6e8c2-8578-4b39-b595-2416fda40c44 2024-09-15T21:59:19.179 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-15T21:59:19.179 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-15T21:59:19.179 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-15T21:59:19.179 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:210: TEST_scrub_snaps: ceph tell 'osd.*' config set osd_shallow_scrub_chunk_max 25 2024-09-15T21:59:19.271 INFO:tasks.workunit.client.0.smithi136.stdout:{ 2024-09-15T21:59:19.271 INFO:tasks.workunit.client.0.smithi136.stdout: "success": "osd_delete_sleep = '' osd_delete_sleep_hdd = '' osd_delete_sleep_hybrid = '' osd_delete_sleep_ssd = '' osd_max_backfills = '' osd_recovery_max_active = '' osd_recovery_max_active_hdd = '' osd_recovery_max_active_ssd = '' osd_recovery_sleep = '' osd_recovery_sleep_hdd = '' osd_recovery_sleep_hybrid = '' osd_recovery_sleep_ssd = '' osd_scrub_sleep = '' osd_shallow_scrub_chunk_max = '' (not observed, change may require restart) osd_snap_trim_sleep = '' osd_snap_trim_sleep_hdd = '' osd_snap_trim_sleep_hybrid = '' osd_snap_trim_sleep_ssd = '' " 2024-09-15T21:59:19.271 INFO:tasks.workunit.client.0.smithi136.stdout:} 2024-09-15T21:59:19.280 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:211: TEST_scrub_snaps: ceph tell 'osd.*' config set osd_shallow_scrub_chunk_min 5 2024-09-15T21:59:19.374 INFO:tasks.workunit.client.0.smithi136.stdout:{ 2024-09-15T21:59:19.374 INFO:tasks.workunit.client.0.smithi136.stdout: "success": "osd_shallow_scrub_chunk_min = '' (not observed, change may require restart) " 2024-09-15T21:59:19.374 INFO:tasks.workunit.client.0.smithi136.stdout:} 2024-09-15T21:59:19.383 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:212: TEST_scrub_snaps: ceph tell 'osd.*' config set osd_pg_stat_report_interval_max_seconds 1 2024-09-15T21:59:19.476 INFO:tasks.workunit.client.0.smithi136.stdout:{ 2024-09-15T21:59:19.476 INFO:tasks.workunit.client.0.smithi136.stdout: "success": "osd_pg_stat_report_interval_max_seconds = '' (not observed, change may require restart) " 2024-09-15T21:59:19.476 INFO:tasks.workunit.client.0.smithi136.stdout:} 2024-09-15T21:59:19.485 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:213: TEST_scrub_snaps: ceph tell 'osd.*' config set osd_pg_stat_report_interval_max_epochs 1 2024-09-15T21:59:19.613 INFO:tasks.workunit.client.0.smithi136.stdout:{ 2024-09-15T21:59:19.613 INFO:tasks.workunit.client.0.smithi136.stdout: "success": "osd_pg_stat_report_interval_max_epochs = '' (not observed, change may require restart) " 2024-09-15T21:59:19.613 INFO:tasks.workunit.client.0.smithi136.stdout:} 2024-09-15T21:59:19.622 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:216: TEST_scrub_snaps: wait_for_clean 2024-09-15T21:59:19.622 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-15T21:59:19.622 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-15T21:59:19.622 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-15T21:59:19.623 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-15T21:59:19.623 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-15T21:59:19.623 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-15T21:59:19.623 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-15T21:59:19.623 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-15T21:59:19.624 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-15T21:59:19.709 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-15T21:59:19.709 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-15T21:59:19.709 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-15T21:59:19.709 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-15T21:59:19.709 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-15T21:59:19.709 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-15T21:59:19.982 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids=0 2024-09-15T21:59:19.982 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-15T21:59:19.982 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T21:59:19.982 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-15T21:59:20.082 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=90194313218 2024-09-15T21:59:20.082 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 90194313218 2024-09-15T21:59:20.082 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-90194313218' 2024-09-15T21:59:20.082 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T21:59:20.082 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-90194313218 2024-09-15T21:59:20.082 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T21:59:20.082 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-15T21:59:20.083 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-90194313218 2024-09-15T21:59:20.083 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T21:59:20.084 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=90194313218 2024-09-15T21:59:20.084 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 90194313218' 2024-09-15T21:59:20.084 INFO:tasks.workunit.client.0.smithi136.stdout:waiting osd.0 seq 90194313218 2024-09-15T21:59:20.085 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T21:59:20.357 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 1 -lt 90194313218 2024-09-15T21:59:20.357 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-15T21:59:21.358 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-15T21:59:21.359 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T21:59:21.642 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 90194313218 -lt 90194313218 2024-09-15T21:59:21.642 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-15T21:59:21.643 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T21:59:21.643 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T21:59:21.986 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 1 == 0 2024-09-15T21:59:21.986 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-15T21:59:21.986 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-15T21:59:21.986 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-15T21:59:21.986 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T21:59:21.986 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T21:59:21.986 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T21:59:21.986 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T21:59:22.255 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=1 2024-09-15T21:59:22.255 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-15T21:59:22.256 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T21:59:22.256 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T21:59:22.599 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 = 1 2024-09-15T21:59:22.599 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-15T21:59:22.600 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-15T21:59:22.600 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:218: TEST_scrub_snaps: ceph tell 'osd.*' config get osd_shallow_scrub_chunk_max 2024-09-15T21:59:22.692 INFO:tasks.workunit.client.0.smithi136.stdout:{ 2024-09-15T21:59:22.693 INFO:tasks.workunit.client.0.smithi136.stdout: "osd_shallow_scrub_chunk_max": "25" 2024-09-15T21:59:22.693 INFO:tasks.workunit.client.0.smithi136.stdout:} 2024-09-15T21:59:22.699 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:219: TEST_scrub_snaps: ceph tell 'osd.*' config get osd_shallow_scrub_chunk_min 2024-09-15T21:59:22.792 INFO:tasks.workunit.client.0.smithi136.stdout:{ 2024-09-15T21:59:22.792 INFO:tasks.workunit.client.0.smithi136.stdout: "osd_shallow_scrub_chunk_min": "5" 2024-09-15T21:59:22.793 INFO:tasks.workunit.client.0.smithi136.stdout:} 2024-09-15T21:59:22.801 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:220: TEST_scrub_snaps: ceph tell 'osd.*' config get osd_pg_stat_report_interval_max_seconds 2024-09-15T21:59:22.902 INFO:tasks.workunit.client.0.smithi136.stdout:{ 2024-09-15T21:59:22.903 INFO:tasks.workunit.client.0.smithi136.stdout: "osd_pg_stat_report_interval_max_seconds": "1" 2024-09-15T21:59:22.903 INFO:tasks.workunit.client.0.smithi136.stdout:} 2024-09-15T21:59:22.911 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:221: TEST_scrub_snaps: ceph tell 'osd.*' config get osd_pg_stat_report_interval_max_epochs 2024-09-15T21:59:23.003 INFO:tasks.workunit.client.0.smithi136.stdout:{ 2024-09-15T21:59:23.003 INFO:tasks.workunit.client.0.smithi136.stdout: "osd_pg_stat_report_interval_max_epochs": "1" 2024-09-15T21:59:23.003 INFO:tasks.workunit.client.0.smithi136.stdout:} 2024-09-15T21:59:23.012 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:222: TEST_scrub_snaps: ceph tell 'osd.*' config get osd_scrub_chunk_max 2024-09-15T21:59:23.104 INFO:tasks.workunit.client.0.smithi136.stdout:{ 2024-09-15T21:59:23.104 INFO:tasks.workunit.client.0.smithi136.stdout: "osd_scrub_chunk_max": "15" 2024-09-15T21:59:23.104 INFO:tasks.workunit.client.0.smithi136.stdout:} 2024-09-15T21:59:23.113 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:223: TEST_scrub_snaps: ceph tell 'osd.*' config get osd_scrub_chunk_min 2024-09-15T21:59:23.205 INFO:tasks.workunit.client.0.smithi136.stdout:{ 2024-09-15T21:59:23.205 INFO:tasks.workunit.client.0.smithi136.stdout: "osd_scrub_chunk_min": "5" 2024-09-15T21:59:23.205 INFO:tasks.workunit.client.0.smithi136.stdout:} 2024-09-15T21:59:23.215 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:225: TEST_scrub_snaps: local pgid=1.0 2024-09-15T21:59:23.215 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:226: TEST_scrub_snaps: pg_scrub 1.0 2024-09-15T21:59:23.215 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1883: pg_scrub: local pgid=1.0 2024-09-15T21:59:23.215 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1884: pg_scrub: get_last_scrub_stamp 1.0 2024-09-15T21:59:23.215 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-15T21:59:23.216 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-15T21:59:23.216 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-15T21:59:23.216 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-15T21:59:23.481 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1884: pg_scrub: local last_scrub=2024-09-15T21:58:18.470030+0000 2024-09-15T21:59:23.481 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1885: pg_scrub: ceph pg scrub 1.0 2024-09-15T21:59:23.745 INFO:tasks.workunit.client.0.smithi136.stderr:instructing pg 1.0 on osd.0 to scrub 2024-09-15T21:59:23.757 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1886: pg_scrub: wait_for_scrub 1.0 2024-09-15T21:58:18.470030+0000 2024-09-15T21:59:23.757 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: local pgid=1.0 2024-09-15T21:59:23.757 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2018: wait_for_scrub: local last_scrub=2024-09-15T21:58:18.470030+0000 2024-09-15T21:59:23.757 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2019: wait_for_scrub: local sname=last_scrub_stamp 2024-09-15T21:59:23.757 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i=0 )) 2024-09-15T21:59:23.757 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-15T21:59:23.757 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-15T21:59:23.757 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-15T21:59:23.758 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-15T21:59:23.758 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-15T21:59:23.758 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-15T21:59:24.029 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-15T21:58:18.470030+0000 '>' 2024-09-15T21:58:18.470030+0000 2024-09-15T21:59:24.029 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-15T21:59:25.030 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-15T21:59:25.030 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-15T21:59:25.031 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-15T21:59:25.031 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-15T21:59:25.031 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-15T21:59:25.031 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-15T21:59:25.031 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-15T21:59:25.308 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-15T21:58:18.470030+0000 '>' 2024-09-15T21:58:18.470030+0000 2024-09-15T21:59:25.309 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-15T21:59:26.310 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-15T21:59:26.310 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-15T21:59:26.310 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-15T21:59:26.310 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-15T21:59:26.310 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-15T21:59:26.311 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-15T21:59:26.311 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-15T21:59:26.592 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-15T21:58:18.470030+0000 '>' 2024-09-15T21:58:18.470030+0000 2024-09-15T21:59:26.592 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-15T21:59:27.591 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-15T21:59:27.591 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-15T21:59:27.591 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-15T21:59:27.591 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-15T21:59:27.591 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-15T21:59:27.592 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-15T21:59:27.592 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-15T21:59:27.874 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-15T21:58:18.470030+0000 '>' 2024-09-15T21:58:18.470030+0000 2024-09-15T21:59:27.874 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-15T21:59:28.876 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-15T21:59:28.876 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-15T21:59:28.876 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-15T21:59:28.876 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-15T21:59:28.876 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-15T21:59:28.876 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-15T21:59:28.876 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-15T21:59:29.155 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-15T21:59:23.929193+0000 '>' 2024-09-15T21:58:18.470030+0000 2024-09-15T21:59:29.155 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2023: wait_for_scrub: return 0 2024-09-15T21:59:29.155 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:230: TEST_scrub_snaps: grep '_scan_snaps start' td/osd-scrub-snaps/osd.0.log 2024-09-15T21:59:29.156 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:230: TEST_scrub_snaps: wc -l 2024-09-15T21:59:29.158 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:230: TEST_scrub_snaps: test 2 = 2 2024-09-15T21:59:29.158 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:232: TEST_scrub_snaps: rados list-inconsistent-pg test 2024-09-15T21:59:29.186 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:234: TEST_scrub_snaps: jq '. | length' td/osd-scrub-snaps/json 2024-09-15T21:59:29.188 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:234: TEST_scrub_snaps: test 1 = 1 2024-09-15T21:59:29.188 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:236: TEST_scrub_snaps: jq -r '.[0]' td/osd-scrub-snaps/json 2024-09-15T21:59:29.190 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:236: TEST_scrub_snaps: test 1.0 = 1.0 2024-09-15T21:59:29.190 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:238: TEST_scrub_snaps: rados list-inconsistent-obj 1.0 2024-09-15T21:59:29.216 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:243: TEST_scrub_snaps: jq .inconsistents 2024-09-15T21:59:29.216 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:243: TEST_scrub_snaps: python3 -c 'import json; import sys ; JSON=sys.stdin.read() ; ud = json.loads(JSON) ; print ( json.dumps(ud, sort_keys=True, indent=2) )' 2024-09-15T21:59:29.238 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:250: TEST_scrub_snaps: jq .inconsistents td/osd-scrub-snaps/json 2024-09-15T21:59:29.238 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:250: TEST_scrub_snaps: python3 -c 'import json; import sys ; JSON=sys.stdin.read() ; ud = json.loads(JSON) ; print ( json.dumps(ud, sort_keys=True, indent=2) )' 2024-09-15T21:59:29.260 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:251: TEST_scrub_snaps: multidiff td/osd-scrub-snaps/checkcsjson td/osd-scrub-snaps/csjson 2024-09-15T21:59:29.260 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2434: multidiff: diff td/osd-scrub-snaps/checkcsjson td/osd-scrub-snaps/csjson 2024-09-15T21:59:29.261 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:253: TEST_scrub_snaps: rados list-inconsistent-snapset 1.0 2024-09-15T21:59:29.287 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:255: TEST_scrub_snaps: jq .inconsistents 2024-09-15T21:59:29.287 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:255: TEST_scrub_snaps: python3 -c 'import json; import sys ; JSON=sys.stdin.read() ; ud = json.loads(JSON) ; print ( json.dumps(ud, sort_keys=True, indent=2) )' 2024-09-15T21:59:29.310 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:634: TEST_scrub_snaps: jq .inconsistents td/osd-scrub-snaps/json 2024-09-15T21:59:29.310 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:634: TEST_scrub_snaps: python3 -c 'import json; import sys ; JSON=sys.stdin.read() ; ud = json.loads(JSON) ; print ( json.dumps(ud, sort_keys=True, indent=2) )' 2024-09-15T21:59:29.332 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:635: TEST_scrub_snaps: multidiff td/osd-scrub-snaps/checkcsjson td/osd-scrub-snaps/csjson 2024-09-15T21:59:29.332 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2434: multidiff: diff td/osd-scrub-snaps/checkcsjson td/osd-scrub-snaps/csjson 2024-09-15T21:59:29.333 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:636: TEST_scrub_snaps: test no = yes 2024-09-15T21:59:29.333 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:641: TEST_scrub_snaps: test '' = yes 2024-09-15T21:59:29.333 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:646: TEST_scrub_snaps: find td/osd-scrub-snaps 2024-09-15T21:59:29.333 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:646: TEST_scrub_snaps: grep 'osd[^/]*\.pid' 2024-09-15T21:59:29.335 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:646: TEST_scrub_snaps: pidfiles=td/osd-scrub-snaps/osd.0.pid 2024-09-15T21:59:29.335 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:647: TEST_scrub_snaps: pids= 2024-09-15T21:59:29.335 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:648: TEST_scrub_snaps: for pidfile in ${pidfiles} 2024-09-15T21:59:29.335 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:650: TEST_scrub_snaps: cat td/osd-scrub-snaps/osd.0.pid 2024-09-15T21:59:29.336 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:650: TEST_scrub_snaps: pids+='478774 ' 2024-09-15T21:59:29.336 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:653: TEST_scrub_snaps: ERRORS=0 2024-09-15T21:59:29.336 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:655: TEST_scrub_snaps: seq 1 7 2024-09-15T21:59:29.337 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:655: TEST_scrub_snaps: for i in `seq 1 7` 2024-09-15T21:59:29.337 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:657: TEST_scrub_snaps: rados -p test rmsnap snap1 2024-09-15T21:59:29.492 INFO:tasks.workunit.client.0.smithi136.stdout:removed pool test snap snap1 2024-09-15T21:59:29.494 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:655: TEST_scrub_snaps: for i in `seq 1 7` 2024-09-15T21:59:29.494 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:657: TEST_scrub_snaps: rados -p test rmsnap snap2 2024-09-15T21:59:29.708 INFO:tasks.workunit.client.0.smithi136.stdout:removed pool test snap snap2 2024-09-15T21:59:29.710 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:655: TEST_scrub_snaps: for i in `seq 1 7` 2024-09-15T21:59:29.710 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:657: TEST_scrub_snaps: rados -p test rmsnap snap3 2024-09-15T21:59:29.926 INFO:tasks.workunit.client.0.smithi136.stdout:removed pool test snap snap3 2024-09-15T21:59:29.928 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:655: TEST_scrub_snaps: for i in `seq 1 7` 2024-09-15T21:59:29.928 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:657: TEST_scrub_snaps: rados -p test rmsnap snap4 2024-09-15T21:59:30.127 INFO:tasks.workunit.client.0.smithi136.stdout:removed pool test snap snap4 2024-09-15T21:59:30.129 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:655: TEST_scrub_snaps: for i in `seq 1 7` 2024-09-15T21:59:30.129 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:657: TEST_scrub_snaps: rados -p test rmsnap snap5 2024-09-15T21:59:30.319 INFO:tasks.workunit.client.0.smithi136.stdout:removed pool test snap snap5 2024-09-15T21:59:30.321 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:655: TEST_scrub_snaps: for i in `seq 1 7` 2024-09-15T21:59:30.321 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:657: TEST_scrub_snaps: rados -p test rmsnap snap6 2024-09-15T21:59:30.520 INFO:tasks.workunit.client.0.smithi136.stdout:removed pool test snap snap6 2024-09-15T21:59:30.522 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:655: TEST_scrub_snaps: for i in `seq 1 7` 2024-09-15T21:59:30.522 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:657: TEST_scrub_snaps: rados -p test rmsnap snap7 2024-09-15T21:59:30.713 INFO:tasks.workunit.client.0.smithi136.stdout:removed pool test snap snap7 2024-09-15T21:59:30.715 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:659: TEST_scrub_snaps: sleep 5 2024-09-15T21:59:35.717 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:660: TEST_scrub_snaps: local -i loop=0 2024-09-15T21:59:35.717 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:661: TEST_scrub_snaps: ceph pg dump pgs 2024-09-15T21:59:35.717 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:661: TEST_scrub_snaps: grep -q snaptrim 2024-09-15T21:59:35.985 INFO:tasks.workunit.client.0.smithi136.stderr:dumped pgs 2024-09-15T21:59:35.996 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:663: TEST_scrub_snaps: ceph pg dump pgs 2024-09-15T21:59:35.996 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:663: TEST_scrub_snaps: grep -q snaptrim_error 2024-09-15T21:59:36.252 INFO:tasks.workunit.client.0.smithi136.stderr:dumped pgs 2024-09-15T21:59:36.264 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:665: TEST_scrub_snaps: break 2024-09-15T21:59:36.264 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:674: TEST_scrub_snaps: ceph pg dump pgs 2024-09-15T21:59:36.520 INFO:tasks.workunit.client.0.smithi136.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-15T21:59:36.520 INFO:tasks.workunit.client.0.smithi136.stdout:1.0 35 0 0 0 0 23416 0 0 60 0 60 active+clean+inconsistent+snaptrim_error 2024-09-15T21:59:30.737692+0000 23'60 29:139 [0] 0 [0] 0 18'56 2024-09-15T21:59:23.929193+0000 0'0 2024-09-15T21:58:18.470030+0000 7 1 periodic scrub scheduled @ 2024-09-17T09:33:45.872819+0000 33 0 2024-09-15T21:59:36.520 INFO:tasks.workunit.client.0.smithi136.stdout: 2024-09-15T21:59:36.520 INFO:tasks.workunit.client.0.smithi136.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-15T21:59:36.521 INFO:tasks.workunit.client.0.smithi136.stderr:dumped pgs 2024-09-15T21:59:36.532 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:676: TEST_scrub_snaps: for pid in $pids 2024-09-15T21:59:36.532 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:678: TEST_scrub_snaps: kill -0 478774 2024-09-15T21:59:36.532 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:685: TEST_scrub_snaps: kill_daemons td/osd-scrub-snaps 2024-09-15T21:59:36.532 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-15T21:59:36.532 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-15T21:59:36.533 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-15T21:59:36.533 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-15T21:59:36.533 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-15T21:59:41.846 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-15T21:59:41.846 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:687: TEST_scrub_snaps: declare -a err_strings 2024-09-15T21:59:41.846 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:688: TEST_scrub_snaps: err_strings[0]='log_channel[(]cluster[)] log [[]ERR[]] : scrub [0-9]*[.]0 .*::obj10:.* : is missing in clone_overlap' 2024-09-15T21:59:41.846 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:689: TEST_scrub_snaps: err_strings[1]='log_channel[(]cluster[)] log [[]ERR[]] : scrub [0-9]*[.]0 .*::obj5:7 : no '\''_'\'' attr' 2024-09-15T21:59:41.846 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:690: TEST_scrub_snaps: err_strings[2]='log_channel[(]cluster[)] log [[]ERR[]] : scrub [0-9]*[.]0 .*::obj5:7 : is an unexpected clone' 2024-09-15T21:59:41.846 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:691: TEST_scrub_snaps: err_strings[3]='log_channel[(]cluster[)] log [[]ERR[]] : scrub [0-9]*[.]0 .*::obj5:4 : on disk size [(]4608[)] does not match object info size [(]512[)] adjusted for ondisk to [(]512[)]' 2024-09-15T21:59:41.846 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:692: TEST_scrub_snaps: err_strings[4]='log_channel[(]cluster[)] log [[]ERR[]] : scrub [0-9]*[.]0 .*:::obj5:head : expected clone .*:::obj5:2' 2024-09-15T21:59:41.846 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:693: TEST_scrub_snaps: err_strings[5]='log_channel[(]cluster[)] log [[]ERR[]] : scrub [0-9]*[.]0 .*:::obj5:head : expected clone .*:::obj5:1' 2024-09-15T21:59:41.847 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:694: TEST_scrub_snaps: err_strings[6]='log_channel[(]cluster[)] log [[]INF[]] : scrub [0-9]*[.]0 .*:::obj5:head : 2 missing clone[(]s[)]' 2024-09-15T21:59:41.847 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:695: TEST_scrub_snaps: err_strings[7]='log_channel[(]cluster[)] log [[]ERR[]] : scrub [0-9]*[.]0 .*:::obj8:head : snaps.seq not set' 2024-09-15T21:59:41.847 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:696: TEST_scrub_snaps: err_strings[8]='log_channel[(]cluster[)] log [[]ERR[]] : scrub [0-9]*[.]0 .*:::obj7:1 : is an unexpected clone' 2024-09-15T21:59:41.847 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:697: TEST_scrub_snaps: err_strings[9]='log_channel[(]cluster[)] log [[]ERR[]] : scrub [0-9]*[.]0 .*:::obj3:head : on disk size [(]3840[)] does not match object info size [(]768[)] adjusted for ondisk to [(]768[)]' 2024-09-15T21:59:41.847 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:698: TEST_scrub_snaps: err_strings[10]='log_channel[(]cluster[)] log [[]ERR[]] : scrub [0-9]*[.]0 .*:::obj6:1 : is an unexpected clone' 2024-09-15T21:59:41.847 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:699: TEST_scrub_snaps: err_strings[11]='log_channel[(]cluster[)] log [[]ERR[]] : scrub [0-9]*[.]0 .*:::obj2:head : no '\''snapset'\'' attr' 2024-09-15T21:59:41.847 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:700: TEST_scrub_snaps: err_strings[12]='log_channel[(]cluster[)] log [[]ERR[]] : scrub [0-9]*[.]0 .*:::obj2:7 : clone ignored due to missing snapset' 2024-09-15T21:59:41.847 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:701: TEST_scrub_snaps: err_strings[13]='log_channel[(]cluster[)] log [[]ERR[]] : scrub [0-9]*[.]0 .*:::obj2:4 : clone ignored due to missing snapset' 2024-09-15T21:59:41.847 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:702: TEST_scrub_snaps: err_strings[14]='log_channel[(]cluster[)] log [[]ERR[]] : scrub [0-9]*[.]0 .*:::obj4:head : expected clone .*:::obj4:7' 2024-09-15T21:59:41.847 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:703: TEST_scrub_snaps: err_strings[15]='log_channel[(]cluster[)] log [[]INF[]] : scrub [0-9]*[.]0 .*:::obj4:head : 1 missing clone[(]s[)]' 2024-09-15T21:59:41.847 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:704: TEST_scrub_snaps: err_strings[16]='log_channel[(]cluster[)] log [[]ERR[]] : scrub [0-9]*[.]0 .*:::obj1:1 : is an unexpected clone' 2024-09-15T21:59:41.847 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:705: TEST_scrub_snaps: err_strings[17]='log_channel[(]cluster[)] log [[]ERR[]] : scrub [0-9]*[.]0 .*:::obj9:1 : is missing in clone_size' 2024-09-15T21:59:41.847 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:706: TEST_scrub_snaps: err_strings[18]='log_channel[(]cluster[)] log [[]ERR[]] : scrub [0-9]*[.]0 .*:::obj11:1 : is an unexpected clone' 2024-09-15T21:59:41.847 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:707: TEST_scrub_snaps: err_strings[19]='log_channel[(]cluster[)] log [[]ERR[]] : scrub [0-9]*[.]0 .*:::obj14:1 : size 1032 != clone_size 1033' 2024-09-15T21:59:41.848 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:708: TEST_scrub_snaps: err_strings[20]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 scrub 20 errors' 2024-09-15T21:59:41.848 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:709: TEST_scrub_snaps: err_strings[21]='log_channel[(]cluster[)] log [[]ERR[]] : scrub [0-9]*[.]0 .*:::obj15:head : can'\''t decode '\''snapset'\'' attr ' 2024-09-15T21:59:41.848 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:710: TEST_scrub_snaps: err_strings[22]='log_channel[(]cluster[)] log [[]ERR[]] : osd[.][0-9]* found snap mapper error on pg 1.0 oid 1:461f8b5e:::obj16:7 snaps missing in mapper, should be: {1, 2, 3, 4, 5, 6, 7} ...repaired' 2024-09-15T21:59:41.848 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:712: TEST_scrub_snaps: for err_string in "${err_strings[@]}" 2024-09-15T21:59:41.848 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:714: TEST_scrub_snaps: grep 'log_channel[(]cluster[)] log [[]ERR[]] : scrub [0-9]*[.]0 .*::obj10:.* : is missing in clone_overlap' td/osd-scrub-snaps/osd.0.log 2024-09-15T21:59:41.848 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:712: TEST_scrub_snaps: for err_string in "${err_strings[@]}" 2024-09-15T21:59:41.848 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:714: TEST_scrub_snaps: grep 'log_channel[(]cluster[)] log [[]ERR[]] : scrub [0-9]*[.]0 .*::obj5:7 : no '\''_'\'' attr' td/osd-scrub-snaps/osd.0.log 2024-09-15T21:59:41.850 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:712: TEST_scrub_snaps: for err_string in "${err_strings[@]}" 2024-09-15T21:59:41.850 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:714: TEST_scrub_snaps: grep 'log_channel[(]cluster[)] log [[]ERR[]] : scrub [0-9]*[.]0 .*::obj5:7 : is an unexpected clone' td/osd-scrub-snaps/osd.0.log 2024-09-15T21:59:41.852 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:712: TEST_scrub_snaps: for err_string in "${err_strings[@]}" 2024-09-15T21:59:41.852 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:714: TEST_scrub_snaps: grep 'log_channel[(]cluster[)] log [[]ERR[]] : scrub [0-9]*[.]0 .*::obj5:4 : on disk size [(]4608[)] does not match object info size [(]512[)] adjusted for ondisk to [(]512[)]' td/osd-scrub-snaps/osd.0.log 2024-09-15T21:59:41.854 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:712: TEST_scrub_snaps: for err_string in "${err_strings[@]}" 2024-09-15T21:59:41.854 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:714: TEST_scrub_snaps: grep 'log_channel[(]cluster[)] log [[]ERR[]] : scrub [0-9]*[.]0 .*:::obj5:head : expected clone .*:::obj5:2' td/osd-scrub-snaps/osd.0.log 2024-09-15T21:59:41.857 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:712: TEST_scrub_snaps: for err_string in "${err_strings[@]}" 2024-09-15T21:59:41.857 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:714: TEST_scrub_snaps: grep 'log_channel[(]cluster[)] log [[]ERR[]] : scrub [0-9]*[.]0 .*:::obj5:head : expected clone .*:::obj5:1' td/osd-scrub-snaps/osd.0.log 2024-09-15T21:59:41.859 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:712: TEST_scrub_snaps: for err_string in "${err_strings[@]}" 2024-09-15T21:59:41.859 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:714: TEST_scrub_snaps: grep 'log_channel[(]cluster[)] log [[]INF[]] : scrub [0-9]*[.]0 .*:::obj5:head : 2 missing clone[(]s[)]' td/osd-scrub-snaps/osd.0.log 2024-09-15T21:59:41.862 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:712: TEST_scrub_snaps: for err_string in "${err_strings[@]}" 2024-09-15T21:59:41.862 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:714: TEST_scrub_snaps: grep 'log_channel[(]cluster[)] log [[]ERR[]] : scrub [0-9]*[.]0 .*:::obj8:head : snaps.seq not set' td/osd-scrub-snaps/osd.0.log 2024-09-15T21:59:41.864 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:712: TEST_scrub_snaps: for err_string in "${err_strings[@]}" 2024-09-15T21:59:41.864 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:714: TEST_scrub_snaps: grep 'log_channel[(]cluster[)] log [[]ERR[]] : scrub [0-9]*[.]0 .*:::obj7:1 : is an unexpected clone' td/osd-scrub-snaps/osd.0.log 2024-09-15T21:59:41.867 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:712: TEST_scrub_snaps: for err_string in "${err_strings[@]}" 2024-09-15T21:59:41.867 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:714: TEST_scrub_snaps: grep 'log_channel[(]cluster[)] log [[]ERR[]] : scrub [0-9]*[.]0 .*:::obj3:head : on disk size [(]3840[)] does not match object info size [(]768[)] adjusted for ondisk to [(]768[)]' td/osd-scrub-snaps/osd.0.log 2024-09-15T21:59:41.869 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:712: TEST_scrub_snaps: for err_string in "${err_strings[@]}" 2024-09-15T21:59:41.869 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:714: TEST_scrub_snaps: grep 'log_channel[(]cluster[)] log [[]ERR[]] : scrub [0-9]*[.]0 .*:::obj6:1 : is an unexpected clone' td/osd-scrub-snaps/osd.0.log 2024-09-15T21:59:41.871 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:712: TEST_scrub_snaps: for err_string in "${err_strings[@]}" 2024-09-15T21:59:41.871 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:714: TEST_scrub_snaps: grep 'log_channel[(]cluster[)] log [[]ERR[]] : scrub [0-9]*[.]0 .*:::obj2:head : no '\''snapset'\'' attr' td/osd-scrub-snaps/osd.0.log 2024-09-15T21:59:41.874 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:712: TEST_scrub_snaps: for err_string in "${err_strings[@]}" 2024-09-15T21:59:41.874 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:714: TEST_scrub_snaps: grep 'log_channel[(]cluster[)] log [[]ERR[]] : scrub [0-9]*[.]0 .*:::obj2:7 : clone ignored due to missing snapset' td/osd-scrub-snaps/osd.0.log 2024-09-15T21:59:41.876 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:712: TEST_scrub_snaps: for err_string in "${err_strings[@]}" 2024-09-15T21:59:41.876 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:714: TEST_scrub_snaps: grep 'log_channel[(]cluster[)] log [[]ERR[]] : scrub [0-9]*[.]0 .*:::obj2:4 : clone ignored due to missing snapset' td/osd-scrub-snaps/osd.0.log 2024-09-15T21:59:41.878 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:712: TEST_scrub_snaps: for err_string in "${err_strings[@]}" 2024-09-15T21:59:41.878 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:714: TEST_scrub_snaps: grep 'log_channel[(]cluster[)] log [[]ERR[]] : scrub [0-9]*[.]0 .*:::obj4:head : expected clone .*:::obj4:7' td/osd-scrub-snaps/osd.0.log 2024-09-15T21:59:41.880 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:712: TEST_scrub_snaps: for err_string in "${err_strings[@]}" 2024-09-15T21:59:41.881 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:714: TEST_scrub_snaps: grep 'log_channel[(]cluster[)] log [[]INF[]] : scrub [0-9]*[.]0 .*:::obj4:head : 1 missing clone[(]s[)]' td/osd-scrub-snaps/osd.0.log 2024-09-15T21:59:41.883 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:712: TEST_scrub_snaps: for err_string in "${err_strings[@]}" 2024-09-15T21:59:41.883 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:714: TEST_scrub_snaps: grep 'log_channel[(]cluster[)] log [[]ERR[]] : scrub [0-9]*[.]0 .*:::obj1:1 : is an unexpected clone' td/osd-scrub-snaps/osd.0.log 2024-09-15T21:59:41.885 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:712: TEST_scrub_snaps: for err_string in "${err_strings[@]}" 2024-09-15T21:59:41.885 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:714: TEST_scrub_snaps: grep 'log_channel[(]cluster[)] log [[]ERR[]] : scrub [0-9]*[.]0 .*:::obj9:1 : is missing in clone_size' td/osd-scrub-snaps/osd.0.log 2024-09-15T21:59:41.887 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:712: TEST_scrub_snaps: for err_string in "${err_strings[@]}" 2024-09-15T21:59:41.888 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:714: TEST_scrub_snaps: grep 'log_channel[(]cluster[)] log [[]ERR[]] : scrub [0-9]*[.]0 .*:::obj11:1 : is an unexpected clone' td/osd-scrub-snaps/osd.0.log 2024-09-15T21:59:41.890 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:712: TEST_scrub_snaps: for err_string in "${err_strings[@]}" 2024-09-15T21:59:41.890 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:714: TEST_scrub_snaps: grep 'log_channel[(]cluster[)] log [[]ERR[]] : scrub [0-9]*[.]0 .*:::obj14:1 : size 1032 != clone_size 1033' td/osd-scrub-snaps/osd.0.log 2024-09-15T21:59:41.892 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:712: TEST_scrub_snaps: for err_string in "${err_strings[@]}" 2024-09-15T21:59:41.892 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:714: TEST_scrub_snaps: grep 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 scrub 20 errors' td/osd-scrub-snaps/osd.0.log 2024-09-15T21:59:41.894 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:712: TEST_scrub_snaps: for err_string in "${err_strings[@]}" 2024-09-15T21:59:41.895 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:714: TEST_scrub_snaps: grep 'log_channel[(]cluster[)] log [[]ERR[]] : scrub [0-9]*[.]0 .*:::obj15:head : can'\''t decode '\''snapset'\'' attr ' td/osd-scrub-snaps/osd.0.log 2024-09-15T21:59:41.897 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:712: TEST_scrub_snaps: for err_string in "${err_strings[@]}" 2024-09-15T21:59:41.897 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:714: TEST_scrub_snaps: grep 'log_channel[(]cluster[)] log [[]ERR[]] : osd[.][0-9]* found snap mapper error on pg 1.0 oid 1:461f8b5e:::obj16:7 snaps missing in mapper, should be: {1, 2, 3, 4, 5, 6, 7} ...repaired' td/osd-scrub-snaps/osd.0.log 2024-09-15T21:59:41.899 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:721: TEST_scrub_snaps: '[' 0 '!=' 0 ']' 2024-09-15T21:59:41.899 INFO:tasks.workunit.client.0.smithi136.stdout:TEST PASSED 2024-09-15T21:59:41.901 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:727: TEST_scrub_snaps: echo 'TEST PASSED' 2024-09-15T21:59:41.901 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:728: TEST_scrub_snaps: return 0 2024-09-15T21:59:41.901 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:40: run: teardown td/osd-scrub-snaps 2024-09-15T21:59:41.901 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/osd-scrub-snaps 2024-09-15T21:59:41.901 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-15T21:59:41.901 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/osd-scrub-snaps KILL 2024-09-15T21:59:41.902 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-15T21:59:41.902 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-15T21:59:41.902 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-15T21:59:41.902 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-15T21:59:41.902 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-15T21:59:41.904 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-15T21:59:41.905 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-15T21:59:41.906 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-15T21:59:41.906 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-15T21:59:41.907 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-15T21:59:41.907 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-15T21:59:41.907 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-15T21:59:41.908 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T21:59:41.908 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-15T21:59:41.909 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-15T21:59:41.909 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T21:59:41.910 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-15T21:59:41.911 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-15T21:59:41.911 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/osd-scrub-snaps 2024-09-15T21:59:41.921 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-15T21:59:41.921 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T21:59:41.921 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.463062 2024-09-15T21:59:41.921 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.463062 2024-09-15T21:59:41.923 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-15T21:59:41.923 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-15T21:59:41.923 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:37: run: for func in $funcs 2024-09-15T21:59:41.923 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:38: run: setup td/osd-scrub-snaps 2024-09-15T21:59:41.923 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/osd-scrub-snaps 2024-09-15T21:59:41.923 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/osd-scrub-snaps 2024-09-15T21:59:41.924 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/osd-scrub-snaps 2024-09-15T21:59:41.924 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-15T21:59:41.924 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/osd-scrub-snaps KILL 2024-09-15T21:59:41.924 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-15T21:59:41.924 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-15T21:59:41.924 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-15T21:59:41.924 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-15T21:59:41.924 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-15T21:59:41.925 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-15T21:59:41.925 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-15T21:59:41.925 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-15T21:59:41.926 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-15T21:59:41.927 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-15T21:59:41.927 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-15T21:59:41.927 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-15T21:59:41.928 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T21:59:41.928 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-15T21:59:41.929 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-15T21:59:41.929 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T21:59:41.929 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-15T21:59:41.930 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-15T21:59:41.930 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/osd-scrub-snaps 2024-09-15T21:59:41.932 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-15T21:59:41.932 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T21:59:41.932 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.463062 2024-09-15T21:59:41.932 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.463062 2024-09-15T21:59:41.932 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-15T21:59:41.933 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-15T21:59:41.933 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/osd-scrub-snaps 2024-09-15T21:59:41.934 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2024-09-15T21:59:41.934 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T21:59:41.934 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.463062 2024-09-15T21:59:41.934 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.463062 2024-09-15T21:59:41.937 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2024-09-15T21:59:41.937 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 4096 -le 1024 ']' 2024-09-15T21:59:41.937 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2024-09-15T21:59:41.937 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/osd-scrub-snaps 1' TERM HUP INT 2024-09-15T21:59:41.937 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:39: run: TEST_scrub_snaps_primary td/osd-scrub-snaps 2024-09-15T21:59:41.938 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1177: TEST_scrub_snaps_primary: local dir=td/osd-scrub-snaps 2024-09-15T21:59:41.938 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1178: TEST_scrub_snaps_primary: ORIG_ARGS='--fsid=0098e432-2807-49bf-8c16-1e2a347d825d --auth-supported=none --mon-host=127.0.0.1:7121 ' 2024-09-15T21:59:41.938 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1179: TEST_scrub_snaps_primary: CEPH_ARGS+=' --osd_scrub_chunk_min=3 --osd_scrub_chunk_max=20 --osd_shallow_scrub_chunk_min=3 --osd_shallow_scrub_chunk_max=3 --osd_pg_stat_report_interval_max_seconds=1 --osd_pg_stat_report_interval_max_epochs=1' 2024-09-15T21:59:41.938 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1180: TEST_scrub_snaps_primary: _scrub_snaps_multi td/osd-scrub-snaps primary 2024-09-15T21:59:41.938 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:732: _scrub_snaps_multi: local dir=td/osd-scrub-snaps 2024-09-15T21:59:41.938 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:733: _scrub_snaps_multi: local poolname=test 2024-09-15T21:59:41.938 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:734: _scrub_snaps_multi: local OBJS=16 2024-09-15T21:59:41.938 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:735: _scrub_snaps_multi: local OSDS=2 2024-09-15T21:59:41.938 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:736: _scrub_snaps_multi: local which=primary 2024-09-15T21:59:41.938 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:738: _scrub_snaps_multi: TESTDATA=testdata.463062 2024-09-15T21:59:41.938 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:740: _scrub_snaps_multi: run_mon td/osd-scrub-snaps a --osd_pool_default_size=2 2024-09-15T21:59:41.938 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/osd-scrub-snaps 2024-09-15T21:59:41.938 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2024-09-15T21:59:41.939 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2024-09-15T21:59:41.939 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2024-09-15T21:59:41.939 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/osd-scrub-snaps/a 2024-09-15T21:59:41.939 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-scrub-snaps/a --run-dir=td/osd-scrub-snaps --osd_pool_default_size=2 2024-09-15T21:59:42.415 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2024-09-15T21:59:42.415 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-15T21:59:42.415 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-15T21:59:42.415 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-15T21:59:42.415 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T21:59:42.415 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.463062 2024-09-15T21:59:42.415 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.463062/$cluster-$name.asok' 2024-09-15T21:59:42.416 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-scrub-snaps/a '--log-file=td/osd-scrub-snaps/$name.log' '--admin-socket=/tmp/ceph-asok.463062/$cluster-$name.asok' --mon-cluster-log-file=td/osd-scrub-snaps/log --run-dir=td/osd-scrub-snaps '--pid-file=td/osd-scrub-snaps/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=2 2024-09-15T21:59:42.624 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2024-09-15T21:59:42.625 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2024-09-15T21:59:42.625 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-15T21:59:42.625 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-15T21:59:42.625 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2024-09-15T21:59:42.625 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2024-09-15T21:59:42.625 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-15T21:59:42.625 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-15T21:59:42.625 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-15T21:59:42.626 INFO:tasks.workunit.client.0.smithi136.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-15T21:59:42.626 INFO:tasks.workunit.client.0.smithi136.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T21:59:42.626 INFO:tasks.workunit.client.0.smithi136.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.463062 2024-09-15T21:59:42.626 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.463062/ceph-mon.a.asok 2024-09-15T21:59:42.626 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-15T21:59:42.627 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.463062/ceph-mon.a.asok config get fsid 2024-09-15T21:59:42.711 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2024-09-15T21:59:42.711 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-15T21:59:42.711 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-15T21:59:42.711 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2024-09-15T21:59:42.711 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2024-09-15T21:59:42.711 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-15T21:59:42.711 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-15T21:59:42.711 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-15T21:59:42.711 INFO:tasks.workunit.client.0.smithi136.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-15T21:59:42.711 INFO:tasks.workunit.client.0.smithi136.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T21:59:42.712 INFO:tasks.workunit.client.0.smithi136.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.463062 2024-09-15T21:59:42.712 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.463062/ceph-mon.a.asok 2024-09-15T21:59:42.712 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-15T21:59:42.712 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.463062/ceph-mon.a.asok config get mon_host 2024-09-15T21:59:42.791 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:741: _scrub_snaps_multi: run_mgr td/osd-scrub-snaps x 2024-09-15T21:59:42.791 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: run_mgr: local dir=td/osd-scrub-snaps 2024-09-15T21:59:42.791 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: run_mgr: shift 2024-09-15T21:59:42.791 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: run_mgr: local id=x 2024-09-15T21:59:42.791 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: run_mgr: shift 2024-09-15T21:59:42.791 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:548: run_mgr: local data=td/osd-scrub-snaps/x 2024-09-15T21:59:42.791 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:550: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-15T21:59:43.617 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: get_asok_path 2024-09-15T21:59:43.617 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-15T21:59:43.617 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-15T21:59:43.617 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-15T21:59:43.617 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T21:59:43.617 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.463062 2024-09-15T21:59:43.617 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.463062/$cluster-$name.asok' 2024-09-15T21:59:43.618 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-15T21:59:43.618 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-scrub-snaps/x '--log-file=td/osd-scrub-snaps/$name.log' '--admin-socket=/tmp/ceph-asok.463062/$cluster-$name.asok' --run-dir=td/osd-scrub-snaps '--pid-file=td/osd-scrub-snaps/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-15T21:59:43.654 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:742: _scrub_snaps_multi: expr 2 - 1 2024-09-15T21:59:43.655 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:742: _scrub_snaps_multi: seq 0 1 2024-09-15T21:59:43.659 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:742: _scrub_snaps_multi: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-09-15T21:59:43.659 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:744: _scrub_snaps_multi: run_osd td/osd-scrub-snaps 0 2024-09-15T21:59:43.659 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-scrub-snaps 2024-09-15T21:59:43.659 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-15T21:59:43.659 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=0 2024-09-15T21:59:43.660 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-15T21:59:43.660 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-scrub-snaps/0 2024-09-15T21:59:43.660 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=0098e432-2807-49bf-8c16-1e2a347d825d --auth-supported=none --mon-host=127.0.0.1:7121 --osd_scrub_chunk_min=3 --osd_scrub_chunk_max=20 --osd_shallow_scrub_chunk_min=3 --osd_shallow_scrub_chunk_max=3 --osd_pg_stat_report_interval_max_seconds=1 --osd_pg_stat_report_interval_max_epochs=1' 2024-09-15T21:59:43.660 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T21:59:43.660 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T21:59:43.660 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T21:59:43.660 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-scrub-snaps/0' 2024-09-15T21:59:43.660 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-snaps/0/journal' 2024-09-15T21:59:43.661 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-15T21:59:43.661 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-15T21:59:43.661 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-scrub-snaps' 2024-09-15T21:59:43.661 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-15T21:59:43.661 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-15T21:59:43.661 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-15T21:59:43.661 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-15T21:59:43.661 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T21:59:43.661 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.463062 2024-09-15T21:59:43.661 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.463062/$cluster-$name.asok' 2024-09-15T21:59:43.662 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.463062/$cluster-$name.asok' 2024-09-15T21:59:43.662 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-15T21:59:43.662 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-15T21:59:43.662 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-15T21:59:43.662 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-scrub-snaps/$name.log' 2024-09-15T21:59:43.662 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-scrub-snaps/$name.pid' 2024-09-15T21:59:43.662 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T21:59:43.662 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T21:59:43.662 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T21:59:43.662 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T21:59:43.662 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-15T21:59:43.663 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-15T21:59:43.663 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-scrub-snaps/0 2024-09-15T21:59:43.663 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-15T21:59:43.663 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=02b49763-9edd-424d-81ca-e2cc12728c66 2024-09-15T21:59:43.663 INFO:tasks.workunit.client.0.smithi136.stdout:add osd0 02b49763-9edd-424d-81ca-e2cc12728c66 2024-09-15T21:59:43.664 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd0 02b49763-9edd-424d-81ca-e2cc12728c66' 2024-09-15T21:59:43.664 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-15T21:59:43.686 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQDPWOdmm8TKKBAABphR4jSau7ZcW78s7bOLPw== 2024-09-15T21:59:43.686 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQDPWOdmm8TKKBAABphR4jSau7ZcW78s7bOLPw=="}' 2024-09-15T21:59:43.686 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 02b49763-9edd-424d-81ca-e2cc12728c66 -i td/osd-scrub-snaps/0/new.json 2024-09-15T21:59:43.931 INFO:tasks.workunit.client.0.smithi136.stdout:0 2024-09-15T21:59:43.941 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-scrub-snaps/0/new.json 2024-09-15T21:59:43.942 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 0 --fsid=0098e432-2807-49bf-8c16-1e2a347d825d --auth-supported=none --mon-host=127.0.0.1:7121 --osd_scrub_chunk_min=3 --osd_scrub_chunk_max=20 --osd_shallow_scrub_chunk_min=3 --osd_shallow_scrub_chunk_max=3 --osd_pg_stat_report_interval_max_seconds=1 --osd_pg_stat_report_interval_max_epochs=1 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-snaps/0 --osd-journal=td/osd-scrub-snaps/0/journal --chdir= --run-dir=td/osd-scrub-snaps '--admin-socket=/tmp/ceph-asok.463062/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-snaps/$name.log' '--pid-file=td/osd-scrub-snaps/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDPWOdmm8TKKBAABphR4jSau7ZcW78s7bOLPw== --osd-uuid 02b49763-9edd-424d-81ca-e2cc12728c66 2024-09-15T21:59:43.982 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:59:43.980+0000 7fa1d81db640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:59:43.984 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:59:43.983+0000 7fa1d81db640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:59:43.985 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:59:43.984+0000 7fa1d81db640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:59:43.986 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:59:43.984+0000 7fa1d81db640 -1 bdev(0x55f1c9dd1c00 td/osd-scrub-snaps/0/block) open stat got: (1) Operation not permitted 2024-09-15T21:59:43.986 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:59:43.984+0000 7fa1d81db640 -1 bluestore(td/osd-scrub-snaps/0) _read_fsid unparsable uuid 2024-09-15T21:59:50.504 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-scrub-snaps/0/keyring 2024-09-15T21:59:50.505 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-15T21:59:50.506 INFO:tasks.workunit.client.0.smithi136.stdout:adding osd0 key to auth repository 2024-09-15T21:59:50.506 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd0 key to auth repository 2024-09-15T21:59:50.506 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-scrub-snaps/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-15T21:59:50.878 INFO:tasks.workunit.client.0.smithi136.stdout:start osd.0 2024-09-15T21:59:50.878 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.0 2024-09-15T21:59:50.879 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 0 --fsid=0098e432-2807-49bf-8c16-1e2a347d825d --auth-supported=none --mon-host=127.0.0.1:7121 --osd_scrub_chunk_min=3 --osd_scrub_chunk_max=20 --osd_shallow_scrub_chunk_min=3 --osd_shallow_scrub_chunk_max=3 --osd_pg_stat_report_interval_max_seconds=1 --osd_pg_stat_report_interval_max_epochs=1 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-snaps/0 --osd-journal=td/osd-scrub-snaps/0/journal --chdir= --run-dir=td/osd-scrub-snaps '--admin-socket=/tmp/ceph-asok.463062/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-snaps/$name.log' '--pid-file=td/osd-scrub-snaps/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T21:59:50.879 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-15T21:59:50.879 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-15T21:59:50.879 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-15T21:59:50.916 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:59:50.914+0000 7f5be4885640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:59:50.945 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:59:50.944+0000 7f5be4885640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:59:50.947 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:59:50.945+0000 7f5be4885640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:59:51.202 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 0 2024-09-15T21:59:51.202 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-15T21:59:51.202 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2024-09-15T21:59:51.202 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-15T21:59:51.202 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-15T21:59:51.202 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:59:51.202 INFO:tasks.workunit.client.0.smithi136.stdout:0 2024-09-15T21:59:51.203 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-15T21:59:51.203 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:59:51.203 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:59:51.488 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:59:52.299 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:59:52.298+0000 7f5be4885640 -1 Falling back to public interface 2024-09-15T21:59:52.489 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:59:52.489 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:59:52.490 INFO:tasks.workunit.client.0.smithi136.stdout:1 2024-09-15T21:59:52.490 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-15T21:59:52.490 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:59:52.490 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:59:52.789 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:59:53.790 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:59:53.790 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:59:53.791 INFO:tasks.workunit.client.0.smithi136.stdout:2 2024-09-15T21:59:53.791 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-15T21:59:53.791 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:59:53.791 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:59:54.088 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:59:55.090 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:59:55.090 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:59:55.090 INFO:tasks.workunit.client.0.smithi136.stdout:3 2024-09-15T21:59:55.090 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-15T21:59:55.091 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:59:55.091 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:59:55.383 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:59:55.452 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T21:59:55.451+0000 7f5be4885640 -1 osd.0 0 log_to_monitors true 2024-09-15T21:59:56.385 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:59:56.385 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:59:56.385 INFO:tasks.workunit.client.0.smithi136.stdout:4 2024-09-15T21:59:56.386 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-15T21:59:56.386 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:59:56.386 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:59:56.683 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:59:57.683 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:59:57.683 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:59:57.684 INFO:tasks.workunit.client.0.smithi136.stdout:5 2024-09-15T21:59:57.684 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-15T21:59:57.684 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:59:57.684 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:59:57.982 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:59:58.984 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:59:58.984 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:59:58.984 INFO:tasks.workunit.client.0.smithi136.stdout:6 2024-09-15T21:59:58.985 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-15T21:59:58.985 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:59:58.985 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:59:59.287 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:00:00.288 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:00:00.288 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:00:00.289 INFO:tasks.workunit.client.0.smithi136.stdout:7 2024-09-15T22:00:00.289 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-15T22:00:00.289 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:00:00.289 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T22:00:00.587 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:00:01.588 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:00:01.588 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:00:01.589 INFO:tasks.workunit.client.0.smithi136.stdout:8 2024-09-15T22:00:01.589 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-15T22:00:01.589 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:00:01.589 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T22:00:01.887 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:00:02.888 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:00:02.889 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:00:02.889 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-15T22:00:02.889 INFO:tasks.workunit.client.0.smithi136.stdout:9 2024-09-15T22:00:02.889 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:00:02.889 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T22:00:03.298 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:00:04.299 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:00:04.299 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:00:04.300 INFO:tasks.workunit.client.0.smithi136.stdout:10 2024-09-15T22:00:04.300 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-15T22:00:04.300 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:00:04.300 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T22:00:04.599 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:00:05.601 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:00:05.601 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:00:05.601 INFO:tasks.workunit.client.0.smithi136.stdout:11 2024-09-15T22:00:05.602 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-15T22:00:05.602 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:00:05.602 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T22:00:05.903 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:00:06.904 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:00:06.904 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:00:06.905 INFO:tasks.workunit.client.0.smithi136.stdout:12 2024-09-15T22:00:06.905 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-15T22:00:06.905 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:00:06.905 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T22:00:07.659 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:00:08.661 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:00:08.661 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:00:08.661 INFO:tasks.workunit.client.0.smithi136.stdout:13 2024-09-15T22:00:08.661 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-15T22:00:08.662 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:00:08.662 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T22:00:09.433 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:00:10.434 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:00:10.435 INFO:tasks.workunit.client.0.smithi136.stdout:14 2024-09-15T22:00:10.435 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:00:10.435 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-15T22:00:10.435 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:00:10.435 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T22:00:11.132 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:00:12.134 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:00:12.134 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:00:12.134 INFO:tasks.workunit.client.0.smithi136.stdout:15 2024-09-15T22:00:12.135 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-15T22:00:12.135 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:00:12.135 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T22:00:12.976 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:00:13.978 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:00:13.978 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:00:13.978 INFO:tasks.workunit.client.0.smithi136.stdout:16 2024-09-15T22:00:13.978 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-15T22:00:13.979 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:00:13.979 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T22:00:14.301 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:00:15.302 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:00:15.302 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:00:15.303 INFO:tasks.workunit.client.0.smithi136.stdout:17 2024-09-15T22:00:15.303 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-15T22:00:15.303 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:00:15.303 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T22:00:15.765 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:00:16.767 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:00:16.767 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:00:16.767 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 18 2024-09-15T22:00:16.768 INFO:tasks.workunit.client.0.smithi136.stdout:18 2024-09-15T22:00:16.768 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:00:16.768 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T22:00:17.566 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:00:18.568 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:00:18.568 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:00:18.568 INFO:tasks.workunit.client.0.smithi136.stdout:19 2024-09-15T22:00:18.569 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 19 2024-09-15T22:00:18.569 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:00:18.569 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T22:00:18.879 INFO:tasks.workunit.client.0.smithi136.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/2127585931,v1:127.0.0.1:6803/2127585931] [v2:127.0.0.1:6804/2127585931,v1:127.0.0.1:6805/2127585931] exists,up 02b49763-9edd-424d-81ca-e2cc12728c66 2024-09-15T22:00:18.883 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-15T22:00:18.883 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-15T22:00:18.883 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-15T22:00:18.883 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:742: _scrub_snaps_multi: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-09-15T22:00:18.883 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:744: _scrub_snaps_multi: run_osd td/osd-scrub-snaps 1 2024-09-15T22:00:18.883 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-scrub-snaps 2024-09-15T22:00:18.883 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-15T22:00:18.883 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=1 2024-09-15T22:00:18.883 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-15T22:00:18.883 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-scrub-snaps/1 2024-09-15T22:00:18.884 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=0098e432-2807-49bf-8c16-1e2a347d825d --auth-supported=none --mon-host=127.0.0.1:7121 --osd_scrub_chunk_min=3 --osd_scrub_chunk_max=20 --osd_shallow_scrub_chunk_min=3 --osd_shallow_scrub_chunk_max=3 --osd_pg_stat_report_interval_max_seconds=1 --osd_pg_stat_report_interval_max_epochs=1' 2024-09-15T22:00:18.884 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T22:00:18.884 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T22:00:18.884 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T22:00:18.884 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-scrub-snaps/1' 2024-09-15T22:00:18.884 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-snaps/1/journal' 2024-09-15T22:00:18.884 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-15T22:00:18.884 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-15T22:00:18.884 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-scrub-snaps' 2024-09-15T22:00:18.884 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-15T22:00:18.884 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-15T22:00:18.884 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-15T22:00:18.884 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-15T22:00:18.885 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T22:00:18.885 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.463062 2024-09-15T22:00:18.885 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.463062/$cluster-$name.asok' 2024-09-15T22:00:18.885 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.463062/$cluster-$name.asok' 2024-09-15T22:00:18.885 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-15T22:00:18.885 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-15T22:00:18.885 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-15T22:00:18.885 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-scrub-snaps/$name.log' 2024-09-15T22:00:18.885 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-scrub-snaps/$name.pid' 2024-09-15T22:00:18.885 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T22:00:18.885 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T22:00:18.885 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T22:00:18.885 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T22:00:18.885 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-15T22:00:18.886 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-15T22:00:18.886 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-scrub-snaps/1 2024-09-15T22:00:18.886 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-15T22:00:18.886 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=e9b95467-ebbe-410e-8da9-b130a4b813e8 2024-09-15T22:00:18.886 INFO:tasks.workunit.client.0.smithi136.stdout:add osd1 e9b95467-ebbe-410e-8da9-b130a4b813e8 2024-09-15T22:00:18.886 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd1 e9b95467-ebbe-410e-8da9-b130a4b813e8' 2024-09-15T22:00:18.886 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-15T22:00:18.909 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQDyWOdmLTgZNhAAUlCMFbsLUqn3PjvDDaLHBg== 2024-09-15T22:00:18.910 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQDyWOdmLTgZNhAAUlCMFbsLUqn3PjvDDaLHBg=="}' 2024-09-15T22:00:18.910 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new e9b95467-ebbe-410e-8da9-b130a4b813e8 -i td/osd-scrub-snaps/1/new.json 2024-09-15T22:00:19.283 INFO:tasks.workunit.client.0.smithi136.stdout:1 2024-09-15T22:00:19.295 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-scrub-snaps/1/new.json 2024-09-15T22:00:19.296 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 1 --fsid=0098e432-2807-49bf-8c16-1e2a347d825d --auth-supported=none --mon-host=127.0.0.1:7121 --osd_scrub_chunk_min=3 --osd_scrub_chunk_max=20 --osd_shallow_scrub_chunk_min=3 --osd_shallow_scrub_chunk_max=3 --osd_pg_stat_report_interval_max_seconds=1 --osd_pg_stat_report_interval_max_epochs=1 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-snaps/1 --osd-journal=td/osd-scrub-snaps/1/journal --chdir= --run-dir=td/osd-scrub-snaps '--admin-socket=/tmp/ceph-asok.463062/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-snaps/$name.log' '--pid-file=td/osd-scrub-snaps/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDyWOdmLTgZNhAAUlCMFbsLUqn3PjvDDaLHBg== --osd-uuid e9b95467-ebbe-410e-8da9-b130a4b813e8 2024-09-15T22:00:19.335 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T22:00:19.333+0000 7f03f7951640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T22:00:19.359 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T22:00:19.357+0000 7f03f7951640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T22:00:19.360 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T22:00:19.359+0000 7f03f7951640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T22:00:19.361 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T22:00:19.359+0000 7f03f7951640 -1 bdev(0x55b5ea4ddc00 td/osd-scrub-snaps/1/block) open stat got: (1) Operation not permitted 2024-09-15T22:00:19.361 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T22:00:19.359+0000 7f03f7951640 -1 bluestore(td/osd-scrub-snaps/1) _read_fsid unparsable uuid 2024-09-15T22:00:25.787 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-scrub-snaps/1/keyring 2024-09-15T22:00:25.787 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-15T22:00:25.788 INFO:tasks.workunit.client.0.smithi136.stdout:adding osd1 key to auth repository 2024-09-15T22:00:25.788 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd1 key to auth repository 2024-09-15T22:00:25.788 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-scrub-snaps/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-15T22:00:26.163 INFO:tasks.workunit.client.0.smithi136.stdout:start osd.1 2024-09-15T22:00:26.163 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.1 2024-09-15T22:00:26.163 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 1 --fsid=0098e432-2807-49bf-8c16-1e2a347d825d --auth-supported=none --mon-host=127.0.0.1:7121 --osd_scrub_chunk_min=3 --osd_scrub_chunk_max=20 --osd_shallow_scrub_chunk_min=3 --osd_shallow_scrub_chunk_max=3 --osd_pg_stat_report_interval_max_seconds=1 --osd_pg_stat_report_interval_max_epochs=1 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-snaps/1 --osd-journal=td/osd-scrub-snaps/1/journal --chdir= --run-dir=td/osd-scrub-snaps '--admin-socket=/tmp/ceph-asok.463062/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-snaps/$name.log' '--pid-file=td/osd-scrub-snaps/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T22:00:26.164 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-15T22:00:26.164 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-15T22:00:26.164 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-15T22:00:26.201 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T22:00:26.199+0000 7fa5bcb1b640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T22:00:26.229 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T22:00:26.227+0000 7fa5bcb1b640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T22:00:26.230 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T22:00:26.229+0000 7fa5bcb1b640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T22:00:26.494 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 1 2024-09-15T22:00:26.494 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-15T22:00:26.494 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2024-09-15T22:00:26.494 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-15T22:00:26.494 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-15T22:00:26.494 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:00:26.494 INFO:tasks.workunit.client.0.smithi136.stdout:0 2024-09-15T22:00:26.495 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-15T22:00:26.495 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:00:26.495 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T22:00:26.816 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:00:27.351 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T22:00:27.350+0000 7fa5bcb1b640 -1 Falling back to public interface 2024-09-15T22:00:27.817 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:00:27.817 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:00:27.818 INFO:tasks.workunit.client.0.smithi136.stdout:1 2024-09-15T22:00:27.818 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-15T22:00:27.818 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:00:27.818 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T22:00:28.111 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:00:29.113 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:00:29.113 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:00:29.113 INFO:tasks.workunit.client.0.smithi136.stdout:2 2024-09-15T22:00:29.114 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-15T22:00:29.114 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:00:29.114 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T22:00:29.419 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:00:30.023 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T22:00:30.021+0000 7fa5bcb1b640 -1 osd.1 0 log_to_monitors true 2024-09-15T22:00:30.421 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:00:30.421 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:00:30.421 INFO:tasks.workunit.client.0.smithi136.stdout:3 2024-09-15T22:00:30.421 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-15T22:00:30.422 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:00:30.422 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T22:00:30.725 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:00:31.726 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:00:31.727 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:00:31.727 INFO:tasks.workunit.client.0.smithi136.stdout:4 2024-09-15T22:00:31.727 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-15T22:00:31.727 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:00:31.727 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T22:00:32.127 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:00:33.128 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:00:33.129 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:00:33.129 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-15T22:00:33.129 INFO:tasks.workunit.client.0.smithi136.stdout:5 2024-09-15T22:00:33.129 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:00:33.129 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T22:00:33.428 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:00:34.430 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:00:34.430 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:00:34.430 INFO:tasks.workunit.client.0.smithi136.stdout:6 2024-09-15T22:00:34.431 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-15T22:00:34.431 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:00:34.431 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T22:00:34.864 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:00:35.866 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:00:35.866 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:00:35.866 INFO:tasks.workunit.client.0.smithi136.stdout:7 2024-09-15T22:00:35.867 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-15T22:00:35.867 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:00:35.867 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T22:00:36.219 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:00:37.220 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:00:37.221 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:00:37.221 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-15T22:00:37.221 INFO:tasks.workunit.client.0.smithi136.stdout:8 2024-09-15T22:00:37.221 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:00:37.221 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T22:00:37.519 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:00:38.521 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:00:38.521 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:00:38.521 INFO:tasks.workunit.client.0.smithi136.stdout:9 2024-09-15T22:00:38.522 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-15T22:00:38.522 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:00:38.522 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T22:00:38.850 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:00:39.851 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:00:39.852 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:00:39.852 INFO:tasks.workunit.client.0.smithi136.stdout:10 2024-09-15T22:00:39.852 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-15T22:00:39.852 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:00:39.853 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T22:00:40.150 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:00:41.151 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:00:41.151 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:00:41.152 INFO:tasks.workunit.client.0.smithi136.stdout:11 2024-09-15T22:00:41.152 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-15T22:00:41.153 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:00:41.153 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T22:00:41.446 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:00:42.447 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:00:42.447 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:00:42.448 INFO:tasks.workunit.client.0.smithi136.stdout:12 2024-09-15T22:00:42.448 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-15T22:00:42.448 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:00:42.448 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T22:00:42.848 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:00:43.850 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:00:43.850 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:00:43.850 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-15T22:00:43.850 INFO:tasks.workunit.client.0.smithi136.stdout:13 2024-09-15T22:00:43.850 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:00:43.851 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T22:00:44.199 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:00:45.200 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:00:45.200 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:00:45.201 INFO:tasks.workunit.client.0.smithi136.stdout:14 2024-09-15T22:00:45.201 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-15T22:00:45.201 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:00:45.201 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T22:00:45.706 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:00:46.708 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:00:46.708 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:00:46.708 INFO:tasks.workunit.client.0.smithi136.stdout:15 2024-09-15T22:00:46.709 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-15T22:00:46.709 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:00:46.709 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T22:00:47.078 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:00:48.080 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:00:48.080 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:00:48.080 INFO:tasks.workunit.client.0.smithi136.stdout:16 2024-09-15T22:00:48.081 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-15T22:00:48.081 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:00:48.081 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T22:00:48.436 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:00:49.438 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:00:49.438 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:00:49.438 INFO:tasks.workunit.client.0.smithi136.stdout:17 2024-09-15T22:00:49.439 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-15T22:00:49.439 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:00:49.439 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T22:00:50.192 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:00:51.193 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:00:51.193 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:00:51.194 INFO:tasks.workunit.client.0.smithi136.stdout:18 2024-09-15T22:00:51.194 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 18 2024-09-15T22:00:51.194 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:00:51.194 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T22:00:51.492 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:00:52.494 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:00:52.494 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:00:52.494 INFO:tasks.workunit.client.0.smithi136.stdout:19 2024-09-15T22:00:52.495 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 19 2024-09-15T22:00:52.495 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:00:52.495 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T22:00:53.168 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:00:54.170 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:00:54.170 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:00:54.170 INFO:tasks.workunit.client.0.smithi136.stdout:20 2024-09-15T22:00:54.171 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 20 2024-09-15T22:00:54.171 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:00:54.171 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T22:00:54.541 INFO:tasks.workunit.client.0.smithi136.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/1528998031,v1:127.0.0.1:6811/1528998031] [v2:127.0.0.1:6812/1528998031,v1:127.0.0.1:6813/1528998031] exists,up e9b95467-ebbe-410e-8da9-b130a4b813e8 2024-09-15T22:00:54.542 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-15T22:00:54.542 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-15T22:00:54.542 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-15T22:00:54.542 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:748: _scrub_snaps_multi: ceph osd set noscrub 2024-09-15T22:00:54.929 INFO:tasks.workunit.client.0.smithi136.stderr:noscrub is set 2024-09-15T22:00:54.940 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:749: _scrub_snaps_multi: ceph osd set nodeep-scrub 2024-09-15T22:00:55.439 INFO:tasks.workunit.client.0.smithi136.stderr:nodeep-scrub is set 2024-09-15T22:00:55.451 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:752: _scrub_snaps_multi: create_pool test 1 1 2024-09-15T22:00:55.451 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create test 1 1 2024-09-15T22:00:56.000 INFO:tasks.workunit.client.0.smithi136.stderr:pool 'test' created 2024-09-15T22:00:56.011 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2024-09-15T22:00:57.014 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:753: _scrub_snaps_multi: wait_for_clean 2024-09-15T22:00:57.014 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-15T22:00:57.014 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-15T22:00:57.014 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-15T22:00:57.014 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-15T22:00:57.015 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-15T22:00:57.015 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-15T22:00:57.015 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-15T22:00:57.015 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-15T22:00:57.015 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-15T22:00:57.112 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-15T22:00:57.113 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-15T22:00:57.113 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-15T22:00:57.113 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-15T22:00:57.113 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-15T22:00:57.113 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-15T22:00:57.437 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-15T22:00:57.437 INFO:tasks.workunit.client.0.smithi136.stderr:1' 2024-09-15T22:00:57.437 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-15T22:00:57.437 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T22:00:57.437 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-15T22:00:57.550 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836489 2024-09-15T22:00:57.550 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836489 2024-09-15T22:00:57.551 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836489' 2024-09-15T22:00:57.551 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T22:00:57.551 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-15T22:00:57.682 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=42949672967 2024-09-15T22:00:57.682 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 42949672967 2024-09-15T22:00:57.683 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836489 1-42949672967' 2024-09-15T22:00:57.683 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T22:00:57.683 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836489 2024-09-15T22:00:57.683 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T22:00:57.683 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-15T22:00:57.684 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836489 2024-09-15T22:00:57.684 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T22:00:57.685 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836489 2024-09-15T22:00:57.685 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836489' 2024-09-15T22:00:57.685 INFO:tasks.workunit.client.0.smithi136.stdout:waiting osd.0 seq 21474836489 2024-09-15T22:00:57.685 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T22:00:57.997 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836489 -lt 21474836489 2024-09-15T22:00:57.997 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T22:00:57.998 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-42949672967 2024-09-15T22:00:57.998 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T22:00:57.999 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-15T22:00:57.999 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-42949672967 2024-09-15T22:00:57.999 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T22:00:58.000 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=42949672967 2024-09-15T22:00:58.000 INFO:tasks.workunit.client.0.smithi136.stdout:waiting osd.1 seq 42949672967 2024-09-15T22:00:58.001 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 42949672967' 2024-09-15T22:00:58.001 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-15T22:00:58.330 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 42949672966 -lt 42949672967 2024-09-15T22:00:58.330 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-15T22:00:59.331 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-15T22:00:59.331 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-15T22:00:59.625 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 42949672966 -lt 42949672967 2024-09-15T22:00:59.625 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-15T22:01:00.626 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 299 -eq 0 ']' 2024-09-15T22:01:00.626 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-15T22:01:00.922 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 42949672968 -lt 42949672967 2024-09-15T22:01:00.922 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-15T22:01:00.923 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T22:01:00.923 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T22:01:01.319 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 1 == 0 2024-09-15T22:01:01.319 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-15T22:01:01.319 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-15T22:01:01.319 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-15T22:01:01.319 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T22:01:01.319 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T22:01:01.319 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T22:01:01.319 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T22:01:01.637 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=1 2024-09-15T22:01:01.638 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-15T22:01:01.638 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T22:01:01.638 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T22:01:02.015 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 = 1 2024-09-15T22:01:02.015 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-15T22:01:02.016 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-15T22:01:02.016 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:754: _scrub_snaps_multi: ceph osd dump 2024-09-15T22:01:02.016 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:754: _scrub_snaps_multi: grep '^pool.*['\'']test['\'']' 2024-09-15T22:01:02.016 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:754: _scrub_snaps_multi: awk '{ print $2 }' 2024-09-15T22:01:02.304 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:754: _scrub_snaps_multi: poolid=1 2024-09-15T22:01:02.304 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:756: _scrub_snaps_multi: dd if=/dev/urandom of=testdata.463062 bs=1032 count=1 2024-09-15T22:01:02.304 INFO:tasks.workunit.client.0.smithi136.stderr:1+0 records in 2024-09-15T22:01:02.305 INFO:tasks.workunit.client.0.smithi136.stderr:1+0 records out 2024-09-15T22:01:02.305 INFO:tasks.workunit.client.0.smithi136.stderr:1032 bytes (1.0 kB, 1.0 KiB) copied, 4.5843e-05 s, 22.5 MB/s 2024-09-15T22:01:02.305 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:757: _scrub_snaps_multi: seq 1 16 2024-09-15T22:01:02.306 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:757: _scrub_snaps_multi: for i in `seq 1 $OBJS` 2024-09-15T22:01:02.306 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:759: _scrub_snaps_multi: rados -p test put obj1 testdata.463062 2024-09-15T22:01:02.462 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:757: _scrub_snaps_multi: for i in `seq 1 $OBJS` 2024-09-15T22:01:02.463 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:759: _scrub_snaps_multi: rados -p test put obj2 testdata.463062 2024-09-15T22:01:02.615 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:757: _scrub_snaps_multi: for i in `seq 1 $OBJS` 2024-09-15T22:01:02.615 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:759: _scrub_snaps_multi: rados -p test put obj3 testdata.463062 2024-09-15T22:01:02.773 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:757: _scrub_snaps_multi: for i in `seq 1 $OBJS` 2024-09-15T22:01:02.773 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:759: _scrub_snaps_multi: rados -p test put obj4 testdata.463062 2024-09-15T22:01:02.915 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:757: _scrub_snaps_multi: for i in `seq 1 $OBJS` 2024-09-15T22:01:02.915 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:759: _scrub_snaps_multi: rados -p test put obj5 testdata.463062 2024-09-15T22:01:03.057 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:757: _scrub_snaps_multi: for i in `seq 1 $OBJS` 2024-09-15T22:01:03.057 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:759: _scrub_snaps_multi: rados -p test put obj6 testdata.463062 2024-09-15T22:01:03.191 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:757: _scrub_snaps_multi: for i in `seq 1 $OBJS` 2024-09-15T22:01:03.191 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:759: _scrub_snaps_multi: rados -p test put obj7 testdata.463062 2024-09-15T22:01:03.325 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:757: _scrub_snaps_multi: for i in `seq 1 $OBJS` 2024-09-15T22:01:03.325 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:759: _scrub_snaps_multi: rados -p test put obj8 testdata.463062 2024-09-15T22:01:03.459 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:757: _scrub_snaps_multi: for i in `seq 1 $OBJS` 2024-09-15T22:01:03.459 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:759: _scrub_snaps_multi: rados -p test put obj9 testdata.463062 2024-09-15T22:01:03.593 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:757: _scrub_snaps_multi: for i in `seq 1 $OBJS` 2024-09-15T22:01:03.593 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:759: _scrub_snaps_multi: rados -p test put obj10 testdata.463062 2024-09-15T22:01:03.744 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:757: _scrub_snaps_multi: for i in `seq 1 $OBJS` 2024-09-15T22:01:03.744 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:759: _scrub_snaps_multi: rados -p test put obj11 testdata.463062 2024-09-15T22:01:03.911 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:757: _scrub_snaps_multi: for i in `seq 1 $OBJS` 2024-09-15T22:01:03.911 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:759: _scrub_snaps_multi: rados -p test put obj12 testdata.463062 2024-09-15T22:01:04.080 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:757: _scrub_snaps_multi: for i in `seq 1 $OBJS` 2024-09-15T22:01:04.080 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:759: _scrub_snaps_multi: rados -p test put obj13 testdata.463062 2024-09-15T22:01:04.220 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:757: _scrub_snaps_multi: for i in `seq 1 $OBJS` 2024-09-15T22:01:04.221 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:759: _scrub_snaps_multi: rados -p test put obj14 testdata.463062 2024-09-15T22:01:04.354 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:757: _scrub_snaps_multi: for i in `seq 1 $OBJS` 2024-09-15T22:01:04.355 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:759: _scrub_snaps_multi: rados -p test put obj15 testdata.463062 2024-09-15T22:01:04.488 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:757: _scrub_snaps_multi: for i in `seq 1 $OBJS` 2024-09-15T22:01:04.488 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:759: _scrub_snaps_multi: rados -p test put obj16 testdata.463062 2024-09-15T22:01:04.622 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:762: _scrub_snaps_multi: get_primary test obj1 2024-09-15T22:01:04.623 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1187: get_primary: local poolname=test 2024-09-15T22:01:04.623 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1188: get_primary: local objectname=obj1 2024-09-15T22:01:04.623 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1190: get_primary: ceph --format json osd map test obj1 2024-09-15T22:01:04.623 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1191: get_primary: jq .acting_primary 2024-09-15T22:01:04.912 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:762: _scrub_snaps_multi: local primary=1 2024-09-15T22:01:04.912 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:763: _scrub_snaps_multi: get_not_primary test obj1 2024-09-15T22:01:04.912 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1220: get_not_primary: local poolname=test 2024-09-15T22:01:04.912 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1221: get_not_primary: local objectname=obj1 2024-09-15T22:01:04.913 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1223: get_not_primary: get_primary test obj1 2024-09-15T22:01:04.913 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1187: get_primary: local poolname=test 2024-09-15T22:01:04.913 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1188: get_primary: local objectname=obj1 2024-09-15T22:01:04.913 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1190: get_primary: ceph --format json osd map test obj1 2024-09-15T22:01:04.913 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1191: get_primary: jq .acting_primary 2024-09-15T22:01:05.243 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1223: get_not_primary: local primary=1 2024-09-15T22:01:05.243 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1224: get_not_primary: ceph --format json osd map test obj1 2024-09-15T22:01:05.243 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1225: get_not_primary: jq '.acting | map(select (. != 1)) | .[0]' 2024-09-15T22:01:05.533 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:763: _scrub_snaps_multi: local replica=0 2024-09-15T22:01:05.533 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:765: _scrub_snaps_multi: eval create_scenario td/osd-scrub-snaps test testdata.463062 '$primary' 2024-09-15T22:01:05.533 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:765: _scrub_snaps_multi: create_scenario td/osd-scrub-snaps test testdata.463062 1 2024-09-15T22:01:05.533 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:45: create_scenario: local dir=td/osd-scrub-snaps 2024-09-15T22:01:05.533 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:46: create_scenario: local poolname=test 2024-09-15T22:01:05.534 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:47: create_scenario: local TESTDATA=testdata.463062 2024-09-15T22:01:05.534 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:48: create_scenario: local osd=1 2024-09-15T22:01:05.534 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:50: create_scenario: SNAP=1 2024-09-15T22:01:05.534 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:51: create_scenario: rados -p test mksnap snap1 2024-09-15T22:01:05.813 INFO:tasks.workunit.client.0.smithi136.stdout:created pool test snap snap1 2024-09-15T22:01:05.815 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:52: create_scenario: dd if=/dev/urandom of=testdata.463062 bs=256 count=1 2024-09-15T22:01:05.816 INFO:tasks.workunit.client.0.smithi136.stderr:1+0 records in 2024-09-15T22:01:05.817 INFO:tasks.workunit.client.0.smithi136.stderr:1+0 records out 2024-09-15T22:01:05.817 INFO:tasks.workunit.client.0.smithi136.stderr:256 bytes copied, 6.3209e-05 s, 4.1 MB/s 2024-09-15T22:01:05.817 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:53: create_scenario: rados -p test put obj1 testdata.463062 2024-09-15T22:01:06.042 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:54: create_scenario: rados -p test put obj5 testdata.463062 2024-09-15T22:01:06.201 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:55: create_scenario: rados -p test put obj3 testdata.463062 2024-09-15T22:01:06.343 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:56: create_scenario: seq 6 14 2024-09-15T22:01:06.344 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:56: create_scenario: for i in `seq 6 14` 2024-09-15T22:01:06.344 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:57: create_scenario: rados -p test put obj6 testdata.463062 2024-09-15T22:01:06.486 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:56: create_scenario: for i in `seq 6 14` 2024-09-15T22:01:06.486 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:57: create_scenario: rados -p test put obj7 testdata.463062 2024-09-15T22:01:06.636 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:56: create_scenario: for i in `seq 6 14` 2024-09-15T22:01:06.636 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:57: create_scenario: rados -p test put obj8 testdata.463062 2024-09-15T22:01:06.770 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:56: create_scenario: for i in `seq 6 14` 2024-09-15T22:01:06.770 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:57: create_scenario: rados -p test put obj9 testdata.463062 2024-09-15T22:01:06.912 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:56: create_scenario: for i in `seq 6 14` 2024-09-15T22:01:06.912 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:57: create_scenario: rados -p test put obj10 testdata.463062 2024-09-15T22:01:07.046 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:56: create_scenario: for i in `seq 6 14` 2024-09-15T22:01:07.046 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:57: create_scenario: rados -p test put obj11 testdata.463062 2024-09-15T22:01:07.196 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:56: create_scenario: for i in `seq 6 14` 2024-09-15T22:01:07.197 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:57: create_scenario: rados -p test put obj12 testdata.463062 2024-09-15T22:01:07.331 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:56: create_scenario: for i in `seq 6 14` 2024-09-15T22:01:07.331 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:57: create_scenario: rados -p test put obj13 testdata.463062 2024-09-15T22:01:07.464 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:56: create_scenario: for i in `seq 6 14` 2024-09-15T22:01:07.464 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:57: create_scenario: rados -p test put obj14 testdata.463062 2024-09-15T22:01:07.615 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:60: create_scenario: SNAP=2 2024-09-15T22:01:07.615 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:61: create_scenario: rados -p test mksnap snap2 2024-09-15T22:01:07.838 INFO:tasks.workunit.client.0.smithi136.stdout:created pool test snap snap2 2024-09-15T22:01:07.840 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:62: create_scenario: dd if=/dev/urandom of=testdata.463062 bs=256 count=2 2024-09-15T22:01:07.842 INFO:tasks.workunit.client.0.smithi136.stderr:2+0 records in 2024-09-15T22:01:07.842 INFO:tasks.workunit.client.0.smithi136.stderr:2+0 records out 2024-09-15T22:01:07.842 INFO:tasks.workunit.client.0.smithi136.stderr:512 bytes copied, 6.5239e-05 s, 7.8 MB/s 2024-09-15T22:01:07.842 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:63: create_scenario: rados -p test put obj5 testdata.463062 2024-09-15T22:01:08.084 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:65: create_scenario: SNAP=3 2024-09-15T22:01:08.084 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:66: create_scenario: rados -p test mksnap snap3 2024-09-15T22:01:08.242 INFO:tasks.workunit.client.0.smithi136.stdout:created pool test snap snap3 2024-09-15T22:01:08.244 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:67: create_scenario: dd if=/dev/urandom of=testdata.463062 bs=256 count=3 2024-09-15T22:01:08.245 INFO:tasks.workunit.client.0.smithi136.stderr:3+0 records in 2024-09-15T22:01:08.245 INFO:tasks.workunit.client.0.smithi136.stderr:3+0 records out 2024-09-15T22:01:08.245 INFO:tasks.workunit.client.0.smithi136.stderr:768 bytes copied, 6.7629e-05 s, 11.4 MB/s 2024-09-15T22:01:08.245 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:68: create_scenario: rados -p test put obj3 testdata.463062 2024-09-15T22:01:08.486 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:70: create_scenario: SNAP=4 2024-09-15T22:01:08.486 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:71: create_scenario: rados -p test mksnap snap4 2024-09-15T22:01:08.661 INFO:tasks.workunit.client.0.smithi136.stdout:created pool test snap snap4 2024-09-15T22:01:08.663 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:72: create_scenario: dd if=/dev/urandom of=testdata.463062 bs=256 count=4 2024-09-15T22:01:08.664 INFO:tasks.workunit.client.0.smithi136.stderr:4+0 records in 2024-09-15T22:01:08.664 INFO:tasks.workunit.client.0.smithi136.stderr:4+0 records out 2024-09-15T22:01:08.664 INFO:tasks.workunit.client.0.smithi136.stderr:1024 bytes (1.0 kB, 1.0 KiB) copied, 6.5918e-05 s, 15.5 MB/s 2024-09-15T22:01:08.664 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:73: create_scenario: rados -p test put obj5 testdata.463062 2024-09-15T22:01:08.913 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:74: create_scenario: rados -p test put obj2 testdata.463062 2024-09-15T22:01:09.047 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:76: create_scenario: SNAP=5 2024-09-15T22:01:09.047 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:77: create_scenario: rados -p test mksnap snap5 2024-09-15T22:01:09.213 INFO:tasks.workunit.client.0.smithi136.stdout:created pool test snap snap5 2024-09-15T22:01:09.215 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:78: create_scenario: SNAP=6 2024-09-15T22:01:09.215 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:79: create_scenario: rados -p test mksnap snap6 2024-09-15T22:01:09.447 INFO:tasks.workunit.client.0.smithi136.stdout:created pool test snap snap6 2024-09-15T22:01:09.449 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:80: create_scenario: dd if=/dev/urandom of=testdata.463062 bs=256 count=6 2024-09-15T22:01:09.450 INFO:tasks.workunit.client.0.smithi136.stderr:6+0 records in 2024-09-15T22:01:09.450 INFO:tasks.workunit.client.0.smithi136.stderr:6+0 records out 2024-09-15T22:01:09.450 INFO:tasks.workunit.client.0.smithi136.stderr:1536 bytes (1.5 kB, 1.5 KiB) copied, 6.9848e-05 s, 22.0 MB/s 2024-09-15T22:01:09.450 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:81: create_scenario: rados -p test put obj5 testdata.463062 2024-09-15T22:01:09.708 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:83: create_scenario: SNAP=7 2024-09-15T22:01:09.709 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:84: create_scenario: rados -p test mksnap snap7 2024-09-15T22:01:10.008 INFO:tasks.workunit.client.0.smithi136.stdout:created pool test snap snap7 2024-09-15T22:01:10.010 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:86: create_scenario: rados -p test rm obj4 2024-09-15T22:01:10.294 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:87: create_scenario: rados -p test rm obj16 2024-09-15T22:01:10.419 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:88: create_scenario: rados -p test rm obj2 2024-09-15T22:01:10.553 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:90: create_scenario: kill_daemons td/osd-scrub-snaps TERM osd 2024-09-15T22:01:10.554 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-15T22:01:10.554 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-15T22:01:10.554 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-15T22:01:10.554 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-15T22:01:10.554 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-15T22:01:10.859 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-15T22:01:10.859 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:94: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/1 --head --op list obj1 2024-09-15T22:01:11.551 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:94: create_scenario: JSON='["1.0",{"oid":"obj1","key":"","snapid":-2,"hash":1828249343,"max":0,"pool":1,"namespace":"","max":0}]' 2024-09-15T22:01:11.551 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:95: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/1 '["1.0",{"oid":"obj1","key":"","snapid":-2,"hash":1828249343,"max":0,"pool":1,"namespace":"","max":0}]' --force remove 2024-09-15T22:01:12.941 INFO:tasks.workunit.client.0.smithi136.stdout:WARNING: only removing head with clones present 2024-09-15T22:01:12.941 INFO:tasks.workunit.client.0.smithi136.stdout:remove #1:ff7b1f36:::obj1:head# 2024-09-15T22:01:13.456 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:97: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/1 --op list obj5 2024-09-15T22:01:13.456 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:97: create_scenario: grep '"snapid":2' 2024-09-15T22:01:14.085 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:97: create_scenario: JSON='["1.0",{"oid":"obj5","key":"","snapid":2,"hash":1718170787,"max":0,"pool":1,"namespace":"","max":0}]' 2024-09-15T22:01:14.085 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:98: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/1 '["1.0",{"oid":"obj5","key":"","snapid":2,"hash":1718170787,"max":0,"pool":1,"namespace":"","max":0}]' remove 2024-09-15T22:01:15.126 INFO:tasks.workunit.client.0.smithi136.stdout:remove #1:c52c9666:::obj5:2# 2024-09-15T22:01:15.643 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:100: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/1 --op list obj5 2024-09-15T22:01:15.643 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:100: create_scenario: grep '"snapid":1' 2024-09-15T22:01:16.233 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:100: create_scenario: JSON='["1.0",{"oid":"obj5","key":"","snapid":1,"hash":1718170787,"max":0,"pool":1,"namespace":"","max":0}]' 2024-09-15T22:01:16.233 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:101: create_scenario: OBJ5SAVE='["1.0",{"oid":"obj5","key":"","snapid":1,"hash":1718170787,"max":0,"pool":1,"namespace":"","max":0}]' 2024-09-15T22:01:16.233 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:103: create_scenario: ceph-kvstore-tool bluestore-kv td/osd-scrub-snaps/1 list 2024-09-15T22:01:17.298 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:104: create_scenario: grep SNA_ td/osd-scrub-snaps/drk.log 2024-09-15T22:01:17.300 INFO:tasks.workunit.client.0.smithi136.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.0779578A.7.obj4.. 2024-09-15T22:01:17.300 INFO:tasks.workunit.client.0.smithi136.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.1BB86F16.1.obj3.. 2024-09-15T22:01:17.300 INFO:tasks.workunit.client.0.smithi136.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.268F1DA7.7.obj16.. 2024-09-15T22:01:17.300 INFO:tasks.workunit.client.0.smithi136.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.3A439666.1.obj5.. 2024-09-15T22:01:17.300 INFO:tasks.workunit.client.0.smithi136.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.4CC52438.1.obj6.. 2024-09-15T22:01:17.300 INFO:tasks.workunit.client.0.smithi136.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.5685B6C7.1.obj13.. 2024-09-15T22:01:17.300 INFO:tasks.workunit.client.0.smithi136.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.5AAE7FD8.1.obj8.. 2024-09-15T22:01:17.300 INFO:tasks.workunit.client.0.smithi136.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.5FAF9A3D.1.obj12.. 2024-09-15T22:01:17.300 INFO:tasks.workunit.client.0.smithi136.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.70522149.1.obj14.. 2024-09-15T22:01:17.301 INFO:tasks.workunit.client.0.smithi136.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.77574592.1.obj11.. 2024-09-15T22:01:17.301 INFO:tasks.workunit.client.0.smithi136.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.802EE1F3.4.obj2.. 2024-09-15T22:01:17.301 INFO:tasks.workunit.client.0.smithi136.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.950988C5.1.obj7.. 2024-09-15T22:01:17.301 INFO:tasks.workunit.client.0.smithi136.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.B8CCECA2.1.obj10.. 2024-09-15T22:01:17.301 INFO:tasks.workunit.client.0.smithi136.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.F7CA874E.1.obj9.. 2024-09-15T22:01:17.301 INFO:tasks.workunit.client.0.smithi136.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.FFED8FC6.1.obj1.. 2024-09-15T22:01:17.301 INFO:tasks.workunit.client.0.smithi136.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000002_0000000000000001.0779578A.7.obj4.. 2024-09-15T22:01:17.301 INFO:tasks.workunit.client.0.smithi136.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000002_0000000000000001.1BB86F16.3.obj3.. 2024-09-15T22:01:17.301 INFO:tasks.workunit.client.0.smithi136.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000002_0000000000000001.268F1DA7.7.obj16.. 2024-09-15T22:01:17.301 INFO:tasks.workunit.client.0.smithi136.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000002_0000000000000001.802EE1F3.4.obj2.. 2024-09-15T22:01:17.301 INFO:tasks.workunit.client.0.smithi136.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.0779578A.7.obj4.. 2024-09-15T22:01:17.301 INFO:tasks.workunit.client.0.smithi136.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.1BB86F16.3.obj3.. 2024-09-15T22:01:17.302 INFO:tasks.workunit.client.0.smithi136.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.268F1DA7.7.obj16.. 2024-09-15T22:01:17.302 INFO:tasks.workunit.client.0.smithi136.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.3A439666.4.obj5.. 2024-09-15T22:01:17.302 INFO:tasks.workunit.client.0.smithi136.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.802EE1F3.4.obj2.. 2024-09-15T22:01:17.302 INFO:tasks.workunit.client.0.smithi136.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000004_0000000000000001.0779578A.7.obj4.. 2024-09-15T22:01:17.302 INFO:tasks.workunit.client.0.smithi136.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000004_0000000000000001.268F1DA7.7.obj16.. 2024-09-15T22:01:17.302 INFO:tasks.workunit.client.0.smithi136.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000004_0000000000000001.3A439666.4.obj5.. 2024-09-15T22:01:17.302 INFO:tasks.workunit.client.0.smithi136.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000004_0000000000000001.802EE1F3.4.obj2.. 2024-09-15T22:01:17.302 INFO:tasks.workunit.client.0.smithi136.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000005_0000000000000001.0779578A.7.obj4.. 2024-09-15T22:01:17.302 INFO:tasks.workunit.client.0.smithi136.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000005_0000000000000001.268F1DA7.7.obj16.. 2024-09-15T22:01:17.302 INFO:tasks.workunit.client.0.smithi136.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000005_0000000000000001.3A439666.6.obj5.. 2024-09-15T22:01:17.302 INFO:tasks.workunit.client.0.smithi136.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000005_0000000000000001.802EE1F3.7.obj2.. 2024-09-15T22:01:17.302 INFO:tasks.workunit.client.0.smithi136.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000006_0000000000000001.0779578A.7.obj4.. 2024-09-15T22:01:17.302 INFO:tasks.workunit.client.0.smithi136.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000006_0000000000000001.268F1DA7.7.obj16.. 2024-09-15T22:01:17.303 INFO:tasks.workunit.client.0.smithi136.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000006_0000000000000001.3A439666.6.obj5.. 2024-09-15T22:01:17.303 INFO:tasks.workunit.client.0.smithi136.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000006_0000000000000001.802EE1F3.7.obj2.. 2024-09-15T22:01:17.303 INFO:tasks.workunit.client.0.smithi136.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000007_0000000000000001.0779578A.7.obj4.. 2024-09-15T22:01:17.303 INFO:tasks.workunit.client.0.smithi136.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000007_0000000000000001.268F1DA7.7.obj16.. 2024-09-15T22:01:17.303 INFO:tasks.workunit.client.0.smithi136.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000007_0000000000000001.802EE1F3.7.obj2.. 2024-09-15T22:01:17.303 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:105: create_scenario: grep '^[pm].*SNA_.*[.]1[.]obj5[.][.]$' td/osd-scrub-snaps/drk.log 2024-09-15T22:01:17.303 INFO:tasks.workunit.client.0.smithi136.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.3A439666.1.obj5.. 2024-09-15T22:01:17.304 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:106: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/1 --rmtype nosnapmap '["1.0",{"oid":"obj5","key":"","snapid":1,"hash":1718170787,"max":0,"pool":1,"namespace":"","max":0}]' remove 2024-09-15T22:01:18.554 INFO:tasks.workunit.client.0.smithi136.stdout:remove #1:c52c9666:::obj5:1# 2024-09-15T22:01:19.069 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:108: create_scenario: ceph-kvstore-tool bluestore-kv td/osd-scrub-snaps/1 list 2024-09-15T22:01:20.133 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:109: create_scenario: grep SNA_ td/osd-scrub-snaps/drk.log 2024-09-15T22:01:20.134 INFO:tasks.workunit.client.0.smithi136.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.0779578A.7.obj4.. 2024-09-15T22:01:20.135 INFO:tasks.workunit.client.0.smithi136.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.1BB86F16.1.obj3.. 2024-09-15T22:01:20.135 INFO:tasks.workunit.client.0.smithi136.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.268F1DA7.7.obj16.. 2024-09-15T22:01:20.135 INFO:tasks.workunit.client.0.smithi136.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.3A439666.1.obj5.. 2024-09-15T22:01:20.135 INFO:tasks.workunit.client.0.smithi136.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.4CC52438.1.obj6.. 2024-09-15T22:01:20.135 INFO:tasks.workunit.client.0.smithi136.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.5685B6C7.1.obj13.. 2024-09-15T22:01:20.135 INFO:tasks.workunit.client.0.smithi136.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.5AAE7FD8.1.obj8.. 2024-09-15T22:01:20.135 INFO:tasks.workunit.client.0.smithi136.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.5FAF9A3D.1.obj12.. 2024-09-15T22:01:20.135 INFO:tasks.workunit.client.0.smithi136.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.70522149.1.obj14.. 2024-09-15T22:01:20.135 INFO:tasks.workunit.client.0.smithi136.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.77574592.1.obj11.. 2024-09-15T22:01:20.135 INFO:tasks.workunit.client.0.smithi136.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.802EE1F3.4.obj2.. 2024-09-15T22:01:20.135 INFO:tasks.workunit.client.0.smithi136.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.950988C5.1.obj7.. 2024-09-15T22:01:20.135 INFO:tasks.workunit.client.0.smithi136.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.B8CCECA2.1.obj10.. 2024-09-15T22:01:20.135 INFO:tasks.workunit.client.0.smithi136.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.F7CA874E.1.obj9.. 2024-09-15T22:01:20.136 INFO:tasks.workunit.client.0.smithi136.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.FFED8FC6.1.obj1.. 2024-09-15T22:01:20.136 INFO:tasks.workunit.client.0.smithi136.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000002_0000000000000001.0779578A.7.obj4.. 2024-09-15T22:01:20.136 INFO:tasks.workunit.client.0.smithi136.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000002_0000000000000001.1BB86F16.3.obj3.. 2024-09-15T22:01:20.136 INFO:tasks.workunit.client.0.smithi136.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000002_0000000000000001.268F1DA7.7.obj16.. 2024-09-15T22:01:20.136 INFO:tasks.workunit.client.0.smithi136.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000002_0000000000000001.802EE1F3.4.obj2.. 2024-09-15T22:01:20.136 INFO:tasks.workunit.client.0.smithi136.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.0779578A.7.obj4.. 2024-09-15T22:01:20.136 INFO:tasks.workunit.client.0.smithi136.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.1BB86F16.3.obj3.. 2024-09-15T22:01:20.136 INFO:tasks.workunit.client.0.smithi136.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.268F1DA7.7.obj16.. 2024-09-15T22:01:20.136 INFO:tasks.workunit.client.0.smithi136.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.3A439666.4.obj5.. 2024-09-15T22:01:20.136 INFO:tasks.workunit.client.0.smithi136.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.802EE1F3.4.obj2.. 2024-09-15T22:01:20.136 INFO:tasks.workunit.client.0.smithi136.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000004_0000000000000001.0779578A.7.obj4.. 2024-09-15T22:01:20.136 INFO:tasks.workunit.client.0.smithi136.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000004_0000000000000001.268F1DA7.7.obj16.. 2024-09-15T22:01:20.136 INFO:tasks.workunit.client.0.smithi136.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000004_0000000000000001.3A439666.4.obj5.. 2024-09-15T22:01:20.136 INFO:tasks.workunit.client.0.smithi136.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000004_0000000000000001.802EE1F3.4.obj2.. 2024-09-15T22:01:20.137 INFO:tasks.workunit.client.0.smithi136.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000005_0000000000000001.0779578A.7.obj4.. 2024-09-15T22:01:20.137 INFO:tasks.workunit.client.0.smithi136.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000005_0000000000000001.268F1DA7.7.obj16.. 2024-09-15T22:01:20.137 INFO:tasks.workunit.client.0.smithi136.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000005_0000000000000001.3A439666.6.obj5.. 2024-09-15T22:01:20.137 INFO:tasks.workunit.client.0.smithi136.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000005_0000000000000001.802EE1F3.7.obj2.. 2024-09-15T22:01:20.137 INFO:tasks.workunit.client.0.smithi136.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000006_0000000000000001.0779578A.7.obj4.. 2024-09-15T22:01:20.137 INFO:tasks.workunit.client.0.smithi136.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000006_0000000000000001.268F1DA7.7.obj16.. 2024-09-15T22:01:20.137 INFO:tasks.workunit.client.0.smithi136.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000006_0000000000000001.3A439666.6.obj5.. 2024-09-15T22:01:20.137 INFO:tasks.workunit.client.0.smithi136.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000006_0000000000000001.802EE1F3.7.obj2.. 2024-09-15T22:01:20.137 INFO:tasks.workunit.client.0.smithi136.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000007_0000000000000001.0779578A.7.obj4.. 2024-09-15T22:01:20.137 INFO:tasks.workunit.client.0.smithi136.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000007_0000000000000001.268F1DA7.7.obj16.. 2024-09-15T22:01:20.137 INFO:tasks.workunit.client.0.smithi136.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000007_0000000000000001.802EE1F3.7.obj2.. 2024-09-15T22:01:20.138 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:110: create_scenario: grep '^[pm].*SNA_.*[.]1[.]obj5[.][.]$' td/osd-scrub-snaps/drk.log 2024-09-15T22:01:20.138 INFO:tasks.workunit.client.0.smithi136.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.3A439666.1.obj5.. 2024-09-15T22:01:20.138 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:111: create_scenario: rm -f td/osd-scrub-snaps/drk.log 2024-09-15T22:01:20.138 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:113: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/1 --op list obj5 2024-09-15T22:01:20.138 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:113: create_scenario: grep '"snapid":4' 2024-09-15T22:01:20.740 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:113: create_scenario: JSON='["1.0",{"oid":"obj5","key":"","snapid":4,"hash":1718170787,"max":0,"pool":1,"namespace":"","max":0}]' 2024-09-15T22:01:20.740 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:114: create_scenario: dd if=/dev/urandom of=testdata.463062 bs=256 count=18 2024-09-15T22:01:20.741 INFO:tasks.workunit.client.0.smithi136.stderr:18+0 records in 2024-09-15T22:01:20.741 INFO:tasks.workunit.client.0.smithi136.stderr:18+0 records out 2024-09-15T22:01:20.741 INFO:tasks.workunit.client.0.smithi136.stderr:4608 bytes (4.6 kB, 4.5 KiB) copied, 0.000133628 s, 34.5 MB/s 2024-09-15T22:01:20.741 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:115: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/1 '["1.0",{"oid":"obj5","key":"","snapid":4,"hash":1718170787,"max":0,"pool":1,"namespace":"","max":0}]' set-bytes testdata.463062 2024-09-15T22:01:22.366 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:117: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/1 --head --op list obj3 2024-09-15T22:01:22.976 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:117: create_scenario: JSON='["1.0",{"oid":"obj3","key":"","snapid":-2,"hash":1643547569,"max":0,"pool":1,"namespace":"","max":0}]' 2024-09-15T22:01:22.976 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:118: create_scenario: dd if=/dev/urandom of=testdata.463062 bs=256 count=15 2024-09-15T22:01:22.977 INFO:tasks.workunit.client.0.smithi136.stderr:15+0 records in 2024-09-15T22:01:22.977 INFO:tasks.workunit.client.0.smithi136.stderr:15+0 records out 2024-09-15T22:01:22.977 INFO:tasks.workunit.client.0.smithi136.stderr:3840 bytes (3.8 kB, 3.8 KiB) copied, 0.000126513 s, 30.4 MB/s 2024-09-15T22:01:22.977 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:119: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/1 '["1.0",{"oid":"obj3","key":"","snapid":-2,"hash":1643547569,"max":0,"pool":1,"namespace":"","max":0}]' set-bytes testdata.463062 2024-09-15T22:01:24.616 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:121: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/1 --op list obj4 2024-09-15T22:01:24.616 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:121: create_scenario: grep '"snapid":7' 2024-09-15T22:01:25.224 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:121: create_scenario: JSON='["1.0",{"oid":"obj4","key":"","snapid":7,"hash":2826278768,"max":0,"pool":1,"namespace":"","max":0}]' 2024-09-15T22:01:25.224 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:122: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/1 '["1.0",{"oid":"obj4","key":"","snapid":7,"hash":2826278768,"max":0,"pool":1,"namespace":"","max":0}]' remove 2024-09-15T22:01:26.469 INFO:tasks.workunit.client.0.smithi136.stdout:remove #1:0ee9ae15:::obj4:7# 2024-09-15T22:01:27.235 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:125: create_scenario: ceph-kvstore-tool bluestore-kv td/osd-scrub-snaps/1 list 2024-09-15T22:01:28.310 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:126: create_scenario: grep SNA_ td/osd-scrub-snaps/drk.log 2024-09-15T22:01:28.311 INFO:tasks.workunit.client.0.smithi136.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.1BB86F16.1.obj3.. 2024-09-15T22:01:28.312 INFO:tasks.workunit.client.0.smithi136.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.268F1DA7.7.obj16.. 2024-09-15T22:01:28.312 INFO:tasks.workunit.client.0.smithi136.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.3A439666.1.obj5.. 2024-09-15T22:01:28.312 INFO:tasks.workunit.client.0.smithi136.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.4CC52438.1.obj6.. 2024-09-15T22:01:28.312 INFO:tasks.workunit.client.0.smithi136.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.5685B6C7.1.obj13.. 2024-09-15T22:01:28.312 INFO:tasks.workunit.client.0.smithi136.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.5AAE7FD8.1.obj8.. 2024-09-15T22:01:28.312 INFO:tasks.workunit.client.0.smithi136.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.5FAF9A3D.1.obj12.. 2024-09-15T22:01:28.312 INFO:tasks.workunit.client.0.smithi136.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.70522149.1.obj14.. 2024-09-15T22:01:28.312 INFO:tasks.workunit.client.0.smithi136.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.77574592.1.obj11.. 2024-09-15T22:01:28.312 INFO:tasks.workunit.client.0.smithi136.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.802EE1F3.4.obj2.. 2024-09-15T22:01:28.312 INFO:tasks.workunit.client.0.smithi136.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.950988C5.1.obj7.. 2024-09-15T22:01:28.312 INFO:tasks.workunit.client.0.smithi136.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.B8CCECA2.1.obj10.. 2024-09-15T22:01:28.312 INFO:tasks.workunit.client.0.smithi136.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.F7CA874E.1.obj9.. 2024-09-15T22:01:28.312 INFO:tasks.workunit.client.0.smithi136.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.FFED8FC6.1.obj1.. 2024-09-15T22:01:28.313 INFO:tasks.workunit.client.0.smithi136.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000002_0000000000000001.1BB86F16.3.obj3.. 2024-09-15T22:01:28.313 INFO:tasks.workunit.client.0.smithi136.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000002_0000000000000001.268F1DA7.7.obj16.. 2024-09-15T22:01:28.313 INFO:tasks.workunit.client.0.smithi136.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000002_0000000000000001.802EE1F3.4.obj2.. 2024-09-15T22:01:28.313 INFO:tasks.workunit.client.0.smithi136.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.1BB86F16.3.obj3.. 2024-09-15T22:01:28.313 INFO:tasks.workunit.client.0.smithi136.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.268F1DA7.7.obj16.. 2024-09-15T22:01:28.313 INFO:tasks.workunit.client.0.smithi136.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.3A439666.4.obj5.. 2024-09-15T22:01:28.313 INFO:tasks.workunit.client.0.smithi136.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.802EE1F3.4.obj2.. 2024-09-15T22:01:28.313 INFO:tasks.workunit.client.0.smithi136.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000004_0000000000000001.268F1DA7.7.obj16.. 2024-09-15T22:01:28.313 INFO:tasks.workunit.client.0.smithi136.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000004_0000000000000001.3A439666.4.obj5.. 2024-09-15T22:01:28.313 INFO:tasks.workunit.client.0.smithi136.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000004_0000000000000001.802EE1F3.4.obj2.. 2024-09-15T22:01:28.313 INFO:tasks.workunit.client.0.smithi136.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000005_0000000000000001.268F1DA7.7.obj16.. 2024-09-15T22:01:28.313 INFO:tasks.workunit.client.0.smithi136.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000005_0000000000000001.3A439666.6.obj5.. 2024-09-15T22:01:28.313 INFO:tasks.workunit.client.0.smithi136.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000005_0000000000000001.802EE1F3.7.obj2.. 2024-09-15T22:01:28.313 INFO:tasks.workunit.client.0.smithi136.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000006_0000000000000001.268F1DA7.7.obj16.. 2024-09-15T22:01:28.314 INFO:tasks.workunit.client.0.smithi136.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000006_0000000000000001.3A439666.6.obj5.. 2024-09-15T22:01:28.314 INFO:tasks.workunit.client.0.smithi136.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000006_0000000000000001.802EE1F3.7.obj2.. 2024-09-15T22:01:28.314 INFO:tasks.workunit.client.0.smithi136.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000007_0000000000000001.268F1DA7.7.obj16.. 2024-09-15T22:01:28.314 INFO:tasks.workunit.client.0.smithi136.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000007_0000000000000001.802EE1F3.7.obj2.. 2024-09-15T22:01:28.314 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:127: create_scenario: grep '^[pm].*SNA_.*[.]7[.]obj16[.][.]$' td/osd-scrub-snaps/drk.log 2024-09-15T22:01:28.314 INFO:tasks.workunit.client.0.smithi136.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.268F1DA7.7.obj16.. 2024-09-15T22:01:28.314 INFO:tasks.workunit.client.0.smithi136.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000002_0000000000000001.268F1DA7.7.obj16.. 2024-09-15T22:01:28.314 INFO:tasks.workunit.client.0.smithi136.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.268F1DA7.7.obj16.. 2024-09-15T22:01:28.314 INFO:tasks.workunit.client.0.smithi136.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000004_0000000000000001.268F1DA7.7.obj16.. 2024-09-15T22:01:28.315 INFO:tasks.workunit.client.0.smithi136.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000005_0000000000000001.268F1DA7.7.obj16.. 2024-09-15T22:01:28.315 INFO:tasks.workunit.client.0.smithi136.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000006_0000000000000001.268F1DA7.7.obj16.. 2024-09-15T22:01:28.315 INFO:tasks.workunit.client.0.smithi136.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000007_0000000000000001.268F1DA7.7.obj16.. 2024-09-15T22:01:28.315 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:128: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/1 --op list obj16 2024-09-15T22:01:28.315 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:128: create_scenario: grep '"snapid":7' 2024-09-15T22:01:28.916 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:128: create_scenario: JSON='["1.0",{"oid":"obj16","key":"","snapid":7,"hash":2060580962,"max":0,"pool":1,"namespace":"","max":0}]' 2024-09-15T22:01:28.916 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:129: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/1 --rmtype snapmap '["1.0",{"oid":"obj16","key":"","snapid":7,"hash":2060580962,"max":0,"pool":1,"namespace":"","max":0}]' remove 2024-09-15T22:01:30.234 INFO:tasks.workunit.client.0.smithi136.stdout:remove #1:461f8b5e:::obj16:7# 2024-09-15T22:01:30.749 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:131: create_scenario: ceph-kvstore-tool bluestore-kv td/osd-scrub-snaps/1 list 2024-09-15T22:01:31.811 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:132: create_scenario: grep SNA_ td/osd-scrub-snaps/drk.log 2024-09-15T22:01:31.812 INFO:tasks.workunit.client.0.smithi136.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.1BB86F16.1.obj3.. 2024-09-15T22:01:31.812 INFO:tasks.workunit.client.0.smithi136.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.3A439666.1.obj5.. 2024-09-15T22:01:31.812 INFO:tasks.workunit.client.0.smithi136.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.4CC52438.1.obj6.. 2024-09-15T22:01:31.813 INFO:tasks.workunit.client.0.smithi136.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.5685B6C7.1.obj13.. 2024-09-15T22:01:31.813 INFO:tasks.workunit.client.0.smithi136.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.5AAE7FD8.1.obj8.. 2024-09-15T22:01:31.813 INFO:tasks.workunit.client.0.smithi136.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.5FAF9A3D.1.obj12.. 2024-09-15T22:01:31.813 INFO:tasks.workunit.client.0.smithi136.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.70522149.1.obj14.. 2024-09-15T22:01:31.813 INFO:tasks.workunit.client.0.smithi136.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.77574592.1.obj11.. 2024-09-15T22:01:31.813 INFO:tasks.workunit.client.0.smithi136.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.802EE1F3.4.obj2.. 2024-09-15T22:01:31.813 INFO:tasks.workunit.client.0.smithi136.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.950988C5.1.obj7.. 2024-09-15T22:01:31.813 INFO:tasks.workunit.client.0.smithi136.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.B8CCECA2.1.obj10.. 2024-09-15T22:01:31.813 INFO:tasks.workunit.client.0.smithi136.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.F7CA874E.1.obj9.. 2024-09-15T22:01:31.813 INFO:tasks.workunit.client.0.smithi136.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.FFED8FC6.1.obj1.. 2024-09-15T22:01:31.813 INFO:tasks.workunit.client.0.smithi136.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000002_0000000000000001.1BB86F16.3.obj3.. 2024-09-15T22:01:31.813 INFO:tasks.workunit.client.0.smithi136.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000002_0000000000000001.802EE1F3.4.obj2.. 2024-09-15T22:01:31.813 INFO:tasks.workunit.client.0.smithi136.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.1BB86F16.3.obj3.. 2024-09-15T22:01:31.814 INFO:tasks.workunit.client.0.smithi136.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.3A439666.4.obj5.. 2024-09-15T22:01:31.814 INFO:tasks.workunit.client.0.smithi136.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.802EE1F3.4.obj2.. 2024-09-15T22:01:31.814 INFO:tasks.workunit.client.0.smithi136.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000004_0000000000000001.3A439666.4.obj5.. 2024-09-15T22:01:31.814 INFO:tasks.workunit.client.0.smithi136.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000004_0000000000000001.802EE1F3.4.obj2.. 2024-09-15T22:01:31.814 INFO:tasks.workunit.client.0.smithi136.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000005_0000000000000001.3A439666.6.obj5.. 2024-09-15T22:01:31.814 INFO:tasks.workunit.client.0.smithi136.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000005_0000000000000001.802EE1F3.7.obj2.. 2024-09-15T22:01:31.814 INFO:tasks.workunit.client.0.smithi136.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000006_0000000000000001.3A439666.6.obj5.. 2024-09-15T22:01:31.814 INFO:tasks.workunit.client.0.smithi136.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000006_0000000000000001.802EE1F3.7.obj2.. 2024-09-15T22:01:31.814 INFO:tasks.workunit.client.0.smithi136.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000007_0000000000000001.802EE1F3.7.obj2.. 2024-09-15T22:01:31.814 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:133: create_scenario: grep '^[pm].*SNA_.*[.]7[.]obj16[.][.]$' td/osd-scrub-snaps/drk.log 2024-09-15T22:01:31.815 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:134: create_scenario: rm -f td/osd-scrub-snaps/drk.log 2024-09-15T22:01:31.815 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:136: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/1 --head --op list obj2 2024-09-15T22:01:32.412 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:136: create_scenario: JSON='["1.0",{"oid":"obj2","key":"","snapid":-2,"hash":1058988552,"max":0,"pool":1,"namespace":"","max":0}]' 2024-09-15T22:01:32.413 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:137: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/1 '["1.0",{"oid":"obj2","key":"","snapid":-2,"hash":1058988552,"max":0,"pool":1,"namespace":"","max":0}]' rm-attr snapset 2024-09-15T22:01:33.859 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:140: create_scenario: echo '["1.0",{"oid":"obj5","key":"","snapid":1,"hash":1718170787,"max":0,"pool":1,"namespace":"","max":0}]' 2024-09-15T22:01:33.860 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:140: create_scenario: sed 's/snapid":1/snapid":7/' 2024-09-15T22:01:33.861 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:140: create_scenario: JSON='["1.0",{"oid":"obj5","key":"","snapid":7,"hash":1718170787,"max":0,"pool":1,"namespace":"","max":0}]' 2024-09-15T22:01:33.861 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:141: create_scenario: dd if=/dev/urandom of=testdata.463062 bs=256 count=7 2024-09-15T22:01:33.862 INFO:tasks.workunit.client.0.smithi136.stderr:7+0 records in 2024-09-15T22:01:33.862 INFO:tasks.workunit.client.0.smithi136.stderr:7+0 records out 2024-09-15T22:01:33.862 INFO:tasks.workunit.client.0.smithi136.stderr:1792 bytes (1.8 kB, 1.8 KiB) copied, 9.1299e-05 s, 19.6 MB/s 2024-09-15T22:01:33.862 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:142: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/1 '["1.0",{"oid":"obj5","key":"","snapid":7,"hash":1718170787,"max":0,"pool":1,"namespace":"","max":0}]' set-bytes testdata.463062 2024-09-15T22:01:35.590 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:144: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/1 --head --op list obj6 2024-09-15T22:01:35.946 INFO:tasks.workunit.client.0.smithi136.stderr:Error getting attr on : 1.0_head,#1:c52c9666:::obj5:7#, (61) No data available 2024-09-15T22:01:36.219 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:144: create_scenario: JSON='["1.0",{"oid":"obj6","key":"","snapid":-2,"hash":2202164420,"max":0,"pool":1,"namespace":"","max":0}]' 2024-09-15T22:01:36.220 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:145: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/1 '["1.0",{"oid":"obj6","key":"","snapid":-2,"hash":2202164420,"max":0,"pool":1,"namespace":"","max":0}]' clear-snapset 2024-09-15T22:01:37.689 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:146: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/1 --head --op list obj7 2024-09-15T22:01:38.054 INFO:tasks.workunit.client.0.smithi136.stderr:Error getting attr on : 1.0_head,#1:c52c9666:::obj5:7#, (61) No data available 2024-09-15T22:01:38.327 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:146: create_scenario: JSON='["1.0",{"oid":"obj7","key":"","snapid":-2,"hash":1552453721,"max":0,"pool":1,"namespace":"","max":0}]' 2024-09-15T22:01:38.328 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:147: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/1 '["1.0",{"oid":"obj7","key":"","snapid":-2,"hash":1552453721,"max":0,"pool":1,"namespace":"","max":0}]' clear-snapset corrupt 2024-09-15T22:01:40.028 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:148: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/1 --head --op list obj8 2024-09-15T22:01:40.360 INFO:tasks.workunit.client.0.smithi136.stderr:Error getting attr on : 1.0_head,#1:c52c9666:::obj5:7#, (61) No data available 2024-09-15T22:01:40.632 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:148: create_scenario: JSON='["1.0",{"oid":"obj8","key":"","snapid":-2,"hash":2381834917,"max":0,"pool":1,"namespace":"","max":0}]' 2024-09-15T22:01:40.632 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:149: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/1 '["1.0",{"oid":"obj8","key":"","snapid":-2,"hash":2381834917,"max":0,"pool":1,"namespace":"","max":0}]' clear-snapset seq 2024-09-15T22:01:42.053 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:150: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/1 --head --op list obj9 2024-09-15T22:01:42.387 INFO:tasks.workunit.client.0.smithi136.stderr:Error getting attr on : 1.0_head,#1:c52c9666:::obj5:7#, (61) No data available 2024-09-15T22:01:42.654 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:150: create_scenario: JSON='["1.0",{"oid":"obj9","key":"","snapid":-2,"hash":3833113727,"max":0,"pool":1,"namespace":"","max":0}]' 2024-09-15T22:01:42.655 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:151: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/1 '["1.0",{"oid":"obj9","key":"","snapid":-2,"hash":3833113727,"max":0,"pool":1,"namespace":"","max":0}]' clear-snapset clone_size 2024-09-15T22:01:44.088 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:152: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/1 --head --op list obj10 2024-09-15T22:01:44.420 INFO:tasks.workunit.client.0.smithi136.stderr:Error getting attr on : 1.0_head,#1:c52c9666:::obj5:7#, (61) No data available 2024-09-15T22:01:44.684 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:152: create_scenario: JSON='["1.0",{"oid":"obj10","key":"","snapid":-2,"hash":718195851,"max":0,"pool":1,"namespace":"","max":0}]' 2024-09-15T22:01:44.685 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:153: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/1 '["1.0",{"oid":"obj10","key":"","snapid":-2,"hash":718195851,"max":0,"pool":1,"namespace":"","max":0}]' clear-snapset clone_overlap 2024-09-15T22:01:46.391 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:154: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/1 --head --op list obj11 2024-09-15T22:01:46.719 INFO:tasks.workunit.client.0.smithi136.stderr:Error getting attr on : 1.0_head,#1:c52c9666:::obj5:7#, (61) No data available 2024-09-15T22:01:46.983 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:154: create_scenario: JSON='["1.0",{"oid":"obj11","key":"","snapid":-2,"hash":693400951,"max":0,"pool":1,"namespace":"","max":0}]' 2024-09-15T22:01:46.984 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:155: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/1 '["1.0",{"oid":"obj11","key":"","snapid":-2,"hash":693400951,"max":0,"pool":1,"namespace":"","max":0}]' clear-snapset clones 2024-09-15T22:01:48.565 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:156: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/1 --head --op list obj12 2024-09-15T22:01:48.893 INFO:tasks.workunit.client.0.smithi136.stderr:Error getting attr on : 1.0_head,#1:c52c9666:::obj5:7#, (61) No data available 2024-09-15T22:01:49.162 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:156: create_scenario: JSON='["1.0",{"oid":"obj12","key":"","snapid":-2,"hash":3551132405,"max":0,"pool":1,"namespace":"","max":0}]' 2024-09-15T22:01:49.163 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:157: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/1 '["1.0",{"oid":"obj12","key":"","snapid":-2,"hash":3551132405,"max":0,"pool":1,"namespace":"","max":0}]' clear-snapset head 2024-09-15T22:01:50.656 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:158: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/1 --head --op list obj13 2024-09-15T22:01:50.985 INFO:tasks.workunit.client.0.smithi136.stderr:Error getting attr on : 1.0_head,#1:c52c9666:::obj5:7#, (61) No data available 2024-09-15T22:01:51.252 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:158: create_scenario: JSON='["1.0",{"oid":"obj13","key":"","snapid":-2,"hash":2087409765,"max":0,"pool":1,"namespace":"","max":0}]' 2024-09-15T22:01:51.253 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:159: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/1 '["1.0",{"oid":"obj13","key":"","snapid":-2,"hash":2087409765,"max":0,"pool":1,"namespace":"","max":0}]' clear-snapset snaps 2024-09-15T22:01:52.712 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:160: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/1 --head --op list obj14 2024-09-15T22:01:53.043 INFO:tasks.workunit.client.0.smithi136.stderr:Error getting attr on : 1.0_head,#1:c52c9666:::obj5:7#, (61) No data available 2024-09-15T22:01:53.308 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:160: create_scenario: JSON='["1.0",{"oid":"obj14","key":"","snapid":-2,"hash":2484217095,"max":0,"pool":1,"namespace":"","max":0}]' 2024-09-15T22:01:53.309 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:161: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/1 '["1.0",{"oid":"obj14","key":"","snapid":-2,"hash":2484217095,"max":0,"pool":1,"namespace":"","max":0}]' clear-snapset size 2024-09-15T22:01:54.995 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:163: create_scenario: echo garbage 2024-09-15T22:01:54.995 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:164: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/1 --head --op list obj15 2024-09-15T22:01:55.326 INFO:tasks.workunit.client.0.smithi136.stderr:Error getting attr on : 1.0_head,#1:c52c9666:::obj5:7#, (61) No data available 2024-09-15T22:01:55.593 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:164: create_scenario: JSON='["1.0",{"oid":"obj15","key":"","snapid":-2,"hash":612772309,"max":0,"pool":1,"namespace":"","max":0}]' 2024-09-15T22:01:55.594 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:165: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/1 '["1.0",{"oid":"obj15","key":"","snapid":-2,"hash":612772309,"max":0,"pool":1,"namespace":"","max":0}]' set-attr snapset td/osd-scrub-snaps/bad 2024-09-15T22:01:57.070 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:166: create_scenario: rm -f td/osd-scrub-snaps/bad 2024-09-15T22:01:57.071 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:167: create_scenario: return 0 2024-09-15T22:01:57.071 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:767: _scrub_snaps_multi: rm -f testdata.463062 2024-09-15T22:01:57.073 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:769: _scrub_snaps_multi: expr 2 - 1 2024-09-15T22:01:57.074 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:769: _scrub_snaps_multi: seq 0 1 2024-09-15T22:01:57.078 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:769: _scrub_snaps_multi: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-09-15T22:01:57.079 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:771: _scrub_snaps_multi: activate_osd td/osd-scrub-snaps 0 2024-09-15T22:01:57.079 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-snaps 2024-09-15T22:01:57.079 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-15T22:01:57.079 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=0 2024-09-15T22:01:57.079 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-15T22:01:57.079 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-snaps/0 2024-09-15T22:01:57.079 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=0098e432-2807-49bf-8c16-1e2a347d825d --auth-supported=none --mon-host=127.0.0.1:7121 --osd_scrub_chunk_min=3 --osd_scrub_chunk_max=20 --osd_shallow_scrub_chunk_min=3 --osd_shallow_scrub_chunk_max=3 --osd_pg_stat_report_interval_max_seconds=1 --osd_pg_stat_report_interval_max_epochs=1' 2024-09-15T22:01:57.079 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T22:01:57.079 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T22:01:57.079 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T22:01:57.080 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-snaps/0' 2024-09-15T22:01:57.080 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-snaps/0/journal' 2024-09-15T22:01:57.080 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-15T22:01:57.080 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-15T22:01:57.080 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-snaps' 2024-09-15T22:01:57.080 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-15T22:01:57.080 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-15T22:01:57.080 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-15T22:01:57.080 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-15T22:01:57.080 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T22:01:57.080 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.463062 2024-09-15T22:01:57.081 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.463062/$cluster-$name.asok' 2024-09-15T22:01:57.081 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.463062/$cluster-$name.asok' 2024-09-15T22:01:57.081 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-15T22:01:57.081 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-snaps/$name.log' 2024-09-15T22:01:57.081 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-snaps/$name.pid' 2024-09-15T22:01:57.081 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T22:01:57.081 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T22:01:57.081 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T22:01:57.081 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T22:01:57.081 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-15T22:01:57.081 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-15T22:01:57.081 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-snaps/0 2024-09-15T22:01:57.082 INFO:tasks.workunit.client.0.smithi136.stdout:start osd.0 2024-09-15T22:01:57.083 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.0 2024-09-15T22:01:57.083 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 0 --fsid=0098e432-2807-49bf-8c16-1e2a347d825d --auth-supported=none --mon-host=127.0.0.1:7121 --osd_scrub_chunk_min=3 --osd_scrub_chunk_max=20 --osd_shallow_scrub_chunk_min=3 --osd_shallow_scrub_chunk_max=3 --osd_pg_stat_report_interval_max_seconds=1 --osd_pg_stat_report_interval_max_epochs=1 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-snaps/0 --osd-journal=td/osd-scrub-snaps/0/journal --chdir= --run-dir=td/osd-scrub-snaps '--admin-socket=/tmp/ceph-asok.463062/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-snaps/$name.log' '--pid-file=td/osd-scrub-snaps/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T22:01:57.083 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-snaps/0/whoami 2024-09-15T22:01:57.083 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 0 = 0 ']' 2024-09-15T22:01:57.083 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-15T22:01:57.083 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-15T22:01:57.083 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-15T22:01:57.120 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T22:01:57.118+0000 7f16cff69640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T22:01:57.122 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T22:01:57.121+0000 7f16cff69640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T22:01:57.124 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T22:01:57.122+0000 7f16cff69640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T22:01:57.378 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 0 2024-09-15T22:01:57.379 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-15T22:01:57.379 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2024-09-15T22:01:57.379 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-15T22:01:57.379 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-15T22:01:57.379 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:01:57.379 INFO:tasks.workunit.client.0.smithi136.stdout:0 2024-09-15T22:01:57.380 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-15T22:01:57.380 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:01:57.380 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T22:01:57.668 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:01:58.226 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T22:01:58.225+0000 7f16cff69640 -1 Falling back to public interface 2024-09-15T22:01:58.670 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:01:58.670 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:01:58.670 INFO:tasks.workunit.client.0.smithi136.stdout:1 2024-09-15T22:01:58.671 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-15T22:01:58.671 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:01:58.671 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T22:01:58.965 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:01:59.966 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:01:59.966 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:01:59.966 INFO:tasks.workunit.client.0.smithi136.stdout:2 2024-09-15T22:01:59.967 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-15T22:01:59.967 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:01:59.967 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T22:02:00.263 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:02:01.264 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:02:01.265 INFO:tasks.workunit.client.0.smithi136.stdout:3 2024-09-15T22:02:01.265 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:02:01.265 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-15T22:02:01.266 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:02:01.266 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T22:02:01.488 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T22:02:01.486+0000 7f16cff69640 -1 osd.0 22 log_to_monitors true 2024-09-15T22:02:01.574 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:02:02.575 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:02:02.575 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:02:02.575 INFO:tasks.workunit.client.0.smithi136.stdout:4 2024-09-15T22:02:02.576 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-15T22:02:02.576 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:02:02.576 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T22:02:02.875 INFO:tasks.workunit.client.0.smithi136.stdout:osd.0 up in weight 1 up_from 25 up_thru 25 down_at 23 last_clean_interval [5,22) [v2:127.0.0.1:6802/137044827,v1:127.0.0.1:6803/137044827] [v2:127.0.0.1:6804/137044827,v1:127.0.0.1:6805/137044827] exists,up 02b49763-9edd-424d-81ca-e2cc12728c66 2024-09-15T22:02:02.879 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-15T22:02:02.879 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-15T22:02:02.879 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-15T22:02:02.879 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:769: _scrub_snaps_multi: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-09-15T22:02:02.879 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:771: _scrub_snaps_multi: activate_osd td/osd-scrub-snaps 1 2024-09-15T22:02:02.879 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-snaps 2024-09-15T22:02:02.879 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-15T22:02:02.879 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=1 2024-09-15T22:02:02.879 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-15T22:02:02.879 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-snaps/1 2024-09-15T22:02:02.879 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=0098e432-2807-49bf-8c16-1e2a347d825d --auth-supported=none --mon-host=127.0.0.1:7121 --osd_scrub_chunk_min=3 --osd_scrub_chunk_max=20 --osd_shallow_scrub_chunk_min=3 --osd_shallow_scrub_chunk_max=3 --osd_pg_stat_report_interval_max_seconds=1 --osd_pg_stat_report_interval_max_epochs=1' 2024-09-15T22:02:02.879 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T22:02:02.880 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T22:02:02.880 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T22:02:02.880 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-snaps/1' 2024-09-15T22:02:02.880 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-snaps/1/journal' 2024-09-15T22:02:02.880 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-15T22:02:02.880 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-15T22:02:02.880 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-snaps' 2024-09-15T22:02:02.880 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-15T22:02:02.880 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-15T22:02:02.880 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-15T22:02:02.880 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-15T22:02:02.880 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T22:02:02.880 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.463062 2024-09-15T22:02:02.880 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.463062/$cluster-$name.asok' 2024-09-15T22:02:02.881 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.463062/$cluster-$name.asok' 2024-09-15T22:02:02.881 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-15T22:02:02.881 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-snaps/$name.log' 2024-09-15T22:02:02.881 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-snaps/$name.pid' 2024-09-15T22:02:02.881 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T22:02:02.881 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T22:02:02.881 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T22:02:02.881 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T22:02:02.881 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-15T22:02:02.881 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-15T22:02:02.881 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-snaps/1 2024-09-15T22:02:02.881 INFO:tasks.workunit.client.0.smithi136.stdout:start osd.1 2024-09-15T22:02:02.882 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.1 2024-09-15T22:02:02.882 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 1 --fsid=0098e432-2807-49bf-8c16-1e2a347d825d --auth-supported=none --mon-host=127.0.0.1:7121 --osd_scrub_chunk_min=3 --osd_scrub_chunk_max=20 --osd_shallow_scrub_chunk_min=3 --osd_shallow_scrub_chunk_max=3 --osd_pg_stat_report_interval_max_seconds=1 --osd_pg_stat_report_interval_max_epochs=1 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-snaps/1 --osd-journal=td/osd-scrub-snaps/1/journal --chdir= --run-dir=td/osd-scrub-snaps '--admin-socket=/tmp/ceph-asok.463062/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-snaps/$name.log' '--pid-file=td/osd-scrub-snaps/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T22:02:02.882 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-snaps/1/whoami 2024-09-15T22:02:02.883 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 1 = 1 ']' 2024-09-15T22:02:02.883 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-15T22:02:02.883 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-15T22:02:02.883 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-15T22:02:02.916 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T22:02:02.914+0000 7fec6a283640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T22:02:02.942 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T22:02:02.940+0000 7fec6a283640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T22:02:02.943 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T22:02:02.942+0000 7fec6a283640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T22:02:03.171 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 1 2024-09-15T22:02:03.171 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-15T22:02:03.171 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2024-09-15T22:02:03.171 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-15T22:02:03.171 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-15T22:02:03.171 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:02:03.171 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-15T22:02:03.171 INFO:tasks.workunit.client.0.smithi136.stdout:0 2024-09-15T22:02:03.172 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:02:03.172 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T22:02:03.494 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:02:04.052 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T22:02:04.051+0000 7fec6a283640 -1 Falling back to public interface 2024-09-15T22:02:04.495 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:02:04.496 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:02:04.496 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-15T22:02:04.496 INFO:tasks.workunit.client.0.smithi136.stdout:1 2024-09-15T22:02:04.496 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:02:04.496 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T22:02:04.794 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:02:05.795 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:02:05.796 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:02:05.796 INFO:tasks.workunit.client.0.smithi136.stdout:2 2024-09-15T22:02:05.796 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-15T22:02:05.797 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:02:05.797 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T22:02:06.096 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:02:06.583 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T22:02:06.581+0000 7fec6a283640 -1 osd.1 22 log_to_monitors true 2024-09-15T22:02:07.097 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:02:07.097 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:02:07.098 INFO:tasks.workunit.client.0.smithi136.stdout:3 2024-09-15T22:02:07.098 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-15T22:02:07.098 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:02:07.098 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T22:02:07.447 INFO:tasks.workunit.client.0.smithi136.stdout:osd.1 up in weight 1 up_from 28 up_thru 28 down_at 23 last_clean_interval [10,22) [v2:127.0.0.1:6810/333972911,v1:127.0.0.1:6811/333972911] [v2:127.0.0.1:6812/333972911,v1:127.0.0.1:6813/333972911] exists,up e9b95467-ebbe-410e-8da9-b130a4b813e8 2024-09-15T22:02:07.447 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-15T22:02:07.447 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-15T22:02:07.448 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-15T22:02:07.448 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:774: _scrub_snaps_multi: ceph tell 'osd.*' config set osd_shallow_scrub_chunk_max 3 2024-09-15T22:02:07.604 INFO:tasks.workunit.client.0.smithi136.stdout:osd.0: { 2024-09-15T22:02:07.605 INFO:tasks.workunit.client.0.smithi136.stdout: "success": "osd_delete_sleep = '' osd_delete_sleep_hdd = '' osd_delete_sleep_hybrid = '' osd_delete_sleep_ssd = '' osd_max_backfills = '' osd_recovery_max_active = '' osd_recovery_max_active_hdd = '' osd_recovery_max_active_ssd = '' osd_recovery_sleep = '' osd_recovery_sleep_hdd = '' osd_recovery_sleep_hybrid = '' osd_recovery_sleep_ssd = '' osd_scrub_sleep = '' osd_shallow_scrub_chunk_max = '' (not observed, change may require restart) osd_snap_trim_sleep = '' osd_snap_trim_sleep_hdd = '' osd_snap_trim_sleep_hybrid = '' osd_snap_trim_sleep_ssd = '' " 2024-09-15T22:02:07.605 INFO:tasks.workunit.client.0.smithi136.stdout:} 2024-09-15T22:02:07.618 INFO:tasks.workunit.client.0.smithi136.stdout:osd.1: { 2024-09-15T22:02:07.618 INFO:tasks.workunit.client.0.smithi136.stdout: "success": "osd_delete_sleep = '' osd_delete_sleep_hdd = '' osd_delete_sleep_hybrid = '' osd_delete_sleep_ssd = '' osd_max_backfills = '' osd_recovery_max_active = '' osd_recovery_max_active_hdd = '' osd_recovery_max_active_ssd = '' osd_recovery_sleep = '' osd_recovery_sleep_hdd = '' osd_recovery_sleep_hybrid = '' osd_recovery_sleep_ssd = '' osd_scrub_sleep = '' osd_shallow_scrub_chunk_max = '' (not observed, change may require restart) osd_snap_trim_sleep = '' osd_snap_trim_sleep_hdd = '' osd_snap_trim_sleep_hybrid = '' osd_snap_trim_sleep_ssd = '' " 2024-09-15T22:02:07.618 INFO:tasks.workunit.client.0.smithi136.stdout:} 2024-09-15T22:02:07.627 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:775: _scrub_snaps_multi: ceph tell 'osd.*' config set osd_shallow_scrub_chunk_min 3 2024-09-15T22:02:07.769 INFO:tasks.workunit.client.0.smithi136.stdout:osd.0: { 2024-09-15T22:02:07.769 INFO:tasks.workunit.client.0.smithi136.stdout: "success": "osd_shallow_scrub_chunk_min = '' (not observed, change may require restart) " 2024-09-15T22:02:07.769 INFO:tasks.workunit.client.0.smithi136.stdout:} 2024-09-15T22:02:07.782 INFO:tasks.workunit.client.0.smithi136.stdout:osd.1: { 2024-09-15T22:02:07.782 INFO:tasks.workunit.client.0.smithi136.stdout: "success": "osd_shallow_scrub_chunk_min = '' (not observed, change may require restart) " 2024-09-15T22:02:07.782 INFO:tasks.workunit.client.0.smithi136.stdout:} 2024-09-15T22:02:07.791 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:776: _scrub_snaps_multi: ceph tell 'osd.*' config set osd_scrub_chunk_min 3 2024-09-15T22:02:07.902 INFO:tasks.workunit.client.0.smithi136.stdout:osd.0: { 2024-09-15T22:02:07.902 INFO:tasks.workunit.client.0.smithi136.stdout: "success": "osd_scrub_chunk_min = '' (not observed, change may require restart) " 2024-09-15T22:02:07.902 INFO:tasks.workunit.client.0.smithi136.stdout:} 2024-09-15T22:02:07.915 INFO:tasks.workunit.client.0.smithi136.stdout:osd.1: { 2024-09-15T22:02:07.915 INFO:tasks.workunit.client.0.smithi136.stdout: "success": "osd_scrub_chunk_min = '' (not observed, change may require restart) " 2024-09-15T22:02:07.915 INFO:tasks.workunit.client.0.smithi136.stdout:} 2024-09-15T22:02:07.924 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:777: _scrub_snaps_multi: ceph tell 'osd.*' config set osd_pg_stat_report_interval_max_seconds 1 2024-09-15T22:02:08.047 INFO:tasks.workunit.client.0.smithi136.stdout:osd.0: { 2024-09-15T22:02:08.047 INFO:tasks.workunit.client.0.smithi136.stdout: "success": "osd_pg_stat_report_interval_max_seconds = '' (not observed, change may require restart) " 2024-09-15T22:02:08.047 INFO:tasks.workunit.client.0.smithi136.stdout:} 2024-09-15T22:02:08.060 INFO:tasks.workunit.client.0.smithi136.stdout:osd.1: { 2024-09-15T22:02:08.060 INFO:tasks.workunit.client.0.smithi136.stdout: "success": "osd_pg_stat_report_interval_max_seconds = '' (not observed, change may require restart) " 2024-09-15T22:02:08.061 INFO:tasks.workunit.client.0.smithi136.stdout:} 2024-09-15T22:02:08.069 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:778: _scrub_snaps_multi: ceph tell 'osd.*' config set osd_pg_stat_report_interval_max_epochs 1 2024-09-15T22:02:08.197 INFO:tasks.workunit.client.0.smithi136.stdout:osd.0: { 2024-09-15T22:02:08.198 INFO:tasks.workunit.client.0.smithi136.stdout: "success": "osd_pg_stat_report_interval_max_epochs = '' (not observed, change may require restart) " 2024-09-15T22:02:08.198 INFO:tasks.workunit.client.0.smithi136.stdout:} 2024-09-15T22:02:08.211 INFO:tasks.workunit.client.0.smithi136.stdout:osd.1: { 2024-09-15T22:02:08.211 INFO:tasks.workunit.client.0.smithi136.stdout: "success": "osd_pg_stat_report_interval_max_epochs = '' (not observed, change may require restart) " 2024-09-15T22:02:08.211 INFO:tasks.workunit.client.0.smithi136.stdout:} 2024-09-15T22:02:08.220 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:779: _scrub_snaps_multi: wait_for_clean 2024-09-15T22:02:08.221 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-15T22:02:08.221 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-15T22:02:08.221 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-15T22:02:08.221 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-15T22:02:08.221 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-15T22:02:08.222 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-15T22:02:08.222 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-15T22:02:08.222 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-15T22:02:08.222 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-15T22:02:08.308 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-15T22:02:08.308 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-15T22:02:08.308 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-15T22:02:08.308 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-15T22:02:08.308 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-15T22:02:08.308 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-15T22:02:08.595 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-15T22:02:08.595 INFO:tasks.workunit.client.0.smithi136.stderr:1' 2024-09-15T22:02:08.595 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-15T22:02:08.595 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T22:02:08.595 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-15T22:02:08.709 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=107374182403 2024-09-15T22:02:08.709 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 107374182403 2024-09-15T22:02:08.709 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-107374182403' 2024-09-15T22:02:08.709 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T22:02:08.709 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-15T22:02:08.824 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=120259084290 2024-09-15T22:02:08.824 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 120259084290 2024-09-15T22:02:08.824 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-107374182403 1-120259084290' 2024-09-15T22:02:08.824 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T22:02:08.824 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-107374182403 2024-09-15T22:02:08.824 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T22:02:08.825 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-15T22:02:08.826 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-107374182403 2024-09-15T22:02:08.826 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T22:02:08.826 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=107374182403 2024-09-15T22:02:08.827 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 107374182403' 2024-09-15T22:02:08.827 INFO:tasks.workunit.client.0.smithi136.stdout:waiting osd.0 seq 107374182403 2024-09-15T22:02:08.827 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T22:02:09.114 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 107374182402 -lt 107374182403 2024-09-15T22:02:09.114 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-15T22:02:10.115 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-15T22:02:10.115 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T22:02:10.407 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 107374182403 -lt 107374182403 2024-09-15T22:02:10.407 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T22:02:10.408 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-120259084290 2024-09-15T22:02:10.408 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T22:02:10.409 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-15T22:02:10.409 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-120259084290 2024-09-15T22:02:10.409 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T22:02:10.410 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=120259084290 2024-09-15T22:02:10.410 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 120259084290' 2024-09-15T22:02:10.410 INFO:tasks.workunit.client.0.smithi136.stdout:waiting osd.1 seq 120259084290 2024-09-15T22:02:10.411 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-15T22:02:10.703 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 120259084290 -lt 120259084290 2024-09-15T22:02:10.703 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-15T22:02:10.703 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T22:02:10.703 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T22:02:11.099 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 1 == 0 2024-09-15T22:02:11.100 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-15T22:02:11.100 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-15T22:02:11.100 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-15T22:02:11.100 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T22:02:11.100 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T22:02:11.100 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T22:02:11.100 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T22:02:11.382 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=1 2024-09-15T22:02:11.382 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-15T22:02:11.382 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T22:02:11.382 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T22:02:11.738 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 = 1 2024-09-15T22:02:11.738 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-15T22:02:11.739 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-15T22:02:11.739 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:781: _scrub_snaps_multi: local pgid=1.0 2024-09-15T22:02:11.739 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:782: _scrub_snaps_multi: pg_scrub 1.0 2024-09-15T22:02:11.739 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1883: pg_scrub: local pgid=1.0 2024-09-15T22:02:11.739 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1884: pg_scrub: get_last_scrub_stamp 1.0 2024-09-15T22:02:11.739 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-15T22:02:11.739 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-15T22:02:11.739 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-15T22:02:11.739 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-15T22:02:12.022 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1884: pg_scrub: local last_scrub=2024-09-15T22:00:55.865337+0000 2024-09-15T22:02:12.022 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1885: pg_scrub: ceph pg scrub 1.0 2024-09-15T22:02:12.296 INFO:tasks.workunit.client.0.smithi136.stderr:instructing pg 1.0 on osd.1 to scrub 2024-09-15T22:02:12.308 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1886: pg_scrub: wait_for_scrub 1.0 2024-09-15T22:00:55.865337+0000 2024-09-15T22:02:12.308 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: local pgid=1.0 2024-09-15T22:02:12.308 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2018: wait_for_scrub: local last_scrub=2024-09-15T22:00:55.865337+0000 2024-09-15T22:02:12.308 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2019: wait_for_scrub: local sname=last_scrub_stamp 2024-09-15T22:02:12.309 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i=0 )) 2024-09-15T22:02:12.309 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-15T22:02:12.309 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-15T22:02:12.309 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-15T22:02:12.309 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-15T22:02:12.309 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-15T22:02:12.309 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-15T22:02:12.592 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-15T22:00:55.865337+0000 '>' 2024-09-15T22:00:55.865337+0000 2024-09-15T22:02:12.592 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-15T22:02:13.594 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-15T22:02:13.594 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-15T22:02:13.594 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-15T22:02:13.594 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-15T22:02:13.595 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-15T22:02:13.595 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-15T22:02:13.595 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-15T22:02:13.907 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-15T22:00:55.865337+0000 '>' 2024-09-15T22:00:55.865337+0000 2024-09-15T22:02:13.907 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-15T22:02:14.908 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-15T22:02:14.908 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-15T22:02:14.909 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-15T22:02:14.909 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-15T22:02:14.909 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-15T22:02:14.909 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-15T22:02:14.909 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-15T22:02:15.201 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-15T22:00:55.865337+0000 '>' 2024-09-15T22:00:55.865337+0000 2024-09-15T22:02:15.201 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-15T22:02:16.202 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-15T22:02:16.202 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-15T22:02:16.202 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-15T22:02:16.203 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-15T22:02:16.203 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-15T22:02:16.203 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-15T22:02:16.203 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-15T22:02:16.493 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-15T22:00:55.865337+0000 '>' 2024-09-15T22:00:55.865337+0000 2024-09-15T22:02:16.493 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-15T22:02:17.494 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-15T22:02:17.494 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-15T22:02:17.495 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-15T22:02:17.495 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-15T22:02:17.495 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-15T22:02:17.495 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-15T22:02:17.495 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-15T22:02:17.792 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-15T22:02:13.145815+0000 '>' 2024-09-15T22:00:55.865337+0000 2024-09-15T22:02:17.792 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2023: wait_for_scrub: return 0 2024-09-15T22:02:17.793 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:786: _scrub_snaps_multi: grep '_scan_snaps start' td/osd-scrub-snaps/osd.1.log 2024-09-15T22:02:17.793 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:786: _scrub_snaps_multi: wc -l 2024-09-15T22:02:17.796 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:786: _scrub_snaps_multi: test 14 -gt 3 2024-09-15T22:02:17.796 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:787: _scrub_snaps_multi: grep '_scan_snaps start' td/osd-scrub-snaps/osd.0.log 2024-09-15T22:02:17.796 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:787: _scrub_snaps_multi: wc -l 2024-09-15T22:02:17.799 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:787: _scrub_snaps_multi: test 14 -gt 3 2024-09-15T22:02:17.799 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:789: _scrub_snaps_multi: rados list-inconsistent-pg test 2024-09-15T22:02:17.835 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:791: _scrub_snaps_multi: jq '. | length' td/osd-scrub-snaps/json 2024-09-15T22:02:17.837 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:791: _scrub_snaps_multi: test 1 = 1 2024-09-15T22:02:17.838 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:793: _scrub_snaps_multi: jq -r '.[0]' td/osd-scrub-snaps/json 2024-09-15T22:02:17.840 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:793: _scrub_snaps_multi: test 1.0 = 1.0 2024-09-15T22:02:17.840 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:795: _scrub_snaps_multi: rados list-inconsistent-obj 1.0 --format=json-pretty 2024-09-15T22:02:17.875 INFO:tasks.workunit.client.0.smithi136.stdout:{ 2024-09-15T22:02:17.875 INFO:tasks.workunit.client.0.smithi136.stdout: "epoch": 28, 2024-09-15T22:02:17.875 INFO:tasks.workunit.client.0.smithi136.stdout: "inconsistents": [ 2024-09-15T22:02:17.875 INFO:tasks.workunit.client.0.smithi136.stdout: { 2024-09-15T22:02:17.875 INFO:tasks.workunit.client.0.smithi136.stdout: "object": { 2024-09-15T22:02:17.875 INFO:tasks.workunit.client.0.smithi136.stdout: "name": "obj5", 2024-09-15T22:02:17.875 INFO:tasks.workunit.client.0.smithi136.stdout: "nspace": "", 2024-09-15T22:02:17.875 INFO:tasks.workunit.client.0.smithi136.stdout: "locator": "", 2024-09-15T22:02:17.876 INFO:tasks.workunit.client.0.smithi136.stdout: "snap": 1, 2024-09-15T22:02:17.876 INFO:tasks.workunit.client.0.smithi136.stdout: "version": 5 2024-09-15T22:02:17.876 INFO:tasks.workunit.client.0.smithi136.stdout: }, 2024-09-15T22:02:17.876 INFO:tasks.workunit.client.0.smithi136.stdout: "errors": [], 2024-09-15T22:02:17.876 INFO:tasks.workunit.client.0.smithi136.stdout: "union_shard_errors": [ 2024-09-15T22:02:17.876 INFO:tasks.workunit.client.0.smithi136.stdout: "missing" 2024-09-15T22:02:17.876 INFO:tasks.workunit.client.0.smithi136.stdout: ], 2024-09-15T22:02:17.876 INFO:tasks.workunit.client.0.smithi136.stdout: "selected_object_info": { 2024-09-15T22:02:17.876 INFO:tasks.workunit.client.0.smithi136.stdout: "oid": { 2024-09-15T22:02:17.876 INFO:tasks.workunit.client.0.smithi136.stdout: "oid": "obj5", 2024-09-15T22:02:17.876 INFO:tasks.workunit.client.0.smithi136.stdout: "key": "", 2024-09-15T22:02:17.876 INFO:tasks.workunit.client.0.smithi136.stdout: "snapid": 1, 2024-09-15T22:02:17.876 INFO:tasks.workunit.client.0.smithi136.stdout: "hash": 1718170787, 2024-09-15T22:02:17.876 INFO:tasks.workunit.client.0.smithi136.stdout: "max": 0, 2024-09-15T22:02:17.877 INFO:tasks.workunit.client.0.smithi136.stdout: "pool": 1, 2024-09-15T22:02:17.877 INFO:tasks.workunit.client.0.smithi136.stdout: "namespace": "" 2024-09-15T22:02:17.877 INFO:tasks.workunit.client.0.smithi136.stdout: }, 2024-09-15T22:02:17.877 INFO:tasks.workunit.client.0.smithi136.stdout: "version": "16'19", 2024-09-15T22:02:17.877 INFO:tasks.workunit.client.0.smithi136.stdout: "prior_version": "15'5", 2024-09-15T22:02:17.877 INFO:tasks.workunit.client.0.smithi136.stdout: "last_reqid": "client.4249.0:1", 2024-09-15T22:02:17.877 INFO:tasks.workunit.client.0.smithi136.stdout: "user_version": 5, 2024-09-15T22:02:17.877 INFO:tasks.workunit.client.0.smithi136.stdout: "size": 1032, 2024-09-15T22:02:17.877 INFO:tasks.workunit.client.0.smithi136.stdout: "mtime": "2024-09-15T22:01:02.949584+0000", 2024-09-15T22:02:17.877 INFO:tasks.workunit.client.0.smithi136.stdout: "local_mtime": "2024-09-15T22:01:02.950137+0000", 2024-09-15T22:02:17.877 INFO:tasks.workunit.client.0.smithi136.stdout: "lost": 0, 2024-09-15T22:02:17.877 INFO:tasks.workunit.client.0.smithi136.stdout: "flags": [ 2024-09-15T22:02:17.877 INFO:tasks.workunit.client.0.smithi136.stdout: "dirty", 2024-09-15T22:02:17.877 INFO:tasks.workunit.client.0.smithi136.stdout: "data_digest" 2024-09-15T22:02:17.878 INFO:tasks.workunit.client.0.smithi136.stdout: ], 2024-09-15T22:02:17.878 INFO:tasks.workunit.client.0.smithi136.stdout: "truncate_seq": 0, 2024-09-15T22:02:17.878 INFO:tasks.workunit.client.0.smithi136.stdout: "truncate_size": 0, 2024-09-15T22:02:17.878 INFO:tasks.workunit.client.0.smithi136.stdout: "data_digest": "0x557795f2", 2024-09-15T22:02:17.878 INFO:tasks.workunit.client.0.smithi136.stdout: "omap_digest": "0xffffffff", 2024-09-15T22:02:17.878 INFO:tasks.workunit.client.0.smithi136.stdout: "expected_object_size": 0, 2024-09-15T22:02:17.878 INFO:tasks.workunit.client.0.smithi136.stdout: "expected_write_size": 0, 2024-09-15T22:02:17.878 INFO:tasks.workunit.client.0.smithi136.stdout: "alloc_hint_flags": 0, 2024-09-15T22:02:17.878 INFO:tasks.workunit.client.0.smithi136.stdout: "manifest": { 2024-09-15T22:02:17.878 INFO:tasks.workunit.client.0.smithi136.stdout: "type": 0 2024-09-15T22:02:17.878 INFO:tasks.workunit.client.0.smithi136.stdout: }, 2024-09-15T22:02:17.878 INFO:tasks.workunit.client.0.smithi136.stdout: "watchers": {} 2024-09-15T22:02:17.879 INFO:tasks.workunit.client.0.smithi136.stdout: }, 2024-09-15T22:02:17.879 INFO:tasks.workunit.client.0.smithi136.stdout: "shards": [ 2024-09-15T22:02:17.879 INFO:tasks.workunit.client.0.smithi136.stdout: { 2024-09-15T22:02:17.879 INFO:tasks.workunit.client.0.smithi136.stdout: "osd": 0, 2024-09-15T22:02:17.879 INFO:tasks.workunit.client.0.smithi136.stdout: "primary": false, 2024-09-15T22:02:17.879 INFO:tasks.workunit.client.0.smithi136.stdout: "errors": [], 2024-09-15T22:02:17.879 INFO:tasks.workunit.client.0.smithi136.stdout: "size": 1032 2024-09-15T22:02:17.879 INFO:tasks.workunit.client.0.smithi136.stdout: }, 2024-09-15T22:02:17.879 INFO:tasks.workunit.client.0.smithi136.stdout: { 2024-09-15T22:02:17.879 INFO:tasks.workunit.client.0.smithi136.stdout: "osd": 1, 2024-09-15T22:02:17.879 INFO:tasks.workunit.client.0.smithi136.stdout: "primary": true, 2024-09-15T22:02:17.879 INFO:tasks.workunit.client.0.smithi136.stdout: "errors": [ 2024-09-15T22:02:17.879 INFO:tasks.workunit.client.0.smithi136.stdout: "missing" 2024-09-15T22:02:17.879 INFO:tasks.workunit.client.0.smithi136.stdout: ] 2024-09-15T22:02:17.879 INFO:tasks.workunit.client.0.smithi136.stdout: } 2024-09-15T22:02:17.880 INFO:tasks.workunit.client.0.smithi136.stdout: ] 2024-09-15T22:02:17.880 INFO:tasks.workunit.client.0.smithi136.stdout: }, 2024-09-15T22:02:17.880 INFO:tasks.workunit.client.0.smithi136.stdout: { 2024-09-15T22:02:17.880 INFO:tasks.workunit.client.0.smithi136.stdout: "object": { 2024-09-15T22:02:17.880 INFO:tasks.workunit.client.0.smithi136.stdout: "name": "obj5", 2024-09-15T22:02:17.880 INFO:tasks.workunit.client.0.smithi136.stdout: "nspace": "", 2024-09-15T22:02:17.880 INFO:tasks.workunit.client.0.smithi136.stdout: "locator": "", 2024-09-15T22:02:17.880 INFO:tasks.workunit.client.0.smithi136.stdout: "snap": 2, 2024-09-15T22:02:17.880 INFO:tasks.workunit.client.0.smithi136.stdout: "version": 20 2024-09-15T22:02:17.880 INFO:tasks.workunit.client.0.smithi136.stdout: }, 2024-09-15T22:02:17.880 INFO:tasks.workunit.client.0.smithi136.stdout: "errors": [], 2024-09-15T22:02:17.880 INFO:tasks.workunit.client.0.smithi136.stdout: "union_shard_errors": [ 2024-09-15T22:02:17.880 INFO:tasks.workunit.client.0.smithi136.stdout: "missing" 2024-09-15T22:02:17.881 INFO:tasks.workunit.client.0.smithi136.stdout: ], 2024-09-15T22:02:17.881 INFO:tasks.workunit.client.0.smithi136.stdout: "selected_object_info": { 2024-09-15T22:02:17.881 INFO:tasks.workunit.client.0.smithi136.stdout: "oid": { 2024-09-15T22:02:17.881 INFO:tasks.workunit.client.0.smithi136.stdout: "oid": "obj5", 2024-09-15T22:02:17.881 INFO:tasks.workunit.client.0.smithi136.stdout: "key": "", 2024-09-15T22:02:17.881 INFO:tasks.workunit.client.0.smithi136.stdout: "snapid": 2, 2024-09-15T22:02:17.881 INFO:tasks.workunit.client.0.smithi136.stdout: "hash": 1718170787, 2024-09-15T22:02:17.881 INFO:tasks.workunit.client.0.smithi136.stdout: "max": 0, 2024-09-15T22:02:17.881 INFO:tasks.workunit.client.0.smithi136.stdout: "pool": 1, 2024-09-15T22:02:17.881 INFO:tasks.workunit.client.0.smithi136.stdout: "namespace": "" 2024-09-15T22:02:17.881 INFO:tasks.workunit.client.0.smithi136.stdout: }, 2024-09-15T22:02:17.881 INFO:tasks.workunit.client.0.smithi136.stdout: "version": "17'41", 2024-09-15T22:02:17.881 INFO:tasks.workunit.client.0.smithi136.stdout: "prior_version": "16'20", 2024-09-15T22:02:17.881 INFO:tasks.workunit.client.0.smithi136.stdout: "last_reqid": "client.4297.0:1", 2024-09-15T22:02:17.881 INFO:tasks.workunit.client.0.smithi136.stdout: "user_version": 20, 2024-09-15T22:02:17.882 INFO:tasks.workunit.client.0.smithi136.stdout: "size": 256, 2024-09-15T22:02:17.882 INFO:tasks.workunit.client.0.smithi136.stdout: "mtime": "2024-09-15T22:01:06.077311+0000", 2024-09-15T22:02:17.882 INFO:tasks.workunit.client.0.smithi136.stdout: "local_mtime": "2024-09-15T22:01:06.077853+0000", 2024-09-15T22:02:17.882 INFO:tasks.workunit.client.0.smithi136.stdout: "lost": 0, 2024-09-15T22:02:17.882 INFO:tasks.workunit.client.0.smithi136.stdout: "flags": [ 2024-09-15T22:02:17.882 INFO:tasks.workunit.client.0.smithi136.stdout: "dirty", 2024-09-15T22:02:17.882 INFO:tasks.workunit.client.0.smithi136.stdout: "data_digest" 2024-09-15T22:02:17.882 INFO:tasks.workunit.client.0.smithi136.stdout: ], 2024-09-15T22:02:17.882 INFO:tasks.workunit.client.0.smithi136.stdout: "truncate_seq": 0, 2024-09-15T22:02:17.882 INFO:tasks.workunit.client.0.smithi136.stdout: "truncate_size": 0, 2024-09-15T22:02:17.882 INFO:tasks.workunit.client.0.smithi136.stdout: "data_digest": "0x29e45974", 2024-09-15T22:02:17.882 INFO:tasks.workunit.client.0.smithi136.stdout: "omap_digest": "0xffffffff", 2024-09-15T22:02:17.882 INFO:tasks.workunit.client.0.smithi136.stdout: "expected_object_size": 0, 2024-09-15T22:02:17.883 INFO:tasks.workunit.client.0.smithi136.stdout: "expected_write_size": 0, 2024-09-15T22:02:17.883 INFO:tasks.workunit.client.0.smithi136.stdout: "alloc_hint_flags": 0, 2024-09-15T22:02:17.883 INFO:tasks.workunit.client.0.smithi136.stdout: "manifest": { 2024-09-15T22:02:17.883 INFO:tasks.workunit.client.0.smithi136.stdout: "type": 0 2024-09-15T22:02:17.883 INFO:tasks.workunit.client.0.smithi136.stdout: }, 2024-09-15T22:02:17.883 INFO:tasks.workunit.client.0.smithi136.stdout: "watchers": {} 2024-09-15T22:02:17.883 INFO:tasks.workunit.client.0.smithi136.stdout: }, 2024-09-15T22:02:17.883 INFO:tasks.workunit.client.0.smithi136.stdout: "shards": [ 2024-09-15T22:02:17.883 INFO:tasks.workunit.client.0.smithi136.stdout: { 2024-09-15T22:02:17.883 INFO:tasks.workunit.client.0.smithi136.stdout: "osd": 0, 2024-09-15T22:02:17.883 INFO:tasks.workunit.client.0.smithi136.stdout: "primary": false, 2024-09-15T22:02:17.883 INFO:tasks.workunit.client.0.smithi136.stdout: "errors": [], 2024-09-15T22:02:17.883 INFO:tasks.workunit.client.0.smithi136.stdout: "size": 256 2024-09-15T22:02:17.884 INFO:tasks.workunit.client.0.smithi136.stdout: }, 2024-09-15T22:02:17.884 INFO:tasks.workunit.client.0.smithi136.stdout: { 2024-09-15T22:02:17.884 INFO:tasks.workunit.client.0.smithi136.stdout: "osd": 1, 2024-09-15T22:02:17.884 INFO:tasks.workunit.client.0.smithi136.stdout: "primary": true, 2024-09-15T22:02:17.884 INFO:tasks.workunit.client.0.smithi136.stdout: "errors": [ 2024-09-15T22:02:17.884 INFO:tasks.workunit.client.0.smithi136.stdout: "missing" 2024-09-15T22:02:17.884 INFO:tasks.workunit.client.0.smithi136.stdout: ] 2024-09-15T22:02:17.884 INFO:tasks.workunit.client.0.smithi136.stdout: } 2024-09-15T22:02:17.884 INFO:tasks.workunit.client.0.smithi136.stdout: ] 2024-09-15T22:02:17.884 INFO:tasks.workunit.client.0.smithi136.stdout: }, 2024-09-15T22:02:17.884 INFO:tasks.workunit.client.0.smithi136.stdout: { 2024-09-15T22:02:17.884 INFO:tasks.workunit.client.0.smithi136.stdout: "object": { 2024-09-15T22:02:17.884 INFO:tasks.workunit.client.0.smithi136.stdout: "name": "obj5", 2024-09-15T22:02:17.884 INFO:tasks.workunit.client.0.smithi136.stdout: "nspace": "", 2024-09-15T22:02:17.885 INFO:tasks.workunit.client.0.smithi136.stdout: "locator": "", 2024-09-15T22:02:17.885 INFO:tasks.workunit.client.0.smithi136.stdout: "snap": 4, 2024-09-15T22:02:17.885 INFO:tasks.workunit.client.0.smithi136.stdout: "version": 42 2024-09-15T22:02:17.885 INFO:tasks.workunit.client.0.smithi136.stdout: }, 2024-09-15T22:02:17.885 INFO:tasks.workunit.client.0.smithi136.stdout: "errors": [ 2024-09-15T22:02:17.885 INFO:tasks.workunit.client.0.smithi136.stdout: "size_mismatch" 2024-09-15T22:02:17.885 INFO:tasks.workunit.client.0.smithi136.stdout: ], 2024-09-15T22:02:17.885 INFO:tasks.workunit.client.0.smithi136.stdout: "union_shard_errors": [ 2024-09-15T22:02:17.885 INFO:tasks.workunit.client.0.smithi136.stdout: "size_mismatch_info", 2024-09-15T22:02:17.885 INFO:tasks.workunit.client.0.smithi136.stdout: "obj_size_info_mismatch" 2024-09-15T22:02:17.885 INFO:tasks.workunit.client.0.smithi136.stdout: ], 2024-09-15T22:02:17.885 INFO:tasks.workunit.client.0.smithi136.stdout: "selected_object_info": { 2024-09-15T22:02:17.885 INFO:tasks.workunit.client.0.smithi136.stdout: "oid": { 2024-09-15T22:02:17.885 INFO:tasks.workunit.client.0.smithi136.stdout: "oid": "obj5", 2024-09-15T22:02:17.886 INFO:tasks.workunit.client.0.smithi136.stdout: "key": "", 2024-09-15T22:02:17.886 INFO:tasks.workunit.client.0.smithi136.stdout: "snapid": 4, 2024-09-15T22:02:17.886 INFO:tasks.workunit.client.0.smithi136.stdout: "hash": 1718170787, 2024-09-15T22:02:17.886 INFO:tasks.workunit.client.0.smithi136.stdout: "max": 0, 2024-09-15T22:02:17.886 INFO:tasks.workunit.client.0.smithi136.stdout: "pool": 1, 2024-09-15T22:02:17.886 INFO:tasks.workunit.client.0.smithi136.stdout: "namespace": "" 2024-09-15T22:02:17.886 INFO:tasks.workunit.client.0.smithi136.stdout: }, 2024-09-15T22:02:17.886 INFO:tasks.workunit.client.0.smithi136.stdout: "version": "19'45", 2024-09-15T22:02:17.886 INFO:tasks.workunit.client.0.smithi136.stdout: "prior_version": "17'42", 2024-09-15T22:02:17.886 INFO:tasks.workunit.client.0.smithi136.stdout: "last_reqid": "client.4333.0:1", 2024-09-15T22:02:17.886 INFO:tasks.workunit.client.0.smithi136.stdout: "user_version": 42, 2024-09-15T22:02:17.886 INFO:tasks.workunit.client.0.smithi136.stdout: "size": 512, 2024-09-15T22:02:17.886 INFO:tasks.workunit.client.0.smithi136.stdout: "mtime": "2024-09-15T22:01:07.877144+0000", 2024-09-15T22:02:17.886 INFO:tasks.workunit.client.0.smithi136.stdout: "local_mtime": "2024-09-15T22:01:07.972396+0000", 2024-09-15T22:02:17.887 INFO:tasks.workunit.client.0.smithi136.stdout: "lost": 0, 2024-09-15T22:02:17.887 INFO:tasks.workunit.client.0.smithi136.stdout: "flags": [ 2024-09-15T22:02:17.887 INFO:tasks.workunit.client.0.smithi136.stdout: "dirty", 2024-09-15T22:02:17.887 INFO:tasks.workunit.client.0.smithi136.stdout: "data_digest" 2024-09-15T22:02:17.887 INFO:tasks.workunit.client.0.smithi136.stdout: ], 2024-09-15T22:02:17.887 INFO:tasks.workunit.client.0.smithi136.stdout: "truncate_seq": 0, 2024-09-15T22:02:17.887 INFO:tasks.workunit.client.0.smithi136.stdout: "truncate_size": 0, 2024-09-15T22:02:17.887 INFO:tasks.workunit.client.0.smithi136.stdout: "data_digest": "0xe054618c", 2024-09-15T22:02:17.887 INFO:tasks.workunit.client.0.smithi136.stdout: "omap_digest": "0xffffffff", 2024-09-15T22:02:17.887 INFO:tasks.workunit.client.0.smithi136.stdout: "expected_object_size": 0, 2024-09-15T22:02:17.887 INFO:tasks.workunit.client.0.smithi136.stdout: "expected_write_size": 0, 2024-09-15T22:02:17.887 INFO:tasks.workunit.client.0.smithi136.stdout: "alloc_hint_flags": 0, 2024-09-15T22:02:17.887 INFO:tasks.workunit.client.0.smithi136.stdout: "manifest": { 2024-09-15T22:02:17.887 INFO:tasks.workunit.client.0.smithi136.stdout: "type": 0 2024-09-15T22:02:17.888 INFO:tasks.workunit.client.0.smithi136.stdout: }, 2024-09-15T22:02:17.888 INFO:tasks.workunit.client.0.smithi136.stdout: "watchers": {} 2024-09-15T22:02:17.888 INFO:tasks.workunit.client.0.smithi136.stdout: }, 2024-09-15T22:02:17.888 INFO:tasks.workunit.client.0.smithi136.stdout: "shards": [ 2024-09-15T22:02:17.888 INFO:tasks.workunit.client.0.smithi136.stdout: { 2024-09-15T22:02:17.888 INFO:tasks.workunit.client.0.smithi136.stdout: "osd": 0, 2024-09-15T22:02:17.888 INFO:tasks.workunit.client.0.smithi136.stdout: "primary": false, 2024-09-15T22:02:17.888 INFO:tasks.workunit.client.0.smithi136.stdout: "errors": [], 2024-09-15T22:02:17.888 INFO:tasks.workunit.client.0.smithi136.stdout: "size": 512 2024-09-15T22:02:17.888 INFO:tasks.workunit.client.0.smithi136.stdout: }, 2024-09-15T22:02:17.888 INFO:tasks.workunit.client.0.smithi136.stdout: { 2024-09-15T22:02:17.888 INFO:tasks.workunit.client.0.smithi136.stdout: "osd": 1, 2024-09-15T22:02:17.888 INFO:tasks.workunit.client.0.smithi136.stdout: "primary": true, 2024-09-15T22:02:17.889 INFO:tasks.workunit.client.0.smithi136.stdout: "errors": [ 2024-09-15T22:02:17.889 INFO:tasks.workunit.client.0.smithi136.stdout: "size_mismatch_info", 2024-09-15T22:02:17.889 INFO:tasks.workunit.client.0.smithi136.stdout: "obj_size_info_mismatch" 2024-09-15T22:02:17.889 INFO:tasks.workunit.client.0.smithi136.stdout: ], 2024-09-15T22:02:17.889 INFO:tasks.workunit.client.0.smithi136.stdout: "size": 4608, 2024-09-15T22:02:17.889 INFO:tasks.workunit.client.0.smithi136.stdout: "object_info": { 2024-09-15T22:02:17.889 INFO:tasks.workunit.client.0.smithi136.stdout: "oid": { 2024-09-15T22:02:17.889 INFO:tasks.workunit.client.0.smithi136.stdout: "oid": "obj5", 2024-09-15T22:02:17.889 INFO:tasks.workunit.client.0.smithi136.stdout: "key": "", 2024-09-15T22:02:17.889 INFO:tasks.workunit.client.0.smithi136.stdout: "snapid": 4, 2024-09-15T22:02:17.889 INFO:tasks.workunit.client.0.smithi136.stdout: "hash": 1718170787, 2024-09-15T22:02:17.889 INFO:tasks.workunit.client.0.smithi136.stdout: "max": 0, 2024-09-15T22:02:17.889 INFO:tasks.workunit.client.0.smithi136.stdout: "pool": 1, 2024-09-15T22:02:17.889 INFO:tasks.workunit.client.0.smithi136.stdout: "namespace": "" 2024-09-15T22:02:17.890 INFO:tasks.workunit.client.0.smithi136.stdout: }, 2024-09-15T22:02:17.890 INFO:tasks.workunit.client.0.smithi136.stdout: "version": "19'45", 2024-09-15T22:02:17.890 INFO:tasks.workunit.client.0.smithi136.stdout: "prior_version": "17'42", 2024-09-15T22:02:17.890 INFO:tasks.workunit.client.0.smithi136.stdout: "last_reqid": "client.4333.0:1", 2024-09-15T22:02:17.890 INFO:tasks.workunit.client.0.smithi136.stdout: "user_version": 42, 2024-09-15T22:02:17.890 INFO:tasks.workunit.client.0.smithi136.stdout: "size": 512, 2024-09-15T22:02:17.890 INFO:tasks.workunit.client.0.smithi136.stdout: "mtime": "2024-09-15T22:01:07.877144+0000", 2024-09-15T22:02:17.890 INFO:tasks.workunit.client.0.smithi136.stdout: "local_mtime": "2024-09-15T22:01:07.972396+0000", 2024-09-15T22:02:17.890 INFO:tasks.workunit.client.0.smithi136.stdout: "lost": 0, 2024-09-15T22:02:17.890 INFO:tasks.workunit.client.0.smithi136.stdout: "flags": [ 2024-09-15T22:02:17.890 INFO:tasks.workunit.client.0.smithi136.stdout: "dirty", 2024-09-15T22:02:17.890 INFO:tasks.workunit.client.0.smithi136.stdout: "data_digest" 2024-09-15T22:02:17.890 INFO:tasks.workunit.client.0.smithi136.stdout: ], 2024-09-15T22:02:17.890 INFO:tasks.workunit.client.0.smithi136.stdout: "truncate_seq": 0, 2024-09-15T22:02:17.890 INFO:tasks.workunit.client.0.smithi136.stdout: "truncate_size": 0, 2024-09-15T22:02:17.891 INFO:tasks.workunit.client.0.smithi136.stdout: "data_digest": "0xe054618c", 2024-09-15T22:02:17.891 INFO:tasks.workunit.client.0.smithi136.stdout: "omap_digest": "0xffffffff", 2024-09-15T22:02:17.891 INFO:tasks.workunit.client.0.smithi136.stdout: "expected_object_size": 0, 2024-09-15T22:02:17.891 INFO:tasks.workunit.client.0.smithi136.stdout: "expected_write_size": 0, 2024-09-15T22:02:17.891 INFO:tasks.workunit.client.0.smithi136.stdout: "alloc_hint_flags": 0, 2024-09-15T22:02:17.891 INFO:tasks.workunit.client.0.smithi136.stdout: "manifest": { 2024-09-15T22:02:17.891 INFO:tasks.workunit.client.0.smithi136.stdout: "type": 0 2024-09-15T22:02:17.891 INFO:tasks.workunit.client.0.smithi136.stdout: }, 2024-09-15T22:02:17.891 INFO:tasks.workunit.client.0.smithi136.stdout: "watchers": {} 2024-09-15T22:02:17.891 INFO:tasks.workunit.client.0.smithi136.stdout: } 2024-09-15T22:02:17.891 INFO:tasks.workunit.client.0.smithi136.stdout: } 2024-09-15T22:02:17.891 INFO:tasks.workunit.client.0.smithi136.stdout: ] 2024-09-15T22:02:17.891 INFO:tasks.workunit.client.0.smithi136.stdout: }, 2024-09-15T22:02:17.891 INFO:tasks.workunit.client.0.smithi136.stdout: { 2024-09-15T22:02:17.892 INFO:tasks.workunit.client.0.smithi136.stdout: "object": { 2024-09-15T22:02:17.892 INFO:tasks.workunit.client.0.smithi136.stdout: "name": "obj4", 2024-09-15T22:02:17.892 INFO:tasks.workunit.client.0.smithi136.stdout: "nspace": "", 2024-09-15T22:02:17.892 INFO:tasks.workunit.client.0.smithi136.stdout: "locator": "", 2024-09-15T22:02:17.892 INFO:tasks.workunit.client.0.smithi136.stdout: "snap": 7, 2024-09-15T22:02:17.892 INFO:tasks.workunit.client.0.smithi136.stdout: "version": 4 2024-09-15T22:02:17.892 INFO:tasks.workunit.client.0.smithi136.stdout: }, 2024-09-15T22:02:17.892 INFO:tasks.workunit.client.0.smithi136.stdout: "errors": [], 2024-09-15T22:02:17.892 INFO:tasks.workunit.client.0.smithi136.stdout: "union_shard_errors": [ 2024-09-15T22:02:17.892 INFO:tasks.workunit.client.0.smithi136.stdout: "missing" 2024-09-15T22:02:17.892 INFO:tasks.workunit.client.0.smithi136.stdout: ], 2024-09-15T22:02:17.892 INFO:tasks.workunit.client.0.smithi136.stdout: "selected_object_info": { 2024-09-15T22:02:17.892 INFO:tasks.workunit.client.0.smithi136.stdout: "oid": { 2024-09-15T22:02:17.892 INFO:tasks.workunit.client.0.smithi136.stdout: "oid": "obj4", 2024-09-15T22:02:17.893 INFO:tasks.workunit.client.0.smithi136.stdout: "key": "", 2024-09-15T22:02:17.893 INFO:tasks.workunit.client.0.smithi136.stdout: "snapid": 7, 2024-09-15T22:02:17.893 INFO:tasks.workunit.client.0.smithi136.stdout: "hash": 2826278768, 2024-09-15T22:02:17.893 INFO:tasks.workunit.client.0.smithi136.stdout: "max": 0, 2024-09-15T22:02:17.893 INFO:tasks.workunit.client.0.smithi136.stdout: "pool": 1, 2024-09-15T22:02:17.893 INFO:tasks.workunit.client.0.smithi136.stdout: "namespace": "" 2024-09-15T22:02:17.893 INFO:tasks.workunit.client.0.smithi136.stdout: }, 2024-09-15T22:02:17.893 INFO:tasks.workunit.client.0.smithi136.stdout: "version": "22'51", 2024-09-15T22:02:17.893 INFO:tasks.workunit.client.0.smithi136.stdout: "prior_version": "15'4", 2024-09-15T22:02:17.893 INFO:tasks.workunit.client.0.smithi136.stdout: "last_reqid": "client.4246.0:1", 2024-09-15T22:02:17.893 INFO:tasks.workunit.client.0.smithi136.stdout: "user_version": 4, 2024-09-15T22:02:17.893 INFO:tasks.workunit.client.0.smithi136.stdout: "size": 1032, 2024-09-15T22:02:17.893 INFO:tasks.workunit.client.0.smithi136.stdout: "mtime": "2024-09-15T22:01:02.807649+0000", 2024-09-15T22:02:17.893 INFO:tasks.workunit.client.0.smithi136.stdout: "local_mtime": "2024-09-15T22:01:02.808218+0000", 2024-09-15T22:02:17.894 INFO:tasks.workunit.client.0.smithi136.stdout: "lost": 0, 2024-09-15T22:02:17.894 INFO:tasks.workunit.client.0.smithi136.stdout: "flags": [ 2024-09-15T22:02:17.894 INFO:tasks.workunit.client.0.smithi136.stdout: "dirty", 2024-09-15T22:02:17.894 INFO:tasks.workunit.client.0.smithi136.stdout: "data_digest" 2024-09-15T22:02:17.894 INFO:tasks.workunit.client.0.smithi136.stdout: ], 2024-09-15T22:02:17.894 INFO:tasks.workunit.client.0.smithi136.stdout: "truncate_seq": 0, 2024-09-15T22:02:17.894 INFO:tasks.workunit.client.0.smithi136.stdout: "truncate_size": 0, 2024-09-15T22:02:17.894 INFO:tasks.workunit.client.0.smithi136.stdout: "data_digest": "0x557795f2", 2024-09-15T22:02:17.894 INFO:tasks.workunit.client.0.smithi136.stdout: "omap_digest": "0xffffffff", 2024-09-15T22:02:17.894 INFO:tasks.workunit.client.0.smithi136.stdout: "expected_object_size": 0, 2024-09-15T22:02:17.894 INFO:tasks.workunit.client.0.smithi136.stdout: "expected_write_size": 0, 2024-09-15T22:02:17.894 INFO:tasks.workunit.client.0.smithi136.stdout: "alloc_hint_flags": 0, 2024-09-15T22:02:17.894 INFO:tasks.workunit.client.0.smithi136.stdout: "manifest": { 2024-09-15T22:02:17.894 INFO:tasks.workunit.client.0.smithi136.stdout: "type": 0 2024-09-15T22:02:17.894 INFO:tasks.workunit.client.0.smithi136.stdout: }, 2024-09-15T22:02:17.895 INFO:tasks.workunit.client.0.smithi136.stdout: "watchers": {} 2024-09-15T22:02:17.895 INFO:tasks.workunit.client.0.smithi136.stdout: }, 2024-09-15T22:02:17.895 INFO:tasks.workunit.client.0.smithi136.stdout: "shards": [ 2024-09-15T22:02:17.895 INFO:tasks.workunit.client.0.smithi136.stdout: { 2024-09-15T22:02:17.895 INFO:tasks.workunit.client.0.smithi136.stdout: "osd": 0, 2024-09-15T22:02:17.895 INFO:tasks.workunit.client.0.smithi136.stdout: "primary": false, 2024-09-15T22:02:17.895 INFO:tasks.workunit.client.0.smithi136.stdout: "errors": [], 2024-09-15T22:02:17.895 INFO:tasks.workunit.client.0.smithi136.stdout: "size": 1032 2024-09-15T22:02:17.895 INFO:tasks.workunit.client.0.smithi136.stdout: }, 2024-09-15T22:02:17.895 INFO:tasks.workunit.client.0.smithi136.stdout: { 2024-09-15T22:02:17.895 INFO:tasks.workunit.client.0.smithi136.stdout: "osd": 1, 2024-09-15T22:02:17.895 INFO:tasks.workunit.client.0.smithi136.stdout: "primary": true, 2024-09-15T22:02:17.895 INFO:tasks.workunit.client.0.smithi136.stdout: "errors": [ 2024-09-15T22:02:17.895 INFO:tasks.workunit.client.0.smithi136.stdout: "missing" 2024-09-15T22:02:17.895 INFO:tasks.workunit.client.0.smithi136.stdout: ] 2024-09-15T22:02:17.896 INFO:tasks.workunit.client.0.smithi136.stdout: } 2024-09-15T22:02:17.896 INFO:tasks.workunit.client.0.smithi136.stdout: ] 2024-09-15T22:02:17.896 INFO:tasks.workunit.client.0.smithi136.stdout: }, 2024-09-15T22:02:17.896 INFO:tasks.workunit.client.0.smithi136.stdout: { 2024-09-15T22:02:17.896 INFO:tasks.workunit.client.0.smithi136.stdout: "object": { 2024-09-15T22:02:17.896 INFO:tasks.workunit.client.0.smithi136.stdout: "name": "obj5", 2024-09-15T22:02:17.896 INFO:tasks.workunit.client.0.smithi136.stdout: "nspace": "", 2024-09-15T22:02:17.896 INFO:tasks.workunit.client.0.smithi136.stdout: "locator": "", 2024-09-15T22:02:17.896 INFO:tasks.workunit.client.0.smithi136.stdout: "snap": 7, 2024-09-15T22:02:17.896 INFO:tasks.workunit.client.0.smithi136.stdout: "version": 0 2024-09-15T22:02:17.896 INFO:tasks.workunit.client.0.smithi136.stdout: }, 2024-09-15T22:02:17.896 INFO:tasks.workunit.client.0.smithi136.stdout: "errors": [], 2024-09-15T22:02:17.896 INFO:tasks.workunit.client.0.smithi136.stdout: "union_shard_errors": [ 2024-09-15T22:02:17.896 INFO:tasks.workunit.client.0.smithi136.stdout: "missing", 2024-09-15T22:02:17.897 INFO:tasks.workunit.client.0.smithi136.stdout: "info_missing" 2024-09-15T22:02:17.897 INFO:tasks.workunit.client.0.smithi136.stdout: ], 2024-09-15T22:02:17.897 INFO:tasks.workunit.client.0.smithi136.stdout: "shards": [ 2024-09-15T22:02:17.897 INFO:tasks.workunit.client.0.smithi136.stdout: { 2024-09-15T22:02:17.897 INFO:tasks.workunit.client.0.smithi136.stdout: "osd": 0, 2024-09-15T22:02:17.897 INFO:tasks.workunit.client.0.smithi136.stdout: "primary": false, 2024-09-15T22:02:17.897 INFO:tasks.workunit.client.0.smithi136.stdout: "errors": [ 2024-09-15T22:02:17.897 INFO:tasks.workunit.client.0.smithi136.stdout: "missing" 2024-09-15T22:02:17.897 INFO:tasks.workunit.client.0.smithi136.stdout: ] 2024-09-15T22:02:17.897 INFO:tasks.workunit.client.0.smithi136.stdout: }, 2024-09-15T22:02:17.897 INFO:tasks.workunit.client.0.smithi136.stdout: { 2024-09-15T22:02:17.897 INFO:tasks.workunit.client.0.smithi136.stdout: "osd": 1, 2024-09-15T22:02:17.897 INFO:tasks.workunit.client.0.smithi136.stdout: "primary": true, 2024-09-15T22:02:17.897 INFO:tasks.workunit.client.0.smithi136.stdout: "errors": [ 2024-09-15T22:02:17.898 INFO:tasks.workunit.client.0.smithi136.stdout: "info_missing" 2024-09-15T22:02:17.898 INFO:tasks.workunit.client.0.smithi136.stdout: ], 2024-09-15T22:02:17.898 INFO:tasks.workunit.client.0.smithi136.stdout: "size": 1792 2024-09-15T22:02:17.898 INFO:tasks.workunit.client.0.smithi136.stdout: } 2024-09-15T22:02:17.898 INFO:tasks.workunit.client.0.smithi136.stdout: ] 2024-09-15T22:02:17.898 INFO:tasks.workunit.client.0.smithi136.stdout: }, 2024-09-15T22:02:17.898 INFO:tasks.workunit.client.0.smithi136.stdout: { 2024-09-15T22:02:17.898 INFO:tasks.workunit.client.0.smithi136.stdout: "object": { 2024-09-15T22:02:17.898 INFO:tasks.workunit.client.0.smithi136.stdout: "name": "obj1", 2024-09-15T22:02:17.898 INFO:tasks.workunit.client.0.smithi136.stdout: "nspace": "", 2024-09-15T22:02:17.898 INFO:tasks.workunit.client.0.smithi136.stdout: "locator": "", 2024-09-15T22:02:17.898 INFO:tasks.workunit.client.0.smithi136.stdout: "snap": "head", 2024-09-15T22:02:17.898 INFO:tasks.workunit.client.0.smithi136.stdout: "version": 18 2024-09-15T22:02:17.898 INFO:tasks.workunit.client.0.smithi136.stdout: }, 2024-09-15T22:02:17.899 INFO:tasks.workunit.client.0.smithi136.stdout: "errors": [], 2024-09-15T22:02:17.899 INFO:tasks.workunit.client.0.smithi136.stdout: "union_shard_errors": [ 2024-09-15T22:02:17.899 INFO:tasks.workunit.client.0.smithi136.stdout: "missing" 2024-09-15T22:02:17.899 INFO:tasks.workunit.client.0.smithi136.stdout: ], 2024-09-15T22:02:17.899 INFO:tasks.workunit.client.0.smithi136.stdout: "selected_object_info": { 2024-09-15T22:02:17.899 INFO:tasks.workunit.client.0.smithi136.stdout: "oid": { 2024-09-15T22:02:17.899 INFO:tasks.workunit.client.0.smithi136.stdout: "oid": "obj1", 2024-09-15T22:02:17.899 INFO:tasks.workunit.client.0.smithi136.stdout: "key": "", 2024-09-15T22:02:17.899 INFO:tasks.workunit.client.0.smithi136.stdout: "snapid": -2, 2024-09-15T22:02:17.899 INFO:tasks.workunit.client.0.smithi136.stdout: "hash": 1828249343, 2024-09-15T22:02:17.899 INFO:tasks.workunit.client.0.smithi136.stdout: "max": 0, 2024-09-15T22:02:17.899 INFO:tasks.workunit.client.0.smithi136.stdout: "pool": 1, 2024-09-15T22:02:17.899 INFO:tasks.workunit.client.0.smithi136.stdout: "namespace": "" 2024-09-15T22:02:17.899 INFO:tasks.workunit.client.0.smithi136.stdout: }, 2024-09-15T22:02:17.899 INFO:tasks.workunit.client.0.smithi136.stdout: "version": "16'18", 2024-09-15T22:02:17.900 INFO:tasks.workunit.client.0.smithi136.stdout: "prior_version": "15'1", 2024-09-15T22:02:17.900 INFO:tasks.workunit.client.0.smithi136.stdout: "last_reqid": "client.4294.0:1", 2024-09-15T22:02:17.900 INFO:tasks.workunit.client.0.smithi136.stdout: "user_version": 18, 2024-09-15T22:02:17.900 INFO:tasks.workunit.client.0.smithi136.stdout: "size": 256, 2024-09-15T22:02:17.900 INFO:tasks.workunit.client.0.smithi136.stdout: "mtime": "2024-09-15T22:01:05.851781+0000", 2024-09-15T22:02:17.900 INFO:tasks.workunit.client.0.smithi136.stdout: "local_mtime": "2024-09-15T22:01:05.852303+0000", 2024-09-15T22:02:17.900 INFO:tasks.workunit.client.0.smithi136.stdout: "lost": 0, 2024-09-15T22:02:17.900 INFO:tasks.workunit.client.0.smithi136.stdout: "flags": [ 2024-09-15T22:02:17.900 INFO:tasks.workunit.client.0.smithi136.stdout: "dirty", 2024-09-15T22:02:17.900 INFO:tasks.workunit.client.0.smithi136.stdout: "data_digest" 2024-09-15T22:02:17.900 INFO:tasks.workunit.client.0.smithi136.stdout: ], 2024-09-15T22:02:17.900 INFO:tasks.workunit.client.0.smithi136.stdout: "truncate_seq": 0, 2024-09-15T22:02:17.900 INFO:tasks.workunit.client.0.smithi136.stdout: "truncate_size": 0, 2024-09-15T22:02:17.900 INFO:tasks.workunit.client.0.smithi136.stdout: "data_digest": "0x29e45974", 2024-09-15T22:02:17.901 INFO:tasks.workunit.client.0.smithi136.stdout: "omap_digest": "0xffffffff", 2024-09-15T22:02:17.901 INFO:tasks.workunit.client.0.smithi136.stdout: "expected_object_size": 0, 2024-09-15T22:02:17.901 INFO:tasks.workunit.client.0.smithi136.stdout: "expected_write_size": 0, 2024-09-15T22:02:17.901 INFO:tasks.workunit.client.0.smithi136.stdout: "alloc_hint_flags": 0, 2024-09-15T22:02:17.901 INFO:tasks.workunit.client.0.smithi136.stdout: "manifest": { 2024-09-15T22:02:17.901 INFO:tasks.workunit.client.0.smithi136.stdout: "type": 0 2024-09-15T22:02:17.901 INFO:tasks.workunit.client.0.smithi136.stdout: }, 2024-09-15T22:02:17.901 INFO:tasks.workunit.client.0.smithi136.stdout: "watchers": {} 2024-09-15T22:02:17.901 INFO:tasks.workunit.client.0.smithi136.stdout: }, 2024-09-15T22:02:17.901 INFO:tasks.workunit.client.0.smithi136.stdout: "shards": [ 2024-09-15T22:02:17.901 INFO:tasks.workunit.client.0.smithi136.stdout: { 2024-09-15T22:02:17.901 INFO:tasks.workunit.client.0.smithi136.stdout: "osd": 0, 2024-09-15T22:02:17.901 INFO:tasks.workunit.client.0.smithi136.stdout: "primary": false, 2024-09-15T22:02:17.901 INFO:tasks.workunit.client.0.smithi136.stdout: "errors": [], 2024-09-15T22:02:17.902 INFO:tasks.workunit.client.0.smithi136.stdout: "size": 256 2024-09-15T22:02:17.902 INFO:tasks.workunit.client.0.smithi136.stdout: }, 2024-09-15T22:02:17.902 INFO:tasks.workunit.client.0.smithi136.stdout: { 2024-09-15T22:02:17.902 INFO:tasks.workunit.client.0.smithi136.stdout: "osd": 1, 2024-09-15T22:02:17.902 INFO:tasks.workunit.client.0.smithi136.stdout: "primary": true, 2024-09-15T22:02:17.902 INFO:tasks.workunit.client.0.smithi136.stdout: "errors": [ 2024-09-15T22:02:17.902 INFO:tasks.workunit.client.0.smithi136.stdout: "missing" 2024-09-15T22:02:17.902 INFO:tasks.workunit.client.0.smithi136.stdout: ] 2024-09-15T22:02:17.902 INFO:tasks.workunit.client.0.smithi136.stdout: } 2024-09-15T22:02:17.902 INFO:tasks.workunit.client.0.smithi136.stdout: ] 2024-09-15T22:02:17.902 INFO:tasks.workunit.client.0.smithi136.stdout: }, 2024-09-15T22:02:17.902 INFO:tasks.workunit.client.0.smithi136.stdout: { 2024-09-15T22:02:17.902 INFO:tasks.workunit.client.0.smithi136.stdout: "object": { 2024-09-15T22:02:17.902 INFO:tasks.workunit.client.0.smithi136.stdout: "name": "obj10", 2024-09-15T22:02:17.902 INFO:tasks.workunit.client.0.smithi136.stdout: "nspace": "", 2024-09-15T22:02:17.903 INFO:tasks.workunit.client.0.smithi136.stdout: "locator": "", 2024-09-15T22:02:17.903 INFO:tasks.workunit.client.0.smithi136.stdout: "snap": "head", 2024-09-15T22:02:17.903 INFO:tasks.workunit.client.0.smithi136.stdout: "version": 32 2024-09-15T22:02:17.903 INFO:tasks.workunit.client.0.smithi136.stdout: }, 2024-09-15T22:02:17.903 INFO:tasks.workunit.client.0.smithi136.stdout: "errors": [ 2024-09-15T22:02:17.903 INFO:tasks.workunit.client.0.smithi136.stdout: "snapset_inconsistency" 2024-09-15T22:02:17.903 INFO:tasks.workunit.client.0.smithi136.stdout: ], 2024-09-15T22:02:17.903 INFO:tasks.workunit.client.0.smithi136.stdout: "union_shard_errors": [], 2024-09-15T22:02:17.903 INFO:tasks.workunit.client.0.smithi136.stdout: "selected_object_info": { 2024-09-15T22:02:17.903 INFO:tasks.workunit.client.0.smithi136.stdout: "oid": { 2024-09-15T22:02:17.903 INFO:tasks.workunit.client.0.smithi136.stdout: "oid": "obj10", 2024-09-15T22:02:17.903 INFO:tasks.workunit.client.0.smithi136.stdout: "key": "", 2024-09-15T22:02:17.903 INFO:tasks.workunit.client.0.smithi136.stdout: "snapid": -2, 2024-09-15T22:02:17.903 INFO:tasks.workunit.client.0.smithi136.stdout: "hash": 718195851, 2024-09-15T22:02:17.903 INFO:tasks.workunit.client.0.smithi136.stdout: "max": 0, 2024-09-15T22:02:17.904 INFO:tasks.workunit.client.0.smithi136.stdout: "pool": 1, 2024-09-15T22:02:17.904 INFO:tasks.workunit.client.0.smithi136.stdout: "namespace": "" 2024-09-15T22:02:17.904 INFO:tasks.workunit.client.0.smithi136.stdout: }, 2024-09-15T22:02:17.904 INFO:tasks.workunit.client.0.smithi136.stdout: "version": "16'32", 2024-09-15T22:02:17.904 INFO:tasks.workunit.client.0.smithi136.stdout: "prior_version": "15'10", 2024-09-15T22:02:17.904 INFO:tasks.workunit.client.0.smithi136.stdout: "last_reqid": "client.4315.0:1", 2024-09-15T22:02:17.904 INFO:tasks.workunit.client.0.smithi136.stdout: "user_version": 32, 2024-09-15T22:02:17.904 INFO:tasks.workunit.client.0.smithi136.stdout: "size": 256, 2024-09-15T22:02:17.904 INFO:tasks.workunit.client.0.smithi136.stdout: "mtime": "2024-09-15T22:01:06.946816+0000", 2024-09-15T22:02:17.904 INFO:tasks.workunit.client.0.smithi136.stdout: "local_mtime": "2024-09-15T22:01:06.947382+0000", 2024-09-15T22:02:17.904 INFO:tasks.workunit.client.0.smithi136.stdout: "lost": 0, 2024-09-15T22:02:17.904 INFO:tasks.workunit.client.0.smithi136.stdout: "flags": [ 2024-09-15T22:02:17.904 INFO:tasks.workunit.client.0.smithi136.stdout: "dirty", 2024-09-15T22:02:17.904 INFO:tasks.workunit.client.0.smithi136.stdout: "data_digest" 2024-09-15T22:02:17.905 INFO:tasks.workunit.client.0.smithi136.stdout: ], 2024-09-15T22:02:17.905 INFO:tasks.workunit.client.0.smithi136.stdout: "truncate_seq": 0, 2024-09-15T22:02:17.905 INFO:tasks.workunit.client.0.smithi136.stdout: "truncate_size": 0, 2024-09-15T22:02:17.905 INFO:tasks.workunit.client.0.smithi136.stdout: "data_digest": "0x29e45974", 2024-09-15T22:02:17.905 INFO:tasks.workunit.client.0.smithi136.stdout: "omap_digest": "0xffffffff", 2024-09-15T22:02:17.905 INFO:tasks.workunit.client.0.smithi136.stdout: "expected_object_size": 0, 2024-09-15T22:02:17.905 INFO:tasks.workunit.client.0.smithi136.stdout: "expected_write_size": 0, 2024-09-15T22:02:17.905 INFO:tasks.workunit.client.0.smithi136.stdout: "alloc_hint_flags": 0, 2024-09-15T22:02:17.905 INFO:tasks.workunit.client.0.smithi136.stdout: "manifest": { 2024-09-15T22:02:17.905 INFO:tasks.workunit.client.0.smithi136.stdout: "type": 0 2024-09-15T22:02:17.906 INFO:tasks.workunit.client.0.smithi136.stdout: }, 2024-09-15T22:02:17.906 INFO:tasks.workunit.client.0.smithi136.stdout: "watchers": {} 2024-09-15T22:02:17.906 INFO:tasks.workunit.client.0.smithi136.stdout: }, 2024-09-15T22:02:17.906 INFO:tasks.workunit.client.0.smithi136.stdout: "shards": [ 2024-09-15T22:02:17.906 INFO:tasks.workunit.client.0.smithi136.stdout: { 2024-09-15T22:02:17.906 INFO:tasks.workunit.client.0.smithi136.stdout: "osd": 0, 2024-09-15T22:02:17.906 INFO:tasks.workunit.client.0.smithi136.stdout: "primary": false, 2024-09-15T22:02:17.906 INFO:tasks.workunit.client.0.smithi136.stdout: "errors": [], 2024-09-15T22:02:17.906 INFO:tasks.workunit.client.0.smithi136.stdout: "size": 256, 2024-09-15T22:02:17.906 INFO:tasks.workunit.client.0.smithi136.stdout: "snapset": { 2024-09-15T22:02:17.906 INFO:tasks.workunit.client.0.smithi136.stdout: "seq": 1, 2024-09-15T22:02:17.906 INFO:tasks.workunit.client.0.smithi136.stdout: "clones": [ 2024-09-15T22:02:17.906 INFO:tasks.workunit.client.0.smithi136.stdout: { 2024-09-15T22:02:17.907 INFO:tasks.workunit.client.0.smithi136.stdout: "snap": 1, 2024-09-15T22:02:17.907 INFO:tasks.workunit.client.0.smithi136.stdout: "size": 1032, 2024-09-15T22:02:17.907 INFO:tasks.workunit.client.0.smithi136.stdout: "overlap": "[]", 2024-09-15T22:02:17.907 INFO:tasks.workunit.client.0.smithi136.stdout: "snaps": [ 2024-09-15T22:02:17.907 INFO:tasks.workunit.client.0.smithi136.stdout: 1 2024-09-15T22:02:17.907 INFO:tasks.workunit.client.0.smithi136.stdout: ] 2024-09-15T22:02:17.907 INFO:tasks.workunit.client.0.smithi136.stdout: } 2024-09-15T22:02:17.907 INFO:tasks.workunit.client.0.smithi136.stdout: ] 2024-09-15T22:02:17.907 INFO:tasks.workunit.client.0.smithi136.stdout: } 2024-09-15T22:02:17.907 INFO:tasks.workunit.client.0.smithi136.stdout: }, 2024-09-15T22:02:17.907 INFO:tasks.workunit.client.0.smithi136.stdout: { 2024-09-15T22:02:17.907 INFO:tasks.workunit.client.0.smithi136.stdout: "osd": 1, 2024-09-15T22:02:17.907 INFO:tasks.workunit.client.0.smithi136.stdout: "primary": true, 2024-09-15T22:02:17.907 INFO:tasks.workunit.client.0.smithi136.stdout: "errors": [], 2024-09-15T22:02:17.908 INFO:tasks.workunit.client.0.smithi136.stdout: "size": 256, 2024-09-15T22:02:17.908 INFO:tasks.workunit.client.0.smithi136.stdout: "snapset": { 2024-09-15T22:02:17.908 INFO:tasks.workunit.client.0.smithi136.stdout: "seq": 1, 2024-09-15T22:02:17.908 INFO:tasks.workunit.client.0.smithi136.stdout: "clones": [ 2024-09-15T22:02:17.908 INFO:tasks.workunit.client.0.smithi136.stdout: { 2024-09-15T22:02:17.908 INFO:tasks.workunit.client.0.smithi136.stdout: "snap": 1, 2024-09-15T22:02:17.908 INFO:tasks.workunit.client.0.smithi136.stdout: "size": 1032, 2024-09-15T22:02:17.908 INFO:tasks.workunit.client.0.smithi136.stdout: "overlap": "????", 2024-09-15T22:02:17.908 INFO:tasks.workunit.client.0.smithi136.stdout: "snaps": [ 2024-09-15T22:02:17.908 INFO:tasks.workunit.client.0.smithi136.stdout: 1 2024-09-15T22:02:17.908 INFO:tasks.workunit.client.0.smithi136.stdout: ] 2024-09-15T22:02:17.908 INFO:tasks.workunit.client.0.smithi136.stdout: } 2024-09-15T22:02:17.908 INFO:tasks.workunit.client.0.smithi136.stdout: ] 2024-09-15T22:02:17.908 INFO:tasks.workunit.client.0.smithi136.stdout: } 2024-09-15T22:02:17.909 INFO:tasks.workunit.client.0.smithi136.stdout: } 2024-09-15T22:02:17.909 INFO:tasks.workunit.client.0.smithi136.stdout: ] 2024-09-15T22:02:17.909 INFO:tasks.workunit.client.0.smithi136.stdout: }, 2024-09-15T22:02:17.909 INFO:tasks.workunit.client.0.smithi136.stdout: { 2024-09-15T22:02:17.909 INFO:tasks.workunit.client.0.smithi136.stdout: "object": { 2024-09-15T22:02:17.909 INFO:tasks.workunit.client.0.smithi136.stdout: "name": "obj11", 2024-09-15T22:02:17.909 INFO:tasks.workunit.client.0.smithi136.stdout: "nspace": "", 2024-09-15T22:02:17.909 INFO:tasks.workunit.client.0.smithi136.stdout: "locator": "", 2024-09-15T22:02:17.909 INFO:tasks.workunit.client.0.smithi136.stdout: "snap": "head", 2024-09-15T22:02:17.909 INFO:tasks.workunit.client.0.smithi136.stdout: "version": 34 2024-09-15T22:02:17.909 INFO:tasks.workunit.client.0.smithi136.stdout: }, 2024-09-15T22:02:17.909 INFO:tasks.workunit.client.0.smithi136.stdout: "errors": [ 2024-09-15T22:02:17.909 INFO:tasks.workunit.client.0.smithi136.stdout: "snapset_inconsistency" 2024-09-15T22:02:17.909 INFO:tasks.workunit.client.0.smithi136.stdout: ], 2024-09-15T22:02:17.909 INFO:tasks.workunit.client.0.smithi136.stdout: "union_shard_errors": [], 2024-09-15T22:02:17.910 INFO:tasks.workunit.client.0.smithi136.stdout: "selected_object_info": { 2024-09-15T22:02:17.910 INFO:tasks.workunit.client.0.smithi136.stdout: "oid": { 2024-09-15T22:02:17.910 INFO:tasks.workunit.client.0.smithi136.stdout: "oid": "obj11", 2024-09-15T22:02:17.910 INFO:tasks.workunit.client.0.smithi136.stdout: "key": "", 2024-09-15T22:02:17.910 INFO:tasks.workunit.client.0.smithi136.stdout: "snapid": -2, 2024-09-15T22:02:17.910 INFO:tasks.workunit.client.0.smithi136.stdout: "hash": 693400951, 2024-09-15T22:02:17.910 INFO:tasks.workunit.client.0.smithi136.stdout: "max": 0, 2024-09-15T22:02:17.910 INFO:tasks.workunit.client.0.smithi136.stdout: "pool": 1, 2024-09-15T22:02:17.910 INFO:tasks.workunit.client.0.smithi136.stdout: "namespace": "" 2024-09-15T22:02:17.910 INFO:tasks.workunit.client.0.smithi136.stdout: }, 2024-09-15T22:02:17.910 INFO:tasks.workunit.client.0.smithi136.stdout: "version": "16'34", 2024-09-15T22:02:17.910 INFO:tasks.workunit.client.0.smithi136.stdout: "prior_version": "15'11", 2024-09-15T22:02:17.910 INFO:tasks.workunit.client.0.smithi136.stdout: "last_reqid": "client.4318.0:1", 2024-09-15T22:02:17.910 INFO:tasks.workunit.client.0.smithi136.stdout: "user_version": 34, 2024-09-15T22:02:17.911 INFO:tasks.workunit.client.0.smithi136.stdout: "size": 256, 2024-09-15T22:02:17.911 INFO:tasks.workunit.client.0.smithi136.stdout: "mtime": "2024-09-15T22:01:07.081184+0000", 2024-09-15T22:02:17.911 INFO:tasks.workunit.client.0.smithi136.stdout: "local_mtime": "2024-09-15T22:01:07.081812+0000", 2024-09-15T22:02:17.911 INFO:tasks.workunit.client.0.smithi136.stdout: "lost": 0, 2024-09-15T22:02:17.911 INFO:tasks.workunit.client.0.smithi136.stdout: "flags": [ 2024-09-15T22:02:17.911 INFO:tasks.workunit.client.0.smithi136.stdout: "dirty", 2024-09-15T22:02:17.911 INFO:tasks.workunit.client.0.smithi136.stdout: "data_digest" 2024-09-15T22:02:17.911 INFO:tasks.workunit.client.0.smithi136.stdout: ], 2024-09-15T22:02:17.911 INFO:tasks.workunit.client.0.smithi136.stdout: "truncate_seq": 0, 2024-09-15T22:02:17.911 INFO:tasks.workunit.client.0.smithi136.stdout: "truncate_size": 0, 2024-09-15T22:02:17.911 INFO:tasks.workunit.client.0.smithi136.stdout: "data_digest": "0x29e45974", 2024-09-15T22:02:17.911 INFO:tasks.workunit.client.0.smithi136.stdout: "omap_digest": "0xffffffff", 2024-09-15T22:02:17.911 INFO:tasks.workunit.client.0.smithi136.stdout: "expected_object_size": 0, 2024-09-15T22:02:17.911 INFO:tasks.workunit.client.0.smithi136.stdout: "expected_write_size": 0, 2024-09-15T22:02:17.911 INFO:tasks.workunit.client.0.smithi136.stdout: "alloc_hint_flags": 0, 2024-09-15T22:02:17.912 INFO:tasks.workunit.client.0.smithi136.stdout: "manifest": { 2024-09-15T22:02:17.912 INFO:tasks.workunit.client.0.smithi136.stdout: "type": 0 2024-09-15T22:02:17.912 INFO:tasks.workunit.client.0.smithi136.stdout: }, 2024-09-15T22:02:17.912 INFO:tasks.workunit.client.0.smithi136.stdout: "watchers": {} 2024-09-15T22:02:17.912 INFO:tasks.workunit.client.0.smithi136.stdout: }, 2024-09-15T22:02:17.912 INFO:tasks.workunit.client.0.smithi136.stdout: "shards": [ 2024-09-15T22:02:17.912 INFO:tasks.workunit.client.0.smithi136.stdout: { 2024-09-15T22:02:17.912 INFO:tasks.workunit.client.0.smithi136.stdout: "osd": 0, 2024-09-15T22:02:17.912 INFO:tasks.workunit.client.0.smithi136.stdout: "primary": false, 2024-09-15T22:02:17.912 INFO:tasks.workunit.client.0.smithi136.stdout: "errors": [], 2024-09-15T22:02:17.912 INFO:tasks.workunit.client.0.smithi136.stdout: "size": 256, 2024-09-15T22:02:17.912 INFO:tasks.workunit.client.0.smithi136.stdout: "snapset": { 2024-09-15T22:02:17.912 INFO:tasks.workunit.client.0.smithi136.stdout: "seq": 1, 2024-09-15T22:02:17.912 INFO:tasks.workunit.client.0.smithi136.stdout: "clones": [ 2024-09-15T22:02:17.913 INFO:tasks.workunit.client.0.smithi136.stdout: { 2024-09-15T22:02:17.913 INFO:tasks.workunit.client.0.smithi136.stdout: "snap": 1, 2024-09-15T22:02:17.913 INFO:tasks.workunit.client.0.smithi136.stdout: "size": 1032, 2024-09-15T22:02:17.913 INFO:tasks.workunit.client.0.smithi136.stdout: "overlap": "[]", 2024-09-15T22:02:17.913 INFO:tasks.workunit.client.0.smithi136.stdout: "snaps": [ 2024-09-15T22:02:17.913 INFO:tasks.workunit.client.0.smithi136.stdout: 1 2024-09-15T22:02:17.913 INFO:tasks.workunit.client.0.smithi136.stdout: ] 2024-09-15T22:02:17.913 INFO:tasks.workunit.client.0.smithi136.stdout: } 2024-09-15T22:02:17.913 INFO:tasks.workunit.client.0.smithi136.stdout: ] 2024-09-15T22:02:17.913 INFO:tasks.workunit.client.0.smithi136.stdout: } 2024-09-15T22:02:17.913 INFO:tasks.workunit.client.0.smithi136.stdout: }, 2024-09-15T22:02:17.913 INFO:tasks.workunit.client.0.smithi136.stdout: { 2024-09-15T22:02:17.913 INFO:tasks.workunit.client.0.smithi136.stdout: "osd": 1, 2024-09-15T22:02:17.913 INFO:tasks.workunit.client.0.smithi136.stdout: "primary": true, 2024-09-15T22:02:17.914 INFO:tasks.workunit.client.0.smithi136.stdout: "errors": [], 2024-09-15T22:02:17.914 INFO:tasks.workunit.client.0.smithi136.stdout: "size": 256, 2024-09-15T22:02:17.914 INFO:tasks.workunit.client.0.smithi136.stdout: "snapset": { 2024-09-15T22:02:17.914 INFO:tasks.workunit.client.0.smithi136.stdout: "seq": 1, 2024-09-15T22:02:17.914 INFO:tasks.workunit.client.0.smithi136.stdout: "clones": [] 2024-09-15T22:02:17.914 INFO:tasks.workunit.client.0.smithi136.stdout: } 2024-09-15T22:02:17.914 INFO:tasks.workunit.client.0.smithi136.stdout: } 2024-09-15T22:02:17.914 INFO:tasks.workunit.client.0.smithi136.stdout: ] 2024-09-15T22:02:17.914 INFO:tasks.workunit.client.0.smithi136.stdout: }, 2024-09-15T22:02:17.914 INFO:tasks.workunit.client.0.smithi136.stdout: { 2024-09-15T22:02:17.914 INFO:tasks.workunit.client.0.smithi136.stdout: "object": { 2024-09-15T22:02:17.914 INFO:tasks.workunit.client.0.smithi136.stdout: "name": "obj13", 2024-09-15T22:02:17.914 INFO:tasks.workunit.client.0.smithi136.stdout: "nspace": "", 2024-09-15T22:02:17.914 INFO:tasks.workunit.client.0.smithi136.stdout: "locator": "", 2024-09-15T22:02:17.915 INFO:tasks.workunit.client.0.smithi136.stdout: "snap": "head", 2024-09-15T22:02:17.915 INFO:tasks.workunit.client.0.smithi136.stdout: "version": 38 2024-09-15T22:02:17.915 INFO:tasks.workunit.client.0.smithi136.stdout: }, 2024-09-15T22:02:17.915 INFO:tasks.workunit.client.0.smithi136.stdout: "errors": [ 2024-09-15T22:02:17.915 INFO:tasks.workunit.client.0.smithi136.stdout: "snapset_inconsistency" 2024-09-15T22:02:17.915 INFO:tasks.workunit.client.0.smithi136.stdout: ], 2024-09-15T22:02:17.915 INFO:tasks.workunit.client.0.smithi136.stdout: "union_shard_errors": [], 2024-09-15T22:02:17.915 INFO:tasks.workunit.client.0.smithi136.stdout: "selected_object_info": { 2024-09-15T22:02:17.915 INFO:tasks.workunit.client.0.smithi136.stdout: "oid": { 2024-09-15T22:02:17.915 INFO:tasks.workunit.client.0.smithi136.stdout: "oid": "obj13", 2024-09-15T22:02:17.915 INFO:tasks.workunit.client.0.smithi136.stdout: "key": "", 2024-09-15T22:02:17.915 INFO:tasks.workunit.client.0.smithi136.stdout: "snapid": -2, 2024-09-15T22:02:17.915 INFO:tasks.workunit.client.0.smithi136.stdout: "hash": 2087409765, 2024-09-15T22:02:17.915 INFO:tasks.workunit.client.0.smithi136.stdout: "max": 0, 2024-09-15T22:02:17.916 INFO:tasks.workunit.client.0.smithi136.stdout: "pool": 1, 2024-09-15T22:02:17.916 INFO:tasks.workunit.client.0.smithi136.stdout: "namespace": "" 2024-09-15T22:02:17.916 INFO:tasks.workunit.client.0.smithi136.stdout: }, 2024-09-15T22:02:17.916 INFO:tasks.workunit.client.0.smithi136.stdout: "version": "16'38", 2024-09-15T22:02:17.916 INFO:tasks.workunit.client.0.smithi136.stdout: "prior_version": "15'13", 2024-09-15T22:02:17.916 INFO:tasks.workunit.client.0.smithi136.stdout: "last_reqid": "client.4324.0:1", 2024-09-15T22:02:17.916 INFO:tasks.workunit.client.0.smithi136.stdout: "user_version": 38, 2024-09-15T22:02:17.916 INFO:tasks.workunit.client.0.smithi136.stdout: "size": 256, 2024-09-15T22:02:17.916 INFO:tasks.workunit.client.0.smithi136.stdout: "mtime": "2024-09-15T22:01:07.365553+0000", 2024-09-15T22:02:17.916 INFO:tasks.workunit.client.0.smithi136.stdout: "local_mtime": "2024-09-15T22:01:07.366194+0000", 2024-09-15T22:02:17.916 INFO:tasks.workunit.client.0.smithi136.stdout: "lost": 0, 2024-09-15T22:02:17.916 INFO:tasks.workunit.client.0.smithi136.stdout: "flags": [ 2024-09-15T22:02:17.916 INFO:tasks.workunit.client.0.smithi136.stdout: "dirty", 2024-09-15T22:02:17.916 INFO:tasks.workunit.client.0.smithi136.stdout: "data_digest" 2024-09-15T22:02:17.916 INFO:tasks.workunit.client.0.smithi136.stdout: ], 2024-09-15T22:02:17.917 INFO:tasks.workunit.client.0.smithi136.stdout: "truncate_seq": 0, 2024-09-15T22:02:17.917 INFO:tasks.workunit.client.0.smithi136.stdout: "truncate_size": 0, 2024-09-15T22:02:17.917 INFO:tasks.workunit.client.0.smithi136.stdout: "data_digest": "0x29e45974", 2024-09-15T22:02:17.917 INFO:tasks.workunit.client.0.smithi136.stdout: "omap_digest": "0xffffffff", 2024-09-15T22:02:17.917 INFO:tasks.workunit.client.0.smithi136.stdout: "expected_object_size": 0, 2024-09-15T22:02:17.917 INFO:tasks.workunit.client.0.smithi136.stdout: "expected_write_size": 0, 2024-09-15T22:02:17.917 INFO:tasks.workunit.client.0.smithi136.stdout: "alloc_hint_flags": 0, 2024-09-15T22:02:17.917 INFO:tasks.workunit.client.0.smithi136.stdout: "manifest": { 2024-09-15T22:02:17.917 INFO:tasks.workunit.client.0.smithi136.stdout: "type": 0 2024-09-15T22:02:17.917 INFO:tasks.workunit.client.0.smithi136.stdout: }, 2024-09-15T22:02:17.917 INFO:tasks.workunit.client.0.smithi136.stdout: "watchers": {} 2024-09-15T22:02:17.917 INFO:tasks.workunit.client.0.smithi136.stdout: }, 2024-09-15T22:02:17.917 INFO:tasks.workunit.client.0.smithi136.stdout: "shards": [ 2024-09-15T22:02:17.917 INFO:tasks.workunit.client.0.smithi136.stdout: { 2024-09-15T22:02:17.918 INFO:tasks.workunit.client.0.smithi136.stdout: "osd": 0, 2024-09-15T22:02:17.918 INFO:tasks.workunit.client.0.smithi136.stdout: "primary": false, 2024-09-15T22:02:17.918 INFO:tasks.workunit.client.0.smithi136.stdout: "errors": [], 2024-09-15T22:02:17.918 INFO:tasks.workunit.client.0.smithi136.stdout: "size": 256, 2024-09-15T22:02:17.918 INFO:tasks.workunit.client.0.smithi136.stdout: "snapset": { 2024-09-15T22:02:17.918 INFO:tasks.workunit.client.0.smithi136.stdout: "seq": 1, 2024-09-15T22:02:17.918 INFO:tasks.workunit.client.0.smithi136.stdout: "clones": [ 2024-09-15T22:02:17.918 INFO:tasks.workunit.client.0.smithi136.stdout: { 2024-09-15T22:02:17.918 INFO:tasks.workunit.client.0.smithi136.stdout: "snap": 1, 2024-09-15T22:02:17.918 INFO:tasks.workunit.client.0.smithi136.stdout: "size": 1032, 2024-09-15T22:02:17.918 INFO:tasks.workunit.client.0.smithi136.stdout: "overlap": "[]", 2024-09-15T22:02:17.918 INFO:tasks.workunit.client.0.smithi136.stdout: "snaps": [ 2024-09-15T22:02:17.918 INFO:tasks.workunit.client.0.smithi136.stdout: 1 2024-09-15T22:02:17.918 INFO:tasks.workunit.client.0.smithi136.stdout: ] 2024-09-15T22:02:17.919 INFO:tasks.workunit.client.0.smithi136.stdout: } 2024-09-15T22:02:17.919 INFO:tasks.workunit.client.0.smithi136.stdout: ] 2024-09-15T22:02:17.919 INFO:tasks.workunit.client.0.smithi136.stdout: } 2024-09-15T22:02:17.919 INFO:tasks.workunit.client.0.smithi136.stdout: }, 2024-09-15T22:02:17.919 INFO:tasks.workunit.client.0.smithi136.stdout: { 2024-09-15T22:02:17.919 INFO:tasks.workunit.client.0.smithi136.stdout: "osd": 1, 2024-09-15T22:02:17.919 INFO:tasks.workunit.client.0.smithi136.stdout: "primary": true, 2024-09-15T22:02:17.919 INFO:tasks.workunit.client.0.smithi136.stdout: "errors": [], 2024-09-15T22:02:17.919 INFO:tasks.workunit.client.0.smithi136.stdout: "size": 256, 2024-09-15T22:02:17.919 INFO:tasks.workunit.client.0.smithi136.stdout: "snapset": { 2024-09-15T22:02:17.919 INFO:tasks.workunit.client.0.smithi136.stdout: "seq": 1, 2024-09-15T22:02:17.919 INFO:tasks.workunit.client.0.smithi136.stdout: "clones": [ 2024-09-15T22:02:17.919 INFO:tasks.workunit.client.0.smithi136.stdout: { 2024-09-15T22:02:17.919 INFO:tasks.workunit.client.0.smithi136.stdout: "snap": 1, 2024-09-15T22:02:17.920 INFO:tasks.workunit.client.0.smithi136.stdout: "size": 1032, 2024-09-15T22:02:17.920 INFO:tasks.workunit.client.0.smithi136.stdout: "overlap": "[]" 2024-09-15T22:02:17.920 INFO:tasks.workunit.client.0.smithi136.stdout: } 2024-09-15T22:02:17.920 INFO:tasks.workunit.client.0.smithi136.stdout: ] 2024-09-15T22:02:17.920 INFO:tasks.workunit.client.0.smithi136.stdout: } 2024-09-15T22:02:17.920 INFO:tasks.workunit.client.0.smithi136.stdout: } 2024-09-15T22:02:17.920 INFO:tasks.workunit.client.0.smithi136.stdout: ] 2024-09-15T22:02:17.920 INFO:tasks.workunit.client.0.smithi136.stdout: }, 2024-09-15T22:02:17.920 INFO:tasks.workunit.client.0.smithi136.stdout: { 2024-09-15T22:02:17.920 INFO:tasks.workunit.client.0.smithi136.stdout: "object": { 2024-09-15T22:02:17.920 INFO:tasks.workunit.client.0.smithi136.stdout: "name": "obj14", 2024-09-15T22:02:17.920 INFO:tasks.workunit.client.0.smithi136.stdout: "nspace": "", 2024-09-15T22:02:17.920 INFO:tasks.workunit.client.0.smithi136.stdout: "locator": "", 2024-09-15T22:02:17.920 INFO:tasks.workunit.client.0.smithi136.stdout: "snap": "head", 2024-09-15T22:02:17.921 INFO:tasks.workunit.client.0.smithi136.stdout: "version": 40 2024-09-15T22:02:17.921 INFO:tasks.workunit.client.0.smithi136.stdout: }, 2024-09-15T22:02:17.921 INFO:tasks.workunit.client.0.smithi136.stdout: "errors": [ 2024-09-15T22:02:17.921 INFO:tasks.workunit.client.0.smithi136.stdout: "snapset_inconsistency" 2024-09-15T22:02:17.921 INFO:tasks.workunit.client.0.smithi136.stdout: ], 2024-09-15T22:02:17.921 INFO:tasks.workunit.client.0.smithi136.stdout: "union_shard_errors": [], 2024-09-15T22:02:17.921 INFO:tasks.workunit.client.0.smithi136.stdout: "selected_object_info": { 2024-09-15T22:02:17.921 INFO:tasks.workunit.client.0.smithi136.stdout: "oid": { 2024-09-15T22:02:17.921 INFO:tasks.workunit.client.0.smithi136.stdout: "oid": "obj14", 2024-09-15T22:02:17.921 INFO:tasks.workunit.client.0.smithi136.stdout: "key": "", 2024-09-15T22:02:17.921 INFO:tasks.workunit.client.0.smithi136.stdout: "snapid": -2, 2024-09-15T22:02:17.921 INFO:tasks.workunit.client.0.smithi136.stdout: "hash": 2484217095, 2024-09-15T22:02:17.921 INFO:tasks.workunit.client.0.smithi136.stdout: "max": 0, 2024-09-15T22:02:17.921 INFO:tasks.workunit.client.0.smithi136.stdout: "pool": 1, 2024-09-15T22:02:17.922 INFO:tasks.workunit.client.0.smithi136.stdout: "namespace": "" 2024-09-15T22:02:17.922 INFO:tasks.workunit.client.0.smithi136.stdout: }, 2024-09-15T22:02:17.922 INFO:tasks.workunit.client.0.smithi136.stdout: "version": "16'40", 2024-09-15T22:02:17.922 INFO:tasks.workunit.client.0.smithi136.stdout: "prior_version": "15'14", 2024-09-15T22:02:17.922 INFO:tasks.workunit.client.0.smithi136.stdout: "last_reqid": "client.4327.0:1", 2024-09-15T22:02:17.922 INFO:tasks.workunit.client.0.smithi136.stdout: "user_version": 40, 2024-09-15T22:02:17.922 INFO:tasks.workunit.client.0.smithi136.stdout: "size": 256, 2024-09-15T22:02:17.922 INFO:tasks.workunit.client.0.smithi136.stdout: "mtime": "2024-09-15T22:01:07.499508+0000", 2024-09-15T22:02:17.922 INFO:tasks.workunit.client.0.smithi136.stdout: "local_mtime": "2024-09-15T22:01:07.500095+0000", 2024-09-15T22:02:17.922 INFO:tasks.workunit.client.0.smithi136.stdout: "lost": 0, 2024-09-15T22:02:17.922 INFO:tasks.workunit.client.0.smithi136.stdout: "flags": [ 2024-09-15T22:02:17.922 INFO:tasks.workunit.client.0.smithi136.stdout: "dirty", 2024-09-15T22:02:17.922 INFO:tasks.workunit.client.0.smithi136.stdout: "data_digest" 2024-09-15T22:02:17.922 INFO:tasks.workunit.client.0.smithi136.stdout: ], 2024-09-15T22:02:17.923 INFO:tasks.workunit.client.0.smithi136.stdout: "truncate_seq": 0, 2024-09-15T22:02:17.923 INFO:tasks.workunit.client.0.smithi136.stdout: "truncate_size": 0, 2024-09-15T22:02:17.923 INFO:tasks.workunit.client.0.smithi136.stdout: "data_digest": "0x29e45974", 2024-09-15T22:02:17.923 INFO:tasks.workunit.client.0.smithi136.stdout: "omap_digest": "0xffffffff", 2024-09-15T22:02:17.923 INFO:tasks.workunit.client.0.smithi136.stdout: "expected_object_size": 0, 2024-09-15T22:02:17.923 INFO:tasks.workunit.client.0.smithi136.stdout: "expected_write_size": 0, 2024-09-15T22:02:17.923 INFO:tasks.workunit.client.0.smithi136.stdout: "alloc_hint_flags": 0, 2024-09-15T22:02:17.923 INFO:tasks.workunit.client.0.smithi136.stdout: "manifest": { 2024-09-15T22:02:17.923 INFO:tasks.workunit.client.0.smithi136.stdout: "type": 0 2024-09-15T22:02:17.923 INFO:tasks.workunit.client.0.smithi136.stdout: }, 2024-09-15T22:02:17.923 INFO:tasks.workunit.client.0.smithi136.stdout: "watchers": {} 2024-09-15T22:02:17.923 INFO:tasks.workunit.client.0.smithi136.stdout: }, 2024-09-15T22:02:17.923 INFO:tasks.workunit.client.0.smithi136.stdout: "shards": [ 2024-09-15T22:02:17.923 INFO:tasks.workunit.client.0.smithi136.stdout: { 2024-09-15T22:02:17.923 INFO:tasks.workunit.client.0.smithi136.stdout: "osd": 0, 2024-09-15T22:02:17.924 INFO:tasks.workunit.client.0.smithi136.stdout: "primary": false, 2024-09-15T22:02:17.924 INFO:tasks.workunit.client.0.smithi136.stdout: "errors": [], 2024-09-15T22:02:17.924 INFO:tasks.workunit.client.0.smithi136.stdout: "size": 256, 2024-09-15T22:02:17.924 INFO:tasks.workunit.client.0.smithi136.stdout: "snapset": { 2024-09-15T22:02:17.924 INFO:tasks.workunit.client.0.smithi136.stdout: "seq": 1, 2024-09-15T22:02:17.924 INFO:tasks.workunit.client.0.smithi136.stdout: "clones": [ 2024-09-15T22:02:17.924 INFO:tasks.workunit.client.0.smithi136.stdout: { 2024-09-15T22:02:17.924 INFO:tasks.workunit.client.0.smithi136.stdout: "snap": 1, 2024-09-15T22:02:17.924 INFO:tasks.workunit.client.0.smithi136.stdout: "size": 1032, 2024-09-15T22:02:17.924 INFO:tasks.workunit.client.0.smithi136.stdout: "overlap": "[]", 2024-09-15T22:02:17.924 INFO:tasks.workunit.client.0.smithi136.stdout: "snaps": [ 2024-09-15T22:02:17.924 INFO:tasks.workunit.client.0.smithi136.stdout: 1 2024-09-15T22:02:17.924 INFO:tasks.workunit.client.0.smithi136.stdout: ] 2024-09-15T22:02:17.924 INFO:tasks.workunit.client.0.smithi136.stdout: } 2024-09-15T22:02:17.925 INFO:tasks.workunit.client.0.smithi136.stdout: ] 2024-09-15T22:02:17.925 INFO:tasks.workunit.client.0.smithi136.stdout: } 2024-09-15T22:02:17.925 INFO:tasks.workunit.client.0.smithi136.stdout: }, 2024-09-15T22:02:17.925 INFO:tasks.workunit.client.0.smithi136.stdout: { 2024-09-15T22:02:17.925 INFO:tasks.workunit.client.0.smithi136.stdout: "osd": 1, 2024-09-15T22:02:17.925 INFO:tasks.workunit.client.0.smithi136.stdout: "primary": true, 2024-09-15T22:02:17.925 INFO:tasks.workunit.client.0.smithi136.stdout: "errors": [], 2024-09-15T22:02:17.925 INFO:tasks.workunit.client.0.smithi136.stdout: "size": 256, 2024-09-15T22:02:17.925 INFO:tasks.workunit.client.0.smithi136.stdout: "snapset": { 2024-09-15T22:02:17.925 INFO:tasks.workunit.client.0.smithi136.stdout: "seq": 1, 2024-09-15T22:02:17.925 INFO:tasks.workunit.client.0.smithi136.stdout: "clones": [ 2024-09-15T22:02:17.925 INFO:tasks.workunit.client.0.smithi136.stdout: { 2024-09-15T22:02:17.925 INFO:tasks.workunit.client.0.smithi136.stdout: "snap": 1, 2024-09-15T22:02:17.925 INFO:tasks.workunit.client.0.smithi136.stdout: "size": 1033, 2024-09-15T22:02:17.926 INFO:tasks.workunit.client.0.smithi136.stdout: "overlap": "[]", 2024-09-15T22:02:17.926 INFO:tasks.workunit.client.0.smithi136.stdout: "snaps": [ 2024-09-15T22:02:17.926 INFO:tasks.workunit.client.0.smithi136.stdout: 1 2024-09-15T22:02:17.926 INFO:tasks.workunit.client.0.smithi136.stdout: ] 2024-09-15T22:02:17.926 INFO:tasks.workunit.client.0.smithi136.stdout: } 2024-09-15T22:02:17.926 INFO:tasks.workunit.client.0.smithi136.stdout: ] 2024-09-15T22:02:17.926 INFO:tasks.workunit.client.0.smithi136.stdout: } 2024-09-15T22:02:17.926 INFO:tasks.workunit.client.0.smithi136.stdout: } 2024-09-15T22:02:17.926 INFO:tasks.workunit.client.0.smithi136.stdout: ] 2024-09-15T22:02:17.926 INFO:tasks.workunit.client.0.smithi136.stdout: }, 2024-09-15T22:02:17.926 INFO:tasks.workunit.client.0.smithi136.stdout: { 2024-09-15T22:02:17.926 INFO:tasks.workunit.client.0.smithi136.stdout: "object": { 2024-09-15T22:02:17.926 INFO:tasks.workunit.client.0.smithi136.stdout: "name": "obj15", 2024-09-15T22:02:17.926 INFO:tasks.workunit.client.0.smithi136.stdout: "nspace": "", 2024-09-15T22:02:17.927 INFO:tasks.workunit.client.0.smithi136.stdout: "locator": "", 2024-09-15T22:02:17.927 INFO:tasks.workunit.client.0.smithi136.stdout: "snap": "head", 2024-09-15T22:02:17.927 INFO:tasks.workunit.client.0.smithi136.stdout: "version": 15 2024-09-15T22:02:17.927 INFO:tasks.workunit.client.0.smithi136.stdout: }, 2024-09-15T22:02:17.927 INFO:tasks.workunit.client.0.smithi136.stdout: "errors": [], 2024-09-15T22:02:17.927 INFO:tasks.workunit.client.0.smithi136.stdout: "union_shard_errors": [ 2024-09-15T22:02:17.927 INFO:tasks.workunit.client.0.smithi136.stdout: "snapset_corrupted" 2024-09-15T22:02:17.927 INFO:tasks.workunit.client.0.smithi136.stdout: ], 2024-09-15T22:02:17.927 INFO:tasks.workunit.client.0.smithi136.stdout: "selected_object_info": { 2024-09-15T22:02:17.927 INFO:tasks.workunit.client.0.smithi136.stdout: "oid": { 2024-09-15T22:02:17.927 INFO:tasks.workunit.client.0.smithi136.stdout: "oid": "obj15", 2024-09-15T22:02:17.927 INFO:tasks.workunit.client.0.smithi136.stdout: "key": "", 2024-09-15T22:02:17.927 INFO:tasks.workunit.client.0.smithi136.stdout: "snapid": -2, 2024-09-15T22:02:17.927 INFO:tasks.workunit.client.0.smithi136.stdout: "hash": 612772309, 2024-09-15T22:02:17.928 INFO:tasks.workunit.client.0.smithi136.stdout: "max": 0, 2024-09-15T22:02:17.928 INFO:tasks.workunit.client.0.smithi136.stdout: "pool": 1, 2024-09-15T22:02:17.928 INFO:tasks.workunit.client.0.smithi136.stdout: "namespace": "" 2024-09-15T22:02:17.928 INFO:tasks.workunit.client.0.smithi136.stdout: }, 2024-09-15T22:02:17.928 INFO:tasks.workunit.client.0.smithi136.stdout: "version": "15'15", 2024-09-15T22:02:17.928 INFO:tasks.workunit.client.0.smithi136.stdout: "prior_version": "0'0", 2024-09-15T22:02:17.928 INFO:tasks.workunit.client.0.smithi136.stdout: "last_reqid": "client.4279.0:1", 2024-09-15T22:02:17.928 INFO:tasks.workunit.client.0.smithi136.stdout: "user_version": 15, 2024-09-15T22:02:17.928 INFO:tasks.workunit.client.0.smithi136.stdout: "size": 1032, 2024-09-15T22:02:17.928 INFO:tasks.workunit.client.0.smithi136.stdout: "mtime": "2024-09-15T22:01:04.389170+0000", 2024-09-15T22:02:17.928 INFO:tasks.workunit.client.0.smithi136.stdout: "local_mtime": "2024-09-15T22:01:04.389736+0000", 2024-09-15T22:02:17.928 INFO:tasks.workunit.client.0.smithi136.stdout: "lost": 0, 2024-09-15T22:02:17.928 INFO:tasks.workunit.client.0.smithi136.stdout: "flags": [ 2024-09-15T22:02:17.928 INFO:tasks.workunit.client.0.smithi136.stdout: "dirty", 2024-09-15T22:02:17.928 INFO:tasks.workunit.client.0.smithi136.stdout: "data_digest" 2024-09-15T22:02:17.929 INFO:tasks.workunit.client.0.smithi136.stdout: ], 2024-09-15T22:02:17.929 INFO:tasks.workunit.client.0.smithi136.stdout: "truncate_seq": 0, 2024-09-15T22:02:17.929 INFO:tasks.workunit.client.0.smithi136.stdout: "truncate_size": 0, 2024-09-15T22:02:17.929 INFO:tasks.workunit.client.0.smithi136.stdout: "data_digest": "0x557795f2", 2024-09-15T22:02:17.929 INFO:tasks.workunit.client.0.smithi136.stdout: "omap_digest": "0xffffffff", 2024-09-15T22:02:17.929 INFO:tasks.workunit.client.0.smithi136.stdout: "expected_object_size": 0, 2024-09-15T22:02:17.929 INFO:tasks.workunit.client.0.smithi136.stdout: "expected_write_size": 0, 2024-09-15T22:02:17.929 INFO:tasks.workunit.client.0.smithi136.stdout: "alloc_hint_flags": 0, 2024-09-15T22:02:17.929 INFO:tasks.workunit.client.0.smithi136.stdout: "manifest": { 2024-09-15T22:02:17.929 INFO:tasks.workunit.client.0.smithi136.stdout: "type": 0 2024-09-15T22:02:17.929 INFO:tasks.workunit.client.0.smithi136.stdout: }, 2024-09-15T22:02:17.929 INFO:tasks.workunit.client.0.smithi136.stdout: "watchers": {} 2024-09-15T22:02:17.929 INFO:tasks.workunit.client.0.smithi136.stdout: }, 2024-09-15T22:02:17.929 INFO:tasks.workunit.client.0.smithi136.stdout: "shards": [ 2024-09-15T22:02:17.930 INFO:tasks.workunit.client.0.smithi136.stdout: { 2024-09-15T22:02:17.930 INFO:tasks.workunit.client.0.smithi136.stdout: "osd": 0, 2024-09-15T22:02:17.930 INFO:tasks.workunit.client.0.smithi136.stdout: "primary": false, 2024-09-15T22:02:17.930 INFO:tasks.workunit.client.0.smithi136.stdout: "errors": [], 2024-09-15T22:02:17.930 INFO:tasks.workunit.client.0.smithi136.stdout: "size": 1032, 2024-09-15T22:02:17.930 INFO:tasks.workunit.client.0.smithi136.stdout: "snapset": { 2024-09-15T22:02:17.930 INFO:tasks.workunit.client.0.smithi136.stdout: "seq": 0, 2024-09-15T22:02:17.930 INFO:tasks.workunit.client.0.smithi136.stdout: "clones": [] 2024-09-15T22:02:17.930 INFO:tasks.workunit.client.0.smithi136.stdout: } 2024-09-15T22:02:17.930 INFO:tasks.workunit.client.0.smithi136.stdout: }, 2024-09-15T22:02:17.930 INFO:tasks.workunit.client.0.smithi136.stdout: { 2024-09-15T22:02:17.930 INFO:tasks.workunit.client.0.smithi136.stdout: "osd": 1, 2024-09-15T22:02:17.930 INFO:tasks.workunit.client.0.smithi136.stdout: "primary": true, 2024-09-15T22:02:17.930 INFO:tasks.workunit.client.0.smithi136.stdout: "errors": [ 2024-09-15T22:02:17.931 INFO:tasks.workunit.client.0.smithi136.stdout: "snapset_corrupted" 2024-09-15T22:02:17.931 INFO:tasks.workunit.client.0.smithi136.stdout: ], 2024-09-15T22:02:17.931 INFO:tasks.workunit.client.0.smithi136.stdout: "size": 1032, 2024-09-15T22:02:17.931 INFO:tasks.workunit.client.0.smithi136.stdout: "snapset": "Z2FyYmFnZQo=" 2024-09-15T22:02:17.931 INFO:tasks.workunit.client.0.smithi136.stdout: } 2024-09-15T22:02:17.931 INFO:tasks.workunit.client.0.smithi136.stdout: ] 2024-09-15T22:02:17.931 INFO:tasks.workunit.client.0.smithi136.stdout: }, 2024-09-15T22:02:17.931 INFO:tasks.workunit.client.0.smithi136.stdout: { 2024-09-15T22:02:17.931 INFO:tasks.workunit.client.0.smithi136.stdout: "object": { 2024-09-15T22:02:17.931 INFO:tasks.workunit.client.0.smithi136.stdout: "name": "obj2", 2024-09-15T22:02:17.931 INFO:tasks.workunit.client.0.smithi136.stdout: "nspace": "", 2024-09-15T22:02:17.931 INFO:tasks.workunit.client.0.smithi136.stdout: "locator": "", 2024-09-15T22:02:17.931 INFO:tasks.workunit.client.0.smithi136.stdout: "snap": "head", 2024-09-15T22:02:17.931 INFO:tasks.workunit.client.0.smithi136.stdout: "version": 56 2024-09-15T22:02:17.931 INFO:tasks.workunit.client.0.smithi136.stdout: }, 2024-09-15T22:02:17.932 INFO:tasks.workunit.client.0.smithi136.stdout: "errors": [], 2024-09-15T22:02:17.932 INFO:tasks.workunit.client.0.smithi136.stdout: "union_shard_errors": [ 2024-09-15T22:02:17.932 INFO:tasks.workunit.client.0.smithi136.stdout: "snapset_missing" 2024-09-15T22:02:17.932 INFO:tasks.workunit.client.0.smithi136.stdout: ], 2024-09-15T22:02:17.932 INFO:tasks.workunit.client.0.smithi136.stdout: "selected_object_info": { 2024-09-15T22:02:17.932 INFO:tasks.workunit.client.0.smithi136.stdout: "oid": { 2024-09-15T22:02:17.932 INFO:tasks.workunit.client.0.smithi136.stdout: "oid": "obj2", 2024-09-15T22:02:17.932 INFO:tasks.workunit.client.0.smithi136.stdout: "key": "", 2024-09-15T22:02:17.932 INFO:tasks.workunit.client.0.smithi136.stdout: "snapid": -2, 2024-09-15T22:02:17.932 INFO:tasks.workunit.client.0.smithi136.stdout: "hash": 1058988552, 2024-09-15T22:02:17.932 INFO:tasks.workunit.client.0.smithi136.stdout: "max": 0, 2024-09-15T22:02:17.932 INFO:tasks.workunit.client.0.smithi136.stdout: "pool": 1, 2024-09-15T22:02:17.932 INFO:tasks.workunit.client.0.smithi136.stdout: "namespace": "" 2024-09-15T22:02:17.932 INFO:tasks.workunit.client.0.smithi136.stdout: }, 2024-09-15T22:02:17.933 INFO:tasks.workunit.client.0.smithi136.stdout: "version": "22'56", 2024-09-15T22:02:17.933 INFO:tasks.workunit.client.0.smithi136.stdout: "prior_version": "19'48", 2024-09-15T22:02:17.933 INFO:tasks.workunit.client.0.smithi136.stdout: "last_reqid": "client.4369.0:1", 2024-09-15T22:02:17.933 INFO:tasks.workunit.client.0.smithi136.stdout: "user_version": 56, 2024-09-15T22:02:17.933 INFO:tasks.workunit.client.0.smithi136.stdout: "size": 0, 2024-09-15T22:02:17.933 INFO:tasks.workunit.client.0.smithi136.stdout: "mtime": "2024-09-15T22:01:10.453262+0000", 2024-09-15T22:02:17.933 INFO:tasks.workunit.client.0.smithi136.stdout: "local_mtime": "2024-09-15T22:01:10.453819+0000", 2024-09-15T22:02:17.933 INFO:tasks.workunit.client.0.smithi136.stdout: "lost": 0, 2024-09-15T22:02:17.933 INFO:tasks.workunit.client.0.smithi136.stdout: "flags": [ 2024-09-15T22:02:17.933 INFO:tasks.workunit.client.0.smithi136.stdout: "whiteout", 2024-09-15T22:02:17.933 INFO:tasks.workunit.client.0.smithi136.stdout: "dirty" 2024-09-15T22:02:17.933 INFO:tasks.workunit.client.0.smithi136.stdout: ], 2024-09-15T22:02:17.933 INFO:tasks.workunit.client.0.smithi136.stdout: "truncate_seq": 0, 2024-09-15T22:02:17.934 INFO:tasks.workunit.client.0.smithi136.stdout: "truncate_size": 0, 2024-09-15T22:02:17.934 INFO:tasks.workunit.client.0.smithi136.stdout: "data_digest": "0xffffffff", 2024-09-15T22:02:17.934 INFO:tasks.workunit.client.0.smithi136.stdout: "omap_digest": "0xffffffff", 2024-09-15T22:02:17.934 INFO:tasks.workunit.client.0.smithi136.stdout: "expected_object_size": 0, 2024-09-15T22:02:17.934 INFO:tasks.workunit.client.0.smithi136.stdout: "expected_write_size": 0, 2024-09-15T22:02:17.934 INFO:tasks.workunit.client.0.smithi136.stdout: "alloc_hint_flags": 0, 2024-09-15T22:02:17.934 INFO:tasks.workunit.client.0.smithi136.stdout: "manifest": { 2024-09-15T22:02:17.934 INFO:tasks.workunit.client.0.smithi136.stdout: "type": 0 2024-09-15T22:02:17.934 INFO:tasks.workunit.client.0.smithi136.stdout: }, 2024-09-15T22:02:17.934 INFO:tasks.workunit.client.0.smithi136.stdout: "watchers": {} 2024-09-15T22:02:17.934 INFO:tasks.workunit.client.0.smithi136.stdout: }, 2024-09-15T22:02:17.934 INFO:tasks.workunit.client.0.smithi136.stdout: "shards": [ 2024-09-15T22:02:17.934 INFO:tasks.workunit.client.0.smithi136.stdout: { 2024-09-15T22:02:17.934 INFO:tasks.workunit.client.0.smithi136.stdout: "osd": 0, 2024-09-15T22:02:17.935 INFO:tasks.workunit.client.0.smithi136.stdout: "primary": false, 2024-09-15T22:02:17.935 INFO:tasks.workunit.client.0.smithi136.stdout: "errors": [], 2024-09-15T22:02:17.935 INFO:tasks.workunit.client.0.smithi136.stdout: "size": 0, 2024-09-15T22:02:17.935 INFO:tasks.workunit.client.0.smithi136.stdout: "snapset": { 2024-09-15T22:02:17.935 INFO:tasks.workunit.client.0.smithi136.stdout: "seq": 7, 2024-09-15T22:02:17.935 INFO:tasks.workunit.client.0.smithi136.stdout: "clones": [ 2024-09-15T22:02:17.935 INFO:tasks.workunit.client.0.smithi136.stdout: { 2024-09-15T22:02:17.935 INFO:tasks.workunit.client.0.smithi136.stdout: "snap": 4, 2024-09-15T22:02:17.935 INFO:tasks.workunit.client.0.smithi136.stdout: "size": 1032, 2024-09-15T22:02:17.935 INFO:tasks.workunit.client.0.smithi136.stdout: "overlap": "[]", 2024-09-15T22:02:17.935 INFO:tasks.workunit.client.0.smithi136.stdout: "snaps": [ 2024-09-15T22:02:17.935 INFO:tasks.workunit.client.0.smithi136.stdout: 4, 2024-09-15T22:02:17.935 INFO:tasks.workunit.client.0.smithi136.stdout: 3, 2024-09-15T22:02:17.935 INFO:tasks.workunit.client.0.smithi136.stdout: 2, 2024-09-15T22:02:17.936 INFO:tasks.workunit.client.0.smithi136.stdout: 1 2024-09-15T22:02:17.936 INFO:tasks.workunit.client.0.smithi136.stdout: ] 2024-09-15T22:02:17.936 INFO:tasks.workunit.client.0.smithi136.stdout: }, 2024-09-15T22:02:17.936 INFO:tasks.workunit.client.0.smithi136.stdout: { 2024-09-15T22:02:17.936 INFO:tasks.workunit.client.0.smithi136.stdout: "snap": 7, 2024-09-15T22:02:17.936 INFO:tasks.workunit.client.0.smithi136.stdout: "size": 1024, 2024-09-15T22:02:17.936 INFO:tasks.workunit.client.0.smithi136.stdout: "overlap": "[]", 2024-09-15T22:02:17.936 INFO:tasks.workunit.client.0.smithi136.stdout: "snaps": [ 2024-09-15T22:02:17.936 INFO:tasks.workunit.client.0.smithi136.stdout: 7, 2024-09-15T22:02:17.936 INFO:tasks.workunit.client.0.smithi136.stdout: 6, 2024-09-15T22:02:17.936 INFO:tasks.workunit.client.0.smithi136.stdout: 5 2024-09-15T22:02:17.936 INFO:tasks.workunit.client.0.smithi136.stdout: ] 2024-09-15T22:02:17.936 INFO:tasks.workunit.client.0.smithi136.stdout: } 2024-09-15T22:02:17.936 INFO:tasks.workunit.client.0.smithi136.stdout: ] 2024-09-15T22:02:17.937 INFO:tasks.workunit.client.0.smithi136.stdout: } 2024-09-15T22:02:17.937 INFO:tasks.workunit.client.0.smithi136.stdout: }, 2024-09-15T22:02:17.937 INFO:tasks.workunit.client.0.smithi136.stdout: { 2024-09-15T22:02:17.937 INFO:tasks.workunit.client.0.smithi136.stdout: "osd": 1, 2024-09-15T22:02:17.937 INFO:tasks.workunit.client.0.smithi136.stdout: "primary": true, 2024-09-15T22:02:17.937 INFO:tasks.workunit.client.0.smithi136.stdout: "errors": [ 2024-09-15T22:02:17.937 INFO:tasks.workunit.client.0.smithi136.stdout: "snapset_missing" 2024-09-15T22:02:17.937 INFO:tasks.workunit.client.0.smithi136.stdout: ], 2024-09-15T22:02:17.937 INFO:tasks.workunit.client.0.smithi136.stdout: "size": 0 2024-09-15T22:02:17.937 INFO:tasks.workunit.client.0.smithi136.stdout: } 2024-09-15T22:02:17.937 INFO:tasks.workunit.client.0.smithi136.stdout: ] 2024-09-15T22:02:17.937 INFO:tasks.workunit.client.0.smithi136.stdout: }, 2024-09-15T22:02:17.937 INFO:tasks.workunit.client.0.smithi136.stdout: { 2024-09-15T22:02:17.937 INFO:tasks.workunit.client.0.smithi136.stdout: "object": { 2024-09-15T22:02:17.938 INFO:tasks.workunit.client.0.smithi136.stdout: "name": "obj3", 2024-09-15T22:02:17.938 INFO:tasks.workunit.client.0.smithi136.stdout: "nspace": "", 2024-09-15T22:02:17.938 INFO:tasks.workunit.client.0.smithi136.stdout: "locator": "", 2024-09-15T22:02:17.938 INFO:tasks.workunit.client.0.smithi136.stdout: "snap": "head", 2024-09-15T22:02:17.938 INFO:tasks.workunit.client.0.smithi136.stdout: "version": 44 2024-09-15T22:02:17.938 INFO:tasks.workunit.client.0.smithi136.stdout: }, 2024-09-15T22:02:17.938 INFO:tasks.workunit.client.0.smithi136.stdout: "errors": [ 2024-09-15T22:02:17.938 INFO:tasks.workunit.client.0.smithi136.stdout: "size_mismatch" 2024-09-15T22:02:17.938 INFO:tasks.workunit.client.0.smithi136.stdout: ], 2024-09-15T22:02:17.938 INFO:tasks.workunit.client.0.smithi136.stdout: "union_shard_errors": [ 2024-09-15T22:02:17.938 INFO:tasks.workunit.client.0.smithi136.stdout: "size_mismatch_info", 2024-09-15T22:02:17.938 INFO:tasks.workunit.client.0.smithi136.stdout: "obj_size_info_mismatch" 2024-09-15T22:02:17.938 INFO:tasks.workunit.client.0.smithi136.stdout: ], 2024-09-15T22:02:17.938 INFO:tasks.workunit.client.0.smithi136.stdout: "selected_object_info": { 2024-09-15T22:02:17.939 INFO:tasks.workunit.client.0.smithi136.stdout: "oid": { 2024-09-15T22:02:17.939 INFO:tasks.workunit.client.0.smithi136.stdout: "oid": "obj3", 2024-09-15T22:02:17.939 INFO:tasks.workunit.client.0.smithi136.stdout: "key": "", 2024-09-15T22:02:17.939 INFO:tasks.workunit.client.0.smithi136.stdout: "snapid": -2, 2024-09-15T22:02:17.939 INFO:tasks.workunit.client.0.smithi136.stdout: "hash": 1643547569, 2024-09-15T22:02:17.939 INFO:tasks.workunit.client.0.smithi136.stdout: "max": 0, 2024-09-15T22:02:17.939 INFO:tasks.workunit.client.0.smithi136.stdout: "pool": 1, 2024-09-15T22:02:17.939 INFO:tasks.workunit.client.0.smithi136.stdout: "namespace": "" 2024-09-15T22:02:17.939 INFO:tasks.workunit.client.0.smithi136.stdout: }, 2024-09-15T22:02:17.939 INFO:tasks.workunit.client.0.smithi136.stdout: "version": "18'44", 2024-09-15T22:02:17.939 INFO:tasks.workunit.client.0.smithi136.stdout: "prior_version": "16'22", 2024-09-15T22:02:17.939 INFO:tasks.workunit.client.0.smithi136.stdout: "last_reqid": "client.4339.0:1", 2024-09-15T22:02:17.939 INFO:tasks.workunit.client.0.smithi136.stdout: "user_version": 44, 2024-09-15T22:02:17.939 INFO:tasks.workunit.client.0.smithi136.stdout: "size": 768, 2024-09-15T22:02:17.939 INFO:tasks.workunit.client.0.smithi136.stdout: "mtime": "2024-09-15T22:01:08.280317+0000", 2024-09-15T22:02:17.940 INFO:tasks.workunit.client.0.smithi136.stdout: "local_mtime": "2024-09-15T22:01:08.290990+0000", 2024-09-15T22:02:17.940 INFO:tasks.workunit.client.0.smithi136.stdout: "lost": 0, 2024-09-15T22:02:17.940 INFO:tasks.workunit.client.0.smithi136.stdout: "flags": [ 2024-09-15T22:02:17.940 INFO:tasks.workunit.client.0.smithi136.stdout: "dirty", 2024-09-15T22:02:17.940 INFO:tasks.workunit.client.0.smithi136.stdout: "data_digest" 2024-09-15T22:02:17.940 INFO:tasks.workunit.client.0.smithi136.stdout: ], 2024-09-15T22:02:17.940 INFO:tasks.workunit.client.0.smithi136.stdout: "truncate_seq": 0, 2024-09-15T22:02:17.940 INFO:tasks.workunit.client.0.smithi136.stdout: "truncate_size": 0, 2024-09-15T22:02:17.940 INFO:tasks.workunit.client.0.smithi136.stdout: "data_digest": "0x87dd7e93", 2024-09-15T22:02:17.940 INFO:tasks.workunit.client.0.smithi136.stdout: "omap_digest": "0xffffffff", 2024-09-15T22:02:17.940 INFO:tasks.workunit.client.0.smithi136.stdout: "expected_object_size": 0, 2024-09-15T22:02:17.940 INFO:tasks.workunit.client.0.smithi136.stdout: "expected_write_size": 0, 2024-09-15T22:02:17.940 INFO:tasks.workunit.client.0.smithi136.stdout: "alloc_hint_flags": 0, 2024-09-15T22:02:17.940 INFO:tasks.workunit.client.0.smithi136.stdout: "manifest": { 2024-09-15T22:02:17.941 INFO:tasks.workunit.client.0.smithi136.stdout: "type": 0 2024-09-15T22:02:17.941 INFO:tasks.workunit.client.0.smithi136.stdout: }, 2024-09-15T22:02:17.941 INFO:tasks.workunit.client.0.smithi136.stdout: "watchers": {} 2024-09-15T22:02:17.941 INFO:tasks.workunit.client.0.smithi136.stdout: }, 2024-09-15T22:02:17.941 INFO:tasks.workunit.client.0.smithi136.stdout: "shards": [ 2024-09-15T22:02:17.944 INFO:tasks.workunit.client.0.smithi136.stdout: { 2024-09-15T22:02:17.944 INFO:tasks.workunit.client.0.smithi136.stdout: "osd": 0, 2024-09-15T22:02:17.944 INFO:tasks.workunit.client.0.smithi136.stdout: "primary": false, 2024-09-15T22:02:17.944 INFO:tasks.workunit.client.0.smithi136.stdout: "errors": [], 2024-09-15T22:02:17.945 INFO:tasks.workunit.client.0.smithi136.stdout: "size": 768 2024-09-15T22:02:17.945 INFO:tasks.workunit.client.0.smithi136.stdout: }, 2024-09-15T22:02:17.945 INFO:tasks.workunit.client.0.smithi136.stdout: { 2024-09-15T22:02:17.945 INFO:tasks.workunit.client.0.smithi136.stdout: "osd": 1, 2024-09-15T22:02:17.945 INFO:tasks.workunit.client.0.smithi136.stdout: "primary": true, 2024-09-15T22:02:17.945 INFO:tasks.workunit.client.0.smithi136.stdout: "errors": [ 2024-09-15T22:02:17.945 INFO:tasks.workunit.client.0.smithi136.stdout: "size_mismatch_info", 2024-09-15T22:02:17.945 INFO:tasks.workunit.client.0.smithi136.stdout: "obj_size_info_mismatch" 2024-09-15T22:02:17.945 INFO:tasks.workunit.client.0.smithi136.stdout: ], 2024-09-15T22:02:17.945 INFO:tasks.workunit.client.0.smithi136.stdout: "size": 3840, 2024-09-15T22:02:17.945 INFO:tasks.workunit.client.0.smithi136.stdout: "object_info": { 2024-09-15T22:02:17.945 INFO:tasks.workunit.client.0.smithi136.stdout: "oid": { 2024-09-15T22:02:17.945 INFO:tasks.workunit.client.0.smithi136.stdout: "oid": "obj3", 2024-09-15T22:02:17.945 INFO:tasks.workunit.client.0.smithi136.stdout: "key": "", 2024-09-15T22:02:17.945 INFO:tasks.workunit.client.0.smithi136.stdout: "snapid": -2, 2024-09-15T22:02:17.946 INFO:tasks.workunit.client.0.smithi136.stdout: "hash": 1643547569, 2024-09-15T22:02:17.946 INFO:tasks.workunit.client.0.smithi136.stdout: "max": 0, 2024-09-15T22:02:17.946 INFO:tasks.workunit.client.0.smithi136.stdout: "pool": 1, 2024-09-15T22:02:17.946 INFO:tasks.workunit.client.0.smithi136.stdout: "namespace": "" 2024-09-15T22:02:17.946 INFO:tasks.workunit.client.0.smithi136.stdout: }, 2024-09-15T22:02:17.946 INFO:tasks.workunit.client.0.smithi136.stdout: "version": "18'44", 2024-09-15T22:02:17.946 INFO:tasks.workunit.client.0.smithi136.stdout: "prior_version": "16'22", 2024-09-15T22:02:17.946 INFO:tasks.workunit.client.0.smithi136.stdout: "last_reqid": "client.4339.0:1", 2024-09-15T22:02:17.946 INFO:tasks.workunit.client.0.smithi136.stdout: "user_version": 44, 2024-09-15T22:02:17.946 INFO:tasks.workunit.client.0.smithi136.stdout: "size": 768, 2024-09-15T22:02:17.946 INFO:tasks.workunit.client.0.smithi136.stdout: "mtime": "2024-09-15T22:01:08.280317+0000", 2024-09-15T22:02:17.946 INFO:tasks.workunit.client.0.smithi136.stdout: "local_mtime": "2024-09-15T22:01:08.290990+0000", 2024-09-15T22:02:17.946 INFO:tasks.workunit.client.0.smithi136.stdout: "lost": 0, 2024-09-15T22:02:17.946 INFO:tasks.workunit.client.0.smithi136.stdout: "flags": [ 2024-09-15T22:02:17.946 INFO:tasks.workunit.client.0.smithi136.stdout: "dirty", 2024-09-15T22:02:17.947 INFO:tasks.workunit.client.0.smithi136.stdout: "data_digest" 2024-09-15T22:02:17.947 INFO:tasks.workunit.client.0.smithi136.stdout: ], 2024-09-15T22:02:17.947 INFO:tasks.workunit.client.0.smithi136.stdout: "truncate_seq": 0, 2024-09-15T22:02:17.947 INFO:tasks.workunit.client.0.smithi136.stdout: "truncate_size": 0, 2024-09-15T22:02:17.947 INFO:tasks.workunit.client.0.smithi136.stdout: "data_digest": "0x87dd7e93", 2024-09-15T22:02:17.947 INFO:tasks.workunit.client.0.smithi136.stdout: "omap_digest": "0xffffffff", 2024-09-15T22:02:17.947 INFO:tasks.workunit.client.0.smithi136.stdout: "expected_object_size": 0, 2024-09-15T22:02:17.947 INFO:tasks.workunit.client.0.smithi136.stdout: "expected_write_size": 0, 2024-09-15T22:02:17.947 INFO:tasks.workunit.client.0.smithi136.stdout: "alloc_hint_flags": 0, 2024-09-15T22:02:17.947 INFO:tasks.workunit.client.0.smithi136.stdout: "manifest": { 2024-09-15T22:02:17.947 INFO:tasks.workunit.client.0.smithi136.stdout: "type": 0 2024-09-15T22:02:17.947 INFO:tasks.workunit.client.0.smithi136.stdout: }, 2024-09-15T22:02:17.947 INFO:tasks.workunit.client.0.smithi136.stdout: "watchers": {} 2024-09-15T22:02:17.947 INFO:tasks.workunit.client.0.smithi136.stdout: } 2024-09-15T22:02:17.948 INFO:tasks.workunit.client.0.smithi136.stdout: } 2024-09-15T22:02:17.948 INFO:tasks.workunit.client.0.smithi136.stdout: ] 2024-09-15T22:02:17.948 INFO:tasks.workunit.client.0.smithi136.stdout: }, 2024-09-15T22:02:17.948 INFO:tasks.workunit.client.0.smithi136.stdout: { 2024-09-15T22:02:17.948 INFO:tasks.workunit.client.0.smithi136.stdout: "object": { 2024-09-15T22:02:17.948 INFO:tasks.workunit.client.0.smithi136.stdout: "name": "obj6", 2024-09-15T22:02:17.948 INFO:tasks.workunit.client.0.smithi136.stdout: "nspace": "", 2024-09-15T22:02:17.948 INFO:tasks.workunit.client.0.smithi136.stdout: "locator": "", 2024-09-15T22:02:17.948 INFO:tasks.workunit.client.0.smithi136.stdout: "snap": "head", 2024-09-15T22:02:17.948 INFO:tasks.workunit.client.0.smithi136.stdout: "version": 24 2024-09-15T22:02:17.948 INFO:tasks.workunit.client.0.smithi136.stdout: }, 2024-09-15T22:02:17.948 INFO:tasks.workunit.client.0.smithi136.stdout: "errors": [ 2024-09-15T22:02:17.948 INFO:tasks.workunit.client.0.smithi136.stdout: "snapset_inconsistency" 2024-09-15T22:02:17.948 INFO:tasks.workunit.client.0.smithi136.stdout: ], 2024-09-15T22:02:17.948 INFO:tasks.workunit.client.0.smithi136.stdout: "union_shard_errors": [], 2024-09-15T22:02:17.949 INFO:tasks.workunit.client.0.smithi136.stdout: "selected_object_info": { 2024-09-15T22:02:17.949 INFO:tasks.workunit.client.0.smithi136.stdout: "oid": { 2024-09-15T22:02:17.949 INFO:tasks.workunit.client.0.smithi136.stdout: "oid": "obj6", 2024-09-15T22:02:17.949 INFO:tasks.workunit.client.0.smithi136.stdout: "key": "", 2024-09-15T22:02:17.949 INFO:tasks.workunit.client.0.smithi136.stdout: "snapid": -2, 2024-09-15T22:02:17.949 INFO:tasks.workunit.client.0.smithi136.stdout: "hash": 2202164420, 2024-09-15T22:02:17.949 INFO:tasks.workunit.client.0.smithi136.stdout: "max": 0, 2024-09-15T22:02:17.949 INFO:tasks.workunit.client.0.smithi136.stdout: "pool": 1, 2024-09-15T22:02:17.949 INFO:tasks.workunit.client.0.smithi136.stdout: "namespace": "" 2024-09-15T22:02:17.949 INFO:tasks.workunit.client.0.smithi136.stdout: }, 2024-09-15T22:02:17.949 INFO:tasks.workunit.client.0.smithi136.stdout: "version": "16'24", 2024-09-15T22:02:17.949 INFO:tasks.workunit.client.0.smithi136.stdout: "prior_version": "15'6", 2024-09-15T22:02:17.949 INFO:tasks.workunit.client.0.smithi136.stdout: "last_reqid": "client.4303.0:1", 2024-09-15T22:02:17.949 INFO:tasks.workunit.client.0.smithi136.stdout: "user_version": 24, 2024-09-15T22:02:17.949 INFO:tasks.workunit.client.0.smithi136.stdout: "size": 256, 2024-09-15T22:02:17.950 INFO:tasks.workunit.client.0.smithi136.stdout: "mtime": "2024-09-15T22:01:06.379506+0000", 2024-09-15T22:02:17.950 INFO:tasks.workunit.client.0.smithi136.stdout: "local_mtime": "2024-09-15T22:01:06.380132+0000", 2024-09-15T22:02:17.950 INFO:tasks.workunit.client.0.smithi136.stdout: "lost": 0, 2024-09-15T22:02:17.950 INFO:tasks.workunit.client.0.smithi136.stdout: "flags": [ 2024-09-15T22:02:17.950 INFO:tasks.workunit.client.0.smithi136.stdout: "dirty", 2024-09-15T22:02:17.950 INFO:tasks.workunit.client.0.smithi136.stdout: "data_digest" 2024-09-15T22:02:17.950 INFO:tasks.workunit.client.0.smithi136.stdout: ], 2024-09-15T22:02:17.950 INFO:tasks.workunit.client.0.smithi136.stdout: "truncate_seq": 0, 2024-09-15T22:02:17.950 INFO:tasks.workunit.client.0.smithi136.stdout: "truncate_size": 0, 2024-09-15T22:02:17.950 INFO:tasks.workunit.client.0.smithi136.stdout: "data_digest": "0x29e45974", 2024-09-15T22:02:17.950 INFO:tasks.workunit.client.0.smithi136.stdout: "omap_digest": "0xffffffff", 2024-09-15T22:02:17.950 INFO:tasks.workunit.client.0.smithi136.stdout: "expected_object_size": 0, 2024-09-15T22:02:17.950 INFO:tasks.workunit.client.0.smithi136.stdout: "expected_write_size": 0, 2024-09-15T22:02:17.950 INFO:tasks.workunit.client.0.smithi136.stdout: "alloc_hint_flags": 0, 2024-09-15T22:02:17.951 INFO:tasks.workunit.client.0.smithi136.stdout: "manifest": { 2024-09-15T22:02:17.951 INFO:tasks.workunit.client.0.smithi136.stdout: "type": 0 2024-09-15T22:02:17.951 INFO:tasks.workunit.client.0.smithi136.stdout: }, 2024-09-15T22:02:17.951 INFO:tasks.workunit.client.0.smithi136.stdout: "watchers": {} 2024-09-15T22:02:17.951 INFO:tasks.workunit.client.0.smithi136.stdout: }, 2024-09-15T22:02:17.951 INFO:tasks.workunit.client.0.smithi136.stdout: "shards": [ 2024-09-15T22:02:17.951 INFO:tasks.workunit.client.0.smithi136.stdout: { 2024-09-15T22:02:17.951 INFO:tasks.workunit.client.0.smithi136.stdout: "osd": 0, 2024-09-15T22:02:17.951 INFO:tasks.workunit.client.0.smithi136.stdout: "primary": false, 2024-09-15T22:02:17.951 INFO:tasks.workunit.client.0.smithi136.stdout: "errors": [], 2024-09-15T22:02:17.951 INFO:tasks.workunit.client.0.smithi136.stdout: "size": 256, 2024-09-15T22:02:17.951 INFO:tasks.workunit.client.0.smithi136.stdout: "snapset": { 2024-09-15T22:02:17.951 INFO:tasks.workunit.client.0.smithi136.stdout: "seq": 1, 2024-09-15T22:02:17.951 INFO:tasks.workunit.client.0.smithi136.stdout: "clones": [ 2024-09-15T22:02:17.951 INFO:tasks.workunit.client.0.smithi136.stdout: { 2024-09-15T22:02:17.952 INFO:tasks.workunit.client.0.smithi136.stdout: "snap": 1, 2024-09-15T22:02:17.952 INFO:tasks.workunit.client.0.smithi136.stdout: "size": 1032, 2024-09-15T22:02:17.952 INFO:tasks.workunit.client.0.smithi136.stdout: "overlap": "[]", 2024-09-15T22:02:17.952 INFO:tasks.workunit.client.0.smithi136.stdout: "snaps": [ 2024-09-15T22:02:17.952 INFO:tasks.workunit.client.0.smithi136.stdout: 1 2024-09-15T22:02:17.952 INFO:tasks.workunit.client.0.smithi136.stdout: ] 2024-09-15T22:02:17.952 INFO:tasks.workunit.client.0.smithi136.stdout: } 2024-09-15T22:02:17.952 INFO:tasks.workunit.client.0.smithi136.stdout: ] 2024-09-15T22:02:17.952 INFO:tasks.workunit.client.0.smithi136.stdout: } 2024-09-15T22:02:17.952 INFO:tasks.workunit.client.0.smithi136.stdout: }, 2024-09-15T22:02:17.952 INFO:tasks.workunit.client.0.smithi136.stdout: { 2024-09-15T22:02:17.952 INFO:tasks.workunit.client.0.smithi136.stdout: "osd": 1, 2024-09-15T22:02:17.952 INFO:tasks.workunit.client.0.smithi136.stdout: "primary": true, 2024-09-15T22:02:17.952 INFO:tasks.workunit.client.0.smithi136.stdout: "errors": [], 2024-09-15T22:02:17.953 INFO:tasks.workunit.client.0.smithi136.stdout: "size": 256, 2024-09-15T22:02:17.953 INFO:tasks.workunit.client.0.smithi136.stdout: "snapset": { 2024-09-15T22:02:17.953 INFO:tasks.workunit.client.0.smithi136.stdout: "seq": 1, 2024-09-15T22:02:17.953 INFO:tasks.workunit.client.0.smithi136.stdout: "clones": [] 2024-09-15T22:02:17.953 INFO:tasks.workunit.client.0.smithi136.stdout: } 2024-09-15T22:02:17.953 INFO:tasks.workunit.client.0.smithi136.stdout: } 2024-09-15T22:02:17.953 INFO:tasks.workunit.client.0.smithi136.stdout: ] 2024-09-15T22:02:17.953 INFO:tasks.workunit.client.0.smithi136.stdout: }, 2024-09-15T22:02:17.953 INFO:tasks.workunit.client.0.smithi136.stdout: { 2024-09-15T22:02:17.953 INFO:tasks.workunit.client.0.smithi136.stdout: "object": { 2024-09-15T22:02:17.953 INFO:tasks.workunit.client.0.smithi136.stdout: "name": "obj7", 2024-09-15T22:02:17.953 INFO:tasks.workunit.client.0.smithi136.stdout: "nspace": "", 2024-09-15T22:02:17.953 INFO:tasks.workunit.client.0.smithi136.stdout: "locator": "", 2024-09-15T22:02:17.953 INFO:tasks.workunit.client.0.smithi136.stdout: "snap": "head", 2024-09-15T22:02:17.953 INFO:tasks.workunit.client.0.smithi136.stdout: "version": 26 2024-09-15T22:02:17.954 INFO:tasks.workunit.client.0.smithi136.stdout: }, 2024-09-15T22:02:17.954 INFO:tasks.workunit.client.0.smithi136.stdout: "errors": [ 2024-09-15T22:02:17.954 INFO:tasks.workunit.client.0.smithi136.stdout: "snapset_inconsistency" 2024-09-15T22:02:17.954 INFO:tasks.workunit.client.0.smithi136.stdout: ], 2024-09-15T22:02:17.954 INFO:tasks.workunit.client.0.smithi136.stdout: "union_shard_errors": [], 2024-09-15T22:02:17.954 INFO:tasks.workunit.client.0.smithi136.stdout: "selected_object_info": { 2024-09-15T22:02:17.954 INFO:tasks.workunit.client.0.smithi136.stdout: "oid": { 2024-09-15T22:02:17.954 INFO:tasks.workunit.client.0.smithi136.stdout: "oid": "obj7", 2024-09-15T22:02:17.954 INFO:tasks.workunit.client.0.smithi136.stdout: "key": "", 2024-09-15T22:02:17.954 INFO:tasks.workunit.client.0.smithi136.stdout: "snapid": -2, 2024-09-15T22:02:17.954 INFO:tasks.workunit.client.0.smithi136.stdout: "hash": 1552453721, 2024-09-15T22:02:17.954 INFO:tasks.workunit.client.0.smithi136.stdout: "max": 0, 2024-09-15T22:02:17.954 INFO:tasks.workunit.client.0.smithi136.stdout: "pool": 1, 2024-09-15T22:02:17.954 INFO:tasks.workunit.client.0.smithi136.stdout: "namespace": "" 2024-09-15T22:02:17.955 INFO:tasks.workunit.client.0.smithi136.stdout: }, 2024-09-15T22:02:17.955 INFO:tasks.workunit.client.0.smithi136.stdout: "version": "16'26", 2024-09-15T22:02:17.955 INFO:tasks.workunit.client.0.smithi136.stdout: "prior_version": "15'7", 2024-09-15T22:02:17.955 INFO:tasks.workunit.client.0.smithi136.stdout: "last_reqid": "client.4306.0:1", 2024-09-15T22:02:17.955 INFO:tasks.workunit.client.0.smithi136.stdout: "user_version": 26, 2024-09-15T22:02:17.955 INFO:tasks.workunit.client.0.smithi136.stdout: "size": 256, 2024-09-15T22:02:17.955 INFO:tasks.workunit.client.0.smithi136.stdout: "mtime": "2024-09-15T22:01:06.520947+0000", 2024-09-15T22:02:17.955 INFO:tasks.workunit.client.0.smithi136.stdout: "local_mtime": "2024-09-15T22:01:06.521567+0000", 2024-09-15T22:02:17.955 INFO:tasks.workunit.client.0.smithi136.stdout: "lost": 0, 2024-09-15T22:02:17.955 INFO:tasks.workunit.client.0.smithi136.stdout: "flags": [ 2024-09-15T22:02:17.955 INFO:tasks.workunit.client.0.smithi136.stdout: "dirty", 2024-09-15T22:02:17.955 INFO:tasks.workunit.client.0.smithi136.stdout: "data_digest" 2024-09-15T22:02:17.955 INFO:tasks.workunit.client.0.smithi136.stdout: ], 2024-09-15T22:02:17.955 INFO:tasks.workunit.client.0.smithi136.stdout: "truncate_seq": 0, 2024-09-15T22:02:17.955 INFO:tasks.workunit.client.0.smithi136.stdout: "truncate_size": 0, 2024-09-15T22:02:17.956 INFO:tasks.workunit.client.0.smithi136.stdout: "data_digest": "0x29e45974", 2024-09-15T22:02:17.956 INFO:tasks.workunit.client.0.smithi136.stdout: "omap_digest": "0xffffffff", 2024-09-15T22:02:17.956 INFO:tasks.workunit.client.0.smithi136.stdout: "expected_object_size": 0, 2024-09-15T22:02:17.956 INFO:tasks.workunit.client.0.smithi136.stdout: "expected_write_size": 0, 2024-09-15T22:02:17.956 INFO:tasks.workunit.client.0.smithi136.stdout: "alloc_hint_flags": 0, 2024-09-15T22:02:17.956 INFO:tasks.workunit.client.0.smithi136.stdout: "manifest": { 2024-09-15T22:02:17.956 INFO:tasks.workunit.client.0.smithi136.stdout: "type": 0 2024-09-15T22:02:17.956 INFO:tasks.workunit.client.0.smithi136.stdout: }, 2024-09-15T22:02:17.956 INFO:tasks.workunit.client.0.smithi136.stdout: "watchers": {} 2024-09-15T22:02:17.956 INFO:tasks.workunit.client.0.smithi136.stdout: }, 2024-09-15T22:02:17.956 INFO:tasks.workunit.client.0.smithi136.stdout: "shards": [ 2024-09-15T22:02:17.956 INFO:tasks.workunit.client.0.smithi136.stdout: { 2024-09-15T22:02:17.956 INFO:tasks.workunit.client.0.smithi136.stdout: "osd": 0, 2024-09-15T22:02:17.956 INFO:tasks.workunit.client.0.smithi136.stdout: "primary": false, 2024-09-15T22:02:17.956 INFO:tasks.workunit.client.0.smithi136.stdout: "errors": [], 2024-09-15T22:02:17.957 INFO:tasks.workunit.client.0.smithi136.stdout: "size": 256, 2024-09-15T22:02:17.957 INFO:tasks.workunit.client.0.smithi136.stdout: "snapset": { 2024-09-15T22:02:17.957 INFO:tasks.workunit.client.0.smithi136.stdout: "seq": 1, 2024-09-15T22:02:17.957 INFO:tasks.workunit.client.0.smithi136.stdout: "clones": [ 2024-09-15T22:02:17.957 INFO:tasks.workunit.client.0.smithi136.stdout: { 2024-09-15T22:02:17.957 INFO:tasks.workunit.client.0.smithi136.stdout: "snap": 1, 2024-09-15T22:02:17.957 INFO:tasks.workunit.client.0.smithi136.stdout: "size": 1032, 2024-09-15T22:02:17.957 INFO:tasks.workunit.client.0.smithi136.stdout: "overlap": "[]", 2024-09-15T22:02:17.957 INFO:tasks.workunit.client.0.smithi136.stdout: "snaps": [ 2024-09-15T22:02:17.957 INFO:tasks.workunit.client.0.smithi136.stdout: 1 2024-09-15T22:02:17.957 INFO:tasks.workunit.client.0.smithi136.stdout: ] 2024-09-15T22:02:17.957 INFO:tasks.workunit.client.0.smithi136.stdout: } 2024-09-15T22:02:17.957 INFO:tasks.workunit.client.0.smithi136.stdout: ] 2024-09-15T22:02:17.957 INFO:tasks.workunit.client.0.smithi136.stdout: } 2024-09-15T22:02:17.958 INFO:tasks.workunit.client.0.smithi136.stdout: }, 2024-09-15T22:02:17.958 INFO:tasks.workunit.client.0.smithi136.stdout: { 2024-09-15T22:02:17.958 INFO:tasks.workunit.client.0.smithi136.stdout: "osd": 1, 2024-09-15T22:02:17.958 INFO:tasks.workunit.client.0.smithi136.stdout: "primary": true, 2024-09-15T22:02:17.958 INFO:tasks.workunit.client.0.smithi136.stdout: "errors": [], 2024-09-15T22:02:17.958 INFO:tasks.workunit.client.0.smithi136.stdout: "size": 256, 2024-09-15T22:02:17.958 INFO:tasks.workunit.client.0.smithi136.stdout: "snapset": { 2024-09-15T22:02:17.958 INFO:tasks.workunit.client.0.smithi136.stdout: "seq": 0, 2024-09-15T22:02:17.958 INFO:tasks.workunit.client.0.smithi136.stdout: "clones": [] 2024-09-15T22:02:17.958 INFO:tasks.workunit.client.0.smithi136.stdout: } 2024-09-15T22:02:17.958 INFO:tasks.workunit.client.0.smithi136.stdout: } 2024-09-15T22:02:17.958 INFO:tasks.workunit.client.0.smithi136.stdout: ] 2024-09-15T22:02:17.958 INFO:tasks.workunit.client.0.smithi136.stdout: }, 2024-09-15T22:02:17.958 INFO:tasks.workunit.client.0.smithi136.stdout: { 2024-09-15T22:02:17.959 INFO:tasks.workunit.client.0.smithi136.stdout: "object": { 2024-09-15T22:02:17.959 INFO:tasks.workunit.client.0.smithi136.stdout: "name": "obj8", 2024-09-15T22:02:17.959 INFO:tasks.workunit.client.0.smithi136.stdout: "nspace": "", 2024-09-15T22:02:17.959 INFO:tasks.workunit.client.0.smithi136.stdout: "locator": "", 2024-09-15T22:02:17.959 INFO:tasks.workunit.client.0.smithi136.stdout: "snap": "head", 2024-09-15T22:02:17.959 INFO:tasks.workunit.client.0.smithi136.stdout: "version": 28 2024-09-15T22:02:17.959 INFO:tasks.workunit.client.0.smithi136.stdout: }, 2024-09-15T22:02:17.959 INFO:tasks.workunit.client.0.smithi136.stdout: "errors": [ 2024-09-15T22:02:17.959 INFO:tasks.workunit.client.0.smithi136.stdout: "snapset_inconsistency" 2024-09-15T22:02:17.959 INFO:tasks.workunit.client.0.smithi136.stdout: ], 2024-09-15T22:02:17.959 INFO:tasks.workunit.client.0.smithi136.stdout: "union_shard_errors": [], 2024-09-15T22:02:17.959 INFO:tasks.workunit.client.0.smithi136.stdout: "selected_object_info": { 2024-09-15T22:02:17.959 INFO:tasks.workunit.client.0.smithi136.stdout: "oid": { 2024-09-15T22:02:17.959 INFO:tasks.workunit.client.0.smithi136.stdout: "oid": "obj8", 2024-09-15T22:02:17.960 INFO:tasks.workunit.client.0.smithi136.stdout: "key": "", 2024-09-15T22:02:17.960 INFO:tasks.workunit.client.0.smithi136.stdout: "snapid": -2, 2024-09-15T22:02:17.960 INFO:tasks.workunit.client.0.smithi136.stdout: "hash": 2381834917, 2024-09-15T22:02:17.960 INFO:tasks.workunit.client.0.smithi136.stdout: "max": 0, 2024-09-15T22:02:17.960 INFO:tasks.workunit.client.0.smithi136.stdout: "pool": 1, 2024-09-15T22:02:17.960 INFO:tasks.workunit.client.0.smithi136.stdout: "namespace": "" 2024-09-15T22:02:17.960 INFO:tasks.workunit.client.0.smithi136.stdout: }, 2024-09-15T22:02:17.960 INFO:tasks.workunit.client.0.smithi136.stdout: "version": "16'28", 2024-09-15T22:02:17.960 INFO:tasks.workunit.client.0.smithi136.stdout: "prior_version": "15'8", 2024-09-15T22:02:17.960 INFO:tasks.workunit.client.0.smithi136.stdout: "last_reqid": "client.4309.0:1", 2024-09-15T22:02:17.960 INFO:tasks.workunit.client.0.smithi136.stdout: "user_version": 28, 2024-09-15T22:02:17.960 INFO:tasks.workunit.client.0.smithi136.stdout: "size": 256, 2024-09-15T22:02:17.960 INFO:tasks.workunit.client.0.smithi136.stdout: "mtime": "2024-09-15T22:01:06.671273+0000", 2024-09-15T22:02:17.960 INFO:tasks.workunit.client.0.smithi136.stdout: "local_mtime": "2024-09-15T22:01:06.671852+0000", 2024-09-15T22:02:17.961 INFO:tasks.workunit.client.0.smithi136.stdout: "lost": 0, 2024-09-15T22:02:17.961 INFO:tasks.workunit.client.0.smithi136.stdout: "flags": [ 2024-09-15T22:02:17.961 INFO:tasks.workunit.client.0.smithi136.stdout: "dirty", 2024-09-15T22:02:17.961 INFO:tasks.workunit.client.0.smithi136.stdout: "data_digest" 2024-09-15T22:02:17.961 INFO:tasks.workunit.client.0.smithi136.stdout: ], 2024-09-15T22:02:17.961 INFO:tasks.workunit.client.0.smithi136.stdout: "truncate_seq": 0, 2024-09-15T22:02:17.961 INFO:tasks.workunit.client.0.smithi136.stdout: "truncate_size": 0, 2024-09-15T22:02:17.961 INFO:tasks.workunit.client.0.smithi136.stdout: "data_digest": "0x29e45974", 2024-09-15T22:02:17.961 INFO:tasks.workunit.client.0.smithi136.stdout: "omap_digest": "0xffffffff", 2024-09-15T22:02:17.961 INFO:tasks.workunit.client.0.smithi136.stdout: "expected_object_size": 0, 2024-09-15T22:02:17.961 INFO:tasks.workunit.client.0.smithi136.stdout: "expected_write_size": 0, 2024-09-15T22:02:17.961 INFO:tasks.workunit.client.0.smithi136.stdout: "alloc_hint_flags": 0, 2024-09-15T22:02:17.961 INFO:tasks.workunit.client.0.smithi136.stdout: "manifest": { 2024-09-15T22:02:17.961 INFO:tasks.workunit.client.0.smithi136.stdout: "type": 0 2024-09-15T22:02:17.961 INFO:tasks.workunit.client.0.smithi136.stdout: }, 2024-09-15T22:02:17.962 INFO:tasks.workunit.client.0.smithi136.stdout: "watchers": {} 2024-09-15T22:02:17.962 INFO:tasks.workunit.client.0.smithi136.stdout: }, 2024-09-15T22:02:17.962 INFO:tasks.workunit.client.0.smithi136.stdout: "shards": [ 2024-09-15T22:02:17.962 INFO:tasks.workunit.client.0.smithi136.stdout: { 2024-09-15T22:02:17.962 INFO:tasks.workunit.client.0.smithi136.stdout: "osd": 0, 2024-09-15T22:02:17.962 INFO:tasks.workunit.client.0.smithi136.stdout: "primary": false, 2024-09-15T22:02:17.962 INFO:tasks.workunit.client.0.smithi136.stdout: "errors": [], 2024-09-15T22:02:17.962 INFO:tasks.workunit.client.0.smithi136.stdout: "size": 256, 2024-09-15T22:02:17.962 INFO:tasks.workunit.client.0.smithi136.stdout: "snapset": { 2024-09-15T22:02:17.962 INFO:tasks.workunit.client.0.smithi136.stdout: "seq": 1, 2024-09-15T22:02:17.962 INFO:tasks.workunit.client.0.smithi136.stdout: "clones": [ 2024-09-15T22:02:17.962 INFO:tasks.workunit.client.0.smithi136.stdout: { 2024-09-15T22:02:17.962 INFO:tasks.workunit.client.0.smithi136.stdout: "snap": 1, 2024-09-15T22:02:17.962 INFO:tasks.workunit.client.0.smithi136.stdout: "size": 1032, 2024-09-15T22:02:17.963 INFO:tasks.workunit.client.0.smithi136.stdout: "overlap": "[]", 2024-09-15T22:02:17.963 INFO:tasks.workunit.client.0.smithi136.stdout: "snaps": [ 2024-09-15T22:02:17.963 INFO:tasks.workunit.client.0.smithi136.stdout: 1 2024-09-15T22:02:17.963 INFO:tasks.workunit.client.0.smithi136.stdout: ] 2024-09-15T22:02:17.963 INFO:tasks.workunit.client.0.smithi136.stdout: } 2024-09-15T22:02:17.963 INFO:tasks.workunit.client.0.smithi136.stdout: ] 2024-09-15T22:02:17.963 INFO:tasks.workunit.client.0.smithi136.stdout: } 2024-09-15T22:02:17.963 INFO:tasks.workunit.client.0.smithi136.stdout: }, 2024-09-15T22:02:17.963 INFO:tasks.workunit.client.0.smithi136.stdout: { 2024-09-15T22:02:17.963 INFO:tasks.workunit.client.0.smithi136.stdout: "osd": 1, 2024-09-15T22:02:17.963 INFO:tasks.workunit.client.0.smithi136.stdout: "primary": true, 2024-09-15T22:02:17.963 INFO:tasks.workunit.client.0.smithi136.stdout: "errors": [], 2024-09-15T22:02:17.963 INFO:tasks.workunit.client.0.smithi136.stdout: "size": 256, 2024-09-15T22:02:17.963 INFO:tasks.workunit.client.0.smithi136.stdout: "snapset": { 2024-09-15T22:02:17.963 INFO:tasks.workunit.client.0.smithi136.stdout: "seq": 0, 2024-09-15T22:02:17.964 INFO:tasks.workunit.client.0.smithi136.stdout: "clones": [ 2024-09-15T22:02:17.964 INFO:tasks.workunit.client.0.smithi136.stdout: { 2024-09-15T22:02:17.964 INFO:tasks.workunit.client.0.smithi136.stdout: "snap": 1, 2024-09-15T22:02:17.964 INFO:tasks.workunit.client.0.smithi136.stdout: "size": 1032, 2024-09-15T22:02:17.964 INFO:tasks.workunit.client.0.smithi136.stdout: "overlap": "[]", 2024-09-15T22:02:17.964 INFO:tasks.workunit.client.0.smithi136.stdout: "snaps": [ 2024-09-15T22:02:17.964 INFO:tasks.workunit.client.0.smithi136.stdout: 1 2024-09-15T22:02:17.964 INFO:tasks.workunit.client.0.smithi136.stdout: ] 2024-09-15T22:02:17.964 INFO:tasks.workunit.client.0.smithi136.stdout: } 2024-09-15T22:02:17.964 INFO:tasks.workunit.client.0.smithi136.stdout: ] 2024-09-15T22:02:17.964 INFO:tasks.workunit.client.0.smithi136.stdout: } 2024-09-15T22:02:17.964 INFO:tasks.workunit.client.0.smithi136.stdout: } 2024-09-15T22:02:17.964 INFO:tasks.workunit.client.0.smithi136.stdout: ] 2024-09-15T22:02:17.964 INFO:tasks.workunit.client.0.smithi136.stdout: }, 2024-09-15T22:02:17.965 INFO:tasks.workunit.client.0.smithi136.stdout: { 2024-09-15T22:02:17.965 INFO:tasks.workunit.client.0.smithi136.stdout: "object": { 2024-09-15T22:02:17.965 INFO:tasks.workunit.client.0.smithi136.stdout: "name": "obj9", 2024-09-15T22:02:17.965 INFO:tasks.workunit.client.0.smithi136.stdout: "nspace": "", 2024-09-15T22:02:17.965 INFO:tasks.workunit.client.0.smithi136.stdout: "locator": "", 2024-09-15T22:02:17.965 INFO:tasks.workunit.client.0.smithi136.stdout: "snap": "head", 2024-09-15T22:02:17.965 INFO:tasks.workunit.client.0.smithi136.stdout: "version": 30 2024-09-15T22:02:17.965 INFO:tasks.workunit.client.0.smithi136.stdout: }, 2024-09-15T22:02:17.965 INFO:tasks.workunit.client.0.smithi136.stdout: "errors": [ 2024-09-15T22:02:17.965 INFO:tasks.workunit.client.0.smithi136.stdout: "snapset_inconsistency" 2024-09-15T22:02:17.965 INFO:tasks.workunit.client.0.smithi136.stdout: ], 2024-09-15T22:02:17.965 INFO:tasks.workunit.client.0.smithi136.stdout: "union_shard_errors": [], 2024-09-15T22:02:17.965 INFO:tasks.workunit.client.0.smithi136.stdout: "selected_object_info": { 2024-09-15T22:02:17.965 INFO:tasks.workunit.client.0.smithi136.stdout: "oid": { 2024-09-15T22:02:17.965 INFO:tasks.workunit.client.0.smithi136.stdout: "oid": "obj9", 2024-09-15T22:02:17.966 INFO:tasks.workunit.client.0.smithi136.stdout: "key": "", 2024-09-15T22:02:17.966 INFO:tasks.workunit.client.0.smithi136.stdout: "snapid": -2, 2024-09-15T22:02:17.966 INFO:tasks.workunit.client.0.smithi136.stdout: "hash": 3833113727, 2024-09-15T22:02:17.966 INFO:tasks.workunit.client.0.smithi136.stdout: "max": 0, 2024-09-15T22:02:17.966 INFO:tasks.workunit.client.0.smithi136.stdout: "pool": 1, 2024-09-15T22:02:17.966 INFO:tasks.workunit.client.0.smithi136.stdout: "namespace": "" 2024-09-15T22:02:17.966 INFO:tasks.workunit.client.0.smithi136.stdout: }, 2024-09-15T22:02:17.966 INFO:tasks.workunit.client.0.smithi136.stdout: "version": "16'30", 2024-09-15T22:02:17.966 INFO:tasks.workunit.client.0.smithi136.stdout: "prior_version": "15'9", 2024-09-15T22:02:17.966 INFO:tasks.workunit.client.0.smithi136.stdout: "last_reqid": "client.4312.0:1", 2024-09-15T22:02:17.966 INFO:tasks.workunit.client.0.smithi136.stdout: "user_version": 30, 2024-09-15T22:02:17.966 INFO:tasks.workunit.client.0.smithi136.stdout: "size": 256, 2024-09-15T22:02:17.966 INFO:tasks.workunit.client.0.smithi136.stdout: "mtime": "2024-09-15T22:01:06.805265+0000", 2024-09-15T22:02:17.966 INFO:tasks.workunit.client.0.smithi136.stdout: "local_mtime": "2024-09-15T22:01:06.805869+0000", 2024-09-15T22:02:17.967 INFO:tasks.workunit.client.0.smithi136.stdout: "lost": 0, 2024-09-15T22:02:17.967 INFO:tasks.workunit.client.0.smithi136.stdout: "flags": [ 2024-09-15T22:02:17.967 INFO:tasks.workunit.client.0.smithi136.stdout: "dirty", 2024-09-15T22:02:17.967 INFO:tasks.workunit.client.0.smithi136.stdout: "data_digest" 2024-09-15T22:02:17.967 INFO:tasks.workunit.client.0.smithi136.stdout: ], 2024-09-15T22:02:17.967 INFO:tasks.workunit.client.0.smithi136.stdout: "truncate_seq": 0, 2024-09-15T22:02:17.967 INFO:tasks.workunit.client.0.smithi136.stdout: "truncate_size": 0, 2024-09-15T22:02:17.967 INFO:tasks.workunit.client.0.smithi136.stdout: "data_digest": "0x29e45974", 2024-09-15T22:02:17.967 INFO:tasks.workunit.client.0.smithi136.stdout: "omap_digest": "0xffffffff", 2024-09-15T22:02:17.967 INFO:tasks.workunit.client.0.smithi136.stdout: "expected_object_size": 0, 2024-09-15T22:02:17.967 INFO:tasks.workunit.client.0.smithi136.stdout: "expected_write_size": 0, 2024-09-15T22:02:17.967 INFO:tasks.workunit.client.0.smithi136.stdout: "alloc_hint_flags": 0, 2024-09-15T22:02:17.967 INFO:tasks.workunit.client.0.smithi136.stdout: "manifest": { 2024-09-15T22:02:17.967 INFO:tasks.workunit.client.0.smithi136.stdout: "type": 0 2024-09-15T22:02:17.967 INFO:tasks.workunit.client.0.smithi136.stdout: }, 2024-09-15T22:02:17.968 INFO:tasks.workunit.client.0.smithi136.stdout: "watchers": {} 2024-09-15T22:02:17.968 INFO:tasks.workunit.client.0.smithi136.stdout: }, 2024-09-15T22:02:17.968 INFO:tasks.workunit.client.0.smithi136.stdout: "shards": [ 2024-09-15T22:02:17.968 INFO:tasks.workunit.client.0.smithi136.stdout: { 2024-09-15T22:02:17.968 INFO:tasks.workunit.client.0.smithi136.stdout: "osd": 0, 2024-09-15T22:02:17.968 INFO:tasks.workunit.client.0.smithi136.stdout: "primary": false, 2024-09-15T22:02:17.968 INFO:tasks.workunit.client.0.smithi136.stdout: "errors": [], 2024-09-15T22:02:17.968 INFO:tasks.workunit.client.0.smithi136.stdout: "size": 256, 2024-09-15T22:02:17.968 INFO:tasks.workunit.client.0.smithi136.stdout: "snapset": { 2024-09-15T22:02:17.968 INFO:tasks.workunit.client.0.smithi136.stdout: "seq": 1, 2024-09-15T22:02:17.969 INFO:tasks.workunit.client.0.smithi136.stdout: "clones": [ 2024-09-15T22:02:17.969 INFO:tasks.workunit.client.0.smithi136.stdout: { 2024-09-15T22:02:17.969 INFO:tasks.workunit.client.0.smithi136.stdout: "snap": 1, 2024-09-15T22:02:17.969 INFO:tasks.workunit.client.0.smithi136.stdout: "size": 1032, 2024-09-15T22:02:17.969 INFO:tasks.workunit.client.0.smithi136.stdout: "overlap": "[]", 2024-09-15T22:02:17.969 INFO:tasks.workunit.client.0.smithi136.stdout: "snaps": [ 2024-09-15T22:02:17.969 INFO:tasks.workunit.client.0.smithi136.stdout: 1 2024-09-15T22:02:17.969 INFO:tasks.workunit.client.0.smithi136.stdout: ] 2024-09-15T22:02:17.969 INFO:tasks.workunit.client.0.smithi136.stdout: } 2024-09-15T22:02:17.969 INFO:tasks.workunit.client.0.smithi136.stdout: ] 2024-09-15T22:02:17.969 INFO:tasks.workunit.client.0.smithi136.stdout: } 2024-09-15T22:02:17.970 INFO:tasks.workunit.client.0.smithi136.stdout: }, 2024-09-15T22:02:17.970 INFO:tasks.workunit.client.0.smithi136.stdout: { 2024-09-15T22:02:17.970 INFO:tasks.workunit.client.0.smithi136.stdout: "osd": 1, 2024-09-15T22:02:17.970 INFO:tasks.workunit.client.0.smithi136.stdout: "primary": true, 2024-09-15T22:02:17.970 INFO:tasks.workunit.client.0.smithi136.stdout: "errors": [], 2024-09-15T22:02:17.970 INFO:tasks.workunit.client.0.smithi136.stdout: "size": 256, 2024-09-15T22:02:17.970 INFO:tasks.workunit.client.0.smithi136.stdout: "snapset": { 2024-09-15T22:02:17.970 INFO:tasks.workunit.client.0.smithi136.stdout: "seq": 1, 2024-09-15T22:02:17.970 INFO:tasks.workunit.client.0.smithi136.stdout: "clones": [ 2024-09-15T22:02:17.970 INFO:tasks.workunit.client.0.smithi136.stdout: { 2024-09-15T22:02:17.970 INFO:tasks.workunit.client.0.smithi136.stdout: "snap": 1, 2024-09-15T22:02:17.970 INFO:tasks.workunit.client.0.smithi136.stdout: "size": "????", 2024-09-15T22:02:17.970 INFO:tasks.workunit.client.0.smithi136.stdout: "overlap": "[]", 2024-09-15T22:02:17.970 INFO:tasks.workunit.client.0.smithi136.stdout: "snaps": [ 2024-09-15T22:02:17.971 INFO:tasks.workunit.client.0.smithi136.stdout: 1 2024-09-15T22:02:17.971 INFO:tasks.workunit.client.0.smithi136.stdout: ] 2024-09-15T22:02:17.971 INFO:tasks.workunit.client.0.smithi136.stdout: } 2024-09-15T22:02:17.971 INFO:tasks.workunit.client.0.smithi136.stdout: ] 2024-09-15T22:02:17.971 INFO:tasks.workunit.client.0.smithi136.stdout: } 2024-09-15T22:02:17.971 INFO:tasks.workunit.client.0.smithi136.stdout: } 2024-09-15T22:02:17.971 INFO:tasks.workunit.client.0.smithi136.stdout: ] 2024-09-15T22:02:17.971 INFO:tasks.workunit.client.0.smithi136.stdout: } 2024-09-15T22:02:17.971 INFO:tasks.workunit.client.0.smithi136.stdout: ] 2024-09-15T22:02:17.971 INFO:tasks.workunit.client.0.smithi136.stdout:} 2024-09-15T22:02:17.973 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:797: _scrub_snaps_multi: rados list-inconsistent-snapset 1.0 2024-09-15T22:02:17.974 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:800: _scrub_snaps_multi: '[' primary = replica ']' 2024-09-15T22:02:17.974 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:811: _scrub_snaps_multi: scruberrors=30 2024-09-15T22:02:17.974 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:812: _scrub_snaps_multi: jq .inconsistents 2024-09-15T22:02:17.974 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:812: _scrub_snaps_multi: python3 -c 'import json; import sys ; JSON=sys.stdin.read() ; ud = json.loads(JSON) ; print ( json.dumps(ud, sort_keys=True, indent=2) )' 2024-09-15T22:02:17.974 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1066: _scrub_snaps_multi: jq .inconsistents td/osd-scrub-snaps/json 2024-09-15T22:02:17.974 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1066: _scrub_snaps_multi: python3 -c 'import json; import sys ; JSON=sys.stdin.read() ; ud = json.loads(JSON) ; print ( json.dumps(ud, sort_keys=True, indent=2) )' 2024-09-15T22:02:17.974 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1067: _scrub_snaps_multi: multidiff td/osd-scrub-snaps/checkcsjson td/osd-scrub-snaps/csjson 2024-09-15T22:02:17.974 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2434: multidiff: diff td/osd-scrub-snaps/checkcsjson td/osd-scrub-snaps/csjson 2024-09-15T22:02:17.974 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1068: _scrub_snaps_multi: test no = yes 2024-09-15T22:02:17.974 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1073: _scrub_snaps_multi: test '' = yes 2024-09-15T22:02:17.974 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1078: _scrub_snaps_multi: find td/osd-scrub-snaps 2024-09-15T22:02:17.974 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1078: _scrub_snaps_multi: grep 'osd[^/]*\.pid' 2024-09-15T22:02:17.974 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1078: _scrub_snaps_multi: pidfiles='td/osd-scrub-snaps/osd.1.pid 2024-09-15T22:02:17.975 INFO:tasks.workunit.client.0.smithi136.stderr:td/osd-scrub-snaps/osd.0.pid' 2024-09-15T22:02:17.975 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1079: _scrub_snaps_multi: pids= 2024-09-15T22:02:17.975 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1080: _scrub_snaps_multi: for pidfile in ${pidfiles} 2024-09-15T22:02:17.975 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1082: _scrub_snaps_multi: cat td/osd-scrub-snaps/osd.1.pid 2024-09-15T22:02:17.975 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1082: _scrub_snaps_multi: pids+='498545 ' 2024-09-15T22:02:17.975 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1080: _scrub_snaps_multi: for pidfile in ${pidfiles} 2024-09-15T22:02:17.975 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1082: _scrub_snaps_multi: cat td/osd-scrub-snaps/osd.0.pid 2024-09-15T22:02:17.975 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1082: _scrub_snaps_multi: pids+='497949 ' 2024-09-15T22:02:17.975 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1085: _scrub_snaps_multi: ERRORS=0 2024-09-15T22:02:17.975 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1089: _scrub_snaps_multi: '[' primary = primary ']' 2024-09-15T22:02:17.975 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1091: _scrub_snaps_multi: seq 1 7 2024-09-15T22:02:17.975 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1091: _scrub_snaps_multi: for i in `seq 1 7` 2024-09-15T22:02:17.975 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1093: _scrub_snaps_multi: rados -p test rmsnap snap1 2024-09-15T22:02:18.134 INFO:tasks.workunit.client.0.smithi136.stdout:removed pool test snap snap1 2024-09-15T22:02:18.136 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1091: _scrub_snaps_multi: for i in `seq 1 7` 2024-09-15T22:02:18.136 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1093: _scrub_snaps_multi: rados -p test rmsnap snap2 2024-09-15T22:02:18.376 INFO:tasks.workunit.client.0.smithi136.stdout:removed pool test snap snap2 2024-09-15T22:02:18.378 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1091: _scrub_snaps_multi: for i in `seq 1 7` 2024-09-15T22:02:18.379 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1093: _scrub_snaps_multi: rados -p test rmsnap snap3 2024-09-15T22:02:18.609 INFO:tasks.workunit.client.0.smithi136.stdout:removed pool test snap snap3 2024-09-15T22:02:18.611 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1091: _scrub_snaps_multi: for i in `seq 1 7` 2024-09-15T22:02:18.611 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1093: _scrub_snaps_multi: rados -p test rmsnap snap4 2024-09-15T22:02:18.810 INFO:tasks.workunit.client.0.smithi136.stdout:removed pool test snap snap4 2024-09-15T22:02:18.812 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1091: _scrub_snaps_multi: for i in `seq 1 7` 2024-09-15T22:02:18.812 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1093: _scrub_snaps_multi: rados -p test rmsnap snap5 2024-09-15T22:02:18.994 INFO:tasks.workunit.client.0.smithi136.stdout:removed pool test snap snap5 2024-09-15T22:02:18.996 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1091: _scrub_snaps_multi: for i in `seq 1 7` 2024-09-15T22:02:18.997 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1093: _scrub_snaps_multi: rados -p test rmsnap snap6 2024-09-15T22:02:19.297 INFO:tasks.workunit.client.0.smithi136.stdout:removed pool test snap snap6 2024-09-15T22:02:19.299 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1091: _scrub_snaps_multi: for i in `seq 1 7` 2024-09-15T22:02:19.300 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1093: _scrub_snaps_multi: rados -p test rmsnap snap7 2024-09-15T22:02:19.480 INFO:tasks.workunit.client.0.smithi136.stdout:removed pool test snap snap7 2024-09-15T22:02:19.482 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1095: _scrub_snaps_multi: sleep 5 2024-09-15T22:02:24.483 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1096: _scrub_snaps_multi: local -i loop=0 2024-09-15T22:02:24.484 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1097: _scrub_snaps_multi: ceph pg dump pgs 2024-09-15T22:02:24.484 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1097: _scrub_snaps_multi: grep -q snaptrim 2024-09-15T22:02:24.766 INFO:tasks.workunit.client.0.smithi136.stderr:dumped pgs 2024-09-15T22:02:24.778 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1099: _scrub_snaps_multi: ceph pg dump pgs 2024-09-15T22:02:24.778 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1099: _scrub_snaps_multi: grep -q snaptrim_error 2024-09-15T22:02:25.050 INFO:tasks.workunit.client.0.smithi136.stderr:dumped pgs 2024-09-15T22:02:25.061 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1101: _scrub_snaps_multi: break 2024-09-15T22:02:25.061 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1111: _scrub_snaps_multi: ceph pg dump pgs 2024-09-15T22:02:25.332 INFO:tasks.workunit.client.0.smithi136.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-15T22:02:25.333 INFO:tasks.workunit.client.0.smithi136.stdout:1.0 36 0 0 0 0 24448 0 0 56 0 56 active+clean+inconsistent+snaptrim_error 2024-09-15T22:02:19.504839+0000 22'56 36:132 [1,0] 1 [1,0] 1 22'56 2024-09-15T22:02:13.145815+0000 0'0 2024-09-15T22:00:55.865337+0000 7 1 periodic scrub scheduled @ 2024-09-17T06:08:11.571458+0000 33 0 2024-09-15T22:02:25.333 INFO:tasks.workunit.client.0.smithi136.stdout: 2024-09-15T22:02:25.333 INFO:tasks.workunit.client.0.smithi136.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-15T22:02:25.333 INFO:tasks.workunit.client.0.smithi136.stderr:dumped pgs 2024-09-15T22:02:25.344 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1113: _scrub_snaps_multi: for pid in $pids 2024-09-15T22:02:25.344 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1115: _scrub_snaps_multi: kill -0 498545 2024-09-15T22:02:25.344 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1113: _scrub_snaps_multi: for pid in $pids 2024-09-15T22:02:25.345 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1115: _scrub_snaps_multi: kill -0 497949 2024-09-15T22:02:25.345 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1122: _scrub_snaps_multi: kill_daemons td/osd-scrub-snaps 2024-09-15T22:02:25.345 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-15T22:02:25.345 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-15T22:02:25.345 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-15T22:02:25.345 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-15T22:02:25.345 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-15T22:02:30.658 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-15T22:02:30.658 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1124: _scrub_snaps_multi: declare -a err_strings 2024-09-15T22:02:30.658 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1125: _scrub_snaps_multi: err_strings[0]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard [0-1] .*:::obj4:7 : missing' 2024-09-15T22:02:30.658 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1126: _scrub_snaps_multi: err_strings[1]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard [0-1] soid .*:::obj3:head : size 3840 != size 768 from auth oi' 2024-09-15T22:02:30.659 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1127: _scrub_snaps_multi: err_strings[2]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard [0-1] .*:::obj5:1 : missing' 2024-09-15T22:02:30.659 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1128: _scrub_snaps_multi: err_strings[3]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard [0-1] .*:::obj5:2 : missing' 2024-09-15T22:02:30.659 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1129: _scrub_snaps_multi: err_strings[4]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard [0-1] soid .*:::obj5:4 : size 4608 != size 512 from auth oi' 2024-09-15T22:02:30.659 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1130: _scrub_snaps_multi: err_strings[5]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid .*:::obj5:7 : failed to pick suitable object info' 2024-09-15T22:02:30.659 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1131: _scrub_snaps_multi: err_strings[6]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard [0-1] .*:::obj1:head : missing' 2024-09-15T22:02:30.659 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1132: _scrub_snaps_multi: err_strings[7]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 scrub 30 errors' 2024-09-15T22:02:30.659 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1134: _scrub_snaps_multi: for err_string in "${err_strings[@]}" 2024-09-15T22:02:30.659 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1136: _scrub_snaps_multi: grep 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard [0-1] .*:::obj4:7 : missing' td/osd-scrub-snaps/osd.1.log 2024-09-15T22:02:30.660 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1134: _scrub_snaps_multi: for err_string in "${err_strings[@]}" 2024-09-15T22:02:30.660 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1136: _scrub_snaps_multi: grep 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard [0-1] soid .*:::obj3:head : size 3840 != size 768 from auth oi' td/osd-scrub-snaps/osd.1.log 2024-09-15T22:02:30.663 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1134: _scrub_snaps_multi: for err_string in "${err_strings[@]}" 2024-09-15T22:02:30.663 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1136: _scrub_snaps_multi: grep 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard [0-1] .*:::obj5:1 : missing' td/osd-scrub-snaps/osd.1.log 2024-09-15T22:02:30.666 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1134: _scrub_snaps_multi: for err_string in "${err_strings[@]}" 2024-09-15T22:02:30.666 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1136: _scrub_snaps_multi: grep 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard [0-1] .*:::obj5:2 : missing' td/osd-scrub-snaps/osd.1.log 2024-09-15T22:02:30.668 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1134: _scrub_snaps_multi: for err_string in "${err_strings[@]}" 2024-09-15T22:02:30.668 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1136: _scrub_snaps_multi: grep 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard [0-1] soid .*:::obj5:4 : size 4608 != size 512 from auth oi' td/osd-scrub-snaps/osd.1.log 2024-09-15T22:02:30.671 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1134: _scrub_snaps_multi: for err_string in "${err_strings[@]}" 2024-09-15T22:02:30.671 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1136: _scrub_snaps_multi: grep 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid .*:::obj5:7 : failed to pick suitable object info' td/osd-scrub-snaps/osd.1.log 2024-09-15T22:02:30.674 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1134: _scrub_snaps_multi: for err_string in "${err_strings[@]}" 2024-09-15T22:02:30.674 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1136: _scrub_snaps_multi: grep 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard [0-1] .*:::obj1:head : missing' td/osd-scrub-snaps/osd.1.log 2024-09-15T22:02:30.676 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1134: _scrub_snaps_multi: for err_string in "${err_strings[@]}" 2024-09-15T22:02:30.677 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1136: _scrub_snaps_multi: grep 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 scrub 30 errors' td/osd-scrub-snaps/osd.1.log 2024-09-15T22:02:30.679 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1144: _scrub_snaps_multi: declare -a rep_err_strings 2024-09-15T22:02:30.680 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1145: _scrub_snaps_multi: eval echo '$primary' 2024-09-15T22:02:30.680 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1145: _scrub_snaps_multi: echo 1 2024-09-15T22:02:30.680 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1145: _scrub_snaps_multi: osd=1 2024-09-15T22:02:30.680 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1146: _scrub_snaps_multi: rep_err_strings[0]='log_channel[(]cluster[)] log [[]ERR[]] : osd[.][0-9]* found snap mapper error on pg 1.0 oid 1:461f8b5e:::obj16:7 snaps missing in mapper, should be: {1, 2, 3, 4, 5, 6, 7} ...repaired' 2024-09-15T22:02:30.680 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1147: _scrub_snaps_multi: for err_string in "${rep_err_strings[@]}" 2024-09-15T22:02:30.681 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1149: _scrub_snaps_multi: grep 'log_channel[(]cluster[)] log [[]ERR[]] : osd[.][0-9]* found snap mapper error on pg 1.0 oid 1:461f8b5e:::obj16:7 snaps missing in mapper, should be: {1, 2, 3, 4, 5, 6, 7} ...repaired' td/osd-scrub-snaps/osd.1.log 2024-09-15T22:02:30.682 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1156: _scrub_snaps_multi: '[' 0 '!=' 0 ']' 2024-09-15T22:02:30.683 INFO:tasks.workunit.client.0.smithi136.stdout:TEST PASSED 2024-09-15T22:02:30.684 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1162: _scrub_snaps_multi: echo 'TEST PASSED' 2024-09-15T22:02:30.684 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1163: _scrub_snaps_multi: return 0 2024-09-15T22:02:30.684 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1181: TEST_scrub_snaps_primary: err=0 2024-09-15T22:02:30.684 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1182: TEST_scrub_snaps_primary: CEPH_ARGS='--fsid=0098e432-2807-49bf-8c16-1e2a347d825d --auth-supported=none --mon-host=127.0.0.1:7121 ' 2024-09-15T22:02:30.684 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1183: TEST_scrub_snaps_primary: return 0 2024-09-15T22:02:30.684 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:40: run: teardown td/osd-scrub-snaps 2024-09-15T22:02:30.685 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/osd-scrub-snaps 2024-09-15T22:02:30.685 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-15T22:02:30.685 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/osd-scrub-snaps KILL 2024-09-15T22:02:30.685 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-15T22:02:30.685 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-15T22:02:30.685 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-15T22:02:30.685 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-15T22:02:30.685 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-15T22:02:30.689 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-15T22:02:30.689 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-15T22:02:30.690 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-15T22:02:30.690 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-15T22:02:30.691 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-15T22:02:30.692 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-15T22:02:30.692 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-15T22:02:30.693 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T22:02:30.693 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-15T22:02:30.693 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-15T22:02:30.693 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T22:02:30.694 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-15T22:02:30.695 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-15T22:02:30.695 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/osd-scrub-snaps 2024-09-15T22:02:30.711 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-15T22:02:30.711 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T22:02:30.711 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.463062 2024-09-15T22:02:30.711 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.463062 2024-09-15T22:02:30.713 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-15T22:02:30.713 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-15T22:02:30.713 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:37: run: for func in $funcs 2024-09-15T22:02:30.713 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:38: run: setup td/osd-scrub-snaps 2024-09-15T22:02:30.713 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/osd-scrub-snaps 2024-09-15T22:02:30.713 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/osd-scrub-snaps 2024-09-15T22:02:30.713 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/osd-scrub-snaps 2024-09-15T22:02:30.713 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-15T22:02:30.713 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/osd-scrub-snaps KILL 2024-09-15T22:02:30.713 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-15T22:02:30.714 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-15T22:02:30.714 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-15T22:02:30.714 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-15T22:02:30.714 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-15T22:02:30.715 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-15T22:02:30.715 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-15T22:02:30.716 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-15T22:02:30.716 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-15T22:02:30.717 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-15T22:02:30.717 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-15T22:02:30.718 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-15T22:02:30.718 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T22:02:30.719 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-15T22:02:30.719 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-15T22:02:30.719 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T22:02:30.720 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-15T22:02:30.721 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-15T22:02:30.721 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/osd-scrub-snaps 2024-09-15T22:02:30.722 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-15T22:02:30.722 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T22:02:30.722 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.463062 2024-09-15T22:02:30.722 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.463062 2024-09-15T22:02:30.723 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-15T22:02:30.723 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-15T22:02:30.723 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/osd-scrub-snaps 2024-09-15T22:02:30.725 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2024-09-15T22:02:30.725 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T22:02:30.725 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.463062 2024-09-15T22:02:30.725 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.463062 2024-09-15T22:02:30.726 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2024-09-15T22:02:30.728 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 4096 -le 1024 ']' 2024-09-15T22:02:30.728 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2024-09-15T22:02:30.728 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/osd-scrub-snaps 1' TERM HUP INT 2024-09-15T22:02:30.728 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:39: run: TEST_scrub_snaps_replica td/osd-scrub-snaps 2024-09-15T22:02:30.728 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1167: TEST_scrub_snaps_replica: local dir=td/osd-scrub-snaps 2024-09-15T22:02:30.728 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1168: TEST_scrub_snaps_replica: ORIG_ARGS='--fsid=0098e432-2807-49bf-8c16-1e2a347d825d --auth-supported=none --mon-host=127.0.0.1:7121 ' 2024-09-15T22:02:30.729 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1169: TEST_scrub_snaps_replica: CEPH_ARGS+=' --osd_scrub_chunk_min=3 --osd_scrub_chunk_max=20 --osd_shallow_scrub_chunk_min=3 --osd_shallow_scrub_chunk_max=3 --osd_pg_stat_report_interval_max_seconds=1 --osd_pg_stat_report_interval_max_epochs=1' 2024-09-15T22:02:30.729 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1170: TEST_scrub_snaps_replica: _scrub_snaps_multi td/osd-scrub-snaps replica 2024-09-15T22:02:30.729 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:732: _scrub_snaps_multi: local dir=td/osd-scrub-snaps 2024-09-15T22:02:30.729 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:733: _scrub_snaps_multi: local poolname=test 2024-09-15T22:02:30.729 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:734: _scrub_snaps_multi: local OBJS=16 2024-09-15T22:02:30.729 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:735: _scrub_snaps_multi: local OSDS=2 2024-09-15T22:02:30.729 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:736: _scrub_snaps_multi: local which=replica 2024-09-15T22:02:30.729 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:738: _scrub_snaps_multi: TESTDATA=testdata.463062 2024-09-15T22:02:30.729 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:740: _scrub_snaps_multi: run_mon td/osd-scrub-snaps a --osd_pool_default_size=2 2024-09-15T22:02:30.729 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/osd-scrub-snaps 2024-09-15T22:02:30.729 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2024-09-15T22:02:30.729 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2024-09-15T22:02:30.729 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2024-09-15T22:02:30.729 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/osd-scrub-snaps/a 2024-09-15T22:02:30.730 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-scrub-snaps/a --run-dir=td/osd-scrub-snaps --osd_pool_default_size=2 2024-09-15T22:02:31.182 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2024-09-15T22:02:31.183 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-15T22:02:31.183 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-15T22:02:31.183 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-15T22:02:31.183 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T22:02:31.183 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.463062 2024-09-15T22:02:31.183 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.463062/$cluster-$name.asok' 2024-09-15T22:02:31.183 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-scrub-snaps/a '--log-file=td/osd-scrub-snaps/$name.log' '--admin-socket=/tmp/ceph-asok.463062/$cluster-$name.asok' --mon-cluster-log-file=td/osd-scrub-snaps/log --run-dir=td/osd-scrub-snaps '--pid-file=td/osd-scrub-snaps/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=2 2024-09-15T22:02:31.383 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2024-09-15T22:02:31.384 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2024-09-15T22:02:31.384 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-15T22:02:31.384 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-15T22:02:31.384 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2024-09-15T22:02:31.385 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2024-09-15T22:02:31.386 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-15T22:02:31.386 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-15T22:02:31.386 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-15T22:02:31.386 INFO:tasks.workunit.client.0.smithi136.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-15T22:02:31.386 INFO:tasks.workunit.client.0.smithi136.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T22:02:31.386 INFO:tasks.workunit.client.0.smithi136.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.463062 2024-09-15T22:02:31.386 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.463062/ceph-mon.a.asok 2024-09-15T22:02:31.386 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-15T22:02:31.386 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.463062/ceph-mon.a.asok config get fsid 2024-09-15T22:02:31.471 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2024-09-15T22:02:31.471 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-15T22:02:31.472 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-15T22:02:31.472 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2024-09-15T22:02:31.472 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2024-09-15T22:02:31.472 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-15T22:02:31.472 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-15T22:02:31.472 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-15T22:02:31.472 INFO:tasks.workunit.client.0.smithi136.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-15T22:02:31.472 INFO:tasks.workunit.client.0.smithi136.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T22:02:31.473 INFO:tasks.workunit.client.0.smithi136.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.463062 2024-09-15T22:02:31.473 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.463062/ceph-mon.a.asok 2024-09-15T22:02:31.473 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-15T22:02:31.473 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.463062/ceph-mon.a.asok config get mon_host 2024-09-15T22:02:31.550 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:741: _scrub_snaps_multi: run_mgr td/osd-scrub-snaps x 2024-09-15T22:02:31.551 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: run_mgr: local dir=td/osd-scrub-snaps 2024-09-15T22:02:31.551 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: run_mgr: shift 2024-09-15T22:02:31.552 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: run_mgr: local id=x 2024-09-15T22:02:31.552 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: run_mgr: shift 2024-09-15T22:02:31.552 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:548: run_mgr: local data=td/osd-scrub-snaps/x 2024-09-15T22:02:31.552 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:550: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-15T22:02:32.367 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: get_asok_path 2024-09-15T22:02:32.367 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-15T22:02:32.367 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-15T22:02:32.367 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-15T22:02:32.367 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T22:02:32.367 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.463062 2024-09-15T22:02:32.367 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.463062/$cluster-$name.asok' 2024-09-15T22:02:32.368 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-15T22:02:32.368 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-scrub-snaps/x '--log-file=td/osd-scrub-snaps/$name.log' '--admin-socket=/tmp/ceph-asok.463062/$cluster-$name.asok' --run-dir=td/osd-scrub-snaps '--pid-file=td/osd-scrub-snaps/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-15T22:02:32.405 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:742: _scrub_snaps_multi: expr 2 - 1 2024-09-15T22:02:32.406 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:742: _scrub_snaps_multi: seq 0 1 2024-09-15T22:02:32.412 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:742: _scrub_snaps_multi: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-09-15T22:02:32.412 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:744: _scrub_snaps_multi: run_osd td/osd-scrub-snaps 0 2024-09-15T22:02:32.412 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-scrub-snaps 2024-09-15T22:02:32.412 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-15T22:02:32.412 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=0 2024-09-15T22:02:32.412 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-15T22:02:32.412 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-scrub-snaps/0 2024-09-15T22:02:32.412 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=0098e432-2807-49bf-8c16-1e2a347d825d --auth-supported=none --mon-host=127.0.0.1:7121 --osd_scrub_chunk_min=3 --osd_scrub_chunk_max=20 --osd_shallow_scrub_chunk_min=3 --osd_shallow_scrub_chunk_max=3 --osd_pg_stat_report_interval_max_seconds=1 --osd_pg_stat_report_interval_max_epochs=1' 2024-09-15T22:02:32.413 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T22:02:32.413 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T22:02:32.413 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T22:02:32.413 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-scrub-snaps/0' 2024-09-15T22:02:32.413 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-snaps/0/journal' 2024-09-15T22:02:32.413 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-15T22:02:32.413 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-15T22:02:32.413 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-scrub-snaps' 2024-09-15T22:02:32.413 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-15T22:02:32.413 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-15T22:02:32.413 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-15T22:02:32.413 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-15T22:02:32.413 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T22:02:32.414 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.463062 2024-09-15T22:02:32.414 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.463062/$cluster-$name.asok' 2024-09-15T22:02:32.414 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.463062/$cluster-$name.asok' 2024-09-15T22:02:32.414 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-15T22:02:32.414 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-15T22:02:32.414 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-15T22:02:32.414 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-scrub-snaps/$name.log' 2024-09-15T22:02:32.414 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-scrub-snaps/$name.pid' 2024-09-15T22:02:32.414 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T22:02:32.414 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T22:02:32.414 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T22:02:32.414 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T22:02:32.414 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-15T22:02:32.414 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-15T22:02:32.415 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-scrub-snaps/0 2024-09-15T22:02:32.415 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-15T22:02:32.415 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=396cb73c-0737-430f-add9-356620de3f92 2024-09-15T22:02:32.415 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd0 396cb73c-0737-430f-add9-356620de3f92' 2024-09-15T22:02:32.415 INFO:tasks.workunit.client.0.smithi136.stdout:add osd0 396cb73c-0737-430f-add9-356620de3f92 2024-09-15T22:02:32.415 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-15T22:02:32.437 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQB4Wedm3uf1GRAAU1LPz+gYAn3f/s9MFGlkwQ== 2024-09-15T22:02:32.437 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQB4Wedm3uf1GRAAU1LPz+gYAn3f/s9MFGlkwQ=="}' 2024-09-15T22:02:32.437 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 396cb73c-0737-430f-add9-356620de3f92 -i td/osd-scrub-snaps/0/new.json 2024-09-15T22:02:32.682 INFO:tasks.workunit.client.0.smithi136.stdout:0 2024-09-15T22:02:32.692 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-scrub-snaps/0/new.json 2024-09-15T22:02:32.693 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 0 --fsid=0098e432-2807-49bf-8c16-1e2a347d825d --auth-supported=none --mon-host=127.0.0.1:7121 --osd_scrub_chunk_min=3 --osd_scrub_chunk_max=20 --osd_shallow_scrub_chunk_min=3 --osd_shallow_scrub_chunk_max=3 --osd_pg_stat_report_interval_max_seconds=1 --osd_pg_stat_report_interval_max_epochs=1 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-snaps/0 --osd-journal=td/osd-scrub-snaps/0/journal --chdir= --run-dir=td/osd-scrub-snaps '--admin-socket=/tmp/ceph-asok.463062/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-snaps/$name.log' '--pid-file=td/osd-scrub-snaps/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQB4Wedm3uf1GRAAU1LPz+gYAn3f/s9MFGlkwQ== --osd-uuid 396cb73c-0737-430f-add9-356620de3f92 2024-09-15T22:02:32.734 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T22:02:32.731+0000 7f13d74ef640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T22:02:32.735 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T22:02:32.733+0000 7f13d74ef640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T22:02:32.736 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T22:02:32.734+0000 7f13d74ef640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T22:02:32.737 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T22:02:32.734+0000 7f13d74ef640 -1 bdev(0x558feb623c00 td/osd-scrub-snaps/0/block) open stat got: (1) Operation not permitted 2024-09-15T22:02:32.737 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T22:02:32.735+0000 7f13d74ef640 -1 bluestore(td/osd-scrub-snaps/0) _read_fsid unparsable uuid 2024-09-15T22:02:38.458 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-scrub-snaps/0/keyring 2024-09-15T22:02:38.458 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-15T22:02:38.459 INFO:tasks.workunit.client.0.smithi136.stdout:adding osd0 key to auth repository 2024-09-15T22:02:38.459 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd0 key to auth repository 2024-09-15T22:02:38.460 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-scrub-snaps/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-15T22:02:38.843 INFO:tasks.workunit.client.0.smithi136.stdout:start osd.0 2024-09-15T22:02:38.844 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.0 2024-09-15T22:02:38.844 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 0 --fsid=0098e432-2807-49bf-8c16-1e2a347d825d --auth-supported=none --mon-host=127.0.0.1:7121 --osd_scrub_chunk_min=3 --osd_scrub_chunk_max=20 --osd_shallow_scrub_chunk_min=3 --osd_shallow_scrub_chunk_max=3 --osd_pg_stat_report_interval_max_seconds=1 --osd_pg_stat_report_interval_max_epochs=1 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-snaps/0 --osd-journal=td/osd-scrub-snaps/0/journal --chdir= --run-dir=td/osd-scrub-snaps '--admin-socket=/tmp/ceph-asok.463062/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-snaps/$name.log' '--pid-file=td/osd-scrub-snaps/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T22:02:38.844 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-15T22:02:38.844 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-15T22:02:38.844 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-15T22:02:38.882 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T22:02:38.880+0000 7f1f6d61e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T22:02:38.906 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T22:02:38.904+0000 7f1f6d61e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T22:02:38.907 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T22:02:38.905+0000 7f1f6d61e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T22:02:39.164 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 0 2024-09-15T22:02:39.164 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-15T22:02:39.164 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2024-09-15T22:02:39.164 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-15T22:02:39.165 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-15T22:02:39.165 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:02:39.165 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-15T22:02:39.165 INFO:tasks.workunit.client.0.smithi136.stdout:0 2024-09-15T22:02:39.165 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:02:39.166 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T22:02:39.486 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:02:40.004 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T22:02:40.003+0000 7f1f6d61e640 -1 Falling back to public interface 2024-09-15T22:02:40.488 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:02:40.488 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:02:40.488 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-15T22:02:40.488 INFO:tasks.workunit.client.0.smithi136.stdout:1 2024-09-15T22:02:40.488 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:02:40.489 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T22:02:40.781 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:02:41.782 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:02:41.782 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:02:41.782 INFO:tasks.workunit.client.0.smithi136.stdout:2 2024-09-15T22:02:41.782 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-15T22:02:41.783 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:02:41.783 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T22:02:42.075 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:02:42.515 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T22:02:42.513+0000 7f1f6d61e640 -1 osd.0 0 log_to_monitors true 2024-09-15T22:02:43.076 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:02:43.076 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:02:43.077 INFO:tasks.workunit.client.0.smithi136.stdout:3 2024-09-15T22:02:43.078 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-15T22:02:43.078 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:02:43.078 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T22:02:43.363 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:02:44.364 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:02:44.364 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:02:44.364 INFO:tasks.workunit.client.0.smithi136.stdout:4 2024-09-15T22:02:44.365 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-15T22:02:44.365 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:02:44.365 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T22:02:44.660 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:02:45.662 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:02:45.662 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:02:45.662 INFO:tasks.workunit.client.0.smithi136.stdout:5 2024-09-15T22:02:45.663 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-15T22:02:45.663 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:02:45.663 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T22:02:45.962 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:02:46.963 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:02:46.963 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:02:46.963 INFO:tasks.workunit.client.0.smithi136.stdout:6 2024-09-15T22:02:46.964 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-15T22:02:46.964 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:02:46.964 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T22:02:47.311 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:02:48.312 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:02:48.312 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:02:48.313 INFO:tasks.workunit.client.0.smithi136.stdout:7 2024-09-15T22:02:48.313 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-15T22:02:48.313 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:02:48.313 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T22:02:48.612 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:02:49.613 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:02:49.614 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:02:49.614 INFO:tasks.workunit.client.0.smithi136.stdout:8 2024-09-15T22:02:49.614 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-15T22:02:49.614 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:02:49.614 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T22:02:49.912 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:02:50.913 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:02:50.913 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:02:50.914 INFO:tasks.workunit.client.0.smithi136.stdout:9 2024-09-15T22:02:50.914 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-15T22:02:50.914 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:02:50.914 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T22:02:51.328 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:02:52.329 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:02:52.329 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:02:52.329 INFO:tasks.workunit.client.0.smithi136.stdout:10 2024-09-15T22:02:52.330 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-15T22:02:52.330 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:02:52.330 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T22:02:52.695 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:02:53.696 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:02:53.696 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:02:53.697 INFO:tasks.workunit.client.0.smithi136.stdout:11 2024-09-15T22:02:53.697 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-15T22:02:53.697 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:02:53.697 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T22:02:53.996 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:02:54.997 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:02:54.997 INFO:tasks.workunit.client.0.smithi136.stdout:12 2024-09-15T22:02:54.998 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:02:54.998 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-15T22:02:54.998 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:02:54.998 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T22:02:55.787 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:02:56.789 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:02:56.789 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:02:56.789 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-15T22:02:56.789 INFO:tasks.workunit.client.0.smithi136.stdout:13 2024-09-15T22:02:56.789 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:02:56.790 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T22:02:57.515 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:02:58.517 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:02:58.517 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:02:58.517 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-15T22:02:58.517 INFO:tasks.workunit.client.0.smithi136.stdout:14 2024-09-15T22:02:58.518 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:02:58.518 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T22:02:59.507 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:03:00.509 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:03:00.509 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:03:00.509 INFO:tasks.workunit.client.0.smithi136.stdout:15 2024-09-15T22:03:00.509 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-15T22:03:00.510 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:03:00.510 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T22:03:01.268 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:03:02.270 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:03:02.270 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:03:02.270 INFO:tasks.workunit.client.0.smithi136.stdout:16 2024-09-15T22:03:02.271 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-15T22:03:02.271 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:03:02.271 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T22:03:02.745 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:03:03.747 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:03:03.747 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:03:03.747 INFO:tasks.workunit.client.0.smithi136.stdout:17 2024-09-15T22:03:03.748 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-15T22:03:03.748 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:03:03.748 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T22:03:04.473 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:03:05.474 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:03:05.474 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:03:05.475 INFO:tasks.workunit.client.0.smithi136.stdout:18 2024-09-15T22:03:05.475 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 18 2024-09-15T22:03:05.475 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:03:05.475 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T22:03:05.776 INFO:tasks.workunit.client.0.smithi136.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/3059087388,v1:127.0.0.1:6803/3059087388] [v2:127.0.0.1:6804/3059087388,v1:127.0.0.1:6805/3059087388] exists,up 396cb73c-0737-430f-add9-356620de3f92 2024-09-15T22:03:05.780 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-15T22:03:05.780 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-15T22:03:05.780 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-15T22:03:05.780 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:742: _scrub_snaps_multi: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-09-15T22:03:05.780 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:744: _scrub_snaps_multi: run_osd td/osd-scrub-snaps 1 2024-09-15T22:03:05.780 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-scrub-snaps 2024-09-15T22:03:05.780 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-15T22:03:05.781 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=1 2024-09-15T22:03:05.781 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-15T22:03:05.781 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-scrub-snaps/1 2024-09-15T22:03:05.781 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=0098e432-2807-49bf-8c16-1e2a347d825d --auth-supported=none --mon-host=127.0.0.1:7121 --osd_scrub_chunk_min=3 --osd_scrub_chunk_max=20 --osd_shallow_scrub_chunk_min=3 --osd_shallow_scrub_chunk_max=3 --osd_pg_stat_report_interval_max_seconds=1 --osd_pg_stat_report_interval_max_epochs=1' 2024-09-15T22:03:05.781 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T22:03:05.781 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T22:03:05.781 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T22:03:05.781 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-scrub-snaps/1' 2024-09-15T22:03:05.781 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-snaps/1/journal' 2024-09-15T22:03:05.781 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-15T22:03:05.781 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-15T22:03:05.781 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-scrub-snaps' 2024-09-15T22:03:05.781 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-15T22:03:05.781 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-15T22:03:05.782 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-15T22:03:05.782 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-15T22:03:05.782 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T22:03:05.782 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.463062 2024-09-15T22:03:05.782 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.463062/$cluster-$name.asok' 2024-09-15T22:03:05.782 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.463062/$cluster-$name.asok' 2024-09-15T22:03:05.782 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-15T22:03:05.782 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-15T22:03:05.782 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-15T22:03:05.782 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-scrub-snaps/$name.log' 2024-09-15T22:03:05.782 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-scrub-snaps/$name.pid' 2024-09-15T22:03:05.782 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T22:03:05.782 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T22:03:05.782 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T22:03:05.782 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T22:03:05.783 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-15T22:03:05.783 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-15T22:03:05.783 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-scrub-snaps/1 2024-09-15T22:03:05.783 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-15T22:03:05.783 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=7a834e93-3154-4b8d-b285-bcb744b7a4d2 2024-09-15T22:03:05.783 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd1 7a834e93-3154-4b8d-b285-bcb744b7a4d2' 2024-09-15T22:03:05.783 INFO:tasks.workunit.client.0.smithi136.stdout:add osd1 7a834e93-3154-4b8d-b285-bcb744b7a4d2 2024-09-15T22:03:05.783 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-15T22:03:05.807 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQCZWedmE7v9LxAAz2KPW3iaea4PJVwoL/aBdg== 2024-09-15T22:03:05.807 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQCZWedmE7v9LxAAz2KPW3iaea4PJVwoL/aBdg=="}' 2024-09-15T22:03:05.807 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 7a834e93-3154-4b8d-b285-bcb744b7a4d2 -i td/osd-scrub-snaps/1/new.json 2024-09-15T22:03:06.143 INFO:tasks.workunit.client.0.smithi136.stdout:1 2024-09-15T22:03:06.155 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-scrub-snaps/1/new.json 2024-09-15T22:03:06.156 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 1 --fsid=0098e432-2807-49bf-8c16-1e2a347d825d --auth-supported=none --mon-host=127.0.0.1:7121 --osd_scrub_chunk_min=3 --osd_scrub_chunk_max=20 --osd_shallow_scrub_chunk_min=3 --osd_shallow_scrub_chunk_max=3 --osd_pg_stat_report_interval_max_seconds=1 --osd_pg_stat_report_interval_max_epochs=1 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-snaps/1 --osd-journal=td/osd-scrub-snaps/1/journal --chdir= --run-dir=td/osd-scrub-snaps '--admin-socket=/tmp/ceph-asok.463062/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-snaps/$name.log' '--pid-file=td/osd-scrub-snaps/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCZWedmE7v9LxAAz2KPW3iaea4PJVwoL/aBdg== --osd-uuid 7a834e93-3154-4b8d-b285-bcb744b7a4d2 2024-09-15T22:03:06.195 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T22:03:06.194+0000 7f2a8c41c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T22:03:06.228 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T22:03:06.226+0000 7f2a8c41c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T22:03:06.229 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T22:03:06.227+0000 7f2a8c41c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T22:03:06.229 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T22:03:06.228+0000 7f2a8c41c640 -1 bdev(0x564b17525c00 td/osd-scrub-snaps/1/block) open stat got: (1) Operation not permitted 2024-09-15T22:03:06.229 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T22:03:06.228+0000 7f2a8c41c640 -1 bluestore(td/osd-scrub-snaps/1) _read_fsid unparsable uuid 2024-09-15T22:03:12.282 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-scrub-snaps/1/keyring 2024-09-15T22:03:12.282 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-15T22:03:12.283 INFO:tasks.workunit.client.0.smithi136.stdout:adding osd1 key to auth repository 2024-09-15T22:03:12.284 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd1 key to auth repository 2024-09-15T22:03:12.284 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-scrub-snaps/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-15T22:03:12.658 INFO:tasks.workunit.client.0.smithi136.stdout:start osd.1 2024-09-15T22:03:12.659 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.1 2024-09-15T22:03:12.659 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 1 --fsid=0098e432-2807-49bf-8c16-1e2a347d825d --auth-supported=none --mon-host=127.0.0.1:7121 --osd_scrub_chunk_min=3 --osd_scrub_chunk_max=20 --osd_shallow_scrub_chunk_min=3 --osd_shallow_scrub_chunk_max=3 --osd_pg_stat_report_interval_max_seconds=1 --osd_pg_stat_report_interval_max_epochs=1 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-snaps/1 --osd-journal=td/osd-scrub-snaps/1/journal --chdir= --run-dir=td/osd-scrub-snaps '--admin-socket=/tmp/ceph-asok.463062/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-snaps/$name.log' '--pid-file=td/osd-scrub-snaps/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T22:03:12.659 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-15T22:03:12.659 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-15T22:03:12.660 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-15T22:03:12.696 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T22:03:12.694+0000 7f417fa5e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T22:03:12.725 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T22:03:12.723+0000 7f417fa5e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T22:03:12.726 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T22:03:12.725+0000 7f417fa5e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T22:03:12.992 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 1 2024-09-15T22:03:12.992 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-15T22:03:12.992 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2024-09-15T22:03:12.992 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-15T22:03:12.992 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-15T22:03:12.993 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:03:12.993 INFO:tasks.workunit.client.0.smithi136.stdout:0 2024-09-15T22:03:12.993 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-15T22:03:12.993 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:03:12.993 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T22:03:13.317 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:03:14.319 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:03:14.319 INFO:tasks.workunit.client.0.smithi136.stdout:1 2024-09-15T22:03:14.320 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:03:14.320 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-15T22:03:14.320 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:03:14.320 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T22:03:14.349 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T22:03:14.348+0000 7f417fa5e640 -1 Falling back to public interface 2024-09-15T22:03:14.608 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:03:15.609 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:03:15.609 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:03:15.610 INFO:tasks.workunit.client.0.smithi136.stdout:2 2024-09-15T22:03:15.610 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-15T22:03:15.610 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:03:15.610 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T22:03:15.905 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:03:16.885 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T22:03:16.883+0000 7f417fa5e640 -1 osd.1 0 log_to_monitors true 2024-09-15T22:03:16.906 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:03:16.907 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:03:16.907 INFO:tasks.workunit.client.0.smithi136.stdout:3 2024-09-15T22:03:16.907 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-15T22:03:16.907 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:03:16.907 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T22:03:17.384 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:03:18.385 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:03:18.385 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:03:18.386 INFO:tasks.workunit.client.0.smithi136.stdout:4 2024-09-15T22:03:18.386 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-15T22:03:18.386 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:03:18.386 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T22:03:18.749 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:03:19.750 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:03:19.750 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:03:19.750 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-15T22:03:19.751 INFO:tasks.workunit.client.0.smithi136.stdout:5 2024-09-15T22:03:19.751 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:03:19.751 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T22:03:20.051 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:03:21.053 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:03:21.053 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:03:21.053 INFO:tasks.workunit.client.0.smithi136.stdout:6 2024-09-15T22:03:21.054 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-15T22:03:21.054 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:03:21.054 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T22:03:21.351 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:03:22.352 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:03:22.353 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:03:22.353 INFO:tasks.workunit.client.0.smithi136.stdout:7 2024-09-15T22:03:22.353 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-15T22:03:22.353 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:03:22.354 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T22:03:22.651 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:03:23.652 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:03:23.652 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:03:23.653 INFO:tasks.workunit.client.0.smithi136.stdout:8 2024-09-15T22:03:23.653 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-15T22:03:23.653 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:03:23.653 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T22:03:23.948 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:03:24.949 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:03:24.949 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:03:24.949 INFO:tasks.workunit.client.0.smithi136.stdout:9 2024-09-15T22:03:24.950 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-15T22:03:24.950 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:03:24.950 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T22:03:25.248 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:03:26.249 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:03:26.249 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:03:26.250 INFO:tasks.workunit.client.0.smithi136.stdout:10 2024-09-15T22:03:26.250 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-15T22:03:26.250 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:03:26.250 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T22:03:26.599 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:03:27.600 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:03:27.600 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:03:27.601 INFO:tasks.workunit.client.0.smithi136.stdout:11 2024-09-15T22:03:27.601 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-15T22:03:27.601 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:03:27.601 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T22:03:27.949 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:03:28.947 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:03:28.947 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:03:28.948 INFO:tasks.workunit.client.0.smithi136.stdout:12 2024-09-15T22:03:28.948 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-15T22:03:28.948 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:03:28.948 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T22:03:29.240 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:03:30.241 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:03:30.242 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:03:30.242 INFO:tasks.workunit.client.0.smithi136.stdout:13 2024-09-15T22:03:30.242 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-15T22:03:30.242 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:03:30.242 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T22:03:31.061 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:03:32.063 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:03:32.063 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:03:32.063 INFO:tasks.workunit.client.0.smithi136.stdout:14 2024-09-15T22:03:32.063 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-15T22:03:32.064 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:03:32.064 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T22:03:32.653 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:03:33.655 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:03:33.655 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:03:33.655 INFO:tasks.workunit.client.0.smithi136.stdout:15 2024-09-15T22:03:33.655 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-15T22:03:33.656 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:03:33.656 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T22:03:34.464 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:03:35.465 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:03:35.465 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:03:35.465 INFO:tasks.workunit.client.0.smithi136.stdout:16 2024-09-15T22:03:35.466 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-15T22:03:35.466 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:03:35.466 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T22:03:36.073 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:03:37.075 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:03:37.075 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:03:37.075 INFO:tasks.workunit.client.0.smithi136.stdout:17 2024-09-15T22:03:37.076 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-15T22:03:37.076 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:03:37.076 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T22:03:37.475 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:03:38.476 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:03:38.477 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:03:38.477 INFO:tasks.workunit.client.0.smithi136.stdout:18 2024-09-15T22:03:38.478 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 18 2024-09-15T22:03:38.478 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:03:38.478 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T22:03:38.860 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:03:39.861 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:03:39.861 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:03:39.862 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 19 2024-09-15T22:03:39.862 INFO:tasks.workunit.client.0.smithi136.stdout:19 2024-09-15T22:03:39.863 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:03:39.863 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T22:03:40.198 INFO:tasks.workunit.client.0.smithi136.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/4108867857,v1:127.0.0.1:6811/4108867857] [v2:127.0.0.1:6812/4108867857,v1:127.0.0.1:6813/4108867857] exists,up 7a834e93-3154-4b8d-b285-bcb744b7a4d2 2024-09-15T22:03:40.199 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-15T22:03:40.199 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-15T22:03:40.199 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-15T22:03:40.199 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:748: _scrub_snaps_multi: ceph osd set noscrub 2024-09-15T22:03:40.562 INFO:tasks.workunit.client.0.smithi136.stderr:noscrub is set 2024-09-15T22:03:40.574 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:749: _scrub_snaps_multi: ceph osd set nodeep-scrub 2024-09-15T22:03:41.148 INFO:tasks.workunit.client.0.smithi136.stderr:nodeep-scrub is set 2024-09-15T22:03:41.160 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:752: _scrub_snaps_multi: create_pool test 1 1 2024-09-15T22:03:41.160 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create test 1 1 2024-09-15T22:03:41.542 INFO:tasks.workunit.client.0.smithi136.stderr:pool 'test' created 2024-09-15T22:03:41.553 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2024-09-15T22:03:42.555 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:753: _scrub_snaps_multi: wait_for_clean 2024-09-15T22:03:42.555 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-15T22:03:42.555 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-15T22:03:42.555 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-15T22:03:42.555 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-15T22:03:42.556 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-15T22:03:42.556 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-15T22:03:42.556 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-15T22:03:42.556 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-15T22:03:42.556 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-15T22:03:42.653 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-15T22:03:42.653 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-15T22:03:42.654 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-15T22:03:42.654 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-15T22:03:42.654 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-15T22:03:42.654 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-15T22:03:42.945 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-15T22:03:42.945 INFO:tasks.workunit.client.0.smithi136.stderr:1' 2024-09-15T22:03:42.945 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-15T22:03:42.945 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T22:03:42.945 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-15T22:03:43.100 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836490 2024-09-15T22:03:43.100 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836490 2024-09-15T22:03:43.100 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836490' 2024-09-15T22:03:43.100 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T22:03:43.100 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-15T22:03:43.215 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=42949672965 2024-09-15T22:03:43.215 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 42949672965 2024-09-15T22:03:43.215 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836490 1-42949672965' 2024-09-15T22:03:43.216 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T22:03:43.216 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836490 2024-09-15T22:03:43.216 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T22:03:43.216 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-15T22:03:43.217 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836490 2024-09-15T22:03:43.217 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T22:03:43.218 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836490 2024-09-15T22:03:43.218 INFO:tasks.workunit.client.0.smithi136.stdout:waiting osd.0 seq 21474836490 2024-09-15T22:03:43.219 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836490' 2024-09-15T22:03:43.219 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T22:03:43.506 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836489 -lt 21474836490 2024-09-15T22:03:43.506 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-15T22:03:44.507 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-15T22:03:44.508 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T22:03:44.807 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836490 -lt 21474836490 2024-09-15T22:03:44.807 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T22:03:44.807 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-42949672965 2024-09-15T22:03:44.807 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T22:03:44.808 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-15T22:03:44.808 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-42949672965 2024-09-15T22:03:44.808 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T22:03:44.809 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=42949672965 2024-09-15T22:03:44.810 INFO:tasks.workunit.client.0.smithi136.stdout:waiting osd.1 seq 42949672965 2024-09-15T22:03:44.810 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 42949672965' 2024-09-15T22:03:44.810 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-15T22:03:45.135 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 42949672965 -lt 42949672965 2024-09-15T22:03:45.136 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-15T22:03:45.136 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T22:03:45.136 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T22:03:45.491 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 1 == 0 2024-09-15T22:03:45.491 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-15T22:03:45.491 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-15T22:03:45.491 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-15T22:03:45.491 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T22:03:45.491 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T22:03:45.492 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T22:03:45.492 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T22:03:45.816 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=1 2024-09-15T22:03:45.817 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-15T22:03:45.817 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T22:03:45.817 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T22:03:46.173 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 = 1 2024-09-15T22:03:46.173 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-15T22:03:46.173 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-15T22:03:46.173 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:754: _scrub_snaps_multi: ceph osd dump 2024-09-15T22:03:46.173 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:754: _scrub_snaps_multi: grep '^pool.*['\'']test['\'']' 2024-09-15T22:03:46.174 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:754: _scrub_snaps_multi: awk '{ print $2 }' 2024-09-15T22:03:46.498 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:754: _scrub_snaps_multi: poolid=1 2024-09-15T22:03:46.498 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:756: _scrub_snaps_multi: dd if=/dev/urandom of=testdata.463062 bs=1032 count=1 2024-09-15T22:03:46.498 INFO:tasks.workunit.client.0.smithi136.stderr:1+0 records in 2024-09-15T22:03:46.499 INFO:tasks.workunit.client.0.smithi136.stderr:1+0 records out 2024-09-15T22:03:46.499 INFO:tasks.workunit.client.0.smithi136.stderr:1032 bytes (1.0 kB, 1.0 KiB) copied, 4.7385e-05 s, 21.8 MB/s 2024-09-15T22:03:46.499 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:757: _scrub_snaps_multi: seq 1 16 2024-09-15T22:03:46.500 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:757: _scrub_snaps_multi: for i in `seq 1 $OBJS` 2024-09-15T22:03:46.500 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:759: _scrub_snaps_multi: rados -p test put obj1 testdata.463062 2024-09-15T22:03:46.656 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:757: _scrub_snaps_multi: for i in `seq 1 $OBJS` 2024-09-15T22:03:46.656 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:759: _scrub_snaps_multi: rados -p test put obj2 testdata.463062 2024-09-15T22:03:46.824 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:757: _scrub_snaps_multi: for i in `seq 1 $OBJS` 2024-09-15T22:03:46.824 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:759: _scrub_snaps_multi: rados -p test put obj3 testdata.463062 2024-09-15T22:03:46.966 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:757: _scrub_snaps_multi: for i in `seq 1 $OBJS` 2024-09-15T22:03:46.966 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:759: _scrub_snaps_multi: rados -p test put obj4 testdata.463062 2024-09-15T22:03:47.092 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:757: _scrub_snaps_multi: for i in `seq 1 $OBJS` 2024-09-15T22:03:47.092 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:759: _scrub_snaps_multi: rados -p test put obj5 testdata.463062 2024-09-15T22:03:47.217 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:757: _scrub_snaps_multi: for i in `seq 1 $OBJS` 2024-09-15T22:03:47.217 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:759: _scrub_snaps_multi: rados -p test put obj6 testdata.463062 2024-09-15T22:03:47.368 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:757: _scrub_snaps_multi: for i in `seq 1 $OBJS` 2024-09-15T22:03:47.369 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:759: _scrub_snaps_multi: rados -p test put obj7 testdata.463062 2024-09-15T22:03:47.502 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:757: _scrub_snaps_multi: for i in `seq 1 $OBJS` 2024-09-15T22:03:47.502 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:759: _scrub_snaps_multi: rados -p test put obj8 testdata.463062 2024-09-15T22:03:47.653 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:757: _scrub_snaps_multi: for i in `seq 1 $OBJS` 2024-09-15T22:03:47.653 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:759: _scrub_snaps_multi: rados -p test put obj9 testdata.463062 2024-09-15T22:03:47.787 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:757: _scrub_snaps_multi: for i in `seq 1 $OBJS` 2024-09-15T22:03:47.787 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:759: _scrub_snaps_multi: rados -p test put obj10 testdata.463062 2024-09-15T22:03:47.912 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:757: _scrub_snaps_multi: for i in `seq 1 $OBJS` 2024-09-15T22:03:47.912 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:759: _scrub_snaps_multi: rados -p test put obj11 testdata.463062 2024-09-15T22:03:48.046 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:757: _scrub_snaps_multi: for i in `seq 1 $OBJS` 2024-09-15T22:03:48.047 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:759: _scrub_snaps_multi: rados -p test put obj12 testdata.463062 2024-09-15T22:03:48.181 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:757: _scrub_snaps_multi: for i in `seq 1 $OBJS` 2024-09-15T22:03:48.181 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:759: _scrub_snaps_multi: rados -p test put obj13 testdata.463062 2024-09-15T22:03:48.322 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:757: _scrub_snaps_multi: for i in `seq 1 $OBJS` 2024-09-15T22:03:48.322 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:759: _scrub_snaps_multi: rados -p test put obj14 testdata.463062 2024-09-15T22:03:48.474 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:757: _scrub_snaps_multi: for i in `seq 1 $OBJS` 2024-09-15T22:03:48.474 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:759: _scrub_snaps_multi: rados -p test put obj15 testdata.463062 2024-09-15T22:03:48.632 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:757: _scrub_snaps_multi: for i in `seq 1 $OBJS` 2024-09-15T22:03:48.633 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:759: _scrub_snaps_multi: rados -p test put obj16 testdata.463062 2024-09-15T22:03:48.767 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:762: _scrub_snaps_multi: get_primary test obj1 2024-09-15T22:03:48.767 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1187: get_primary: local poolname=test 2024-09-15T22:03:48.767 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1188: get_primary: local objectname=obj1 2024-09-15T22:03:48.767 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1190: get_primary: ceph --format json osd map test obj1 2024-09-15T22:03:48.767 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1191: get_primary: jq .acting_primary 2024-09-15T22:03:49.058 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:762: _scrub_snaps_multi: local primary=1 2024-09-15T22:03:49.058 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:763: _scrub_snaps_multi: get_not_primary test obj1 2024-09-15T22:03:49.058 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1220: get_not_primary: local poolname=test 2024-09-15T22:03:49.058 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1221: get_not_primary: local objectname=obj1 2024-09-15T22:03:49.059 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1223: get_not_primary: get_primary test obj1 2024-09-15T22:03:49.059 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1187: get_primary: local poolname=test 2024-09-15T22:03:49.059 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1188: get_primary: local objectname=obj1 2024-09-15T22:03:49.059 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1190: get_primary: ceph --format json osd map test obj1 2024-09-15T22:03:49.059 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1191: get_primary: jq .acting_primary 2024-09-15T22:03:49.349 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1223: get_not_primary: local primary=1 2024-09-15T22:03:49.349 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1224: get_not_primary: ceph --format json osd map test obj1 2024-09-15T22:03:49.349 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1225: get_not_primary: jq '.acting | map(select (. != 1)) | .[0]' 2024-09-15T22:03:49.681 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:763: _scrub_snaps_multi: local replica=0 2024-09-15T22:03:49.681 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:765: _scrub_snaps_multi: eval create_scenario td/osd-scrub-snaps test testdata.463062 '$replica' 2024-09-15T22:03:49.681 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:765: _scrub_snaps_multi: create_scenario td/osd-scrub-snaps test testdata.463062 0 2024-09-15T22:03:49.681 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:45: create_scenario: local dir=td/osd-scrub-snaps 2024-09-15T22:03:49.681 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:46: create_scenario: local poolname=test 2024-09-15T22:03:49.682 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:47: create_scenario: local TESTDATA=testdata.463062 2024-09-15T22:03:49.682 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:48: create_scenario: local osd=0 2024-09-15T22:03:49.682 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:50: create_scenario: SNAP=1 2024-09-15T22:03:49.682 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:51: create_scenario: rados -p test mksnap snap1 2024-09-15T22:03:49.892 INFO:tasks.workunit.client.0.smithi136.stdout:created pool test snap snap1 2024-09-15T22:03:49.894 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:52: create_scenario: dd if=/dev/urandom of=testdata.463062 bs=256 count=1 2024-09-15T22:03:49.895 INFO:tasks.workunit.client.0.smithi136.stderr:1+0 records in 2024-09-15T22:03:49.895 INFO:tasks.workunit.client.0.smithi136.stderr:1+0 records out 2024-09-15T22:03:49.895 INFO:tasks.workunit.client.0.smithi136.stderr:256 bytes copied, 6.3851e-05 s, 4.0 MB/s 2024-09-15T22:03:49.896 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:53: create_scenario: rados -p test put obj1 testdata.463062 2024-09-15T22:03:50.120 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:54: create_scenario: rados -p test put obj5 testdata.463062 2024-09-15T22:03:50.262 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:55: create_scenario: rados -p test put obj3 testdata.463062 2024-09-15T22:03:50.397 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:56: create_scenario: seq 6 14 2024-09-15T22:03:50.397 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:56: create_scenario: for i in `seq 6 14` 2024-09-15T22:03:50.397 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:57: create_scenario: rados -p test put obj6 testdata.463062 2024-09-15T22:03:50.556 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:56: create_scenario: for i in `seq 6 14` 2024-09-15T22:03:50.556 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:57: create_scenario: rados -p test put obj7 testdata.463062 2024-09-15T22:03:50.723 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:56: create_scenario: for i in `seq 6 14` 2024-09-15T22:03:50.723 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:57: create_scenario: rados -p test put obj8 testdata.463062 2024-09-15T22:03:50.865 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:56: create_scenario: for i in `seq 6 14` 2024-09-15T22:03:50.866 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:57: create_scenario: rados -p test put obj9 testdata.463062 2024-09-15T22:03:50.991 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:56: create_scenario: for i in `seq 6 14` 2024-09-15T22:03:50.991 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:57: create_scenario: rados -p test put obj10 testdata.463062 2024-09-15T22:03:51.117 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:56: create_scenario: for i in `seq 6 14` 2024-09-15T22:03:51.117 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:57: create_scenario: rados -p test put obj11 testdata.463062 2024-09-15T22:03:51.242 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:56: create_scenario: for i in `seq 6 14` 2024-09-15T22:03:51.242 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:57: create_scenario: rados -p test put obj12 testdata.463062 2024-09-15T22:03:51.393 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:56: create_scenario: for i in `seq 6 14` 2024-09-15T22:03:51.394 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:57: create_scenario: rados -p test put obj13 testdata.463062 2024-09-15T22:03:51.535 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:56: create_scenario: for i in `seq 6 14` 2024-09-15T22:03:51.536 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:57: create_scenario: rados -p test put obj14 testdata.463062 2024-09-15T22:03:51.694 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:60: create_scenario: SNAP=2 2024-09-15T22:03:51.694 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:61: create_scenario: rados -p test mksnap snap2 2024-09-15T22:03:51.852 INFO:tasks.workunit.client.0.smithi136.stdout:created pool test snap snap2 2024-09-15T22:03:51.854 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:62: create_scenario: dd if=/dev/urandom of=testdata.463062 bs=256 count=2 2024-09-15T22:03:51.855 INFO:tasks.workunit.client.0.smithi136.stderr:2+0 records in 2024-09-15T22:03:51.855 INFO:tasks.workunit.client.0.smithi136.stderr:2+0 records out 2024-09-15T22:03:51.855 INFO:tasks.workunit.client.0.smithi136.stderr:512 bytes copied, 6.3224e-05 s, 8.1 MB/s 2024-09-15T22:03:51.855 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:63: create_scenario: rados -p test put obj5 testdata.463062 2024-09-15T22:03:52.121 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:65: create_scenario: SNAP=3 2024-09-15T22:03:52.121 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:66: create_scenario: rados -p test mksnap snap3 2024-09-15T22:03:52.288 INFO:tasks.workunit.client.0.smithi136.stdout:created pool test snap snap3 2024-09-15T22:03:52.290 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:67: create_scenario: dd if=/dev/urandom of=testdata.463062 bs=256 count=3 2024-09-15T22:03:52.291 INFO:tasks.workunit.client.0.smithi136.stderr:3+0 records in 2024-09-15T22:03:52.291 INFO:tasks.workunit.client.0.smithi136.stderr:3+0 records out 2024-09-15T22:03:52.291 INFO:tasks.workunit.client.0.smithi136.stderr:768 bytes copied, 6.6525e-05 s, 11.5 MB/s 2024-09-15T22:03:52.291 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:68: create_scenario: rados -p test put obj3 testdata.463062 2024-09-15T22:03:52.540 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:70: create_scenario: SNAP=4 2024-09-15T22:03:52.540 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:71: create_scenario: rados -p test mksnap snap4 2024-09-15T22:03:52.797 INFO:tasks.workunit.client.0.smithi136.stdout:created pool test snap snap4 2024-09-15T22:03:52.799 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:72: create_scenario: dd if=/dev/urandom of=testdata.463062 bs=256 count=4 2024-09-15T22:03:52.800 INFO:tasks.workunit.client.0.smithi136.stderr:4+0 records in 2024-09-15T22:03:52.800 INFO:tasks.workunit.client.0.smithi136.stderr:4+0 records out 2024-09-15T22:03:52.800 INFO:tasks.workunit.client.0.smithi136.stderr:1024 bytes (1.0 kB, 1.0 KiB) copied, 6.7898e-05 s, 15.1 MB/s 2024-09-15T22:03:52.800 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:73: create_scenario: rados -p test put obj5 testdata.463062 2024-09-15T22:03:53.026 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:74: create_scenario: rados -p test put obj2 testdata.463062 2024-09-15T22:03:53.202 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:76: create_scenario: SNAP=5 2024-09-15T22:03:53.202 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:77: create_scenario: rados -p test mksnap snap5 2024-09-15T22:03:53.368 INFO:tasks.workunit.client.0.smithi136.stdout:created pool test snap snap5 2024-09-15T22:03:53.370 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:78: create_scenario: SNAP=6 2024-09-15T22:03:53.370 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:79: create_scenario: rados -p test mksnap snap6 2024-09-15T22:03:53.753 INFO:tasks.workunit.client.0.smithi136.stdout:created pool test snap snap6 2024-09-15T22:03:53.755 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:80: create_scenario: dd if=/dev/urandom of=testdata.463062 bs=256 count=6 2024-09-15T22:03:53.756 INFO:tasks.workunit.client.0.smithi136.stderr:6+0 records in 2024-09-15T22:03:53.756 INFO:tasks.workunit.client.0.smithi136.stderr:6+0 records out 2024-09-15T22:03:53.756 INFO:tasks.workunit.client.0.smithi136.stderr:1536 bytes (1.5 kB, 1.5 KiB) copied, 7.2081e-05 s, 21.3 MB/s 2024-09-15T22:03:53.756 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:81: create_scenario: rados -p test put obj5 testdata.463062 2024-09-15T22:03:53.997 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:83: create_scenario: SNAP=7 2024-09-15T22:03:53.997 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:84: create_scenario: rados -p test mksnap snap7 2024-09-15T22:03:54.163 INFO:tasks.workunit.client.0.smithi136.stdout:created pool test snap snap7 2024-09-15T22:03:54.165 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:86: create_scenario: rados -p test rm obj4 2024-09-15T22:03:54.407 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:87: create_scenario: rados -p test rm obj16 2024-09-15T22:03:54.557 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:88: create_scenario: rados -p test rm obj2 2024-09-15T22:03:54.701 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:90: create_scenario: kill_daemons td/osd-scrub-snaps TERM osd 2024-09-15T22:03:54.701 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-15T22:03:54.701 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-15T22:03:54.701 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-15T22:03:54.701 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-15T22:03:54.701 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-15T22:03:55.007 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-15T22:03:55.007 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:94: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 --head --op list obj1 2024-09-15T22:03:55.640 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:94: create_scenario: JSON='["1.0",{"oid":"obj1","key":"","snapid":-2,"hash":1828249343,"max":0,"pool":1,"namespace":"","max":0}]' 2024-09-15T22:03:55.640 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:95: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 '["1.0",{"oid":"obj1","key":"","snapid":-2,"hash":1828249343,"max":0,"pool":1,"namespace":"","max":0}]' --force remove 2024-09-15T22:03:57.163 INFO:tasks.workunit.client.0.smithi136.stdout:WARNING: only removing head with clones present 2024-09-15T22:03:57.164 INFO:tasks.workunit.client.0.smithi136.stdout:remove #1:ff7b1f36:::obj1:head# 2024-09-15T22:03:57.679 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:97: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 --op list obj5 2024-09-15T22:03:57.679 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:97: create_scenario: grep '"snapid":2' 2024-09-15T22:03:58.284 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:97: create_scenario: JSON='["1.0",{"oid":"obj5","key":"","snapid":2,"hash":1718170787,"max":0,"pool":1,"namespace":"","max":0}]' 2024-09-15T22:03:58.284 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:98: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 '["1.0",{"oid":"obj5","key":"","snapid":2,"hash":1718170787,"max":0,"pool":1,"namespace":"","max":0}]' remove 2024-09-15T22:03:59.334 INFO:tasks.workunit.client.0.smithi136.stdout:remove #1:c52c9666:::obj5:2# 2024-09-15T22:03:59.849 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:100: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 --op list obj5 2024-09-15T22:03:59.849 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:100: create_scenario: grep '"snapid":1' 2024-09-15T22:04:00.455 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:100: create_scenario: JSON='["1.0",{"oid":"obj5","key":"","snapid":1,"hash":1718170787,"max":0,"pool":1,"namespace":"","max":0}]' 2024-09-15T22:04:00.455 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:101: create_scenario: OBJ5SAVE='["1.0",{"oid":"obj5","key":"","snapid":1,"hash":1718170787,"max":0,"pool":1,"namespace":"","max":0}]' 2024-09-15T22:04:00.455 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:103: create_scenario: ceph-kvstore-tool bluestore-kv td/osd-scrub-snaps/0 list 2024-09-15T22:04:01.516 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:104: create_scenario: grep SNA_ td/osd-scrub-snaps/drk.log 2024-09-15T22:04:01.517 INFO:tasks.workunit.client.0.smithi136.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.0779578A.7.obj4.. 2024-09-15T22:04:01.517 INFO:tasks.workunit.client.0.smithi136.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.1BB86F16.1.obj3.. 2024-09-15T22:04:01.517 INFO:tasks.workunit.client.0.smithi136.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.268F1DA7.7.obj16.. 2024-09-15T22:04:01.517 INFO:tasks.workunit.client.0.smithi136.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.3A439666.1.obj5.. 2024-09-15T22:04:01.518 INFO:tasks.workunit.client.0.smithi136.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.4CC52438.1.obj6.. 2024-09-15T22:04:01.518 INFO:tasks.workunit.client.0.smithi136.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.5685B6C7.1.obj13.. 2024-09-15T22:04:01.518 INFO:tasks.workunit.client.0.smithi136.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.5AAE7FD8.1.obj8.. 2024-09-15T22:04:01.518 INFO:tasks.workunit.client.0.smithi136.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.5FAF9A3D.1.obj12.. 2024-09-15T22:04:01.518 INFO:tasks.workunit.client.0.smithi136.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.70522149.1.obj14.. 2024-09-15T22:04:01.518 INFO:tasks.workunit.client.0.smithi136.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.77574592.1.obj11.. 2024-09-15T22:04:01.518 INFO:tasks.workunit.client.0.smithi136.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.802EE1F3.4.obj2.. 2024-09-15T22:04:01.518 INFO:tasks.workunit.client.0.smithi136.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.950988C5.1.obj7.. 2024-09-15T22:04:01.518 INFO:tasks.workunit.client.0.smithi136.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.B8CCECA2.1.obj10.. 2024-09-15T22:04:01.518 INFO:tasks.workunit.client.0.smithi136.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.F7CA874E.1.obj9.. 2024-09-15T22:04:01.518 INFO:tasks.workunit.client.0.smithi136.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.FFED8FC6.1.obj1.. 2024-09-15T22:04:01.518 INFO:tasks.workunit.client.0.smithi136.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000002_0000000000000001.0779578A.7.obj4.. 2024-09-15T22:04:01.519 INFO:tasks.workunit.client.0.smithi136.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000002_0000000000000001.1BB86F16.3.obj3.. 2024-09-15T22:04:01.519 INFO:tasks.workunit.client.0.smithi136.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000002_0000000000000001.268F1DA7.7.obj16.. 2024-09-15T22:04:01.519 INFO:tasks.workunit.client.0.smithi136.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000002_0000000000000001.802EE1F3.4.obj2.. 2024-09-15T22:04:01.519 INFO:tasks.workunit.client.0.smithi136.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.0779578A.7.obj4.. 2024-09-15T22:04:01.519 INFO:tasks.workunit.client.0.smithi136.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.1BB86F16.3.obj3.. 2024-09-15T22:04:01.519 INFO:tasks.workunit.client.0.smithi136.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.268F1DA7.7.obj16.. 2024-09-15T22:04:01.519 INFO:tasks.workunit.client.0.smithi136.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.3A439666.4.obj5.. 2024-09-15T22:04:01.519 INFO:tasks.workunit.client.0.smithi136.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.802EE1F3.4.obj2.. 2024-09-15T22:04:01.519 INFO:tasks.workunit.client.0.smithi136.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000004_0000000000000001.0779578A.7.obj4.. 2024-09-15T22:04:01.519 INFO:tasks.workunit.client.0.smithi136.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000004_0000000000000001.268F1DA7.7.obj16.. 2024-09-15T22:04:01.519 INFO:tasks.workunit.client.0.smithi136.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000004_0000000000000001.3A439666.4.obj5.. 2024-09-15T22:04:01.519 INFO:tasks.workunit.client.0.smithi136.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000004_0000000000000001.802EE1F3.4.obj2.. 2024-09-15T22:04:01.519 INFO:tasks.workunit.client.0.smithi136.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000005_0000000000000001.0779578A.7.obj4.. 2024-09-15T22:04:01.520 INFO:tasks.workunit.client.0.smithi136.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000005_0000000000000001.268F1DA7.7.obj16.. 2024-09-15T22:04:01.520 INFO:tasks.workunit.client.0.smithi136.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000005_0000000000000001.3A439666.6.obj5.. 2024-09-15T22:04:01.520 INFO:tasks.workunit.client.0.smithi136.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000005_0000000000000001.802EE1F3.7.obj2.. 2024-09-15T22:04:01.520 INFO:tasks.workunit.client.0.smithi136.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000006_0000000000000001.0779578A.7.obj4.. 2024-09-15T22:04:01.520 INFO:tasks.workunit.client.0.smithi136.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000006_0000000000000001.268F1DA7.7.obj16.. 2024-09-15T22:04:01.520 INFO:tasks.workunit.client.0.smithi136.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000006_0000000000000001.3A439666.6.obj5.. 2024-09-15T22:04:01.520 INFO:tasks.workunit.client.0.smithi136.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000006_0000000000000001.802EE1F3.7.obj2.. 2024-09-15T22:04:01.520 INFO:tasks.workunit.client.0.smithi136.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000007_0000000000000001.0779578A.7.obj4.. 2024-09-15T22:04:01.520 INFO:tasks.workunit.client.0.smithi136.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000007_0000000000000001.268F1DA7.7.obj16.. 2024-09-15T22:04:01.520 INFO:tasks.workunit.client.0.smithi136.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000007_0000000000000001.802EE1F3.7.obj2.. 2024-09-15T22:04:01.520 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:105: create_scenario: grep '^[pm].*SNA_.*[.]1[.]obj5[.][.]$' td/osd-scrub-snaps/drk.log 2024-09-15T22:04:01.521 INFO:tasks.workunit.client.0.smithi136.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.3A439666.1.obj5.. 2024-09-15T22:04:01.521 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:106: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 --rmtype nosnapmap '["1.0",{"oid":"obj5","key":"","snapid":1,"hash":1718170787,"max":0,"pool":1,"namespace":"","max":0}]' remove 2024-09-15T22:04:02.720 INFO:tasks.workunit.client.0.smithi136.stdout:remove #1:c52c9666:::obj5:1# 2024-09-15T22:04:03.235 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:108: create_scenario: ceph-kvstore-tool bluestore-kv td/osd-scrub-snaps/0 list 2024-09-15T22:04:04.304 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:109: create_scenario: grep SNA_ td/osd-scrub-snaps/drk.log 2024-09-15T22:04:04.306 INFO:tasks.workunit.client.0.smithi136.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.0779578A.7.obj4.. 2024-09-15T22:04:04.306 INFO:tasks.workunit.client.0.smithi136.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.1BB86F16.1.obj3.. 2024-09-15T22:04:04.306 INFO:tasks.workunit.client.0.smithi136.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.268F1DA7.7.obj16.. 2024-09-15T22:04:04.306 INFO:tasks.workunit.client.0.smithi136.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.3A439666.1.obj5.. 2024-09-15T22:04:04.306 INFO:tasks.workunit.client.0.smithi136.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.4CC52438.1.obj6.. 2024-09-15T22:04:04.306 INFO:tasks.workunit.client.0.smithi136.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.5685B6C7.1.obj13.. 2024-09-15T22:04:04.306 INFO:tasks.workunit.client.0.smithi136.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.5AAE7FD8.1.obj8.. 2024-09-15T22:04:04.306 INFO:tasks.workunit.client.0.smithi136.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.5FAF9A3D.1.obj12.. 2024-09-15T22:04:04.306 INFO:tasks.workunit.client.0.smithi136.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.70522149.1.obj14.. 2024-09-15T22:04:04.306 INFO:tasks.workunit.client.0.smithi136.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.77574592.1.obj11.. 2024-09-15T22:04:04.306 INFO:tasks.workunit.client.0.smithi136.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.802EE1F3.4.obj2.. 2024-09-15T22:04:04.307 INFO:tasks.workunit.client.0.smithi136.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.950988C5.1.obj7.. 2024-09-15T22:04:04.307 INFO:tasks.workunit.client.0.smithi136.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.B8CCECA2.1.obj10.. 2024-09-15T22:04:04.307 INFO:tasks.workunit.client.0.smithi136.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.F7CA874E.1.obj9.. 2024-09-15T22:04:04.307 INFO:tasks.workunit.client.0.smithi136.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.FFED8FC6.1.obj1.. 2024-09-15T22:04:04.307 INFO:tasks.workunit.client.0.smithi136.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000002_0000000000000001.0779578A.7.obj4.. 2024-09-15T22:04:04.307 INFO:tasks.workunit.client.0.smithi136.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000002_0000000000000001.1BB86F16.3.obj3.. 2024-09-15T22:04:04.307 INFO:tasks.workunit.client.0.smithi136.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000002_0000000000000001.268F1DA7.7.obj16.. 2024-09-15T22:04:04.307 INFO:tasks.workunit.client.0.smithi136.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000002_0000000000000001.802EE1F3.4.obj2.. 2024-09-15T22:04:04.307 INFO:tasks.workunit.client.0.smithi136.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.0779578A.7.obj4.. 2024-09-15T22:04:04.307 INFO:tasks.workunit.client.0.smithi136.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.1BB86F16.3.obj3.. 2024-09-15T22:04:04.307 INFO:tasks.workunit.client.0.smithi136.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.268F1DA7.7.obj16.. 2024-09-15T22:04:04.307 INFO:tasks.workunit.client.0.smithi136.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.3A439666.4.obj5.. 2024-09-15T22:04:04.308 INFO:tasks.workunit.client.0.smithi136.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.802EE1F3.4.obj2.. 2024-09-15T22:04:04.308 INFO:tasks.workunit.client.0.smithi136.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000004_0000000000000001.0779578A.7.obj4.. 2024-09-15T22:04:04.308 INFO:tasks.workunit.client.0.smithi136.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000004_0000000000000001.268F1DA7.7.obj16.. 2024-09-15T22:04:04.308 INFO:tasks.workunit.client.0.smithi136.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000004_0000000000000001.3A439666.4.obj5.. 2024-09-15T22:04:04.308 INFO:tasks.workunit.client.0.smithi136.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000004_0000000000000001.802EE1F3.4.obj2.. 2024-09-15T22:04:04.308 INFO:tasks.workunit.client.0.smithi136.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000005_0000000000000001.0779578A.7.obj4.. 2024-09-15T22:04:04.308 INFO:tasks.workunit.client.0.smithi136.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000005_0000000000000001.268F1DA7.7.obj16.. 2024-09-15T22:04:04.308 INFO:tasks.workunit.client.0.smithi136.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000005_0000000000000001.3A439666.6.obj5.. 2024-09-15T22:04:04.308 INFO:tasks.workunit.client.0.smithi136.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000005_0000000000000001.802EE1F3.7.obj2.. 2024-09-15T22:04:04.308 INFO:tasks.workunit.client.0.smithi136.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000006_0000000000000001.0779578A.7.obj4.. 2024-09-15T22:04:04.308 INFO:tasks.workunit.client.0.smithi136.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000006_0000000000000001.268F1DA7.7.obj16.. 2024-09-15T22:04:04.309 INFO:tasks.workunit.client.0.smithi136.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000006_0000000000000001.3A439666.6.obj5.. 2024-09-15T22:04:04.309 INFO:tasks.workunit.client.0.smithi136.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000006_0000000000000001.802EE1F3.7.obj2.. 2024-09-15T22:04:04.309 INFO:tasks.workunit.client.0.smithi136.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000007_0000000000000001.0779578A.7.obj4.. 2024-09-15T22:04:04.309 INFO:tasks.workunit.client.0.smithi136.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000007_0000000000000001.268F1DA7.7.obj16.. 2024-09-15T22:04:04.309 INFO:tasks.workunit.client.0.smithi136.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000007_0000000000000001.802EE1F3.7.obj2.. 2024-09-15T22:04:04.309 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:110: create_scenario: grep '^[pm].*SNA_.*[.]1[.]obj5[.][.]$' td/osd-scrub-snaps/drk.log 2024-09-15T22:04:04.309 INFO:tasks.workunit.client.0.smithi136.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.3A439666.1.obj5.. 2024-09-15T22:04:04.310 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:111: create_scenario: rm -f td/osd-scrub-snaps/drk.log 2024-09-15T22:04:04.310 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:113: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 --op list obj5 2024-09-15T22:04:04.310 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:113: create_scenario: grep '"snapid":4' 2024-09-15T22:04:04.901 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:113: create_scenario: JSON='["1.0",{"oid":"obj5","key":"","snapid":4,"hash":1718170787,"max":0,"pool":1,"namespace":"","max":0}]' 2024-09-15T22:04:04.901 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:114: create_scenario: dd if=/dev/urandom of=testdata.463062 bs=256 count=18 2024-09-15T22:04:04.901 INFO:tasks.workunit.client.0.smithi136.stderr:18+0 records in 2024-09-15T22:04:04.901 INFO:tasks.workunit.client.0.smithi136.stderr:18+0 records out 2024-09-15T22:04:04.902 INFO:tasks.workunit.client.0.smithi136.stderr:4608 bytes (4.6 kB, 4.5 KiB) copied, 0.000131135 s, 35.1 MB/s 2024-09-15T22:04:04.902 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:115: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 '["1.0",{"oid":"obj5","key":"","snapid":4,"hash":1718170787,"max":0,"pool":1,"namespace":"","max":0}]' set-bytes testdata.463062 2024-09-15T22:04:06.379 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:117: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 --head --op list obj3 2024-09-15T22:04:06.965 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:117: create_scenario: JSON='["1.0",{"oid":"obj3","key":"","snapid":-2,"hash":1643547569,"max":0,"pool":1,"namespace":"","max":0}]' 2024-09-15T22:04:06.966 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:118: create_scenario: dd if=/dev/urandom of=testdata.463062 bs=256 count=15 2024-09-15T22:04:06.966 INFO:tasks.workunit.client.0.smithi136.stderr:15+0 records in 2024-09-15T22:04:06.966 INFO:tasks.workunit.client.0.smithi136.stderr:15+0 records out 2024-09-15T22:04:06.967 INFO:tasks.workunit.client.0.smithi136.stderr:3840 bytes (3.8 kB, 3.8 KiB) copied, 0.000141874 s, 27.1 MB/s 2024-09-15T22:04:06.967 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:119: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 '["1.0",{"oid":"obj3","key":"","snapid":-2,"hash":1643547569,"max":0,"pool":1,"namespace":"","max":0}]' set-bytes testdata.463062 2024-09-15T22:04:08.510 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:121: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 --op list obj4 2024-09-15T22:04:08.510 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:121: create_scenario: grep '"snapid":7' 2024-09-15T22:04:09.138 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:121: create_scenario: JSON='["1.0",{"oid":"obj4","key":"","snapid":7,"hash":2826278768,"max":0,"pool":1,"namespace":"","max":0}]' 2024-09-15T22:04:09.138 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:122: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 '["1.0",{"oid":"obj4","key":"","snapid":7,"hash":2826278768,"max":0,"pool":1,"namespace":"","max":0}]' remove 2024-09-15T22:04:10.107 INFO:tasks.workunit.client.0.smithi136.stdout:remove #1:0ee9ae15:::obj4:7# 2024-09-15T22:04:10.875 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:125: create_scenario: ceph-kvstore-tool bluestore-kv td/osd-scrub-snaps/0 list 2024-09-15T22:04:11.936 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:126: create_scenario: grep SNA_ td/osd-scrub-snaps/drk.log 2024-09-15T22:04:11.937 INFO:tasks.workunit.client.0.smithi136.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.1BB86F16.1.obj3.. 2024-09-15T22:04:11.937 INFO:tasks.workunit.client.0.smithi136.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.268F1DA7.7.obj16.. 2024-09-15T22:04:11.938 INFO:tasks.workunit.client.0.smithi136.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.3A439666.1.obj5.. 2024-09-15T22:04:11.938 INFO:tasks.workunit.client.0.smithi136.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.4CC52438.1.obj6.. 2024-09-15T22:04:11.938 INFO:tasks.workunit.client.0.smithi136.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.5685B6C7.1.obj13.. 2024-09-15T22:04:11.938 INFO:tasks.workunit.client.0.smithi136.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.5AAE7FD8.1.obj8.. 2024-09-15T22:04:11.938 INFO:tasks.workunit.client.0.smithi136.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.5FAF9A3D.1.obj12.. 2024-09-15T22:04:11.938 INFO:tasks.workunit.client.0.smithi136.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.70522149.1.obj14.. 2024-09-15T22:04:11.938 INFO:tasks.workunit.client.0.smithi136.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.77574592.1.obj11.. 2024-09-15T22:04:11.938 INFO:tasks.workunit.client.0.smithi136.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.802EE1F3.4.obj2.. 2024-09-15T22:04:11.938 INFO:tasks.workunit.client.0.smithi136.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.950988C5.1.obj7.. 2024-09-15T22:04:11.938 INFO:tasks.workunit.client.0.smithi136.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.B8CCECA2.1.obj10.. 2024-09-15T22:04:11.938 INFO:tasks.workunit.client.0.smithi136.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.F7CA874E.1.obj9.. 2024-09-15T22:04:11.938 INFO:tasks.workunit.client.0.smithi136.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.FFED8FC6.1.obj1.. 2024-09-15T22:04:11.939 INFO:tasks.workunit.client.0.smithi136.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000002_0000000000000001.1BB86F16.3.obj3.. 2024-09-15T22:04:11.939 INFO:tasks.workunit.client.0.smithi136.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000002_0000000000000001.268F1DA7.7.obj16.. 2024-09-15T22:04:11.939 INFO:tasks.workunit.client.0.smithi136.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000002_0000000000000001.802EE1F3.4.obj2.. 2024-09-15T22:04:11.939 INFO:tasks.workunit.client.0.smithi136.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.1BB86F16.3.obj3.. 2024-09-15T22:04:11.939 INFO:tasks.workunit.client.0.smithi136.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.268F1DA7.7.obj16.. 2024-09-15T22:04:11.939 INFO:tasks.workunit.client.0.smithi136.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.3A439666.4.obj5.. 2024-09-15T22:04:11.939 INFO:tasks.workunit.client.0.smithi136.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.802EE1F3.4.obj2.. 2024-09-15T22:04:11.939 INFO:tasks.workunit.client.0.smithi136.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000004_0000000000000001.268F1DA7.7.obj16.. 2024-09-15T22:04:11.939 INFO:tasks.workunit.client.0.smithi136.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000004_0000000000000001.3A439666.4.obj5.. 2024-09-15T22:04:11.939 INFO:tasks.workunit.client.0.smithi136.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000004_0000000000000001.802EE1F3.4.obj2.. 2024-09-15T22:04:11.939 INFO:tasks.workunit.client.0.smithi136.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000005_0000000000000001.268F1DA7.7.obj16.. 2024-09-15T22:04:11.939 INFO:tasks.workunit.client.0.smithi136.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000005_0000000000000001.3A439666.6.obj5.. 2024-09-15T22:04:11.939 INFO:tasks.workunit.client.0.smithi136.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000005_0000000000000001.802EE1F3.7.obj2.. 2024-09-15T22:04:11.939 INFO:tasks.workunit.client.0.smithi136.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000006_0000000000000001.268F1DA7.7.obj16.. 2024-09-15T22:04:11.940 INFO:tasks.workunit.client.0.smithi136.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000006_0000000000000001.3A439666.6.obj5.. 2024-09-15T22:04:11.940 INFO:tasks.workunit.client.0.smithi136.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000006_0000000000000001.802EE1F3.7.obj2.. 2024-09-15T22:04:11.940 INFO:tasks.workunit.client.0.smithi136.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000007_0000000000000001.268F1DA7.7.obj16.. 2024-09-15T22:04:11.940 INFO:tasks.workunit.client.0.smithi136.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000007_0000000000000001.802EE1F3.7.obj2.. 2024-09-15T22:04:11.940 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:127: create_scenario: grep '^[pm].*SNA_.*[.]7[.]obj16[.][.]$' td/osd-scrub-snaps/drk.log 2024-09-15T22:04:11.940 INFO:tasks.workunit.client.0.smithi136.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.268F1DA7.7.obj16.. 2024-09-15T22:04:11.940 INFO:tasks.workunit.client.0.smithi136.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000002_0000000000000001.268F1DA7.7.obj16.. 2024-09-15T22:04:11.940 INFO:tasks.workunit.client.0.smithi136.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.268F1DA7.7.obj16.. 2024-09-15T22:04:11.941 INFO:tasks.workunit.client.0.smithi136.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000004_0000000000000001.268F1DA7.7.obj16.. 2024-09-15T22:04:11.941 INFO:tasks.workunit.client.0.smithi136.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000005_0000000000000001.268F1DA7.7.obj16.. 2024-09-15T22:04:11.941 INFO:tasks.workunit.client.0.smithi136.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000006_0000000000000001.268F1DA7.7.obj16.. 2024-09-15T22:04:11.941 INFO:tasks.workunit.client.0.smithi136.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000007_0000000000000001.268F1DA7.7.obj16.. 2024-09-15T22:04:11.941 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:128: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 --op list obj16 2024-09-15T22:04:11.941 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:128: create_scenario: grep '"snapid":7' 2024-09-15T22:04:12.534 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:128: create_scenario: JSON='["1.0",{"oid":"obj16","key":"","snapid":7,"hash":2060580962,"max":0,"pool":1,"namespace":"","max":0}]' 2024-09-15T22:04:12.534 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:129: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 --rmtype snapmap '["1.0",{"oid":"obj16","key":"","snapid":7,"hash":2060580962,"max":0,"pool":1,"namespace":"","max":0}]' remove 2024-09-15T22:04:13.479 INFO:tasks.workunit.client.0.smithi136.stdout:remove #1:461f8b5e:::obj16:7# 2024-09-15T22:04:13.993 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:131: create_scenario: ceph-kvstore-tool bluestore-kv td/osd-scrub-snaps/0 list 2024-09-15T22:04:15.057 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:132: create_scenario: grep SNA_ td/osd-scrub-snaps/drk.log 2024-09-15T22:04:15.058 INFO:tasks.workunit.client.0.smithi136.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.1BB86F16.1.obj3.. 2024-09-15T22:04:15.058 INFO:tasks.workunit.client.0.smithi136.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.3A439666.1.obj5.. 2024-09-15T22:04:15.059 INFO:tasks.workunit.client.0.smithi136.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.4CC52438.1.obj6.. 2024-09-15T22:04:15.059 INFO:tasks.workunit.client.0.smithi136.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.5685B6C7.1.obj13.. 2024-09-15T22:04:15.059 INFO:tasks.workunit.client.0.smithi136.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.5AAE7FD8.1.obj8.. 2024-09-15T22:04:15.059 INFO:tasks.workunit.client.0.smithi136.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.5FAF9A3D.1.obj12.. 2024-09-15T22:04:15.059 INFO:tasks.workunit.client.0.smithi136.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.70522149.1.obj14.. 2024-09-15T22:04:15.059 INFO:tasks.workunit.client.0.smithi136.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.77574592.1.obj11.. 2024-09-15T22:04:15.059 INFO:tasks.workunit.client.0.smithi136.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.802EE1F3.4.obj2.. 2024-09-15T22:04:15.059 INFO:tasks.workunit.client.0.smithi136.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.950988C5.1.obj7.. 2024-09-15T22:04:15.059 INFO:tasks.workunit.client.0.smithi136.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.B8CCECA2.1.obj10.. 2024-09-15T22:04:15.059 INFO:tasks.workunit.client.0.smithi136.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.F7CA874E.1.obj9.. 2024-09-15T22:04:15.059 INFO:tasks.workunit.client.0.smithi136.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.FFED8FC6.1.obj1.. 2024-09-15T22:04:15.059 INFO:tasks.workunit.client.0.smithi136.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000002_0000000000000001.1BB86F16.3.obj3.. 2024-09-15T22:04:15.059 INFO:tasks.workunit.client.0.smithi136.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000002_0000000000000001.802EE1F3.4.obj2.. 2024-09-15T22:04:15.060 INFO:tasks.workunit.client.0.smithi136.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.1BB86F16.3.obj3.. 2024-09-15T22:04:15.060 INFO:tasks.workunit.client.0.smithi136.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.3A439666.4.obj5.. 2024-09-15T22:04:15.060 INFO:tasks.workunit.client.0.smithi136.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.802EE1F3.4.obj2.. 2024-09-15T22:04:15.060 INFO:tasks.workunit.client.0.smithi136.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000004_0000000000000001.3A439666.4.obj5.. 2024-09-15T22:04:15.060 INFO:tasks.workunit.client.0.smithi136.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000004_0000000000000001.802EE1F3.4.obj2.. 2024-09-15T22:04:15.060 INFO:tasks.workunit.client.0.smithi136.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000005_0000000000000001.3A439666.6.obj5.. 2024-09-15T22:04:15.060 INFO:tasks.workunit.client.0.smithi136.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000005_0000000000000001.802EE1F3.7.obj2.. 2024-09-15T22:04:15.060 INFO:tasks.workunit.client.0.smithi136.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000006_0000000000000001.3A439666.6.obj5.. 2024-09-15T22:04:15.060 INFO:tasks.workunit.client.0.smithi136.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000006_0000000000000001.802EE1F3.7.obj2.. 2024-09-15T22:04:15.060 INFO:tasks.workunit.client.0.smithi136.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000007_0000000000000001.802EE1F3.7.obj2.. 2024-09-15T22:04:15.061 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:133: create_scenario: grep '^[pm].*SNA_.*[.]7[.]obj16[.][.]$' td/osd-scrub-snaps/drk.log 2024-09-15T22:04:15.061 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:134: create_scenario: rm -f td/osd-scrub-snaps/drk.log 2024-09-15T22:04:15.061 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:136: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 --head --op list obj2 2024-09-15T22:04:15.663 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:136: create_scenario: JSON='["1.0",{"oid":"obj2","key":"","snapid":-2,"hash":1058988552,"max":0,"pool":1,"namespace":"","max":0}]' 2024-09-15T22:04:15.664 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:137: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 '["1.0",{"oid":"obj2","key":"","snapid":-2,"hash":1058988552,"max":0,"pool":1,"namespace":"","max":0}]' rm-attr snapset 2024-09-15T22:04:17.230 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:140: create_scenario: echo '["1.0",{"oid":"obj5","key":"","snapid":1,"hash":1718170787,"max":0,"pool":1,"namespace":"","max":0}]' 2024-09-15T22:04:17.231 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:140: create_scenario: sed 's/snapid":1/snapid":7/' 2024-09-15T22:04:17.232 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:140: create_scenario: JSON='["1.0",{"oid":"obj5","key":"","snapid":7,"hash":1718170787,"max":0,"pool":1,"namespace":"","max":0}]' 2024-09-15T22:04:17.232 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:141: create_scenario: dd if=/dev/urandom of=testdata.463062 bs=256 count=7 2024-09-15T22:04:17.233 INFO:tasks.workunit.client.0.smithi136.stderr:7+0 records in 2024-09-15T22:04:17.233 INFO:tasks.workunit.client.0.smithi136.stderr:7+0 records out 2024-09-15T22:04:17.233 INFO:tasks.workunit.client.0.smithi136.stderr:1792 bytes (1.8 kB, 1.8 KiB) copied, 0.000100401 s, 17.8 MB/s 2024-09-15T22:04:17.233 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:142: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 '["1.0",{"oid":"obj5","key":"","snapid":7,"hash":1718170787,"max":0,"pool":1,"namespace":"","max":0}]' set-bytes testdata.463062 2024-09-15T22:04:19.003 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:144: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 --head --op list obj6 2024-09-15T22:04:19.332 INFO:tasks.workunit.client.0.smithi136.stderr:Error getting attr on : 1.0_head,#1:c52c9666:::obj5:7#, (61) No data available 2024-09-15T22:04:19.596 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:144: create_scenario: JSON='["1.0",{"oid":"obj6","key":"","snapid":-2,"hash":2202164420,"max":0,"pool":1,"namespace":"","max":0}]' 2024-09-15T22:04:19.597 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:145: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 '["1.0",{"oid":"obj6","key":"","snapid":-2,"hash":2202164420,"max":0,"pool":1,"namespace":"","max":0}]' clear-snapset 2024-09-15T22:04:21.242 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:146: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 --head --op list obj7 2024-09-15T22:04:21.574 INFO:tasks.workunit.client.0.smithi136.stderr:Error getting attr on : 1.0_head,#1:c52c9666:::obj5:7#, (61) No data available 2024-09-15T22:04:21.844 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:146: create_scenario: JSON='["1.0",{"oid":"obj7","key":"","snapid":-2,"hash":1552453721,"max":0,"pool":1,"namespace":"","max":0}]' 2024-09-15T22:04:21.845 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:147: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 '["1.0",{"oid":"obj7","key":"","snapid":-2,"hash":1552453721,"max":0,"pool":1,"namespace":"","max":0}]' clear-snapset corrupt 2024-09-15T22:04:23.670 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:148: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 --head --op list obj8 2024-09-15T22:04:23.999 INFO:tasks.workunit.client.0.smithi136.stderr:Error getting attr on : 1.0_head,#1:c52c9666:::obj5:7#, (61) No data available 2024-09-15T22:04:24.263 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:148: create_scenario: JSON='["1.0",{"oid":"obj8","key":"","snapid":-2,"hash":2381834917,"max":0,"pool":1,"namespace":"","max":0}]' 2024-09-15T22:04:24.263 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:149: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 '["1.0",{"oid":"obj8","key":"","snapid":-2,"hash":2381834917,"max":0,"pool":1,"namespace":"","max":0}]' clear-snapset seq 2024-09-15T22:04:25.713 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:150: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 --head --op list obj9 2024-09-15T22:04:26.057 INFO:tasks.workunit.client.0.smithi136.stderr:Error getting attr on : 1.0_head,#1:c52c9666:::obj5:7#, (61) No data available 2024-09-15T22:04:26.326 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:150: create_scenario: JSON='["1.0",{"oid":"obj9","key":"","snapid":-2,"hash":3833113727,"max":0,"pool":1,"namespace":"","max":0}]' 2024-09-15T22:04:26.327 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:151: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 '["1.0",{"oid":"obj9","key":"","snapid":-2,"hash":3833113727,"max":0,"pool":1,"namespace":"","max":0}]' clear-snapset clone_size 2024-09-15T22:04:27.749 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:152: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 --head --op list obj10 2024-09-15T22:04:28.073 INFO:tasks.workunit.client.0.smithi136.stderr:Error getting attr on : 1.0_head,#1:c52c9666:::obj5:7#, (61) No data available 2024-09-15T22:04:28.340 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:152: create_scenario: JSON='["1.0",{"oid":"obj10","key":"","snapid":-2,"hash":718195851,"max":0,"pool":1,"namespace":"","max":0}]' 2024-09-15T22:04:28.340 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:153: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 '["1.0",{"oid":"obj10","key":"","snapid":-2,"hash":718195851,"max":0,"pool":1,"namespace":"","max":0}]' clear-snapset clone_overlap 2024-09-15T22:04:30.044 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:154: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 --head --op list obj11 2024-09-15T22:04:30.380 INFO:tasks.workunit.client.0.smithi136.stderr:Error getting attr on : 1.0_head,#1:c52c9666:::obj5:7#, (61) No data available 2024-09-15T22:04:30.647 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:154: create_scenario: JSON='["1.0",{"oid":"obj11","key":"","snapid":-2,"hash":693400951,"max":0,"pool":1,"namespace":"","max":0}]' 2024-09-15T22:04:30.647 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:155: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 '["1.0",{"oid":"obj11","key":"","snapid":-2,"hash":693400951,"max":0,"pool":1,"namespace":"","max":0}]' clear-snapset clones 2024-09-15T22:04:32.074 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:156: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 --head --op list obj12 2024-09-15T22:04:32.439 INFO:tasks.workunit.client.0.smithi136.stderr:Error getting attr on : 1.0_head,#1:c52c9666:::obj5:7#, (61) No data available 2024-09-15T22:04:32.703 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:156: create_scenario: JSON='["1.0",{"oid":"obj12","key":"","snapid":-2,"hash":3551132405,"max":0,"pool":1,"namespace":"","max":0}]' 2024-09-15T22:04:32.704 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:157: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 '["1.0",{"oid":"obj12","key":"","snapid":-2,"hash":3551132405,"max":0,"pool":1,"namespace":"","max":0}]' clear-snapset head 2024-09-15T22:04:34.133 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:158: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 --head --op list obj13 2024-09-15T22:04:34.464 INFO:tasks.workunit.client.0.smithi136.stderr:Error getting attr on : 1.0_head,#1:c52c9666:::obj5:7#, (61) No data available 2024-09-15T22:04:34.728 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:158: create_scenario: JSON='["1.0",{"oid":"obj13","key":"","snapid":-2,"hash":2087409765,"max":0,"pool":1,"namespace":"","max":0}]' 2024-09-15T22:04:34.729 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:159: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 '["1.0",{"oid":"obj13","key":"","snapid":-2,"hash":2087409765,"max":0,"pool":1,"namespace":"","max":0}]' clear-snapset snaps 2024-09-15T22:04:36.152 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:160: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 --head --op list obj14 2024-09-15T22:04:36.486 INFO:tasks.workunit.client.0.smithi136.stderr:Error getting attr on : 1.0_head,#1:c52c9666:::obj5:7#, (61) No data available 2024-09-15T22:04:36.750 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:160: create_scenario: JSON='["1.0",{"oid":"obj14","key":"","snapid":-2,"hash":2484217095,"max":0,"pool":1,"namespace":"","max":0}]' 2024-09-15T22:04:36.751 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:161: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 '["1.0",{"oid":"obj14","key":"","snapid":-2,"hash":2484217095,"max":0,"pool":1,"namespace":"","max":0}]' clear-snapset size 2024-09-15T22:04:38.446 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:163: create_scenario: echo garbage 2024-09-15T22:04:38.446 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:164: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 --head --op list obj15 2024-09-15T22:04:38.772 INFO:tasks.workunit.client.0.smithi136.stderr:Error getting attr on : 1.0_head,#1:c52c9666:::obj5:7#, (61) No data available 2024-09-15T22:04:39.037 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:164: create_scenario: JSON='["1.0",{"oid":"obj15","key":"","snapid":-2,"hash":612772309,"max":0,"pool":1,"namespace":"","max":0}]' 2024-09-15T22:04:39.038 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:165: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 '["1.0",{"oid":"obj15","key":"","snapid":-2,"hash":612772309,"max":0,"pool":1,"namespace":"","max":0}]' set-attr snapset td/osd-scrub-snaps/bad 2024-09-15T22:04:40.514 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:166: create_scenario: rm -f td/osd-scrub-snaps/bad 2024-09-15T22:04:40.515 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:167: create_scenario: return 0 2024-09-15T22:04:40.515 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:767: _scrub_snaps_multi: rm -f testdata.463062 2024-09-15T22:04:40.517 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:769: _scrub_snaps_multi: expr 2 - 1 2024-09-15T22:04:40.518 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:769: _scrub_snaps_multi: seq 0 1 2024-09-15T22:04:40.522 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:769: _scrub_snaps_multi: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-09-15T22:04:40.522 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:771: _scrub_snaps_multi: activate_osd td/osd-scrub-snaps 0 2024-09-15T22:04:40.522 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-snaps 2024-09-15T22:04:40.522 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-15T22:04:40.522 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=0 2024-09-15T22:04:40.522 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-15T22:04:40.522 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-snaps/0 2024-09-15T22:04:40.522 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=0098e432-2807-49bf-8c16-1e2a347d825d --auth-supported=none --mon-host=127.0.0.1:7121 --osd_scrub_chunk_min=3 --osd_scrub_chunk_max=20 --osd_shallow_scrub_chunk_min=3 --osd_shallow_scrub_chunk_max=3 --osd_pg_stat_report_interval_max_seconds=1 --osd_pg_stat_report_interval_max_epochs=1' 2024-09-15T22:04:40.522 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T22:04:40.522 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T22:04:40.522 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T22:04:40.522 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-snaps/0' 2024-09-15T22:04:40.523 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-snaps/0/journal' 2024-09-15T22:04:40.523 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-15T22:04:40.523 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-15T22:04:40.523 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-snaps' 2024-09-15T22:04:40.523 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-15T22:04:40.523 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-15T22:04:40.523 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-15T22:04:40.523 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-15T22:04:40.523 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T22:04:40.523 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.463062 2024-09-15T22:04:40.523 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.463062/$cluster-$name.asok' 2024-09-15T22:04:40.523 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.463062/$cluster-$name.asok' 2024-09-15T22:04:40.523 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-15T22:04:40.523 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-snaps/$name.log' 2024-09-15T22:04:40.524 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-snaps/$name.pid' 2024-09-15T22:04:40.524 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T22:04:40.524 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T22:04:40.524 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T22:04:40.524 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T22:04:40.524 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-15T22:04:40.524 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-15T22:04:40.524 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-snaps/0 2024-09-15T22:04:40.524 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.0 2024-09-15T22:04:40.525 INFO:tasks.workunit.client.0.smithi136.stdout:start osd.0 2024-09-15T22:04:40.525 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 0 --fsid=0098e432-2807-49bf-8c16-1e2a347d825d --auth-supported=none --mon-host=127.0.0.1:7121 --osd_scrub_chunk_min=3 --osd_scrub_chunk_max=20 --osd_shallow_scrub_chunk_min=3 --osd_shallow_scrub_chunk_max=3 --osd_pg_stat_report_interval_max_seconds=1 --osd_pg_stat_report_interval_max_epochs=1 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-snaps/0 --osd-journal=td/osd-scrub-snaps/0/journal --chdir= --run-dir=td/osd-scrub-snaps '--admin-socket=/tmp/ceph-asok.463062/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-snaps/$name.log' '--pid-file=td/osd-scrub-snaps/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T22:04:40.525 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-snaps/0/whoami 2024-09-15T22:04:40.526 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 0 = 0 ']' 2024-09-15T22:04:40.526 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-15T22:04:40.526 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-15T22:04:40.526 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-15T22:04:40.563 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T22:04:40.561+0000 7f8ee9ea5640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T22:04:40.569 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T22:04:40.568+0000 7f8ee9ea5640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T22:04:40.572 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T22:04:40.569+0000 7f8ee9ea5640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T22:04:40.821 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 0 2024-09-15T22:04:40.821 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-15T22:04:40.821 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2024-09-15T22:04:40.821 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-15T22:04:40.821 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-15T22:04:40.821 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:04:40.821 INFO:tasks.workunit.client.0.smithi136.stdout:0 2024-09-15T22:04:40.822 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-15T22:04:40.822 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:04:40.822 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T22:04:41.109 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:04:42.110 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:04:42.110 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:04:42.110 INFO:tasks.workunit.client.0.smithi136.stdout:1 2024-09-15T22:04:42.111 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-15T22:04:42.111 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:04:42.111 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T22:04:42.160 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T22:04:42.159+0000 7f8ee9ea5640 -1 Falling back to public interface 2024-09-15T22:04:42.421 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:04:43.423 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:04:43.423 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:04:43.423 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-15T22:04:43.423 INFO:tasks.workunit.client.0.smithi136.stdout:2 2024-09-15T22:04:43.424 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:04:43.424 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T22:04:43.717 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:04:44.718 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:04:44.719 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:04:44.719 INFO:tasks.workunit.client.0.smithi136.stdout:3 2024-09-15T22:04:44.719 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-15T22:04:44.719 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:04:44.720 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T22:04:44.921 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T22:04:44.919+0000 7f8ee9ea5640 -1 osd.0 23 log_to_monitors true 2024-09-15T22:04:45.022 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:04:46.023 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:04:46.023 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:04:46.023 INFO:tasks.workunit.client.0.smithi136.stdout:4 2024-09-15T22:04:46.024 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-15T22:04:46.024 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:04:46.024 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T22:04:46.317 INFO:tasks.workunit.client.0.smithi136.stdout:osd.0 up in weight 1 up_from 26 up_thru 26 down_at 24 last_clean_interval [5,23) [v2:127.0.0.1:6802/2129161786,v1:127.0.0.1:6803/2129161786] [v2:127.0.0.1:6804/2129161786,v1:127.0.0.1:6805/2129161786] exists,up 396cb73c-0737-430f-add9-356620de3f92 2024-09-15T22:04:46.321 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-15T22:04:46.321 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-15T22:04:46.321 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-15T22:04:46.321 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:769: _scrub_snaps_multi: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-09-15T22:04:46.321 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:771: _scrub_snaps_multi: activate_osd td/osd-scrub-snaps 1 2024-09-15T22:04:46.321 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-snaps 2024-09-15T22:04:46.321 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-15T22:04:46.321 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=1 2024-09-15T22:04:46.322 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-15T22:04:46.322 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-snaps/1 2024-09-15T22:04:46.322 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=0098e432-2807-49bf-8c16-1e2a347d825d --auth-supported=none --mon-host=127.0.0.1:7121 --osd_scrub_chunk_min=3 --osd_scrub_chunk_max=20 --osd_shallow_scrub_chunk_min=3 --osd_shallow_scrub_chunk_max=3 --osd_pg_stat_report_interval_max_seconds=1 --osd_pg_stat_report_interval_max_epochs=1' 2024-09-15T22:04:46.322 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T22:04:46.322 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T22:04:46.322 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T22:04:46.322 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-snaps/1' 2024-09-15T22:04:46.322 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-snaps/1/journal' 2024-09-15T22:04:46.322 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-15T22:04:46.322 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-15T22:04:46.322 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-snaps' 2024-09-15T22:04:46.322 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-15T22:04:46.322 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-15T22:04:46.322 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-15T22:04:46.323 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-15T22:04:46.323 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T22:04:46.323 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.463062 2024-09-15T22:04:46.323 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.463062/$cluster-$name.asok' 2024-09-15T22:04:46.323 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.463062/$cluster-$name.asok' 2024-09-15T22:04:46.323 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-15T22:04:46.323 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-snaps/$name.log' 2024-09-15T22:04:46.323 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-snaps/$name.pid' 2024-09-15T22:04:46.323 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T22:04:46.323 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T22:04:46.323 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T22:04:46.323 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T22:04:46.323 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-15T22:04:46.323 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-15T22:04:46.324 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-snaps/1 2024-09-15T22:04:46.324 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.1 2024-09-15T22:04:46.324 INFO:tasks.workunit.client.0.smithi136.stdout:start osd.1 2024-09-15T22:04:46.325 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 1 --fsid=0098e432-2807-49bf-8c16-1e2a347d825d --auth-supported=none --mon-host=127.0.0.1:7121 --osd_scrub_chunk_min=3 --osd_scrub_chunk_max=20 --osd_shallow_scrub_chunk_min=3 --osd_shallow_scrub_chunk_max=3 --osd_pg_stat_report_interval_max_seconds=1 --osd_pg_stat_report_interval_max_epochs=1 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-snaps/1 --osd-journal=td/osd-scrub-snaps/1/journal --chdir= --run-dir=td/osd-scrub-snaps '--admin-socket=/tmp/ceph-asok.463062/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-snaps/$name.log' '--pid-file=td/osd-scrub-snaps/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T22:04:46.325 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-snaps/1/whoami 2024-09-15T22:04:46.325 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 1 = 1 ']' 2024-09-15T22:04:46.325 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-15T22:04:46.325 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-15T22:04:46.325 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-15T22:04:46.359 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T22:04:46.357+0000 7f8dcd40d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T22:04:46.387 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T22:04:46.386+0000 7f8dcd40d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T22:04:46.389 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T22:04:46.387+0000 7f8dcd40d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T22:04:46.627 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 1 2024-09-15T22:04:46.627 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-15T22:04:46.627 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2024-09-15T22:04:46.627 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-15T22:04:46.627 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-15T22:04:46.627 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:04:46.627 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-15T22:04:46.628 INFO:tasks.workunit.client.0.smithi136.stdout:0 2024-09-15T22:04:46.628 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:04:46.628 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T22:04:46.915 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:04:47.755 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T22:04:47.754+0000 7f8dcd40d640 -1 Falling back to public interface 2024-09-15T22:04:47.917 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:04:47.917 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:04:47.917 INFO:tasks.workunit.client.0.smithi136.stdout:1 2024-09-15T22:04:47.918 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-15T22:04:47.918 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:04:47.918 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T22:04:48.209 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:04:49.210 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:04:49.210 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:04:49.211 INFO:tasks.workunit.client.0.smithi136.stdout:2 2024-09-15T22:04:49.211 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-15T22:04:49.211 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:04:49.211 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T22:04:49.504 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:04:50.505 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:04:50.505 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:04:50.505 INFO:tasks.workunit.client.0.smithi136.stdout:3 2024-09-15T22:04:50.506 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-15T22:04:50.506 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:04:50.506 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T22:04:50.886 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:04:51.271 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T22:04:51.269+0000 7f8dcd40d640 -1 osd.1 23 log_to_monitors true 2024-09-15T22:04:51.888 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:04:51.888 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:04:51.888 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-15T22:04:51.888 INFO:tasks.workunit.client.0.smithi136.stdout:4 2024-09-15T22:04:51.889 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:04:51.889 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T22:04:52.213 INFO:tasks.workunit.client.0.smithi136.stdout:osd.1 up in weight 1 up_from 29 up_thru 29 down_at 24 last_clean_interval [10,23) [v2:127.0.0.1:6810/1406138721,v1:127.0.0.1:6811/1406138721] [v2:127.0.0.1:6812/1406138721,v1:127.0.0.1:6813/1406138721] exists,up 7a834e93-3154-4b8d-b285-bcb744b7a4d2 2024-09-15T22:04:52.213 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-15T22:04:52.213 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-15T22:04:52.214 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-15T22:04:52.214 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:774: _scrub_snaps_multi: ceph tell 'osd.*' config set osd_shallow_scrub_chunk_max 3 2024-09-15T22:04:52.321 INFO:tasks.workunit.client.0.smithi136.stdout:osd.0: { 2024-09-15T22:04:52.321 INFO:tasks.workunit.client.0.smithi136.stdout: "success": "osd_delete_sleep = '' osd_delete_sleep_hdd = '' osd_delete_sleep_hybrid = '' osd_delete_sleep_ssd = '' osd_max_backfills = '' osd_recovery_max_active = '' osd_recovery_max_active_hdd = '' osd_recovery_max_active_ssd = '' osd_recovery_sleep = '' osd_recovery_sleep_hdd = '' osd_recovery_sleep_hybrid = '' osd_recovery_sleep_ssd = '' osd_scrub_sleep = '' osd_shallow_scrub_chunk_max = '' (not observed, change may require restart) osd_snap_trim_sleep = '' osd_snap_trim_sleep_hdd = '' osd_snap_trim_sleep_hybrid = '' osd_snap_trim_sleep_ssd = '' " 2024-09-15T22:04:52.321 INFO:tasks.workunit.client.0.smithi136.stdout:} 2024-09-15T22:04:52.334 INFO:tasks.workunit.client.0.smithi136.stdout:osd.1: { 2024-09-15T22:04:52.335 INFO:tasks.workunit.client.0.smithi136.stdout: "success": "osd_delete_sleep = '' osd_delete_sleep_hdd = '' osd_delete_sleep_hybrid = '' osd_delete_sleep_ssd = '' osd_max_backfills = '' osd_recovery_max_active = '' osd_recovery_max_active_hdd = '' osd_recovery_max_active_ssd = '' osd_recovery_sleep = '' osd_recovery_sleep_hdd = '' osd_recovery_sleep_hybrid = '' osd_recovery_sleep_ssd = '' osd_scrub_sleep = '' osd_shallow_scrub_chunk_max = '' (not observed, change may require restart) osd_snap_trim_sleep = '' osd_snap_trim_sleep_hdd = '' osd_snap_trim_sleep_hybrid = '' osd_snap_trim_sleep_ssd = '' " 2024-09-15T22:04:52.335 INFO:tasks.workunit.client.0.smithi136.stdout:} 2024-09-15T22:04:52.344 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:775: _scrub_snaps_multi: ceph tell 'osd.*' config set osd_shallow_scrub_chunk_min 3 2024-09-15T22:04:52.501 INFO:tasks.workunit.client.0.smithi136.stdout:osd.0: { 2024-09-15T22:04:52.501 INFO:tasks.workunit.client.0.smithi136.stdout: "success": "osd_shallow_scrub_chunk_min = '' (not observed, change may require restart) " 2024-09-15T22:04:52.501 INFO:tasks.workunit.client.0.smithi136.stdout:} 2024-09-15T22:04:52.515 INFO:tasks.workunit.client.0.smithi136.stdout:osd.1: { 2024-09-15T22:04:52.515 INFO:tasks.workunit.client.0.smithi136.stdout: "success": "osd_shallow_scrub_chunk_min = '' (not observed, change may require restart) " 2024-09-15T22:04:52.515 INFO:tasks.workunit.client.0.smithi136.stdout:} 2024-09-15T22:04:52.524 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:776: _scrub_snaps_multi: ceph tell 'osd.*' config set osd_scrub_chunk_min 3 2024-09-15T22:04:52.632 INFO:tasks.workunit.client.0.smithi136.stdout:osd.0: { 2024-09-15T22:04:52.632 INFO:tasks.workunit.client.0.smithi136.stdout: "success": "osd_scrub_chunk_min = '' (not observed, change may require restart) " 2024-09-15T22:04:52.632 INFO:tasks.workunit.client.0.smithi136.stdout:} 2024-09-15T22:04:52.645 INFO:tasks.workunit.client.0.smithi136.stdout:osd.1: { 2024-09-15T22:04:52.645 INFO:tasks.workunit.client.0.smithi136.stdout: "success": "osd_scrub_chunk_min = '' (not observed, change may require restart) " 2024-09-15T22:04:52.645 INFO:tasks.workunit.client.0.smithi136.stdout:} 2024-09-15T22:04:52.654 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:777: _scrub_snaps_multi: ceph tell 'osd.*' config set osd_pg_stat_report_interval_max_seconds 1 2024-09-15T22:04:52.770 INFO:tasks.workunit.client.0.smithi136.stdout:osd.0: { 2024-09-15T22:04:52.770 INFO:tasks.workunit.client.0.smithi136.stdout: "success": "osd_pg_stat_report_interval_max_seconds = '' (not observed, change may require restart) " 2024-09-15T22:04:52.770 INFO:tasks.workunit.client.0.smithi136.stdout:} 2024-09-15T22:04:52.783 INFO:tasks.workunit.client.0.smithi136.stdout:osd.1: { 2024-09-15T22:04:52.783 INFO:tasks.workunit.client.0.smithi136.stdout: "success": "osd_pg_stat_report_interval_max_seconds = '' (not observed, change may require restart) " 2024-09-15T22:04:52.783 INFO:tasks.workunit.client.0.smithi136.stdout:} 2024-09-15T22:04:52.792 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:778: _scrub_snaps_multi: ceph tell 'osd.*' config set osd_pg_stat_report_interval_max_epochs 1 2024-09-15T22:04:52.921 INFO:tasks.workunit.client.0.smithi136.stdout:osd.0: { 2024-09-15T22:04:52.921 INFO:tasks.workunit.client.0.smithi136.stdout: "success": "osd_pg_stat_report_interval_max_epochs = '' (not observed, change may require restart) " 2024-09-15T22:04:52.921 INFO:tasks.workunit.client.0.smithi136.stdout:} 2024-09-15T22:04:52.934 INFO:tasks.workunit.client.0.smithi136.stdout:osd.1: { 2024-09-15T22:04:52.934 INFO:tasks.workunit.client.0.smithi136.stdout: "success": "osd_pg_stat_report_interval_max_epochs = '' (not observed, change may require restart) " 2024-09-15T22:04:52.935 INFO:tasks.workunit.client.0.smithi136.stdout:} 2024-09-15T22:04:52.943 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:779: _scrub_snaps_multi: wait_for_clean 2024-09-15T22:04:52.944 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-15T22:04:52.944 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-15T22:04:52.944 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-15T22:04:52.944 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-15T22:04:52.945 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-15T22:04:52.945 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-15T22:04:52.945 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-15T22:04:52.945 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-15T22:04:52.945 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-15T22:04:53.033 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-15T22:04:53.033 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-15T22:04:53.033 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-15T22:04:53.033 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-15T22:04:53.033 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-15T22:04:53.033 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-15T22:04:53.321 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-15T22:04:53.322 INFO:tasks.workunit.client.0.smithi136.stderr:1' 2024-09-15T22:04:53.322 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-15T22:04:53.322 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T22:04:53.322 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-15T22:04:53.466 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=111669149699 2024-09-15T22:04:53.466 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 111669149699 2024-09-15T22:04:53.466 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-111669149699' 2024-09-15T22:04:53.467 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T22:04:53.467 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-15T22:04:53.582 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=124554051586 2024-09-15T22:04:53.582 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 124554051586 2024-09-15T22:04:53.582 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-111669149699 1-124554051586' 2024-09-15T22:04:53.582 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T22:04:53.582 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-111669149699 2024-09-15T22:04:53.583 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T22:04:53.583 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-15T22:04:53.584 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-111669149699 2024-09-15T22:04:53.584 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T22:04:53.585 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=111669149699 2024-09-15T22:04:53.585 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 111669149699' 2024-09-15T22:04:53.585 INFO:tasks.workunit.client.0.smithi136.stdout:waiting osd.0 seq 111669149699 2024-09-15T22:04:53.585 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T22:04:53.869 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 111669149698 -lt 111669149699 2024-09-15T22:04:53.869 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-15T22:04:54.870 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-15T22:04:54.871 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T22:04:55.164 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 111669149699 -lt 111669149699 2024-09-15T22:04:55.164 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T22:04:55.165 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-124554051586 2024-09-15T22:04:55.165 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T22:04:55.166 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-15T22:04:55.166 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-124554051586 2024-09-15T22:04:55.166 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T22:04:55.167 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=124554051586 2024-09-15T22:04:55.167 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 124554051586' 2024-09-15T22:04:55.168 INFO:tasks.workunit.client.0.smithi136.stdout:waiting osd.1 seq 124554051586 2024-09-15T22:04:55.168 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-15T22:04:55.489 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 124554051586 -lt 124554051586 2024-09-15T22:04:55.490 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-15T22:04:55.490 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T22:04:55.490 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T22:04:55.848 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 1 == 0 2024-09-15T22:04:55.848 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-15T22:04:55.848 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-15T22:04:55.848 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-15T22:04:55.848 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T22:04:55.848 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T22:04:55.848 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T22:04:55.849 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T22:04:56.171 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=1 2024-09-15T22:04:56.172 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-15T22:04:56.172 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T22:04:56.172 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T22:04:56.528 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 = 1 2024-09-15T22:04:56.528 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-15T22:04:56.528 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-15T22:04:56.528 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:781: _scrub_snaps_multi: local pgid=1.0 2024-09-15T22:04:56.528 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:782: _scrub_snaps_multi: pg_scrub 1.0 2024-09-15T22:04:56.528 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1883: pg_scrub: local pgid=1.0 2024-09-15T22:04:56.528 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1884: pg_scrub: get_last_scrub_stamp 1.0 2024-09-15T22:04:56.529 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-15T22:04:56.529 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-15T22:04:56.529 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-15T22:04:56.529 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-15T22:04:56.814 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1884: pg_scrub: local last_scrub=2024-09-15T22:03:41.490650+0000 2024-09-15T22:04:56.814 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1885: pg_scrub: ceph pg scrub 1.0 2024-09-15T22:04:57.088 INFO:tasks.workunit.client.0.smithi136.stderr:instructing pg 1.0 on osd.1 to scrub 2024-09-15T22:04:57.100 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1886: pg_scrub: wait_for_scrub 1.0 2024-09-15T22:03:41.490650+0000 2024-09-15T22:04:57.100 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: local pgid=1.0 2024-09-15T22:04:57.100 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2018: wait_for_scrub: local last_scrub=2024-09-15T22:03:41.490650+0000 2024-09-15T22:04:57.100 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2019: wait_for_scrub: local sname=last_scrub_stamp 2024-09-15T22:04:57.100 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i=0 )) 2024-09-15T22:04:57.100 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-15T22:04:57.100 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-15T22:04:57.101 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-15T22:04:57.101 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-15T22:04:57.101 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-15T22:04:57.101 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-15T22:04:57.385 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-15T22:03:41.490650+0000 '>' 2024-09-15T22:03:41.490650+0000 2024-09-15T22:04:57.385 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-15T22:04:58.386 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-15T22:04:58.386 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-15T22:04:58.387 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-15T22:04:58.387 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-15T22:04:58.387 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-15T22:04:58.387 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-15T22:04:58.387 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-15T22:04:58.694 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-15T22:03:41.490650+0000 '>' 2024-09-15T22:03:41.490650+0000 2024-09-15T22:04:58.694 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-15T22:04:59.695 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-15T22:04:59.695 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-15T22:04:59.696 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-15T22:04:59.696 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-15T22:04:59.696 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-15T22:04:59.696 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-15T22:04:59.696 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-15T22:04:59.994 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-15T22:03:41.490650+0000 '>' 2024-09-15T22:03:41.490650+0000 2024-09-15T22:04:59.994 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-15T22:05:00.995 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-15T22:05:00.995 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-15T22:05:00.996 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-15T22:05:00.996 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-15T22:05:00.996 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-15T22:05:00.996 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-15T22:05:00.996 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-15T22:05:01.291 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-15T22:03:41.490650+0000 '>' 2024-09-15T22:03:41.490650+0000 2024-09-15T22:05:01.291 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-15T22:05:02.292 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-15T22:05:02.292 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-15T22:05:02.293 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-15T22:05:02.293 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-15T22:05:02.293 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-15T22:05:02.293 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-15T22:05:02.293 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-15T22:05:02.612 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-15T22:04:57.936033+0000 '>' 2024-09-15T22:03:41.490650+0000 2024-09-15T22:05:02.612 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2023: wait_for_scrub: return 0 2024-09-15T22:05:02.613 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:786: _scrub_snaps_multi: grep '_scan_snaps start' td/osd-scrub-snaps/osd.1.log 2024-09-15T22:05:02.613 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:786: _scrub_snaps_multi: wc -l 2024-09-15T22:05:02.616 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:786: _scrub_snaps_multi: test 16 -gt 3 2024-09-15T22:05:02.616 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:787: _scrub_snaps_multi: grep '_scan_snaps start' td/osd-scrub-snaps/osd.0.log 2024-09-15T22:05:02.616 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:787: _scrub_snaps_multi: wc -l 2024-09-15T22:05:02.619 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:787: _scrub_snaps_multi: test 16 -gt 3 2024-09-15T22:05:02.619 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:789: _scrub_snaps_multi: rados list-inconsistent-pg test 2024-09-15T22:05:02.655 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:791: _scrub_snaps_multi: jq '. | length' td/osd-scrub-snaps/json 2024-09-15T22:05:02.657 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:791: _scrub_snaps_multi: test 1 = 1 2024-09-15T22:05:02.657 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:793: _scrub_snaps_multi: jq -r '.[0]' td/osd-scrub-snaps/json 2024-09-15T22:05:02.660 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:793: _scrub_snaps_multi: test 1.0 = 1.0 2024-09-15T22:05:02.660 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:795: _scrub_snaps_multi: rados list-inconsistent-obj 1.0 --format=json-pretty 2024-09-15T22:05:02.695 INFO:tasks.workunit.client.0.smithi136.stdout:{ 2024-09-15T22:05:02.695 INFO:tasks.workunit.client.0.smithi136.stdout: "epoch": 29, 2024-09-15T22:05:02.695 INFO:tasks.workunit.client.0.smithi136.stdout: "inconsistents": [ 2024-09-15T22:05:02.695 INFO:tasks.workunit.client.0.smithi136.stdout: { 2024-09-15T22:05:02.695 INFO:tasks.workunit.client.0.smithi136.stdout: "object": { 2024-09-15T22:05:02.695 INFO:tasks.workunit.client.0.smithi136.stdout: "name": "obj5", 2024-09-15T22:05:02.696 INFO:tasks.workunit.client.0.smithi136.stdout: "nspace": "", 2024-09-15T22:05:02.696 INFO:tasks.workunit.client.0.smithi136.stdout: "locator": "", 2024-09-15T22:05:02.696 INFO:tasks.workunit.client.0.smithi136.stdout: "snap": 1, 2024-09-15T22:05:02.696 INFO:tasks.workunit.client.0.smithi136.stdout: "version": 5 2024-09-15T22:05:02.696 INFO:tasks.workunit.client.0.smithi136.stdout: }, 2024-09-15T22:05:02.696 INFO:tasks.workunit.client.0.smithi136.stdout: "errors": [], 2024-09-15T22:05:02.696 INFO:tasks.workunit.client.0.smithi136.stdout: "union_shard_errors": [ 2024-09-15T22:05:02.696 INFO:tasks.workunit.client.0.smithi136.stdout: "missing" 2024-09-15T22:05:02.696 INFO:tasks.workunit.client.0.smithi136.stdout: ], 2024-09-15T22:05:02.696 INFO:tasks.workunit.client.0.smithi136.stdout: "selected_object_info": { 2024-09-15T22:05:02.696 INFO:tasks.workunit.client.0.smithi136.stdout: "oid": { 2024-09-15T22:05:02.696 INFO:tasks.workunit.client.0.smithi136.stdout: "oid": "obj5", 2024-09-15T22:05:02.696 INFO:tasks.workunit.client.0.smithi136.stdout: "key": "", 2024-09-15T22:05:02.697 INFO:tasks.workunit.client.0.smithi136.stdout: "snapid": 1, 2024-09-15T22:05:02.697 INFO:tasks.workunit.client.0.smithi136.stdout: "hash": 1718170787, 2024-09-15T22:05:02.697 INFO:tasks.workunit.client.0.smithi136.stdout: "max": 0, 2024-09-15T22:05:02.697 INFO:tasks.workunit.client.0.smithi136.stdout: "pool": 1, 2024-09-15T22:05:02.697 INFO:tasks.workunit.client.0.smithi136.stdout: "namespace": "" 2024-09-15T22:05:02.697 INFO:tasks.workunit.client.0.smithi136.stdout: }, 2024-09-15T22:05:02.697 INFO:tasks.workunit.client.0.smithi136.stdout: "version": "17'19", 2024-09-15T22:05:02.697 INFO:tasks.workunit.client.0.smithi136.stdout: "prior_version": "16'5", 2024-09-15T22:05:02.697 INFO:tasks.workunit.client.0.smithi136.stdout: "last_reqid": "client.4243.0:1", 2024-09-15T22:05:02.697 INFO:tasks.workunit.client.0.smithi136.stdout: "user_version": 5, 2024-09-15T22:05:02.697 INFO:tasks.workunit.client.0.smithi136.stdout: "size": 1032, 2024-09-15T22:05:02.697 INFO:tasks.workunit.client.0.smithi136.stdout: "mtime": "2024-09-15T22:03:47.126752+0000", 2024-09-15T22:05:02.697 INFO:tasks.workunit.client.0.smithi136.stdout: "local_mtime": "2024-09-15T22:03:47.127291+0000", 2024-09-15T22:05:02.697 INFO:tasks.workunit.client.0.smithi136.stdout: "lost": 0, 2024-09-15T22:05:02.698 INFO:tasks.workunit.client.0.smithi136.stdout: "flags": [ 2024-09-15T22:05:02.698 INFO:tasks.workunit.client.0.smithi136.stdout: "dirty", 2024-09-15T22:05:02.698 INFO:tasks.workunit.client.0.smithi136.stdout: "data_digest" 2024-09-15T22:05:02.698 INFO:tasks.workunit.client.0.smithi136.stdout: ], 2024-09-15T22:05:02.698 INFO:tasks.workunit.client.0.smithi136.stdout: "truncate_seq": 0, 2024-09-15T22:05:02.698 INFO:tasks.workunit.client.0.smithi136.stdout: "truncate_size": 0, 2024-09-15T22:05:02.698 INFO:tasks.workunit.client.0.smithi136.stdout: "data_digest": "0x0e25be29", 2024-09-15T22:05:02.698 INFO:tasks.workunit.client.0.smithi136.stdout: "omap_digest": "0xffffffff", 2024-09-15T22:05:02.698 INFO:tasks.workunit.client.0.smithi136.stdout: "expected_object_size": 0, 2024-09-15T22:05:02.698 INFO:tasks.workunit.client.0.smithi136.stdout: "expected_write_size": 0, 2024-09-15T22:05:02.698 INFO:tasks.workunit.client.0.smithi136.stdout: "alloc_hint_flags": 0, 2024-09-15T22:05:02.698 INFO:tasks.workunit.client.0.smithi136.stdout: "manifest": { 2024-09-15T22:05:02.698 INFO:tasks.workunit.client.0.smithi136.stdout: "type": 0 2024-09-15T22:05:02.698 INFO:tasks.workunit.client.0.smithi136.stdout: }, 2024-09-15T22:05:02.698 INFO:tasks.workunit.client.0.smithi136.stdout: "watchers": {} 2024-09-15T22:05:02.699 INFO:tasks.workunit.client.0.smithi136.stdout: }, 2024-09-15T22:05:02.699 INFO:tasks.workunit.client.0.smithi136.stdout: "shards": [ 2024-09-15T22:05:02.699 INFO:tasks.workunit.client.0.smithi136.stdout: { 2024-09-15T22:05:02.699 INFO:tasks.workunit.client.0.smithi136.stdout: "osd": 0, 2024-09-15T22:05:02.699 INFO:tasks.workunit.client.0.smithi136.stdout: "primary": false, 2024-09-15T22:05:02.699 INFO:tasks.workunit.client.0.smithi136.stdout: "errors": [ 2024-09-15T22:05:02.699 INFO:tasks.workunit.client.0.smithi136.stdout: "missing" 2024-09-15T22:05:02.699 INFO:tasks.workunit.client.0.smithi136.stdout: ] 2024-09-15T22:05:02.699 INFO:tasks.workunit.client.0.smithi136.stdout: }, 2024-09-15T22:05:02.699 INFO:tasks.workunit.client.0.smithi136.stdout: { 2024-09-15T22:05:02.699 INFO:tasks.workunit.client.0.smithi136.stdout: "osd": 1, 2024-09-15T22:05:02.699 INFO:tasks.workunit.client.0.smithi136.stdout: "primary": true, 2024-09-15T22:05:02.699 INFO:tasks.workunit.client.0.smithi136.stdout: "errors": [], 2024-09-15T22:05:02.700 INFO:tasks.workunit.client.0.smithi136.stdout: "size": 1032 2024-09-15T22:05:02.700 INFO:tasks.workunit.client.0.smithi136.stdout: } 2024-09-15T22:05:02.700 INFO:tasks.workunit.client.0.smithi136.stdout: ] 2024-09-15T22:05:02.700 INFO:tasks.workunit.client.0.smithi136.stdout: }, 2024-09-15T22:05:02.700 INFO:tasks.workunit.client.0.smithi136.stdout: { 2024-09-15T22:05:02.700 INFO:tasks.workunit.client.0.smithi136.stdout: "object": { 2024-09-15T22:05:02.700 INFO:tasks.workunit.client.0.smithi136.stdout: "name": "obj5", 2024-09-15T22:05:02.700 INFO:tasks.workunit.client.0.smithi136.stdout: "nspace": "", 2024-09-15T22:05:02.700 INFO:tasks.workunit.client.0.smithi136.stdout: "locator": "", 2024-09-15T22:05:02.700 INFO:tasks.workunit.client.0.smithi136.stdout: "snap": 2, 2024-09-15T22:05:02.700 INFO:tasks.workunit.client.0.smithi136.stdout: "version": 20 2024-09-15T22:05:02.700 INFO:tasks.workunit.client.0.smithi136.stdout: }, 2024-09-15T22:05:02.700 INFO:tasks.workunit.client.0.smithi136.stdout: "errors": [], 2024-09-15T22:05:02.700 INFO:tasks.workunit.client.0.smithi136.stdout: "union_shard_errors": [ 2024-09-15T22:05:02.701 INFO:tasks.workunit.client.0.smithi136.stdout: "missing" 2024-09-15T22:05:02.701 INFO:tasks.workunit.client.0.smithi136.stdout: ], 2024-09-15T22:05:02.701 INFO:tasks.workunit.client.0.smithi136.stdout: "selected_object_info": { 2024-09-15T22:05:02.701 INFO:tasks.workunit.client.0.smithi136.stdout: "oid": { 2024-09-15T22:05:02.701 INFO:tasks.workunit.client.0.smithi136.stdout: "oid": "obj5", 2024-09-15T22:05:02.701 INFO:tasks.workunit.client.0.smithi136.stdout: "key": "", 2024-09-15T22:05:02.701 INFO:tasks.workunit.client.0.smithi136.stdout: "snapid": 2, 2024-09-15T22:05:02.701 INFO:tasks.workunit.client.0.smithi136.stdout: "hash": 1718170787, 2024-09-15T22:05:02.701 INFO:tasks.workunit.client.0.smithi136.stdout: "max": 0, 2024-09-15T22:05:02.701 INFO:tasks.workunit.client.0.smithi136.stdout: "pool": 1, 2024-09-15T22:05:02.701 INFO:tasks.workunit.client.0.smithi136.stdout: "namespace": "" 2024-09-15T22:05:02.701 INFO:tasks.workunit.client.0.smithi136.stdout: }, 2024-09-15T22:05:02.701 INFO:tasks.workunit.client.0.smithi136.stdout: "version": "18'41", 2024-09-15T22:05:02.701 INFO:tasks.workunit.client.0.smithi136.stdout: "prior_version": "17'20", 2024-09-15T22:05:02.702 INFO:tasks.workunit.client.0.smithi136.stdout: "last_reqid": "client.4291.0:1", 2024-09-15T22:05:02.702 INFO:tasks.workunit.client.0.smithi136.stdout: "user_version": 20, 2024-09-15T22:05:02.702 INFO:tasks.workunit.client.0.smithi136.stdout: "size": 256, 2024-09-15T22:05:02.702 INFO:tasks.workunit.client.0.smithi136.stdout: "mtime": "2024-09-15T22:03:50.156969+0000", 2024-09-15T22:05:02.702 INFO:tasks.workunit.client.0.smithi136.stdout: "local_mtime": "2024-09-15T22:03:50.157591+0000", 2024-09-15T22:05:02.702 INFO:tasks.workunit.client.0.smithi136.stdout: "lost": 0, 2024-09-15T22:05:02.702 INFO:tasks.workunit.client.0.smithi136.stdout: "flags": [ 2024-09-15T22:05:02.702 INFO:tasks.workunit.client.0.smithi136.stdout: "dirty", 2024-09-15T22:05:02.702 INFO:tasks.workunit.client.0.smithi136.stdout: "data_digest" 2024-09-15T22:05:02.702 INFO:tasks.workunit.client.0.smithi136.stdout: ], 2024-09-15T22:05:02.702 INFO:tasks.workunit.client.0.smithi136.stdout: "truncate_seq": 0, 2024-09-15T22:05:02.702 INFO:tasks.workunit.client.0.smithi136.stdout: "truncate_size": 0, 2024-09-15T22:05:02.702 INFO:tasks.workunit.client.0.smithi136.stdout: "data_digest": "0x8d0e1329", 2024-09-15T22:05:02.702 INFO:tasks.workunit.client.0.smithi136.stdout: "omap_digest": "0xffffffff", 2024-09-15T22:05:02.703 INFO:tasks.workunit.client.0.smithi136.stdout: "expected_object_size": 0, 2024-09-15T22:05:02.703 INFO:tasks.workunit.client.0.smithi136.stdout: "expected_write_size": 0, 2024-09-15T22:05:02.703 INFO:tasks.workunit.client.0.smithi136.stdout: "alloc_hint_flags": 0, 2024-09-15T22:05:02.703 INFO:tasks.workunit.client.0.smithi136.stdout: "manifest": { 2024-09-15T22:05:02.703 INFO:tasks.workunit.client.0.smithi136.stdout: "type": 0 2024-09-15T22:05:02.703 INFO:tasks.workunit.client.0.smithi136.stdout: }, 2024-09-15T22:05:02.703 INFO:tasks.workunit.client.0.smithi136.stdout: "watchers": {} 2024-09-15T22:05:02.703 INFO:tasks.workunit.client.0.smithi136.stdout: }, 2024-09-15T22:05:02.703 INFO:tasks.workunit.client.0.smithi136.stdout: "shards": [ 2024-09-15T22:05:02.703 INFO:tasks.workunit.client.0.smithi136.stdout: { 2024-09-15T22:05:02.703 INFO:tasks.workunit.client.0.smithi136.stdout: "osd": 0, 2024-09-15T22:05:02.703 INFO:tasks.workunit.client.0.smithi136.stdout: "primary": false, 2024-09-15T22:05:02.703 INFO:tasks.workunit.client.0.smithi136.stdout: "errors": [ 2024-09-15T22:05:02.703 INFO:tasks.workunit.client.0.smithi136.stdout: "missing" 2024-09-15T22:05:02.704 INFO:tasks.workunit.client.0.smithi136.stdout: ] 2024-09-15T22:05:02.704 INFO:tasks.workunit.client.0.smithi136.stdout: }, 2024-09-15T22:05:02.704 INFO:tasks.workunit.client.0.smithi136.stdout: { 2024-09-15T22:05:02.704 INFO:tasks.workunit.client.0.smithi136.stdout: "osd": 1, 2024-09-15T22:05:02.704 INFO:tasks.workunit.client.0.smithi136.stdout: "primary": true, 2024-09-15T22:05:02.704 INFO:tasks.workunit.client.0.smithi136.stdout: "errors": [], 2024-09-15T22:05:02.704 INFO:tasks.workunit.client.0.smithi136.stdout: "size": 256 2024-09-15T22:05:02.704 INFO:tasks.workunit.client.0.smithi136.stdout: } 2024-09-15T22:05:02.704 INFO:tasks.workunit.client.0.smithi136.stdout: ] 2024-09-15T22:05:02.704 INFO:tasks.workunit.client.0.smithi136.stdout: }, 2024-09-15T22:05:02.704 INFO:tasks.workunit.client.0.smithi136.stdout: { 2024-09-15T22:05:02.704 INFO:tasks.workunit.client.0.smithi136.stdout: "object": { 2024-09-15T22:05:02.704 INFO:tasks.workunit.client.0.smithi136.stdout: "name": "obj5", 2024-09-15T22:05:02.704 INFO:tasks.workunit.client.0.smithi136.stdout: "nspace": "", 2024-09-15T22:05:02.705 INFO:tasks.workunit.client.0.smithi136.stdout: "locator": "", 2024-09-15T22:05:02.705 INFO:tasks.workunit.client.0.smithi136.stdout: "snap": 4, 2024-09-15T22:05:02.705 INFO:tasks.workunit.client.0.smithi136.stdout: "version": 42 2024-09-15T22:05:02.705 INFO:tasks.workunit.client.0.smithi136.stdout: }, 2024-09-15T22:05:02.705 INFO:tasks.workunit.client.0.smithi136.stdout: "errors": [ 2024-09-15T22:05:02.705 INFO:tasks.workunit.client.0.smithi136.stdout: "size_mismatch" 2024-09-15T22:05:02.705 INFO:tasks.workunit.client.0.smithi136.stdout: ], 2024-09-15T22:05:02.705 INFO:tasks.workunit.client.0.smithi136.stdout: "union_shard_errors": [ 2024-09-15T22:05:02.705 INFO:tasks.workunit.client.0.smithi136.stdout: "size_mismatch_info", 2024-09-15T22:05:02.705 INFO:tasks.workunit.client.0.smithi136.stdout: "obj_size_info_mismatch" 2024-09-15T22:05:02.705 INFO:tasks.workunit.client.0.smithi136.stdout: ], 2024-09-15T22:05:02.705 INFO:tasks.workunit.client.0.smithi136.stdout: "selected_object_info": { 2024-09-15T22:05:02.705 INFO:tasks.workunit.client.0.smithi136.stdout: "oid": { 2024-09-15T22:05:02.706 INFO:tasks.workunit.client.0.smithi136.stdout: "oid": "obj5", 2024-09-15T22:05:02.706 INFO:tasks.workunit.client.0.smithi136.stdout: "key": "", 2024-09-15T22:05:02.706 INFO:tasks.workunit.client.0.smithi136.stdout: "snapid": 4, 2024-09-15T22:05:02.706 INFO:tasks.workunit.client.0.smithi136.stdout: "hash": 1718170787, 2024-09-15T22:05:02.706 INFO:tasks.workunit.client.0.smithi136.stdout: "max": 0, 2024-09-15T22:05:02.706 INFO:tasks.workunit.client.0.smithi136.stdout: "pool": 1, 2024-09-15T22:05:02.706 INFO:tasks.workunit.client.0.smithi136.stdout: "namespace": "" 2024-09-15T22:05:02.706 INFO:tasks.workunit.client.0.smithi136.stdout: }, 2024-09-15T22:05:02.706 INFO:tasks.workunit.client.0.smithi136.stdout: "version": "20'45", 2024-09-15T22:05:02.706 INFO:tasks.workunit.client.0.smithi136.stdout: "prior_version": "18'42", 2024-09-15T22:05:02.706 INFO:tasks.workunit.client.0.smithi136.stdout: "last_reqid": "client.4327.0:1", 2024-09-15T22:05:02.706 INFO:tasks.workunit.client.0.smithi136.stdout: "user_version": 42, 2024-09-15T22:05:02.706 INFO:tasks.workunit.client.0.smithi136.stdout: "size": 512, 2024-09-15T22:05:02.706 INFO:tasks.workunit.client.0.smithi136.stdout: "mtime": "2024-09-15T22:03:51.890487+0000", 2024-09-15T22:05:02.706 INFO:tasks.workunit.client.0.smithi136.stdout: "local_mtime": "2024-09-15T22:03:51.917406+0000", 2024-09-15T22:05:02.707 INFO:tasks.workunit.client.0.smithi136.stdout: "lost": 0, 2024-09-15T22:05:02.707 INFO:tasks.workunit.client.0.smithi136.stdout: "flags": [ 2024-09-15T22:05:02.707 INFO:tasks.workunit.client.0.smithi136.stdout: "dirty", 2024-09-15T22:05:02.707 INFO:tasks.workunit.client.0.smithi136.stdout: "data_digest" 2024-09-15T22:05:02.707 INFO:tasks.workunit.client.0.smithi136.stdout: ], 2024-09-15T22:05:02.707 INFO:tasks.workunit.client.0.smithi136.stdout: "truncate_seq": 0, 2024-09-15T22:05:02.707 INFO:tasks.workunit.client.0.smithi136.stdout: "truncate_size": 0, 2024-09-15T22:05:02.707 INFO:tasks.workunit.client.0.smithi136.stdout: "data_digest": "0x20d369d1", 2024-09-15T22:05:02.707 INFO:tasks.workunit.client.0.smithi136.stdout: "omap_digest": "0xffffffff", 2024-09-15T22:05:02.707 INFO:tasks.workunit.client.0.smithi136.stdout: "expected_object_size": 0, 2024-09-15T22:05:02.707 INFO:tasks.workunit.client.0.smithi136.stdout: "expected_write_size": 0, 2024-09-15T22:05:02.707 INFO:tasks.workunit.client.0.smithi136.stdout: "alloc_hint_flags": 0, 2024-09-15T22:05:02.707 INFO:tasks.workunit.client.0.smithi136.stdout: "manifest": { 2024-09-15T22:05:02.707 INFO:tasks.workunit.client.0.smithi136.stdout: "type": 0 2024-09-15T22:05:02.708 INFO:tasks.workunit.client.0.smithi136.stdout: }, 2024-09-15T22:05:02.708 INFO:tasks.workunit.client.0.smithi136.stdout: "watchers": {} 2024-09-15T22:05:02.708 INFO:tasks.workunit.client.0.smithi136.stdout: }, 2024-09-15T22:05:02.708 INFO:tasks.workunit.client.0.smithi136.stdout: "shards": [ 2024-09-15T22:05:02.708 INFO:tasks.workunit.client.0.smithi136.stdout: { 2024-09-15T22:05:02.708 INFO:tasks.workunit.client.0.smithi136.stdout: "osd": 0, 2024-09-15T22:05:02.708 INFO:tasks.workunit.client.0.smithi136.stdout: "primary": false, 2024-09-15T22:05:02.708 INFO:tasks.workunit.client.0.smithi136.stdout: "errors": [ 2024-09-15T22:05:02.708 INFO:tasks.workunit.client.0.smithi136.stdout: "size_mismatch_info", 2024-09-15T22:05:02.708 INFO:tasks.workunit.client.0.smithi136.stdout: "obj_size_info_mismatch" 2024-09-15T22:05:02.708 INFO:tasks.workunit.client.0.smithi136.stdout: ], 2024-09-15T22:05:02.708 INFO:tasks.workunit.client.0.smithi136.stdout: "size": 4608, 2024-09-15T22:05:02.708 INFO:tasks.workunit.client.0.smithi136.stdout: "object_info": { 2024-09-15T22:05:02.709 INFO:tasks.workunit.client.0.smithi136.stdout: "oid": { 2024-09-15T22:05:02.709 INFO:tasks.workunit.client.0.smithi136.stdout: "oid": "obj5", 2024-09-15T22:05:02.709 INFO:tasks.workunit.client.0.smithi136.stdout: "key": "", 2024-09-15T22:05:02.709 INFO:tasks.workunit.client.0.smithi136.stdout: "snapid": 4, 2024-09-15T22:05:02.709 INFO:tasks.workunit.client.0.smithi136.stdout: "hash": 1718170787, 2024-09-15T22:05:02.709 INFO:tasks.workunit.client.0.smithi136.stdout: "max": 0, 2024-09-15T22:05:02.709 INFO:tasks.workunit.client.0.smithi136.stdout: "pool": 1, 2024-09-15T22:05:02.709 INFO:tasks.workunit.client.0.smithi136.stdout: "namespace": "" 2024-09-15T22:05:02.709 INFO:tasks.workunit.client.0.smithi136.stdout: }, 2024-09-15T22:05:02.710 INFO:tasks.workunit.client.0.smithi136.stdout: "version": "20'45", 2024-09-15T22:05:02.710 INFO:tasks.workunit.client.0.smithi136.stdout: "prior_version": "18'42", 2024-09-15T22:05:02.710 INFO:tasks.workunit.client.0.smithi136.stdout: "last_reqid": "client.4327.0:1", 2024-09-15T22:05:02.710 INFO:tasks.workunit.client.0.smithi136.stdout: "user_version": 42, 2024-09-15T22:05:02.710 INFO:tasks.workunit.client.0.smithi136.stdout: "size": 512, 2024-09-15T22:05:02.710 INFO:tasks.workunit.client.0.smithi136.stdout: "mtime": "2024-09-15T22:03:51.890487+0000", 2024-09-15T22:05:02.710 INFO:tasks.workunit.client.0.smithi136.stdout: "local_mtime": "2024-09-15T22:03:51.917406+0000", 2024-09-15T22:05:02.710 INFO:tasks.workunit.client.0.smithi136.stdout: "lost": 0, 2024-09-15T22:05:02.710 INFO:tasks.workunit.client.0.smithi136.stdout: "flags": [ 2024-09-15T22:05:02.710 INFO:tasks.workunit.client.0.smithi136.stdout: "dirty", 2024-09-15T22:05:02.710 INFO:tasks.workunit.client.0.smithi136.stdout: "data_digest" 2024-09-15T22:05:02.710 INFO:tasks.workunit.client.0.smithi136.stdout: ], 2024-09-15T22:05:02.710 INFO:tasks.workunit.client.0.smithi136.stdout: "truncate_seq": 0, 2024-09-15T22:05:02.710 INFO:tasks.workunit.client.0.smithi136.stdout: "truncate_size": 0, 2024-09-15T22:05:02.711 INFO:tasks.workunit.client.0.smithi136.stdout: "data_digest": "0x20d369d1", 2024-09-15T22:05:02.711 INFO:tasks.workunit.client.0.smithi136.stdout: "omap_digest": "0xffffffff", 2024-09-15T22:05:02.711 INFO:tasks.workunit.client.0.smithi136.stdout: "expected_object_size": 0, 2024-09-15T22:05:02.711 INFO:tasks.workunit.client.0.smithi136.stdout: "expected_write_size": 0, 2024-09-15T22:05:02.711 INFO:tasks.workunit.client.0.smithi136.stdout: "alloc_hint_flags": 0, 2024-09-15T22:05:02.711 INFO:tasks.workunit.client.0.smithi136.stdout: "manifest": { 2024-09-15T22:05:02.711 INFO:tasks.workunit.client.0.smithi136.stdout: "type": 0 2024-09-15T22:05:02.711 INFO:tasks.workunit.client.0.smithi136.stdout: }, 2024-09-15T22:05:02.711 INFO:tasks.workunit.client.0.smithi136.stdout: "watchers": {} 2024-09-15T22:05:02.711 INFO:tasks.workunit.client.0.smithi136.stdout: } 2024-09-15T22:05:02.711 INFO:tasks.workunit.client.0.smithi136.stdout: }, 2024-09-15T22:05:02.711 INFO:tasks.workunit.client.0.smithi136.stdout: { 2024-09-15T22:05:02.711 INFO:tasks.workunit.client.0.smithi136.stdout: "osd": 1, 2024-09-15T22:05:02.711 INFO:tasks.workunit.client.0.smithi136.stdout: "primary": true, 2024-09-15T22:05:02.712 INFO:tasks.workunit.client.0.smithi136.stdout: "errors": [], 2024-09-15T22:05:02.712 INFO:tasks.workunit.client.0.smithi136.stdout: "size": 512 2024-09-15T22:05:02.712 INFO:tasks.workunit.client.0.smithi136.stdout: } 2024-09-15T22:05:02.712 INFO:tasks.workunit.client.0.smithi136.stdout: ] 2024-09-15T22:05:02.712 INFO:tasks.workunit.client.0.smithi136.stdout: }, 2024-09-15T22:05:02.712 INFO:tasks.workunit.client.0.smithi136.stdout: { 2024-09-15T22:05:02.712 INFO:tasks.workunit.client.0.smithi136.stdout: "object": { 2024-09-15T22:05:02.712 INFO:tasks.workunit.client.0.smithi136.stdout: "name": "obj4", 2024-09-15T22:05:02.712 INFO:tasks.workunit.client.0.smithi136.stdout: "nspace": "", 2024-09-15T22:05:02.712 INFO:tasks.workunit.client.0.smithi136.stdout: "locator": "", 2024-09-15T22:05:02.712 INFO:tasks.workunit.client.0.smithi136.stdout: "snap": 7, 2024-09-15T22:05:02.712 INFO:tasks.workunit.client.0.smithi136.stdout: "version": 4 2024-09-15T22:05:02.712 INFO:tasks.workunit.client.0.smithi136.stdout: }, 2024-09-15T22:05:02.712 INFO:tasks.workunit.client.0.smithi136.stdout: "errors": [], 2024-09-15T22:05:02.713 INFO:tasks.workunit.client.0.smithi136.stdout: "union_shard_errors": [ 2024-09-15T22:05:02.713 INFO:tasks.workunit.client.0.smithi136.stdout: "missing" 2024-09-15T22:05:02.713 INFO:tasks.workunit.client.0.smithi136.stdout: ], 2024-09-15T22:05:02.713 INFO:tasks.workunit.client.0.smithi136.stdout: "selected_object_info": { 2024-09-15T22:05:02.713 INFO:tasks.workunit.client.0.smithi136.stdout: "oid": { 2024-09-15T22:05:02.713 INFO:tasks.workunit.client.0.smithi136.stdout: "oid": "obj4", 2024-09-15T22:05:02.713 INFO:tasks.workunit.client.0.smithi136.stdout: "key": "", 2024-09-15T22:05:02.713 INFO:tasks.workunit.client.0.smithi136.stdout: "snapid": 7, 2024-09-15T22:05:02.713 INFO:tasks.workunit.client.0.smithi136.stdout: "hash": 2826278768, 2024-09-15T22:05:02.713 INFO:tasks.workunit.client.0.smithi136.stdout: "max": 0, 2024-09-15T22:05:02.713 INFO:tasks.workunit.client.0.smithi136.stdout: "pool": 1, 2024-09-15T22:05:02.713 INFO:tasks.workunit.client.0.smithi136.stdout: "namespace": "" 2024-09-15T22:05:02.713 INFO:tasks.workunit.client.0.smithi136.stdout: }, 2024-09-15T22:05:02.713 INFO:tasks.workunit.client.0.smithi136.stdout: "version": "23'51", 2024-09-15T22:05:02.713 INFO:tasks.workunit.client.0.smithi136.stdout: "prior_version": "16'4", 2024-09-15T22:05:02.714 INFO:tasks.workunit.client.0.smithi136.stdout: "last_reqid": "client.4240.0:1", 2024-09-15T22:05:02.714 INFO:tasks.workunit.client.0.smithi136.stdout: "user_version": 4, 2024-09-15T22:05:02.714 INFO:tasks.workunit.client.0.smithi136.stdout: "size": 1032, 2024-09-15T22:05:02.714 INFO:tasks.workunit.client.0.smithi136.stdout: "mtime": "2024-09-15T22:03:47.001394+0000", 2024-09-15T22:05:02.714 INFO:tasks.workunit.client.0.smithi136.stdout: "local_mtime": "2024-09-15T22:03:47.001968+0000", 2024-09-15T22:05:02.714 INFO:tasks.workunit.client.0.smithi136.stdout: "lost": 0, 2024-09-15T22:05:02.714 INFO:tasks.workunit.client.0.smithi136.stdout: "flags": [ 2024-09-15T22:05:02.714 INFO:tasks.workunit.client.0.smithi136.stdout: "dirty", 2024-09-15T22:05:02.714 INFO:tasks.workunit.client.0.smithi136.stdout: "data_digest" 2024-09-15T22:05:02.714 INFO:tasks.workunit.client.0.smithi136.stdout: ], 2024-09-15T22:05:02.714 INFO:tasks.workunit.client.0.smithi136.stdout: "truncate_seq": 0, 2024-09-15T22:05:02.714 INFO:tasks.workunit.client.0.smithi136.stdout: "truncate_size": 0, 2024-09-15T22:05:02.714 INFO:tasks.workunit.client.0.smithi136.stdout: "data_digest": "0x0e25be29", 2024-09-15T22:05:02.715 INFO:tasks.workunit.client.0.smithi136.stdout: "omap_digest": "0xffffffff", 2024-09-15T22:05:02.715 INFO:tasks.workunit.client.0.smithi136.stdout: "expected_object_size": 0, 2024-09-15T22:05:02.715 INFO:tasks.workunit.client.0.smithi136.stdout: "expected_write_size": 0, 2024-09-15T22:05:02.715 INFO:tasks.workunit.client.0.smithi136.stdout: "alloc_hint_flags": 0, 2024-09-15T22:05:02.715 INFO:tasks.workunit.client.0.smithi136.stdout: "manifest": { 2024-09-15T22:05:02.715 INFO:tasks.workunit.client.0.smithi136.stdout: "type": 0 2024-09-15T22:05:02.715 INFO:tasks.workunit.client.0.smithi136.stdout: }, 2024-09-15T22:05:02.715 INFO:tasks.workunit.client.0.smithi136.stdout: "watchers": {} 2024-09-15T22:05:02.715 INFO:tasks.workunit.client.0.smithi136.stdout: }, 2024-09-15T22:05:02.715 INFO:tasks.workunit.client.0.smithi136.stdout: "shards": [ 2024-09-15T22:05:02.715 INFO:tasks.workunit.client.0.smithi136.stdout: { 2024-09-15T22:05:02.715 INFO:tasks.workunit.client.0.smithi136.stdout: "osd": 0, 2024-09-15T22:05:02.715 INFO:tasks.workunit.client.0.smithi136.stdout: "primary": false, 2024-09-15T22:05:02.715 INFO:tasks.workunit.client.0.smithi136.stdout: "errors": [ 2024-09-15T22:05:02.716 INFO:tasks.workunit.client.0.smithi136.stdout: "missing" 2024-09-15T22:05:02.716 INFO:tasks.workunit.client.0.smithi136.stdout: ] 2024-09-15T22:05:02.716 INFO:tasks.workunit.client.0.smithi136.stdout: }, 2024-09-15T22:05:02.716 INFO:tasks.workunit.client.0.smithi136.stdout: { 2024-09-15T22:05:02.716 INFO:tasks.workunit.client.0.smithi136.stdout: "osd": 1, 2024-09-15T22:05:02.716 INFO:tasks.workunit.client.0.smithi136.stdout: "primary": true, 2024-09-15T22:05:02.716 INFO:tasks.workunit.client.0.smithi136.stdout: "errors": [], 2024-09-15T22:05:02.716 INFO:tasks.workunit.client.0.smithi136.stdout: "size": 1032 2024-09-15T22:05:02.716 INFO:tasks.workunit.client.0.smithi136.stdout: } 2024-09-15T22:05:02.716 INFO:tasks.workunit.client.0.smithi136.stdout: ] 2024-09-15T22:05:02.716 INFO:tasks.workunit.client.0.smithi136.stdout: }, 2024-09-15T22:05:02.716 INFO:tasks.workunit.client.0.smithi136.stdout: { 2024-09-15T22:05:02.716 INFO:tasks.workunit.client.0.smithi136.stdout: "object": { 2024-09-15T22:05:02.716 INFO:tasks.workunit.client.0.smithi136.stdout: "name": "obj5", 2024-09-15T22:05:02.717 INFO:tasks.workunit.client.0.smithi136.stdout: "nspace": "", 2024-09-15T22:05:02.717 INFO:tasks.workunit.client.0.smithi136.stdout: "locator": "", 2024-09-15T22:05:02.717 INFO:tasks.workunit.client.0.smithi136.stdout: "snap": 7, 2024-09-15T22:05:02.717 INFO:tasks.workunit.client.0.smithi136.stdout: "version": 0 2024-09-15T22:05:02.717 INFO:tasks.workunit.client.0.smithi136.stdout: }, 2024-09-15T22:05:02.717 INFO:tasks.workunit.client.0.smithi136.stdout: "errors": [], 2024-09-15T22:05:02.717 INFO:tasks.workunit.client.0.smithi136.stdout: "union_shard_errors": [ 2024-09-15T22:05:02.717 INFO:tasks.workunit.client.0.smithi136.stdout: "missing", 2024-09-15T22:05:02.717 INFO:tasks.workunit.client.0.smithi136.stdout: "info_missing" 2024-09-15T22:05:02.717 INFO:tasks.workunit.client.0.smithi136.stdout: ], 2024-09-15T22:05:02.717 INFO:tasks.workunit.client.0.smithi136.stdout: "shards": [ 2024-09-15T22:05:02.717 INFO:tasks.workunit.client.0.smithi136.stdout: { 2024-09-15T22:05:02.717 INFO:tasks.workunit.client.0.smithi136.stdout: "osd": 0, 2024-09-15T22:05:02.717 INFO:tasks.workunit.client.0.smithi136.stdout: "primary": false, 2024-09-15T22:05:02.718 INFO:tasks.workunit.client.0.smithi136.stdout: "errors": [ 2024-09-15T22:05:02.718 INFO:tasks.workunit.client.0.smithi136.stdout: "info_missing" 2024-09-15T22:05:02.718 INFO:tasks.workunit.client.0.smithi136.stdout: ], 2024-09-15T22:05:02.718 INFO:tasks.workunit.client.0.smithi136.stdout: "size": 1792 2024-09-15T22:05:02.718 INFO:tasks.workunit.client.0.smithi136.stdout: }, 2024-09-15T22:05:02.718 INFO:tasks.workunit.client.0.smithi136.stdout: { 2024-09-15T22:05:02.718 INFO:tasks.workunit.client.0.smithi136.stdout: "osd": 1, 2024-09-15T22:05:02.718 INFO:tasks.workunit.client.0.smithi136.stdout: "primary": true, 2024-09-15T22:05:02.718 INFO:tasks.workunit.client.0.smithi136.stdout: "errors": [ 2024-09-15T22:05:02.718 INFO:tasks.workunit.client.0.smithi136.stdout: "missing" 2024-09-15T22:05:02.718 INFO:tasks.workunit.client.0.smithi136.stdout: ] 2024-09-15T22:05:02.718 INFO:tasks.workunit.client.0.smithi136.stdout: } 2024-09-15T22:05:02.718 INFO:tasks.workunit.client.0.smithi136.stdout: ] 2024-09-15T22:05:02.718 INFO:tasks.workunit.client.0.smithi136.stdout: }, 2024-09-15T22:05:02.719 INFO:tasks.workunit.client.0.smithi136.stdout: { 2024-09-15T22:05:02.719 INFO:tasks.workunit.client.0.smithi136.stdout: "object": { 2024-09-15T22:05:02.719 INFO:tasks.workunit.client.0.smithi136.stdout: "name": "obj1", 2024-09-15T22:05:02.719 INFO:tasks.workunit.client.0.smithi136.stdout: "nspace": "", 2024-09-15T22:05:02.719 INFO:tasks.workunit.client.0.smithi136.stdout: "locator": "", 2024-09-15T22:05:02.719 INFO:tasks.workunit.client.0.smithi136.stdout: "snap": "head", 2024-09-15T22:05:02.719 INFO:tasks.workunit.client.0.smithi136.stdout: "version": 18 2024-09-15T22:05:02.719 INFO:tasks.workunit.client.0.smithi136.stdout: }, 2024-09-15T22:05:02.719 INFO:tasks.workunit.client.0.smithi136.stdout: "errors": [], 2024-09-15T22:05:02.719 INFO:tasks.workunit.client.0.smithi136.stdout: "union_shard_errors": [ 2024-09-15T22:05:02.719 INFO:tasks.workunit.client.0.smithi136.stdout: "missing" 2024-09-15T22:05:02.719 INFO:tasks.workunit.client.0.smithi136.stdout: ], 2024-09-15T22:05:02.719 INFO:tasks.workunit.client.0.smithi136.stdout: "selected_object_info": { 2024-09-15T22:05:02.719 INFO:tasks.workunit.client.0.smithi136.stdout: "oid": { 2024-09-15T22:05:02.719 INFO:tasks.workunit.client.0.smithi136.stdout: "oid": "obj1", 2024-09-15T22:05:02.720 INFO:tasks.workunit.client.0.smithi136.stdout: "key": "", 2024-09-15T22:05:02.720 INFO:tasks.workunit.client.0.smithi136.stdout: "snapid": -2, 2024-09-15T22:05:02.720 INFO:tasks.workunit.client.0.smithi136.stdout: "hash": 1828249343, 2024-09-15T22:05:02.720 INFO:tasks.workunit.client.0.smithi136.stdout: "max": 0, 2024-09-15T22:05:02.720 INFO:tasks.workunit.client.0.smithi136.stdout: "pool": 1, 2024-09-15T22:05:02.720 INFO:tasks.workunit.client.0.smithi136.stdout: "namespace": "" 2024-09-15T22:05:02.720 INFO:tasks.workunit.client.0.smithi136.stdout: }, 2024-09-15T22:05:02.720 INFO:tasks.workunit.client.0.smithi136.stdout: "version": "17'18", 2024-09-15T22:05:02.720 INFO:tasks.workunit.client.0.smithi136.stdout: "prior_version": "16'1", 2024-09-15T22:05:02.720 INFO:tasks.workunit.client.0.smithi136.stdout: "last_reqid": "client.4288.0:1", 2024-09-15T22:05:02.720 INFO:tasks.workunit.client.0.smithi136.stdout: "user_version": 18, 2024-09-15T22:05:02.720 INFO:tasks.workunit.client.0.smithi136.stdout: "size": 256, 2024-09-15T22:05:02.720 INFO:tasks.workunit.client.0.smithi136.stdout: "mtime": "2024-09-15T22:03:49.931346+0000", 2024-09-15T22:05:02.720 INFO:tasks.workunit.client.0.smithi136.stdout: "local_mtime": "2024-09-15T22:03:50.025379+0000", 2024-09-15T22:05:02.721 INFO:tasks.workunit.client.0.smithi136.stdout: "lost": 0, 2024-09-15T22:05:02.721 INFO:tasks.workunit.client.0.smithi136.stdout: "flags": [ 2024-09-15T22:05:02.721 INFO:tasks.workunit.client.0.smithi136.stdout: "dirty", 2024-09-15T22:05:02.721 INFO:tasks.workunit.client.0.smithi136.stdout: "data_digest" 2024-09-15T22:05:02.721 INFO:tasks.workunit.client.0.smithi136.stdout: ], 2024-09-15T22:05:02.721 INFO:tasks.workunit.client.0.smithi136.stdout: "truncate_seq": 0, 2024-09-15T22:05:02.721 INFO:tasks.workunit.client.0.smithi136.stdout: "truncate_size": 0, 2024-09-15T22:05:02.721 INFO:tasks.workunit.client.0.smithi136.stdout: "data_digest": "0x8d0e1329", 2024-09-15T22:05:02.721 INFO:tasks.workunit.client.0.smithi136.stdout: "omap_digest": "0xffffffff", 2024-09-15T22:05:02.721 INFO:tasks.workunit.client.0.smithi136.stdout: "expected_object_size": 0, 2024-09-15T22:05:02.721 INFO:tasks.workunit.client.0.smithi136.stdout: "expected_write_size": 0, 2024-09-15T22:05:02.721 INFO:tasks.workunit.client.0.smithi136.stdout: "alloc_hint_flags": 0, 2024-09-15T22:05:02.721 INFO:tasks.workunit.client.0.smithi136.stdout: "manifest": { 2024-09-15T22:05:02.722 INFO:tasks.workunit.client.0.smithi136.stdout: "type": 0 2024-09-15T22:05:02.722 INFO:tasks.workunit.client.0.smithi136.stdout: }, 2024-09-15T22:05:02.722 INFO:tasks.workunit.client.0.smithi136.stdout: "watchers": {} 2024-09-15T22:05:02.722 INFO:tasks.workunit.client.0.smithi136.stdout: }, 2024-09-15T22:05:02.722 INFO:tasks.workunit.client.0.smithi136.stdout: "shards": [ 2024-09-15T22:05:02.722 INFO:tasks.workunit.client.0.smithi136.stdout: { 2024-09-15T22:05:02.722 INFO:tasks.workunit.client.0.smithi136.stdout: "osd": 0, 2024-09-15T22:05:02.722 INFO:tasks.workunit.client.0.smithi136.stdout: "primary": false, 2024-09-15T22:05:02.722 INFO:tasks.workunit.client.0.smithi136.stdout: "errors": [ 2024-09-15T22:05:02.722 INFO:tasks.workunit.client.0.smithi136.stdout: "missing" 2024-09-15T22:05:02.722 INFO:tasks.workunit.client.0.smithi136.stdout: ] 2024-09-15T22:05:02.722 INFO:tasks.workunit.client.0.smithi136.stdout: }, 2024-09-15T22:05:02.722 INFO:tasks.workunit.client.0.smithi136.stdout: { 2024-09-15T22:05:02.722 INFO:tasks.workunit.client.0.smithi136.stdout: "osd": 1, 2024-09-15T22:05:02.723 INFO:tasks.workunit.client.0.smithi136.stdout: "primary": true, 2024-09-15T22:05:02.723 INFO:tasks.workunit.client.0.smithi136.stdout: "errors": [], 2024-09-15T22:05:02.723 INFO:tasks.workunit.client.0.smithi136.stdout: "size": 256 2024-09-15T22:05:02.723 INFO:tasks.workunit.client.0.smithi136.stdout: } 2024-09-15T22:05:02.723 INFO:tasks.workunit.client.0.smithi136.stdout: ] 2024-09-15T22:05:02.723 INFO:tasks.workunit.client.0.smithi136.stdout: }, 2024-09-15T22:05:02.723 INFO:tasks.workunit.client.0.smithi136.stdout: { 2024-09-15T22:05:02.723 INFO:tasks.workunit.client.0.smithi136.stdout: "object": { 2024-09-15T22:05:02.723 INFO:tasks.workunit.client.0.smithi136.stdout: "name": "obj10", 2024-09-15T22:05:02.723 INFO:tasks.workunit.client.0.smithi136.stdout: "nspace": "", 2024-09-15T22:05:02.723 INFO:tasks.workunit.client.0.smithi136.stdout: "locator": "", 2024-09-15T22:05:02.723 INFO:tasks.workunit.client.0.smithi136.stdout: "snap": "head", 2024-09-15T22:05:02.723 INFO:tasks.workunit.client.0.smithi136.stdout: "version": 32 2024-09-15T22:05:02.723 INFO:tasks.workunit.client.0.smithi136.stdout: }, 2024-09-15T22:05:02.724 INFO:tasks.workunit.client.0.smithi136.stdout: "errors": [ 2024-09-15T22:05:02.724 INFO:tasks.workunit.client.0.smithi136.stdout: "snapset_inconsistency" 2024-09-15T22:05:02.724 INFO:tasks.workunit.client.0.smithi136.stdout: ], 2024-09-15T22:05:02.724 INFO:tasks.workunit.client.0.smithi136.stdout: "union_shard_errors": [], 2024-09-15T22:05:02.724 INFO:tasks.workunit.client.0.smithi136.stdout: "selected_object_info": { 2024-09-15T22:05:02.724 INFO:tasks.workunit.client.0.smithi136.stdout: "oid": { 2024-09-15T22:05:02.724 INFO:tasks.workunit.client.0.smithi136.stdout: "oid": "obj10", 2024-09-15T22:05:02.724 INFO:tasks.workunit.client.0.smithi136.stdout: "key": "", 2024-09-15T22:05:02.724 INFO:tasks.workunit.client.0.smithi136.stdout: "snapid": -2, 2024-09-15T22:05:02.724 INFO:tasks.workunit.client.0.smithi136.stdout: "hash": 718195851, 2024-09-15T22:05:02.724 INFO:tasks.workunit.client.0.smithi136.stdout: "max": 0, 2024-09-15T22:05:02.724 INFO:tasks.workunit.client.0.smithi136.stdout: "pool": 1, 2024-09-15T22:05:02.724 INFO:tasks.workunit.client.0.smithi136.stdout: "namespace": "" 2024-09-15T22:05:02.724 INFO:tasks.workunit.client.0.smithi136.stdout: }, 2024-09-15T22:05:02.724 INFO:tasks.workunit.client.0.smithi136.stdout: "version": "17'32", 2024-09-15T22:05:02.725 INFO:tasks.workunit.client.0.smithi136.stdout: "prior_version": "16'10", 2024-09-15T22:05:02.725 INFO:tasks.workunit.client.0.smithi136.stdout: "last_reqid": "client.4309.0:1", 2024-09-15T22:05:02.725 INFO:tasks.workunit.client.0.smithi136.stdout: "user_version": 32, 2024-09-15T22:05:02.725 INFO:tasks.workunit.client.0.smithi136.stdout: "size": 256, 2024-09-15T22:05:02.725 INFO:tasks.workunit.client.0.smithi136.stdout: "mtime": "2024-09-15T22:03:51.026198+0000", 2024-09-15T22:05:02.725 INFO:tasks.workunit.client.0.smithi136.stdout: "local_mtime": "2024-09-15T22:03:51.026828+0000", 2024-09-15T22:05:02.725 INFO:tasks.workunit.client.0.smithi136.stdout: "lost": 0, 2024-09-15T22:05:02.725 INFO:tasks.workunit.client.0.smithi136.stdout: "flags": [ 2024-09-15T22:05:02.725 INFO:tasks.workunit.client.0.smithi136.stdout: "dirty", 2024-09-15T22:05:02.725 INFO:tasks.workunit.client.0.smithi136.stdout: "data_digest" 2024-09-15T22:05:02.725 INFO:tasks.workunit.client.0.smithi136.stdout: ], 2024-09-15T22:05:02.725 INFO:tasks.workunit.client.0.smithi136.stdout: "truncate_seq": 0, 2024-09-15T22:05:02.725 INFO:tasks.workunit.client.0.smithi136.stdout: "truncate_size": 0, 2024-09-15T22:05:02.726 INFO:tasks.workunit.client.0.smithi136.stdout: "data_digest": "0x8d0e1329", 2024-09-15T22:05:02.726 INFO:tasks.workunit.client.0.smithi136.stdout: "omap_digest": "0xffffffff", 2024-09-15T22:05:02.726 INFO:tasks.workunit.client.0.smithi136.stdout: "expected_object_size": 0, 2024-09-15T22:05:02.726 INFO:tasks.workunit.client.0.smithi136.stdout: "expected_write_size": 0, 2024-09-15T22:05:02.726 INFO:tasks.workunit.client.0.smithi136.stdout: "alloc_hint_flags": 0, 2024-09-15T22:05:02.726 INFO:tasks.workunit.client.0.smithi136.stdout: "manifest": { 2024-09-15T22:05:02.726 INFO:tasks.workunit.client.0.smithi136.stdout: "type": 0 2024-09-15T22:05:02.726 INFO:tasks.workunit.client.0.smithi136.stdout: }, 2024-09-15T22:05:02.726 INFO:tasks.workunit.client.0.smithi136.stdout: "watchers": {} 2024-09-15T22:05:02.726 INFO:tasks.workunit.client.0.smithi136.stdout: }, 2024-09-15T22:05:02.726 INFO:tasks.workunit.client.0.smithi136.stdout: "shards": [ 2024-09-15T22:05:02.726 INFO:tasks.workunit.client.0.smithi136.stdout: { 2024-09-15T22:05:02.726 INFO:tasks.workunit.client.0.smithi136.stdout: "osd": 0, 2024-09-15T22:05:02.726 INFO:tasks.workunit.client.0.smithi136.stdout: "primary": false, 2024-09-15T22:05:02.726 INFO:tasks.workunit.client.0.smithi136.stdout: "errors": [], 2024-09-15T22:05:02.727 INFO:tasks.workunit.client.0.smithi136.stdout: "size": 256, 2024-09-15T22:05:02.727 INFO:tasks.workunit.client.0.smithi136.stdout: "snapset": { 2024-09-15T22:05:02.727 INFO:tasks.workunit.client.0.smithi136.stdout: "seq": 1, 2024-09-15T22:05:02.727 INFO:tasks.workunit.client.0.smithi136.stdout: "clones": [ 2024-09-15T22:05:02.727 INFO:tasks.workunit.client.0.smithi136.stdout: { 2024-09-15T22:05:02.727 INFO:tasks.workunit.client.0.smithi136.stdout: "snap": 1, 2024-09-15T22:05:02.727 INFO:tasks.workunit.client.0.smithi136.stdout: "size": 1032, 2024-09-15T22:05:02.727 INFO:tasks.workunit.client.0.smithi136.stdout: "overlap": "????", 2024-09-15T22:05:02.727 INFO:tasks.workunit.client.0.smithi136.stdout: "snaps": [ 2024-09-15T22:05:02.727 INFO:tasks.workunit.client.0.smithi136.stdout: 1 2024-09-15T22:05:02.727 INFO:tasks.workunit.client.0.smithi136.stdout: ] 2024-09-15T22:05:02.727 INFO:tasks.workunit.client.0.smithi136.stdout: } 2024-09-15T22:05:02.727 INFO:tasks.workunit.client.0.smithi136.stdout: ] 2024-09-15T22:05:02.727 INFO:tasks.workunit.client.0.smithi136.stdout: } 2024-09-15T22:05:02.728 INFO:tasks.workunit.client.0.smithi136.stdout: }, 2024-09-15T22:05:02.728 INFO:tasks.workunit.client.0.smithi136.stdout: { 2024-09-15T22:05:02.728 INFO:tasks.workunit.client.0.smithi136.stdout: "osd": 1, 2024-09-15T22:05:02.728 INFO:tasks.workunit.client.0.smithi136.stdout: "primary": true, 2024-09-15T22:05:02.728 INFO:tasks.workunit.client.0.smithi136.stdout: "errors": [], 2024-09-15T22:05:02.728 INFO:tasks.workunit.client.0.smithi136.stdout: "size": 256, 2024-09-15T22:05:02.728 INFO:tasks.workunit.client.0.smithi136.stdout: "snapset": { 2024-09-15T22:05:02.728 INFO:tasks.workunit.client.0.smithi136.stdout: "seq": 1, 2024-09-15T22:05:02.728 INFO:tasks.workunit.client.0.smithi136.stdout: "clones": [ 2024-09-15T22:05:02.728 INFO:tasks.workunit.client.0.smithi136.stdout: { 2024-09-15T22:05:02.728 INFO:tasks.workunit.client.0.smithi136.stdout: "snap": 1, 2024-09-15T22:05:02.728 INFO:tasks.workunit.client.0.smithi136.stdout: "size": 1032, 2024-09-15T22:05:02.728 INFO:tasks.workunit.client.0.smithi136.stdout: "overlap": "[]", 2024-09-15T22:05:02.728 INFO:tasks.workunit.client.0.smithi136.stdout: "snaps": [ 2024-09-15T22:05:02.729 INFO:tasks.workunit.client.0.smithi136.stdout: 1 2024-09-15T22:05:02.729 INFO:tasks.workunit.client.0.smithi136.stdout: ] 2024-09-15T22:05:02.729 INFO:tasks.workunit.client.0.smithi136.stdout: } 2024-09-15T22:05:02.729 INFO:tasks.workunit.client.0.smithi136.stdout: ] 2024-09-15T22:05:02.729 INFO:tasks.workunit.client.0.smithi136.stdout: } 2024-09-15T22:05:02.729 INFO:tasks.workunit.client.0.smithi136.stdout: } 2024-09-15T22:05:02.729 INFO:tasks.workunit.client.0.smithi136.stdout: ] 2024-09-15T22:05:02.729 INFO:tasks.workunit.client.0.smithi136.stdout: }, 2024-09-15T22:05:02.729 INFO:tasks.workunit.client.0.smithi136.stdout: { 2024-09-15T22:05:02.729 INFO:tasks.workunit.client.0.smithi136.stdout: "object": { 2024-09-15T22:05:02.729 INFO:tasks.workunit.client.0.smithi136.stdout: "name": "obj11", 2024-09-15T22:05:02.729 INFO:tasks.workunit.client.0.smithi136.stdout: "nspace": "", 2024-09-15T22:05:02.729 INFO:tasks.workunit.client.0.smithi136.stdout: "locator": "", 2024-09-15T22:05:02.729 INFO:tasks.workunit.client.0.smithi136.stdout: "snap": "head", 2024-09-15T22:05:02.730 INFO:tasks.workunit.client.0.smithi136.stdout: "version": 34 2024-09-15T22:05:02.730 INFO:tasks.workunit.client.0.smithi136.stdout: }, 2024-09-15T22:05:02.730 INFO:tasks.workunit.client.0.smithi136.stdout: "errors": [ 2024-09-15T22:05:02.730 INFO:tasks.workunit.client.0.smithi136.stdout: "snapset_inconsistency" 2024-09-15T22:05:02.730 INFO:tasks.workunit.client.0.smithi136.stdout: ], 2024-09-15T22:05:02.730 INFO:tasks.workunit.client.0.smithi136.stdout: "union_shard_errors": [], 2024-09-15T22:05:02.730 INFO:tasks.workunit.client.0.smithi136.stdout: "selected_object_info": { 2024-09-15T22:05:02.730 INFO:tasks.workunit.client.0.smithi136.stdout: "oid": { 2024-09-15T22:05:02.730 INFO:tasks.workunit.client.0.smithi136.stdout: "oid": "obj11", 2024-09-15T22:05:02.730 INFO:tasks.workunit.client.0.smithi136.stdout: "key": "", 2024-09-15T22:05:02.730 INFO:tasks.workunit.client.0.smithi136.stdout: "snapid": -2, 2024-09-15T22:05:02.730 INFO:tasks.workunit.client.0.smithi136.stdout: "hash": 693400951, 2024-09-15T22:05:02.730 INFO:tasks.workunit.client.0.smithi136.stdout: "max": 0, 2024-09-15T22:05:02.730 INFO:tasks.workunit.client.0.smithi136.stdout: "pool": 1, 2024-09-15T22:05:02.730 INFO:tasks.workunit.client.0.smithi136.stdout: "namespace": "" 2024-09-15T22:05:02.731 INFO:tasks.workunit.client.0.smithi136.stdout: }, 2024-09-15T22:05:02.731 INFO:tasks.workunit.client.0.smithi136.stdout: "version": "17'34", 2024-09-15T22:05:02.731 INFO:tasks.workunit.client.0.smithi136.stdout: "prior_version": "16'11", 2024-09-15T22:05:02.731 INFO:tasks.workunit.client.0.smithi136.stdout: "last_reqid": "client.4312.0:1", 2024-09-15T22:05:02.731 INFO:tasks.workunit.client.0.smithi136.stdout: "user_version": 34, 2024-09-15T22:05:02.731 INFO:tasks.workunit.client.0.smithi136.stdout: "size": 256, 2024-09-15T22:05:02.731 INFO:tasks.workunit.client.0.smithi136.stdout: "mtime": "2024-09-15T22:03:51.152023+0000", 2024-09-15T22:05:02.731 INFO:tasks.workunit.client.0.smithi136.stdout: "local_mtime": "2024-09-15T22:03:51.152594+0000", 2024-09-15T22:05:02.731 INFO:tasks.workunit.client.0.smithi136.stdout: "lost": 0, 2024-09-15T22:05:02.731 INFO:tasks.workunit.client.0.smithi136.stdout: "flags": [ 2024-09-15T22:05:02.731 INFO:tasks.workunit.client.0.smithi136.stdout: "dirty", 2024-09-15T22:05:02.731 INFO:tasks.workunit.client.0.smithi136.stdout: "data_digest" 2024-09-15T22:05:02.731 INFO:tasks.workunit.client.0.smithi136.stdout: ], 2024-09-15T22:05:02.731 INFO:tasks.workunit.client.0.smithi136.stdout: "truncate_seq": 0, 2024-09-15T22:05:02.732 INFO:tasks.workunit.client.0.smithi136.stdout: "truncate_size": 0, 2024-09-15T22:05:02.732 INFO:tasks.workunit.client.0.smithi136.stdout: "data_digest": "0x8d0e1329", 2024-09-15T22:05:02.732 INFO:tasks.workunit.client.0.smithi136.stdout: "omap_digest": "0xffffffff", 2024-09-15T22:05:02.732 INFO:tasks.workunit.client.0.smithi136.stdout: "expected_object_size": 0, 2024-09-15T22:05:02.732 INFO:tasks.workunit.client.0.smithi136.stdout: "expected_write_size": 0, 2024-09-15T22:05:02.732 INFO:tasks.workunit.client.0.smithi136.stdout: "alloc_hint_flags": 0, 2024-09-15T22:05:02.732 INFO:tasks.workunit.client.0.smithi136.stdout: "manifest": { 2024-09-15T22:05:02.732 INFO:tasks.workunit.client.0.smithi136.stdout: "type": 0 2024-09-15T22:05:02.732 INFO:tasks.workunit.client.0.smithi136.stdout: }, 2024-09-15T22:05:02.732 INFO:tasks.workunit.client.0.smithi136.stdout: "watchers": {} 2024-09-15T22:05:02.732 INFO:tasks.workunit.client.0.smithi136.stdout: }, 2024-09-15T22:05:02.732 INFO:tasks.workunit.client.0.smithi136.stdout: "shards": [ 2024-09-15T22:05:02.732 INFO:tasks.workunit.client.0.smithi136.stdout: { 2024-09-15T22:05:02.732 INFO:tasks.workunit.client.0.smithi136.stdout: "osd": 0, 2024-09-15T22:05:02.733 INFO:tasks.workunit.client.0.smithi136.stdout: "primary": false, 2024-09-15T22:05:02.733 INFO:tasks.workunit.client.0.smithi136.stdout: "errors": [], 2024-09-15T22:05:02.733 INFO:tasks.workunit.client.0.smithi136.stdout: "size": 256, 2024-09-15T22:05:02.733 INFO:tasks.workunit.client.0.smithi136.stdout: "snapset": { 2024-09-15T22:05:02.733 INFO:tasks.workunit.client.0.smithi136.stdout: "seq": 1, 2024-09-15T22:05:02.733 INFO:tasks.workunit.client.0.smithi136.stdout: "clones": [] 2024-09-15T22:05:02.733 INFO:tasks.workunit.client.0.smithi136.stdout: } 2024-09-15T22:05:02.733 INFO:tasks.workunit.client.0.smithi136.stdout: }, 2024-09-15T22:05:02.733 INFO:tasks.workunit.client.0.smithi136.stdout: { 2024-09-15T22:05:02.733 INFO:tasks.workunit.client.0.smithi136.stdout: "osd": 1, 2024-09-15T22:05:02.733 INFO:tasks.workunit.client.0.smithi136.stdout: "primary": true, 2024-09-15T22:05:02.733 INFO:tasks.workunit.client.0.smithi136.stdout: "errors": [], 2024-09-15T22:05:02.733 INFO:tasks.workunit.client.0.smithi136.stdout: "size": 256, 2024-09-15T22:05:02.733 INFO:tasks.workunit.client.0.smithi136.stdout: "snapset": { 2024-09-15T22:05:02.734 INFO:tasks.workunit.client.0.smithi136.stdout: "seq": 1, 2024-09-15T22:05:02.734 INFO:tasks.workunit.client.0.smithi136.stdout: "clones": [ 2024-09-15T22:05:02.734 INFO:tasks.workunit.client.0.smithi136.stdout: { 2024-09-15T22:05:02.734 INFO:tasks.workunit.client.0.smithi136.stdout: "snap": 1, 2024-09-15T22:05:02.734 INFO:tasks.workunit.client.0.smithi136.stdout: "size": 1032, 2024-09-15T22:05:02.734 INFO:tasks.workunit.client.0.smithi136.stdout: "overlap": "[]", 2024-09-15T22:05:02.734 INFO:tasks.workunit.client.0.smithi136.stdout: "snaps": [ 2024-09-15T22:05:02.734 INFO:tasks.workunit.client.0.smithi136.stdout: 1 2024-09-15T22:05:02.734 INFO:tasks.workunit.client.0.smithi136.stdout: ] 2024-09-15T22:05:02.734 INFO:tasks.workunit.client.0.smithi136.stdout: } 2024-09-15T22:05:02.734 INFO:tasks.workunit.client.0.smithi136.stdout: ] 2024-09-15T22:05:02.734 INFO:tasks.workunit.client.0.smithi136.stdout: } 2024-09-15T22:05:02.734 INFO:tasks.workunit.client.0.smithi136.stdout: } 2024-09-15T22:05:02.734 INFO:tasks.workunit.client.0.smithi136.stdout: ] 2024-09-15T22:05:02.735 INFO:tasks.workunit.client.0.smithi136.stdout: }, 2024-09-15T22:05:02.735 INFO:tasks.workunit.client.0.smithi136.stdout: { 2024-09-15T22:05:02.735 INFO:tasks.workunit.client.0.smithi136.stdout: "object": { 2024-09-15T22:05:02.735 INFO:tasks.workunit.client.0.smithi136.stdout: "name": "obj13", 2024-09-15T22:05:02.735 INFO:tasks.workunit.client.0.smithi136.stdout: "nspace": "", 2024-09-15T22:05:02.735 INFO:tasks.workunit.client.0.smithi136.stdout: "locator": "", 2024-09-15T22:05:02.735 INFO:tasks.workunit.client.0.smithi136.stdout: "snap": "head", 2024-09-15T22:05:02.735 INFO:tasks.workunit.client.0.smithi136.stdout: "version": 38 2024-09-15T22:05:02.735 INFO:tasks.workunit.client.0.smithi136.stdout: }, 2024-09-15T22:05:02.735 INFO:tasks.workunit.client.0.smithi136.stdout: "errors": [ 2024-09-15T22:05:02.735 INFO:tasks.workunit.client.0.smithi136.stdout: "snapset_inconsistency" 2024-09-15T22:05:02.735 INFO:tasks.workunit.client.0.smithi136.stdout: ], 2024-09-15T22:05:02.735 INFO:tasks.workunit.client.0.smithi136.stdout: "union_shard_errors": [], 2024-09-15T22:05:02.736 INFO:tasks.workunit.client.0.smithi136.stdout: "selected_object_info": { 2024-09-15T22:05:02.736 INFO:tasks.workunit.client.0.smithi136.stdout: "oid": { 2024-09-15T22:05:02.736 INFO:tasks.workunit.client.0.smithi136.stdout: "oid": "obj13", 2024-09-15T22:05:02.736 INFO:tasks.workunit.client.0.smithi136.stdout: "key": "", 2024-09-15T22:05:02.736 INFO:tasks.workunit.client.0.smithi136.stdout: "snapid": -2, 2024-09-15T22:05:02.736 INFO:tasks.workunit.client.0.smithi136.stdout: "hash": 2087409765, 2024-09-15T22:05:02.736 INFO:tasks.workunit.client.0.smithi136.stdout: "max": 0, 2024-09-15T22:05:02.736 INFO:tasks.workunit.client.0.smithi136.stdout: "pool": 1, 2024-09-15T22:05:02.736 INFO:tasks.workunit.client.0.smithi136.stdout: "namespace": "" 2024-09-15T22:05:02.736 INFO:tasks.workunit.client.0.smithi136.stdout: }, 2024-09-15T22:05:02.736 INFO:tasks.workunit.client.0.smithi136.stdout: "version": "17'38", 2024-09-15T22:05:02.736 INFO:tasks.workunit.client.0.smithi136.stdout: "prior_version": "16'13", 2024-09-15T22:05:02.736 INFO:tasks.workunit.client.0.smithi136.stdout: "last_reqid": "client.4318.0:1", 2024-09-15T22:05:02.736 INFO:tasks.workunit.client.0.smithi136.stdout: "user_version": 38, 2024-09-15T22:05:02.737 INFO:tasks.workunit.client.0.smithi136.stdout: "size": 256, 2024-09-15T22:05:02.737 INFO:tasks.workunit.client.0.smithi136.stdout: "mtime": "2024-09-15T22:03:51.428089+0000", 2024-09-15T22:05:02.737 INFO:tasks.workunit.client.0.smithi136.stdout: "local_mtime": "2024-09-15T22:03:51.428658+0000", 2024-09-15T22:05:02.737 INFO:tasks.workunit.client.0.smithi136.stdout: "lost": 0, 2024-09-15T22:05:02.737 INFO:tasks.workunit.client.0.smithi136.stdout: "flags": [ 2024-09-15T22:05:02.737 INFO:tasks.workunit.client.0.smithi136.stdout: "dirty", 2024-09-15T22:05:02.737 INFO:tasks.workunit.client.0.smithi136.stdout: "data_digest" 2024-09-15T22:05:02.737 INFO:tasks.workunit.client.0.smithi136.stdout: ], 2024-09-15T22:05:02.737 INFO:tasks.workunit.client.0.smithi136.stdout: "truncate_seq": 0, 2024-09-15T22:05:02.737 INFO:tasks.workunit.client.0.smithi136.stdout: "truncate_size": 0, 2024-09-15T22:05:02.737 INFO:tasks.workunit.client.0.smithi136.stdout: "data_digest": "0x8d0e1329", 2024-09-15T22:05:02.737 INFO:tasks.workunit.client.0.smithi136.stdout: "omap_digest": "0xffffffff", 2024-09-15T22:05:02.737 INFO:tasks.workunit.client.0.smithi136.stdout: "expected_object_size": 0, 2024-09-15T22:05:02.737 INFO:tasks.workunit.client.0.smithi136.stdout: "expected_write_size": 0, 2024-09-15T22:05:02.738 INFO:tasks.workunit.client.0.smithi136.stdout: "alloc_hint_flags": 0, 2024-09-15T22:05:02.738 INFO:tasks.workunit.client.0.smithi136.stdout: "manifest": { 2024-09-15T22:05:02.738 INFO:tasks.workunit.client.0.smithi136.stdout: "type": 0 2024-09-15T22:05:02.738 INFO:tasks.workunit.client.0.smithi136.stdout: }, 2024-09-15T22:05:02.738 INFO:tasks.workunit.client.0.smithi136.stdout: "watchers": {} 2024-09-15T22:05:02.738 INFO:tasks.workunit.client.0.smithi136.stdout: }, 2024-09-15T22:05:02.738 INFO:tasks.workunit.client.0.smithi136.stdout: "shards": [ 2024-09-15T22:05:02.738 INFO:tasks.workunit.client.0.smithi136.stdout: { 2024-09-15T22:05:02.738 INFO:tasks.workunit.client.0.smithi136.stdout: "osd": 0, 2024-09-15T22:05:02.738 INFO:tasks.workunit.client.0.smithi136.stdout: "primary": false, 2024-09-15T22:05:02.738 INFO:tasks.workunit.client.0.smithi136.stdout: "errors": [], 2024-09-15T22:05:02.738 INFO:tasks.workunit.client.0.smithi136.stdout: "size": 256, 2024-09-15T22:05:02.738 INFO:tasks.workunit.client.0.smithi136.stdout: "snapset": { 2024-09-15T22:05:02.738 INFO:tasks.workunit.client.0.smithi136.stdout: "seq": 1, 2024-09-15T22:05:02.739 INFO:tasks.workunit.client.0.smithi136.stdout: "clones": [ 2024-09-15T22:05:02.739 INFO:tasks.workunit.client.0.smithi136.stdout: { 2024-09-15T22:05:02.739 INFO:tasks.workunit.client.0.smithi136.stdout: "snap": 1, 2024-09-15T22:05:02.739 INFO:tasks.workunit.client.0.smithi136.stdout: "size": 1032, 2024-09-15T22:05:02.739 INFO:tasks.workunit.client.0.smithi136.stdout: "overlap": "[]" 2024-09-15T22:05:02.739 INFO:tasks.workunit.client.0.smithi136.stdout: } 2024-09-15T22:05:02.739 INFO:tasks.workunit.client.0.smithi136.stdout: ] 2024-09-15T22:05:02.739 INFO:tasks.workunit.client.0.smithi136.stdout: } 2024-09-15T22:05:02.739 INFO:tasks.workunit.client.0.smithi136.stdout: }, 2024-09-15T22:05:02.739 INFO:tasks.workunit.client.0.smithi136.stdout: { 2024-09-15T22:05:02.739 INFO:tasks.workunit.client.0.smithi136.stdout: "osd": 1, 2024-09-15T22:05:02.739 INFO:tasks.workunit.client.0.smithi136.stdout: "primary": true, 2024-09-15T22:05:02.739 INFO:tasks.workunit.client.0.smithi136.stdout: "errors": [], 2024-09-15T22:05:02.739 INFO:tasks.workunit.client.0.smithi136.stdout: "size": 256, 2024-09-15T22:05:02.740 INFO:tasks.workunit.client.0.smithi136.stdout: "snapset": { 2024-09-15T22:05:02.740 INFO:tasks.workunit.client.0.smithi136.stdout: "seq": 1, 2024-09-15T22:05:02.740 INFO:tasks.workunit.client.0.smithi136.stdout: "clones": [ 2024-09-15T22:05:02.740 INFO:tasks.workunit.client.0.smithi136.stdout: { 2024-09-15T22:05:02.740 INFO:tasks.workunit.client.0.smithi136.stdout: "snap": 1, 2024-09-15T22:05:02.740 INFO:tasks.workunit.client.0.smithi136.stdout: "size": 1032, 2024-09-15T22:05:02.740 INFO:tasks.workunit.client.0.smithi136.stdout: "overlap": "[]", 2024-09-15T22:05:02.740 INFO:tasks.workunit.client.0.smithi136.stdout: "snaps": [ 2024-09-15T22:05:02.740 INFO:tasks.workunit.client.0.smithi136.stdout: 1 2024-09-15T22:05:02.740 INFO:tasks.workunit.client.0.smithi136.stdout: ] 2024-09-15T22:05:02.740 INFO:tasks.workunit.client.0.smithi136.stdout: } 2024-09-15T22:05:02.740 INFO:tasks.workunit.client.0.smithi136.stdout: ] 2024-09-15T22:05:02.740 INFO:tasks.workunit.client.0.smithi136.stdout: } 2024-09-15T22:05:02.741 INFO:tasks.workunit.client.0.smithi136.stdout: } 2024-09-15T22:05:02.741 INFO:tasks.workunit.client.0.smithi136.stdout: ] 2024-09-15T22:05:02.741 INFO:tasks.workunit.client.0.smithi136.stdout: }, 2024-09-15T22:05:02.741 INFO:tasks.workunit.client.0.smithi136.stdout: { 2024-09-15T22:05:02.741 INFO:tasks.workunit.client.0.smithi136.stdout: "object": { 2024-09-15T22:05:02.741 INFO:tasks.workunit.client.0.smithi136.stdout: "name": "obj14", 2024-09-15T22:05:02.741 INFO:tasks.workunit.client.0.smithi136.stdout: "nspace": "", 2024-09-15T22:05:02.741 INFO:tasks.workunit.client.0.smithi136.stdout: "locator": "", 2024-09-15T22:05:02.741 INFO:tasks.workunit.client.0.smithi136.stdout: "snap": "head", 2024-09-15T22:05:02.741 INFO:tasks.workunit.client.0.smithi136.stdout: "version": 40 2024-09-15T22:05:02.741 INFO:tasks.workunit.client.0.smithi136.stdout: }, 2024-09-15T22:05:02.741 INFO:tasks.workunit.client.0.smithi136.stdout: "errors": [ 2024-09-15T22:05:02.741 INFO:tasks.workunit.client.0.smithi136.stdout: "snapset_inconsistency" 2024-09-15T22:05:02.741 INFO:tasks.workunit.client.0.smithi136.stdout: ], 2024-09-15T22:05:02.741 INFO:tasks.workunit.client.0.smithi136.stdout: "union_shard_errors": [], 2024-09-15T22:05:02.742 INFO:tasks.workunit.client.0.smithi136.stdout: "selected_object_info": { 2024-09-15T22:05:02.742 INFO:tasks.workunit.client.0.smithi136.stdout: "oid": { 2024-09-15T22:05:02.742 INFO:tasks.workunit.client.0.smithi136.stdout: "oid": "obj14", 2024-09-15T22:05:02.742 INFO:tasks.workunit.client.0.smithi136.stdout: "key": "", 2024-09-15T22:05:02.742 INFO:tasks.workunit.client.0.smithi136.stdout: "snapid": -2, 2024-09-15T22:05:02.742 INFO:tasks.workunit.client.0.smithi136.stdout: "hash": 2484217095, 2024-09-15T22:05:02.742 INFO:tasks.workunit.client.0.smithi136.stdout: "max": 0, 2024-09-15T22:05:02.742 INFO:tasks.workunit.client.0.smithi136.stdout: "pool": 1, 2024-09-15T22:05:02.742 INFO:tasks.workunit.client.0.smithi136.stdout: "namespace": "" 2024-09-15T22:05:02.742 INFO:tasks.workunit.client.0.smithi136.stdout: }, 2024-09-15T22:05:02.742 INFO:tasks.workunit.client.0.smithi136.stdout: "version": "17'40", 2024-09-15T22:05:02.742 INFO:tasks.workunit.client.0.smithi136.stdout: "prior_version": "16'14", 2024-09-15T22:05:02.742 INFO:tasks.workunit.client.0.smithi136.stdout: "last_reqid": "client.4321.0:1", 2024-09-15T22:05:02.743 INFO:tasks.workunit.client.0.smithi136.stdout: "user_version": 40, 2024-09-15T22:05:02.743 INFO:tasks.workunit.client.0.smithi136.stdout: "size": 256, 2024-09-15T22:05:02.743 INFO:tasks.workunit.client.0.smithi136.stdout: "mtime": "2024-09-15T22:03:51.570404+0000", 2024-09-15T22:05:02.743 INFO:tasks.workunit.client.0.smithi136.stdout: "local_mtime": "2024-09-15T22:03:51.570961+0000", 2024-09-15T22:05:02.743 INFO:tasks.workunit.client.0.smithi136.stdout: "lost": 0, 2024-09-15T22:05:02.743 INFO:tasks.workunit.client.0.smithi136.stdout: "flags": [ 2024-09-15T22:05:02.743 INFO:tasks.workunit.client.0.smithi136.stdout: "dirty", 2024-09-15T22:05:02.743 INFO:tasks.workunit.client.0.smithi136.stdout: "data_digest" 2024-09-15T22:05:02.743 INFO:tasks.workunit.client.0.smithi136.stdout: ], 2024-09-15T22:05:02.743 INFO:tasks.workunit.client.0.smithi136.stdout: "truncate_seq": 0, 2024-09-15T22:05:02.743 INFO:tasks.workunit.client.0.smithi136.stdout: "truncate_size": 0, 2024-09-15T22:05:02.743 INFO:tasks.workunit.client.0.smithi136.stdout: "data_digest": "0x8d0e1329", 2024-09-15T22:05:02.743 INFO:tasks.workunit.client.0.smithi136.stdout: "omap_digest": "0xffffffff", 2024-09-15T22:05:02.743 INFO:tasks.workunit.client.0.smithi136.stdout: "expected_object_size": 0, 2024-09-15T22:05:02.743 INFO:tasks.workunit.client.0.smithi136.stdout: "expected_write_size": 0, 2024-09-15T22:05:02.744 INFO:tasks.workunit.client.0.smithi136.stdout: "alloc_hint_flags": 0, 2024-09-15T22:05:02.744 INFO:tasks.workunit.client.0.smithi136.stdout: "manifest": { 2024-09-15T22:05:02.744 INFO:tasks.workunit.client.0.smithi136.stdout: "type": 0 2024-09-15T22:05:02.744 INFO:tasks.workunit.client.0.smithi136.stdout: }, 2024-09-15T22:05:02.744 INFO:tasks.workunit.client.0.smithi136.stdout: "watchers": {} 2024-09-15T22:05:02.744 INFO:tasks.workunit.client.0.smithi136.stdout: }, 2024-09-15T22:05:02.744 INFO:tasks.workunit.client.0.smithi136.stdout: "shards": [ 2024-09-15T22:05:02.744 INFO:tasks.workunit.client.0.smithi136.stdout: { 2024-09-15T22:05:02.744 INFO:tasks.workunit.client.0.smithi136.stdout: "osd": 0, 2024-09-15T22:05:02.744 INFO:tasks.workunit.client.0.smithi136.stdout: "primary": false, 2024-09-15T22:05:02.744 INFO:tasks.workunit.client.0.smithi136.stdout: "errors": [], 2024-09-15T22:05:02.744 INFO:tasks.workunit.client.0.smithi136.stdout: "size": 256, 2024-09-15T22:05:02.744 INFO:tasks.workunit.client.0.smithi136.stdout: "snapset": { 2024-09-15T22:05:02.744 INFO:tasks.workunit.client.0.smithi136.stdout: "seq": 1, 2024-09-15T22:05:02.745 INFO:tasks.workunit.client.0.smithi136.stdout: "clones": [ 2024-09-15T22:05:02.745 INFO:tasks.workunit.client.0.smithi136.stdout: { 2024-09-15T22:05:02.745 INFO:tasks.workunit.client.0.smithi136.stdout: "snap": 1, 2024-09-15T22:05:02.745 INFO:tasks.workunit.client.0.smithi136.stdout: "size": 1033, 2024-09-15T22:05:02.745 INFO:tasks.workunit.client.0.smithi136.stdout: "overlap": "[]", 2024-09-15T22:05:02.745 INFO:tasks.workunit.client.0.smithi136.stdout: "snaps": [ 2024-09-15T22:05:02.745 INFO:tasks.workunit.client.0.smithi136.stdout: 1 2024-09-15T22:05:02.745 INFO:tasks.workunit.client.0.smithi136.stdout: ] 2024-09-15T22:05:02.745 INFO:tasks.workunit.client.0.smithi136.stdout: } 2024-09-15T22:05:02.745 INFO:tasks.workunit.client.0.smithi136.stdout: ] 2024-09-15T22:05:02.745 INFO:tasks.workunit.client.0.smithi136.stdout: } 2024-09-15T22:05:02.745 INFO:tasks.workunit.client.0.smithi136.stdout: }, 2024-09-15T22:05:02.745 INFO:tasks.workunit.client.0.smithi136.stdout: { 2024-09-15T22:05:02.745 INFO:tasks.workunit.client.0.smithi136.stdout: "osd": 1, 2024-09-15T22:05:02.746 INFO:tasks.workunit.client.0.smithi136.stdout: "primary": true, 2024-09-15T22:05:02.746 INFO:tasks.workunit.client.0.smithi136.stdout: "errors": [], 2024-09-15T22:05:02.746 INFO:tasks.workunit.client.0.smithi136.stdout: "size": 256, 2024-09-15T22:05:02.746 INFO:tasks.workunit.client.0.smithi136.stdout: "snapset": { 2024-09-15T22:05:02.746 INFO:tasks.workunit.client.0.smithi136.stdout: "seq": 1, 2024-09-15T22:05:02.746 INFO:tasks.workunit.client.0.smithi136.stdout: "clones": [ 2024-09-15T22:05:02.746 INFO:tasks.workunit.client.0.smithi136.stdout: { 2024-09-15T22:05:02.746 INFO:tasks.workunit.client.0.smithi136.stdout: "snap": 1, 2024-09-15T22:05:02.746 INFO:tasks.workunit.client.0.smithi136.stdout: "size": 1032, 2024-09-15T22:05:02.746 INFO:tasks.workunit.client.0.smithi136.stdout: "overlap": "[]", 2024-09-15T22:05:02.746 INFO:tasks.workunit.client.0.smithi136.stdout: "snaps": [ 2024-09-15T22:05:02.746 INFO:tasks.workunit.client.0.smithi136.stdout: 1 2024-09-15T22:05:02.746 INFO:tasks.workunit.client.0.smithi136.stdout: ] 2024-09-15T22:05:02.746 INFO:tasks.workunit.client.0.smithi136.stdout: } 2024-09-15T22:05:02.747 INFO:tasks.workunit.client.0.smithi136.stdout: ] 2024-09-15T22:05:02.747 INFO:tasks.workunit.client.0.smithi136.stdout: } 2024-09-15T22:05:02.747 INFO:tasks.workunit.client.0.smithi136.stdout: } 2024-09-15T22:05:02.747 INFO:tasks.workunit.client.0.smithi136.stdout: ] 2024-09-15T22:05:02.747 INFO:tasks.workunit.client.0.smithi136.stdout: }, 2024-09-15T22:05:02.747 INFO:tasks.workunit.client.0.smithi136.stdout: { 2024-09-15T22:05:02.747 INFO:tasks.workunit.client.0.smithi136.stdout: "object": { 2024-09-15T22:05:02.747 INFO:tasks.workunit.client.0.smithi136.stdout: "name": "obj15", 2024-09-15T22:05:02.747 INFO:tasks.workunit.client.0.smithi136.stdout: "nspace": "", 2024-09-15T22:05:02.747 INFO:tasks.workunit.client.0.smithi136.stdout: "locator": "", 2024-09-15T22:05:02.747 INFO:tasks.workunit.client.0.smithi136.stdout: "snap": "head", 2024-09-15T22:05:02.747 INFO:tasks.workunit.client.0.smithi136.stdout: "version": 15 2024-09-15T22:05:02.747 INFO:tasks.workunit.client.0.smithi136.stdout: }, 2024-09-15T22:05:02.747 INFO:tasks.workunit.client.0.smithi136.stdout: "errors": [], 2024-09-15T22:05:02.747 INFO:tasks.workunit.client.0.smithi136.stdout: "union_shard_errors": [ 2024-09-15T22:05:02.748 INFO:tasks.workunit.client.0.smithi136.stdout: "snapset_corrupted" 2024-09-15T22:05:02.748 INFO:tasks.workunit.client.0.smithi136.stdout: ], 2024-09-15T22:05:02.748 INFO:tasks.workunit.client.0.smithi136.stdout: "selected_object_info": { 2024-09-15T22:05:02.748 INFO:tasks.workunit.client.0.smithi136.stdout: "oid": { 2024-09-15T22:05:02.748 INFO:tasks.workunit.client.0.smithi136.stdout: "oid": "obj15", 2024-09-15T22:05:02.748 INFO:tasks.workunit.client.0.smithi136.stdout: "key": "", 2024-09-15T22:05:02.748 INFO:tasks.workunit.client.0.smithi136.stdout: "snapid": -2, 2024-09-15T22:05:02.748 INFO:tasks.workunit.client.0.smithi136.stdout: "hash": 612772309, 2024-09-15T22:05:02.748 INFO:tasks.workunit.client.0.smithi136.stdout: "max": 0, 2024-09-15T22:05:02.748 INFO:tasks.workunit.client.0.smithi136.stdout: "pool": 1, 2024-09-15T22:05:02.748 INFO:tasks.workunit.client.0.smithi136.stdout: "namespace": "" 2024-09-15T22:05:02.748 INFO:tasks.workunit.client.0.smithi136.stdout: }, 2024-09-15T22:05:02.748 INFO:tasks.workunit.client.0.smithi136.stdout: "version": "16'15", 2024-09-15T22:05:02.748 INFO:tasks.workunit.client.0.smithi136.stdout: "prior_version": "0'0", 2024-09-15T22:05:02.749 INFO:tasks.workunit.client.0.smithi136.stdout: "last_reqid": "client.4273.0:1", 2024-09-15T22:05:02.749 INFO:tasks.workunit.client.0.smithi136.stdout: "user_version": 15, 2024-09-15T22:05:02.749 INFO:tasks.workunit.client.0.smithi136.stdout: "size": 1032, 2024-09-15T22:05:02.749 INFO:tasks.workunit.client.0.smithi136.stdout: "mtime": "2024-09-15T22:03:48.525165+0000", 2024-09-15T22:05:02.749 INFO:tasks.workunit.client.0.smithi136.stdout: "local_mtime": "2024-09-15T22:03:48.525722+0000", 2024-09-15T22:05:02.749 INFO:tasks.workunit.client.0.smithi136.stdout: "lost": 0, 2024-09-15T22:05:02.749 INFO:tasks.workunit.client.0.smithi136.stdout: "flags": [ 2024-09-15T22:05:02.749 INFO:tasks.workunit.client.0.smithi136.stdout: "dirty", 2024-09-15T22:05:02.749 INFO:tasks.workunit.client.0.smithi136.stdout: "data_digest" 2024-09-15T22:05:02.749 INFO:tasks.workunit.client.0.smithi136.stdout: ], 2024-09-15T22:05:02.749 INFO:tasks.workunit.client.0.smithi136.stdout: "truncate_seq": 0, 2024-09-15T22:05:02.749 INFO:tasks.workunit.client.0.smithi136.stdout: "truncate_size": 0, 2024-09-15T22:05:02.749 INFO:tasks.workunit.client.0.smithi136.stdout: "data_digest": "0x0e25be29", 2024-09-15T22:05:02.749 INFO:tasks.workunit.client.0.smithi136.stdout: "omap_digest": "0xffffffff", 2024-09-15T22:05:02.749 INFO:tasks.workunit.client.0.smithi136.stdout: "expected_object_size": 0, 2024-09-15T22:05:02.750 INFO:tasks.workunit.client.0.smithi136.stdout: "expected_write_size": 0, 2024-09-15T22:05:02.750 INFO:tasks.workunit.client.0.smithi136.stdout: "alloc_hint_flags": 0, 2024-09-15T22:05:02.750 INFO:tasks.workunit.client.0.smithi136.stdout: "manifest": { 2024-09-15T22:05:02.750 INFO:tasks.workunit.client.0.smithi136.stdout: "type": 0 2024-09-15T22:05:02.750 INFO:tasks.workunit.client.0.smithi136.stdout: }, 2024-09-15T22:05:02.750 INFO:tasks.workunit.client.0.smithi136.stdout: "watchers": {} 2024-09-15T22:05:02.750 INFO:tasks.workunit.client.0.smithi136.stdout: }, 2024-09-15T22:05:02.750 INFO:tasks.workunit.client.0.smithi136.stdout: "shards": [ 2024-09-15T22:05:02.750 INFO:tasks.workunit.client.0.smithi136.stdout: { 2024-09-15T22:05:02.750 INFO:tasks.workunit.client.0.smithi136.stdout: "osd": 0, 2024-09-15T22:05:02.750 INFO:tasks.workunit.client.0.smithi136.stdout: "primary": false, 2024-09-15T22:05:02.750 INFO:tasks.workunit.client.0.smithi136.stdout: "errors": [ 2024-09-15T22:05:02.750 INFO:tasks.workunit.client.0.smithi136.stdout: "snapset_corrupted" 2024-09-15T22:05:02.750 INFO:tasks.workunit.client.0.smithi136.stdout: ], 2024-09-15T22:05:02.750 INFO:tasks.workunit.client.0.smithi136.stdout: "size": 1032, 2024-09-15T22:05:02.751 INFO:tasks.workunit.client.0.smithi136.stdout: "snapset": "Z2FyYmFnZQo=" 2024-09-15T22:05:02.751 INFO:tasks.workunit.client.0.smithi136.stdout: }, 2024-09-15T22:05:02.751 INFO:tasks.workunit.client.0.smithi136.stdout: { 2024-09-15T22:05:02.751 INFO:tasks.workunit.client.0.smithi136.stdout: "osd": 1, 2024-09-15T22:05:02.751 INFO:tasks.workunit.client.0.smithi136.stdout: "primary": true, 2024-09-15T22:05:02.751 INFO:tasks.workunit.client.0.smithi136.stdout: "errors": [], 2024-09-15T22:05:02.751 INFO:tasks.workunit.client.0.smithi136.stdout: "size": 1032, 2024-09-15T22:05:02.751 INFO:tasks.workunit.client.0.smithi136.stdout: "snapset": { 2024-09-15T22:05:02.751 INFO:tasks.workunit.client.0.smithi136.stdout: "seq": 0, 2024-09-15T22:05:02.751 INFO:tasks.workunit.client.0.smithi136.stdout: "clones": [] 2024-09-15T22:05:02.751 INFO:tasks.workunit.client.0.smithi136.stdout: } 2024-09-15T22:05:02.751 INFO:tasks.workunit.client.0.smithi136.stdout: } 2024-09-15T22:05:02.751 INFO:tasks.workunit.client.0.smithi136.stdout: ] 2024-09-15T22:05:02.751 INFO:tasks.workunit.client.0.smithi136.stdout: }, 2024-09-15T22:05:02.751 INFO:tasks.workunit.client.0.smithi136.stdout: { 2024-09-15T22:05:02.752 INFO:tasks.workunit.client.0.smithi136.stdout: "object": { 2024-09-15T22:05:02.752 INFO:tasks.workunit.client.0.smithi136.stdout: "name": "obj2", 2024-09-15T22:05:02.752 INFO:tasks.workunit.client.0.smithi136.stdout: "nspace": "", 2024-09-15T22:05:02.752 INFO:tasks.workunit.client.0.smithi136.stdout: "locator": "", 2024-09-15T22:05:02.752 INFO:tasks.workunit.client.0.smithi136.stdout: "snap": "head", 2024-09-15T22:05:02.752 INFO:tasks.workunit.client.0.smithi136.stdout: "version": 56 2024-09-15T22:05:02.752 INFO:tasks.workunit.client.0.smithi136.stdout: }, 2024-09-15T22:05:02.752 INFO:tasks.workunit.client.0.smithi136.stdout: "errors": [], 2024-09-15T22:05:02.752 INFO:tasks.workunit.client.0.smithi136.stdout: "union_shard_errors": [ 2024-09-15T22:05:02.752 INFO:tasks.workunit.client.0.smithi136.stdout: "snapset_missing" 2024-09-15T22:05:02.752 INFO:tasks.workunit.client.0.smithi136.stdout: ], 2024-09-15T22:05:02.752 INFO:tasks.workunit.client.0.smithi136.stdout: "selected_object_info": { 2024-09-15T22:05:02.752 INFO:tasks.workunit.client.0.smithi136.stdout: "oid": { 2024-09-15T22:05:02.752 INFO:tasks.workunit.client.0.smithi136.stdout: "oid": "obj2", 2024-09-15T22:05:02.752 INFO:tasks.workunit.client.0.smithi136.stdout: "key": "", 2024-09-15T22:05:02.753 INFO:tasks.workunit.client.0.smithi136.stdout: "snapid": -2, 2024-09-15T22:05:02.753 INFO:tasks.workunit.client.0.smithi136.stdout: "hash": 1058988552, 2024-09-15T22:05:02.753 INFO:tasks.workunit.client.0.smithi136.stdout: "max": 0, 2024-09-15T22:05:02.753 INFO:tasks.workunit.client.0.smithi136.stdout: "pool": 1, 2024-09-15T22:05:02.753 INFO:tasks.workunit.client.0.smithi136.stdout: "namespace": "" 2024-09-15T22:05:02.753 INFO:tasks.workunit.client.0.smithi136.stdout: }, 2024-09-15T22:05:02.753 INFO:tasks.workunit.client.0.smithi136.stdout: "version": "23'56", 2024-09-15T22:05:02.753 INFO:tasks.workunit.client.0.smithi136.stdout: "prior_version": "20'48", 2024-09-15T22:05:02.753 INFO:tasks.workunit.client.0.smithi136.stdout: "last_reqid": "client.4363.0:1", 2024-09-15T22:05:02.753 INFO:tasks.workunit.client.0.smithi136.stdout: "user_version": 56, 2024-09-15T22:05:02.753 INFO:tasks.workunit.client.0.smithi136.stdout: "size": 0, 2024-09-15T22:05:02.753 INFO:tasks.workunit.client.0.smithi136.stdout: "mtime": "2024-09-15T22:03:54.601099+0000", 2024-09-15T22:05:02.753 INFO:tasks.workunit.client.0.smithi136.stdout: "local_mtime": "2024-09-15T22:03:54.601640+0000", 2024-09-15T22:05:02.754 INFO:tasks.workunit.client.0.smithi136.stdout: "lost": 0, 2024-09-15T22:05:02.754 INFO:tasks.workunit.client.0.smithi136.stdout: "flags": [ 2024-09-15T22:05:02.754 INFO:tasks.workunit.client.0.smithi136.stdout: "whiteout", 2024-09-15T22:05:02.754 INFO:tasks.workunit.client.0.smithi136.stdout: "dirty" 2024-09-15T22:05:02.754 INFO:tasks.workunit.client.0.smithi136.stdout: ], 2024-09-15T22:05:02.754 INFO:tasks.workunit.client.0.smithi136.stdout: "truncate_seq": 0, 2024-09-15T22:05:02.754 INFO:tasks.workunit.client.0.smithi136.stdout: "truncate_size": 0, 2024-09-15T22:05:02.754 INFO:tasks.workunit.client.0.smithi136.stdout: "data_digest": "0xffffffff", 2024-09-15T22:05:02.754 INFO:tasks.workunit.client.0.smithi136.stdout: "omap_digest": "0xffffffff", 2024-09-15T22:05:02.754 INFO:tasks.workunit.client.0.smithi136.stdout: "expected_object_size": 0, 2024-09-15T22:05:02.754 INFO:tasks.workunit.client.0.smithi136.stdout: "expected_write_size": 0, 2024-09-15T22:05:02.754 INFO:tasks.workunit.client.0.smithi136.stdout: "alloc_hint_flags": 0, 2024-09-15T22:05:02.754 INFO:tasks.workunit.client.0.smithi136.stdout: "manifest": { 2024-09-15T22:05:02.754 INFO:tasks.workunit.client.0.smithi136.stdout: "type": 0 2024-09-15T22:05:02.755 INFO:tasks.workunit.client.0.smithi136.stdout: }, 2024-09-15T22:05:02.755 INFO:tasks.workunit.client.0.smithi136.stdout: "watchers": {} 2024-09-15T22:05:02.755 INFO:tasks.workunit.client.0.smithi136.stdout: }, 2024-09-15T22:05:02.755 INFO:tasks.workunit.client.0.smithi136.stdout: "shards": [ 2024-09-15T22:05:02.755 INFO:tasks.workunit.client.0.smithi136.stdout: { 2024-09-15T22:05:02.755 INFO:tasks.workunit.client.0.smithi136.stdout: "osd": 0, 2024-09-15T22:05:02.755 INFO:tasks.workunit.client.0.smithi136.stdout: "primary": false, 2024-09-15T22:05:02.755 INFO:tasks.workunit.client.0.smithi136.stdout: "errors": [ 2024-09-15T22:05:02.755 INFO:tasks.workunit.client.0.smithi136.stdout: "snapset_missing" 2024-09-15T22:05:02.755 INFO:tasks.workunit.client.0.smithi136.stdout: ], 2024-09-15T22:05:02.755 INFO:tasks.workunit.client.0.smithi136.stdout: "size": 0 2024-09-15T22:05:02.755 INFO:tasks.workunit.client.0.smithi136.stdout: }, 2024-09-15T22:05:02.755 INFO:tasks.workunit.client.0.smithi136.stdout: { 2024-09-15T22:05:02.755 INFO:tasks.workunit.client.0.smithi136.stdout: "osd": 1, 2024-09-15T22:05:02.756 INFO:tasks.workunit.client.0.smithi136.stdout: "primary": true, 2024-09-15T22:05:02.756 INFO:tasks.workunit.client.0.smithi136.stdout: "errors": [], 2024-09-15T22:05:02.756 INFO:tasks.workunit.client.0.smithi136.stdout: "size": 0, 2024-09-15T22:05:02.756 INFO:tasks.workunit.client.0.smithi136.stdout: "snapset": { 2024-09-15T22:05:02.756 INFO:tasks.workunit.client.0.smithi136.stdout: "seq": 7, 2024-09-15T22:05:02.756 INFO:tasks.workunit.client.0.smithi136.stdout: "clones": [ 2024-09-15T22:05:02.756 INFO:tasks.workunit.client.0.smithi136.stdout: { 2024-09-15T22:05:02.756 INFO:tasks.workunit.client.0.smithi136.stdout: "snap": 4, 2024-09-15T22:05:02.756 INFO:tasks.workunit.client.0.smithi136.stdout: "size": 1032, 2024-09-15T22:05:02.756 INFO:tasks.workunit.client.0.smithi136.stdout: "overlap": "[]", 2024-09-15T22:05:02.756 INFO:tasks.workunit.client.0.smithi136.stdout: "snaps": [ 2024-09-15T22:05:02.756 INFO:tasks.workunit.client.0.smithi136.stdout: 4, 2024-09-15T22:05:02.756 INFO:tasks.workunit.client.0.smithi136.stdout: 3, 2024-09-15T22:05:02.756 INFO:tasks.workunit.client.0.smithi136.stdout: 2, 2024-09-15T22:05:02.757 INFO:tasks.workunit.client.0.smithi136.stdout: 1 2024-09-15T22:05:02.757 INFO:tasks.workunit.client.0.smithi136.stdout: ] 2024-09-15T22:05:02.757 INFO:tasks.workunit.client.0.smithi136.stdout: }, 2024-09-15T22:05:02.757 INFO:tasks.workunit.client.0.smithi136.stdout: { 2024-09-15T22:05:02.757 INFO:tasks.workunit.client.0.smithi136.stdout: "snap": 7, 2024-09-15T22:05:02.757 INFO:tasks.workunit.client.0.smithi136.stdout: "size": 1024, 2024-09-15T22:05:02.757 INFO:tasks.workunit.client.0.smithi136.stdout: "overlap": "[]", 2024-09-15T22:05:02.757 INFO:tasks.workunit.client.0.smithi136.stdout: "snaps": [ 2024-09-15T22:05:02.757 INFO:tasks.workunit.client.0.smithi136.stdout: 7, 2024-09-15T22:05:02.757 INFO:tasks.workunit.client.0.smithi136.stdout: 6, 2024-09-15T22:05:02.757 INFO:tasks.workunit.client.0.smithi136.stdout: 5 2024-09-15T22:05:02.757 INFO:tasks.workunit.client.0.smithi136.stdout: ] 2024-09-15T22:05:02.757 INFO:tasks.workunit.client.0.smithi136.stdout: } 2024-09-15T22:05:02.758 INFO:tasks.workunit.client.0.smithi136.stdout: ] 2024-09-15T22:05:02.758 INFO:tasks.workunit.client.0.smithi136.stdout: } 2024-09-15T22:05:02.758 INFO:tasks.workunit.client.0.smithi136.stdout: } 2024-09-15T22:05:02.758 INFO:tasks.workunit.client.0.smithi136.stdout: ] 2024-09-15T22:05:02.758 INFO:tasks.workunit.client.0.smithi136.stdout: }, 2024-09-15T22:05:02.758 INFO:tasks.workunit.client.0.smithi136.stdout: { 2024-09-15T22:05:02.758 INFO:tasks.workunit.client.0.smithi136.stdout: "object": { 2024-09-15T22:05:02.758 INFO:tasks.workunit.client.0.smithi136.stdout: "name": "obj3", 2024-09-15T22:05:02.758 INFO:tasks.workunit.client.0.smithi136.stdout: "nspace": "", 2024-09-15T22:05:02.758 INFO:tasks.workunit.client.0.smithi136.stdout: "locator": "", 2024-09-15T22:05:02.758 INFO:tasks.workunit.client.0.smithi136.stdout: "snap": "head", 2024-09-15T22:05:02.758 INFO:tasks.workunit.client.0.smithi136.stdout: "version": 44 2024-09-15T22:05:02.758 INFO:tasks.workunit.client.0.smithi136.stdout: }, 2024-09-15T22:05:02.759 INFO:tasks.workunit.client.0.smithi136.stdout: "errors": [ 2024-09-15T22:05:02.759 INFO:tasks.workunit.client.0.smithi136.stdout: "size_mismatch" 2024-09-15T22:05:02.759 INFO:tasks.workunit.client.0.smithi136.stdout: ], 2024-09-15T22:05:02.759 INFO:tasks.workunit.client.0.smithi136.stdout: "union_shard_errors": [ 2024-09-15T22:05:02.759 INFO:tasks.workunit.client.0.smithi136.stdout: "size_mismatch_info", 2024-09-15T22:05:02.759 INFO:tasks.workunit.client.0.smithi136.stdout: "obj_size_info_mismatch" 2024-09-15T22:05:02.759 INFO:tasks.workunit.client.0.smithi136.stdout: ], 2024-09-15T22:05:02.759 INFO:tasks.workunit.client.0.smithi136.stdout: "selected_object_info": { 2024-09-15T22:05:02.759 INFO:tasks.workunit.client.0.smithi136.stdout: "oid": { 2024-09-15T22:05:02.759 INFO:tasks.workunit.client.0.smithi136.stdout: "oid": "obj3", 2024-09-15T22:05:02.759 INFO:tasks.workunit.client.0.smithi136.stdout: "key": "", 2024-09-15T22:05:02.759 INFO:tasks.workunit.client.0.smithi136.stdout: "snapid": -2, 2024-09-15T22:05:02.759 INFO:tasks.workunit.client.0.smithi136.stdout: "hash": 1643547569, 2024-09-15T22:05:02.759 INFO:tasks.workunit.client.0.smithi136.stdout: "max": 0, 2024-09-15T22:05:02.760 INFO:tasks.workunit.client.0.smithi136.stdout: "pool": 1, 2024-09-15T22:05:02.760 INFO:tasks.workunit.client.0.smithi136.stdout: "namespace": "" 2024-09-15T22:05:02.760 INFO:tasks.workunit.client.0.smithi136.stdout: }, 2024-09-15T22:05:02.760 INFO:tasks.workunit.client.0.smithi136.stdout: "version": "19'44", 2024-09-15T22:05:02.760 INFO:tasks.workunit.client.0.smithi136.stdout: "prior_version": "17'22", 2024-09-15T22:05:02.760 INFO:tasks.workunit.client.0.smithi136.stdout: "last_reqid": "client.4333.0:1", 2024-09-15T22:05:02.760 INFO:tasks.workunit.client.0.smithi136.stdout: "user_version": 44, 2024-09-15T22:05:02.760 INFO:tasks.workunit.client.0.smithi136.stdout: "size": 768, 2024-09-15T22:05:02.760 INFO:tasks.workunit.client.0.smithi136.stdout: "mtime": "2024-09-15T22:03:52.326331+0000", 2024-09-15T22:05:02.760 INFO:tasks.workunit.client.0.smithi136.stdout: "local_mtime": "2024-09-15T22:03:52.344917+0000", 2024-09-15T22:05:02.760 INFO:tasks.workunit.client.0.smithi136.stdout: "lost": 0, 2024-09-15T22:05:02.760 INFO:tasks.workunit.client.0.smithi136.stdout: "flags": [ 2024-09-15T22:05:02.760 INFO:tasks.workunit.client.0.smithi136.stdout: "dirty", 2024-09-15T22:05:02.761 INFO:tasks.workunit.client.0.smithi136.stdout: "data_digest" 2024-09-15T22:05:02.761 INFO:tasks.workunit.client.0.smithi136.stdout: ], 2024-09-15T22:05:02.761 INFO:tasks.workunit.client.0.smithi136.stdout: "truncate_seq": 0, 2024-09-15T22:05:02.761 INFO:tasks.workunit.client.0.smithi136.stdout: "truncate_size": 0, 2024-09-15T22:05:02.761 INFO:tasks.workunit.client.0.smithi136.stdout: "data_digest": "0x582becc6", 2024-09-15T22:05:02.761 INFO:tasks.workunit.client.0.smithi136.stdout: "omap_digest": "0xffffffff", 2024-09-15T22:05:02.761 INFO:tasks.workunit.client.0.smithi136.stdout: "expected_object_size": 0, 2024-09-15T22:05:02.761 INFO:tasks.workunit.client.0.smithi136.stdout: "expected_write_size": 0, 2024-09-15T22:05:02.761 INFO:tasks.workunit.client.0.smithi136.stdout: "alloc_hint_flags": 0, 2024-09-15T22:05:02.761 INFO:tasks.workunit.client.0.smithi136.stdout: "manifest": { 2024-09-15T22:05:02.761 INFO:tasks.workunit.client.0.smithi136.stdout: "type": 0 2024-09-15T22:05:02.761 INFO:tasks.workunit.client.0.smithi136.stdout: }, 2024-09-15T22:05:02.761 INFO:tasks.workunit.client.0.smithi136.stdout: "watchers": {} 2024-09-15T22:05:02.761 INFO:tasks.workunit.client.0.smithi136.stdout: }, 2024-09-15T22:05:02.761 INFO:tasks.workunit.client.0.smithi136.stdout: "shards": [ 2024-09-15T22:05:02.762 INFO:tasks.workunit.client.0.smithi136.stdout: { 2024-09-15T22:05:02.762 INFO:tasks.workunit.client.0.smithi136.stdout: "osd": 0, 2024-09-15T22:05:02.762 INFO:tasks.workunit.client.0.smithi136.stdout: "primary": false, 2024-09-15T22:05:02.762 INFO:tasks.workunit.client.0.smithi136.stdout: "errors": [ 2024-09-15T22:05:02.762 INFO:tasks.workunit.client.0.smithi136.stdout: "size_mismatch_info", 2024-09-15T22:05:02.762 INFO:tasks.workunit.client.0.smithi136.stdout: "obj_size_info_mismatch" 2024-09-15T22:05:02.762 INFO:tasks.workunit.client.0.smithi136.stdout: ], 2024-09-15T22:05:02.762 INFO:tasks.workunit.client.0.smithi136.stdout: "size": 3840, 2024-09-15T22:05:02.762 INFO:tasks.workunit.client.0.smithi136.stdout: "object_info": { 2024-09-15T22:05:02.762 INFO:tasks.workunit.client.0.smithi136.stdout: "oid": { 2024-09-15T22:05:02.762 INFO:tasks.workunit.client.0.smithi136.stdout: "oid": "obj3", 2024-09-15T22:05:02.762 INFO:tasks.workunit.client.0.smithi136.stdout: "key": "", 2024-09-15T22:05:02.762 INFO:tasks.workunit.client.0.smithi136.stdout: "snapid": -2, 2024-09-15T22:05:02.762 INFO:tasks.workunit.client.0.smithi136.stdout: "hash": 1643547569, 2024-09-15T22:05:02.762 INFO:tasks.workunit.client.0.smithi136.stdout: "max": 0, 2024-09-15T22:05:02.763 INFO:tasks.workunit.client.0.smithi136.stdout: "pool": 1, 2024-09-15T22:05:02.763 INFO:tasks.workunit.client.0.smithi136.stdout: "namespace": "" 2024-09-15T22:05:02.763 INFO:tasks.workunit.client.0.smithi136.stdout: }, 2024-09-15T22:05:02.763 INFO:tasks.workunit.client.0.smithi136.stdout: "version": "19'44", 2024-09-15T22:05:02.763 INFO:tasks.workunit.client.0.smithi136.stdout: "prior_version": "17'22", 2024-09-15T22:05:02.763 INFO:tasks.workunit.client.0.smithi136.stdout: "last_reqid": "client.4333.0:1", 2024-09-15T22:05:02.763 INFO:tasks.workunit.client.0.smithi136.stdout: "user_version": 44, 2024-09-15T22:05:02.763 INFO:tasks.workunit.client.0.smithi136.stdout: "size": 768, 2024-09-15T22:05:02.763 INFO:tasks.workunit.client.0.smithi136.stdout: "mtime": "2024-09-15T22:03:52.326331+0000", 2024-09-15T22:05:02.763 INFO:tasks.workunit.client.0.smithi136.stdout: "local_mtime": "2024-09-15T22:03:52.344917+0000", 2024-09-15T22:05:02.763 INFO:tasks.workunit.client.0.smithi136.stdout: "lost": 0, 2024-09-15T22:05:02.763 INFO:tasks.workunit.client.0.smithi136.stdout: "flags": [ 2024-09-15T22:05:02.763 INFO:tasks.workunit.client.0.smithi136.stdout: "dirty", 2024-09-15T22:05:02.763 INFO:tasks.workunit.client.0.smithi136.stdout: "data_digest" 2024-09-15T22:05:02.764 INFO:tasks.workunit.client.0.smithi136.stdout: ], 2024-09-15T22:05:02.764 INFO:tasks.workunit.client.0.smithi136.stdout: "truncate_seq": 0, 2024-09-15T22:05:02.764 INFO:tasks.workunit.client.0.smithi136.stdout: "truncate_size": 0, 2024-09-15T22:05:02.764 INFO:tasks.workunit.client.0.smithi136.stdout: "data_digest": "0x582becc6", 2024-09-15T22:05:02.764 INFO:tasks.workunit.client.0.smithi136.stdout: "omap_digest": "0xffffffff", 2024-09-15T22:05:02.764 INFO:tasks.workunit.client.0.smithi136.stdout: "expected_object_size": 0, 2024-09-15T22:05:02.764 INFO:tasks.workunit.client.0.smithi136.stdout: "expected_write_size": 0, 2024-09-15T22:05:02.764 INFO:tasks.workunit.client.0.smithi136.stdout: "alloc_hint_flags": 0, 2024-09-15T22:05:02.764 INFO:tasks.workunit.client.0.smithi136.stdout: "manifest": { 2024-09-15T22:05:02.764 INFO:tasks.workunit.client.0.smithi136.stdout: "type": 0 2024-09-15T22:05:02.764 INFO:tasks.workunit.client.0.smithi136.stdout: }, 2024-09-15T22:05:02.764 INFO:tasks.workunit.client.0.smithi136.stdout: "watchers": {} 2024-09-15T22:05:02.764 INFO:tasks.workunit.client.0.smithi136.stdout: } 2024-09-15T22:05:02.764 INFO:tasks.workunit.client.0.smithi136.stdout: }, 2024-09-15T22:05:02.764 INFO:tasks.workunit.client.0.smithi136.stdout: { 2024-09-15T22:05:02.765 INFO:tasks.workunit.client.0.smithi136.stdout: "osd": 1, 2024-09-15T22:05:02.765 INFO:tasks.workunit.client.0.smithi136.stdout: "primary": true, 2024-09-15T22:05:02.765 INFO:tasks.workunit.client.0.smithi136.stdout: "errors": [], 2024-09-15T22:05:02.765 INFO:tasks.workunit.client.0.smithi136.stdout: "size": 768 2024-09-15T22:05:02.765 INFO:tasks.workunit.client.0.smithi136.stdout: } 2024-09-15T22:05:02.765 INFO:tasks.workunit.client.0.smithi136.stdout: ] 2024-09-15T22:05:02.765 INFO:tasks.workunit.client.0.smithi136.stdout: }, 2024-09-15T22:05:02.765 INFO:tasks.workunit.client.0.smithi136.stdout: { 2024-09-15T22:05:02.765 INFO:tasks.workunit.client.0.smithi136.stdout: "object": { 2024-09-15T22:05:02.765 INFO:tasks.workunit.client.0.smithi136.stdout: "name": "obj6", 2024-09-15T22:05:02.765 INFO:tasks.workunit.client.0.smithi136.stdout: "nspace": "", 2024-09-15T22:05:02.765 INFO:tasks.workunit.client.0.smithi136.stdout: "locator": "", 2024-09-15T22:05:02.765 INFO:tasks.workunit.client.0.smithi136.stdout: "snap": "head", 2024-09-15T22:05:02.765 INFO:tasks.workunit.client.0.smithi136.stdout: "version": 24 2024-09-15T22:05:02.766 INFO:tasks.workunit.client.0.smithi136.stdout: }, 2024-09-15T22:05:02.766 INFO:tasks.workunit.client.0.smithi136.stdout: "errors": [ 2024-09-15T22:05:02.766 INFO:tasks.workunit.client.0.smithi136.stdout: "snapset_inconsistency" 2024-09-15T22:05:02.766 INFO:tasks.workunit.client.0.smithi136.stdout: ], 2024-09-15T22:05:02.766 INFO:tasks.workunit.client.0.smithi136.stdout: "union_shard_errors": [], 2024-09-15T22:05:02.766 INFO:tasks.workunit.client.0.smithi136.stdout: "selected_object_info": { 2024-09-15T22:05:02.766 INFO:tasks.workunit.client.0.smithi136.stdout: "oid": { 2024-09-15T22:05:02.766 INFO:tasks.workunit.client.0.smithi136.stdout: "oid": "obj6", 2024-09-15T22:05:02.766 INFO:tasks.workunit.client.0.smithi136.stdout: "key": "", 2024-09-15T22:05:02.766 INFO:tasks.workunit.client.0.smithi136.stdout: "snapid": -2, 2024-09-15T22:05:02.766 INFO:tasks.workunit.client.0.smithi136.stdout: "hash": 2202164420, 2024-09-15T22:05:02.766 INFO:tasks.workunit.client.0.smithi136.stdout: "max": 0, 2024-09-15T22:05:02.766 INFO:tasks.workunit.client.0.smithi136.stdout: "pool": 1, 2024-09-15T22:05:02.766 INFO:tasks.workunit.client.0.smithi136.stdout: "namespace": "" 2024-09-15T22:05:02.767 INFO:tasks.workunit.client.0.smithi136.stdout: }, 2024-09-15T22:05:02.767 INFO:tasks.workunit.client.0.smithi136.stdout: "version": "17'24", 2024-09-15T22:05:02.767 INFO:tasks.workunit.client.0.smithi136.stdout: "prior_version": "16'6", 2024-09-15T22:05:02.767 INFO:tasks.workunit.client.0.smithi136.stdout: "last_reqid": "client.4297.0:1", 2024-09-15T22:05:02.767 INFO:tasks.workunit.client.0.smithi136.stdout: "user_version": 24, 2024-09-15T22:05:02.767 INFO:tasks.workunit.client.0.smithi136.stdout: "size": 256, 2024-09-15T22:05:02.767 INFO:tasks.workunit.client.0.smithi136.stdout: "mtime": "2024-09-15T22:03:50.432442+0000", 2024-09-15T22:05:02.767 INFO:tasks.workunit.client.0.smithi136.stdout: "local_mtime": "2024-09-15T22:03:50.433013+0000", 2024-09-15T22:05:02.767 INFO:tasks.workunit.client.0.smithi136.stdout: "lost": 0, 2024-09-15T22:05:02.767 INFO:tasks.workunit.client.0.smithi136.stdout: "flags": [ 2024-09-15T22:05:02.767 INFO:tasks.workunit.client.0.smithi136.stdout: "dirty", 2024-09-15T22:05:02.767 INFO:tasks.workunit.client.0.smithi136.stdout: "data_digest" 2024-09-15T22:05:02.767 INFO:tasks.workunit.client.0.smithi136.stdout: ], 2024-09-15T22:05:02.767 INFO:tasks.workunit.client.0.smithi136.stdout: "truncate_seq": 0, 2024-09-15T22:05:02.767 INFO:tasks.workunit.client.0.smithi136.stdout: "truncate_size": 0, 2024-09-15T22:05:02.768 INFO:tasks.workunit.client.0.smithi136.stdout: "data_digest": "0x8d0e1329", 2024-09-15T22:05:02.768 INFO:tasks.workunit.client.0.smithi136.stdout: "omap_digest": "0xffffffff", 2024-09-15T22:05:02.768 INFO:tasks.workunit.client.0.smithi136.stdout: "expected_object_size": 0, 2024-09-15T22:05:02.768 INFO:tasks.workunit.client.0.smithi136.stdout: "expected_write_size": 0, 2024-09-15T22:05:02.768 INFO:tasks.workunit.client.0.smithi136.stdout: "alloc_hint_flags": 0, 2024-09-15T22:05:02.768 INFO:tasks.workunit.client.0.smithi136.stdout: "manifest": { 2024-09-15T22:05:02.768 INFO:tasks.workunit.client.0.smithi136.stdout: "type": 0 2024-09-15T22:05:02.768 INFO:tasks.workunit.client.0.smithi136.stdout: }, 2024-09-15T22:05:02.768 INFO:tasks.workunit.client.0.smithi136.stdout: "watchers": {} 2024-09-15T22:05:02.768 INFO:tasks.workunit.client.0.smithi136.stdout: }, 2024-09-15T22:05:02.768 INFO:tasks.workunit.client.0.smithi136.stdout: "shards": [ 2024-09-15T22:05:02.768 INFO:tasks.workunit.client.0.smithi136.stdout: { 2024-09-15T22:05:02.768 INFO:tasks.workunit.client.0.smithi136.stdout: "osd": 0, 2024-09-15T22:05:02.768 INFO:tasks.workunit.client.0.smithi136.stdout: "primary": false, 2024-09-15T22:05:02.769 INFO:tasks.workunit.client.0.smithi136.stdout: "errors": [], 2024-09-15T22:05:02.769 INFO:tasks.workunit.client.0.smithi136.stdout: "size": 256, 2024-09-15T22:05:02.769 INFO:tasks.workunit.client.0.smithi136.stdout: "snapset": { 2024-09-15T22:05:02.769 INFO:tasks.workunit.client.0.smithi136.stdout: "seq": 1, 2024-09-15T22:05:02.769 INFO:tasks.workunit.client.0.smithi136.stdout: "clones": [] 2024-09-15T22:05:02.769 INFO:tasks.workunit.client.0.smithi136.stdout: } 2024-09-15T22:05:02.769 INFO:tasks.workunit.client.0.smithi136.stdout: }, 2024-09-15T22:05:02.769 INFO:tasks.workunit.client.0.smithi136.stdout: { 2024-09-15T22:05:02.769 INFO:tasks.workunit.client.0.smithi136.stdout: "osd": 1, 2024-09-15T22:05:02.769 INFO:tasks.workunit.client.0.smithi136.stdout: "primary": true, 2024-09-15T22:05:02.769 INFO:tasks.workunit.client.0.smithi136.stdout: "errors": [], 2024-09-15T22:05:02.769 INFO:tasks.workunit.client.0.smithi136.stdout: "size": 256, 2024-09-15T22:05:02.769 INFO:tasks.workunit.client.0.smithi136.stdout: "snapset": { 2024-09-15T22:05:02.769 INFO:tasks.workunit.client.0.smithi136.stdout: "seq": 1, 2024-09-15T22:05:02.769 INFO:tasks.workunit.client.0.smithi136.stdout: "clones": [ 2024-09-15T22:05:02.770 INFO:tasks.workunit.client.0.smithi136.stdout: { 2024-09-15T22:05:02.770 INFO:tasks.workunit.client.0.smithi136.stdout: "snap": 1, 2024-09-15T22:05:02.770 INFO:tasks.workunit.client.0.smithi136.stdout: "size": 1032, 2024-09-15T22:05:02.770 INFO:tasks.workunit.client.0.smithi136.stdout: "overlap": "[]", 2024-09-15T22:05:02.770 INFO:tasks.workunit.client.0.smithi136.stdout: "snaps": [ 2024-09-15T22:05:02.770 INFO:tasks.workunit.client.0.smithi136.stdout: 1 2024-09-15T22:05:02.770 INFO:tasks.workunit.client.0.smithi136.stdout: ] 2024-09-15T22:05:02.770 INFO:tasks.workunit.client.0.smithi136.stdout: } 2024-09-15T22:05:02.770 INFO:tasks.workunit.client.0.smithi136.stdout: ] 2024-09-15T22:05:02.770 INFO:tasks.workunit.client.0.smithi136.stdout: } 2024-09-15T22:05:02.770 INFO:tasks.workunit.client.0.smithi136.stdout: } 2024-09-15T22:05:02.770 INFO:tasks.workunit.client.0.smithi136.stdout: ] 2024-09-15T22:05:02.770 INFO:tasks.workunit.client.0.smithi136.stdout: }, 2024-09-15T22:05:02.770 INFO:tasks.workunit.client.0.smithi136.stdout: { 2024-09-15T22:05:02.771 INFO:tasks.workunit.client.0.smithi136.stdout: "object": { 2024-09-15T22:05:02.771 INFO:tasks.workunit.client.0.smithi136.stdout: "name": "obj7", 2024-09-15T22:05:02.771 INFO:tasks.workunit.client.0.smithi136.stdout: "nspace": "", 2024-09-15T22:05:02.771 INFO:tasks.workunit.client.0.smithi136.stdout: "locator": "", 2024-09-15T22:05:02.771 INFO:tasks.workunit.client.0.smithi136.stdout: "snap": "head", 2024-09-15T22:05:02.771 INFO:tasks.workunit.client.0.smithi136.stdout: "version": 26 2024-09-15T22:05:02.771 INFO:tasks.workunit.client.0.smithi136.stdout: }, 2024-09-15T22:05:02.771 INFO:tasks.workunit.client.0.smithi136.stdout: "errors": [ 2024-09-15T22:05:02.771 INFO:tasks.workunit.client.0.smithi136.stdout: "snapset_inconsistency" 2024-09-15T22:05:02.771 INFO:tasks.workunit.client.0.smithi136.stdout: ], 2024-09-15T22:05:02.771 INFO:tasks.workunit.client.0.smithi136.stdout: "union_shard_errors": [], 2024-09-15T22:05:02.771 INFO:tasks.workunit.client.0.smithi136.stdout: "selected_object_info": { 2024-09-15T22:05:02.771 INFO:tasks.workunit.client.0.smithi136.stdout: "oid": { 2024-09-15T22:05:02.771 INFO:tasks.workunit.client.0.smithi136.stdout: "oid": "obj7", 2024-09-15T22:05:02.772 INFO:tasks.workunit.client.0.smithi136.stdout: "key": "", 2024-09-15T22:05:02.772 INFO:tasks.workunit.client.0.smithi136.stdout: "snapid": -2, 2024-09-15T22:05:02.772 INFO:tasks.workunit.client.0.smithi136.stdout: "hash": 1552453721, 2024-09-15T22:05:02.772 INFO:tasks.workunit.client.0.smithi136.stdout: "max": 0, 2024-09-15T22:05:02.772 INFO:tasks.workunit.client.0.smithi136.stdout: "pool": 1, 2024-09-15T22:05:02.772 INFO:tasks.workunit.client.0.smithi136.stdout: "namespace": "" 2024-09-15T22:05:02.772 INFO:tasks.workunit.client.0.smithi136.stdout: }, 2024-09-15T22:05:02.772 INFO:tasks.workunit.client.0.smithi136.stdout: "version": "17'26", 2024-09-15T22:05:02.772 INFO:tasks.workunit.client.0.smithi136.stdout: "prior_version": "16'7", 2024-09-15T22:05:02.772 INFO:tasks.workunit.client.0.smithi136.stdout: "last_reqid": "client.4300.0:1", 2024-09-15T22:05:02.772 INFO:tasks.workunit.client.0.smithi136.stdout: "user_version": 26, 2024-09-15T22:05:02.772 INFO:tasks.workunit.client.0.smithi136.stdout: "size": 256, 2024-09-15T22:05:02.772 INFO:tasks.workunit.client.0.smithi136.stdout: "mtime": "2024-09-15T22:03:50.600719+0000", 2024-09-15T22:05:02.772 INFO:tasks.workunit.client.0.smithi136.stdout: "local_mtime": "2024-09-15T22:03:50.601342+0000", 2024-09-15T22:05:02.772 INFO:tasks.workunit.client.0.smithi136.stdout: "lost": 0, 2024-09-15T22:05:02.773 INFO:tasks.workunit.client.0.smithi136.stdout: "flags": [ 2024-09-15T22:05:02.773 INFO:tasks.workunit.client.0.smithi136.stdout: "dirty", 2024-09-15T22:05:02.773 INFO:tasks.workunit.client.0.smithi136.stdout: "data_digest" 2024-09-15T22:05:02.773 INFO:tasks.workunit.client.0.smithi136.stdout: ], 2024-09-15T22:05:02.773 INFO:tasks.workunit.client.0.smithi136.stdout: "truncate_seq": 0, 2024-09-15T22:05:02.773 INFO:tasks.workunit.client.0.smithi136.stdout: "truncate_size": 0, 2024-09-15T22:05:02.773 INFO:tasks.workunit.client.0.smithi136.stdout: "data_digest": "0x8d0e1329", 2024-09-15T22:05:02.773 INFO:tasks.workunit.client.0.smithi136.stdout: "omap_digest": "0xffffffff", 2024-09-15T22:05:02.773 INFO:tasks.workunit.client.0.smithi136.stdout: "expected_object_size": 0, 2024-09-15T22:05:02.773 INFO:tasks.workunit.client.0.smithi136.stdout: "expected_write_size": 0, 2024-09-15T22:05:02.773 INFO:tasks.workunit.client.0.smithi136.stdout: "alloc_hint_flags": 0, 2024-09-15T22:05:02.773 INFO:tasks.workunit.client.0.smithi136.stdout: "manifest": { 2024-09-15T22:05:02.773 INFO:tasks.workunit.client.0.smithi136.stdout: "type": 0 2024-09-15T22:05:02.774 INFO:tasks.workunit.client.0.smithi136.stdout: }, 2024-09-15T22:05:02.774 INFO:tasks.workunit.client.0.smithi136.stdout: "watchers": {} 2024-09-15T22:05:02.774 INFO:tasks.workunit.client.0.smithi136.stdout: }, 2024-09-15T22:05:02.774 INFO:tasks.workunit.client.0.smithi136.stdout: "shards": [ 2024-09-15T22:05:02.774 INFO:tasks.workunit.client.0.smithi136.stdout: { 2024-09-15T22:05:02.774 INFO:tasks.workunit.client.0.smithi136.stdout: "osd": 0, 2024-09-15T22:05:02.774 INFO:tasks.workunit.client.0.smithi136.stdout: "primary": false, 2024-09-15T22:05:02.775 INFO:tasks.workunit.client.0.smithi136.stdout: "errors": [], 2024-09-15T22:05:02.775 INFO:tasks.workunit.client.0.smithi136.stdout: "size": 256, 2024-09-15T22:05:02.775 INFO:tasks.workunit.client.0.smithi136.stdout: "snapset": { 2024-09-15T22:05:02.775 INFO:tasks.workunit.client.0.smithi136.stdout: "seq": 0, 2024-09-15T22:05:02.775 INFO:tasks.workunit.client.0.smithi136.stdout: "clones": [] 2024-09-15T22:05:02.775 INFO:tasks.workunit.client.0.smithi136.stdout: } 2024-09-15T22:05:02.775 INFO:tasks.workunit.client.0.smithi136.stdout: }, 2024-09-15T22:05:02.775 INFO:tasks.workunit.client.0.smithi136.stdout: { 2024-09-15T22:05:02.775 INFO:tasks.workunit.client.0.smithi136.stdout: "osd": 1, 2024-09-15T22:05:02.775 INFO:tasks.workunit.client.0.smithi136.stdout: "primary": true, 2024-09-15T22:05:02.775 INFO:tasks.workunit.client.0.smithi136.stdout: "errors": [], 2024-09-15T22:05:02.775 INFO:tasks.workunit.client.0.smithi136.stdout: "size": 256, 2024-09-15T22:05:02.775 INFO:tasks.workunit.client.0.smithi136.stdout: "snapset": { 2024-09-15T22:05:02.776 INFO:tasks.workunit.client.0.smithi136.stdout: "seq": 1, 2024-09-15T22:05:02.776 INFO:tasks.workunit.client.0.smithi136.stdout: "clones": [ 2024-09-15T22:05:02.776 INFO:tasks.workunit.client.0.smithi136.stdout: { 2024-09-15T22:05:02.776 INFO:tasks.workunit.client.0.smithi136.stdout: "snap": 1, 2024-09-15T22:05:02.776 INFO:tasks.workunit.client.0.smithi136.stdout: "size": 1032, 2024-09-15T22:05:02.776 INFO:tasks.workunit.client.0.smithi136.stdout: "overlap": "[]", 2024-09-15T22:05:02.776 INFO:tasks.workunit.client.0.smithi136.stdout: "snaps": [ 2024-09-15T22:05:02.776 INFO:tasks.workunit.client.0.smithi136.stdout: 1 2024-09-15T22:05:02.776 INFO:tasks.workunit.client.0.smithi136.stdout: ] 2024-09-15T22:05:02.776 INFO:tasks.workunit.client.0.smithi136.stdout: } 2024-09-15T22:05:02.776 INFO:tasks.workunit.client.0.smithi136.stdout: ] 2024-09-15T22:05:02.776 INFO:tasks.workunit.client.0.smithi136.stdout: } 2024-09-15T22:05:02.776 INFO:tasks.workunit.client.0.smithi136.stdout: } 2024-09-15T22:05:02.777 INFO:tasks.workunit.client.0.smithi136.stdout: ] 2024-09-15T22:05:02.777 INFO:tasks.workunit.client.0.smithi136.stdout: }, 2024-09-15T22:05:02.777 INFO:tasks.workunit.client.0.smithi136.stdout: { 2024-09-15T22:05:02.777 INFO:tasks.workunit.client.0.smithi136.stdout: "object": { 2024-09-15T22:05:02.777 INFO:tasks.workunit.client.0.smithi136.stdout: "name": "obj8", 2024-09-15T22:05:02.777 INFO:tasks.workunit.client.0.smithi136.stdout: "nspace": "", 2024-09-15T22:05:02.777 INFO:tasks.workunit.client.0.smithi136.stdout: "locator": "", 2024-09-15T22:05:02.777 INFO:tasks.workunit.client.0.smithi136.stdout: "snap": "head", 2024-09-15T22:05:02.777 INFO:tasks.workunit.client.0.smithi136.stdout: "version": 28 2024-09-15T22:05:02.777 INFO:tasks.workunit.client.0.smithi136.stdout: }, 2024-09-15T22:05:02.777 INFO:tasks.workunit.client.0.smithi136.stdout: "errors": [ 2024-09-15T22:05:02.777 INFO:tasks.workunit.client.0.smithi136.stdout: "snapset_inconsistency" 2024-09-15T22:05:02.778 INFO:tasks.workunit.client.0.smithi136.stdout: ], 2024-09-15T22:05:02.778 INFO:tasks.workunit.client.0.smithi136.stdout: "union_shard_errors": [], 2024-09-15T22:05:02.778 INFO:tasks.workunit.client.0.smithi136.stdout: "selected_object_info": { 2024-09-15T22:05:02.778 INFO:tasks.workunit.client.0.smithi136.stdout: "oid": { 2024-09-15T22:05:02.778 INFO:tasks.workunit.client.0.smithi136.stdout: "oid": "obj8", 2024-09-15T22:05:02.778 INFO:tasks.workunit.client.0.smithi136.stdout: "key": "", 2024-09-15T22:05:02.778 INFO:tasks.workunit.client.0.smithi136.stdout: "snapid": -2, 2024-09-15T22:05:02.778 INFO:tasks.workunit.client.0.smithi136.stdout: "hash": 2381834917, 2024-09-15T22:05:02.778 INFO:tasks.workunit.client.0.smithi136.stdout: "max": 0, 2024-09-15T22:05:02.778 INFO:tasks.workunit.client.0.smithi136.stdout: "pool": 1, 2024-09-15T22:05:02.778 INFO:tasks.workunit.client.0.smithi136.stdout: "namespace": "" 2024-09-15T22:05:02.778 INFO:tasks.workunit.client.0.smithi136.stdout: }, 2024-09-15T22:05:02.778 INFO:tasks.workunit.client.0.smithi136.stdout: "version": "17'28", 2024-09-15T22:05:02.778 INFO:tasks.workunit.client.0.smithi136.stdout: "prior_version": "16'8", 2024-09-15T22:05:02.779 INFO:tasks.workunit.client.0.smithi136.stdout: "last_reqid": "client.4303.0:1", 2024-09-15T22:05:02.779 INFO:tasks.workunit.client.0.smithi136.stdout: "user_version": 28, 2024-09-15T22:05:02.779 INFO:tasks.workunit.client.0.smithi136.stdout: "size": 256, 2024-09-15T22:05:02.779 INFO:tasks.workunit.client.0.smithi136.stdout: "mtime": "2024-09-15T22:03:50.758587+0000", 2024-09-15T22:05:02.779 INFO:tasks.workunit.client.0.smithi136.stdout: "local_mtime": "2024-09-15T22:03:50.759163+0000", 2024-09-15T22:05:02.779 INFO:tasks.workunit.client.0.smithi136.stdout: "lost": 0, 2024-09-15T22:05:02.779 INFO:tasks.workunit.client.0.smithi136.stdout: "flags": [ 2024-09-15T22:05:02.779 INFO:tasks.workunit.client.0.smithi136.stdout: "dirty", 2024-09-15T22:05:02.779 INFO:tasks.workunit.client.0.smithi136.stdout: "data_digest" 2024-09-15T22:05:02.779 INFO:tasks.workunit.client.0.smithi136.stdout: ], 2024-09-15T22:05:02.779 INFO:tasks.workunit.client.0.smithi136.stdout: "truncate_seq": 0, 2024-09-15T22:05:02.779 INFO:tasks.workunit.client.0.smithi136.stdout: "truncate_size": 0, 2024-09-15T22:05:02.779 INFO:tasks.workunit.client.0.smithi136.stdout: "data_digest": "0x8d0e1329", 2024-09-15T22:05:02.779 INFO:tasks.workunit.client.0.smithi136.stdout: "omap_digest": "0xffffffff", 2024-09-15T22:05:02.780 INFO:tasks.workunit.client.0.smithi136.stdout: "expected_object_size": 0, 2024-09-15T22:05:02.780 INFO:tasks.workunit.client.0.smithi136.stdout: "expected_write_size": 0, 2024-09-15T22:05:02.780 INFO:tasks.workunit.client.0.smithi136.stdout: "alloc_hint_flags": 0, 2024-09-15T22:05:02.780 INFO:tasks.workunit.client.0.smithi136.stdout: "manifest": { 2024-09-15T22:05:02.780 INFO:tasks.workunit.client.0.smithi136.stdout: "type": 0 2024-09-15T22:05:02.780 INFO:tasks.workunit.client.0.smithi136.stdout: }, 2024-09-15T22:05:02.780 INFO:tasks.workunit.client.0.smithi136.stdout: "watchers": {} 2024-09-15T22:05:02.780 INFO:tasks.workunit.client.0.smithi136.stdout: }, 2024-09-15T22:05:02.780 INFO:tasks.workunit.client.0.smithi136.stdout: "shards": [ 2024-09-15T22:05:02.780 INFO:tasks.workunit.client.0.smithi136.stdout: { 2024-09-15T22:05:02.780 INFO:tasks.workunit.client.0.smithi136.stdout: "osd": 0, 2024-09-15T22:05:02.780 INFO:tasks.workunit.client.0.smithi136.stdout: "primary": false, 2024-09-15T22:05:02.780 INFO:tasks.workunit.client.0.smithi136.stdout: "errors": [], 2024-09-15T22:05:02.781 INFO:tasks.workunit.client.0.smithi136.stdout: "size": 256, 2024-09-15T22:05:02.781 INFO:tasks.workunit.client.0.smithi136.stdout: "snapset": { 2024-09-15T22:05:02.781 INFO:tasks.workunit.client.0.smithi136.stdout: "seq": 0, 2024-09-15T22:05:02.781 INFO:tasks.workunit.client.0.smithi136.stdout: "clones": [ 2024-09-15T22:05:02.781 INFO:tasks.workunit.client.0.smithi136.stdout: { 2024-09-15T22:05:02.781 INFO:tasks.workunit.client.0.smithi136.stdout: "snap": 1, 2024-09-15T22:05:02.781 INFO:tasks.workunit.client.0.smithi136.stdout: "size": 1032, 2024-09-15T22:05:02.781 INFO:tasks.workunit.client.0.smithi136.stdout: "overlap": "[]", 2024-09-15T22:05:02.781 INFO:tasks.workunit.client.0.smithi136.stdout: "snaps": [ 2024-09-15T22:05:02.781 INFO:tasks.workunit.client.0.smithi136.stdout: 1 2024-09-15T22:05:02.781 INFO:tasks.workunit.client.0.smithi136.stdout: ] 2024-09-15T22:05:02.781 INFO:tasks.workunit.client.0.smithi136.stdout: } 2024-09-15T22:05:02.781 INFO:tasks.workunit.client.0.smithi136.stdout: ] 2024-09-15T22:05:02.781 INFO:tasks.workunit.client.0.smithi136.stdout: } 2024-09-15T22:05:02.781 INFO:tasks.workunit.client.0.smithi136.stdout: }, 2024-09-15T22:05:02.782 INFO:tasks.workunit.client.0.smithi136.stdout: { 2024-09-15T22:05:02.782 INFO:tasks.workunit.client.0.smithi136.stdout: "osd": 1, 2024-09-15T22:05:02.782 INFO:tasks.workunit.client.0.smithi136.stdout: "primary": true, 2024-09-15T22:05:02.782 INFO:tasks.workunit.client.0.smithi136.stdout: "errors": [], 2024-09-15T22:05:02.782 INFO:tasks.workunit.client.0.smithi136.stdout: "size": 256, 2024-09-15T22:05:02.782 INFO:tasks.workunit.client.0.smithi136.stdout: "snapset": { 2024-09-15T22:05:02.782 INFO:tasks.workunit.client.0.smithi136.stdout: "seq": 1, 2024-09-15T22:05:02.782 INFO:tasks.workunit.client.0.smithi136.stdout: "clones": [ 2024-09-15T22:05:02.782 INFO:tasks.workunit.client.0.smithi136.stdout: { 2024-09-15T22:05:02.782 INFO:tasks.workunit.client.0.smithi136.stdout: "snap": 1, 2024-09-15T22:05:02.782 INFO:tasks.workunit.client.0.smithi136.stdout: "size": 1032, 2024-09-15T22:05:02.782 INFO:tasks.workunit.client.0.smithi136.stdout: "overlap": "[]", 2024-09-15T22:05:02.782 INFO:tasks.workunit.client.0.smithi136.stdout: "snaps": [ 2024-09-15T22:05:02.783 INFO:tasks.workunit.client.0.smithi136.stdout: 1 2024-09-15T22:05:02.783 INFO:tasks.workunit.client.0.smithi136.stdout: ] 2024-09-15T22:05:02.783 INFO:tasks.workunit.client.0.smithi136.stdout: } 2024-09-15T22:05:02.783 INFO:tasks.workunit.client.0.smithi136.stdout: ] 2024-09-15T22:05:02.783 INFO:tasks.workunit.client.0.smithi136.stdout: } 2024-09-15T22:05:02.783 INFO:tasks.workunit.client.0.smithi136.stdout: } 2024-09-15T22:05:02.783 INFO:tasks.workunit.client.0.smithi136.stdout: ] 2024-09-15T22:05:02.783 INFO:tasks.workunit.client.0.smithi136.stdout: }, 2024-09-15T22:05:02.783 INFO:tasks.workunit.client.0.smithi136.stdout: { 2024-09-15T22:05:02.783 INFO:tasks.workunit.client.0.smithi136.stdout: "object": { 2024-09-15T22:05:02.783 INFO:tasks.workunit.client.0.smithi136.stdout: "name": "obj9", 2024-09-15T22:05:02.783 INFO:tasks.workunit.client.0.smithi136.stdout: "nspace": "", 2024-09-15T22:05:02.783 INFO:tasks.workunit.client.0.smithi136.stdout: "locator": "", 2024-09-15T22:05:02.783 INFO:tasks.workunit.client.0.smithi136.stdout: "snap": "head", 2024-09-15T22:05:02.783 INFO:tasks.workunit.client.0.smithi136.stdout: "version": 30 2024-09-15T22:05:02.784 INFO:tasks.workunit.client.0.smithi136.stdout: }, 2024-09-15T22:05:02.784 INFO:tasks.workunit.client.0.smithi136.stdout: "errors": [ 2024-09-15T22:05:02.784 INFO:tasks.workunit.client.0.smithi136.stdout: "snapset_inconsistency" 2024-09-15T22:05:02.784 INFO:tasks.workunit.client.0.smithi136.stdout: ], 2024-09-15T22:05:02.784 INFO:tasks.workunit.client.0.smithi136.stdout: "union_shard_errors": [], 2024-09-15T22:05:02.784 INFO:tasks.workunit.client.0.smithi136.stdout: "selected_object_info": { 2024-09-15T22:05:02.784 INFO:tasks.workunit.client.0.smithi136.stdout: "oid": { 2024-09-15T22:05:02.784 INFO:tasks.workunit.client.0.smithi136.stdout: "oid": "obj9", 2024-09-15T22:05:02.784 INFO:tasks.workunit.client.0.smithi136.stdout: "key": "", 2024-09-15T22:05:02.784 INFO:tasks.workunit.client.0.smithi136.stdout: "snapid": -2, 2024-09-15T22:05:02.784 INFO:tasks.workunit.client.0.smithi136.stdout: "hash": 3833113727, 2024-09-15T22:05:02.784 INFO:tasks.workunit.client.0.smithi136.stdout: "max": 0, 2024-09-15T22:05:02.784 INFO:tasks.workunit.client.0.smithi136.stdout: "pool": 1, 2024-09-15T22:05:02.784 INFO:tasks.workunit.client.0.smithi136.stdout: "namespace": "" 2024-09-15T22:05:02.785 INFO:tasks.workunit.client.0.smithi136.stdout: }, 2024-09-15T22:05:02.785 INFO:tasks.workunit.client.0.smithi136.stdout: "version": "17'30", 2024-09-15T22:05:02.785 INFO:tasks.workunit.client.0.smithi136.stdout: "prior_version": "16'9", 2024-09-15T22:05:02.785 INFO:tasks.workunit.client.0.smithi136.stdout: "last_reqid": "client.4306.0:1", 2024-09-15T22:05:02.785 INFO:tasks.workunit.client.0.smithi136.stdout: "user_version": 30, 2024-09-15T22:05:02.785 INFO:tasks.workunit.client.0.smithi136.stdout: "size": 256, 2024-09-15T22:05:02.785 INFO:tasks.workunit.client.0.smithi136.stdout: "mtime": "2024-09-15T22:03:50.900657+0000", 2024-09-15T22:05:02.785 INFO:tasks.workunit.client.0.smithi136.stdout: "local_mtime": "2024-09-15T22:03:50.901282+0000", 2024-09-15T22:05:02.785 INFO:tasks.workunit.client.0.smithi136.stdout: "lost": 0, 2024-09-15T22:05:02.785 INFO:tasks.workunit.client.0.smithi136.stdout: "flags": [ 2024-09-15T22:05:02.785 INFO:tasks.workunit.client.0.smithi136.stdout: "dirty", 2024-09-15T22:05:02.785 INFO:tasks.workunit.client.0.smithi136.stdout: "data_digest" 2024-09-15T22:05:02.785 INFO:tasks.workunit.client.0.smithi136.stdout: ], 2024-09-15T22:05:02.785 INFO:tasks.workunit.client.0.smithi136.stdout: "truncate_seq": 0, 2024-09-15T22:05:02.786 INFO:tasks.workunit.client.0.smithi136.stdout: "truncate_size": 0, 2024-09-15T22:05:02.786 INFO:tasks.workunit.client.0.smithi136.stdout: "data_digest": "0x8d0e1329", 2024-09-15T22:05:02.786 INFO:tasks.workunit.client.0.smithi136.stdout: "omap_digest": "0xffffffff", 2024-09-15T22:05:02.786 INFO:tasks.workunit.client.0.smithi136.stdout: "expected_object_size": 0, 2024-09-15T22:05:02.786 INFO:tasks.workunit.client.0.smithi136.stdout: "expected_write_size": 0, 2024-09-15T22:05:02.786 INFO:tasks.workunit.client.0.smithi136.stdout: "alloc_hint_flags": 0, 2024-09-15T22:05:02.786 INFO:tasks.workunit.client.0.smithi136.stdout: "manifest": { 2024-09-15T22:05:02.786 INFO:tasks.workunit.client.0.smithi136.stdout: "type": 0 2024-09-15T22:05:02.786 INFO:tasks.workunit.client.0.smithi136.stdout: }, 2024-09-15T22:05:02.786 INFO:tasks.workunit.client.0.smithi136.stdout: "watchers": {} 2024-09-15T22:05:02.786 INFO:tasks.workunit.client.0.smithi136.stdout: }, 2024-09-15T22:05:02.786 INFO:tasks.workunit.client.0.smithi136.stdout: "shards": [ 2024-09-15T22:05:02.786 INFO:tasks.workunit.client.0.smithi136.stdout: { 2024-09-15T22:05:02.786 INFO:tasks.workunit.client.0.smithi136.stdout: "osd": 0, 2024-09-15T22:05:02.786 INFO:tasks.workunit.client.0.smithi136.stdout: "primary": false, 2024-09-15T22:05:02.787 INFO:tasks.workunit.client.0.smithi136.stdout: "errors": [], 2024-09-15T22:05:02.787 INFO:tasks.workunit.client.0.smithi136.stdout: "size": 256, 2024-09-15T22:05:02.787 INFO:tasks.workunit.client.0.smithi136.stdout: "snapset": { 2024-09-15T22:05:02.787 INFO:tasks.workunit.client.0.smithi136.stdout: "seq": 1, 2024-09-15T22:05:02.787 INFO:tasks.workunit.client.0.smithi136.stdout: "clones": [ 2024-09-15T22:05:02.787 INFO:tasks.workunit.client.0.smithi136.stdout: { 2024-09-15T22:05:02.787 INFO:tasks.workunit.client.0.smithi136.stdout: "snap": 1, 2024-09-15T22:05:02.788 INFO:tasks.workunit.client.0.smithi136.stdout: "size": "????", 2024-09-15T22:05:02.788 INFO:tasks.workunit.client.0.smithi136.stdout: "overlap": "[]", 2024-09-15T22:05:02.788 INFO:tasks.workunit.client.0.smithi136.stdout: "snaps": [ 2024-09-15T22:05:02.788 INFO:tasks.workunit.client.0.smithi136.stdout: 1 2024-09-15T22:05:02.788 INFO:tasks.workunit.client.0.smithi136.stdout: ] 2024-09-15T22:05:02.788 INFO:tasks.workunit.client.0.smithi136.stdout: } 2024-09-15T22:05:02.788 INFO:tasks.workunit.client.0.smithi136.stdout: ] 2024-09-15T22:05:02.788 INFO:tasks.workunit.client.0.smithi136.stdout: } 2024-09-15T22:05:02.788 INFO:tasks.workunit.client.0.smithi136.stdout: }, 2024-09-15T22:05:02.788 INFO:tasks.workunit.client.0.smithi136.stdout: { 2024-09-15T22:05:02.788 INFO:tasks.workunit.client.0.smithi136.stdout: "osd": 1, 2024-09-15T22:05:02.788 INFO:tasks.workunit.client.0.smithi136.stdout: "primary": true, 2024-09-15T22:05:02.788 INFO:tasks.workunit.client.0.smithi136.stdout: "errors": [], 2024-09-15T22:05:02.788 INFO:tasks.workunit.client.0.smithi136.stdout: "size": 256, 2024-09-15T22:05:02.789 INFO:tasks.workunit.client.0.smithi136.stdout: "snapset": { 2024-09-15T22:05:02.789 INFO:tasks.workunit.client.0.smithi136.stdout: "seq": 1, 2024-09-15T22:05:02.789 INFO:tasks.workunit.client.0.smithi136.stdout: "clones": [ 2024-09-15T22:05:02.789 INFO:tasks.workunit.client.0.smithi136.stdout: { 2024-09-15T22:05:02.789 INFO:tasks.workunit.client.0.smithi136.stdout: "snap": 1, 2024-09-15T22:05:02.789 INFO:tasks.workunit.client.0.smithi136.stdout: "size": 1032, 2024-09-15T22:05:02.789 INFO:tasks.workunit.client.0.smithi136.stdout: "overlap": "[]", 2024-09-15T22:05:02.789 INFO:tasks.workunit.client.0.smithi136.stdout: "snaps": [ 2024-09-15T22:05:02.789 INFO:tasks.workunit.client.0.smithi136.stdout: 1 2024-09-15T22:05:02.789 INFO:tasks.workunit.client.0.smithi136.stdout: ] 2024-09-15T22:05:02.789 INFO:tasks.workunit.client.0.smithi136.stdout: } 2024-09-15T22:05:02.789 INFO:tasks.workunit.client.0.smithi136.stdout: ] 2024-09-15T22:05:02.789 INFO:tasks.workunit.client.0.smithi136.stdout: } 2024-09-15T22:05:02.789 INFO:tasks.workunit.client.0.smithi136.stdout: } 2024-09-15T22:05:02.790 INFO:tasks.workunit.client.0.smithi136.stdout: ] 2024-09-15T22:05:02.790 INFO:tasks.workunit.client.0.smithi136.stdout: } 2024-09-15T22:05:02.790 INFO:tasks.workunit.client.0.smithi136.stdout: ] 2024-09-15T22:05:02.790 INFO:tasks.workunit.client.0.smithi136.stdout:} 2024-09-15T22:05:02.792 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:797: _scrub_snaps_multi: rados list-inconsistent-snapset 1.0 2024-09-15T22:05:02.792 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:800: _scrub_snaps_multi: '[' replica = replica ']' 2024-09-15T22:05:02.792 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:802: _scrub_snaps_multi: scruberrors=20 2024-09-15T22:05:02.792 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:803: _scrub_snaps_multi: jq .inconsistents 2024-09-15T22:05:02.792 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:803: _scrub_snaps_multi: python3 -c 'import json; import sys ; JSON=sys.stdin.read() ; ud = json.loads(JSON) ; print ( json.dumps(ud, sort_keys=True, indent=2) )' 2024-09-15T22:05:02.792 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1066: _scrub_snaps_multi: jq .inconsistents td/osd-scrub-snaps/json 2024-09-15T22:05:02.792 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1066: _scrub_snaps_multi: python3 -c 'import json; import sys ; JSON=sys.stdin.read() ; ud = json.loads(JSON) ; print ( json.dumps(ud, sort_keys=True, indent=2) )' 2024-09-15T22:05:02.792 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1067: _scrub_snaps_multi: multidiff td/osd-scrub-snaps/checkcsjson td/osd-scrub-snaps/csjson 2024-09-15T22:05:02.793 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2434: multidiff: diff td/osd-scrub-snaps/checkcsjson td/osd-scrub-snaps/csjson 2024-09-15T22:05:02.793 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1068: _scrub_snaps_multi: test no = yes 2024-09-15T22:05:02.793 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1073: _scrub_snaps_multi: test '' = yes 2024-09-15T22:05:02.793 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1078: _scrub_snaps_multi: find td/osd-scrub-snaps 2024-09-15T22:05:02.793 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1078: _scrub_snaps_multi: grep 'osd[^/]*\.pid' 2024-09-15T22:05:02.793 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1078: _scrub_snaps_multi: pidfiles='td/osd-scrub-snaps/osd.1.pid 2024-09-15T22:05:02.793 INFO:tasks.workunit.client.0.smithi136.stderr:td/osd-scrub-snaps/osd.0.pid' 2024-09-15T22:05:02.793 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1079: _scrub_snaps_multi: pids= 2024-09-15T22:05:02.793 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1080: _scrub_snaps_multi: for pidfile in ${pidfiles} 2024-09-15T22:05:02.793 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1082: _scrub_snaps_multi: cat td/osd-scrub-snaps/osd.1.pid 2024-09-15T22:05:02.793 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1082: _scrub_snaps_multi: pids+='518155 ' 2024-09-15T22:05:02.793 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1080: _scrub_snaps_multi: for pidfile in ${pidfiles} 2024-09-15T22:05:02.793 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1082: _scrub_snaps_multi: cat td/osd-scrub-snaps/osd.0.pid 2024-09-15T22:05:02.793 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1082: _scrub_snaps_multi: pids+='517558 ' 2024-09-15T22:05:02.794 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1085: _scrub_snaps_multi: ERRORS=0 2024-09-15T22:05:02.794 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1089: _scrub_snaps_multi: '[' replica = primary ']' 2024-09-15T22:05:02.794 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1111: _scrub_snaps_multi: ceph pg dump pgs 2024-09-15T22:05:03.056 INFO:tasks.workunit.client.0.smithi136.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-15T22:05:03.057 INFO:tasks.workunit.client.0.smithi136.stdout:1.0 36 0 0 0 0 24448 0 0 56 0 56 active+clean+inconsistent 2024-09-15T22:04:57.936088+0000 23'56 30:103 [1,0] 1 [1,0] 1 23'56 2024-09-15T22:04:57.936033+0000 0'0 2024-09-15T22:03:41.490650+0000 0 1 periodic scrub scheduled @ 2024-09-17T04:49:06.438375+0000 36 0 2024-09-15T22:05:03.057 INFO:tasks.workunit.client.0.smithi136.stdout: 2024-09-15T22:05:03.057 INFO:tasks.workunit.client.0.smithi136.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-15T22:05:03.057 INFO:tasks.workunit.client.0.smithi136.stderr:dumped pgs 2024-09-15T22:05:03.068 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1113: _scrub_snaps_multi: for pid in $pids 2024-09-15T22:05:03.069 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1115: _scrub_snaps_multi: kill -0 518155 2024-09-15T22:05:03.069 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1113: _scrub_snaps_multi: for pid in $pids 2024-09-15T22:05:03.069 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1115: _scrub_snaps_multi: kill -0 517558 2024-09-15T22:05:03.069 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1122: _scrub_snaps_multi: kill_daemons td/osd-scrub-snaps 2024-09-15T22:05:03.069 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-15T22:05:03.069 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-15T22:05:03.069 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-15T22:05:03.069 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-15T22:05:03.069 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-15T22:05:08.382 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-15T22:05:08.382 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1124: _scrub_snaps_multi: declare -a err_strings 2024-09-15T22:05:08.382 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1125: _scrub_snaps_multi: err_strings[0]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard [0-1] .*:::obj4:7 : missing' 2024-09-15T22:05:08.382 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1126: _scrub_snaps_multi: err_strings[1]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard [0-1] soid .*:::obj3:head : size 3840 != size 768 from auth oi' 2024-09-15T22:05:08.383 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1127: _scrub_snaps_multi: err_strings[2]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard [0-1] .*:::obj5:1 : missing' 2024-09-15T22:05:08.383 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1128: _scrub_snaps_multi: err_strings[3]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard [0-1] .*:::obj5:2 : missing' 2024-09-15T22:05:08.383 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1129: _scrub_snaps_multi: err_strings[4]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard [0-1] soid .*:::obj5:4 : size 4608 != size 512 from auth oi' 2024-09-15T22:05:08.383 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1130: _scrub_snaps_multi: err_strings[5]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid .*:::obj5:7 : failed to pick suitable object info' 2024-09-15T22:05:08.383 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1131: _scrub_snaps_multi: err_strings[6]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard [0-1] .*:::obj1:head : missing' 2024-09-15T22:05:08.383 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1132: _scrub_snaps_multi: err_strings[7]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 scrub 20 errors' 2024-09-15T22:05:08.383 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1134: _scrub_snaps_multi: for err_string in "${err_strings[@]}" 2024-09-15T22:05:08.383 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1136: _scrub_snaps_multi: grep 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard [0-1] .*:::obj4:7 : missing' td/osd-scrub-snaps/osd.1.log 2024-09-15T22:05:08.384 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1134: _scrub_snaps_multi: for err_string in "${err_strings[@]}" 2024-09-15T22:05:08.384 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1136: _scrub_snaps_multi: grep 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard [0-1] soid .*:::obj3:head : size 3840 != size 768 from auth oi' td/osd-scrub-snaps/osd.1.log 2024-09-15T22:05:08.387 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1134: _scrub_snaps_multi: for err_string in "${err_strings[@]}" 2024-09-15T22:05:08.387 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1136: _scrub_snaps_multi: grep 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard [0-1] .*:::obj5:1 : missing' td/osd-scrub-snaps/osd.1.log 2024-09-15T22:05:08.390 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1134: _scrub_snaps_multi: for err_string in "${err_strings[@]}" 2024-09-15T22:05:08.390 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1136: _scrub_snaps_multi: grep 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard [0-1] .*:::obj5:2 : missing' td/osd-scrub-snaps/osd.1.log 2024-09-15T22:05:08.393 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1134: _scrub_snaps_multi: for err_string in "${err_strings[@]}" 2024-09-15T22:05:08.393 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1136: _scrub_snaps_multi: grep 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard [0-1] soid .*:::obj5:4 : size 4608 != size 512 from auth oi' td/osd-scrub-snaps/osd.1.log 2024-09-15T22:05:08.395 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1134: _scrub_snaps_multi: for err_string in "${err_strings[@]}" 2024-09-15T22:05:08.395 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1136: _scrub_snaps_multi: grep 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid .*:::obj5:7 : failed to pick suitable object info' td/osd-scrub-snaps/osd.1.log 2024-09-15T22:05:08.398 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1134: _scrub_snaps_multi: for err_string in "${err_strings[@]}" 2024-09-15T22:05:08.398 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1136: _scrub_snaps_multi: grep 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard [0-1] .*:::obj1:head : missing' td/osd-scrub-snaps/osd.1.log 2024-09-15T22:05:08.401 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1134: _scrub_snaps_multi: for err_string in "${err_strings[@]}" 2024-09-15T22:05:08.401 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1136: _scrub_snaps_multi: grep 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 scrub 20 errors' td/osd-scrub-snaps/osd.1.log 2024-09-15T22:05:08.403 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1144: _scrub_snaps_multi: declare -a rep_err_strings 2024-09-15T22:05:08.405 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1145: _scrub_snaps_multi: eval echo '$replica' 2024-09-15T22:05:08.405 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1145: _scrub_snaps_multi: echo 0 2024-09-15T22:05:08.405 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1145: _scrub_snaps_multi: osd=0 2024-09-15T22:05:08.405 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1146: _scrub_snaps_multi: rep_err_strings[0]='log_channel[(]cluster[)] log [[]ERR[]] : osd[.][0-9]* found snap mapper error on pg 1.0 oid 1:461f8b5e:::obj16:7 snaps missing in mapper, should be: {1, 2, 3, 4, 5, 6, 7} ...repaired' 2024-09-15T22:05:08.405 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1147: _scrub_snaps_multi: for err_string in "${rep_err_strings[@]}" 2024-09-15T22:05:08.405 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1149: _scrub_snaps_multi: grep 'log_channel[(]cluster[)] log [[]ERR[]] : osd[.][0-9]* found snap mapper error on pg 1.0 oid 1:461f8b5e:::obj16:7 snaps missing in mapper, should be: {1, 2, 3, 4, 5, 6, 7} ...repaired' td/osd-scrub-snaps/osd.0.log 2024-09-15T22:05:08.407 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1156: _scrub_snaps_multi: '[' 0 '!=' 0 ']' 2024-09-15T22:05:08.407 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1162: _scrub_snaps_multi: echo 'TEST PASSED' 2024-09-15T22:05:08.407 INFO:tasks.workunit.client.0.smithi136.stdout:TEST PASSED 2024-09-15T22:05:08.408 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1163: _scrub_snaps_multi: return 0 2024-09-15T22:05:08.408 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1171: TEST_scrub_snaps_replica: err=0 2024-09-15T22:05:08.408 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1172: TEST_scrub_snaps_replica: CEPH_ARGS='--fsid=0098e432-2807-49bf-8c16-1e2a347d825d --auth-supported=none --mon-host=127.0.0.1:7121 ' 2024-09-15T22:05:08.408 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1173: TEST_scrub_snaps_replica: return 0 2024-09-15T22:05:08.409 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:40: run: teardown td/osd-scrub-snaps 2024-09-15T22:05:08.409 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/osd-scrub-snaps 2024-09-15T22:05:08.409 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-15T22:05:08.409 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/osd-scrub-snaps KILL 2024-09-15T22:05:08.409 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-15T22:05:08.409 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-15T22:05:08.409 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-15T22:05:08.409 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-15T22:05:08.409 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-15T22:05:08.413 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-15T22:05:08.413 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-15T22:05:08.414 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-15T22:05:08.415 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-15T22:05:08.416 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-15T22:05:08.416 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-15T22:05:08.416 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-15T22:05:08.417 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T22:05:08.417 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-15T22:05:08.417 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-15T22:05:08.418 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T22:05:08.418 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-15T22:05:08.419 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-15T22:05:08.420 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/osd-scrub-snaps 2024-09-15T22:05:08.435 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-15T22:05:08.435 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T22:05:08.435 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.463062 2024-09-15T22:05:08.435 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.463062 2024-09-15T22:05:08.437 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-15T22:05:08.437 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-15T22:05:08.437 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2322: main: code=0 2024-09-15T22:05:08.437 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2326: main: teardown td/osd-scrub-snaps 0 2024-09-15T22:05:08.437 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/osd-scrub-snaps 2024-09-15T22:05:08.437 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs=0 2024-09-15T22:05:08.437 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/osd-scrub-snaps KILL 2024-09-15T22:05:08.437 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-15T22:05:08.437 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-15T22:05:08.437 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-15T22:05:08.437 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-15T22:05:08.437 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-15T22:05:08.439 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-15T22:05:08.439 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-15T22:05:08.440 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-15T22:05:08.440 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-15T22:05:08.441 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-15T22:05:08.441 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-15T22:05:08.441 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-15T22:05:08.442 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T22:05:08.442 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-15T22:05:08.443 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-15T22:05:08.443 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T22:05:08.444 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-15T22:05:08.445 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o 0 = 1 ']' 2024-09-15T22:05:08.445 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/osd-scrub-snaps 2024-09-15T22:05:08.446 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-15T22:05:08.446 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T22:05:08.446 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.463062 2024-09-15T22:05:08.446 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.463062 2024-09-15T22:05:08.448 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-15T22:05:08.448 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-15T22:05:08.448 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2327: main: return 0 2024-09-15T22:05:08.448 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-09-15T22:05:08.449 DEBUG:teuthology.orchestra.run.smithi136:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-09-15T22:05:08.475 INFO:tasks.workunit:Running workunit scrub/osd-scrub-test.sh... 2024-09-15T22:05:08.475 DEBUG:teuthology.orchestra.run.smithi136:workunit test scrub/osd-scrub-test.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=7498493279982dfe87f00616198a5967475f6169 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh 2024-09-15T22:05:08.539 INFO:tasks.workunit.client.0.smithi136.stderr:stty: 'standard input': Inappropriate ioctl for device 2024-09-15T22:05:08.545 INFO:tasks.workunit.client.0.smithi136.stderr:+ PS4='${BASH_SOURCE[0]}:$LINENO: ${FUNCNAME[0]}: ' 2024-09-15T22:05:08.545 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2315: main: export PATH=.:/home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin 2024-09-15T22:05:08.545 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2315: main: PATH=.:/home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin 2024-09-15T22:05:08.546 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2316: main: export PYTHONWARNINGS=ignore 2024-09-15T22:05:08.546 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2316: main: PYTHONWARNINGS=ignore 2024-09-15T22:05:08.546 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2317: main: export CEPH_CONF=/dev/null 2024-09-15T22:05:08.546 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2317: main: CEPH_CONF=/dev/null 2024-09-15T22:05:08.546 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2318: main: unset CEPH_ARGS 2024-09-15T22:05:08.546 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2320: main: local code 2024-09-15T22:05:08.546 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2321: main: run td/osd-scrub-test 2024-09-15T22:05:08.546 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:21: run: local dir=td/osd-scrub-test 2024-09-15T22:05:08.546 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:22: run: shift 2024-09-15T22:05:08.546 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:24: run: export CEPH_MON=127.0.0.1:7138 2024-09-15T22:05:08.546 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:24: run: CEPH_MON=127.0.0.1:7138 2024-09-15T22:05:08.546 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:25: run: export CEPH_ARGS 2024-09-15T22:05:08.546 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:26: run: uuidgen 2024-09-15T22:05:08.547 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:26: run: CEPH_ARGS+='--fsid=79d14839-1c9a-4907-b0ca-211bfb85130f --auth-supported=none ' 2024-09-15T22:05:08.547 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:27: run: CEPH_ARGS+='--mon-host=127.0.0.1:7138 ' 2024-09-15T22:05:08.547 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:29: run: export -n CEPH_CLI_TEST_DUP_COMMAND 2024-09-15T22:05:08.547 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:30: run: set 2024-09-15T22:05:08.547 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:30: run: sed -n -e 's/^\(TEST_[0-9a-z_]*\) .*/\1/p' 2024-09-15T22:05:08.548 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:30: run: local 'funcs=TEST_deep_scrub_abort 2024-09-15T22:05:08.548 INFO:tasks.workunit.client.0.smithi136.stderr:TEST_dump_scrub_schedule 2024-09-15T22:05:08.548 INFO:tasks.workunit.client.0.smithi136.stderr:TEST_interval_changes 2024-09-15T22:05:08.549 INFO:tasks.workunit.client.0.smithi136.stderr:TEST_just_deep_scrubs 2024-09-15T22:05:08.549 INFO:tasks.workunit.client.0.smithi136.stderr:TEST_pg_dump_objects_scrubbed 2024-09-15T22:05:08.549 INFO:tasks.workunit.client.0.smithi136.stderr:TEST_scrub_abort 2024-09-15T22:05:08.549 INFO:tasks.workunit.client.0.smithi136.stderr:TEST_scrub_permit_time 2024-09-15T22:05:08.549 INFO:tasks.workunit.client.0.smithi136.stderr:TEST_scrub_test' 2024-09-15T22:05:08.549 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:31: run: for func in $funcs 2024-09-15T22:05:08.549 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:32: run: echo '-------------- Prepare Test TEST_deep_scrub_abort -------------------' 2024-09-15T22:05:08.549 INFO:tasks.workunit.client.0.smithi136.stdout:-------------- Prepare Test TEST_deep_scrub_abort ------------------- 2024-09-15T22:05:08.550 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:33: run: setup td/osd-scrub-test 2024-09-15T22:05:08.551 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/osd-scrub-test 2024-09-15T22:05:08.551 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/osd-scrub-test 2024-09-15T22:05:08.551 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/osd-scrub-test 2024-09-15T22:05:08.551 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-15T22:05:08.551 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/osd-scrub-test KILL 2024-09-15T22:05:08.551 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-15T22:05:08.551 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-15T22:05:08.551 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-15T22:05:08.551 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-15T22:05:08.551 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-15T22:05:08.551 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-15T22:05:08.551 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-15T22:05:08.552 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-15T22:05:08.553 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-15T22:05:08.553 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-15T22:05:08.553 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-15T22:05:08.553 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-15T22:05:08.553 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T22:05:08.554 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-15T22:05:08.554 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-15T22:05:08.554 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T22:05:08.554 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-15T22:05:08.555 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-15T22:05:08.556 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/osd-scrub-test 2024-09-15T22:05:08.556 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-15T22:05:08.557 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T22:05:08.557 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.519518 2024-09-15T22:05:08.557 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.519518 2024-09-15T22:05:08.557 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-15T22:05:08.557 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-15T22:05:08.557 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/osd-scrub-test 2024-09-15T22:05:08.559 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2024-09-15T22:05:08.559 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T22:05:08.559 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.519518 2024-09-15T22:05:08.559 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.519518 2024-09-15T22:05:08.560 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2024-09-15T22:05:08.560 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 1024 -le 1024 ']' 2024-09-15T22:05:08.560 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:127: setup: ulimit -n 4096 2024-09-15T22:05:08.561 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2024-09-15T22:05:08.561 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/osd-scrub-test 1' TERM HUP INT 2024-09-15T22:05:08.561 INFO:tasks.workunit.client.0.smithi136.stdout:-------------- Run Test TEST_deep_scrub_abort ----------------------- 2024-09-15T22:05:08.563 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:34: run: echo '-------------- Run Test TEST_deep_scrub_abort -----------------------' 2024-09-15T22:05:08.563 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:35: run: TEST_deep_scrub_abort td/osd-scrub-test 2024-09-15T22:05:08.563 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:416: TEST_deep_scrub_abort: local dir=td/osd-scrub-test 2024-09-15T22:05:08.563 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:417: TEST_deep_scrub_abort: _scrub_abort td/osd-scrub-test deep-scrub 2024-09-15T22:05:08.563 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:300: _scrub_abort: local dir=td/osd-scrub-test 2024-09-15T22:05:08.563 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:301: _scrub_abort: local poolname=test 2024-09-15T22:05:08.563 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:302: _scrub_abort: local OSDS=3 2024-09-15T22:05:08.563 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:303: _scrub_abort: local objects=1000 2024-09-15T22:05:08.563 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:304: _scrub_abort: local type=deep-scrub 2024-09-15T22:05:08.563 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:306: _scrub_abort: TESTDATA=testdata.519518 2024-09-15T22:05:08.563 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:307: _scrub_abort: test deep-scrub = scrub 2024-09-15T22:05:08.563 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:312: _scrub_abort: stopscrub=nodeep-scrub 2024-09-15T22:05:08.564 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: check=nodeep_scrub 2024-09-15T22:05:08.564 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:316: _scrub_abort: run_mon td/osd-scrub-test a --osd_pool_default_size=3 2024-09-15T22:05:08.564 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/osd-scrub-test 2024-09-15T22:05:08.564 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2024-09-15T22:05:08.564 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2024-09-15T22:05:08.564 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2024-09-15T22:05:08.564 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/osd-scrub-test/a 2024-09-15T22:05:08.564 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-scrub-test/a --run-dir=td/osd-scrub-test --osd_pool_default_size=3 2024-09-15T22:05:09.007 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2024-09-15T22:05:09.007 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-15T22:05:09.007 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-15T22:05:09.007 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-15T22:05:09.007 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T22:05:09.007 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.519518 2024-09-15T22:05:09.007 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.519518/$cluster-$name.asok' 2024-09-15T22:05:09.007 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-scrub-test/a '--log-file=td/osd-scrub-test/$name.log' '--admin-socket=/tmp/ceph-asok.519518/$cluster-$name.asok' --mon-cluster-log-file=td/osd-scrub-test/log --run-dir=td/osd-scrub-test '--pid-file=td/osd-scrub-test/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=3 2024-09-15T22:05:09.174 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2024-09-15T22:05:09.176 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2024-09-15T22:05:09.176 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-15T22:05:09.176 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-15T22:05:09.176 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2024-09-15T22:05:09.176 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2024-09-15T22:05:09.176 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-15T22:05:09.176 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-15T22:05:09.176 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-15T22:05:09.176 INFO:tasks.workunit.client.0.smithi136.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-15T22:05:09.176 INFO:tasks.workunit.client.0.smithi136.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T22:05:09.177 INFO:tasks.workunit.client.0.smithi136.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.519518 2024-09-15T22:05:09.177 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.519518/ceph-mon.a.asok 2024-09-15T22:05:09.177 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-15T22:05:09.177 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.519518/ceph-mon.a.asok config get fsid 2024-09-15T22:05:09.257 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2024-09-15T22:05:09.257 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-15T22:05:09.257 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-15T22:05:09.257 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2024-09-15T22:05:09.257 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2024-09-15T22:05:09.257 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-15T22:05:09.257 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-15T22:05:09.257 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-15T22:05:09.257 INFO:tasks.workunit.client.0.smithi136.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-15T22:05:09.257 INFO:tasks.workunit.client.0.smithi136.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T22:05:09.257 INFO:tasks.workunit.client.0.smithi136.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.519518 2024-09-15T22:05:09.257 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.519518/ceph-mon.a.asok 2024-09-15T22:05:09.258 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-15T22:05:09.258 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.519518/ceph-mon.a.asok config get mon_host 2024-09-15T22:05:09.335 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:317: _scrub_abort: run_mgr td/osd-scrub-test x 2024-09-15T22:05:09.336 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: run_mgr: local dir=td/osd-scrub-test 2024-09-15T22:05:09.336 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: run_mgr: shift 2024-09-15T22:05:09.336 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: run_mgr: local id=x 2024-09-15T22:05:09.336 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: run_mgr: shift 2024-09-15T22:05:09.336 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:548: run_mgr: local data=td/osd-scrub-test/x 2024-09-15T22:05:09.336 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:550: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-15T22:05:10.142 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: get_asok_path 2024-09-15T22:05:10.142 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-15T22:05:10.142 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-15T22:05:10.142 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-15T22:05:10.142 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T22:05:10.143 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.519518 2024-09-15T22:05:10.143 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.519518/$cluster-$name.asok' 2024-09-15T22:05:10.143 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-15T22:05:10.143 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-scrub-test/x '--log-file=td/osd-scrub-test/$name.log' '--admin-socket=/tmp/ceph-asok.519518/$cluster-$name.asok' --run-dir=td/osd-scrub-test '--pid-file=td/osd-scrub-test/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-15T22:05:10.169 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:318: _scrub_abort: expr 3 - 1 2024-09-15T22:05:10.170 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:318: _scrub_abort: seq 0 2 2024-09-15T22:05:10.174 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:318: _scrub_abort: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-09-15T22:05:10.174 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:324: _scrub_abort: run_osd td/osd-scrub-test 0 --osd_pool_default_pg_autoscale_mode=off --osd_deep_scrub_randomize_ratio=0.0 --osd_scrub_sleep=5.0 --osd_scrub_interval_randomize_ratio=0 --osd_op_queue=wpq 2024-09-15T22:05:10.175 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-scrub-test 2024-09-15T22:05:10.175 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-15T22:05:10.175 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=0 2024-09-15T22:05:10.175 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-15T22:05:10.175 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-scrub-test/0 2024-09-15T22:05:10.175 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=79d14839-1c9a-4907-b0ca-211bfb85130f --auth-supported=none --mon-host=127.0.0.1:7138 ' 2024-09-15T22:05:10.175 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T22:05:10.175 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T22:05:10.175 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T22:05:10.175 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-scrub-test/0' 2024-09-15T22:05:10.175 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-test/0/journal' 2024-09-15T22:05:10.175 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-15T22:05:10.175 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-15T22:05:10.175 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-scrub-test' 2024-09-15T22:05:10.176 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-15T22:05:10.176 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-15T22:05:10.176 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-15T22:05:10.176 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-15T22:05:10.176 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T22:05:10.176 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.519518 2024-09-15T22:05:10.176 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.519518/$cluster-$name.asok' 2024-09-15T22:05:10.176 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.519518/$cluster-$name.asok' 2024-09-15T22:05:10.176 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-15T22:05:10.176 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-15T22:05:10.176 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-15T22:05:10.176 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-scrub-test/$name.log' 2024-09-15T22:05:10.177 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-scrub-test/$name.pid' 2024-09-15T22:05:10.177 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T22:05:10.177 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T22:05:10.177 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T22:05:10.177 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T22:05:10.177 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-15T22:05:10.177 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+='--osd_pool_default_pg_autoscale_mode=off --osd_deep_scrub_randomize_ratio=0.0 --osd_scrub_sleep=5.0 --osd_scrub_interval_randomize_ratio=0 --osd_op_queue=wpq' 2024-09-15T22:05:10.177 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-scrub-test/0 2024-09-15T22:05:10.177 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-15T22:05:10.177 INFO:tasks.workunit.client.0.smithi136.stdout:add osd0 5475b161-78a4-4250-90c3-a518fb249d9f 2024-09-15T22:05:10.178 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=5475b161-78a4-4250-90c3-a518fb249d9f 2024-09-15T22:05:10.178 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd0 5475b161-78a4-4250-90c3-a518fb249d9f' 2024-09-15T22:05:10.178 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-15T22:05:10.192 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQAWWudmwFpYCxAAzIX1cid9Zc4BZU6sJkjojg== 2024-09-15T22:05:10.192 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQAWWudmwFpYCxAAzIX1cid9Zc4BZU6sJkjojg=="}' 2024-09-15T22:05:10.192 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 5475b161-78a4-4250-90c3-a518fb249d9f -i td/osd-scrub-test/0/new.json 2024-09-15T22:05:10.389 INFO:tasks.workunit.client.0.smithi136.stdout:0 2024-09-15T22:05:10.399 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-scrub-test/0/new.json 2024-09-15T22:05:10.400 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 0 --fsid=79d14839-1c9a-4907-b0ca-211bfb85130f --auth-supported=none --mon-host=127.0.0.1:7138 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-test/0 --osd-journal=td/osd-scrub-test/0/journal --chdir= --run-dir=td/osd-scrub-test '--admin-socket=/tmp/ceph-asok.519518/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-test/$name.log' '--pid-file=td/osd-scrub-test/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_pool_default_pg_autoscale_mode=off --osd_deep_scrub_randomize_ratio=0.0 --osd_scrub_sleep=5.0 --osd_scrub_interval_randomize_ratio=0 --osd_op_queue=wpq --mkfs --key AQAWWudmwFpYCxAAzIX1cid9Zc4BZU6sJkjojg== --osd-uuid 5475b161-78a4-4250-90c3-a518fb249d9f 2024-09-15T22:05:10.431 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T22:05:10.429+0000 7fd7a3a18640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T22:05:10.440 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T22:05:10.438+0000 7fd7a3a18640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T22:05:10.441 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T22:05:10.440+0000 7fd7a3a18640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T22:05:10.441 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T22:05:10.440+0000 7fd7a3a18640 -1 bdev(0x556fe9e45c00 td/osd-scrub-test/0/block) open stat got: (1) Operation not permitted 2024-09-15T22:05:10.441 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T22:05:10.440+0000 7fd7a3a18640 -1 bluestore(td/osd-scrub-test/0) _read_fsid unparsable uuid 2024-09-15T22:05:16.506 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-scrub-test/0/keyring 2024-09-15T22:05:16.506 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-15T22:05:16.507 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd0 key to auth repository 2024-09-15T22:05:16.507 INFO:tasks.workunit.client.0.smithi136.stdout:adding osd0 key to auth repository 2024-09-15T22:05:16.507 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-scrub-test/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-15T22:05:16.865 INFO:tasks.workunit.client.0.smithi136.stdout:start osd.0 2024-09-15T22:05:16.866 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.0 2024-09-15T22:05:16.866 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 0 --fsid=79d14839-1c9a-4907-b0ca-211bfb85130f --auth-supported=none --mon-host=127.0.0.1:7138 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-test/0 --osd-journal=td/osd-scrub-test/0/journal --chdir= --run-dir=td/osd-scrub-test '--admin-socket=/tmp/ceph-asok.519518/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-test/$name.log' '--pid-file=td/osd-scrub-test/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_pool_default_pg_autoscale_mode=off --osd_deep_scrub_randomize_ratio=0.0 --osd_scrub_sleep=5.0 --osd_scrub_interval_randomize_ratio=0 --osd_op_queue=wpq 2024-09-15T22:05:16.866 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-15T22:05:16.866 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-15T22:05:16.866 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-15T22:05:16.893 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T22:05:16.891+0000 7f162dce0640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T22:05:16.895 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T22:05:16.893+0000 7f162dce0640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T22:05:16.896 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T22:05:16.894+0000 7f162dce0640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T22:05:17.140 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 0 2024-09-15T22:05:17.140 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-15T22:05:17.140 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2024-09-15T22:05:17.140 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-15T22:05:17.141 INFO:tasks.workunit.client.0.smithi136.stdout:0 2024-09-15T22:05:17.141 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-15T22:05:17.141 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:05:17.141 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-15T22:05:17.141 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:05:17.141 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T22:05:17.415 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:05:18.244 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T22:05:18.242+0000 7f162dce0640 -1 Falling back to public interface 2024-09-15T22:05:18.417 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:05:18.417 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:05:18.417 INFO:tasks.workunit.client.0.smithi136.stdout:1 2024-09-15T22:05:18.417 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-15T22:05:18.418 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:05:18.418 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T22:05:18.702 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:05:19.703 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:05:19.704 INFO:tasks.workunit.client.0.smithi136.stdout:2 2024-09-15T22:05:19.704 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:05:19.704 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-15T22:05:19.704 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:05:19.704 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T22:05:19.982 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:05:20.881 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T22:05:20.879+0000 7f162dce0640 -1 osd.0 0 log_to_monitors true 2024-09-15T22:05:20.983 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:05:20.983 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:05:20.983 INFO:tasks.workunit.client.0.smithi136.stdout:3 2024-09-15T22:05:20.984 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-15T22:05:20.984 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:05:20.984 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T22:05:21.359 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:05:22.360 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:05:22.360 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:05:22.360 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-15T22:05:22.360 INFO:tasks.workunit.client.0.smithi136.stdout:4 2024-09-15T22:05:22.361 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:05:22.361 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T22:05:22.643 INFO:tasks.workunit.client.0.smithi136.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/2850141095,v1:127.0.0.1:6803/2850141095] [v2:127.0.0.1:6804/2850141095,v1:127.0.0.1:6805/2850141095] exists,up 5475b161-78a4-4250-90c3-a518fb249d9f 2024-09-15T22:05:22.647 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-15T22:05:22.647 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-15T22:05:22.647 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-15T22:05:22.647 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:318: _scrub_abort: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-09-15T22:05:22.647 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:324: _scrub_abort: run_osd td/osd-scrub-test 1 --osd_pool_default_pg_autoscale_mode=off --osd_deep_scrub_randomize_ratio=0.0 --osd_scrub_sleep=5.0 --osd_scrub_interval_randomize_ratio=0 --osd_op_queue=wpq 2024-09-15T22:05:22.647 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-scrub-test 2024-09-15T22:05:22.647 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-15T22:05:22.647 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=1 2024-09-15T22:05:22.648 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-15T22:05:22.648 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-scrub-test/1 2024-09-15T22:05:22.648 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=79d14839-1c9a-4907-b0ca-211bfb85130f --auth-supported=none --mon-host=127.0.0.1:7138 ' 2024-09-15T22:05:22.648 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T22:05:22.648 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T22:05:22.648 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T22:05:22.648 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-scrub-test/1' 2024-09-15T22:05:22.648 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-test/1/journal' 2024-09-15T22:05:22.648 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-15T22:05:22.648 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-15T22:05:22.648 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-scrub-test' 2024-09-15T22:05:22.648 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-15T22:05:22.649 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-15T22:05:22.649 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-15T22:05:22.649 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-15T22:05:22.649 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T22:05:22.649 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.519518 2024-09-15T22:05:22.649 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.519518/$cluster-$name.asok' 2024-09-15T22:05:22.649 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.519518/$cluster-$name.asok' 2024-09-15T22:05:22.649 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-15T22:05:22.649 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-15T22:05:22.649 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-15T22:05:22.649 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-scrub-test/$name.log' 2024-09-15T22:05:22.649 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-scrub-test/$name.pid' 2024-09-15T22:05:22.649 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T22:05:22.650 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T22:05:22.650 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T22:05:22.650 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T22:05:22.650 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-15T22:05:22.650 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+='--osd_pool_default_pg_autoscale_mode=off --osd_deep_scrub_randomize_ratio=0.0 --osd_scrub_sleep=5.0 --osd_scrub_interval_randomize_ratio=0 --osd_op_queue=wpq' 2024-09-15T22:05:22.650 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-scrub-test/1 2024-09-15T22:05:22.650 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-15T22:05:22.650 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=f496a610-436d-4f7b-9801-7464bdb5a9c7 2024-09-15T22:05:22.650 INFO:tasks.workunit.client.0.smithi136.stdout:add osd1 f496a610-436d-4f7b-9801-7464bdb5a9c7 2024-09-15T22:05:22.651 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd1 f496a610-436d-4f7b-9801-7464bdb5a9c7' 2024-09-15T22:05:22.651 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-15T22:05:22.663 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQAiWudmTUt4JxAAZ2rjLjHAugBBkQGdoOZE3g== 2024-09-15T22:05:22.663 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQAiWudmTUt4JxAAZ2rjLjHAugBBkQGdoOZE3g=="}' 2024-09-15T22:05:22.664 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new f496a610-436d-4f7b-9801-7464bdb5a9c7 -i td/osd-scrub-test/1/new.json 2024-09-15T22:05:22.982 INFO:tasks.workunit.client.0.smithi136.stdout:1 2024-09-15T22:05:22.994 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-scrub-test/1/new.json 2024-09-15T22:05:22.995 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 1 --fsid=79d14839-1c9a-4907-b0ca-211bfb85130f --auth-supported=none --mon-host=127.0.0.1:7138 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-test/1 --osd-journal=td/osd-scrub-test/1/journal --chdir= --run-dir=td/osd-scrub-test '--admin-socket=/tmp/ceph-asok.519518/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-test/$name.log' '--pid-file=td/osd-scrub-test/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_pool_default_pg_autoscale_mode=off --osd_deep_scrub_randomize_ratio=0.0 --osd_scrub_sleep=5.0 --osd_scrub_interval_randomize_ratio=0 --osd_op_queue=wpq --mkfs --key AQAiWudmTUt4JxAAZ2rjLjHAugBBkQGdoOZE3g== --osd-uuid f496a610-436d-4f7b-9801-7464bdb5a9c7 2024-09-15T22:05:23.025 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T22:05:23.023+0000 7efe193a6640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T22:05:23.058 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T22:05:23.056+0000 7efe193a6640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T22:05:23.059 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T22:05:23.057+0000 7efe193a6640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T22:05:23.059 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T22:05:23.058+0000 7efe193a6640 -1 bdev(0x55cdf578fc00 td/osd-scrub-test/1/block) open stat got: (1) Operation not permitted 2024-09-15T22:05:23.059 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T22:05:23.058+0000 7efe193a6640 -1 bluestore(td/osd-scrub-test/1) _read_fsid unparsable uuid 2024-09-15T22:05:29.385 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-scrub-test/1/keyring 2024-09-15T22:05:29.385 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-15T22:05:29.386 INFO:tasks.workunit.client.0.smithi136.stdout:adding osd1 key to auth repository 2024-09-15T22:05:29.387 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd1 key to auth repository 2024-09-15T22:05:29.387 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-scrub-test/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-15T22:05:29.751 INFO:tasks.workunit.client.0.smithi136.stdout:start osd.1 2024-09-15T22:05:29.752 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.1 2024-09-15T22:05:29.752 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 1 --fsid=79d14839-1c9a-4907-b0ca-211bfb85130f --auth-supported=none --mon-host=127.0.0.1:7138 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-test/1 --osd-journal=td/osd-scrub-test/1/journal --chdir= --run-dir=td/osd-scrub-test '--admin-socket=/tmp/ceph-asok.519518/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-test/$name.log' '--pid-file=td/osd-scrub-test/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_pool_default_pg_autoscale_mode=off --osd_deep_scrub_randomize_ratio=0.0 --osd_scrub_sleep=5.0 --osd_scrub_interval_randomize_ratio=0 --osd_op_queue=wpq 2024-09-15T22:05:29.752 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-15T22:05:29.752 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-15T22:05:29.752 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-15T22:05:29.780 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T22:05:29.779+0000 7f5884b58640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T22:05:29.782 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T22:05:29.781+0000 7f5884b58640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T22:05:29.784 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T22:05:29.782+0000 7f5884b58640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T22:05:30.030 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 1 2024-09-15T22:05:30.030 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-15T22:05:30.030 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2024-09-15T22:05:30.030 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-15T22:05:30.030 INFO:tasks.workunit.client.0.smithi136.stdout:0 2024-09-15T22:05:30.031 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-15T22:05:30.031 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:05:30.031 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-15T22:05:30.031 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:05:30.031 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T22:05:30.332 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:05:30.663 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T22:05:30.662+0000 7f5884b58640 -1 Falling back to public interface 2024-09-15T22:05:31.333 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:05:31.333 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:05:31.334 INFO:tasks.workunit.client.0.smithi136.stdout:1 2024-09-15T22:05:31.334 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-15T22:05:31.334 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:05:31.334 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T22:05:31.615 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:05:32.616 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:05:32.616 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:05:32.616 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-15T22:05:32.617 INFO:tasks.workunit.client.0.smithi136.stdout:2 2024-09-15T22:05:32.617 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:05:32.617 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T22:05:32.898 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:05:33.369 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T22:05:33.368+0000 7f5884b58640 -1 osd.1 0 log_to_monitors true 2024-09-15T22:05:33.898 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:05:33.899 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:05:33.899 INFO:tasks.workunit.client.0.smithi136.stdout:3 2024-09-15T22:05:33.899 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-15T22:05:33.899 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:05:33.900 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T22:05:34.190 INFO:tasks.workunit.client.0.smithi136.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/698497346,v1:127.0.0.1:6811/698497346] [v2:127.0.0.1:6812/698497346,v1:127.0.0.1:6813/698497346] exists,up f496a610-436d-4f7b-9801-7464bdb5a9c7 2024-09-15T22:05:34.194 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-15T22:05:34.194 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-15T22:05:34.195 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-15T22:05:34.195 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:318: _scrub_abort: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-09-15T22:05:34.195 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:324: _scrub_abort: run_osd td/osd-scrub-test 2 --osd_pool_default_pg_autoscale_mode=off --osd_deep_scrub_randomize_ratio=0.0 --osd_scrub_sleep=5.0 --osd_scrub_interval_randomize_ratio=0 --osd_op_queue=wpq 2024-09-15T22:05:34.195 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-scrub-test 2024-09-15T22:05:34.195 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-15T22:05:34.195 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=2 2024-09-15T22:05:34.195 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-15T22:05:34.195 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-scrub-test/2 2024-09-15T22:05:34.195 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=79d14839-1c9a-4907-b0ca-211bfb85130f --auth-supported=none --mon-host=127.0.0.1:7138 ' 2024-09-15T22:05:34.195 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T22:05:34.195 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T22:05:34.195 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T22:05:34.195 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-scrub-test/2' 2024-09-15T22:05:34.195 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-test/2/journal' 2024-09-15T22:05:34.196 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-15T22:05:34.196 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-15T22:05:34.196 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-scrub-test' 2024-09-15T22:05:34.196 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-15T22:05:34.196 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-15T22:05:34.196 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-15T22:05:34.196 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-15T22:05:34.196 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T22:05:34.196 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.519518 2024-09-15T22:05:34.196 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.519518/$cluster-$name.asok' 2024-09-15T22:05:34.196 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.519518/$cluster-$name.asok' 2024-09-15T22:05:34.196 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-15T22:05:34.196 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-15T22:05:34.196 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-15T22:05:34.197 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-scrub-test/$name.log' 2024-09-15T22:05:34.197 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-scrub-test/$name.pid' 2024-09-15T22:05:34.197 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T22:05:34.197 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T22:05:34.197 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T22:05:34.197 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T22:05:34.197 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-15T22:05:34.197 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+='--osd_pool_default_pg_autoscale_mode=off --osd_deep_scrub_randomize_ratio=0.0 --osd_scrub_sleep=5.0 --osd_scrub_interval_randomize_ratio=0 --osd_op_queue=wpq' 2024-09-15T22:05:34.197 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-scrub-test/2 2024-09-15T22:05:34.197 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-15T22:05:34.197 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=6c41eaee-d659-409d-9f03-a0159055219d 2024-09-15T22:05:34.197 INFO:tasks.workunit.client.0.smithi136.stdout:add osd2 6c41eaee-d659-409d-9f03-a0159055219d 2024-09-15T22:05:34.198 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd2 6c41eaee-d659-409d-9f03-a0159055219d' 2024-09-15T22:05:34.198 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-15T22:05:34.211 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQAuWudmYDGIDBAAO+U7w3I1h1CjR74p1gEE2A== 2024-09-15T22:05:34.212 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQAuWudmYDGIDBAAO+U7w3I1h1CjR74p1gEE2A=="}' 2024-09-15T22:05:34.212 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 6c41eaee-d659-409d-9f03-a0159055219d -i td/osd-scrub-test/2/new.json 2024-09-15T22:05:34.587 INFO:tasks.workunit.client.0.smithi136.stdout:2 2024-09-15T22:05:34.601 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-scrub-test/2/new.json 2024-09-15T22:05:34.601 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 2 --fsid=79d14839-1c9a-4907-b0ca-211bfb85130f --auth-supported=none --mon-host=127.0.0.1:7138 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-test/2 --osd-journal=td/osd-scrub-test/2/journal --chdir= --run-dir=td/osd-scrub-test '--admin-socket=/tmp/ceph-asok.519518/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-test/$name.log' '--pid-file=td/osd-scrub-test/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_pool_default_pg_autoscale_mode=off --osd_deep_scrub_randomize_ratio=0.0 --osd_scrub_sleep=5.0 --osd_scrub_interval_randomize_ratio=0 --osd_op_queue=wpq --mkfs --key AQAuWudmYDGIDBAAO+U7w3I1h1CjR74p1gEE2A== --osd-uuid 6c41eaee-d659-409d-9f03-a0159055219d 2024-09-15T22:05:34.630 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T22:05:34.628+0000 7efccf413640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T22:05:34.663 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T22:05:34.662+0000 7efccf413640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T22:05:34.664 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T22:05:34.663+0000 7efccf413640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T22:05:34.665 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T22:05:34.663+0000 7efccf413640 -1 bdev(0x561b3edebc00 td/osd-scrub-test/2/block) open stat got: (1) Operation not permitted 2024-09-15T22:05:34.665 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T22:05:34.663+0000 7efccf413640 -1 bluestore(td/osd-scrub-test/2) _read_fsid unparsable uuid 2024-09-15T22:05:40.329 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-scrub-test/2/keyring 2024-09-15T22:05:40.329 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-15T22:05:40.330 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd2 key to auth repository 2024-09-15T22:05:40.330 INFO:tasks.workunit.client.0.smithi136.stdout:adding osd2 key to auth repository 2024-09-15T22:05:40.331 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-scrub-test/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-15T22:05:40.691 INFO:tasks.workunit.client.0.smithi136.stdout:start osd.2 2024-09-15T22:05:40.691 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.2 2024-09-15T22:05:40.691 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 2 --fsid=79d14839-1c9a-4907-b0ca-211bfb85130f --auth-supported=none --mon-host=127.0.0.1:7138 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-test/2 --osd-journal=td/osd-scrub-test/2/journal --chdir= --run-dir=td/osd-scrub-test '--admin-socket=/tmp/ceph-asok.519518/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-test/$name.log' '--pid-file=td/osd-scrub-test/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_pool_default_pg_autoscale_mode=off --osd_deep_scrub_randomize_ratio=0.0 --osd_scrub_sleep=5.0 --osd_scrub_interval_randomize_ratio=0 --osd_op_queue=wpq 2024-09-15T22:05:40.691 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-15T22:05:40.692 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-15T22:05:40.692 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-15T22:05:40.719 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T22:05:40.717+0000 7faa3b216640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T22:05:40.721 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T22:05:40.719+0000 7faa3b216640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T22:05:40.722 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T22:05:40.720+0000 7faa3b216640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T22:05:40.964 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 2 2024-09-15T22:05:40.964 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-15T22:05:40.964 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=2 2024-09-15T22:05:40.964 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-15T22:05:40.964 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-15T22:05:40.964 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:05:40.965 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-15T22:05:40.965 INFO:tasks.workunit.client.0.smithi136.stdout:0 2024-09-15T22:05:40.965 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:05:40.965 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T22:05:41.238 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:05:42.239 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:05:42.239 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:05:42.240 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-15T22:05:42.240 INFO:tasks.workunit.client.0.smithi136.stdout:1 2024-09-15T22:05:42.240 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:05:42.240 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T22:05:42.361 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T22:05:42.360+0000 7faa3b216640 -1 Falling back to public interface 2024-09-15T22:05:42.530 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:05:43.531 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:05:43.532 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:05:43.532 INFO:tasks.workunit.client.0.smithi136.stdout:2 2024-09-15T22:05:43.532 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-15T22:05:43.532 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:05:43.533 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T22:05:43.816 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:05:44.817 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:05:44.817 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:05:44.818 INFO:tasks.workunit.client.0.smithi136.stdout:3 2024-09-15T22:05:44.818 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-15T22:05:44.818 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:05:44.818 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T22:05:45.034 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T22:05:45.033+0000 7faa3b216640 -1 osd.2 0 log_to_monitors true 2024-09-15T22:05:45.125 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:05:46.127 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:05:46.127 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T22:05:46.127 INFO:tasks.workunit.client.0.smithi136.stdout:4 2024-09-15T22:05:46.128 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-15T22:05:46.128 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:05:46.128 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T22:05:46.435 INFO:tasks.workunit.client.0.smithi136.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/2224361528,v1:127.0.0.1:6819/2224361528] [v2:127.0.0.1:6820/2224361528,v1:127.0.0.1:6821/2224361528] exists,up 6c41eaee-d659-409d-9f03-a0159055219d 2024-09-15T22:05:46.435 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-15T22:05:46.436 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-15T22:05:46.436 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-15T22:05:46.436 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: create_pool test 1 1 2024-09-15T22:05:46.436 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create test 1 1 2024-09-15T22:05:46.821 INFO:tasks.workunit.client.0.smithi136.stderr:pool 'test' created 2024-09-15T22:05:46.842 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2024-09-15T22:05:47.833 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:333: _scrub_abort: wait_for_clean 2024-09-15T22:05:47.833 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-15T22:05:47.833 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-15T22:05:47.833 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-15T22:05:47.834 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-15T22:05:47.834 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-15T22:05:47.834 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-15T22:05:47.834 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-15T22:05:47.834 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-15T22:05:47.835 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-15T22:05:47.926 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-15T22:05:47.926 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-15T22:05:47.927 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-15T22:05:47.927 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-15T22:05:47.927 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-15T22:05:47.927 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-15T22:05:48.387 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-15T22:05:48.388 INFO:tasks.workunit.client.0.smithi136.stderr:1 2024-09-15T22:05:48.388 INFO:tasks.workunit.client.0.smithi136.stderr:2' 2024-09-15T22:05:48.388 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-15T22:05:48.388 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T22:05:48.388 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-15T22:05:48.555 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836487 2024-09-15T22:05:48.555 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836487 2024-09-15T22:05:48.555 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836487' 2024-09-15T22:05:48.555 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T22:05:48.555 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-15T22:05:48.696 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=42949672964 2024-09-15T22:05:48.697 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 42949672964 2024-09-15T22:05:48.697 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836487 1-42949672964' 2024-09-15T22:05:48.697 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T22:05:48.697 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-15T22:05:48.814 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=64424509442 2024-09-15T22:05:48.814 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 64424509442 2024-09-15T22:05:48.814 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836487 1-42949672964 2-64424509442' 2024-09-15T22:05:48.814 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T22:05:48.815 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836487 2024-09-15T22:05:48.815 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T22:05:48.815 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-15T22:05:48.816 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836487 2024-09-15T22:05:48.816 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T22:05:48.817 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836487 2024-09-15T22:05:48.817 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836487' 2024-09-15T22:05:48.817 INFO:tasks.workunit.client.0.smithi136.stdout:waiting osd.0 seq 21474836487 2024-09-15T22:05:48.818 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T22:05:49.089 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836486 -lt 21474836487 2024-09-15T22:05:49.089 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-15T22:05:50.090 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-15T22:05:50.090 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T22:05:50.394 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836487 -lt 21474836487 2024-09-15T22:05:50.394 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T22:05:50.395 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-42949672964 2024-09-15T22:05:50.395 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T22:05:50.395 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-15T22:05:50.396 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-42949672964 2024-09-15T22:05:50.396 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T22:05:50.397 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=42949672964 2024-09-15T22:05:50.397 INFO:tasks.workunit.client.0.smithi136.stdout:waiting osd.1 seq 42949672964 2024-09-15T22:05:50.397 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 42949672964' 2024-09-15T22:05:50.398 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-15T22:05:50.688 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 42949672965 -lt 42949672964 2024-09-15T22:05:50.688 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T22:05:50.689 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-64424509442 2024-09-15T22:05:50.689 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T22:05:50.689 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-15T22:05:50.690 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-64424509442 2024-09-15T22:05:50.690 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T22:05:50.691 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=64424509442 2024-09-15T22:05:50.691 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 64424509442' 2024-09-15T22:05:50.692 INFO:tasks.workunit.client.0.smithi136.stdout:waiting osd.2 seq 64424509442 2024-09-15T22:05:50.692 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-15T22:05:50.967 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 64424509442 -lt 64424509442 2024-09-15T22:05:50.968 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-15T22:05:50.968 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T22:05:50.968 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T22:05:51.311 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 1 == 0 2024-09-15T22:05:51.312 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-15T22:05:51.312 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-15T22:05:51.312 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-15T22:05:51.312 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T22:05:51.312 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T22:05:51.312 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T22:05:51.312 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T22:05:51.584 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=1 2024-09-15T22:05:51.585 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-15T22:05:51.585 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T22:05:51.585 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T22:05:51.928 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 = 1 2024-09-15T22:05:51.972 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-15T22:05:51.972 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-15T22:05:51.973 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:334: _scrub_abort: ceph osd dump 2024-09-15T22:05:51.973 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:334: _scrub_abort: grep '^pool.*['\'']test['\'']' 2024-09-15T22:05:51.973 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:334: _scrub_abort: awk '{ print $2 }' 2024-09-15T22:05:52.202 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:334: _scrub_abort: poolid=1 2024-09-15T22:05:52.202 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:336: _scrub_abort: dd if=/dev/urandom of=testdata.519518 bs=1032 count=1 2024-09-15T22:05:52.202 INFO:tasks.workunit.client.0.smithi136.stderr:1+0 records in 2024-09-15T22:05:52.202 INFO:tasks.workunit.client.0.smithi136.stderr:1+0 records out 2024-09-15T22:05:52.203 INFO:tasks.workunit.client.0.smithi136.stderr:1032 bytes (1.0 kB, 1.0 KiB) copied, 4.6268e-05 s, 22.3 MB/s 2024-09-15T22:05:52.203 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: seq 1 1000 2024-09-15T22:05:52.204 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:05:52.204 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj1 testdata.519518 2024-09-15T22:05:52.360 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:05:52.360 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj2 testdata.519518 2024-09-15T22:05:52.550 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:05:52.550 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj3 testdata.519518 2024-09-15T22:05:52.678 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:05:52.678 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj4 testdata.519518 2024-09-15T22:05:52.804 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:05:52.804 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj5 testdata.519518 2024-09-15T22:05:52.930 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:05:52.930 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj6 testdata.519518 2024-09-15T22:05:53.072 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:05:53.073 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj7 testdata.519518 2024-09-15T22:05:53.198 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:05:53.198 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj8 testdata.519518 2024-09-15T22:05:53.323 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:05:53.324 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj9 testdata.519518 2024-09-15T22:05:53.458 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:05:53.458 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj10 testdata.519518 2024-09-15T22:05:53.583 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:05:53.583 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj11 testdata.519518 2024-09-15T22:05:53.709 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:05:53.709 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj12 testdata.519518 2024-09-15T22:05:53.843 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:05:53.843 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj13 testdata.519518 2024-09-15T22:05:53.985 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:05:53.986 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj14 testdata.519518 2024-09-15T22:05:54.128 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:05:54.128 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj15 testdata.519518 2024-09-15T22:05:54.270 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:05:54.271 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj16 testdata.519518 2024-09-15T22:05:54.422 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:05:54.422 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj17 testdata.519518 2024-09-15T22:05:54.572 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:05:54.573 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj18 testdata.519518 2024-09-15T22:05:54.706 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:05:54.706 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj19 testdata.519518 2024-09-15T22:05:54.832 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:05:54.832 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj20 testdata.519518 2024-09-15T22:05:54.958 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:05:54.958 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj21 testdata.519518 2024-09-15T22:05:55.083 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:05:55.084 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj22 testdata.519518 2024-09-15T22:05:55.217 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:05:55.217 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj23 testdata.519518 2024-09-15T22:05:55.360 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:05:55.360 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj24 testdata.519518 2024-09-15T22:05:55.494 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:05:55.494 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj25 testdata.519518 2024-09-15T22:05:55.628 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:05:55.628 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj26 testdata.519518 2024-09-15T22:05:55.754 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:05:55.754 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj27 testdata.519518 2024-09-15T22:05:55.879 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:05:55.879 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj28 testdata.519518 2024-09-15T22:05:56.013 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:05:56.013 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj29 testdata.519518 2024-09-15T22:05:56.147 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:05:56.148 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj30 testdata.519518 2024-09-15T22:05:56.315 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:05:56.315 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj31 testdata.519518 2024-09-15T22:05:56.474 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:05:56.474 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj32 testdata.519518 2024-09-15T22:05:56.633 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:05:56.633 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj33 testdata.519518 2024-09-15T22:05:56.767 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:05:56.768 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj34 testdata.519518 2024-09-15T22:05:56.902 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:05:56.902 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj35 testdata.519518 2024-09-15T22:05:57.061 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:05:57.061 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj36 testdata.519518 2024-09-15T22:05:57.194 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:05:57.195 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj37 testdata.519518 2024-09-15T22:05:57.320 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:05:57.321 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj38 testdata.519518 2024-09-15T22:05:57.471 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:05:57.471 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj39 testdata.519518 2024-09-15T22:05:57.605 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:05:57.605 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj40 testdata.519518 2024-09-15T22:05:57.731 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:05:57.731 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj41 testdata.519518 2024-09-15T22:05:57.865 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:05:57.866 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj42 testdata.519518 2024-09-15T22:05:57.999 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:05:57.999 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj43 testdata.519518 2024-09-15T22:05:58.125 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:05:58.125 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj44 testdata.519518 2024-09-15T22:05:58.267 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:05:58.267 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj45 testdata.519518 2024-09-15T22:05:58.410 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:05:58.410 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj46 testdata.519518 2024-09-15T22:05:58.569 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:05:58.569 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj47 testdata.519518 2024-09-15T22:05:58.703 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:05:58.703 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj48 testdata.519518 2024-09-15T22:05:58.828 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:05:58.828 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj49 testdata.519518 2024-09-15T22:05:58.957 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:05:58.957 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj50 testdata.519518 2024-09-15T22:05:59.091 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:05:59.091 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj51 testdata.519518 2024-09-15T22:05:59.216 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:05:59.216 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj52 testdata.519518 2024-09-15T22:05:59.342 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:05:59.342 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj53 testdata.519518 2024-09-15T22:05:59.476 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:05:59.476 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj54 testdata.519518 2024-09-15T22:05:59.618 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:05:59.619 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj55 testdata.519518 2024-09-15T22:05:59.761 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:05:59.761 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj56 testdata.519518 2024-09-15T22:05:59.886 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:05:59.886 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj57 testdata.519518 2024-09-15T22:06:00.037 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:06:00.038 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj58 testdata.519518 2024-09-15T22:06:00.171 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:06:00.171 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj59 testdata.519518 2024-09-15T22:06:00.331 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:06:00.331 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj60 testdata.519518 2024-09-15T22:06:00.507 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:06:00.507 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj61 testdata.519518 2024-09-15T22:06:00.682 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:06:00.682 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj62 testdata.519518 2024-09-15T22:06:00.808 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:06:00.808 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj63 testdata.519518 2024-09-15T22:06:00.942 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:06:00.942 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj64 testdata.519518 2024-09-15T22:06:01.068 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:06:01.068 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj65 testdata.519518 2024-09-15T22:06:01.210 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:06:01.210 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj66 testdata.519518 2024-09-15T22:06:01.344 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:06:01.344 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj67 testdata.519518 2024-09-15T22:06:01.470 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:06:01.470 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj68 testdata.519518 2024-09-15T22:06:01.595 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:06:01.595 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj69 testdata.519518 2024-09-15T22:06:01.730 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:06:01.730 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj70 testdata.519518 2024-09-15T22:06:01.864 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:06:01.864 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj71 testdata.519518 2024-09-15T22:06:02.006 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:06:02.006 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj72 testdata.519518 2024-09-15T22:06:02.140 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:06:02.140 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj73 testdata.519518 2024-09-15T22:06:02.316 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:06:02.316 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj74 testdata.519518 2024-09-15T22:06:02.484 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:06:02.484 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj75 testdata.519518 2024-09-15T22:06:02.893 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:06:02.894 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj76 testdata.519518 2024-09-15T22:06:03.044 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:06:03.044 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj77 testdata.519518 2024-09-15T22:06:03.195 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:06:03.195 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj78 testdata.519518 2024-09-15T22:06:03.329 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:06:03.329 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj79 testdata.519518 2024-09-15T22:06:03.471 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:06:03.471 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj80 testdata.519518 2024-09-15T22:06:03.613 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:06:03.613 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj81 testdata.519518 2024-09-15T22:06:03.748 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:06:03.748 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj82 testdata.519518 2024-09-15T22:06:03.890 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:06:03.890 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj83 testdata.519518 2024-09-15T22:06:04.032 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:06:04.032 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj84 testdata.519518 2024-09-15T22:06:04.166 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:06:04.166 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj85 testdata.519518 2024-09-15T22:06:04.309 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:06:04.309 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj86 testdata.519518 2024-09-15T22:06:04.493 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:06:04.493 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj87 testdata.519518 2024-09-15T22:06:04.660 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:06:04.660 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj88 testdata.519518 2024-09-15T22:06:04.786 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:06:04.786 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj89 testdata.519518 2024-09-15T22:06:04.912 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:06:04.912 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj90 testdata.519518 2024-09-15T22:06:05.047 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:06:05.047 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj91 testdata.519518 2024-09-15T22:06:05.173 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:06:05.173 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj92 testdata.519518 2024-09-15T22:06:05.308 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:06:05.308 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj93 testdata.519518 2024-09-15T22:06:05.442 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:06:05.442 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj94 testdata.519518 2024-09-15T22:06:05.577 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:06:05.577 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj95 testdata.519518 2024-09-15T22:06:05.704 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:06:05.704 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj96 testdata.519518 2024-09-15T22:06:05.838 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:06:05.838 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj97 testdata.519518 2024-09-15T22:06:05.974 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:06:05.974 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj98 testdata.519518 2024-09-15T22:06:06.124 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:06:06.124 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj99 testdata.519518 2024-09-15T22:06:06.267 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:06:06.267 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj100 testdata.519518 2024-09-15T22:06:06.409 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:06:06.409 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj101 testdata.519518 2024-09-15T22:06:06.554 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:06:06.554 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj102 testdata.519518 2024-09-15T22:06:06.688 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:06:06.688 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj103 testdata.519518 2024-09-15T22:06:06.825 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:06:06.825 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj104 testdata.519518 2024-09-15T22:06:06.959 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:06:06.959 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj105 testdata.519518 2024-09-15T22:06:07.104 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:06:07.104 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj106 testdata.519518 2024-09-15T22:06:07.236 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:06:07.236 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj107 testdata.519518 2024-09-15T22:06:07.378 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:06:07.378 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj108 testdata.519518 2024-09-15T22:06:07.504 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:06:07.504 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj109 testdata.519518 2024-09-15T22:06:07.629 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:06:07.630 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj110 testdata.519518 2024-09-15T22:06:07.764 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:06:07.764 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj111 testdata.519518 2024-09-15T22:06:07.889 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:06:07.890 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj112 testdata.519518 2024-09-15T22:06:08.025 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:06:08.025 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj113 testdata.519518 2024-09-15T22:06:08.159 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:06:08.160 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj114 testdata.519518 2024-09-15T22:06:08.344 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:06:08.344 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj115 testdata.519518 2024-09-15T22:06:08.512 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:06:08.512 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj116 testdata.519518 2024-09-15T22:06:08.646 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:06:08.646 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj117 testdata.519518 2024-09-15T22:06:08.780 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:06:08.780 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj118 testdata.519518 2024-09-15T22:06:08.905 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:06:08.906 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj119 testdata.519518 2024-09-15T22:06:09.040 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:06:09.040 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj120 testdata.519518 2024-09-15T22:06:09.174 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:06:09.174 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj121 testdata.519518 2024-09-15T22:06:09.308 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:06:09.308 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj122 testdata.519518 2024-09-15T22:06:09.453 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:06:09.453 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj123 testdata.519518 2024-09-15T22:06:09.578 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:06:09.578 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj124 testdata.519518 2024-09-15T22:06:09.713 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:06:09.713 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj125 testdata.519518 2024-09-15T22:06:09.847 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:06:09.847 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj126 testdata.519518 2024-09-15T22:06:09.973 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:06:09.973 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj127 testdata.519518 2024-09-15T22:06:10.098 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:06:10.098 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj128 testdata.519518 2024-09-15T22:06:10.241 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:06:10.241 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj129 testdata.519518 2024-09-15T22:06:10.409 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:06:10.409 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj130 testdata.519518 2024-09-15T22:06:10.543 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:06:10.543 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj131 testdata.519518 2024-09-15T22:06:10.677 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:06:10.677 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj132 testdata.519518 2024-09-15T22:06:10.803 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:06:10.803 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj133 testdata.519518 2024-09-15T22:06:10.929 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:06:10.929 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj134 testdata.519518 2024-09-15T22:06:11.063 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:06:11.063 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj135 testdata.519518 2024-09-15T22:06:11.197 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:06:11.197 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj136 testdata.519518 2024-09-15T22:06:11.332 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:06:11.332 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj137 testdata.519518 2024-09-15T22:06:11.466 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:06:11.466 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj138 testdata.519518 2024-09-15T22:06:11.600 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:06:11.600 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj139 testdata.519518 2024-09-15T22:06:11.734 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:06:11.734 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj140 testdata.519518 2024-09-15T22:06:11.877 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:06:11.878 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj141 testdata.519518 2024-09-15T22:06:12.012 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:06:12.012 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj142 testdata.519518 2024-09-15T22:06:12.147 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:06:12.147 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj143 testdata.519518 2024-09-15T22:06:12.306 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:06:12.307 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj144 testdata.519518 2024-09-15T22:06:12.484 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:06:12.484 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj145 testdata.519518 2024-09-15T22:06:12.618 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:06:12.619 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj146 testdata.519518 2024-09-15T22:06:12.778 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:06:12.778 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj147 testdata.519518 2024-09-15T22:06:12.903 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:06:12.903 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj148 testdata.519518 2024-09-15T22:06:13.029 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:06:13.029 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj149 testdata.519518 2024-09-15T22:06:13.163 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:06:13.163 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj150 testdata.519518 2024-09-15T22:06:13.289 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:06:13.289 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj151 testdata.519518 2024-09-15T22:06:13.423 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:06:13.423 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj152 testdata.519518 2024-09-15T22:06:13.557 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:06:13.558 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj153 testdata.519518 2024-09-15T22:06:13.691 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:06:13.692 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj154 testdata.519518 2024-09-15T22:06:13.817 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:06:13.817 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj155 testdata.519518 2024-09-15T22:06:13.960 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:06:13.960 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj156 testdata.519518 2024-09-15T22:06:14.103 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:06:14.103 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj157 testdata.519518 2024-09-15T22:06:14.237 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:06:14.237 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj158 testdata.519518 2024-09-15T22:06:14.413 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:06:14.413 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj159 testdata.519518 2024-09-15T22:06:14.539 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:06:14.539 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj160 testdata.519518 2024-09-15T22:06:14.673 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:06:14.676 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj161 testdata.519518 2024-09-15T22:06:14.816 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:06:14.816 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj162 testdata.519518 2024-09-15T22:06:14.950 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:06:14.950 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj163 testdata.519518 2024-09-15T22:06:15.093 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:06:15.093 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj164 testdata.519518 2024-09-15T22:06:15.227 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:06:15.227 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj165 testdata.519518 2024-09-15T22:06:15.361 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:06:15.361 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj166 testdata.519518 2024-09-15T22:06:15.496 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:06:15.497 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj167 testdata.519518 2024-09-15T22:06:15.629 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:06:15.629 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj168 testdata.519518 2024-09-15T22:06:15.755 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:06:15.756 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj169 testdata.519518 2024-09-15T22:06:15.881 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:06:15.881 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj170 testdata.519518 2024-09-15T22:06:16.007 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:06:16.007 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj171 testdata.519518 2024-09-15T22:06:16.141 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:06:16.142 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj172 testdata.519518 2024-09-15T22:06:16.303 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:06:16.303 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj173 testdata.519518 2024-09-15T22:06:16.435 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:06:16.436 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj174 testdata.519518 2024-09-15T22:06:16.561 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:06:16.561 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj175 testdata.519518 2024-09-15T22:06:16.687 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:06:16.687 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj176 testdata.519518 2024-09-15T22:06:16.846 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:06:16.846 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj177 testdata.519518 2024-09-15T22:06:16.980 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:06:16.980 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj178 testdata.519518 2024-09-15T22:06:17.122 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:06:17.123 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj179 testdata.519518 2024-09-15T22:06:17.258 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:06:17.258 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj180 testdata.519518 2024-09-15T22:06:17.391 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:06:17.392 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj181 testdata.519518 2024-09-15T22:06:17.526 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:06:17.526 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj182 testdata.519518 2024-09-15T22:06:17.660 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:06:17.660 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj183 testdata.519518 2024-09-15T22:06:17.794 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:06:17.794 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj184 testdata.519518 2024-09-15T22:06:17.920 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:06:17.920 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj185 testdata.519518 2024-09-15T22:06:18.062 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:06:18.062 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj186 testdata.519518 2024-09-15T22:06:18.213 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:06:18.213 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj187 testdata.519518 2024-09-15T22:06:18.357 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:06:18.357 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj188 testdata.519518 2024-09-15T22:06:18.524 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:06:18.524 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj189 testdata.519518 2024-09-15T22:06:18.658 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:06:18.658 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj190 testdata.519518 2024-09-15T22:06:18.792 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:06:18.792 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj191 testdata.519518 2024-09-15T22:06:18.926 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:06:18.926 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj192 testdata.519518 2024-09-15T22:06:19.060 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:06:19.060 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj193 testdata.519518 2024-09-15T22:06:19.186 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:06:19.186 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj194 testdata.519518 2024-09-15T22:06:19.320 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:06:19.320 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj195 testdata.519518 2024-09-15T22:06:19.446 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:06:19.446 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj196 testdata.519518 2024-09-15T22:06:19.572 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:06:19.572 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj197 testdata.519518 2024-09-15T22:06:19.697 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:06:19.697 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj198 testdata.519518 2024-09-15T22:06:19.824 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:06:19.824 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj199 testdata.519518 2024-09-15T22:06:19.958 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:06:19.958 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj200 testdata.519518 2024-09-15T22:06:20.092 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:06:20.093 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj201 testdata.519518 2024-09-15T22:06:20.252 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:06:20.252 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj202 testdata.519518 2024-09-15T22:06:20.428 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:06:20.428 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj203 testdata.519518 2024-09-15T22:06:20.562 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:06:20.562 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj204 testdata.519518 2024-09-15T22:06:20.713 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:06:20.713 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj205 testdata.519518 2024-09-15T22:06:20.881 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:06:20.881 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj206 testdata.519518 2024-09-15T22:06:21.015 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:06:21.015 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj207 testdata.519518 2024-09-15T22:06:21.166 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:06:21.166 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj208 testdata.519518 2024-09-15T22:06:21.292 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:06:21.292 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj209 testdata.519518 2024-09-15T22:06:21.418 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:06:21.418 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj210 testdata.519518 2024-09-15T22:06:21.568 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:06:21.568 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj211 testdata.519518 2024-09-15T22:06:21.694 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:06:21.694 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj212 testdata.519518 2024-09-15T22:06:21.820 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:06:21.820 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj213 testdata.519518 2024-09-15T22:06:21.954 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:06:21.954 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj214 testdata.519518 2024-09-15T22:06:22.080 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:06:22.080 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj215 testdata.519518 2024-09-15T22:06:22.222 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:06:22.222 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj216 testdata.519518 2024-09-15T22:06:22.365 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:06:22.365 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj217 testdata.519518 2024-09-15T22:06:22.525 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:06:22.525 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj218 testdata.519518 2024-09-15T22:06:22.650 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:06:22.651 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj219 testdata.519518 2024-09-15T22:06:22.785 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:06:22.785 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj220 testdata.519518 2024-09-15T22:06:22.910 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:06:22.911 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj221 testdata.519518 2024-09-15T22:06:23.045 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:06:23.045 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj222 testdata.519518 2024-09-15T22:06:23.179 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:06:23.179 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj223 testdata.519518 2024-09-15T22:06:23.313 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:06:23.313 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj224 testdata.519518 2024-09-15T22:06:23.447 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:06:23.447 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj225 testdata.519518 2024-09-15T22:06:23.581 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:06:23.581 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj226 testdata.519518 2024-09-15T22:06:23.724 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:06:23.724 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj227 testdata.519518 2024-09-15T22:06:23.858 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:06:23.858 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj228 testdata.519518 2024-09-15T22:06:23.984 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:06:23.984 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj229 testdata.519518 2024-09-15T22:06:24.135 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:06:24.135 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj230 testdata.519518 2024-09-15T22:06:24.288 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:06:24.288 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj231 testdata.519518 2024-09-15T22:06:24.445 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:06:24.445 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj232 testdata.519518 2024-09-15T22:06:24.579 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:06:24.579 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj233 testdata.519518 2024-09-15T22:06:24.704 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:06:24.705 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj234 testdata.519518 2024-09-15T22:06:24.872 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:06:24.872 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj235 testdata.519518 2024-09-15T22:06:25.007 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:06:25.007 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj236 testdata.519518 2024-09-15T22:06:25.149 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:06:25.149 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj237 testdata.519518 2024-09-15T22:06:25.292 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:06:25.292 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj238 testdata.519518 2024-09-15T22:06:25.417 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:06:25.417 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj239 testdata.519518 2024-09-15T22:06:25.560 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:06:25.560 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj240 testdata.519518 2024-09-15T22:06:25.694 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:06:25.694 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj241 testdata.519518 2024-09-15T22:06:25.819 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:06:25.820 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj242 testdata.519518 2024-09-15T22:06:25.954 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:06:25.954 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj243 testdata.519518 2024-09-15T22:06:26.088 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:06:26.088 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj244 testdata.519518 2024-09-15T22:06:26.222 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:06:26.222 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj245 testdata.519518 2024-09-15T22:06:26.381 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:06:26.381 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj246 testdata.519518 2024-09-15T22:06:26.541 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:06:26.541 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj247 testdata.519518 2024-09-15T22:06:26.667 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:06:26.667 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj248 testdata.519518 2024-09-15T22:06:26.801 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:06:26.801 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj249 testdata.519518 2024-09-15T22:06:26.952 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:06:26.952 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj250 testdata.519518 2024-09-15T22:06:27.077 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:06:27.078 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj251 testdata.519518 2024-09-15T22:06:27.228 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:06:27.229 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj252 testdata.519518 2024-09-15T22:06:27.363 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:06:27.363 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj253 testdata.519518 2024-09-15T22:06:27.502 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:06:27.503 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj254 testdata.519518 2024-09-15T22:06:27.639 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:06:27.639 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj255 testdata.519518 2024-09-15T22:06:27.773 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:06:27.773 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj256 testdata.519518 2024-09-15T22:06:27.916 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:06:27.916 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj257 testdata.519518 2024-09-15T22:06:28.050 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:06:28.050 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj258 testdata.519518 2024-09-15T22:06:28.193 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:06:28.193 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj259 testdata.519518 2024-09-15T22:06:28.335 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:06:28.335 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj260 testdata.519518 2024-09-15T22:06:28.503 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:06:28.503 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj261 testdata.519518 2024-09-15T22:06:28.637 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:06:28.637 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj262 testdata.519518 2024-09-15T22:06:28.763 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:06:28.763 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj263 testdata.519518 2024-09-15T22:06:28.939 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:06:28.939 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj264 testdata.519518 2024-09-15T22:06:29.064 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:06:29.065 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj265 testdata.519518 2024-09-15T22:06:29.207 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:06:29.207 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj266 testdata.519518 2024-09-15T22:06:29.333 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:06:29.333 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj267 testdata.519518 2024-09-15T22:06:29.467 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:06:29.468 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj268 testdata.519518 2024-09-15T22:06:29.601 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:06:29.602 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj269 testdata.519518 2024-09-15T22:06:29.735 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:06:29.735 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj270 testdata.519518 2024-09-15T22:06:29.861 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:06:29.861 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj271 testdata.519518 2024-09-15T22:06:29.995 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:06:29.995 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj272 testdata.519518 2024-09-15T22:06:30.130 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:06:30.130 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj273 testdata.519518 2024-09-15T22:06:30.272 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:06:30.272 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj274 testdata.519518 2024-09-15T22:06:30.406 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:06:30.406 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj275 testdata.519518 2024-09-15T22:06:30.566 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:06:30.566 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj276 testdata.519518 2024-09-15T22:06:30.700 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:06:30.701 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj277 testdata.519518 2024-09-15T22:06:30.837 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:06:30.837 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj278 testdata.519518 2024-09-15T22:06:30.968 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:06:30.968 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj279 testdata.519518 2024-09-15T22:06:31.111 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:06:31.111 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj280 testdata.519518 2024-09-15T22:06:31.245 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:06:31.245 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj281 testdata.519518 2024-09-15T22:06:31.379 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:06:31.379 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj282 testdata.519518 2024-09-15T22:06:31.538 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:06:31.538 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj283 testdata.519518 2024-09-15T22:06:31.663 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:06:31.663 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj284 testdata.519518 2024-09-15T22:06:31.789 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:06:31.789 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj285 testdata.519518 2024-09-15T22:06:31.932 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:06:31.932 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj286 testdata.519518 2024-09-15T22:06:32.066 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:06:32.066 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj287 testdata.519518 2024-09-15T22:06:32.201 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:06:32.201 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj288 testdata.519518 2024-09-15T22:06:32.343 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:06:32.343 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj289 testdata.519518 2024-09-15T22:06:32.519 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:06:32.520 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj290 testdata.519518 2024-09-15T22:06:32.662 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:06:32.662 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj291 testdata.519518 2024-09-15T22:06:32.789 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:06:32.789 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj292 testdata.519518 2024-09-15T22:06:32.946 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:06:32.947 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj293 testdata.519518 2024-09-15T22:06:33.081 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:06:33.081 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj294 testdata.519518 2024-09-15T22:06:33.215 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:06:33.215 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj295 testdata.519518 2024-09-15T22:06:33.349 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:06:33.349 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj296 testdata.519518 2024-09-15T22:06:33.484 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:06:33.484 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj297 testdata.519518 2024-09-15T22:06:33.617 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:06:33.618 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj298 testdata.519518 2024-09-15T22:06:33.752 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:06:33.752 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj299 testdata.519518 2024-09-15T22:06:34.312 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:06:34.312 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj300 testdata.519518 2024-09-15T22:06:34.471 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:06:34.471 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj301 testdata.519518 2024-09-15T22:06:34.623 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:06:34.623 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj302 testdata.519518 2024-09-15T22:06:34.756 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:06:34.756 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj303 testdata.519518 2024-09-15T22:06:34.899 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:06:34.899 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj304 testdata.519518 2024-09-15T22:06:35.033 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:06:35.033 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj305 testdata.519518 2024-09-15T22:06:35.167 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:06:35.167 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj306 testdata.519518 2024-09-15T22:06:35.318 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:06:35.318 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj307 testdata.519518 2024-09-15T22:06:35.460 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:06:35.460 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj308 testdata.519518 2024-09-15T22:06:35.594 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:06:35.594 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj309 testdata.519518 2024-09-15T22:06:35.737 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:06:35.737 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj310 testdata.519518 2024-09-15T22:06:35.862 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:06:35.862 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj311 testdata.519518 2024-09-15T22:06:35.988 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:06:35.988 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj312 testdata.519518 2024-09-15T22:06:36.114 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:06:36.114 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj313 testdata.519518 2024-09-15T22:06:36.265 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:06:36.265 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj314 testdata.519518 2024-09-15T22:06:36.408 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:06:36.408 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj315 testdata.519518 2024-09-15T22:06:36.550 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:06:36.550 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj316 testdata.519518 2024-09-15T22:06:36.709 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:06:36.710 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj317 testdata.519518 2024-09-15T22:06:36.844 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:06:36.844 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj318 testdata.519518 2024-09-15T22:06:36.995 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:06:36.995 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj319 testdata.519518 2024-09-15T22:06:37.129 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:06:37.129 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj320 testdata.519518 2024-09-15T22:06:37.263 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:06:37.263 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj321 testdata.519518 2024-09-15T22:06:37.397 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:06:37.397 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj322 testdata.519518 2024-09-15T22:06:37.523 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:06:37.523 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj323 testdata.519518 2024-09-15T22:06:37.648 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:06:37.649 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj324 testdata.519518 2024-09-15T22:06:37.782 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:06:37.783 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj325 testdata.519518 2024-09-15T22:06:37.908 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:06:37.908 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj326 testdata.519518 2024-09-15T22:06:38.059 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:06:38.059 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj327 testdata.519518 2024-09-15T22:06:38.218 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:06:38.218 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj328 testdata.519518 2024-09-15T22:06:38.378 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:06:38.378 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj329 testdata.519518 2024-09-15T22:06:38.529 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:06:38.529 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj330 testdata.519518 2024-09-15T22:06:38.663 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:06:38.663 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj331 testdata.519518 2024-09-15T22:06:38.797 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:06:38.797 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj332 testdata.519518 2024-09-15T22:06:38.931 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:06:38.932 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj333 testdata.519518 2024-09-15T22:06:39.074 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:06:39.074 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj334 testdata.519518 2024-09-15T22:06:39.208 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:06:39.208 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj335 testdata.519518 2024-09-15T22:06:39.351 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:06:39.351 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj336 testdata.519518 2024-09-15T22:06:39.510 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:06:39.510 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj337 testdata.519518 2024-09-15T22:06:39.644 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:06:39.644 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj338 testdata.519518 2024-09-15T22:06:39.786 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:06:39.787 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj339 testdata.519518 2024-09-15T22:06:39.912 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:06:39.912 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj340 testdata.519518 2024-09-15T22:06:40.046 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:06:40.046 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj341 testdata.519518 2024-09-15T22:06:40.181 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:06:40.181 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj342 testdata.519518 2024-09-15T22:06:40.331 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:06:40.332 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj343 testdata.519518 2024-09-15T22:06:40.499 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:06:40.500 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj344 testdata.519518 2024-09-15T22:06:40.634 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:06:40.634 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj345 testdata.519518 2024-09-15T22:06:40.768 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:06:40.768 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj346 testdata.519518 2024-09-15T22:06:40.902 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:06:40.902 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj347 testdata.519518 2024-09-15T22:06:41.053 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:06:41.053 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj348 testdata.519518 2024-09-15T22:06:41.178 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:06:41.178 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj349 testdata.519518 2024-09-15T22:06:41.305 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:06:41.305 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj350 testdata.519518 2024-09-15T22:06:41.430 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:06:41.430 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj351 testdata.519518 2024-09-15T22:06:41.564 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:06:41.564 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj352 testdata.519518 2024-09-15T22:06:41.699 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:06:41.699 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj353 testdata.519518 2024-09-15T22:06:41.832 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:06:41.833 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj354 testdata.519518 2024-09-15T22:06:41.958 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:06:41.959 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj355 testdata.519518 2024-09-15T22:06:42.101 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:06:42.101 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj356 testdata.519518 2024-09-15T22:06:42.252 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:06:42.252 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj357 testdata.519518 2024-09-15T22:06:42.419 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:06:42.420 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj358 testdata.519518 2024-09-15T22:06:42.553 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:06:42.554 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj359 testdata.519518 2024-09-15T22:06:42.679 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:06:42.679 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj360 testdata.519518 2024-09-15T22:06:42.813 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:06:42.814 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj361 testdata.519518 2024-09-15T22:06:42.956 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:06:42.956 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj362 testdata.519518 2024-09-15T22:06:43.082 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:06:43.082 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj363 testdata.519518 2024-09-15T22:06:43.208 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:06:43.208 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj364 testdata.519518 2024-09-15T22:06:43.342 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:06:43.342 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj365 testdata.519518 2024-09-15T22:06:43.476 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:06:43.476 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj366 testdata.519518 2024-09-15T22:06:43.618 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:06:43.619 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj367 testdata.519518 2024-09-15T22:06:43.753 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:06:43.753 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj368 testdata.519518 2024-09-15T22:06:43.886 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:06:43.887 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj369 testdata.519518 2024-09-15T22:06:44.012 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:06:44.012 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj370 testdata.519518 2024-09-15T22:06:44.138 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:06:44.138 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj371 testdata.519518 2024-09-15T22:06:44.281 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:06:44.282 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj372 testdata.519518 2024-09-15T22:06:44.432 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:06:44.432 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj373 testdata.519518 2024-09-15T22:06:44.558 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:06:44.558 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj374 testdata.519518 2024-09-15T22:06:44.692 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:06:44.692 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj375 testdata.519518 2024-09-15T22:06:44.826 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:06:44.826 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj376 testdata.519518 2024-09-15T22:06:44.977 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:06:44.977 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj377 testdata.519518 2024-09-15T22:06:45.119 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:06:45.120 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj378 testdata.519518 2024-09-15T22:06:45.253 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:06:45.254 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj379 testdata.519518 2024-09-15T22:06:45.388 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:06:45.388 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj380 testdata.519518 2024-09-15T22:06:45.514 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:06:45.514 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj381 testdata.519518 2024-09-15T22:06:45.656 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:06:45.657 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj382 testdata.519518 2024-09-15T22:06:45.790 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:06:45.790 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj383 testdata.519518 2024-09-15T22:06:45.925 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:06:45.925 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj384 testdata.519518 2024-09-15T22:06:46.058 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:06:46.059 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj385 testdata.519518 2024-09-15T22:06:46.201 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:06:46.201 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj386 testdata.519518 2024-09-15T22:06:46.361 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:06:46.361 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj387 testdata.519518 2024-09-15T22:06:46.537 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:06:46.537 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj388 testdata.519518 2024-09-15T22:06:46.662 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:06:46.662 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj389 testdata.519518 2024-09-15T22:06:46.797 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:06:46.797 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj390 testdata.519518 2024-09-15T22:06:46.931 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:06:46.931 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj391 testdata.519518 2024-09-15T22:06:47.073 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:06:47.073 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj392 testdata.519518 2024-09-15T22:06:47.199 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:06:47.199 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj393 testdata.519518 2024-09-15T22:06:47.325 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:06:47.325 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj394 testdata.519518 2024-09-15T22:06:47.459 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:06:47.459 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj395 testdata.519518 2024-09-15T22:06:47.585 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:06:47.585 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj396 testdata.519518 2024-09-15T22:06:47.727 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:06:47.727 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj397 testdata.519518 2024-09-15T22:06:47.853 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:06:47.853 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj398 testdata.519518 2024-09-15T22:06:47.979 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:06:47.979 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj399 testdata.519518 2024-09-15T22:06:48.122 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:06:48.122 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj400 testdata.519518 2024-09-15T22:06:48.272 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:06:48.273 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj401 testdata.519518 2024-09-15T22:06:48.415 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:06:48.415 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj402 testdata.519518 2024-09-15T22:06:48.551 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:06:48.551 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj403 testdata.519518 2024-09-15T22:06:48.692 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:06:48.692 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj404 testdata.519518 2024-09-15T22:06:48.835 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:06:48.835 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj405 testdata.519518 2024-09-15T22:06:48.994 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:06:48.994 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj406 testdata.519518 2024-09-15T22:06:49.128 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:06:49.128 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj407 testdata.519518 2024-09-15T22:06:49.253 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:06:49.254 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj408 testdata.519518 2024-09-15T22:06:49.387 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:06:49.388 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj409 testdata.519518 2024-09-15T22:06:49.513 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:06:49.513 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj410 testdata.519518 2024-09-15T22:06:49.647 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:06:49.648 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj411 testdata.519518 2024-09-15T22:06:49.784 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:06:49.785 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj412 testdata.519518 2024-09-15T22:06:49.918 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:06:49.918 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj413 testdata.519518 2024-09-15T22:06:50.061 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:06:50.061 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj414 testdata.519518 2024-09-15T22:06:50.203 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:06:50.203 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj415 testdata.519518 2024-09-15T22:06:50.346 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:06:50.346 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj416 testdata.519518 2024-09-15T22:06:50.522 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:06:50.522 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj417 testdata.519518 2024-09-15T22:06:50.665 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:06:50.665 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj418 testdata.519518 2024-09-15T22:06:50.790 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:06:50.791 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj419 testdata.519518 2024-09-15T22:06:50.916 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:06:50.916 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj420 testdata.519518 2024-09-15T22:06:51.059 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:06:51.059 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj421 testdata.519518 2024-09-15T22:06:51.201 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:06:51.201 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj422 testdata.519518 2024-09-15T22:06:51.335 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:06:51.335 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj423 testdata.519518 2024-09-15T22:06:51.469 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:06:51.469 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj424 testdata.519518 2024-09-15T22:06:51.612 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:06:51.612 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj425 testdata.519518 2024-09-15T22:06:51.746 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:06:51.746 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj426 testdata.519518 2024-09-15T22:06:51.871 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:06:51.872 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj427 testdata.519518 2024-09-15T22:06:52.005 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:06:52.006 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj428 testdata.519518 2024-09-15T22:06:52.140 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:06:52.140 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj429 testdata.519518 2024-09-15T22:06:52.282 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:06:52.282 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj430 testdata.519518 2024-09-15T22:06:52.425 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:06:52.425 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj431 testdata.519518 2024-09-15T22:06:52.567 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:06:52.568 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj432 testdata.519518 2024-09-15T22:06:52.711 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:06:52.711 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj433 testdata.519518 2024-09-15T22:06:52.836 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:06:52.836 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj434 testdata.519518 2024-09-15T22:06:52.962 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:06:52.962 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj435 testdata.519518 2024-09-15T22:06:53.121 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:06:53.121 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj436 testdata.519518 2024-09-15T22:06:53.247 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:06:53.276 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj437 testdata.519518 2024-09-15T22:06:53.381 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:06:53.381 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj438 testdata.519518 2024-09-15T22:06:53.515 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:06:53.515 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj439 testdata.519518 2024-09-15T22:06:53.649 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:06:53.649 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj440 testdata.519518 2024-09-15T22:06:53.775 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:06:53.775 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj441 testdata.519518 2024-09-15T22:06:53.909 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:06:53.909 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj442 testdata.519518 2024-09-15T22:06:54.043 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:06:54.043 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj443 testdata.519518 2024-09-15T22:06:54.220 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:06:54.220 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj444 testdata.519518 2024-09-15T22:06:54.370 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:06:54.370 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj445 testdata.519518 2024-09-15T22:06:54.513 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:06:54.513 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj446 testdata.519518 2024-09-15T22:06:54.647 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:06:54.647 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj447 testdata.519518 2024-09-15T22:06:54.781 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:06:54.781 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj448 testdata.519518 2024-09-15T22:06:54.915 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:06:54.915 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj449 testdata.519518 2024-09-15T22:06:55.041 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:06:55.041 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj450 testdata.519518 2024-09-15T22:06:55.175 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:06:55.176 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj451 testdata.519518 2024-09-15T22:06:55.309 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:06:55.310 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj452 testdata.519518 2024-09-15T22:06:55.460 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:06:55.461 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj453 testdata.519518 2024-09-15T22:06:55.586 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:06:55.586 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj454 testdata.519518 2024-09-15T22:06:55.712 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:06:55.712 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj455 testdata.519518 2024-09-15T22:06:55.846 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:06:55.846 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj456 testdata.519518 2024-09-15T22:06:55.971 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:06:55.971 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj457 testdata.519518 2024-09-15T22:06:56.105 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:06:56.106 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj458 testdata.519518 2024-09-15T22:06:56.265 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:06:56.265 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj459 testdata.519518 2024-09-15T22:06:56.416 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:06:56.416 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj460 testdata.519518 2024-09-15T22:06:56.584 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:06:56.584 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj461 testdata.519518 2024-09-15T22:06:56.726 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:06:56.726 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj462 testdata.519518 2024-09-15T22:06:56.852 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:06:56.852 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj463 testdata.519518 2024-09-15T22:06:56.986 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:06:56.986 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj464 testdata.519518 2024-09-15T22:06:57.162 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:06:57.162 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj465 testdata.519518 2024-09-15T22:06:57.288 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:06:57.288 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj466 testdata.519518 2024-09-15T22:06:57.422 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:06:57.422 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj467 testdata.519518 2024-09-15T22:06:57.567 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:06:57.567 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj468 testdata.519518 2024-09-15T22:06:57.698 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:06:57.698 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj469 testdata.519518 2024-09-15T22:06:57.824 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:06:57.825 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj470 testdata.519518 2024-09-15T22:06:57.950 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:06:57.950 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj471 testdata.519518 2024-09-15T22:06:58.101 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:06:58.101 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj472 testdata.519518 2024-09-15T22:06:58.252 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:06:58.252 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj473 testdata.519518 2024-09-15T22:06:58.419 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:06:58.420 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj474 testdata.519518 2024-09-15T22:06:58.545 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:06:58.545 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj475 testdata.519518 2024-09-15T22:06:58.671 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:06:58.671 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj476 testdata.519518 2024-09-15T22:06:58.805 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:06:58.805 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj477 testdata.519518 2024-09-15T22:06:58.939 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:06:58.940 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj478 testdata.519518 2024-09-15T22:06:59.083 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:06:59.083 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj479 testdata.519518 2024-09-15T22:06:59.228 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:06:59.229 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj480 testdata.519518 2024-09-15T22:06:59.355 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:06:59.355 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj481 testdata.519518 2024-09-15T22:06:59.491 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:06:59.491 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj482 testdata.519518 2024-09-15T22:06:59.634 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:06:59.634 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj483 testdata.519518 2024-09-15T22:06:59.770 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:06:59.770 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj484 testdata.519518 2024-09-15T22:06:59.905 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:06:59.905 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj485 testdata.519518 2024-09-15T22:07:00.031 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:07:00.032 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj486 testdata.519518 2024-09-15T22:07:00.169 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:07:00.169 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj487 testdata.519518 2024-09-15T22:07:00.345 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:07:00.345 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj488 testdata.519518 2024-09-15T22:07:00.513 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:07:00.513 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj489 testdata.519518 2024-09-15T22:07:00.647 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:07:00.647 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj490 testdata.519518 2024-09-15T22:07:00.781 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:07:00.781 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj491 testdata.519518 2024-09-15T22:07:00.915 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:07:00.915 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj492 testdata.519518 2024-09-15T22:07:01.049 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:07:01.049 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj493 testdata.519518 2024-09-15T22:07:01.208 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:07:01.208 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj494 testdata.519518 2024-09-15T22:07:01.351 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:07:01.351 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj495 testdata.519518 2024-09-15T22:07:01.510 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:07:01.510 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj496 testdata.519518 2024-09-15T22:07:01.636 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:07:01.636 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj497 testdata.519518 2024-09-15T22:07:01.773 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:07:01.773 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj498 testdata.519518 2024-09-15T22:07:01.910 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:07:01.910 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj499 testdata.519518 2024-09-15T22:07:02.046 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:07:02.046 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj500 testdata.519518 2024-09-15T22:07:02.210 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:07:02.210 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj501 testdata.519518 2024-09-15T22:07:02.376 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:07:02.376 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj502 testdata.519518 2024-09-15T22:07:02.562 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:07:02.563 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj503 testdata.519518 2024-09-15T22:07:02.689 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:07:02.689 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj504 testdata.519518 2024-09-15T22:07:02.814 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:07:02.814 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj505 testdata.519518 2024-09-15T22:07:02.940 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:07:02.940 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj506 testdata.519518 2024-09-15T22:07:03.141 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:07:03.141 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj507 testdata.519518 2024-09-15T22:07:03.300 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:07:03.300 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj508 testdata.519518 2024-09-15T22:07:03.451 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:07:03.451 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj509 testdata.519518 2024-09-15T22:07:03.594 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:07:03.594 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj510 testdata.519518 2024-09-15T22:07:03.728 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:07:03.728 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj511 testdata.519518 2024-09-15T22:07:03.862 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:07:03.862 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj512 testdata.519518 2024-09-15T22:07:04.004 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:07:04.004 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj513 testdata.519518 2024-09-15T22:07:04.138 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:07:04.139 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj514 testdata.519518 2024-09-15T22:07:04.289 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:07:04.289 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj515 testdata.519518 2024-09-15T22:07:04.451 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:07:04.451 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj516 testdata.519518 2024-09-15T22:07:04.585 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:07:04.585 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj517 testdata.519518 2024-09-15T22:07:04.719 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:07:04.719 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj518 testdata.519518 2024-09-15T22:07:04.862 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:07:04.862 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj519 testdata.519518 2024-09-15T22:07:04.996 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:07:04.996 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj520 testdata.519518 2024-09-15T22:07:05.575 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:07:05.575 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj521 testdata.519518 2024-09-15T22:07:05.723 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:07:05.723 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj522 testdata.519518 2024-09-15T22:07:05.857 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:07:05.857 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj523 testdata.519518 2024-09-15T22:07:05.991 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:07:05.992 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj524 testdata.519518 2024-09-15T22:07:06.142 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:07:06.142 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj525 testdata.519518 2024-09-15T22:07:06.302 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:07:06.302 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj526 testdata.519518 2024-09-15T22:07:06.461 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:07:06.461 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj527 testdata.519518 2024-09-15T22:07:06.612 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:07:06.612 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj528 testdata.519518 2024-09-15T22:07:06.746 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:07:06.746 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj529 testdata.519518 2024-09-15T22:07:06.888 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:07:06.888 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj530 testdata.519518 2024-09-15T22:07:07.039 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:07:07.039 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj531 testdata.519518 2024-09-15T22:07:07.173 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:07:07.173 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj532 testdata.519518 2024-09-15T22:07:07.316 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:07:07.316 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj533 testdata.519518 2024-09-15T22:07:07.450 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:07:07.450 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj534 testdata.519518 2024-09-15T22:07:07.592 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:07:07.592 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj535 testdata.519518 2024-09-15T22:07:07.735 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:07:07.735 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj536 testdata.519518 2024-09-15T22:07:07.877 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:07:07.877 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj537 testdata.519518 2024-09-15T22:07:08.019 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:07:08.019 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj538 testdata.519518 2024-09-15T22:07:08.162 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:07:08.162 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj539 testdata.519518 2024-09-15T22:07:08.321 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:07:08.321 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj540 testdata.519518 2024-09-15T22:07:08.489 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:07:08.489 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj541 testdata.519518 2024-09-15T22:07:08.615 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:07:08.615 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj542 testdata.519518 2024-09-15T22:07:08.741 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:07:08.741 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj543 testdata.519518 2024-09-15T22:07:08.883 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:07:08.883 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj544 testdata.519518 2024-09-15T22:07:09.017 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:07:09.017 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj545 testdata.519518 2024-09-15T22:07:09.160 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:07:09.160 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj546 testdata.519518 2024-09-15T22:07:09.319 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:07:09.319 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj547 testdata.519518 2024-09-15T22:07:09.445 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:07:09.445 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj548 testdata.519518 2024-09-15T22:07:09.571 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:07:09.571 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj549 testdata.519518 2024-09-15T22:07:09.696 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:07:09.697 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj550 testdata.519518 2024-09-15T22:07:09.822 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:07:09.822 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj551 testdata.519518 2024-09-15T22:07:09.956 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:07:09.956 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj552 testdata.519518 2024-09-15T22:07:10.082 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:07:10.082 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj553 testdata.519518 2024-09-15T22:07:10.216 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:07:10.216 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj554 testdata.519518 2024-09-15T22:07:10.359 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:07:10.359 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj555 testdata.519518 2024-09-15T22:07:10.510 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:07:10.510 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj556 testdata.519518 2024-09-15T22:07:10.636 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:07:10.636 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj557 testdata.519518 2024-09-15T22:07:10.804 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:07:10.804 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj558 testdata.519518 2024-09-15T22:07:10.937 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:07:10.938 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj559 testdata.519518 2024-09-15T22:07:11.072 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:07:11.072 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj560 testdata.519518 2024-09-15T22:07:11.197 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:07:11.198 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj561 testdata.519518 2024-09-15T22:07:11.323 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:07:11.323 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj562 testdata.519518 2024-09-15T22:07:11.449 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:07:11.449 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj563 testdata.519518 2024-09-15T22:07:11.591 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:07:11.591 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj564 testdata.519518 2024-09-15T22:07:11.734 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:07:11.734 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj565 testdata.519518 2024-09-15T22:07:11.859 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:07:11.860 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj566 testdata.519518 2024-09-15T22:07:11.985 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:07:11.985 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj567 testdata.519518 2024-09-15T22:07:12.111 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:07:12.111 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj568 testdata.519518 2024-09-15T22:07:12.271 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:07:12.271 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj569 testdata.519518 2024-09-15T22:07:12.446 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:07:12.446 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj570 testdata.519518 2024-09-15T22:07:12.581 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:07:12.581 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj571 testdata.519518 2024-09-15T22:07:12.706 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:07:12.707 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj572 testdata.519518 2024-09-15T22:07:12.832 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:07:12.832 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj573 testdata.519518 2024-09-15T22:07:12.958 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:07:12.958 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj574 testdata.519518 2024-09-15T22:07:13.100 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:07:13.100 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj575 testdata.519518 2024-09-15T22:07:13.234 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:07:13.234 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj576 testdata.519518 2024-09-15T22:07:13.402 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:07:13.402 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj577 testdata.519518 2024-09-15T22:07:13.536 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:07:13.536 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj578 testdata.519518 2024-09-15T22:07:13.662 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:07:13.662 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj579 testdata.519518 2024-09-15T22:07:13.796 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:07:13.796 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj580 testdata.519518 2024-09-15T22:07:13.938 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:07:13.938 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj581 testdata.519518 2024-09-15T22:07:14.072 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:07:14.072 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj582 testdata.519518 2024-09-15T22:07:14.207 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:07:14.207 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj583 testdata.519518 2024-09-15T22:07:14.349 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:07:14.349 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj584 testdata.519518 2024-09-15T22:07:14.525 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:07:14.525 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj585 testdata.519518 2024-09-15T22:07:14.659 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:07:14.659 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj586 testdata.519518 2024-09-15T22:07:14.793 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:07:14.793 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj587 testdata.519518 2024-09-15T22:07:14.927 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:07:14.927 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj588 testdata.519518 2024-09-15T22:07:15.053 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:07:15.054 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj589 testdata.519518 2024-09-15T22:07:15.187 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:07:15.187 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj590 testdata.519518 2024-09-15T22:07:15.338 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:07:15.338 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj591 testdata.519518 2024-09-15T22:07:15.472 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:07:15.472 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj592 testdata.519518 2024-09-15T22:07:15.607 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:07:15.607 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj593 testdata.519518 2024-09-15T22:07:15.732 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:07:15.732 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj594 testdata.519518 2024-09-15T22:07:15.858 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:07:15.858 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj595 testdata.519518 2024-09-15T22:07:15.991 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:07:15.992 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj596 testdata.519518 2024-09-15T22:07:16.134 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:07:16.134 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj597 testdata.519518 2024-09-15T22:07:16.277 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:07:16.277 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj598 testdata.519518 2024-09-15T22:07:16.419 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:07:16.419 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj599 testdata.519518 2024-09-15T22:07:16.562 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:07:16.562 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj600 testdata.519518 2024-09-15T22:07:16.696 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:07:16.696 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj601 testdata.519518 2024-09-15T22:07:16.838 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:07:16.839 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj602 testdata.519518 2024-09-15T22:07:16.981 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:07:16.981 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj603 testdata.519518 2024-09-15T22:07:17.115 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:07:17.115 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj604 testdata.519518 2024-09-15T22:07:17.241 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:07:17.241 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj605 testdata.519518 2024-09-15T22:07:17.399 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:07:17.400 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj606 testdata.519518 2024-09-15T22:07:17.550 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:07:17.550 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj607 testdata.519518 2024-09-15T22:07:17.684 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:07:17.684 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj608 testdata.519518 2024-09-15T22:07:17.810 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:07:17.810 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj609 testdata.519518 2024-09-15T22:07:17.938 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:07:17.938 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj610 testdata.519518 2024-09-15T22:07:18.072 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:07:18.072 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj611 testdata.519518 2024-09-15T22:07:18.223 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:07:18.223 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj612 testdata.519518 2024-09-15T22:07:18.374 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:07:18.374 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj613 testdata.519518 2024-09-15T22:07:18.542 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:07:18.542 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj614 testdata.519518 2024-09-15T22:07:18.676 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:07:18.676 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj615 testdata.519518 2024-09-15T22:07:18.810 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:07:18.810 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj616 testdata.519518 2024-09-15T22:07:18.935 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:07:18.935 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj617 testdata.519518 2024-09-15T22:07:19.061 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:07:19.061 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj618 testdata.519518 2024-09-15T22:07:19.195 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:07:19.195 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj619 testdata.519518 2024-09-15T22:07:19.337 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:07:19.337 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj620 testdata.519518 2024-09-15T22:07:19.463 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:07:19.464 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj621 testdata.519518 2024-09-15T22:07:19.597 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:07:19.597 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj622 testdata.519518 2024-09-15T22:07:19.731 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:07:19.731 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj623 testdata.519518 2024-09-15T22:07:19.865 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:07:19.866 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj624 testdata.519518 2024-09-15T22:07:19.991 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:07:19.991 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj625 testdata.519518 2024-09-15T22:07:20.141 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:07:20.142 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj626 testdata.519518 2024-09-15T22:07:20.284 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:07:20.284 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj627 testdata.519518 2024-09-15T22:07:20.435 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:07:20.435 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj628 testdata.519518 2024-09-15T22:07:20.569 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:07:20.569 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj629 testdata.519518 2024-09-15T22:07:20.703 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:07:20.703 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj630 testdata.519518 2024-09-15T22:07:20.837 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:07:20.837 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj631 testdata.519518 2024-09-15T22:07:20.980 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:07:20.980 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj632 testdata.519518 2024-09-15T22:07:21.114 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:07:21.114 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj633 testdata.519518 2024-09-15T22:07:21.239 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:07:21.240 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj634 testdata.519518 2024-09-15T22:07:21.440 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:07:21.441 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj635 testdata.519518 2024-09-15T22:07:21.566 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:07:21.566 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj636 testdata.519518 2024-09-15T22:07:21.708 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:07:21.708 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj637 testdata.519518 2024-09-15T22:07:21.850 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:07:21.850 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj638 testdata.519518 2024-09-15T22:07:21.976 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:07:21.976 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj639 testdata.519518 2024-09-15T22:07:22.102 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:07:22.102 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj640 testdata.519518 2024-09-15T22:07:22.244 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:07:22.244 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj641 testdata.519518 2024-09-15T22:07:22.395 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:07:22.395 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj642 testdata.519518 2024-09-15T22:07:22.546 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:07:22.546 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj643 testdata.519518 2024-09-15T22:07:22.680 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:07:22.680 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj644 testdata.519518 2024-09-15T22:07:22.822 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:07:22.822 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj645 testdata.519518 2024-09-15T22:07:22.973 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:07:22.973 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj646 testdata.519518 2024-09-15T22:07:23.107 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:07:23.107 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj647 testdata.519518 2024-09-15T22:07:23.233 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:07:23.233 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj648 testdata.519518 2024-09-15T22:07:23.367 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:07:23.367 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj649 testdata.519518 2024-09-15T22:07:23.501 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:07:23.501 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj650 testdata.519518 2024-09-15T22:07:23.634 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:07:23.635 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj651 testdata.519518 2024-09-15T22:07:23.769 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:07:23.769 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj652 testdata.519518 2024-09-15T22:07:23.894 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:07:23.895 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj653 testdata.519518 2024-09-15T22:07:24.028 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:07:24.028 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj654 testdata.519518 2024-09-15T22:07:24.171 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:07:24.171 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj655 testdata.519518 2024-09-15T22:07:24.313 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:07:24.314 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj656 testdata.519518 2024-09-15T22:07:24.489 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:07:24.490 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj657 testdata.519518 2024-09-15T22:07:24.624 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:07:24.624 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj658 testdata.519518 2024-09-15T22:07:24.749 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:07:24.750 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj659 testdata.519518 2024-09-15T22:07:24.875 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:07:24.875 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj660 testdata.519518 2024-09-15T22:07:25.001 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:07:25.001 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj661 testdata.519518 2024-09-15T22:07:25.134 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:07:25.135 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj662 testdata.519518 2024-09-15T22:07:25.260 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:07:25.261 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj663 testdata.519518 2024-09-15T22:07:25.413 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:07:25.413 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj664 testdata.519518 2024-09-15T22:07:25.554 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:07:25.554 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj665 testdata.519518 2024-09-15T22:07:25.696 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:07:25.696 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj666 testdata.519518 2024-09-15T22:07:25.838 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:07:25.838 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj667 testdata.519518 2024-09-15T22:07:25.989 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:07:25.989 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj668 testdata.519518 2024-09-15T22:07:26.123 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:07:26.123 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj669 testdata.519518 2024-09-15T22:07:26.274 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:07:26.274 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj670 testdata.519518 2024-09-15T22:07:26.433 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:07:26.433 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj671 testdata.519518 2024-09-15T22:07:26.567 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:07:26.567 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj672 testdata.519518 2024-09-15T22:07:26.701 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:07:26.701 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj673 testdata.519518 2024-09-15T22:07:26.826 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:07:26.827 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj674 testdata.519518 2024-09-15T22:07:26.952 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:07:26.952 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj675 testdata.519518 2024-09-15T22:07:27.086 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:07:27.086 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj676 testdata.519518 2024-09-15T22:07:27.228 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:07:27.229 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj677 testdata.519518 2024-09-15T22:07:27.380 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:07:27.380 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj678 testdata.519518 2024-09-15T22:07:27.514 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:07:27.514 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj679 testdata.519518 2024-09-15T22:07:27.656 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:07:27.656 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj680 testdata.519518 2024-09-15T22:07:27.782 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:07:27.782 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj681 testdata.519518 2024-09-15T22:07:27.907 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:07:27.907 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj682 testdata.519518 2024-09-15T22:07:28.041 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:07:28.042 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj683 testdata.519518 2024-09-15T22:07:28.167 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:07:28.167 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj684 testdata.519518 2024-09-15T22:07:28.343 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:07:28.343 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj685 testdata.519518 2024-09-15T22:07:28.511 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:07:28.511 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj686 testdata.519518 2024-09-15T22:07:28.645 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:07:28.645 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj687 testdata.519518 2024-09-15T22:07:28.778 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:07:28.779 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj688 testdata.519518 2024-09-15T22:07:28.913 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:07:28.913 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj689 testdata.519518 2024-09-15T22:07:29.055 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:07:29.055 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj690 testdata.519518 2024-09-15T22:07:29.189 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:07:29.190 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj691 testdata.519518 2024-09-15T22:07:29.315 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:07:29.315 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj692 testdata.519518 2024-09-15T22:07:29.466 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:07:29.466 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj693 testdata.519518 2024-09-15T22:07:29.608 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:07:29.608 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj694 testdata.519518 2024-09-15T22:07:29.742 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:07:29.742 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj695 testdata.519518 2024-09-15T22:07:29.867 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:07:29.868 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj696 testdata.519518 2024-09-15T22:07:29.993 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:07:29.993 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj697 testdata.519518 2024-09-15T22:07:30.119 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:07:30.119 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj698 testdata.519518 2024-09-15T22:07:30.261 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:07:30.262 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj699 testdata.519518 2024-09-15T22:07:30.430 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:07:30.430 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj700 testdata.519518 2024-09-15T22:07:30.580 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:07:30.581 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj701 testdata.519518 2024-09-15T22:07:30.714 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:07:30.715 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj702 testdata.519518 2024-09-15T22:07:30.848 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:07:30.849 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj703 testdata.519518 2024-09-15T22:07:30.982 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:07:30.983 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj704 testdata.519518 2024-09-15T22:07:31.116 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:07:31.116 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj705 testdata.519518 2024-09-15T22:07:31.242 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:07:31.242 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj706 testdata.519518 2024-09-15T22:07:31.367 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:07:31.368 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj707 testdata.519518 2024-09-15T22:07:31.493 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:07:31.493 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj708 testdata.519518 2024-09-15T22:07:31.627 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:07:31.627 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj709 testdata.519518 2024-09-15T22:07:31.761 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:07:31.761 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj710 testdata.519518 2024-09-15T22:07:31.895 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:07:31.895 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj711 testdata.519518 2024-09-15T22:07:32.037 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:07:32.038 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj712 testdata.519518 2024-09-15T22:07:32.189 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:07:32.189 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj713 testdata.519518 2024-09-15T22:07:32.348 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:07:32.348 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj714 testdata.519518 2024-09-15T22:07:32.515 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:07:32.515 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj715 testdata.519518 2024-09-15T22:07:32.649 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:07:32.649 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj716 testdata.519518 2024-09-15T22:07:32.783 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:07:32.783 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj717 testdata.519518 2024-09-15T22:07:32.909 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:07:32.909 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj718 testdata.519518 2024-09-15T22:07:33.043 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:07:33.043 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj719 testdata.519518 2024-09-15T22:07:33.169 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:07:33.169 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj720 testdata.519518 2024-09-15T22:07:33.311 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:07:33.311 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj721 testdata.519518 2024-09-15T22:07:33.445 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:07:33.445 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj722 testdata.519518 2024-09-15T22:07:33.596 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:07:33.596 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj723 testdata.519518 2024-09-15T22:07:33.730 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:07:33.730 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj724 testdata.519518 2024-09-15T22:07:33.864 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:07:33.864 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj725 testdata.519518 2024-09-15T22:07:33.990 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:07:33.990 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj726 testdata.519518 2024-09-15T22:07:34.140 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:07:34.140 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj727 testdata.519518 2024-09-15T22:07:34.282 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:07:34.283 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj728 testdata.519518 2024-09-15T22:07:34.417 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:07:34.417 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj729 testdata.519518 2024-09-15T22:07:34.543 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:07:34.543 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj730 testdata.519518 2024-09-15T22:07:34.677 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:07:34.677 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj731 testdata.519518 2024-09-15T22:07:34.811 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:07:34.811 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj732 testdata.519518 2024-09-15T22:07:34.936 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:07:34.937 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj733 testdata.519518 2024-09-15T22:07:35.079 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:07:35.079 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj734 testdata.519518 2024-09-15T22:07:35.213 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:07:35.213 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj735 testdata.519518 2024-09-15T22:07:35.347 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:07:35.347 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj736 testdata.519518 2024-09-15T22:07:35.489 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:07:35.489 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj737 testdata.519518 2024-09-15T22:07:35.632 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:07:35.632 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj738 testdata.519518 2024-09-15T22:07:35.791 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:07:35.791 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj739 testdata.519518 2024-09-15T22:07:35.925 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:07:35.925 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj740 testdata.519518 2024-09-15T22:07:36.050 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:07:36.050 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj741 testdata.519518 2024-09-15T22:07:36.180 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:07:36.181 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj742 testdata.519518 2024-09-15T22:07:36.672 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:07:36.673 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj743 testdata.519518 2024-09-15T22:07:36.920 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:07:36.920 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj744 testdata.519518 2024-09-15T22:07:37.071 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:07:37.071 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj745 testdata.519518 2024-09-15T22:07:37.222 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:07:37.222 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj746 testdata.519518 2024-09-15T22:07:37.364 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:07:37.364 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj747 testdata.519518 2024-09-15T22:07:37.540 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:07:37.540 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj748 testdata.519518 2024-09-15T22:07:37.682 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:07:37.682 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj749 testdata.519518 2024-09-15T22:07:37.825 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:07:37.825 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj750 testdata.519518 2024-09-15T22:07:37.967 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:07:37.967 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj751 testdata.519518 2024-09-15T22:07:38.109 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:07:38.109 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj752 testdata.519518 2024-09-15T22:07:38.260 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:07:38.260 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj753 testdata.519518 2024-09-15T22:07:38.427 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:07:38.428 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj754 testdata.519518 2024-09-15T22:07:38.570 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:07:38.570 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj755 testdata.519518 2024-09-15T22:07:38.713 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:07:38.713 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj756 testdata.519518 2024-09-15T22:07:38.863 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:07:38.863 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj757 testdata.519518 2024-09-15T22:07:39.014 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:07:39.014 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj758 testdata.519518 2024-09-15T22:07:39.165 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:07:39.165 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj759 testdata.519518 2024-09-15T22:07:39.315 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:07:39.315 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj760 testdata.519518 2024-09-15T22:07:39.466 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:07:39.466 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj761 testdata.519518 2024-09-15T22:07:39.608 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:07:39.608 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj762 testdata.519518 2024-09-15T22:07:39.751 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:07:39.751 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj763 testdata.519518 2024-09-15T22:07:39.885 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:07:39.885 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj764 testdata.519518 2024-09-15T22:07:40.027 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:07:40.027 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj765 testdata.519518 2024-09-15T22:07:40.178 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:07:40.178 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj766 testdata.519518 2024-09-15T22:07:40.337 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:07:40.337 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj767 testdata.519518 2024-09-15T22:07:40.530 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:07:40.530 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj768 testdata.519518 2024-09-15T22:07:40.680 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:07:40.680 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj769 testdata.519518 2024-09-15T22:07:40.823 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:07:40.823 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj770 testdata.519518 2024-09-15T22:07:40.956 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:07:40.956 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj771 testdata.519518 2024-09-15T22:07:41.099 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:07:41.099 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj772 testdata.519518 2024-09-15T22:07:41.241 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:07:41.241 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj773 testdata.519518 2024-09-15T22:07:41.367 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:07:41.367 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj774 testdata.519518 2024-09-15T22:07:41.501 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:07:41.501 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj775 testdata.519518 2024-09-15T22:07:41.651 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:07:41.651 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj776 testdata.519518 2024-09-15T22:07:41.785 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:07:41.785 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj777 testdata.519518 2024-09-15T22:07:41.936 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:07:41.936 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj778 testdata.519518 2024-09-15T22:07:42.078 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:07:42.078 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj779 testdata.519518 2024-09-15T22:07:42.221 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:07:42.221 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj780 testdata.519518 2024-09-15T22:07:42.372 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:07:42.372 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj781 testdata.519518 2024-09-15T22:07:42.548 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:07:42.548 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj782 testdata.519518 2024-09-15T22:07:42.707 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:07:42.707 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj783 testdata.519518 2024-09-15T22:07:42.841 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:07:42.841 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj784 testdata.519518 2024-09-15T22:07:42.966 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:07:42.966 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj785 testdata.519518 2024-09-15T22:07:43.092 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:07:43.092 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj786 testdata.519518 2024-09-15T22:07:43.218 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:07:43.218 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj787 testdata.519518 2024-09-15T22:07:43.352 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:07:43.352 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj788 testdata.519518 2024-09-15T22:07:43.486 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:07:43.486 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj789 testdata.519518 2024-09-15T22:07:43.620 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:07:43.620 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj790 testdata.519518 2024-09-15T22:07:43.746 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:07:43.746 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj791 testdata.519518 2024-09-15T22:07:43.879 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:07:43.880 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj792 testdata.519518 2024-09-15T22:07:44.014 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:07:44.014 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj793 testdata.519518 2024-09-15T22:07:44.173 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:07:44.173 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj794 testdata.519518 2024-09-15T22:07:44.332 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:07:44.332 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj795 testdata.519518 2024-09-15T22:07:44.500 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:07:44.500 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj796 testdata.519518 2024-09-15T22:07:44.625 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:07:44.626 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj797 testdata.519518 2024-09-15T22:07:44.760 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:07:44.760 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj798 testdata.519518 2024-09-15T22:07:44.902 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:07:44.902 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj799 testdata.519518 2024-09-15T22:07:45.036 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:07:45.036 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj800 testdata.519518 2024-09-15T22:07:45.178 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:07:45.179 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj801 testdata.519518 2024-09-15T22:07:45.337 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:07:45.337 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj802 testdata.519518 2024-09-15T22:07:45.480 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:07:45.480 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj803 testdata.519518 2024-09-15T22:07:45.639 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:07:45.639 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj804 testdata.519518 2024-09-15T22:07:45.773 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:07:45.773 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj805 testdata.519518 2024-09-15T22:07:45.915 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:07:45.915 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj806 testdata.519518 2024-09-15T22:07:46.041 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:07:46.041 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj807 testdata.519518 2024-09-15T22:07:46.166 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:07:46.167 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj808 testdata.519518 2024-09-15T22:07:46.342 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:07:46.343 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj809 testdata.519518 2024-09-15T22:07:46.510 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:07:46.510 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj810 testdata.519518 2024-09-15T22:07:46.644 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:07:46.644 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj811 testdata.519518 2024-09-15T22:07:46.778 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:07:46.779 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj812 testdata.519518 2024-09-15T22:07:46.929 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:07:46.929 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj813 testdata.519518 2024-09-15T22:07:47.079 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:07:47.080 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj814 testdata.519518 2024-09-15T22:07:47.214 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:07:47.214 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj815 testdata.519518 2024-09-15T22:07:47.356 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:07:47.356 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj816 testdata.519518 2024-09-15T22:07:47.490 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:07:47.490 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj817 testdata.519518 2024-09-15T22:07:47.624 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:07:47.624 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj818 testdata.519518 2024-09-15T22:07:47.774 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:07:47.774 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj819 testdata.519518 2024-09-15T22:07:47.901 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:07:47.901 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj820 testdata.519518 2024-09-15T22:07:48.026 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:07:48.026 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj821 testdata.519518 2024-09-15T22:07:48.160 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:07:48.160 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj822 testdata.519518 2024-09-15T22:07:48.353 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:07:48.353 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj823 testdata.519518 2024-09-15T22:07:48.512 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:07:48.513 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj824 testdata.519518 2024-09-15T22:07:48.647 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:07:48.647 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj825 testdata.519518 2024-09-15T22:07:48.781 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:07:48.781 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj826 testdata.519518 2024-09-15T22:07:48.922 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:07:48.923 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj827 testdata.519518 2024-09-15T22:07:49.057 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:07:49.057 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj828 testdata.519518 2024-09-15T22:07:49.191 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:07:49.191 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj829 testdata.519518 2024-09-15T22:07:49.316 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:07:49.317 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj830 testdata.519518 2024-09-15T22:07:49.442 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:07:49.442 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj831 testdata.519518 2024-09-15T22:07:49.601 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:07:49.601 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj832 testdata.519518 2024-09-15T22:07:49.743 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:07:49.744 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj833 testdata.519518 2024-09-15T22:07:49.878 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:07:49.878 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj834 testdata.519518 2024-09-15T22:07:50.020 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:07:50.020 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj835 testdata.519518 2024-09-15T22:07:50.146 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:07:50.146 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj836 testdata.519518 2024-09-15T22:07:50.313 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:07:50.313 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj837 testdata.519518 2024-09-15T22:07:50.447 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:07:50.448 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj838 testdata.519518 2024-09-15T22:07:50.581 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:07:50.582 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj839 testdata.519518 2024-09-15T22:07:50.715 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:07:50.715 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj840 testdata.519518 2024-09-15T22:07:50.858 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:07:50.858 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj841 testdata.519518 2024-09-15T22:07:50.992 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:07:50.992 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj842 testdata.519518 2024-09-15T22:07:51.117 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:07:51.117 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj843 testdata.519518 2024-09-15T22:07:51.251 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:07:51.251 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj844 testdata.519518 2024-09-15T22:07:51.394 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:07:51.394 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj845 testdata.519518 2024-09-15T22:07:51.536 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:07:51.536 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj846 testdata.519518 2024-09-15T22:07:51.670 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:07:51.670 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj847 testdata.519518 2024-09-15T22:07:51.813 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:07:51.813 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj848 testdata.519518 2024-09-15T22:07:51.955 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:07:51.955 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj849 testdata.519518 2024-09-15T22:07:52.097 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:07:52.097 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj850 testdata.519518 2024-09-15T22:07:52.248 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:07:52.248 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj851 testdata.519518 2024-09-15T22:07:52.390 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:07:52.391 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj852 testdata.519518 2024-09-15T22:07:52.550 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:07:52.550 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj853 testdata.519518 2024-09-15T22:07:52.684 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:07:52.684 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj854 testdata.519518 2024-09-15T22:07:52.818 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:07:52.818 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj855 testdata.519518 2024-09-15T22:07:52.952 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:07:52.952 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj856 testdata.519518 2024-09-15T22:07:53.103 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:07:53.103 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj857 testdata.519518 2024-09-15T22:07:53.237 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:07:53.237 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj858 testdata.519518 2024-09-15T22:07:53.379 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:07:53.379 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj859 testdata.519518 2024-09-15T22:07:53.513 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:07:53.513 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj860 testdata.519518 2024-09-15T22:07:53.689 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:07:53.689 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj861 testdata.519518 2024-09-15T22:07:53.814 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:07:53.815 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj862 testdata.519518 2024-09-15T22:07:53.949 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:07:53.949 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj863 testdata.519518 2024-09-15T22:07:54.082 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:07:54.082 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj864 testdata.519518 2024-09-15T22:07:54.237 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:07:54.237 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj865 testdata.519518 2024-09-15T22:07:54.393 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:07:54.393 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj866 testdata.519518 2024-09-15T22:07:54.569 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:07:54.569 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj867 testdata.519518 2024-09-15T22:07:54.712 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:07:54.712 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj868 testdata.519518 2024-09-15T22:07:54.837 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:07:54.837 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj869 testdata.519518 2024-09-15T22:07:54.979 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:07:54.980 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj870 testdata.519518 2024-09-15T22:07:55.122 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:07:55.122 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj871 testdata.519518 2024-09-15T22:07:55.272 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:07:55.272 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj872 testdata.519518 2024-09-15T22:07:55.406 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:07:55.406 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj873 testdata.519518 2024-09-15T22:07:55.532 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:07:55.532 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj874 testdata.519518 2024-09-15T22:07:55.674 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:07:55.674 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj875 testdata.519518 2024-09-15T22:07:55.825 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:07:55.825 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj876 testdata.519518 2024-09-15T22:07:55.959 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:07:55.959 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj877 testdata.519518 2024-09-15T22:07:56.101 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:07:56.101 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj878 testdata.519518 2024-09-15T22:07:56.244 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:07:56.244 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj879 testdata.519518 2024-09-15T22:07:56.411 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:07:56.412 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj880 testdata.519518 2024-09-15T22:07:56.612 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:07:56.613 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj881 testdata.519518 2024-09-15T22:07:56.747 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:07:56.747 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj882 testdata.519518 2024-09-15T22:07:56.880 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:07:56.880 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj883 testdata.519518 2024-09-15T22:07:57.014 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:07:57.014 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj884 testdata.519518 2024-09-15T22:07:57.148 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:07:57.148 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj885 testdata.519518 2024-09-15T22:07:57.291 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:07:57.291 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj886 testdata.519518 2024-09-15T22:07:57.433 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:07:57.433 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj887 testdata.519518 2024-09-15T22:07:57.567 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:07:57.567 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj888 testdata.519518 2024-09-15T22:07:57.726 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:07:57.726 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj889 testdata.519518 2024-09-15T22:07:57.860 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:07:57.860 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj890 testdata.519518 2024-09-15T22:07:57.986 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:07:57.986 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj891 testdata.519518 2024-09-15T22:07:58.112 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:07:58.112 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj892 testdata.519518 2024-09-15T22:07:58.262 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:07:58.263 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj893 testdata.519518 2024-09-15T22:07:58.430 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:07:58.430 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj894 testdata.519518 2024-09-15T22:07:58.556 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:07:58.556 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj895 testdata.519518 2024-09-15T22:07:58.681 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:07:58.681 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj896 testdata.519518 2024-09-15T22:07:58.816 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:07:58.816 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj897 testdata.519518 2024-09-15T22:07:58.958 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:07:58.958 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj898 testdata.519518 2024-09-15T22:07:59.092 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:07:59.092 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj899 testdata.519518 2024-09-15T22:07:59.226 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:07:59.226 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj900 testdata.519518 2024-09-15T22:07:59.360 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:07:59.360 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj901 testdata.519518 2024-09-15T22:07:59.502 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:07:59.503 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj902 testdata.519518 2024-09-15T22:07:59.637 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:07:59.637 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj903 testdata.519518 2024-09-15T22:07:59.762 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:07:59.762 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj904 testdata.519518 2024-09-15T22:07:59.896 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:07:59.896 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj905 testdata.519518 2024-09-15T22:08:00.047 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:08:00.047 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj906 testdata.519518 2024-09-15T22:08:00.181 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:08:00.181 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj907 testdata.519518 2024-09-15T22:08:00.340 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:08:00.340 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj908 testdata.519518 2024-09-15T22:08:00.508 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:08:00.508 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj909 testdata.519518 2024-09-15T22:08:00.642 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:08:00.642 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj910 testdata.519518 2024-09-15T22:08:00.776 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:08:00.776 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj911 testdata.519518 2024-09-15T22:08:00.910 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:08:00.910 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj912 testdata.519518 2024-09-15T22:08:01.036 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:08:01.036 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj913 testdata.519518 2024-09-15T22:08:01.162 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:08:01.162 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj914 testdata.519518 2024-09-15T22:08:01.304 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:08:01.304 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj915 testdata.519518 2024-09-15T22:08:01.438 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:08:01.438 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj916 testdata.519518 2024-09-15T22:08:01.572 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:08:01.572 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj917 testdata.519518 2024-09-15T22:08:01.722 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:08:01.723 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj918 testdata.519518 2024-09-15T22:08:01.848 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:08:01.848 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj919 testdata.519518 2024-09-15T22:08:01.974 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:08:01.974 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj920 testdata.519518 2024-09-15T22:08:02.116 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:08:02.116 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj921 testdata.519518 2024-09-15T22:08:02.258 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:08:02.259 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj922 testdata.519518 2024-09-15T22:08:02.426 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:08:02.426 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj923 testdata.519518 2024-09-15T22:08:02.561 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:08:02.561 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj924 testdata.519518 2024-09-15T22:08:02.694 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:08:02.695 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj925 testdata.519518 2024-09-15T22:08:02.828 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:08:02.829 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj926 testdata.519518 2024-09-15T22:08:02.971 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:08:02.971 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj927 testdata.519518 2024-09-15T22:08:03.130 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:08:03.130 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj928 testdata.519518 2024-09-15T22:08:03.280 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:08:03.281 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj929 testdata.519518 2024-09-15T22:08:03.440 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:08:03.440 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj930 testdata.519518 2024-09-15T22:08:03.573 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:08:03.574 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj931 testdata.519518 2024-09-15T22:08:03.724 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:08:03.724 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj932 testdata.519518 2024-09-15T22:08:03.858 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:08:03.858 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj933 testdata.519518 2024-09-15T22:08:03.992 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:08:03.992 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj934 testdata.519518 2024-09-15T22:08:04.126 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:08:04.126 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj935 testdata.519518 2024-09-15T22:08:04.277 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:08:04.277 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj936 testdata.519518 2024-09-15T22:08:04.453 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:08:04.453 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj937 testdata.519518 2024-09-15T22:08:04.595 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:08:04.595 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj938 testdata.519518 2024-09-15T22:08:04.730 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:08:04.730 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj939 testdata.519518 2024-09-15T22:08:04.872 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:08:04.872 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj940 testdata.519518 2024-09-15T22:08:04.997 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:08:04.997 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj941 testdata.519518 2024-09-15T22:08:05.123 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:08:05.123 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj942 testdata.519518 2024-09-15T22:08:05.249 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:08:05.249 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj943 testdata.519518 2024-09-15T22:08:05.416 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:08:05.416 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj944 testdata.519518 2024-09-15T22:08:05.558 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:08:05.558 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj945 testdata.519518 2024-09-15T22:08:05.692 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:08:05.692 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj946 testdata.519518 2024-09-15T22:08:05.868 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:08:05.868 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj947 testdata.519518 2024-09-15T22:08:06.010 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:08:06.010 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj948 testdata.519518 2024-09-15T22:08:06.136 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:08:06.136 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj949 testdata.519518 2024-09-15T22:08:06.286 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:08:06.287 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj950 testdata.519518 2024-09-15T22:08:06.437 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:08:06.437 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj951 testdata.519518 2024-09-15T22:08:06.571 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:08:06.572 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj952 testdata.519518 2024-09-15T22:08:06.719 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:08:06.719 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj953 testdata.519518 2024-09-15T22:08:06.848 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:08:06.848 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj954 testdata.519518 2024-09-15T22:08:06.973 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:08:06.973 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj955 testdata.519518 2024-09-15T22:08:07.108 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:08:07.108 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj956 testdata.519518 2024-09-15T22:08:07.233 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:08:07.233 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj957 testdata.519518 2024-09-15T22:08:07.359 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:08:07.359 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj958 testdata.519518 2024-09-15T22:08:07.635 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:08:07.635 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj959 testdata.519518 2024-09-15T22:08:08.072 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:08:08.072 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj960 testdata.519518 2024-09-15T22:08:08.222 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:08:08.222 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj961 testdata.519518 2024-09-15T22:08:08.373 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:08:08.373 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj962 testdata.519518 2024-09-15T22:08:08.557 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:08:08.557 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj963 testdata.519518 2024-09-15T22:08:08.700 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:08:08.700 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj964 testdata.519518 2024-09-15T22:08:08.833 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:08:08.834 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj965 testdata.519518 2024-09-15T22:08:08.976 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:08:08.976 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj966 testdata.519518 2024-09-15T22:08:09.127 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:08:09.127 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj967 testdata.519518 2024-09-15T22:08:09.269 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:08:09.269 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj968 testdata.519518 2024-09-15T22:08:09.411 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:08:09.412 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj969 testdata.519518 2024-09-15T22:08:09.571 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:08:09.571 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj970 testdata.519518 2024-09-15T22:08:09.705 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:08:09.705 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj971 testdata.519518 2024-09-15T22:08:09.855 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:08:09.856 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj972 testdata.519518 2024-09-15T22:08:09.981 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:08:09.981 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj973 testdata.519518 2024-09-15T22:08:10.115 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:08:10.115 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj974 testdata.519518 2024-09-15T22:08:10.249 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:08:10.249 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj975 testdata.519518 2024-09-15T22:08:10.392 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:08:10.392 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj976 testdata.519518 2024-09-15T22:08:10.551 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:08:10.552 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj977 testdata.519518 2024-09-15T22:08:10.677 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:08:10.677 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj978 testdata.519518 2024-09-15T22:08:10.802 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:08:10.803 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj979 testdata.519518 2024-09-15T22:08:10.937 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:08:10.937 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj980 testdata.519518 2024-09-15T22:08:11.079 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:08:11.079 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj981 testdata.519518 2024-09-15T22:08:11.213 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:08:11.214 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj982 testdata.519518 2024-09-15T22:08:11.339 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:08:11.340 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj983 testdata.519518 2024-09-15T22:08:11.465 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:08:11.465 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj984 testdata.519518 2024-09-15T22:08:11.599 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:08:11.599 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj985 testdata.519518 2024-09-15T22:08:11.725 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:08:11.725 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj986 testdata.519518 2024-09-15T22:08:11.868 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:08:11.868 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj987 testdata.519518 2024-09-15T22:08:12.001 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:08:12.001 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj988 testdata.519518 2024-09-15T22:08:12.126 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:08:12.127 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj989 testdata.519518 2024-09-15T22:08:12.286 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:08:12.286 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj990 testdata.519518 2024-09-15T22:08:12.446 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:08:12.446 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj991 testdata.519518 2024-09-15T22:08:12.579 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:08:12.580 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj992 testdata.519518 2024-09-15T22:08:12.714 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:08:12.714 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj993 testdata.519518 2024-09-15T22:08:12.839 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:08:12.840 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj994 testdata.519518 2024-09-15T22:08:12.982 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:08:12.982 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj995 testdata.519518 2024-09-15T22:08:13.124 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:08:13.124 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj996 testdata.519518 2024-09-15T22:08:13.250 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:08:13.250 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj997 testdata.519518 2024-09-15T22:08:13.375 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:08:13.376 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj998 testdata.519518 2024-09-15T22:08:13.501 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:08:13.501 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj999 testdata.519518 2024-09-15T22:08:13.627 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-15T22:08:13.628 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj1000 testdata.519518 2024-09-15T22:08:13.753 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:341: _scrub_abort: rm -f testdata.519518 2024-09-15T22:08:13.755 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:343: _scrub_abort: get_primary test obj1 2024-09-15T22:08:13.755 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1187: get_primary: local poolname=test 2024-09-15T22:08:13.755 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1188: get_primary: local objectname=obj1 2024-09-15T22:08:13.755 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1190: get_primary: ceph --format json osd map test obj1 2024-09-15T22:08:13.755 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1191: get_primary: jq .acting_primary 2024-09-15T22:08:14.029 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:343: _scrub_abort: local primary=1 2024-09-15T22:08:14.029 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:344: _scrub_abort: local pgid=1.0 2024-09-15T22:08:14.029 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:346: _scrub_abort: ceph tell 1.0 schedule-deep-scrub 2024-09-15T22:08:14.124 INFO:tasks.workunit.client.0.smithi136.stdout:{ 2024-09-15T22:08:14.124 INFO:tasks.workunit.client.0.smithi136.stdout: "deep": true, 2024-09-15T22:08:14.124 INFO:tasks.workunit.client.0.smithi136.stdout: "must": false, 2024-09-15T22:08:14.124 INFO:tasks.workunit.client.0.smithi136.stdout: "stamp": "2024-09-01T22:06:34.123489+0000" 2024-09-15T22:08:14.124 INFO:tasks.workunit.client.0.smithi136.stdout:} 2024-09-15T22:08:14.133 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:349: _scrub_abort: set -o pipefail 2024-09-15T22:08:14.133 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:350: _scrub_abort: found=no 2024-09-15T22:08:14.133 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:351: _scrub_abort: seq 0 200 2024-09-15T22:08:14.134 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:351: _scrub_abort: for i in $(seq 0 200) 2024-09-15T22:08:14.134 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:353: _scrub_abort: flush_pg_stats 2024-09-15T22:08:14.135 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-15T22:08:14.135 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-15T22:08:14.410 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-15T22:08:14.410 INFO:tasks.workunit.client.0.smithi136.stderr:1 2024-09-15T22:08:14.410 INFO:tasks.workunit.client.0.smithi136.stderr:2' 2024-09-15T22:08:14.410 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-15T22:08:14.410 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T22:08:14.410 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-15T22:08:14.510 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836517 2024-09-15T22:08:14.510 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836517 2024-09-15T22:08:14.510 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836517' 2024-09-15T22:08:14.511 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T22:08:14.511 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-15T22:08:14.611 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=42949672994 2024-09-15T22:08:14.612 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 42949672994 2024-09-15T22:08:14.612 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836517 1-42949672994' 2024-09-15T22:08:14.612 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T22:08:14.612 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-15T22:08:14.713 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=64424509472 2024-09-15T22:08:14.713 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 64424509472 2024-09-15T22:08:14.713 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836517 1-42949672994 2-64424509472' 2024-09-15T22:08:14.713 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T22:08:14.713 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836517 2024-09-15T22:08:14.714 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T22:08:14.715 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-15T22:08:14.715 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836517 2024-09-15T22:08:14.716 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T22:08:14.716 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836517 2024-09-15T22:08:14.716 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836517' 2024-09-15T22:08:14.717 INFO:tasks.workunit.client.0.smithi136.stdout:waiting osd.0 seq 21474836517 2024-09-15T22:08:14.717 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T22:08:14.987 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836516 -lt 21474836517 2024-09-15T22:08:14.987 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-15T22:08:15.988 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-15T22:08:15.989 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T22:08:16.274 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836517 -lt 21474836517 2024-09-15T22:08:16.274 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T22:08:16.275 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-42949672994 2024-09-15T22:08:16.275 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T22:08:16.275 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-15T22:08:16.276 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-42949672994 2024-09-15T22:08:16.276 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T22:08:16.277 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=42949672994 2024-09-15T22:08:16.277 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 42949672994' 2024-09-15T22:08:16.277 INFO:tasks.workunit.client.0.smithi136.stdout:waiting osd.1 seq 42949672994 2024-09-15T22:08:16.277 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-15T22:08:16.569 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 42949672995 -lt 42949672994 2024-09-15T22:08:16.569 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T22:08:16.569 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-64424509472 2024-09-15T22:08:16.569 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T22:08:16.569 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-15T22:08:16.569 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-64424509472 2024-09-15T22:08:16.569 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T22:08:16.569 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=64424509472 2024-09-15T22:08:16.569 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 64424509472' 2024-09-15T22:08:16.570 INFO:tasks.workunit.client.0.smithi136.stdout:waiting osd.2 seq 64424509472 2024-09-15T22:08:16.570 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-15T22:08:16.843 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 64424509472 -lt 64424509472 2024-09-15T22:08:16.843 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:354: _scrub_abort: ceph pg dump pgs 2024-09-15T22:08:16.843 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:354: _scrub_abort: grep '^1.0' 2024-09-15T22:08:16.843 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:354: _scrub_abort: grep -q scrubbing 2024-09-15T22:08:17.101 INFO:tasks.workunit.client.0.smithi136.stderr:dumped pgs 2024-09-15T22:08:17.112 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:351: _scrub_abort: for i in $(seq 0 200) 2024-09-15T22:08:17.113 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:353: _scrub_abort: flush_pg_stats 2024-09-15T22:08:17.113 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-15T22:08:17.113 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-15T22:08:17.387 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-15T22:08:17.387 INFO:tasks.workunit.client.0.smithi136.stderr:1 2024-09-15T22:08:17.387 INFO:tasks.workunit.client.0.smithi136.stderr:2' 2024-09-15T22:08:17.387 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-15T22:08:17.387 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T22:08:17.387 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-15T22:08:17.518 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836519 2024-09-15T22:08:17.519 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836519 2024-09-15T22:08:17.519 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836519' 2024-09-15T22:08:17.519 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T22:08:17.519 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-15T22:08:17.618 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=42949672996 2024-09-15T22:08:17.618 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 42949672996 2024-09-15T22:08:17.618 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836519 1-42949672996' 2024-09-15T22:08:17.618 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T22:08:17.618 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-15T22:08:17.719 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=64424509474 2024-09-15T22:08:17.719 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 64424509474 2024-09-15T22:08:17.719 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836519 1-42949672996 2-64424509474' 2024-09-15T22:08:17.719 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T22:08:17.719 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836519 2024-09-15T22:08:17.720 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T22:08:17.720 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-15T22:08:17.721 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836519 2024-09-15T22:08:17.721 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T22:08:17.722 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836519 2024-09-15T22:08:17.722 INFO:tasks.workunit.client.0.smithi136.stdout:waiting osd.0 seq 21474836519 2024-09-15T22:08:17.722 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836519' 2024-09-15T22:08:17.722 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T22:08:17.995 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836517 -lt 21474836519 2024-09-15T22:08:17.996 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-15T22:08:18.997 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-15T22:08:18.997 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T22:08:19.279 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836519 -lt 21474836519 2024-09-15T22:08:19.279 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T22:08:19.280 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-42949672996 2024-09-15T22:08:19.280 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T22:08:19.280 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-15T22:08:19.281 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-42949672996 2024-09-15T22:08:19.281 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T22:08:19.282 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=42949672996 2024-09-15T22:08:19.282 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 42949672996' 2024-09-15T22:08:19.282 INFO:tasks.workunit.client.0.smithi136.stdout:waiting osd.1 seq 42949672996 2024-09-15T22:08:19.282 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-15T22:08:19.553 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 42949672996 -lt 42949672996 2024-09-15T22:08:19.554 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T22:08:19.554 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-64424509474 2024-09-15T22:08:19.554 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T22:08:19.555 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-15T22:08:19.556 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-64424509474 2024-09-15T22:08:19.556 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T22:08:19.557 INFO:tasks.workunit.client.0.smithi136.stdout:waiting osd.2 seq 64424509474 2024-09-15T22:08:19.557 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=64424509474 2024-09-15T22:08:19.557 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 64424509474' 2024-09-15T22:08:19.557 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-15T22:08:19.835 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 64424509474 -lt 64424509474 2024-09-15T22:08:19.835 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:354: _scrub_abort: ceph pg dump pgs 2024-09-15T22:08:19.835 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:354: _scrub_abort: grep '^1.0' 2024-09-15T22:08:19.835 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:354: _scrub_abort: grep -q scrubbing 2024-09-15T22:08:20.092 INFO:tasks.workunit.client.0.smithi136.stderr:dumped pgs 2024-09-15T22:08:20.104 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:351: _scrub_abort: for i in $(seq 0 200) 2024-09-15T22:08:20.104 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:353: _scrub_abort: flush_pg_stats 2024-09-15T22:08:20.104 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-15T22:08:20.104 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-15T22:08:20.378 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-15T22:08:20.378 INFO:tasks.workunit.client.0.smithi136.stderr:1 2024-09-15T22:08:20.379 INFO:tasks.workunit.client.0.smithi136.stderr:2' 2024-09-15T22:08:20.379 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-15T22:08:20.379 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T22:08:20.379 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-15T22:08:20.478 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836520 2024-09-15T22:08:20.478 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836520 2024-09-15T22:08:20.478 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836520' 2024-09-15T22:08:20.478 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T22:08:20.479 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-15T22:08:20.580 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=42949672998 2024-09-15T22:08:20.580 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 42949672998 2024-09-15T22:08:20.580 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836520 1-42949672998' 2024-09-15T22:08:20.580 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T22:08:20.580 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-15T22:08:20.680 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=64424509475 2024-09-15T22:08:20.680 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 64424509475 2024-09-15T22:08:20.680 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836520 1-42949672998 2-64424509475' 2024-09-15T22:08:20.680 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T22:08:20.681 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836520 2024-09-15T22:08:20.681 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T22:08:20.681 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-15T22:08:20.682 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836520 2024-09-15T22:08:20.682 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T22:08:20.683 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836520 2024-09-15T22:08:20.683 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836520' 2024-09-15T22:08:20.684 INFO:tasks.workunit.client.0.smithi136.stdout:waiting osd.0 seq 21474836520 2024-09-15T22:08:20.684 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T22:08:20.957 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836519 -lt 21474836520 2024-09-15T22:08:20.957 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-15T22:08:21.958 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-15T22:08:21.959 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T22:08:22.247 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836519 -lt 21474836520 2024-09-15T22:08:22.247 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-15T22:08:23.249 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 299 -eq 0 ']' 2024-09-15T22:08:23.249 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T22:08:23.535 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836520 -lt 21474836520 2024-09-15T22:08:23.535 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T22:08:23.536 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-42949672998 2024-09-15T22:08:23.536 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T22:08:23.536 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-15T22:08:23.537 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-42949672998 2024-09-15T22:08:23.537 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T22:08:23.538 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=42949672998 2024-09-15T22:08:23.538 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 42949672998' 2024-09-15T22:08:23.539 INFO:tasks.workunit.client.0.smithi136.stdout:waiting osd.1 seq 42949672998 2024-09-15T22:08:23.539 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-15T22:08:23.812 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 42949672998 -lt 42949672998 2024-09-15T22:08:23.813 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T22:08:23.813 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-64424509475 2024-09-15T22:08:23.813 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T22:08:23.814 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-15T22:08:23.815 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-64424509475 2024-09-15T22:08:23.815 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T22:08:23.816 INFO:tasks.workunit.client.0.smithi136.stdout:waiting osd.2 seq 64424509475 2024-09-15T22:08:23.816 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=64424509475 2024-09-15T22:08:23.816 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 64424509475' 2024-09-15T22:08:23.816 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-15T22:08:24.091 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 64424509476 -lt 64424509475 2024-09-15T22:08:24.091 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:354: _scrub_abort: ceph pg dump pgs 2024-09-15T22:08:24.091 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:354: _scrub_abort: grep '^1.0' 2024-09-15T22:08:24.091 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:354: _scrub_abort: grep -q scrubbing 2024-09-15T22:08:24.353 INFO:tasks.workunit.client.0.smithi136.stderr:dumped pgs 2024-09-15T22:08:24.365 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:356: _scrub_abort: found=yes 2024-09-15T22:08:24.365 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:358: _scrub_abort: break 2024-09-15T22:08:24.366 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:361: _scrub_abort: set +o pipefail 2024-09-15T22:08:24.366 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:363: _scrub_abort: test yes = no 2024-09-15T22:08:24.366 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:369: _scrub_abort: ceph osd set nodeep-scrub 2024-09-15T22:08:24.733 INFO:tasks.workunit.client.0.smithi136.stderr:nodeep-scrub is set 2024-09-15T22:08:24.744 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:370: _scrub_abort: '[' deep-scrub = deep-scrub ']' 2024-09-15T22:08:24.744 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:372: _scrub_abort: ceph osd set noscrub 2024-09-15T22:08:25.252 INFO:tasks.workunit.client.0.smithi136.stderr:noscrub is set 2024-09-15T22:08:25.263 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:376: _scrub_abort: set -o pipefail 2024-09-15T22:08:25.264 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:377: _scrub_abort: seq 0 200 2024-09-15T22:08:25.265 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:377: _scrub_abort: for i in $(seq 0 200) 2024-09-15T22:08:25.265 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:379: _scrub_abort: flush_pg_stats 2024-09-15T22:08:25.265 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-15T22:08:25.265 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-15T22:08:25.575 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-15T22:08:25.576 INFO:tasks.workunit.client.0.smithi136.stderr:1 2024-09-15T22:08:25.576 INFO:tasks.workunit.client.0.smithi136.stderr:2' 2024-09-15T22:08:25.576 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-15T22:08:25.576 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T22:08:25.576 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-15T22:08:25.680 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836522 2024-09-15T22:08:25.680 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836522 2024-09-15T22:08:25.680 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836522' 2024-09-15T22:08:25.680 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T22:08:25.681 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-15T22:08:25.782 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=42949673000 2024-09-15T22:08:25.782 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 42949673000 2024-09-15T22:08:25.782 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836522 1-42949673000' 2024-09-15T22:08:25.782 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T22:08:25.782 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-15T22:08:25.883 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=64424509477 2024-09-15T22:08:25.883 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 64424509477 2024-09-15T22:08:25.883 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836522 1-42949673000 2-64424509477' 2024-09-15T22:08:25.883 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T22:08:25.883 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836522 2024-09-15T22:08:25.883 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T22:08:25.884 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-15T22:08:25.885 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836522 2024-09-15T22:08:25.885 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T22:08:25.885 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836522 2024-09-15T22:08:25.886 INFO:tasks.workunit.client.0.smithi136.stdout:waiting osd.0 seq 21474836522 2024-09-15T22:08:25.886 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836522' 2024-09-15T22:08:25.886 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T22:08:26.157 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836521 -lt 21474836522 2024-09-15T22:08:26.157 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-15T22:08:27.158 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-15T22:08:27.158 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T22:08:27.443 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836522 -lt 21474836522 2024-09-15T22:08:27.443 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T22:08:27.443 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-42949673000 2024-09-15T22:08:27.443 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T22:08:27.444 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-15T22:08:27.445 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-42949673000 2024-09-15T22:08:27.445 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T22:08:27.446 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=42949673000 2024-09-15T22:08:27.446 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 42949673000' 2024-09-15T22:08:27.446 INFO:tasks.workunit.client.0.smithi136.stdout:waiting osd.1 seq 42949673000 2024-09-15T22:08:27.446 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-15T22:08:27.719 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 42949673000 -lt 42949673000 2024-09-15T22:08:27.719 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T22:08:27.719 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-64424509477 2024-09-15T22:08:27.720 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T22:08:27.720 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-15T22:08:27.721 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-64424509477 2024-09-15T22:08:27.721 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T22:08:27.722 INFO:tasks.workunit.client.0.smithi136.stdout:waiting osd.2 seq 64424509477 2024-09-15T22:08:27.722 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=64424509477 2024-09-15T22:08:27.722 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 64424509477' 2024-09-15T22:08:27.722 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-15T22:08:27.996 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 64424509477 -lt 64424509477 2024-09-15T22:08:27.996 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:380: _scrub_abort: ceph pg dump pgs 2024-09-15T22:08:27.996 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:380: _scrub_abort: grep '^1.0' 2024-09-15T22:08:27.996 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:380: _scrub_abort: grep -q scrubbing 2024-09-15T22:08:28.257 INFO:tasks.workunit.client.0.smithi136.stderr:dumped pgs 2024-09-15T22:08:28.268 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:382: _scrub_abort: continue 2024-09-15T22:08:28.269 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:377: _scrub_abort: for i in $(seq 0 200) 2024-09-15T22:08:28.269 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:379: _scrub_abort: flush_pg_stats 2024-09-15T22:08:28.269 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-15T22:08:28.269 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-15T22:08:28.553 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-15T22:08:28.553 INFO:tasks.workunit.client.0.smithi136.stderr:1 2024-09-15T22:08:28.553 INFO:tasks.workunit.client.0.smithi136.stderr:2' 2024-09-15T22:08:28.553 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-15T22:08:28.553 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T22:08:28.553 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-15T22:08:28.653 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836524 2024-09-15T22:08:28.654 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836524 2024-09-15T22:08:28.654 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836524' 2024-09-15T22:08:28.654 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T22:08:28.654 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-15T22:08:28.755 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=42949673001 2024-09-15T22:08:28.755 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 42949673001 2024-09-15T22:08:28.755 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836524 1-42949673001' 2024-09-15T22:08:28.755 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T22:08:28.755 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-15T22:08:28.855 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=64424509479 2024-09-15T22:08:28.855 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 64424509479 2024-09-15T22:08:28.855 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836524 1-42949673001 2-64424509479' 2024-09-15T22:08:28.855 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T22:08:28.856 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836524 2024-09-15T22:08:28.856 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T22:08:28.856 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-15T22:08:28.857 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836524 2024-09-15T22:08:28.857 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T22:08:28.858 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836524 2024-09-15T22:08:28.858 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836524' 2024-09-15T22:08:28.858 INFO:tasks.workunit.client.0.smithi136.stdout:waiting osd.0 seq 21474836524 2024-09-15T22:08:28.859 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T22:08:29.130 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836523 -lt 21474836524 2024-09-15T22:08:29.131 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-15T22:08:30.131 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-15T22:08:30.132 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T22:08:30.419 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836524 -lt 21474836524 2024-09-15T22:08:30.420 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T22:08:30.420 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-42949673001 2024-09-15T22:08:30.420 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T22:08:30.421 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-15T22:08:30.422 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-42949673001 2024-09-15T22:08:30.422 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T22:08:30.423 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=42949673001 2024-09-15T22:08:30.423 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 42949673001' 2024-09-15T22:08:30.423 INFO:tasks.workunit.client.0.smithi136.stdout:waiting osd.1 seq 42949673001 2024-09-15T22:08:30.423 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-15T22:08:30.711 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 42949673002 -lt 42949673001 2024-09-15T22:08:30.711 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T22:08:30.711 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-64424509479 2024-09-15T22:08:30.711 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T22:08:30.712 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-15T22:08:30.713 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-64424509479 2024-09-15T22:08:30.713 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T22:08:30.714 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=64424509479 2024-09-15T22:08:30.714 INFO:tasks.workunit.client.0.smithi136.stdout:waiting osd.2 seq 64424509479 2024-09-15T22:08:30.714 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 64424509479' 2024-09-15T22:08:30.715 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-15T22:08:30.988 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 64424509479 -lt 64424509479 2024-09-15T22:08:30.988 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:380: _scrub_abort: ceph pg dump pgs 2024-09-15T22:08:30.988 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:380: _scrub_abort: grep '^1.0' 2024-09-15T22:08:30.988 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:380: _scrub_abort: grep -q scrubbing 2024-09-15T22:08:31.245 INFO:tasks.workunit.client.0.smithi136.stderr:dumped pgs 2024-09-15T22:08:31.257 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:382: _scrub_abort: continue 2024-09-15T22:08:31.257 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:377: _scrub_abort: for i in $(seq 0 200) 2024-09-15T22:08:31.257 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:379: _scrub_abort: flush_pg_stats 2024-09-15T22:08:31.257 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-15T22:08:31.257 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-15T22:08:31.530 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-15T22:08:31.531 INFO:tasks.workunit.client.0.smithi136.stderr:1 2024-09-15T22:08:31.531 INFO:tasks.workunit.client.0.smithi136.stderr:2' 2024-09-15T22:08:31.531 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-15T22:08:31.531 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T22:08:31.531 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-15T22:08:31.631 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836525 2024-09-15T22:08:31.631 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836525 2024-09-15T22:08:31.631 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836525' 2024-09-15T22:08:31.632 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T22:08:31.632 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-15T22:08:31.731 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=42949673003 2024-09-15T22:08:31.732 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 42949673003 2024-09-15T22:08:31.732 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836525 1-42949673003' 2024-09-15T22:08:31.732 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T22:08:31.732 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-15T22:08:31.832 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=64424509481 2024-09-15T22:08:31.832 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 64424509481 2024-09-15T22:08:31.832 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836525 1-42949673003 2-64424509481' 2024-09-15T22:08:31.832 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T22:08:31.833 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836525 2024-09-15T22:08:31.833 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T22:08:31.834 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-15T22:08:31.834 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836525 2024-09-15T22:08:31.834 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T22:08:31.835 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836525 2024-09-15T22:08:31.836 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836525' 2024-09-15T22:08:31.836 INFO:tasks.workunit.client.0.smithi136.stdout:waiting osd.0 seq 21474836525 2024-09-15T22:08:31.836 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T22:08:32.111 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836524 -lt 21474836525 2024-09-15T22:08:32.111 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-15T22:08:33.112 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-15T22:08:33.112 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T22:08:33.395 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836525 -lt 21474836525 2024-09-15T22:08:33.396 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T22:08:33.396 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-42949673003 2024-09-15T22:08:33.396 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T22:08:33.397 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-15T22:08:33.398 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-42949673003 2024-09-15T22:08:33.398 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T22:08:33.398 INFO:tasks.workunit.client.0.smithi136.stdout:waiting osd.1 seq 42949673003 2024-09-15T22:08:33.399 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=42949673003 2024-09-15T22:08:33.399 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 42949673003' 2024-09-15T22:08:33.399 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-15T22:08:33.671 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 42949673003 -lt 42949673003 2024-09-15T22:08:33.672 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T22:08:33.672 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-64424509481 2024-09-15T22:08:33.672 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T22:08:33.673 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-15T22:08:33.674 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-64424509481 2024-09-15T22:08:33.674 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T22:08:33.674 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=64424509481 2024-09-15T22:08:33.675 INFO:tasks.workunit.client.0.smithi136.stdout:waiting osd.2 seq 64424509481 2024-09-15T22:08:33.675 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 64424509481' 2024-09-15T22:08:33.675 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-15T22:08:33.949 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 64424509481 -lt 64424509481 2024-09-15T22:08:33.949 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:380: _scrub_abort: ceph pg dump pgs 2024-09-15T22:08:33.949 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:380: _scrub_abort: grep '^1.0' 2024-09-15T22:08:33.949 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:380: _scrub_abort: grep -q scrubbing 2024-09-15T22:08:34.208 INFO:tasks.workunit.client.0.smithi136.stderr:dumped pgs 2024-09-15T22:08:34.219 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:382: _scrub_abort: continue 2024-09-15T22:08:34.219 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:377: _scrub_abort: for i in $(seq 0 200) 2024-09-15T22:08:34.220 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:379: _scrub_abort: flush_pg_stats 2024-09-15T22:08:34.220 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-15T22:08:34.220 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-15T22:08:34.508 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-15T22:08:34.509 INFO:tasks.workunit.client.0.smithi136.stderr:1 2024-09-15T22:08:34.509 INFO:tasks.workunit.client.0.smithi136.stderr:2' 2024-09-15T22:08:34.509 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-15T22:08:34.509 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T22:08:34.509 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-15T22:08:34.620 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836527 2024-09-15T22:08:34.620 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836527 2024-09-15T22:08:34.620 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836527' 2024-09-15T22:08:34.621 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T22:08:34.621 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-15T22:08:34.721 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=42949673004 2024-09-15T22:08:34.721 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 42949673004 2024-09-15T22:08:34.722 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836527 1-42949673004' 2024-09-15T22:08:34.722 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T22:08:34.722 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-15T22:08:34.821 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=64424509482 2024-09-15T22:08:34.821 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 64424509482 2024-09-15T22:08:34.821 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836527 1-42949673004 2-64424509482' 2024-09-15T22:08:34.821 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T22:08:34.822 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836527 2024-09-15T22:08:34.822 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T22:08:34.822 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-15T22:08:34.823 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836527 2024-09-15T22:08:34.823 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T22:08:34.824 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836527 2024-09-15T22:08:34.824 INFO:tasks.workunit.client.0.smithi136.stdout:waiting osd.0 seq 21474836527 2024-09-15T22:08:34.824 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836527' 2024-09-15T22:08:34.824 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T22:08:35.100 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836526 -lt 21474836527 2024-09-15T22:08:35.100 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-15T22:08:36.101 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-15T22:08:36.102 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T22:08:36.388 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836527 -lt 21474836527 2024-09-15T22:08:36.388 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T22:08:36.389 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-42949673004 2024-09-15T22:08:36.389 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T22:08:36.389 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-15T22:08:36.390 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-42949673004 2024-09-15T22:08:36.390 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T22:08:36.391 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=42949673004 2024-09-15T22:08:36.391 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 42949673004' 2024-09-15T22:08:36.391 INFO:tasks.workunit.client.0.smithi136.stdout:waiting osd.1 seq 42949673004 2024-09-15T22:08:36.391 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-15T22:08:36.664 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 42949673005 -lt 42949673004 2024-09-15T22:08:36.665 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T22:08:36.665 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-64424509482 2024-09-15T22:08:36.665 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T22:08:36.666 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-15T22:08:36.666 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-64424509482 2024-09-15T22:08:36.667 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T22:08:36.667 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=64424509482 2024-09-15T22:08:36.668 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 64424509482' 2024-09-15T22:08:36.668 INFO:tasks.workunit.client.0.smithi136.stdout:waiting osd.2 seq 64424509482 2024-09-15T22:08:36.668 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-15T22:08:36.941 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 64424509482 -lt 64424509482 2024-09-15T22:08:36.941 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:380: _scrub_abort: ceph pg dump pgs 2024-09-15T22:08:36.941 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:380: _scrub_abort: grep '^1.0' 2024-09-15T22:08:36.941 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:380: _scrub_abort: grep -q scrubbing 2024-09-15T22:08:37.201 INFO:tasks.workunit.client.0.smithi136.stderr:dumped pgs 2024-09-15T22:08:37.213 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:382: _scrub_abort: continue 2024-09-15T22:08:37.213 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:377: _scrub_abort: for i in $(seq 0 200) 2024-09-15T22:08:37.213 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:379: _scrub_abort: flush_pg_stats 2024-09-15T22:08:37.213 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-15T22:08:37.213 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-15T22:08:37.484 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-15T22:08:37.485 INFO:tasks.workunit.client.0.smithi136.stderr:1 2024-09-15T22:08:37.485 INFO:tasks.workunit.client.0.smithi136.stderr:2' 2024-09-15T22:08:37.485 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-15T22:08:37.485 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T22:08:37.485 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-15T22:08:37.585 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836529 2024-09-15T22:08:37.585 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836529 2024-09-15T22:08:37.585 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836529' 2024-09-15T22:08:37.585 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T22:08:37.585 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-15T22:08:37.685 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=42949673006 2024-09-15T22:08:37.685 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 42949673006 2024-09-15T22:08:37.685 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836529 1-42949673006' 2024-09-15T22:08:37.685 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T22:08:37.685 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-15T22:08:37.785 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=64424509484 2024-09-15T22:08:37.785 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 64424509484 2024-09-15T22:08:37.785 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836529 1-42949673006 2-64424509484' 2024-09-15T22:08:37.785 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T22:08:37.786 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836529 2024-09-15T22:08:37.786 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T22:08:37.786 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-15T22:08:37.787 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836529 2024-09-15T22:08:37.787 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T22:08:37.788 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836529 2024-09-15T22:08:37.788 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836529' 2024-09-15T22:08:37.788 INFO:tasks.workunit.client.0.smithi136.stdout:waiting osd.0 seq 21474836529 2024-09-15T22:08:37.788 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T22:08:38.062 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836527 -lt 21474836529 2024-09-15T22:08:38.062 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-15T22:08:39.063 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-15T22:08:39.064 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T22:08:39.349 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836529 -lt 21474836529 2024-09-15T22:08:39.349 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T22:08:39.350 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-42949673006 2024-09-15T22:08:39.350 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T22:08:39.351 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-15T22:08:39.351 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-42949673006 2024-09-15T22:08:39.351 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T22:08:39.352 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=42949673006 2024-09-15T22:08:39.352 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 42949673006' 2024-09-15T22:08:39.353 INFO:tasks.workunit.client.0.smithi136.stdout:waiting osd.1 seq 42949673006 2024-09-15T22:08:39.353 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-15T22:08:39.628 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 42949673006 -lt 42949673006 2024-09-15T22:08:39.628 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T22:08:39.628 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-64424509484 2024-09-15T22:08:39.628 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T22:08:39.629 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-15T22:08:39.630 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-64424509484 2024-09-15T22:08:39.630 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T22:08:39.631 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=64424509484 2024-09-15T22:08:39.631 INFO:tasks.workunit.client.0.smithi136.stdout:waiting osd.2 seq 64424509484 2024-09-15T22:08:39.631 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 64424509484' 2024-09-15T22:08:39.632 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-15T22:08:39.905 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 64424509484 -lt 64424509484 2024-09-15T22:08:39.905 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:380: _scrub_abort: ceph pg dump pgs 2024-09-15T22:08:39.905 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:380: _scrub_abort: grep '^1.0' 2024-09-15T22:08:39.905 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:380: _scrub_abort: grep -q scrubbing 2024-09-15T22:08:40.166 INFO:tasks.workunit.client.0.smithi136.stderr:dumped pgs 2024-09-15T22:08:40.178 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:382: _scrub_abort: continue 2024-09-15T22:08:40.178 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:377: _scrub_abort: for i in $(seq 0 200) 2024-09-15T22:08:40.178 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:379: _scrub_abort: flush_pg_stats 2024-09-15T22:08:40.178 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-15T22:08:40.179 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-15T22:08:40.456 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-15T22:08:40.456 INFO:tasks.workunit.client.0.smithi136.stderr:1 2024-09-15T22:08:40.456 INFO:tasks.workunit.client.0.smithi136.stderr:2' 2024-09-15T22:08:40.456 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-15T22:08:40.456 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T22:08:40.457 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-15T22:08:40.557 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836530 2024-09-15T22:08:40.557 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836530 2024-09-15T22:08:40.557 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836530' 2024-09-15T22:08:40.557 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T22:08:40.557 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-15T22:08:40.658 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=42949673008 2024-09-15T22:08:40.658 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 42949673008 2024-09-15T22:08:40.658 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836530 1-42949673008' 2024-09-15T22:08:40.658 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T22:08:40.658 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-15T22:08:40.757 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=64424509485 2024-09-15T22:08:40.758 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 64424509485 2024-09-15T22:08:40.758 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836530 1-42949673008 2-64424509485' 2024-09-15T22:08:40.758 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T22:08:40.758 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836530 2024-09-15T22:08:40.758 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T22:08:40.759 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-15T22:08:40.759 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836530 2024-09-15T22:08:40.760 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T22:08:40.760 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836530 2024-09-15T22:08:40.761 INFO:tasks.workunit.client.0.smithi136.stdout:waiting osd.0 seq 21474836530 2024-09-15T22:08:40.761 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836530' 2024-09-15T22:08:40.761 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T22:08:41.034 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836529 -lt 21474836530 2024-09-15T22:08:41.035 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-15T22:08:42.035 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-15T22:08:42.036 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T22:08:42.325 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836530 -lt 21474836530 2024-09-15T22:08:42.325 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T22:08:42.325 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-42949673008 2024-09-15T22:08:42.325 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T22:08:42.325 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-15T22:08:42.325 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-42949673008 2024-09-15T22:08:42.325 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T22:08:42.326 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=42949673008 2024-09-15T22:08:42.326 INFO:tasks.workunit.client.0.smithi136.stdout:waiting osd.1 seq 42949673008 2024-09-15T22:08:42.326 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 42949673008' 2024-09-15T22:08:42.327 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-15T22:08:42.634 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 42949673008 -lt 42949673008 2024-09-15T22:08:42.635 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T22:08:42.635 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-64424509485 2024-09-15T22:08:42.635 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T22:08:42.636 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-15T22:08:42.637 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-64424509485 2024-09-15T22:08:42.637 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T22:08:42.637 INFO:tasks.workunit.client.0.smithi136.stdout:waiting osd.2 seq 64424509485 2024-09-15T22:08:42.638 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=64424509485 2024-09-15T22:08:42.638 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 64424509485' 2024-09-15T22:08:42.638 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-15T22:08:42.914 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 64424509486 -lt 64424509485 2024-09-15T22:08:42.914 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:380: _scrub_abort: ceph pg dump pgs 2024-09-15T22:08:42.914 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:380: _scrub_abort: grep '^1.0' 2024-09-15T22:08:42.915 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:380: _scrub_abort: grep -q scrubbing 2024-09-15T22:08:43.171 INFO:tasks.workunit.client.0.smithi136.stderr:dumped pgs 2024-09-15T22:08:43.183 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:382: _scrub_abort: continue 2024-09-15T22:08:43.183 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:377: _scrub_abort: for i in $(seq 0 200) 2024-09-15T22:08:43.183 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:379: _scrub_abort: flush_pg_stats 2024-09-15T22:08:43.184 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-15T22:08:43.184 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-15T22:08:43.472 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-15T22:08:43.472 INFO:tasks.workunit.client.0.smithi136.stderr:1 2024-09-15T22:08:43.472 INFO:tasks.workunit.client.0.smithi136.stderr:2' 2024-09-15T22:08:43.472 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-15T22:08:43.472 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T22:08:43.473 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-15T22:08:43.573 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836532 2024-09-15T22:08:43.573 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836532 2024-09-15T22:08:43.573 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836532' 2024-09-15T22:08:43.573 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T22:08:43.573 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-15T22:08:43.674 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=42949673009 2024-09-15T22:08:43.674 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 42949673009 2024-09-15T22:08:43.674 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836532 1-42949673009' 2024-09-15T22:08:43.674 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T22:08:43.674 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-15T22:08:43.775 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=64424509487 2024-09-15T22:08:43.775 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 64424509487 2024-09-15T22:08:43.775 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836532 1-42949673009 2-64424509487' 2024-09-15T22:08:43.775 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T22:08:43.775 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836532 2024-09-15T22:08:43.775 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T22:08:43.776 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-15T22:08:43.777 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836532 2024-09-15T22:08:43.777 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T22:08:43.778 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836532 2024-09-15T22:08:43.778 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836532' 2024-09-15T22:08:43.778 INFO:tasks.workunit.client.0.smithi136.stdout:waiting osd.0 seq 21474836532 2024-09-15T22:08:43.778 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T22:08:44.050 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836530 -lt 21474836532 2024-09-15T22:08:44.051 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-15T22:08:45.052 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-15T22:08:45.053 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T22:08:45.333 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836532 -lt 21474836532 2024-09-15T22:08:45.333 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T22:08:45.333 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-42949673009 2024-09-15T22:08:45.333 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T22:08:45.334 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-15T22:08:45.335 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-42949673009 2024-09-15T22:08:45.335 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T22:08:45.336 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=42949673009 2024-09-15T22:08:45.336 INFO:tasks.workunit.client.0.smithi136.stdout:waiting osd.1 seq 42949673009 2024-09-15T22:08:45.336 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 42949673009' 2024-09-15T22:08:45.336 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-15T22:08:45.617 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 42949673009 -lt 42949673009 2024-09-15T22:08:45.617 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T22:08:45.618 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-64424509487 2024-09-15T22:08:45.618 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T22:08:45.619 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-15T22:08:45.620 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-64424509487 2024-09-15T22:08:45.620 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T22:08:45.620 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=64424509487 2024-09-15T22:08:45.620 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 64424509487' 2024-09-15T22:08:45.621 INFO:tasks.workunit.client.0.smithi136.stdout:waiting osd.2 seq 64424509487 2024-09-15T22:08:45.621 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-15T22:08:45.896 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 64424509487 -lt 64424509487 2024-09-15T22:08:45.896 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:380: _scrub_abort: ceph pg dump pgs 2024-09-15T22:08:45.896 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:380: _scrub_abort: grep '^1.0' 2024-09-15T22:08:45.896 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:380: _scrub_abort: grep -q scrubbing 2024-09-15T22:08:46.153 INFO:tasks.workunit.client.0.smithi136.stderr:dumped pgs 2024-09-15T22:08:46.166 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:382: _scrub_abort: continue 2024-09-15T22:08:46.168 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:377: _scrub_abort: for i in $(seq 0 200) 2024-09-15T22:08:46.169 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:379: _scrub_abort: flush_pg_stats 2024-09-15T22:08:46.170 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-15T22:08:46.170 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-15T22:08:46.441 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-15T22:08:46.441 INFO:tasks.workunit.client.0.smithi136.stderr:1 2024-09-15T22:08:46.441 INFO:tasks.workunit.client.0.smithi136.stderr:2' 2024-09-15T22:08:46.441 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-15T22:08:46.442 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T22:08:46.442 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-15T22:08:46.541 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836533 2024-09-15T22:08:46.541 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836533 2024-09-15T22:08:46.541 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836533' 2024-09-15T22:08:46.541 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T22:08:46.541 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-15T22:08:46.642 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=42949673011 2024-09-15T22:08:46.642 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 42949673011 2024-09-15T22:08:46.642 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836533 1-42949673011' 2024-09-15T22:08:46.642 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T22:08:46.642 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-15T22:08:46.741 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=64424509489 2024-09-15T22:08:46.741 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 64424509489 2024-09-15T22:08:46.741 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836533 1-42949673011 2-64424509489' 2024-09-15T22:08:46.741 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T22:08:46.742 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836533 2024-09-15T22:08:46.742 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T22:08:46.743 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-15T22:08:46.743 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836533 2024-09-15T22:08:46.743 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T22:08:46.744 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836533 2024-09-15T22:08:46.745 INFO:tasks.workunit.client.0.smithi136.stdout:waiting osd.0 seq 21474836533 2024-09-15T22:08:46.745 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836533' 2024-09-15T22:08:46.745 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T22:08:47.016 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836532 -lt 21474836533 2024-09-15T22:08:47.016 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-15T22:08:48.017 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-15T22:08:48.017 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T22:08:48.306 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836534 -lt 21474836533 2024-09-15T22:08:48.306 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T22:08:48.306 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-42949673011 2024-09-15T22:08:48.306 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T22:08:48.307 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-15T22:08:48.308 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-42949673011 2024-09-15T22:08:48.308 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T22:08:48.309 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=42949673011 2024-09-15T22:08:48.309 INFO:tasks.workunit.client.0.smithi136.stdout:waiting osd.1 seq 42949673011 2024-09-15T22:08:48.309 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 42949673011' 2024-09-15T22:08:48.309 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-15T22:08:48.581 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 42949673011 -lt 42949673011 2024-09-15T22:08:48.581 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T22:08:48.582 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-64424509489 2024-09-15T22:08:48.582 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T22:08:48.583 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-15T22:08:48.583 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-64424509489 2024-09-15T22:08:48.584 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T22:08:48.585 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=64424509489 2024-09-15T22:08:48.585 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 64424509489' 2024-09-15T22:08:48.585 INFO:tasks.workunit.client.0.smithi136.stdout:waiting osd.2 seq 64424509489 2024-09-15T22:08:48.585 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-15T22:08:48.879 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 64424509489 -lt 64424509489 2024-09-15T22:08:48.879 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:380: _scrub_abort: ceph pg dump pgs 2024-09-15T22:08:48.879 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:380: _scrub_abort: grep '^1.0' 2024-09-15T22:08:48.879 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:380: _scrub_abort: grep -q scrubbing 2024-09-15T22:08:49.136 INFO:tasks.workunit.client.0.smithi136.stderr:dumped pgs 2024-09-15T22:08:49.148 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:382: _scrub_abort: continue 2024-09-15T22:08:49.148 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:377: _scrub_abort: for i in $(seq 0 200) 2024-09-15T22:08:49.148 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:379: _scrub_abort: flush_pg_stats 2024-09-15T22:08:49.148 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-15T22:08:49.149 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-15T22:08:49.425 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-15T22:08:49.425 INFO:tasks.workunit.client.0.smithi136.stderr:1 2024-09-15T22:08:49.425 INFO:tasks.workunit.client.0.smithi136.stderr:2' 2024-09-15T22:08:49.425 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-15T22:08:49.425 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T22:08:49.425 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-15T22:08:49.531 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836535 2024-09-15T22:08:49.531 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836535 2024-09-15T22:08:49.531 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836535' 2024-09-15T22:08:49.531 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T22:08:49.531 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-15T22:08:49.632 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=42949673012 2024-09-15T22:08:49.633 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 42949673012 2024-09-15T22:08:49.633 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836535 1-42949673012' 2024-09-15T22:08:49.633 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T22:08:49.633 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-15T22:08:49.733 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=64424509490 2024-09-15T22:08:49.733 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 64424509490 2024-09-15T22:08:49.733 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836535 1-42949673012 2-64424509490' 2024-09-15T22:08:49.733 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T22:08:49.733 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836535 2024-09-15T22:08:49.734 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T22:08:49.734 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-15T22:08:49.735 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836535 2024-09-15T22:08:49.735 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T22:08:49.735 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836535 2024-09-15T22:08:49.736 INFO:tasks.workunit.client.0.smithi136.stdout:waiting osd.0 seq 21474836535 2024-09-15T22:08:49.736 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836535' 2024-09-15T22:08:49.736 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T22:08:50.008 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836534 -lt 21474836535 2024-09-15T22:08:50.008 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-15T22:08:51.009 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-15T22:08:51.010 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T22:08:51.290 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836535 -lt 21474836535 2024-09-15T22:08:51.290 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T22:08:51.291 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-42949673012 2024-09-15T22:08:51.291 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T22:08:51.291 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-15T22:08:51.292 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-42949673012 2024-09-15T22:08:51.292 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T22:08:51.293 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=42949673012 2024-09-15T22:08:51.293 INFO:tasks.workunit.client.0.smithi136.stdout:waiting osd.1 seq 42949673012 2024-09-15T22:08:51.294 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 42949673012' 2024-09-15T22:08:51.294 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-15T22:08:51.571 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 42949673013 -lt 42949673012 2024-09-15T22:08:51.571 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T22:08:51.571 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-64424509490 2024-09-15T22:08:51.571 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T22:08:51.572 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-15T22:08:51.573 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-64424509490 2024-09-15T22:08:51.573 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T22:08:51.574 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=64424509490 2024-09-15T22:08:51.574 INFO:tasks.workunit.client.0.smithi136.stdout:waiting osd.2 seq 64424509490 2024-09-15T22:08:51.574 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 64424509490' 2024-09-15T22:08:51.574 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-15T22:08:51.850 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 64424509490 -lt 64424509490 2024-09-15T22:08:51.850 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:380: _scrub_abort: ceph pg dump pgs 2024-09-15T22:08:51.850 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:380: _scrub_abort: grep '^1.0' 2024-09-15T22:08:51.850 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:380: _scrub_abort: grep -q scrubbing 2024-09-15T22:08:52.109 INFO:tasks.workunit.client.0.smithi136.stderr:dumped pgs 2024-09-15T22:08:52.121 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:382: _scrub_abort: continue 2024-09-15T22:08:52.121 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:377: _scrub_abort: for i in $(seq 0 200) 2024-09-15T22:08:52.121 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:379: _scrub_abort: flush_pg_stats 2024-09-15T22:08:52.121 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-15T22:08:52.121 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-15T22:08:52.425 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-15T22:08:52.426 INFO:tasks.workunit.client.0.smithi136.stderr:1 2024-09-15T22:08:52.426 INFO:tasks.workunit.client.0.smithi136.stderr:2' 2024-09-15T22:08:52.426 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-15T22:08:52.426 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T22:08:52.426 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-15T22:08:52.544 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836537 2024-09-15T22:08:52.545 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836537 2024-09-15T22:08:52.545 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836537' 2024-09-15T22:08:52.545 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T22:08:52.545 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-15T22:08:52.646 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=42949673014 2024-09-15T22:08:52.647 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 42949673014 2024-09-15T22:08:52.647 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836537 1-42949673014' 2024-09-15T22:08:52.647 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T22:08:52.647 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-15T22:08:52.747 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=64424509492 2024-09-15T22:08:52.748 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 64424509492 2024-09-15T22:08:52.748 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836537 1-42949673014 2-64424509492' 2024-09-15T22:08:52.748 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T22:08:52.748 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836537 2024-09-15T22:08:52.748 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T22:08:52.749 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-15T22:08:52.750 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836537 2024-09-15T22:08:52.750 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T22:08:52.750 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836537 2024-09-15T22:08:52.751 INFO:tasks.workunit.client.0.smithi136.stdout:waiting osd.0 seq 21474836537 2024-09-15T22:08:52.751 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836537' 2024-09-15T22:08:52.751 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T22:08:53.025 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836535 -lt 21474836537 2024-09-15T22:08:53.025 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-15T22:08:54.026 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-15T22:08:54.026 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T22:08:54.315 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836537 -lt 21474836537 2024-09-15T22:08:54.315 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T22:08:54.315 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-42949673014 2024-09-15T22:08:54.315 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T22:08:54.316 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-15T22:08:54.317 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-42949673014 2024-09-15T22:08:54.317 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T22:08:54.318 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=42949673014 2024-09-15T22:08:54.318 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 42949673014' 2024-09-15T22:08:54.318 INFO:tasks.workunit.client.0.smithi136.stdout:waiting osd.1 seq 42949673014 2024-09-15T22:08:54.319 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-15T22:08:54.591 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 42949673014 -lt 42949673014 2024-09-15T22:08:54.591 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T22:08:54.592 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-64424509492 2024-09-15T22:08:54.592 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T22:08:54.592 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-15T22:08:54.593 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-64424509492 2024-09-15T22:08:54.593 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T22:08:54.596 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=64424509492 2024-09-15T22:08:54.596 INFO:tasks.workunit.client.0.smithi136.stdout:waiting osd.2 seq 64424509492 2024-09-15T22:08:54.596 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 64424509492' 2024-09-15T22:08:54.596 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-15T22:08:54.869 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 64424509492 -lt 64424509492 2024-09-15T22:08:54.870 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:380: _scrub_abort: ceph pg dump pgs 2024-09-15T22:08:54.870 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:380: _scrub_abort: grep '^1.0' 2024-09-15T22:08:54.870 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:380: _scrub_abort: grep -q scrubbing 2024-09-15T22:08:55.130 INFO:tasks.workunit.client.0.smithi136.stderr:dumped pgs 2024-09-15T22:08:55.141 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:382: _scrub_abort: continue 2024-09-15T22:08:55.142 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:377: _scrub_abort: for i in $(seq 0 200) 2024-09-15T22:08:55.142 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:379: _scrub_abort: flush_pg_stats 2024-09-15T22:08:55.142 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-15T22:08:55.142 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-15T22:08:55.416 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-15T22:08:55.416 INFO:tasks.workunit.client.0.smithi136.stderr:1 2024-09-15T22:08:55.417 INFO:tasks.workunit.client.0.smithi136.stderr:2' 2024-09-15T22:08:55.417 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-15T22:08:55.417 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T22:08:55.417 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-15T22:08:55.514 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836538 2024-09-15T22:08:55.515 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836538 2024-09-15T22:08:55.515 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836538' 2024-09-15T22:08:55.515 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T22:08:55.515 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-15T22:08:55.615 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=42949673016 2024-09-15T22:08:55.616 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 42949673016 2024-09-15T22:08:55.616 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836538 1-42949673016' 2024-09-15T22:08:55.616 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T22:08:55.616 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-15T22:08:55.717 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=64424509493 2024-09-15T22:08:55.717 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 64424509493 2024-09-15T22:08:55.717 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836538 1-42949673016 2-64424509493' 2024-09-15T22:08:55.717 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T22:08:55.718 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836538 2024-09-15T22:08:55.718 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T22:08:55.718 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-15T22:08:55.719 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836538 2024-09-15T22:08:55.719 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T22:08:55.720 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836538 2024-09-15T22:08:55.720 INFO:tasks.workunit.client.0.smithi136.stdout:waiting osd.0 seq 21474836538 2024-09-15T22:08:55.720 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836538' 2024-09-15T22:08:55.721 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T22:08:55.991 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836537 -lt 21474836538 2024-09-15T22:08:55.992 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-15T22:08:56.993 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-15T22:08:56.993 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T22:08:57.275 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836538 -lt 21474836538 2024-09-15T22:08:57.275 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T22:08:57.276 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-42949673016 2024-09-15T22:08:57.276 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T22:08:57.277 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-15T22:08:57.278 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-42949673016 2024-09-15T22:08:57.278 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T22:08:57.278 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=42949673016 2024-09-15T22:08:57.279 INFO:tasks.workunit.client.0.smithi136.stdout:waiting osd.1 seq 42949673016 2024-09-15T22:08:57.279 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 42949673016' 2024-09-15T22:08:57.279 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-15T22:08:57.552 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 42949673016 -lt 42949673016 2024-09-15T22:08:57.552 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T22:08:57.552 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-64424509493 2024-09-15T22:08:57.552 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T22:08:57.553 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-15T22:08:57.554 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-64424509493 2024-09-15T22:08:57.554 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T22:08:57.555 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=64424509493 2024-09-15T22:08:57.555 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 64424509493' 2024-09-15T22:08:57.555 INFO:tasks.workunit.client.0.smithi136.stdout:waiting osd.2 seq 64424509493 2024-09-15T22:08:57.555 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-15T22:08:57.829 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 64424509493 -lt 64424509493 2024-09-15T22:08:57.829 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:380: _scrub_abort: ceph pg dump pgs 2024-09-15T22:08:57.829 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:380: _scrub_abort: grep '^1.0' 2024-09-15T22:08:57.829 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:380: _scrub_abort: grep -q scrubbing 2024-09-15T22:08:58.086 INFO:tasks.workunit.client.0.smithi136.stderr:dumped pgs 2024-09-15T22:08:58.098 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:382: _scrub_abort: continue 2024-09-15T22:08:58.098 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:377: _scrub_abort: for i in $(seq 0 200) 2024-09-15T22:08:58.098 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:379: _scrub_abort: flush_pg_stats 2024-09-15T22:08:58.098 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-15T22:08:58.098 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-15T22:08:58.388 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-15T22:08:58.388 INFO:tasks.workunit.client.0.smithi136.stderr:1 2024-09-15T22:08:58.388 INFO:tasks.workunit.client.0.smithi136.stderr:2' 2024-09-15T22:08:58.388 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-15T22:08:58.388 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T22:08:58.388 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-15T22:08:58.489 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836540 2024-09-15T22:08:58.489 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836540 2024-09-15T22:08:58.489 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836540' 2024-09-15T22:08:58.490 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T22:08:58.490 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-15T22:08:58.589 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=42949673017 2024-09-15T22:08:58.589 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 42949673017 2024-09-15T22:08:58.589 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836540 1-42949673017' 2024-09-15T22:08:58.589 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T22:08:58.590 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-15T22:08:58.692 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=64424509495 2024-09-15T22:08:58.692 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 64424509495 2024-09-15T22:08:58.692 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836540 1-42949673017 2-64424509495' 2024-09-15T22:08:58.692 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T22:08:58.693 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836540 2024-09-15T22:08:58.693 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T22:08:58.693 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-15T22:08:58.693 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836540 2024-09-15T22:08:58.694 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T22:08:58.694 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836540 2024-09-15T22:08:58.694 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836540' 2024-09-15T22:08:58.694 INFO:tasks.workunit.client.0.smithi136.stdout:waiting osd.0 seq 21474836540 2024-09-15T22:08:58.694 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T22:08:58.966 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836539 -lt 21474836540 2024-09-15T22:08:58.966 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-15T22:08:59.966 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-15T22:08:59.967 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T22:09:00.251 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836539 -lt 21474836540 2024-09-15T22:09:00.252 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-15T22:09:01.253 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 299 -eq 0 ']' 2024-09-15T22:09:01.253 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T22:09:01.537 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836540 -lt 21474836540 2024-09-15T22:09:01.538 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T22:09:01.538 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-42949673017 2024-09-15T22:09:01.538 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T22:09:01.539 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-15T22:09:01.539 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-42949673017 2024-09-15T22:09:01.540 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T22:09:01.541 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=42949673017 2024-09-15T22:09:01.541 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 42949673017' 2024-09-15T22:09:01.541 INFO:tasks.workunit.client.0.smithi136.stdout:waiting osd.1 seq 42949673017 2024-09-15T22:09:01.541 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-15T22:09:01.812 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 42949673018 -lt 42949673017 2024-09-15T22:09:01.812 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T22:09:01.813 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-64424509495 2024-09-15T22:09:01.813 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T22:09:01.814 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-15T22:09:01.815 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-64424509495 2024-09-15T22:09:01.815 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T22:09:01.815 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=64424509495 2024-09-15T22:09:01.816 INFO:tasks.workunit.client.0.smithi136.stdout:waiting osd.2 seq 64424509495 2024-09-15T22:09:01.816 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 64424509495' 2024-09-15T22:09:01.816 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-15T22:09:02.093 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 64424509495 -lt 64424509495 2024-09-15T22:09:02.093 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:380: _scrub_abort: ceph pg dump pgs 2024-09-15T22:09:02.093 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:380: _scrub_abort: grep '^1.0' 2024-09-15T22:09:02.093 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:380: _scrub_abort: grep -q scrubbing 2024-09-15T22:09:02.379 INFO:tasks.workunit.client.0.smithi136.stderr:dumped pgs 2024-09-15T22:09:02.391 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:382: _scrub_abort: continue 2024-09-15T22:09:02.391 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:377: _scrub_abort: for i in $(seq 0 200) 2024-09-15T22:09:02.391 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:379: _scrub_abort: flush_pg_stats 2024-09-15T22:09:02.391 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-15T22:09:02.391 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-15T22:09:02.666 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-15T22:09:02.666 INFO:tasks.workunit.client.0.smithi136.stderr:1 2024-09-15T22:09:02.666 INFO:tasks.workunit.client.0.smithi136.stderr:2' 2024-09-15T22:09:02.666 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-15T22:09:02.666 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T22:09:02.666 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-15T22:09:02.767 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836542 2024-09-15T22:09:02.767 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836542 2024-09-15T22:09:02.767 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836542' 2024-09-15T22:09:02.767 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T22:09:02.767 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-15T22:09:02.868 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=42949673019 2024-09-15T22:09:02.868 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 42949673019 2024-09-15T22:09:02.868 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836542 1-42949673019' 2024-09-15T22:09:02.868 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T22:09:02.868 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-15T22:09:02.967 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=64424509497 2024-09-15T22:09:02.968 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 64424509497 2024-09-15T22:09:02.968 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836542 1-42949673019 2-64424509497' 2024-09-15T22:09:02.968 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T22:09:02.968 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836542 2024-09-15T22:09:02.968 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T22:09:02.969 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-15T22:09:02.969 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836542 2024-09-15T22:09:02.970 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T22:09:02.970 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836542 2024-09-15T22:09:02.971 INFO:tasks.workunit.client.0.smithi136.stdout:waiting osd.0 seq 21474836542 2024-09-15T22:09:02.971 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836542' 2024-09-15T22:09:02.971 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T22:09:03.243 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836540 -lt 21474836542 2024-09-15T22:09:03.244 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-15T22:09:04.245 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-15T22:09:04.245 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T22:09:04.566 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836542 -lt 21474836542 2024-09-15T22:09:04.566 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T22:09:04.566 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-42949673019 2024-09-15T22:09:04.566 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T22:09:04.567 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-15T22:09:04.568 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-42949673019 2024-09-15T22:09:04.568 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T22:09:04.569 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=42949673019 2024-09-15T22:09:04.569 INFO:tasks.workunit.client.0.smithi136.stdout:waiting osd.1 seq 42949673019 2024-09-15T22:09:04.569 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 42949673019' 2024-09-15T22:09:04.569 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-15T22:09:04.842 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 42949673019 -lt 42949673019 2024-09-15T22:09:04.842 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T22:09:04.842 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-64424509497 2024-09-15T22:09:04.842 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T22:09:04.843 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-15T22:09:04.844 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-64424509497 2024-09-15T22:09:04.845 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T22:09:04.845 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=64424509497 2024-09-15T22:09:04.845 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 64424509497' 2024-09-15T22:09:04.845 INFO:tasks.workunit.client.0.smithi136.stdout:waiting osd.2 seq 64424509497 2024-09-15T22:09:04.846 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-15T22:09:05.118 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 64424509497 -lt 64424509497 2024-09-15T22:09:05.119 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:380: _scrub_abort: ceph pg dump pgs 2024-09-15T22:09:05.119 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:380: _scrub_abort: grep '^1.0' 2024-09-15T22:09:05.119 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:380: _scrub_abort: grep -q scrubbing 2024-09-15T22:09:05.377 INFO:tasks.workunit.client.0.smithi136.stderr:dumped pgs 2024-09-15T22:09:05.389 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:382: _scrub_abort: continue 2024-09-15T22:09:05.389 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:377: _scrub_abort: for i in $(seq 0 200) 2024-09-15T22:09:05.389 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:379: _scrub_abort: flush_pg_stats 2024-09-15T22:09:05.389 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-15T22:09:05.390 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-15T22:09:05.662 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-15T22:09:05.662 INFO:tasks.workunit.client.0.smithi136.stderr:1 2024-09-15T22:09:05.662 INFO:tasks.workunit.client.0.smithi136.stderr:2' 2024-09-15T22:09:05.662 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-15T22:09:05.662 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T22:09:05.663 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-15T22:09:05.764 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836543 2024-09-15T22:09:05.764 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836543 2024-09-15T22:09:05.764 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836543' 2024-09-15T22:09:05.764 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T22:09:05.764 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-15T22:09:05.865 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=42949673021 2024-09-15T22:09:05.866 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 42949673021 2024-09-15T22:09:05.866 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836543 1-42949673021' 2024-09-15T22:09:05.866 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T22:09:05.866 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-15T22:09:05.966 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=64424509498 2024-09-15T22:09:05.966 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 64424509498 2024-09-15T22:09:05.966 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836543 1-42949673021 2-64424509498' 2024-09-15T22:09:05.966 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T22:09:05.967 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836543 2024-09-15T22:09:05.967 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T22:09:05.967 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-15T22:09:05.968 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836543 2024-09-15T22:09:05.968 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T22:09:05.969 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836543 2024-09-15T22:09:05.969 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836543' 2024-09-15T22:09:05.969 INFO:tasks.workunit.client.0.smithi136.stdout:waiting osd.0 seq 21474836543 2024-09-15T22:09:05.970 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T22:09:06.247 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836542 -lt 21474836543 2024-09-15T22:09:06.247 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-15T22:09:07.248 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-15T22:09:07.249 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T22:09:07.535 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836543 -lt 21474836543 2024-09-15T22:09:07.535 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T22:09:07.536 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-42949673021 2024-09-15T22:09:07.536 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T22:09:07.537 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-15T22:09:07.537 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-42949673021 2024-09-15T22:09:07.537 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T22:09:07.538 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=42949673021 2024-09-15T22:09:07.538 INFO:tasks.workunit.client.0.smithi136.stdout:waiting osd.1 seq 42949673021 2024-09-15T22:09:07.539 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 42949673021' 2024-09-15T22:09:07.539 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-15T22:09:07.816 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 42949673021 -lt 42949673021 2024-09-15T22:09:07.816 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T22:09:07.817 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-64424509498 2024-09-15T22:09:07.817 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T22:09:07.817 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-15T22:09:07.818 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-64424509498 2024-09-15T22:09:07.818 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T22:09:07.819 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=64424509498 2024-09-15T22:09:07.819 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 64424509498' 2024-09-15T22:09:07.820 INFO:tasks.workunit.client.0.smithi136.stdout:waiting osd.2 seq 64424509498 2024-09-15T22:09:07.820 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-15T22:09:08.094 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 64424509498 -lt 64424509498 2024-09-15T22:09:08.094 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:380: _scrub_abort: ceph pg dump pgs 2024-09-15T22:09:08.095 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:380: _scrub_abort: grep '^1.0' 2024-09-15T22:09:08.095 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:380: _scrub_abort: grep -q scrubbing 2024-09-15T22:09:08.356 INFO:tasks.workunit.client.0.smithi136.stderr:dumped pgs 2024-09-15T22:09:08.367 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:382: _scrub_abort: continue 2024-09-15T22:09:08.368 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:377: _scrub_abort: for i in $(seq 0 200) 2024-09-15T22:09:08.368 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:379: _scrub_abort: flush_pg_stats 2024-09-15T22:09:08.368 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-15T22:09:08.368 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-15T22:09:08.639 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-15T22:09:08.639 INFO:tasks.workunit.client.0.smithi136.stderr:1 2024-09-15T22:09:08.639 INFO:tasks.workunit.client.0.smithi136.stderr:2' 2024-09-15T22:09:08.639 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-15T22:09:08.639 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T22:09:08.639 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-15T22:09:08.740 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836545 2024-09-15T22:09:08.740 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836545 2024-09-15T22:09:08.740 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836545' 2024-09-15T22:09:08.740 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T22:09:08.741 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-15T22:09:08.841 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=42949673022 2024-09-15T22:09:08.841 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 42949673022 2024-09-15T22:09:08.841 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836545 1-42949673022' 2024-09-15T22:09:08.842 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T22:09:08.842 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-15T22:09:08.941 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=64424509500 2024-09-15T22:09:08.942 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 64424509500 2024-09-15T22:09:08.942 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836545 1-42949673022 2-64424509500' 2024-09-15T22:09:08.942 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T22:09:08.942 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836545 2024-09-15T22:09:08.942 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T22:09:08.943 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-15T22:09:08.944 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836545 2024-09-15T22:09:08.944 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T22:09:08.945 INFO:tasks.workunit.client.0.smithi136.stdout:waiting osd.0 seq 21474836545 2024-09-15T22:09:08.945 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836545 2024-09-15T22:09:08.945 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836545' 2024-09-15T22:09:08.945 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T22:09:09.218 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836544 -lt 21474836545 2024-09-15T22:09:09.218 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-15T22:09:10.219 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-15T22:09:10.220 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T22:09:10.521 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836545 -lt 21474836545 2024-09-15T22:09:10.521 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T22:09:10.521 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-42949673022 2024-09-15T22:09:10.521 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T22:09:10.522 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-15T22:09:10.523 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-42949673022 2024-09-15T22:09:10.523 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T22:09:10.524 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=42949673022 2024-09-15T22:09:10.524 INFO:tasks.workunit.client.0.smithi136.stdout:waiting osd.1 seq 42949673022 2024-09-15T22:09:10.524 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 42949673022' 2024-09-15T22:09:10.524 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-15T22:09:10.798 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 42949673023 -lt 42949673022 2024-09-15T22:09:10.798 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T22:09:10.799 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-64424509500 2024-09-15T22:09:10.799 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T22:09:10.800 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-15T22:09:10.800 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-64424509500 2024-09-15T22:09:10.801 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T22:09:10.801 INFO:tasks.workunit.client.0.smithi136.stdout:waiting osd.2 seq 64424509500 2024-09-15T22:09:10.802 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=64424509500 2024-09-15T22:09:10.802 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 64424509500' 2024-09-15T22:09:10.802 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-15T22:09:11.074 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 64424509500 -lt 64424509500 2024-09-15T22:09:11.074 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:380: _scrub_abort: ceph pg dump pgs 2024-09-15T22:09:11.074 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:380: _scrub_abort: grep '^1.0' 2024-09-15T22:09:11.074 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:380: _scrub_abort: grep -q scrubbing 2024-09-15T22:09:11.333 INFO:tasks.workunit.client.0.smithi136.stderr:dumped pgs 2024-09-15T22:09:11.344 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:382: _scrub_abort: continue 2024-09-15T22:09:11.344 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:377: _scrub_abort: for i in $(seq 0 200) 2024-09-15T22:09:11.344 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:379: _scrub_abort: flush_pg_stats 2024-09-15T22:09:11.344 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-15T22:09:11.345 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-15T22:09:11.619 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-15T22:09:11.619 INFO:tasks.workunit.client.0.smithi136.stderr:1 2024-09-15T22:09:11.619 INFO:tasks.workunit.client.0.smithi136.stderr:2' 2024-09-15T22:09:11.619 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-15T22:09:11.619 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T22:09:11.619 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-15T22:09:11.720 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836546 2024-09-15T22:09:11.720 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836546 2024-09-15T22:09:11.720 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836546' 2024-09-15T22:09:11.720 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T22:09:11.720 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-15T22:09:11.821 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=42949673024 2024-09-15T22:09:11.821 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 42949673024 2024-09-15T22:09:11.821 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836546 1-42949673024' 2024-09-15T22:09:11.821 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T22:09:11.821 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-15T22:09:11.921 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=64424509502 2024-09-15T22:09:11.921 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 64424509502 2024-09-15T22:09:11.921 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836546 1-42949673024 2-64424509502' 2024-09-15T22:09:11.921 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T22:09:11.922 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836546 2024-09-15T22:09:11.922 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T22:09:11.922 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-15T22:09:11.923 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836546 2024-09-15T22:09:11.923 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T22:09:11.924 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836546 2024-09-15T22:09:11.924 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836546' 2024-09-15T22:09:11.924 INFO:tasks.workunit.client.0.smithi136.stdout:waiting osd.0 seq 21474836546 2024-09-15T22:09:11.924 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T22:09:12.197 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836545 -lt 21474836546 2024-09-15T22:09:12.197 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-15T22:09:13.198 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-15T22:09:13.198 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T22:09:13.484 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836546 -lt 21474836546 2024-09-15T22:09:13.484 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T22:09:13.484 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-42949673024 2024-09-15T22:09:13.485 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T22:09:13.485 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-15T22:09:13.486 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-42949673024 2024-09-15T22:09:13.486 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T22:09:13.487 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=42949673024 2024-09-15T22:09:13.487 INFO:tasks.workunit.client.0.smithi136.stdout:waiting osd.1 seq 42949673024 2024-09-15T22:09:13.487 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 42949673024' 2024-09-15T22:09:13.487 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-15T22:09:13.760 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 42949673024 -lt 42949673024 2024-09-15T22:09:13.760 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T22:09:13.761 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-64424509502 2024-09-15T22:09:13.761 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T22:09:13.761 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-15T22:09:13.762 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-64424509502 2024-09-15T22:09:13.762 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T22:09:13.763 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=64424509502 2024-09-15T22:09:13.763 INFO:tasks.workunit.client.0.smithi136.stdout:waiting osd.2 seq 64424509502 2024-09-15T22:09:13.764 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 64424509502' 2024-09-15T22:09:13.764 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-15T22:09:14.037 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 64424509502 -lt 64424509502 2024-09-15T22:09:14.037 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:380: _scrub_abort: ceph pg dump pgs 2024-09-15T22:09:14.037 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:380: _scrub_abort: grep '^1.0' 2024-09-15T22:09:14.037 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:380: _scrub_abort: grep -q scrubbing 2024-09-15T22:09:14.296 INFO:tasks.workunit.client.0.smithi136.stderr:dumped pgs 2024-09-15T22:09:14.307 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:382: _scrub_abort: continue 2024-09-15T22:09:14.307 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:377: _scrub_abort: for i in $(seq 0 200) 2024-09-15T22:09:14.308 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:379: _scrub_abort: flush_pg_stats 2024-09-15T22:09:14.308 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-15T22:09:14.308 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-15T22:09:14.604 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-15T22:09:14.604 INFO:tasks.workunit.client.0.smithi136.stderr:1 2024-09-15T22:09:14.605 INFO:tasks.workunit.client.0.smithi136.stderr:2' 2024-09-15T22:09:14.605 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-15T22:09:14.605 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T22:09:14.605 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-15T22:09:14.706 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836548 2024-09-15T22:09:14.706 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836548 2024-09-15T22:09:14.707 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836548' 2024-09-15T22:09:14.707 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T22:09:14.707 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-15T22:09:14.808 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=42949673025 2024-09-15T22:09:14.808 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 42949673025 2024-09-15T22:09:14.808 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836548 1-42949673025' 2024-09-15T22:09:14.808 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T22:09:14.809 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-15T22:09:14.909 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=64424509503 2024-09-15T22:09:14.909 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 64424509503 2024-09-15T22:09:14.909 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836548 1-42949673025 2-64424509503' 2024-09-15T22:09:14.909 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T22:09:14.910 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836548 2024-09-15T22:09:14.910 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T22:09:14.910 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-15T22:09:14.911 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836548 2024-09-15T22:09:14.911 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T22:09:14.913 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836548 2024-09-15T22:09:14.913 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836548' 2024-09-15T22:09:14.913 INFO:tasks.workunit.client.0.smithi136.stdout:waiting osd.0 seq 21474836548 2024-09-15T22:09:14.913 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T22:09:15.185 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836547 -lt 21474836548 2024-09-15T22:09:15.185 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-15T22:09:16.187 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-15T22:09:16.187 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T22:09:16.505 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836548 -lt 21474836548 2024-09-15T22:09:16.505 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T22:09:16.505 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-42949673025 2024-09-15T22:09:16.505 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T22:09:16.506 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-15T22:09:16.507 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-42949673025 2024-09-15T22:09:16.507 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T22:09:16.508 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=42949673025 2024-09-15T22:09:16.508 INFO:tasks.workunit.client.0.smithi136.stdout:waiting osd.1 seq 42949673025 2024-09-15T22:09:16.508 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 42949673025' 2024-09-15T22:09:16.508 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-15T22:09:16.781 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 42949673026 -lt 42949673025 2024-09-15T22:09:16.781 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T22:09:16.782 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-64424509503 2024-09-15T22:09:16.782 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T22:09:16.783 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-15T22:09:16.783 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-64424509503 2024-09-15T22:09:16.783 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T22:09:16.784 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=64424509503 2024-09-15T22:09:16.784 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 64424509503' 2024-09-15T22:09:16.784 INFO:tasks.workunit.client.0.smithi136.stdout:waiting osd.2 seq 64424509503 2024-09-15T22:09:16.785 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-15T22:09:17.059 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 64424509503 -lt 64424509503 2024-09-15T22:09:17.059 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:380: _scrub_abort: ceph pg dump pgs 2024-09-15T22:09:17.060 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:380: _scrub_abort: grep '^1.0' 2024-09-15T22:09:17.060 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:380: _scrub_abort: grep -q scrubbing 2024-09-15T22:09:17.318 INFO:tasks.workunit.client.0.smithi136.stderr:dumped pgs 2024-09-15T22:09:17.330 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:382: _scrub_abort: continue 2024-09-15T22:09:17.330 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:377: _scrub_abort: for i in $(seq 0 200) 2024-09-15T22:09:17.330 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:379: _scrub_abort: flush_pg_stats 2024-09-15T22:09:17.330 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-15T22:09:17.331 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-15T22:09:17.601 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-15T22:09:17.601 INFO:tasks.workunit.client.0.smithi136.stderr:1 2024-09-15T22:09:17.601 INFO:tasks.workunit.client.0.smithi136.stderr:2' 2024-09-15T22:09:17.601 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-15T22:09:17.601 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T22:09:17.601 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-15T22:09:17.701 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836550 2024-09-15T22:09:17.702 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836550 2024-09-15T22:09:17.702 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836550' 2024-09-15T22:09:17.702 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T22:09:17.702 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-15T22:09:17.803 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=42949673027 2024-09-15T22:09:17.803 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 42949673027 2024-09-15T22:09:17.803 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836550 1-42949673027' 2024-09-15T22:09:17.803 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T22:09:17.803 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-15T22:09:17.904 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=64424509505 2024-09-15T22:09:17.905 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 64424509505 2024-09-15T22:09:17.905 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836550 1-42949673027 2-64424509505' 2024-09-15T22:09:17.905 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T22:09:17.905 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836550 2024-09-15T22:09:17.905 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T22:09:17.906 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-15T22:09:17.906 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836550 2024-09-15T22:09:17.907 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T22:09:17.908 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836550 2024-09-15T22:09:17.909 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836550' 2024-09-15T22:09:17.909 INFO:tasks.workunit.client.0.smithi136.stdout:waiting osd.0 seq 21474836550 2024-09-15T22:09:17.909 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T22:09:18.180 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836548 -lt 21474836550 2024-09-15T22:09:18.180 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-15T22:09:19.181 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-15T22:09:19.181 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T22:09:19.468 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836550 -lt 21474836550 2024-09-15T22:09:19.468 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T22:09:19.469 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-42949673027 2024-09-15T22:09:19.469 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T22:09:19.470 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-15T22:09:19.470 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-42949673027 2024-09-15T22:09:19.470 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T22:09:19.471 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=42949673027 2024-09-15T22:09:19.472 INFO:tasks.workunit.client.0.smithi136.stdout:waiting osd.1 seq 42949673027 2024-09-15T22:09:19.472 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 42949673027' 2024-09-15T22:09:19.472 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-15T22:09:19.744 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 42949673027 -lt 42949673027 2024-09-15T22:09:19.744 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T22:09:19.744 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-64424509505 2024-09-15T22:09:19.745 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T22:09:19.745 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-15T22:09:19.746 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-64424509505 2024-09-15T22:09:19.746 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T22:09:19.747 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=64424509505 2024-09-15T22:09:19.747 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 64424509505' 2024-09-15T22:09:19.747 INFO:tasks.workunit.client.0.smithi136.stdout:waiting osd.2 seq 64424509505 2024-09-15T22:09:19.747 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-15T22:09:20.027 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 64424509505 -lt 64424509505 2024-09-15T22:09:20.027 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:380: _scrub_abort: ceph pg dump pgs 2024-09-15T22:09:20.027 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:380: _scrub_abort: grep '^1.0' 2024-09-15T22:09:20.027 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:380: _scrub_abort: grep -q scrubbing 2024-09-15T22:09:20.283 INFO:tasks.workunit.client.0.smithi136.stderr:dumped pgs 2024-09-15T22:09:20.295 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:382: _scrub_abort: continue 2024-09-15T22:09:20.295 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:377: _scrub_abort: for i in $(seq 0 200) 2024-09-15T22:09:20.295 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:379: _scrub_abort: flush_pg_stats 2024-09-15T22:09:20.295 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-15T22:09:20.295 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-15T22:09:20.567 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-15T22:09:20.568 INFO:tasks.workunit.client.0.smithi136.stderr:1 2024-09-15T22:09:20.568 INFO:tasks.workunit.client.0.smithi136.stderr:2' 2024-09-15T22:09:20.568 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-15T22:09:20.568 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T22:09:20.568 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-15T22:09:20.669 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836551 2024-09-15T22:09:20.669 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836551 2024-09-15T22:09:20.669 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836551' 2024-09-15T22:09:20.669 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T22:09:20.669 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-15T22:09:20.770 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=42949673029 2024-09-15T22:09:20.770 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 42949673029 2024-09-15T22:09:20.770 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836551 1-42949673029' 2024-09-15T22:09:20.770 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T22:09:20.771 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-15T22:09:20.871 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=64424509506 2024-09-15T22:09:20.871 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 64424509506 2024-09-15T22:09:20.871 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836551 1-42949673029 2-64424509506' 2024-09-15T22:09:20.871 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T22:09:20.872 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836551 2024-09-15T22:09:20.872 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T22:09:20.872 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-15T22:09:20.873 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836551 2024-09-15T22:09:20.873 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T22:09:20.874 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836551 2024-09-15T22:09:20.874 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836551' 2024-09-15T22:09:20.874 INFO:tasks.workunit.client.0.smithi136.stdout:waiting osd.0 seq 21474836551 2024-09-15T22:09:20.874 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T22:09:21.146 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836550 -lt 21474836551 2024-09-15T22:09:21.147 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-15T22:09:22.147 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-15T22:09:22.148 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T22:09:22.428 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836551 -lt 21474836551 2024-09-15T22:09:22.428 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T22:09:22.429 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-42949673029 2024-09-15T22:09:22.429 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T22:09:22.429 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-15T22:09:22.429 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-42949673029 2024-09-15T22:09:22.430 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T22:09:22.431 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=42949673029 2024-09-15T22:09:22.431 INFO:tasks.workunit.client.0.smithi136.stdout:waiting osd.1 seq 42949673029 2024-09-15T22:09:22.431 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 42949673029' 2024-09-15T22:09:22.431 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-15T22:09:22.736 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 42949673029 -lt 42949673029 2024-09-15T22:09:22.737 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T22:09:22.737 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-64424509506 2024-09-15T22:09:22.737 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T22:09:22.738 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-15T22:09:22.739 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-64424509506 2024-09-15T22:09:22.739 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T22:09:22.739 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=64424509506 2024-09-15T22:09:22.740 INFO:tasks.workunit.client.0.smithi136.stdout:waiting osd.2 seq 64424509506 2024-09-15T22:09:22.740 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 64424509506' 2024-09-15T22:09:22.740 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-15T22:09:23.015 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 64424509507 -lt 64424509506 2024-09-15T22:09:23.015 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:380: _scrub_abort: ceph pg dump pgs 2024-09-15T22:09:23.015 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:380: _scrub_abort: grep '^1.0' 2024-09-15T22:09:23.015 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:380: _scrub_abort: grep -q scrubbing 2024-09-15T22:09:23.274 INFO:tasks.workunit.client.0.smithi136.stderr:dumped pgs 2024-09-15T22:09:23.286 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:382: _scrub_abort: continue 2024-09-15T22:09:23.286 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:377: _scrub_abort: for i in $(seq 0 200) 2024-09-15T22:09:23.286 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:379: _scrub_abort: flush_pg_stats 2024-09-15T22:09:23.286 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-15T22:09:23.286 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-15T22:09:23.560 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-15T22:09:23.560 INFO:tasks.workunit.client.0.smithi136.stderr:1 2024-09-15T22:09:23.560 INFO:tasks.workunit.client.0.smithi136.stderr:2' 2024-09-15T22:09:23.560 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-15T22:09:23.560 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T22:09:23.560 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-15T22:09:23.662 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836553 2024-09-15T22:09:23.663 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836553 2024-09-15T22:09:23.663 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836553' 2024-09-15T22:09:23.663 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T22:09:23.663 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-15T22:09:23.764 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=42949673030 2024-09-15T22:09:23.765 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 42949673030 2024-09-15T22:09:23.765 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836553 1-42949673030' 2024-09-15T22:09:23.765 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T22:09:23.765 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-15T22:09:23.866 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=64424509508 2024-09-15T22:09:23.867 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 64424509508 2024-09-15T22:09:23.867 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836553 1-42949673030 2-64424509508' 2024-09-15T22:09:23.867 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T22:09:23.867 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836553 2024-09-15T22:09:23.867 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T22:09:23.868 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-15T22:09:23.868 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836553 2024-09-15T22:09:23.869 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T22:09:23.869 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836553 2024-09-15T22:09:23.870 INFO:tasks.workunit.client.0.smithi136.stdout:waiting osd.0 seq 21474836553 2024-09-15T22:09:23.870 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836553' 2024-09-15T22:09:23.870 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T22:09:24.141 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836551 -lt 21474836553 2024-09-15T22:09:24.141 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-15T22:09:25.142 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-15T22:09:25.142 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T22:09:25.419 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836553 -lt 21474836553 2024-09-15T22:09:25.419 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T22:09:25.420 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-42949673030 2024-09-15T22:09:25.420 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T22:09:25.421 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-15T22:09:25.421 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-42949673030 2024-09-15T22:09:25.421 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T22:09:25.423 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=42949673030 2024-09-15T22:09:25.423 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 42949673030' 2024-09-15T22:09:25.423 INFO:tasks.workunit.client.0.smithi136.stdout:waiting osd.1 seq 42949673030 2024-09-15T22:09:25.423 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-15T22:09:25.701 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 42949673030 -lt 42949673030 2024-09-15T22:09:25.701 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T22:09:25.701 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-64424509508 2024-09-15T22:09:25.701 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T22:09:25.701 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-15T22:09:25.701 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-64424509508 2024-09-15T22:09:25.701 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T22:09:25.702 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=64424509508 2024-09-15T22:09:25.702 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 64424509508' 2024-09-15T22:09:25.702 INFO:tasks.workunit.client.0.smithi136.stdout:waiting osd.2 seq 64424509508 2024-09-15T22:09:25.702 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-15T22:09:25.976 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 64424509508 -lt 64424509508 2024-09-15T22:09:25.976 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:380: _scrub_abort: ceph pg dump pgs 2024-09-15T22:09:25.976 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:380: _scrub_abort: grep '^1.0' 2024-09-15T22:09:25.976 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:380: _scrub_abort: grep -q scrubbing 2024-09-15T22:09:26.236 INFO:tasks.workunit.client.0.smithi136.stderr:dumped pgs 2024-09-15T22:09:26.248 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:382: _scrub_abort: continue 2024-09-15T22:09:26.248 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:377: _scrub_abort: for i in $(seq 0 200) 2024-09-15T22:09:26.248 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:379: _scrub_abort: flush_pg_stats 2024-09-15T22:09:26.248 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-15T22:09:26.248 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-15T22:09:26.585 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-15T22:09:26.585 INFO:tasks.workunit.client.0.smithi136.stderr:1 2024-09-15T22:09:26.585 INFO:tasks.workunit.client.0.smithi136.stderr:2' 2024-09-15T22:09:26.586 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-15T22:09:26.586 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T22:09:26.586 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-15T22:09:26.686 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836554 2024-09-15T22:09:26.686 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836554 2024-09-15T22:09:26.686 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836554' 2024-09-15T22:09:26.687 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T22:09:26.687 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-15T22:09:26.787 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=42949673032 2024-09-15T22:09:26.787 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 42949673032 2024-09-15T22:09:26.787 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836554 1-42949673032' 2024-09-15T22:09:26.787 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T22:09:26.787 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-15T22:09:26.888 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=64424509510 2024-09-15T22:09:26.888 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 64424509510 2024-09-15T22:09:26.888 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836554 1-42949673032 2-64424509510' 2024-09-15T22:09:26.888 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T22:09:26.889 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836554 2024-09-15T22:09:26.889 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T22:09:26.889 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-15T22:09:26.890 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836554 2024-09-15T22:09:26.890 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T22:09:26.891 INFO:tasks.workunit.client.0.smithi136.stdout:waiting osd.0 seq 21474836554 2024-09-15T22:09:26.891 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836554 2024-09-15T22:09:26.891 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836554' 2024-09-15T22:09:26.891 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T22:09:27.163 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836553 -lt 21474836554 2024-09-15T22:09:27.163 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-15T22:09:28.164 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-15T22:09:28.164 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T22:09:28.448 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836555 -lt 21474836554 2024-09-15T22:09:28.448 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T22:09:28.448 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-42949673032 2024-09-15T22:09:28.448 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T22:09:28.449 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-15T22:09:28.450 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-42949673032 2024-09-15T22:09:28.450 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T22:09:28.451 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=42949673032 2024-09-15T22:09:28.451 INFO:tasks.workunit.client.0.smithi136.stdout:waiting osd.1 seq 42949673032 2024-09-15T22:09:28.451 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 42949673032' 2024-09-15T22:09:28.452 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-15T22:09:28.726 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 42949673032 -lt 42949673032 2024-09-15T22:09:28.726 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T22:09:28.726 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-64424509510 2024-09-15T22:09:28.727 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T22:09:28.727 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-15T22:09:28.728 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-64424509510 2024-09-15T22:09:28.728 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T22:09:28.729 INFO:tasks.workunit.client.0.smithi136.stdout:waiting osd.2 seq 64424509510 2024-09-15T22:09:28.729 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=64424509510 2024-09-15T22:09:28.729 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 64424509510' 2024-09-15T22:09:28.729 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-15T22:09:29.006 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 64424509510 -lt 64424509510 2024-09-15T22:09:29.006 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:380: _scrub_abort: ceph pg dump pgs 2024-09-15T22:09:29.006 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:380: _scrub_abort: grep '^1.0' 2024-09-15T22:09:29.006 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:380: _scrub_abort: grep -q scrubbing 2024-09-15T22:09:29.269 INFO:tasks.workunit.client.0.smithi136.stderr:dumped pgs 2024-09-15T22:09:29.280 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:382: _scrub_abort: continue 2024-09-15T22:09:29.281 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:377: _scrub_abort: for i in $(seq 0 200) 2024-09-15T22:09:29.281 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:379: _scrub_abort: flush_pg_stats 2024-09-15T22:09:29.281 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-15T22:09:29.281 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-15T22:09:29.554 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-15T22:09:29.554 INFO:tasks.workunit.client.0.smithi136.stderr:1 2024-09-15T22:09:29.554 INFO:tasks.workunit.client.0.smithi136.stderr:2' 2024-09-15T22:09:29.554 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-15T22:09:29.554 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T22:09:29.555 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-15T22:09:29.655 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836556 2024-09-15T22:09:29.655 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836556 2024-09-15T22:09:29.655 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836556' 2024-09-15T22:09:29.655 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T22:09:29.655 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-15T22:09:29.756 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=42949673033 2024-09-15T22:09:29.756 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 42949673033 2024-09-15T22:09:29.756 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836556 1-42949673033' 2024-09-15T22:09:29.756 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T22:09:29.756 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-15T22:09:29.856 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=64424509511 2024-09-15T22:09:29.856 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 64424509511 2024-09-15T22:09:29.856 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836556 1-42949673033 2-64424509511' 2024-09-15T22:09:29.856 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T22:09:29.857 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836556 2024-09-15T22:09:29.857 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T22:09:29.857 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-15T22:09:29.858 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836556 2024-09-15T22:09:29.859 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T22:09:29.859 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836556 2024-09-15T22:09:29.859 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836556' 2024-09-15T22:09:29.859 INFO:tasks.workunit.client.0.smithi136.stdout:waiting osd.0 seq 21474836556 2024-09-15T22:09:29.860 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T22:09:30.132 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836555 -lt 21474836556 2024-09-15T22:09:30.132 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-15T22:09:31.133 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-15T22:09:31.133 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T22:09:31.418 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836556 -lt 21474836556 2024-09-15T22:09:31.418 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T22:09:31.418 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-42949673033 2024-09-15T22:09:31.418 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T22:09:31.419 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-15T22:09:31.419 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-42949673033 2024-09-15T22:09:31.420 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T22:09:31.420 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=42949673033 2024-09-15T22:09:31.421 INFO:tasks.workunit.client.0.smithi136.stdout:waiting osd.1 seq 42949673033 2024-09-15T22:09:31.421 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 42949673033' 2024-09-15T22:09:31.421 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-15T22:09:31.692 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 42949673034 -lt 42949673033 2024-09-15T22:09:31.692 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T22:09:31.693 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-64424509511 2024-09-15T22:09:31.693 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T22:09:31.693 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-15T22:09:31.694 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-64424509511 2024-09-15T22:09:31.694 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T22:09:31.695 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=64424509511 2024-09-15T22:09:31.695 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 64424509511' 2024-09-15T22:09:31.696 INFO:tasks.workunit.client.0.smithi136.stdout:waiting osd.2 seq 64424509511 2024-09-15T22:09:31.696 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-15T22:09:31.969 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 64424509511 -lt 64424509511 2024-09-15T22:09:31.969 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:380: _scrub_abort: ceph pg dump pgs 2024-09-15T22:09:31.970 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:380: _scrub_abort: grep '^1.0' 2024-09-15T22:09:31.970 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:380: _scrub_abort: grep -q scrubbing 2024-09-15T22:09:32.229 INFO:tasks.workunit.client.0.smithi136.stderr:dumped pgs 2024-09-15T22:09:32.240 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:382: _scrub_abort: continue 2024-09-15T22:09:32.240 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:377: _scrub_abort: for i in $(seq 0 200) 2024-09-15T22:09:32.240 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:379: _scrub_abort: flush_pg_stats 2024-09-15T22:09:32.240 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-15T22:09:32.241 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-15T22:09:32.544 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-15T22:09:32.544 INFO:tasks.workunit.client.0.smithi136.stderr:1 2024-09-15T22:09:32.544 INFO:tasks.workunit.client.0.smithi136.stderr:2' 2024-09-15T22:09:32.544 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-15T22:09:32.544 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T22:09:32.544 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-15T22:09:32.644 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836558 2024-09-15T22:09:32.644 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836558 2024-09-15T22:09:32.644 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836558' 2024-09-15T22:09:32.644 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T22:09:32.644 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-15T22:09:32.747 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=42949673035 2024-09-15T22:09:32.747 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 42949673035 2024-09-15T22:09:32.747 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836558 1-42949673035' 2024-09-15T22:09:32.747 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T22:09:32.747 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-15T22:09:32.846 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=64424509513 2024-09-15T22:09:32.846 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 64424509513 2024-09-15T22:09:32.846 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836558 1-42949673035 2-64424509513' 2024-09-15T22:09:32.846 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T22:09:32.847 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836558 2024-09-15T22:09:32.847 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T22:09:32.847 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-15T22:09:32.848 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836558 2024-09-15T22:09:32.848 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T22:09:32.849 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836558 2024-09-15T22:09:32.849 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836558' 2024-09-15T22:09:32.850 INFO:tasks.workunit.client.0.smithi136.stdout:waiting osd.0 seq 21474836558 2024-09-15T22:09:32.850 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T22:09:33.125 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836556 -lt 21474836558 2024-09-15T22:09:33.125 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-15T22:09:34.126 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-15T22:09:34.127 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T22:09:34.424 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836558 -lt 21474836558 2024-09-15T22:09:34.424 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T22:09:34.424 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-42949673035 2024-09-15T22:09:34.424 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T22:09:34.425 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-15T22:09:34.425 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-42949673035 2024-09-15T22:09:34.426 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T22:09:34.427 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=42949673035 2024-09-15T22:09:34.427 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 42949673035' 2024-09-15T22:09:34.427 INFO:tasks.workunit.client.0.smithi136.stdout:waiting osd.1 seq 42949673035 2024-09-15T22:09:34.427 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-15T22:09:34.727 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 42949673035 -lt 42949673035 2024-09-15T22:09:34.727 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T22:09:34.727 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-64424509513 2024-09-15T22:09:34.728 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T22:09:34.728 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-15T22:09:34.729 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-64424509513 2024-09-15T22:09:34.730 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T22:09:34.730 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=64424509513 2024-09-15T22:09:34.730 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 64424509513' 2024-09-15T22:09:34.730 INFO:tasks.workunit.client.0.smithi136.stdout:waiting osd.2 seq 64424509513 2024-09-15T22:09:34.731 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-15T22:09:35.006 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 64424509513 -lt 64424509513 2024-09-15T22:09:35.006 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:380: _scrub_abort: ceph pg dump pgs 2024-09-15T22:09:35.007 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:380: _scrub_abort: grep '^1.0' 2024-09-15T22:09:35.007 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:380: _scrub_abort: grep -q scrubbing 2024-09-15T22:09:35.292 INFO:tasks.workunit.client.0.smithi136.stderr:dumped pgs 2024-09-15T22:09:35.304 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:382: _scrub_abort: continue 2024-09-15T22:09:35.304 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:377: _scrub_abort: for i in $(seq 0 200) 2024-09-15T22:09:35.304 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:379: _scrub_abort: flush_pg_stats 2024-09-15T22:09:35.304 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-15T22:09:35.304 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-15T22:09:35.577 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-15T22:09:35.577 INFO:tasks.workunit.client.0.smithi136.stderr:1 2024-09-15T22:09:35.577 INFO:tasks.workunit.client.0.smithi136.stderr:2' 2024-09-15T22:09:35.577 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-15T22:09:35.577 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T22:09:35.577 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-15T22:09:35.677 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836559 2024-09-15T22:09:35.677 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836559 2024-09-15T22:09:35.677 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836559' 2024-09-15T22:09:35.677 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T22:09:35.678 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-15T22:09:35.778 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=42949673037 2024-09-15T22:09:35.778 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 42949673037 2024-09-15T22:09:35.779 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836559 1-42949673037' 2024-09-15T22:09:35.779 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T22:09:35.779 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-15T22:09:35.878 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=64424509514 2024-09-15T22:09:35.879 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 64424509514 2024-09-15T22:09:35.879 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836559 1-42949673037 2-64424509514' 2024-09-15T22:09:35.879 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T22:09:35.879 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836559 2024-09-15T22:09:35.879 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T22:09:35.880 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-15T22:09:35.880 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836559 2024-09-15T22:09:35.881 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T22:09:35.881 INFO:tasks.workunit.client.0.smithi136.stdout:waiting osd.0 seq 21474836559 2024-09-15T22:09:35.882 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836559 2024-09-15T22:09:35.882 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836559' 2024-09-15T22:09:35.882 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T22:09:36.160 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836558 -lt 21474836559 2024-09-15T22:09:36.160 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-15T22:09:37.161 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-15T22:09:37.161 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T22:09:37.447 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836559 -lt 21474836559 2024-09-15T22:09:37.447 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T22:09:37.448 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-42949673037 2024-09-15T22:09:37.448 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T22:09:37.449 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-15T22:09:37.449 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-42949673037 2024-09-15T22:09:37.449 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T22:09:37.450 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=42949673037 2024-09-15T22:09:37.451 INFO:tasks.workunit.client.0.smithi136.stdout:waiting osd.1 seq 42949673037 2024-09-15T22:09:37.451 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 42949673037' 2024-09-15T22:09:37.451 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-15T22:09:37.726 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 42949673037 -lt 42949673037 2024-09-15T22:09:37.726 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T22:09:37.726 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-64424509514 2024-09-15T22:09:37.726 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T22:09:37.727 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-15T22:09:37.728 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-64424509514 2024-09-15T22:09:37.728 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T22:09:37.729 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=64424509514 2024-09-15T22:09:37.729 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 64424509514' 2024-09-15T22:09:37.729 INFO:tasks.workunit.client.0.smithi136.stdout:waiting osd.2 seq 64424509514 2024-09-15T22:09:37.730 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-15T22:09:38.004 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 64424509514 -lt 64424509514 2024-09-15T22:09:38.004 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:380: _scrub_abort: ceph pg dump pgs 2024-09-15T22:09:38.004 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:380: _scrub_abort: grep '^1.0' 2024-09-15T22:09:38.004 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:380: _scrub_abort: grep -q scrubbing 2024-09-15T22:09:38.263 INFO:tasks.workunit.client.0.smithi136.stderr:dumped pgs 2024-09-15T22:09:38.275 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:382: _scrub_abort: continue 2024-09-15T22:09:38.275 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:377: _scrub_abort: for i in $(seq 0 200) 2024-09-15T22:09:38.276 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:379: _scrub_abort: flush_pg_stats 2024-09-15T22:09:38.276 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-15T22:09:38.276 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-15T22:09:38.569 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-15T22:09:38.569 INFO:tasks.workunit.client.0.smithi136.stderr:1 2024-09-15T22:09:38.569 INFO:tasks.workunit.client.0.smithi136.stderr:2' 2024-09-15T22:09:38.569 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-15T22:09:38.569 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T22:09:38.570 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-15T22:09:38.670 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836561 2024-09-15T22:09:38.670 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836561 2024-09-15T22:09:38.670 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836561' 2024-09-15T22:09:38.671 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T22:09:38.671 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-15T22:09:38.772 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=42949673038 2024-09-15T22:09:38.772 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 42949673038 2024-09-15T22:09:38.772 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836561 1-42949673038' 2024-09-15T22:09:38.772 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T22:09:38.772 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-15T22:09:38.872 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=64424509516 2024-09-15T22:09:38.872 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 64424509516 2024-09-15T22:09:38.872 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836561 1-42949673038 2-64424509516' 2024-09-15T22:09:38.873 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T22:09:38.873 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836561 2024-09-15T22:09:38.873 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T22:09:38.874 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-15T22:09:38.874 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836561 2024-09-15T22:09:38.875 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T22:09:38.875 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836561 2024-09-15T22:09:38.875 INFO:tasks.workunit.client.0.smithi136.stdout:waiting osd.0 seq 21474836561 2024-09-15T22:09:38.876 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836561' 2024-09-15T22:09:38.876 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T22:09:39.147 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836560 -lt 21474836561 2024-09-15T22:09:39.147 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-15T22:09:40.148 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-15T22:09:40.148 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T22:09:40.434 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836561 -lt 21474836561 2024-09-15T22:09:40.434 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T22:09:40.435 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-42949673038 2024-09-15T22:09:40.435 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T22:09:40.435 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-15T22:09:40.436 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-42949673038 2024-09-15T22:09:40.436 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T22:09:40.437 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=42949673038 2024-09-15T22:09:40.437 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 42949673038' 2024-09-15T22:09:40.437 INFO:tasks.workunit.client.0.smithi136.stdout:waiting osd.1 seq 42949673038 2024-09-15T22:09:40.438 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-15T22:09:40.722 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 42949673039 -lt 42949673038 2024-09-15T22:09:40.722 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T22:09:40.722 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-64424509516 2024-09-15T22:09:40.722 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T22:09:40.723 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-15T22:09:40.724 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-64424509516 2024-09-15T22:09:40.724 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T22:09:40.725 INFO:tasks.workunit.client.0.smithi136.stdout:waiting osd.2 seq 64424509516 2024-09-15T22:09:40.725 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=64424509516 2024-09-15T22:09:40.725 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 64424509516' 2024-09-15T22:09:40.725 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-15T22:09:41.000 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 64424509516 -lt 64424509516 2024-09-15T22:09:41.000 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:380: _scrub_abort: ceph pg dump pgs 2024-09-15T22:09:41.000 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:380: _scrub_abort: grep '^1.0' 2024-09-15T22:09:41.000 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:380: _scrub_abort: grep -q scrubbing 2024-09-15T22:09:41.257 INFO:tasks.workunit.client.0.smithi136.stderr:dumped pgs 2024-09-15T22:09:41.270 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:382: _scrub_abort: continue 2024-09-15T22:09:41.270 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:377: _scrub_abort: for i in $(seq 0 200) 2024-09-15T22:09:41.271 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:379: _scrub_abort: flush_pg_stats 2024-09-15T22:09:41.271 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-15T22:09:41.271 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-15T22:09:41.545 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-15T22:09:41.545 INFO:tasks.workunit.client.0.smithi136.stderr:1 2024-09-15T22:09:41.545 INFO:tasks.workunit.client.0.smithi136.stderr:2' 2024-09-15T22:09:41.546 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-15T22:09:41.546 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T22:09:41.546 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-15T22:09:41.647 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836562 2024-09-15T22:09:41.647 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836562 2024-09-15T22:09:41.647 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836562' 2024-09-15T22:09:41.647 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T22:09:41.647 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-15T22:09:41.748 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=42949673040 2024-09-15T22:09:41.749 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 42949673040 2024-09-15T22:09:41.749 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836562 1-42949673040' 2024-09-15T22:09:41.749 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T22:09:41.749 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-15T22:09:41.848 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=64424509518 2024-09-15T22:09:41.848 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 64424509518 2024-09-15T22:09:41.848 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836562 1-42949673040 2-64424509518' 2024-09-15T22:09:41.849 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T22:09:41.849 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836562 2024-09-15T22:09:41.849 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T22:09:41.850 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-15T22:09:41.850 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836562 2024-09-15T22:09:41.850 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T22:09:41.851 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836562 2024-09-15T22:09:41.852 INFO:tasks.workunit.client.0.smithi136.stdout:waiting osd.0 seq 21474836562 2024-09-15T22:09:41.852 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836562' 2024-09-15T22:09:41.852 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T22:09:42.127 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836561 -lt 21474836562 2024-09-15T22:09:42.127 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-15T22:09:43.128 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-15T22:09:43.128 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T22:09:43.413 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836562 -lt 21474836562 2024-09-15T22:09:43.413 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T22:09:43.414 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-42949673040 2024-09-15T22:09:43.414 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T22:09:43.414 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-15T22:09:43.415 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-42949673040 2024-09-15T22:09:43.415 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T22:09:43.416 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=42949673040 2024-09-15T22:09:43.416 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 42949673040' 2024-09-15T22:09:43.416 INFO:tasks.workunit.client.0.smithi136.stdout:waiting osd.1 seq 42949673040 2024-09-15T22:09:43.417 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-15T22:09:43.692 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 42949673040 -lt 42949673040 2024-09-15T22:09:43.692 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T22:09:43.692 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-64424509518 2024-09-15T22:09:43.692 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T22:09:43.693 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-15T22:09:43.694 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-64424509518 2024-09-15T22:09:43.694 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T22:09:43.695 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=64424509518 2024-09-15T22:09:43.695 INFO:tasks.workunit.client.0.smithi136.stdout:waiting osd.2 seq 64424509518 2024-09-15T22:09:43.696 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 64424509518' 2024-09-15T22:09:43.696 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-15T22:09:43.969 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 64424509518 -lt 64424509518 2024-09-15T22:09:43.969 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:380: _scrub_abort: ceph pg dump pgs 2024-09-15T22:09:43.969 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:380: _scrub_abort: grep '^1.0' 2024-09-15T22:09:43.969 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:380: _scrub_abort: grep -q scrubbing 2024-09-15T22:09:44.231 INFO:tasks.workunit.client.0.smithi136.stderr:dumped pgs 2024-09-15T22:09:44.243 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:382: _scrub_abort: continue 2024-09-15T22:09:44.243 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:377: _scrub_abort: for i in $(seq 0 200) 2024-09-15T22:09:44.243 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:379: _scrub_abort: flush_pg_stats 2024-09-15T22:09:44.243 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-15T22:09:44.243 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-15T22:09:44.518 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-15T22:09:44.518 INFO:tasks.workunit.client.0.smithi136.stderr:1 2024-09-15T22:09:44.518 INFO:tasks.workunit.client.0.smithi136.stderr:2' 2024-09-15T22:09:44.519 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-15T22:09:44.519 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T22:09:44.519 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-15T22:09:44.621 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836564 2024-09-15T22:09:44.621 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836564 2024-09-15T22:09:44.621 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836564' 2024-09-15T22:09:44.621 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T22:09:44.621 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-15T22:09:44.722 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=42949673041 2024-09-15T22:09:44.723 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 42949673041 2024-09-15T22:09:44.723 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836564 1-42949673041' 2024-09-15T22:09:44.723 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T22:09:44.723 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-15T22:09:44.823 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=64424509519 2024-09-15T22:09:44.823 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 64424509519 2024-09-15T22:09:44.823 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836564 1-42949673041 2-64424509519' 2024-09-15T22:09:44.823 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T22:09:44.824 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836564 2024-09-15T22:09:44.824 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T22:09:44.824 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-15T22:09:44.825 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836564 2024-09-15T22:09:44.825 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T22:09:44.826 INFO:tasks.workunit.client.0.smithi136.stdout:waiting osd.0 seq 21474836564 2024-09-15T22:09:44.826 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836564 2024-09-15T22:09:44.826 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836564' 2024-09-15T22:09:44.826 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T22:09:45.100 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836563 -lt 21474836564 2024-09-15T22:09:45.101 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-15T22:09:46.102 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-15T22:09:46.102 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T22:09:46.389 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836564 -lt 21474836564 2024-09-15T22:09:46.389 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T22:09:46.389 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-42949673041 2024-09-15T22:09:46.389 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T22:09:46.390 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-15T22:09:46.391 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-42949673041 2024-09-15T22:09:46.391 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T22:09:46.392 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=42949673041 2024-09-15T22:09:46.392 INFO:tasks.workunit.client.0.smithi136.stdout:waiting osd.1 seq 42949673041 2024-09-15T22:09:46.392 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 42949673041' 2024-09-15T22:09:46.392 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-15T22:09:46.667 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 42949673042 -lt 42949673041 2024-09-15T22:09:46.667 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T22:09:46.667 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-64424509519 2024-09-15T22:09:46.667 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T22:09:46.668 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-15T22:09:46.669 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-64424509519 2024-09-15T22:09:46.669 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T22:09:46.670 INFO:tasks.workunit.client.0.smithi136.stdout:waiting osd.2 seq 64424509519 2024-09-15T22:09:46.670 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=64424509519 2024-09-15T22:09:46.670 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 64424509519' 2024-09-15T22:09:46.670 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-15T22:09:46.944 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 64424509519 -lt 64424509519 2024-09-15T22:09:46.944 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:380: _scrub_abort: ceph pg dump pgs 2024-09-15T22:09:46.944 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:380: _scrub_abort: grep '^1.0' 2024-09-15T22:09:46.944 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:380: _scrub_abort: grep -q scrubbing 2024-09-15T22:09:47.204 INFO:tasks.workunit.client.0.smithi136.stderr:dumped pgs 2024-09-15T22:09:47.216 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:382: _scrub_abort: continue 2024-09-15T22:09:47.216 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:377: _scrub_abort: for i in $(seq 0 200) 2024-09-15T22:09:47.217 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:379: _scrub_abort: flush_pg_stats 2024-09-15T22:09:47.217 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-15T22:09:47.217 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-15T22:09:47.490 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-15T22:09:47.490 INFO:tasks.workunit.client.0.smithi136.stderr:1 2024-09-15T22:09:47.490 INFO:tasks.workunit.client.0.smithi136.stderr:2' 2024-09-15T22:09:47.490 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-15T22:09:47.491 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T22:09:47.491 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-15T22:09:47.589 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836566 2024-09-15T22:09:47.590 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836566 2024-09-15T22:09:47.590 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836566' 2024-09-15T22:09:47.590 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T22:09:47.590 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-15T22:09:47.691 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=42949673043 2024-09-15T22:09:47.691 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 42949673043 2024-09-15T22:09:47.691 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836566 1-42949673043' 2024-09-15T22:09:47.691 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T22:09:47.691 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-15T22:09:47.792 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=64424509521 2024-09-15T22:09:47.792 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 64424509521 2024-09-15T22:09:47.792 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836566 1-42949673043 2-64424509521' 2024-09-15T22:09:47.792 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T22:09:47.792 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836566 2024-09-15T22:09:47.792 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T22:09:47.793 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-15T22:09:47.794 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836566 2024-09-15T22:09:47.794 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T22:09:47.795 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836566 2024-09-15T22:09:47.795 INFO:tasks.workunit.client.0.smithi136.stdout:waiting osd.0 seq 21474836566 2024-09-15T22:09:47.795 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836566' 2024-09-15T22:09:47.795 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T22:09:48.069 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836564 -lt 21474836566 2024-09-15T22:09:48.069 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-15T22:09:49.070 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-15T22:09:49.070 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T22:09:49.347 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836566 -lt 21474836566 2024-09-15T22:09:49.347 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T22:09:49.348 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-42949673043 2024-09-15T22:09:49.348 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T22:09:49.349 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-15T22:09:49.349 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-42949673043 2024-09-15T22:09:49.349 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T22:09:49.350 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=42949673043 2024-09-15T22:09:49.350 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 42949673043' 2024-09-15T22:09:49.351 INFO:tasks.workunit.client.0.smithi136.stdout:waiting osd.1 seq 42949673043 2024-09-15T22:09:49.351 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-15T22:09:49.625 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 42949673043 -lt 42949673043 2024-09-15T22:09:49.625 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T22:09:49.626 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-64424509521 2024-09-15T22:09:49.626 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T22:09:49.627 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-15T22:09:49.627 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-64424509521 2024-09-15T22:09:49.628 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T22:09:49.628 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=64424509521 2024-09-15T22:09:49.629 INFO:tasks.workunit.client.0.smithi136.stdout:waiting osd.2 seq 64424509521 2024-09-15T22:09:49.629 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 64424509521' 2024-09-15T22:09:49.629 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-15T22:09:49.905 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 64424509521 -lt 64424509521 2024-09-15T22:09:49.905 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:380: _scrub_abort: ceph pg dump pgs 2024-09-15T22:09:49.905 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:380: _scrub_abort: grep '^1.0' 2024-09-15T22:09:49.905 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:380: _scrub_abort: grep -q scrubbing 2024-09-15T22:09:50.163 INFO:tasks.workunit.client.0.smithi136.stderr:dumped pgs 2024-09-15T22:09:50.174 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:382: _scrub_abort: continue 2024-09-15T22:09:50.175 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:377: _scrub_abort: for i in $(seq 0 200) 2024-09-15T22:09:50.175 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:379: _scrub_abort: flush_pg_stats 2024-09-15T22:09:50.175 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-15T22:09:50.175 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-15T22:09:50.450 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-15T22:09:50.450 INFO:tasks.workunit.client.0.smithi136.stderr:1 2024-09-15T22:09:50.451 INFO:tasks.workunit.client.0.smithi136.stderr:2' 2024-09-15T22:09:50.451 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-15T22:09:50.451 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T22:09:50.451 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-15T22:09:50.553 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836567 2024-09-15T22:09:50.553 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836567 2024-09-15T22:09:50.553 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836567' 2024-09-15T22:09:50.553 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T22:09:50.553 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-15T22:09:50.654 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=42949673045 2024-09-15T22:09:50.654 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 42949673045 2024-09-15T22:09:50.654 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836567 1-42949673045' 2024-09-15T22:09:50.654 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T22:09:50.655 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-15T22:09:50.756 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=64424509522 2024-09-15T22:09:50.756 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 64424509522 2024-09-15T22:09:50.756 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836567 1-42949673045 2-64424509522' 2024-09-15T22:09:50.756 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T22:09:50.756 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836567 2024-09-15T22:09:50.757 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T22:09:50.757 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-15T22:09:50.758 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836567 2024-09-15T22:09:50.758 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T22:09:50.759 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836567 2024-09-15T22:09:50.759 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836567' 2024-09-15T22:09:50.759 INFO:tasks.workunit.client.0.smithi136.stdout:waiting osd.0 seq 21474836567 2024-09-15T22:09:50.759 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T22:09:51.031 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836566 -lt 21474836567 2024-09-15T22:09:51.031 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-15T22:09:52.032 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-15T22:09:52.033 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T22:09:52.325 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836567 -lt 21474836567 2024-09-15T22:09:52.325 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T22:09:52.325 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-42949673045 2024-09-15T22:09:52.325 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T22:09:52.326 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-15T22:09:52.327 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-42949673045 2024-09-15T22:09:52.327 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T22:09:52.328 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=42949673045 2024-09-15T22:09:52.328 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 42949673045' 2024-09-15T22:09:52.328 INFO:tasks.workunit.client.0.smithi136.stdout:waiting osd.1 seq 42949673045 2024-09-15T22:09:52.328 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-15T22:09:52.603 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 42949673045 -lt 42949673045 2024-09-15T22:09:52.603 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T22:09:52.604 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-64424509522 2024-09-15T22:09:52.604 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T22:09:52.605 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-15T22:09:52.605 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-64424509522 2024-09-15T22:09:52.605 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T22:09:52.606 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=64424509522 2024-09-15T22:09:52.607 INFO:tasks.workunit.client.0.smithi136.stdout:waiting osd.2 seq 64424509522 2024-09-15T22:09:52.607 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 64424509522' 2024-09-15T22:09:52.607 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-15T22:09:52.888 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 64424509523 -lt 64424509522 2024-09-15T22:09:52.888 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:380: _scrub_abort: ceph pg dump pgs 2024-09-15T22:09:52.888 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:380: _scrub_abort: grep '^1.0' 2024-09-15T22:09:52.888 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:380: _scrub_abort: grep -q scrubbing 2024-09-15T22:09:53.147 INFO:tasks.workunit.client.0.smithi136.stderr:dumped pgs 2024-09-15T22:09:53.158 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:382: _scrub_abort: continue 2024-09-15T22:09:53.158 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:377: _scrub_abort: for i in $(seq 0 200) 2024-09-15T22:09:53.158 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:379: _scrub_abort: flush_pg_stats 2024-09-15T22:09:53.158 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-15T22:09:53.159 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-15T22:09:53.433 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-15T22:09:53.433 INFO:tasks.workunit.client.0.smithi136.stderr:1 2024-09-15T22:09:53.433 INFO:tasks.workunit.client.0.smithi136.stderr:2' 2024-09-15T22:09:53.433 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-15T22:09:53.433 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T22:09:53.433 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-15T22:09:53.534 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836569 2024-09-15T22:09:53.534 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836569 2024-09-15T22:09:53.534 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836569' 2024-09-15T22:09:53.534 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T22:09:53.534 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-15T22:09:53.635 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=42949673046 2024-09-15T22:09:53.635 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 42949673046 2024-09-15T22:09:53.635 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836569 1-42949673046' 2024-09-15T22:09:53.635 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T22:09:53.635 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-15T22:09:53.735 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=64424509524 2024-09-15T22:09:53.735 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 64424509524 2024-09-15T22:09:53.735 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836569 1-42949673046 2-64424509524' 2024-09-15T22:09:53.735 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T22:09:53.736 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836569 2024-09-15T22:09:53.736 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T22:09:53.736 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-15T22:09:53.737 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836569 2024-09-15T22:09:53.737 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T22:09:53.738 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836569 2024-09-15T22:09:53.738 INFO:tasks.workunit.client.0.smithi136.stdout:waiting osd.0 seq 21474836569 2024-09-15T22:09:53.739 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836569' 2024-09-15T22:09:53.739 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T22:09:54.014 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836567 -lt 21474836569 2024-09-15T22:09:54.014 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-15T22:09:55.015 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-15T22:09:55.015 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T22:09:55.298 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836569 -lt 21474836569 2024-09-15T22:09:55.299 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T22:09:55.299 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-42949673046 2024-09-15T22:09:55.299 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T22:09:55.300 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-15T22:09:55.300 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-42949673046 2024-09-15T22:09:55.301 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T22:09:55.302 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=42949673046 2024-09-15T22:09:55.302 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 42949673046' 2024-09-15T22:09:55.302 INFO:tasks.workunit.client.0.smithi136.stdout:waiting osd.1 seq 42949673046 2024-09-15T22:09:55.302 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-15T22:09:55.575 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 42949673046 -lt 42949673046 2024-09-15T22:09:55.575 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T22:09:55.576 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-64424509524 2024-09-15T22:09:55.576 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T22:09:55.577 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-15T22:09:55.577 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-64424509524 2024-09-15T22:09:55.577 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T22:09:55.578 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=64424509524 2024-09-15T22:09:55.578 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 64424509524' 2024-09-15T22:09:55.579 INFO:tasks.workunit.client.0.smithi136.stdout:waiting osd.2 seq 64424509524 2024-09-15T22:09:55.579 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-15T22:09:55.871 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 64424509524 -lt 64424509524 2024-09-15T22:09:55.871 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:380: _scrub_abort: ceph pg dump pgs 2024-09-15T22:09:55.872 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:380: _scrub_abort: grep '^1.0' 2024-09-15T22:09:55.872 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:380: _scrub_abort: grep -q scrubbing 2024-09-15T22:09:56.130 INFO:tasks.workunit.client.0.smithi136.stderr:dumped pgs 2024-09-15T22:09:56.142 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:382: _scrub_abort: continue 2024-09-15T22:09:56.142 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:377: _scrub_abort: for i in $(seq 0 200) 2024-09-15T22:09:56.142 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:379: _scrub_abort: flush_pg_stats 2024-09-15T22:09:56.142 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-15T22:09:56.143 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-15T22:09:56.428 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-15T22:09:56.428 INFO:tasks.workunit.client.0.smithi136.stderr:1 2024-09-15T22:09:56.429 INFO:tasks.workunit.client.0.smithi136.stderr:2' 2024-09-15T22:09:56.429 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-15T22:09:56.429 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T22:09:56.429 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-15T22:09:56.558 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836570 2024-09-15T22:09:56.558 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836570 2024-09-15T22:09:56.558 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836570' 2024-09-15T22:09:56.558 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T22:09:56.558 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-15T22:09:56.658 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=42949673048 2024-09-15T22:09:56.658 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 42949673048 2024-09-15T22:09:56.658 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836570 1-42949673048' 2024-09-15T22:09:56.658 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T22:09:56.658 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-15T22:09:56.758 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=64424509526 2024-09-15T22:09:56.758 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 64424509526 2024-09-15T22:09:56.758 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836570 1-42949673048 2-64424509526' 2024-09-15T22:09:56.758 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T22:09:56.759 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836570 2024-09-15T22:09:56.759 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T22:09:56.760 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-15T22:09:56.760 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836570 2024-09-15T22:09:56.760 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T22:09:56.761 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836570 2024-09-15T22:09:56.761 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836570' 2024-09-15T22:09:56.762 INFO:tasks.workunit.client.0.smithi136.stdout:waiting osd.0 seq 21474836570 2024-09-15T22:09:56.762 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T22:09:57.037 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836569 -lt 21474836570 2024-09-15T22:09:57.038 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-15T22:09:58.038 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-15T22:09:58.039 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T22:09:58.325 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836571 -lt 21474836570 2024-09-15T22:09:58.325 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T22:09:58.325 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-42949673048 2024-09-15T22:09:58.325 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T22:09:58.326 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-15T22:09:58.327 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-42949673048 2024-09-15T22:09:58.327 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T22:09:58.328 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=42949673048 2024-09-15T22:09:58.328 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 42949673048' 2024-09-15T22:09:58.328 INFO:tasks.workunit.client.0.smithi136.stdout:waiting osd.1 seq 42949673048 2024-09-15T22:09:58.329 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-15T22:09:58.604 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 42949673048 -lt 42949673048 2024-09-15T22:09:58.604 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T22:09:58.605 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-64424509526 2024-09-15T22:09:58.605 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T22:09:58.606 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-15T22:09:58.606 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-64424509526 2024-09-15T22:09:58.607 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T22:09:58.607 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=64424509526 2024-09-15T22:09:58.608 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 64424509526' 2024-09-15T22:09:58.608 INFO:tasks.workunit.client.0.smithi136.stdout:waiting osd.2 seq 64424509526 2024-09-15T22:09:58.608 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-15T22:09:58.883 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 64424509526 -lt 64424509526 2024-09-15T22:09:58.883 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:380: _scrub_abort: ceph pg dump pgs 2024-09-15T22:09:58.883 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:380: _scrub_abort: grep '^1.0' 2024-09-15T22:09:58.883 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:380: _scrub_abort: grep -q scrubbing 2024-09-15T22:09:59.144 INFO:tasks.workunit.client.0.smithi136.stderr:dumped pgs 2024-09-15T22:09:59.156 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:382: _scrub_abort: continue 2024-09-15T22:09:59.156 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:377: _scrub_abort: for i in $(seq 0 200) 2024-09-15T22:09:59.156 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:379: _scrub_abort: flush_pg_stats 2024-09-15T22:09:59.156 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-15T22:09:59.156 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-15T22:09:59.430 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-15T22:09:59.430 INFO:tasks.workunit.client.0.smithi136.stderr:1 2024-09-15T22:09:59.430 INFO:tasks.workunit.client.0.smithi136.stderr:2' 2024-09-15T22:09:59.430 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-15T22:09:59.430 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T22:09:59.430 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-15T22:09:59.545 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836572 2024-09-15T22:09:59.546 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836572 2024-09-15T22:09:59.546 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836572' 2024-09-15T22:09:59.546 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T22:09:59.546 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-15T22:09:59.645 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=42949673049 2024-09-15T22:09:59.645 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 42949673049 2024-09-15T22:09:59.646 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836572 1-42949673049' 2024-09-15T22:09:59.646 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T22:09:59.646 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-15T22:09:59.745 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=64424509527 2024-09-15T22:09:59.745 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 64424509527 2024-09-15T22:09:59.745 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836572 1-42949673049 2-64424509527' 2024-09-15T22:09:59.745 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T22:09:59.745 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836572 2024-09-15T22:09:59.745 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T22:09:59.746 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-15T22:09:59.747 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836572 2024-09-15T22:09:59.747 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T22:09:59.747 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836572 2024-09-15T22:09:59.748 INFO:tasks.workunit.client.0.smithi136.stdout:waiting osd.0 seq 21474836572 2024-09-15T22:09:59.748 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836572' 2024-09-15T22:09:59.748 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T22:10:00.019 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836571 -lt 21474836572 2024-09-15T22:10:00.020 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-15T22:10:01.021 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-15T22:10:01.021 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T22:10:01.307 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836572 -lt 21474836572 2024-09-15T22:10:01.307 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T22:10:01.307 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-42949673049 2024-09-15T22:10:01.307 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T22:10:01.308 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-15T22:10:01.309 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-42949673049 2024-09-15T22:10:01.309 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T22:10:01.310 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=42949673049 2024-09-15T22:10:01.310 INFO:tasks.workunit.client.0.smithi136.stdout:waiting osd.1 seq 42949673049 2024-09-15T22:10:01.310 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 42949673049' 2024-09-15T22:10:01.311 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-15T22:10:01.584 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 42949673050 -lt 42949673049 2024-09-15T22:10:01.584 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T22:10:01.585 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-64424509527 2024-09-15T22:10:01.585 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T22:10:01.585 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-15T22:10:01.586 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-64424509527 2024-09-15T22:10:01.586 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T22:10:01.587 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=64424509527 2024-09-15T22:10:01.587 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 64424509527' 2024-09-15T22:10:01.588 INFO:tasks.workunit.client.0.smithi136.stdout:waiting osd.2 seq 64424509527 2024-09-15T22:10:01.588 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-15T22:10:01.864 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 64424509527 -lt 64424509527 2024-09-15T22:10:01.865 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:380: _scrub_abort: ceph pg dump pgs 2024-09-15T22:10:01.865 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:380: _scrub_abort: grep '^1.0' 2024-09-15T22:10:01.865 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:380: _scrub_abort: grep -q scrubbing 2024-09-15T22:10:02.123 INFO:tasks.workunit.client.0.smithi136.stderr:dumped pgs 2024-09-15T22:10:02.134 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:382: _scrub_abort: continue 2024-09-15T22:10:02.134 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:377: _scrub_abort: for i in $(seq 0 200) 2024-09-15T22:10:02.135 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:379: _scrub_abort: flush_pg_stats 2024-09-15T22:10:02.135 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-15T22:10:02.135 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-15T22:10:02.432 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-15T22:10:02.432 INFO:tasks.workunit.client.0.smithi136.stderr:1 2024-09-15T22:10:02.432 INFO:tasks.workunit.client.0.smithi136.stderr:2' 2024-09-15T22:10:02.432 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-15T22:10:02.433 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T22:10:02.433 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-15T22:10:02.549 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836574 2024-09-15T22:10:02.549 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836574 2024-09-15T22:10:02.549 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836574' 2024-09-15T22:10:02.549 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T22:10:02.550 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-15T22:10:02.649 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=42949673051 2024-09-15T22:10:02.649 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 42949673051 2024-09-15T22:10:02.650 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836574 1-42949673051' 2024-09-15T22:10:02.650 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T22:10:02.650 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-15T22:10:02.750 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=64424509529 2024-09-15T22:10:02.750 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 64424509529 2024-09-15T22:10:02.750 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836574 1-42949673051 2-64424509529' 2024-09-15T22:10:02.751 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T22:10:02.751 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836574 2024-09-15T22:10:02.751 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T22:10:02.752 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-15T22:10:02.753 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836574 2024-09-15T22:10:02.753 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T22:10:02.753 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836574 2024-09-15T22:10:02.754 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836574' 2024-09-15T22:10:02.754 INFO:tasks.workunit.client.0.smithi136.stdout:waiting osd.0 seq 21474836574 2024-09-15T22:10:02.754 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T22:10:03.026 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836572 -lt 21474836574 2024-09-15T22:10:03.026 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-15T22:10:04.027 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-15T22:10:04.027 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T22:10:04.318 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836574 -lt 21474836574 2024-09-15T22:10:04.318 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T22:10:04.319 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-42949673051 2024-09-15T22:10:04.319 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T22:10:04.320 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-15T22:10:04.320 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-42949673051 2024-09-15T22:10:04.320 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T22:10:04.321 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=42949673051 2024-09-15T22:10:04.322 INFO:tasks.workunit.client.0.smithi136.stdout:waiting osd.1 seq 42949673051 2024-09-15T22:10:04.322 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 42949673051' 2024-09-15T22:10:04.322 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-15T22:10:04.596 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 42949673051 -lt 42949673051 2024-09-15T22:10:04.596 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T22:10:04.596 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-64424509529 2024-09-15T22:10:04.596 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T22:10:04.597 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-15T22:10:04.597 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-64424509529 2024-09-15T22:10:04.597 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T22:10:04.598 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=64424509529 2024-09-15T22:10:04.598 INFO:tasks.workunit.client.0.smithi136.stdout:waiting osd.2 seq 64424509529 2024-09-15T22:10:04.598 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 64424509529' 2024-09-15T22:10:04.598 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-15T22:10:04.874 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 64424509529 -lt 64424509529 2024-09-15T22:10:04.874 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:380: _scrub_abort: ceph pg dump pgs 2024-09-15T22:10:04.874 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:380: _scrub_abort: grep '^1.0' 2024-09-15T22:10:04.874 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:380: _scrub_abort: grep -q scrubbing 2024-09-15T22:10:05.134 INFO:tasks.workunit.client.0.smithi136.stderr:dumped pgs 2024-09-15T22:10:05.145 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:382: _scrub_abort: continue 2024-09-15T22:10:05.146 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:377: _scrub_abort: for i in $(seq 0 200) 2024-09-15T22:10:05.146 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:379: _scrub_abort: flush_pg_stats 2024-09-15T22:10:05.146 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-15T22:10:05.146 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-15T22:10:05.420 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-15T22:10:05.420 INFO:tasks.workunit.client.0.smithi136.stderr:1 2024-09-15T22:10:05.420 INFO:tasks.workunit.client.0.smithi136.stderr:2' 2024-09-15T22:10:05.420 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-15T22:10:05.421 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T22:10:05.421 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-15T22:10:05.520 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836575 2024-09-15T22:10:05.520 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836575 2024-09-15T22:10:05.521 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836575' 2024-09-15T22:10:05.521 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T22:10:05.521 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-15T22:10:05.621 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=42949673053 2024-09-15T22:10:05.621 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 42949673053 2024-09-15T22:10:05.621 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836575 1-42949673053' 2024-09-15T22:10:05.622 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T22:10:05.622 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-15T22:10:05.722 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=64424509530 2024-09-15T22:10:05.723 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 64424509530 2024-09-15T22:10:05.723 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836575 1-42949673053 2-64424509530' 2024-09-15T22:10:05.723 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T22:10:05.723 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836575 2024-09-15T22:10:05.723 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T22:10:05.724 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-15T22:10:05.725 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836575 2024-09-15T22:10:05.725 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T22:10:05.726 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836575 2024-09-15T22:10:05.726 INFO:tasks.workunit.client.0.smithi136.stdout:waiting osd.0 seq 21474836575 2024-09-15T22:10:05.727 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836575' 2024-09-15T22:10:05.727 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T22:10:06.001 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836574 -lt 21474836575 2024-09-15T22:10:06.001 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-15T22:10:07.002 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-15T22:10:07.002 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T22:10:07.290 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836575 -lt 21474836575 2024-09-15T22:10:07.290 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T22:10:07.290 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-42949673053 2024-09-15T22:10:07.290 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T22:10:07.291 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-15T22:10:07.292 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-42949673053 2024-09-15T22:10:07.292 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T22:10:07.293 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=42949673053 2024-09-15T22:10:07.293 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 42949673053' 2024-09-15T22:10:07.293 INFO:tasks.workunit.client.0.smithi136.stdout:waiting osd.1 seq 42949673053 2024-09-15T22:10:07.293 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-15T22:10:07.571 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 42949673053 -lt 42949673053 2024-09-15T22:10:07.571 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T22:10:07.571 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-64424509530 2024-09-15T22:10:07.571 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T22:10:07.572 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-15T22:10:07.573 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-64424509530 2024-09-15T22:10:07.573 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T22:10:07.574 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=64424509530 2024-09-15T22:10:07.574 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 64424509530' 2024-09-15T22:10:07.574 INFO:tasks.workunit.client.0.smithi136.stdout:waiting osd.2 seq 64424509530 2024-09-15T22:10:07.574 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-15T22:10:07.848 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 64424509530 -lt 64424509530 2024-09-15T22:10:07.848 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:380: _scrub_abort: ceph pg dump pgs 2024-09-15T22:10:07.849 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:380: _scrub_abort: grep '^1.0' 2024-09-15T22:10:07.849 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:380: _scrub_abort: grep -q scrubbing 2024-09-15T22:10:08.103 INFO:tasks.workunit.client.0.smithi136.stderr:dumped pgs 2024-09-15T22:10:08.115 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:382: _scrub_abort: continue 2024-09-15T22:10:08.115 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:377: _scrub_abort: for i in $(seq 0 200) 2024-09-15T22:10:08.115 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:379: _scrub_abort: flush_pg_stats 2024-09-15T22:10:08.115 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-15T22:10:08.116 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-15T22:10:08.394 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-15T22:10:08.394 INFO:tasks.workunit.client.0.smithi136.stderr:1 2024-09-15T22:10:08.394 INFO:tasks.workunit.client.0.smithi136.stderr:2' 2024-09-15T22:10:08.394 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-15T22:10:08.394 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T22:10:08.395 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-15T22:10:08.495 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836577 2024-09-15T22:10:08.495 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836577 2024-09-15T22:10:08.495 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836577' 2024-09-15T22:10:08.495 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T22:10:08.496 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-15T22:10:08.596 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=42949673054 2024-09-15T22:10:08.596 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 42949673054 2024-09-15T22:10:08.596 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836577 1-42949673054' 2024-09-15T22:10:08.596 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T22:10:08.596 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-15T22:10:08.697 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=64424509532 2024-09-15T22:10:08.697 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 64424509532 2024-09-15T22:10:08.697 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836577 1-42949673054 2-64424509532' 2024-09-15T22:10:08.697 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T22:10:08.698 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836577 2024-09-15T22:10:08.698 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T22:10:08.698 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-15T22:10:08.699 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836577 2024-09-15T22:10:08.699 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T22:10:08.700 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836577 2024-09-15T22:10:08.700 INFO:tasks.workunit.client.0.smithi136.stdout:waiting osd.0 seq 21474836577 2024-09-15T22:10:08.700 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836577' 2024-09-15T22:10:08.700 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T22:10:08.974 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836576 -lt 21474836577 2024-09-15T22:10:08.974 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-15T22:10:09.975 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-15T22:10:09.975 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T22:10:10.264 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836576 -lt 21474836577 2024-09-15T22:10:10.264 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-15T22:10:11.265 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 299 -eq 0 ']' 2024-09-15T22:10:11.265 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T22:10:11.549 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836577 -lt 21474836577 2024-09-15T22:10:11.549 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T22:10:11.550 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-42949673054 2024-09-15T22:10:11.550 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T22:10:11.552 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-15T22:10:11.552 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-42949673054 2024-09-15T22:10:11.552 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T22:10:11.552 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=42949673054 2024-09-15T22:10:11.552 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 42949673054' 2024-09-15T22:10:11.553 INFO:tasks.workunit.client.0.smithi136.stdout:waiting osd.1 seq 42949673054 2024-09-15T22:10:11.553 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-15T22:10:11.827 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 42949673055 -lt 42949673054 2024-09-15T22:10:11.828 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T22:10:11.828 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-64424509532 2024-09-15T22:10:11.828 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T22:10:11.829 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-15T22:10:11.829 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-64424509532 2024-09-15T22:10:11.830 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T22:10:11.831 INFO:tasks.workunit.client.0.smithi136.stdout:waiting osd.2 seq 64424509532 2024-09-15T22:10:11.831 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=64424509532 2024-09-15T22:10:11.831 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 64424509532' 2024-09-15T22:10:11.831 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-15T22:10:12.106 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 64424509532 -lt 64424509532 2024-09-15T22:10:12.106 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:380: _scrub_abort: ceph pg dump pgs 2024-09-15T22:10:12.106 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:380: _scrub_abort: grep '^1.0' 2024-09-15T22:10:12.107 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:380: _scrub_abort: grep -q scrubbing 2024-09-15T22:10:12.368 INFO:tasks.workunit.client.0.smithi136.stderr:dumped pgs 2024-09-15T22:10:12.380 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:382: _scrub_abort: continue 2024-09-15T22:10:12.381 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:377: _scrub_abort: for i in $(seq 0 200) 2024-09-15T22:10:12.381 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:379: _scrub_abort: flush_pg_stats 2024-09-15T22:10:12.381 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-15T22:10:12.381 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-15T22:10:12.654 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-15T22:10:12.654 INFO:tasks.workunit.client.0.smithi136.stderr:1 2024-09-15T22:10:12.654 INFO:tasks.workunit.client.0.smithi136.stderr:2' 2024-09-15T22:10:12.654 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-15T22:10:12.654 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T22:10:12.654 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-15T22:10:12.754 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836579 2024-09-15T22:10:12.754 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836579 2024-09-15T22:10:12.754 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836579' 2024-09-15T22:10:12.754 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T22:10:12.754 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-15T22:10:12.855 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=42949673056 2024-09-15T22:10:12.855 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 42949673056 2024-09-15T22:10:12.856 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836579 1-42949673056' 2024-09-15T22:10:12.856 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T22:10:12.856 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-15T22:10:12.955 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=64424509534 2024-09-15T22:10:12.955 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 64424509534 2024-09-15T22:10:12.956 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836579 1-42949673056 2-64424509534' 2024-09-15T22:10:12.956 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T22:10:12.956 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836579 2024-09-15T22:10:12.956 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T22:10:12.957 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-15T22:10:12.957 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836579 2024-09-15T22:10:12.958 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T22:10:12.958 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836579 2024-09-15T22:10:12.959 INFO:tasks.workunit.client.0.smithi136.stdout:waiting osd.0 seq 21474836579 2024-09-15T22:10:12.959 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836579' 2024-09-15T22:10:12.959 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T22:10:13.232 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836577 -lt 21474836579 2024-09-15T22:10:13.232 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-15T22:10:14.234 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-15T22:10:14.234 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T22:10:14.512 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836579 -lt 21474836579 2024-09-15T22:10:14.512 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T22:10:14.512 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-42949673056 2024-09-15T22:10:14.513 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T22:10:14.513 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-15T22:10:14.514 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-42949673056 2024-09-15T22:10:14.514 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T22:10:14.515 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=42949673056 2024-09-15T22:10:14.515 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 42949673056' 2024-09-15T22:10:14.515 INFO:tasks.workunit.client.0.smithi136.stdout:waiting osd.1 seq 42949673056 2024-09-15T22:10:14.516 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-15T22:10:14.791 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 42949673056 -lt 42949673056 2024-09-15T22:10:14.791 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T22:10:14.791 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-64424509534 2024-09-15T22:10:14.791 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T22:10:14.792 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-15T22:10:14.793 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-64424509534 2024-09-15T22:10:14.793 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T22:10:14.794 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=64424509534 2024-09-15T22:10:14.794 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 64424509534' 2024-09-15T22:10:14.794 INFO:tasks.workunit.client.0.smithi136.stdout:waiting osd.2 seq 64424509534 2024-09-15T22:10:14.794 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-15T22:10:15.067 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 64424509534 -lt 64424509534 2024-09-15T22:10:15.068 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:380: _scrub_abort: ceph pg dump pgs 2024-09-15T22:10:15.068 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:380: _scrub_abort: grep '^1.0' 2024-09-15T22:10:15.068 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:380: _scrub_abort: grep -q scrubbing 2024-09-15T22:10:15.325 INFO:tasks.workunit.client.0.smithi136.stderr:dumped pgs 2024-09-15T22:10:15.336 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:382: _scrub_abort: continue 2024-09-15T22:10:15.336 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:377: _scrub_abort: for i in $(seq 0 200) 2024-09-15T22:10:15.336 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:379: _scrub_abort: flush_pg_stats 2024-09-15T22:10:15.336 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-15T22:10:15.336 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-15T22:10:15.610 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-15T22:10:15.610 INFO:tasks.workunit.client.0.smithi136.stderr:1 2024-09-15T22:10:15.610 INFO:tasks.workunit.client.0.smithi136.stderr:2' 2024-09-15T22:10:15.610 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-15T22:10:15.610 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T22:10:15.610 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-15T22:10:15.711 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836580 2024-09-15T22:10:15.711 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836580 2024-09-15T22:10:15.711 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836580' 2024-09-15T22:10:15.711 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T22:10:15.711 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-15T22:10:15.810 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=42949673058 2024-09-15T22:10:15.810 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 42949673058 2024-09-15T22:10:15.810 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836580 1-42949673058' 2024-09-15T22:10:15.810 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T22:10:15.811 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-15T22:10:15.910 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=64424509535 2024-09-15T22:10:15.910 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 64424509535 2024-09-15T22:10:15.911 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836580 1-42949673058 2-64424509535' 2024-09-15T22:10:15.911 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T22:10:15.911 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836580 2024-09-15T22:10:15.911 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T22:10:15.912 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-15T22:10:15.912 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836580 2024-09-15T22:10:15.912 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T22:10:15.913 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836580 2024-09-15T22:10:15.913 INFO:tasks.workunit.client.0.smithi136.stdout:waiting osd.0 seq 21474836580 2024-09-15T22:10:15.914 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836580' 2024-09-15T22:10:15.914 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T22:10:16.189 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836579 -lt 21474836580 2024-09-15T22:10:16.189 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-15T22:10:17.190 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-15T22:10:17.190 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T22:10:17.472 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836580 -lt 21474836580 2024-09-15T22:10:17.472 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T22:10:17.473 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-42949673058 2024-09-15T22:10:17.473 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T22:10:17.474 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-15T22:10:17.474 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-42949673058 2024-09-15T22:10:17.474 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T22:10:17.475 INFO:tasks.workunit.client.0.smithi136.stdout:waiting osd.1 seq 42949673058 2024-09-15T22:10:17.476 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=42949673058 2024-09-15T22:10:17.476 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 42949673058' 2024-09-15T22:10:17.476 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-15T22:10:17.749 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 42949673058 -lt 42949673058 2024-09-15T22:10:17.749 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T22:10:17.749 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-64424509535 2024-09-15T22:10:17.750 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T22:10:17.751 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-15T22:10:17.751 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-64424509535 2024-09-15T22:10:17.751 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T22:10:17.752 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=64424509535 2024-09-15T22:10:17.752 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 64424509535' 2024-09-15T22:10:17.752 INFO:tasks.workunit.client.0.smithi136.stdout:waiting osd.2 seq 64424509535 2024-09-15T22:10:17.753 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-15T22:10:18.027 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 64424509535 -lt 64424509535 2024-09-15T22:10:18.027 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:380: _scrub_abort: ceph pg dump pgs 2024-09-15T22:10:18.027 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:380: _scrub_abort: grep '^1.0' 2024-09-15T22:10:18.027 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:380: _scrub_abort: grep -q scrubbing 2024-09-15T22:10:18.287 INFO:tasks.workunit.client.0.smithi136.stderr:dumped pgs 2024-09-15T22:10:18.299 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:382: _scrub_abort: continue 2024-09-15T22:10:18.299 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:377: _scrub_abort: for i in $(seq 0 200) 2024-09-15T22:10:18.299 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:379: _scrub_abort: flush_pg_stats 2024-09-15T22:10:18.299 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-15T22:10:18.299 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-15T22:10:18.572 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-15T22:10:18.573 INFO:tasks.workunit.client.0.smithi136.stderr:1 2024-09-15T22:10:18.573 INFO:tasks.workunit.client.0.smithi136.stderr:2' 2024-09-15T22:10:18.573 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-15T22:10:18.573 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T22:10:18.573 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-15T22:10:18.673 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836582 2024-09-15T22:10:18.673 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836582 2024-09-15T22:10:18.673 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836582' 2024-09-15T22:10:18.673 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T22:10:18.673 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-15T22:10:18.774 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=42949673059 2024-09-15T22:10:18.774 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 42949673059 2024-09-15T22:10:18.774 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836582 1-42949673059' 2024-09-15T22:10:18.774 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T22:10:18.774 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-15T22:10:18.874 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=64424509537 2024-09-15T22:10:18.874 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 64424509537 2024-09-15T22:10:18.874 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836582 1-42949673059 2-64424509537' 2024-09-15T22:10:18.874 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T22:10:18.875 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836582 2024-09-15T22:10:18.875 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T22:10:18.875 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-15T22:10:18.876 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836582 2024-09-15T22:10:18.876 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T22:10:18.877 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836582 2024-09-15T22:10:18.877 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836582' 2024-09-15T22:10:18.877 INFO:tasks.workunit.client.0.smithi136.stdout:waiting osd.0 seq 21474836582 2024-09-15T22:10:18.877 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T22:10:19.149 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836581 -lt 21474836582 2024-09-15T22:10:19.149 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-15T22:10:20.150 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-15T22:10:20.150 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T22:10:20.439 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836582 -lt 21474836582 2024-09-15T22:10:20.439 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T22:10:20.439 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-42949673059 2024-09-15T22:10:20.439 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T22:10:20.440 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-15T22:10:20.441 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-42949673059 2024-09-15T22:10:20.441 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T22:10:20.442 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=42949673059 2024-09-15T22:10:20.442 INFO:tasks.workunit.client.0.smithi136.stdout:waiting osd.1 seq 42949673059 2024-09-15T22:10:20.442 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 42949673059' 2024-09-15T22:10:20.442 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-15T22:10:20.725 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 42949673060 -lt 42949673059 2024-09-15T22:10:20.725 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T22:10:20.726 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-64424509537 2024-09-15T22:10:20.726 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T22:10:20.727 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-15T22:10:20.727 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-64424509537 2024-09-15T22:10:20.727 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T22:10:20.728 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=64424509537 2024-09-15T22:10:20.729 INFO:tasks.workunit.client.0.smithi136.stdout:waiting osd.2 seq 64424509537 2024-09-15T22:10:20.729 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 64424509537' 2024-09-15T22:10:20.729 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-15T22:10:21.004 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 64424509537 -lt 64424509537 2024-09-15T22:10:21.004 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:380: _scrub_abort: ceph pg dump pgs 2024-09-15T22:10:21.004 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:380: _scrub_abort: grep '^1.0' 2024-09-15T22:10:21.004 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:380: _scrub_abort: grep -q scrubbing 2024-09-15T22:10:21.262 INFO:tasks.workunit.client.0.smithi136.stderr:dumped pgs 2024-09-15T22:10:21.273 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:385: _scrub_abort: break 2024-09-15T22:10:21.273 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:387: _scrub_abort: set +o pipefail 2024-09-15T22:10:21.273 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:389: _scrub_abort: sleep 5 2024-09-15T22:10:26.275 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:391: _scrub_abort: grep 'nodeep_scrub set, aborting' td/osd-scrub-test/osd.1.log 2024-09-15T22:10:26.290 INFO:tasks.workunit.client.0.smithi136.stdout:2024-09-15T22:08:27.746+0000 7f586b2cf640 10 osd.1 pg_epoch: 21 pg[1.0( v 19'1000 (0'0,19'1000] local-lis/les=17/18 n=1000 ec=17/17 lis/c=17/17 les/c/f=18/18/0 sis=17) [1,0,2] r=0 lpr=17 crt=19'1000 lcod 19'999 mlcod 19'999 active+clean+scrubbing+deep [ 1.0: ] ] scrubber: nodeep_scrub set, aborting 2024-09-15T22:10:26.291 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:397: _scrub_abort: get_last_scrub_stamp 1.0 2024-09-15T22:10:26.291 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-15T22:10:26.291 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-15T22:10:26.292 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-15T22:10:26.292 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-15T22:10:26.570 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:397: _scrub_abort: local last_scrub=2024-09-01T22:06:34.123489+0000 2024-09-15T22:10:26.570 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:398: _scrub_abort: ceph config set osd osd_scrub_sleep 0.1 2024-09-15T22:10:26.919 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:400: _scrub_abort: ceph osd unset nodeep-scrub 2024-09-15T22:10:27.307 INFO:tasks.workunit.client.0.smithi136.stderr:nodeep-scrub is unset 2024-09-15T22:10:27.318 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:401: _scrub_abort: '[' deep-scrub = deep-scrub ']' 2024-09-15T22:10:27.318 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:403: _scrub_abort: ceph osd unset noscrub 2024-09-15T22:10:27.834 INFO:tasks.workunit.client.0.smithi136.stderr:noscrub is unset 2024-09-15T22:10:27.847 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:405: _scrub_abort: TIMEOUT=500 2024-09-15T22:10:27.847 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:406: _scrub_abort: wait_for_scrub 1.0 2024-09-01T22:06:34.123489+0000 2024-09-15T22:10:27.847 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: local pgid=1.0 2024-09-15T22:10:27.847 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2018: wait_for_scrub: local last_scrub=2024-09-01T22:06:34.123489+0000 2024-09-15T22:10:27.847 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2019: wait_for_scrub: local sname=last_scrub_stamp 2024-09-15T22:10:27.847 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i=0 )) 2024-09-15T22:10:27.847 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-15T22:10:27.847 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-15T22:10:27.848 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-15T22:10:27.848 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-15T22:10:27.848 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-15T22:10:27.848 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-15T22:10:28.126 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-01T22:06:34.123489+0000 '>' 2024-09-01T22:06:34.123489+0000 2024-09-15T22:10:28.126 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-15T22:10:29.127 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-15T22:10:29.128 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-15T22:10:29.128 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-15T22:10:29.129 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-15T22:10:29.129 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-15T22:10:29.129 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-15T22:10:29.129 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-15T22:10:29.410 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-01T22:06:34.123489+0000 '>' 2024-09-01T22:06:34.123489+0000 2024-09-15T22:10:29.410 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-15T22:10:30.411 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-15T22:10:30.411 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-15T22:10:30.412 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-15T22:10:30.412 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-15T22:10:30.412 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-15T22:10:30.412 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-15T22:10:30.412 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-15T22:10:30.692 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-01T22:06:34.123489+0000 '>' 2024-09-01T22:06:34.123489+0000 2024-09-15T22:10:30.692 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-15T22:10:31.693 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-15T22:10:31.693 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-15T22:10:31.694 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-15T22:10:31.694 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-15T22:10:31.694 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-15T22:10:31.694 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-15T22:10:31.694 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-15T22:10:31.972 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-01T22:06:34.123489+0000 '>' 2024-09-01T22:06:34.123489+0000 2024-09-15T22:10:31.972 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-15T22:10:32.973 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-15T22:10:32.973 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-15T22:10:32.974 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-15T22:10:32.974 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-15T22:10:32.974 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-15T22:10:32.975 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-15T22:10:32.975 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-15T22:10:33.252 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-01T22:06:34.123489+0000 '>' 2024-09-01T22:06:34.123489+0000 2024-09-15T22:10:33.252 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-15T22:10:34.253 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-15T22:10:34.253 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-15T22:10:34.254 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-15T22:10:34.254 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-15T22:10:34.254 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-15T22:10:34.254 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-15T22:10:34.254 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-15T22:10:34.536 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-01T22:06:34.123489+0000 '>' 2024-09-01T22:06:34.123489+0000 2024-09-15T22:10:34.536 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-15T22:10:35.537 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-15T22:10:35.537 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-15T22:10:35.538 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-15T22:10:35.538 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-15T22:10:35.538 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-15T22:10:35.538 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-15T22:10:35.538 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-15T22:10:35.817 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-01T22:06:34.123489+0000 '>' 2024-09-01T22:06:34.123489+0000 2024-09-15T22:10:35.817 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-15T22:10:36.818 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-15T22:10:36.818 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-15T22:10:36.819 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-15T22:10:36.819 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-15T22:10:36.820 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-15T22:10:36.820 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-15T22:10:36.820 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-15T22:10:37.102 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-01T22:06:34.123489+0000 '>' 2024-09-01T22:06:34.123489+0000 2024-09-15T22:10:37.102 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-15T22:10:38.104 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-15T22:10:38.104 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-15T22:10:38.105 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-15T22:10:38.105 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-15T22:10:38.105 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-15T22:10:38.105 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-15T22:10:38.105 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-15T22:10:38.383 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-01T22:06:34.123489+0000 '>' 2024-09-01T22:06:34.123489+0000 2024-09-15T22:10:38.383 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-15T22:10:39.384 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-15T22:10:39.384 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-15T22:10:39.385 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-15T22:10:39.385 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-15T22:10:39.385 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-15T22:10:39.385 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-15T22:10:39.385 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-15T22:10:39.666 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-01T22:06:34.123489+0000 '>' 2024-09-01T22:06:34.123489+0000 2024-09-15T22:10:39.666 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-15T22:10:40.667 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-15T22:10:40.667 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-15T22:10:40.668 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-15T22:10:40.668 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-15T22:10:40.668 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-15T22:10:40.668 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-15T22:10:40.668 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-15T22:10:40.948 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-01T22:06:34.123489+0000 '>' 2024-09-01T22:06:34.123489+0000 2024-09-15T22:10:40.948 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-15T22:10:41.949 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-15T22:10:41.949 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-15T22:10:41.950 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-15T22:10:41.950 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-15T22:10:41.950 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-15T22:10:41.950 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-15T22:10:41.950 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-15T22:10:42.231 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-01T22:06:34.123489+0000 '>' 2024-09-01T22:06:34.123489+0000 2024-09-15T22:10:42.231 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-15T22:10:43.232 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-15T22:10:43.232 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-15T22:10:43.233 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-15T22:10:43.233 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-15T22:10:43.233 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-15T22:10:43.233 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-15T22:10:43.233 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-15T22:10:43.552 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-01T22:06:34.123489+0000 '>' 2024-09-01T22:06:34.123489+0000 2024-09-15T22:10:43.552 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-15T22:10:44.553 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-15T22:10:44.553 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-15T22:10:44.554 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-15T22:10:44.554 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-15T22:10:44.554 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-15T22:10:44.554 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-15T22:10:44.554 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-15T22:10:44.837 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-01T22:06:34.123489+0000 '>' 2024-09-01T22:06:34.123489+0000 2024-09-15T22:10:44.837 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-15T22:10:45.838 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-15T22:10:45.838 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-15T22:10:45.839 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-15T22:10:45.839 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-15T22:10:45.839 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-15T22:10:45.839 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-15T22:10:45.839 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-15T22:10:46.118 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-01T22:06:34.123489+0000 '>' 2024-09-01T22:06:34.123489+0000 2024-09-15T22:10:46.118 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-15T22:10:47.119 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-15T22:10:47.119 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-15T22:10:47.120 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-15T22:10:47.120 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-15T22:10:47.120 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-15T22:10:47.120 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-15T22:10:47.120 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-15T22:10:47.403 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-01T22:06:34.123489+0000 '>' 2024-09-01T22:06:34.123489+0000 2024-09-15T22:10:47.403 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-15T22:10:48.404 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-15T22:10:48.405 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-15T22:10:48.405 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-15T22:10:48.405 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-15T22:10:48.405 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-15T22:10:48.406 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-15T22:10:48.406 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-15T22:10:48.686 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-01T22:06:34.123489+0000 '>' 2024-09-01T22:06:34.123489+0000 2024-09-15T22:10:48.686 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-15T22:10:49.687 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-15T22:10:49.687 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-15T22:10:49.688 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-15T22:10:49.688 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-15T22:10:49.688 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-15T22:10:49.688 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-15T22:10:49.688 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-15T22:10:49.967 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-01T22:06:34.123489+0000 '>' 2024-09-01T22:06:34.123489+0000 2024-09-15T22:10:49.967 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-15T22:10:50.968 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-15T22:10:50.968 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-15T22:10:50.969 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-15T22:10:50.969 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-15T22:10:50.969 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-15T22:10:50.969 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-15T22:10:50.969 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-15T22:10:51.249 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-01T22:06:34.123489+0000 '>' 2024-09-01T22:06:34.123489+0000 2024-09-15T22:10:51.250 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-15T22:10:52.250 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-15T22:10:52.250 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-15T22:10:52.251 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-15T22:10:52.251 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-15T22:10:52.251 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-15T22:10:52.252 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-15T22:10:52.252 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-15T22:10:52.533 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-01T22:06:34.123489+0000 '>' 2024-09-01T22:06:34.123489+0000 2024-09-15T22:10:52.533 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-15T22:10:53.534 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-15T22:10:53.535 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-15T22:10:53.535 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-15T22:10:53.535 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-15T22:10:53.536 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-15T22:10:53.536 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-15T22:10:53.536 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-15T22:10:53.818 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-01T22:06:34.123489+0000 '>' 2024-09-01T22:06:34.123489+0000 2024-09-15T22:10:53.819 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-15T22:10:54.820 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-15T22:10:54.820 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-15T22:10:54.821 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-15T22:10:54.821 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-15T22:10:54.821 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-15T22:10:54.821 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-15T22:10:54.821 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-15T22:10:55.098 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-01T22:06:34.123489+0000 '>' 2024-09-01T22:06:34.123489+0000 2024-09-15T22:10:55.098 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-15T22:10:56.099 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-15T22:10:56.099 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-15T22:10:56.100 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-15T22:10:56.100 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-15T22:10:56.100 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-15T22:10:56.100 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-15T22:10:56.100 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-15T22:10:56.385 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-01T22:06:34.123489+0000 '>' 2024-09-01T22:06:34.123489+0000 2024-09-15T22:10:56.385 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-15T22:10:57.386 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-15T22:10:57.386 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-15T22:10:57.387 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-15T22:10:57.387 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-15T22:10:57.387 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-15T22:10:57.387 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-15T22:10:57.387 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-15T22:10:57.668 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-01T22:06:34.123489+0000 '>' 2024-09-01T22:06:34.123489+0000 2024-09-15T22:10:57.668 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-15T22:10:58.668 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-15T22:10:58.669 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-15T22:10:58.669 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-15T22:10:58.670 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-15T22:10:58.670 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-15T22:10:58.670 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-15T22:10:58.670 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-15T22:10:58.949 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-01T22:06:34.123489+0000 '>' 2024-09-01T22:06:34.123489+0000 2024-09-15T22:10:58.949 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-15T22:10:59.950 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-15T22:10:59.950 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-15T22:10:59.950 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-15T22:10:59.950 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-15T22:10:59.951 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-15T22:10:59.951 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-15T22:10:59.951 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-15T22:11:00.232 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-01T22:06:34.123489+0000 '>' 2024-09-01T22:06:34.123489+0000 2024-09-15T22:11:00.233 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-15T22:11:01.233 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-15T22:11:01.233 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-15T22:11:01.234 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-15T22:11:01.234 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-15T22:11:01.234 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-15T22:11:01.234 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-15T22:11:01.234 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-15T22:11:01.510 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-01T22:06:34.123489+0000 '>' 2024-09-01T22:06:34.123489+0000 2024-09-15T22:11:01.511 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-15T22:11:02.512 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-15T22:11:02.512 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-15T22:11:02.512 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-15T22:11:02.513 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-15T22:11:02.513 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-15T22:11:02.513 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-15T22:11:02.513 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-15T22:11:02.792 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-01T22:06:34.123489+0000 '>' 2024-09-01T22:06:34.123489+0000 2024-09-15T22:11:02.792 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-15T22:11:03.793 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-15T22:11:03.793 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-15T22:11:03.794 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-15T22:11:03.794 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-15T22:11:03.794 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-15T22:11:03.794 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-15T22:11:03.795 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-15T22:11:04.074 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-01T22:06:34.123489+0000 '>' 2024-09-01T22:06:34.123489+0000 2024-09-15T22:11:04.074 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-15T22:11:05.075 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-15T22:11:05.075 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-15T22:11:05.076 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-15T22:11:05.076 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-15T22:11:05.076 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-15T22:11:05.076 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-15T22:11:05.076 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-15T22:11:05.356 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-01T22:06:34.123489+0000 '>' 2024-09-01T22:06:34.123489+0000 2024-09-15T22:11:05.356 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-15T22:11:06.357 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-15T22:11:06.357 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-15T22:11:06.358 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-15T22:11:06.358 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-15T22:11:06.358 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-15T22:11:06.358 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-15T22:11:06.358 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-15T22:11:06.634 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-01T22:06:34.123489+0000 '>' 2024-09-01T22:06:34.123489+0000 2024-09-15T22:11:06.635 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-15T22:11:07.636 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-15T22:11:07.636 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-15T22:11:07.636 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-15T22:11:07.637 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-15T22:11:07.637 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-15T22:11:07.637 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-15T22:11:07.637 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-15T22:11:07.914 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-01T22:06:34.123489+0000 '>' 2024-09-01T22:06:34.123489+0000 2024-09-15T22:11:07.914 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-15T22:11:08.915 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-15T22:11:08.915 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-15T22:11:08.915 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-15T22:11:08.916 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-15T22:11:08.916 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-15T22:11:08.916 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-15T22:11:08.916 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-15T22:11:09.198 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-01T22:06:34.123489+0000 '>' 2024-09-01T22:06:34.123489+0000 2024-09-15T22:11:09.198 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-15T22:11:10.199 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-15T22:11:10.199 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-15T22:11:10.200 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-15T22:11:10.200 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-15T22:11:10.200 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-15T22:11:10.201 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-15T22:11:10.201 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-15T22:11:10.475 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-01T22:06:34.123489+0000 '>' 2024-09-01T22:06:34.123489+0000 2024-09-15T22:11:10.475 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-15T22:11:11.476 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-15T22:11:11.476 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-15T22:11:11.477 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-15T22:11:11.477 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-15T22:11:11.477 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-15T22:11:11.478 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-15T22:11:11.478 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-15T22:11:11.757 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-01T22:06:34.123489+0000 '>' 2024-09-01T22:06:34.123489+0000 2024-09-15T22:11:11.757 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-15T22:11:12.758 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-15T22:11:12.758 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-15T22:11:12.758 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-15T22:11:12.759 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-15T22:11:12.759 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-15T22:11:12.759 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-15T22:11:12.759 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-15T22:11:13.041 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-01T22:06:34.123489+0000 '>' 2024-09-01T22:06:34.123489+0000 2024-09-15T22:11:13.041 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-15T22:11:14.041 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-15T22:11:14.042 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-15T22:11:14.042 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-15T22:11:14.042 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-15T22:11:14.042 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-15T22:11:14.043 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-15T22:11:14.043 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-15T22:11:14.324 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-01T22:06:34.123489+0000 '>' 2024-09-01T22:06:34.123489+0000 2024-09-15T22:11:14.325 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-15T22:11:15.325 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-15T22:11:15.325 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-15T22:11:15.326 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-15T22:11:15.326 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-15T22:11:15.326 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-15T22:11:15.326 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-15T22:11:15.326 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-15T22:11:15.607 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-01T22:06:34.123489+0000 '>' 2024-09-01T22:06:34.123489+0000 2024-09-15T22:11:15.607 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-15T22:11:16.608 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-15T22:11:16.608 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-15T22:11:16.609 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-15T22:11:16.609 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-15T22:11:16.609 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-15T22:11:16.609 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-15T22:11:16.609 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-15T22:11:16.883 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-01T22:06:34.123489+0000 '>' 2024-09-01T22:06:34.123489+0000 2024-09-15T22:11:16.883 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-15T22:11:17.885 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-15T22:11:17.885 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-15T22:11:17.885 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-15T22:11:17.886 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-15T22:11:17.886 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-15T22:11:17.886 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-15T22:11:17.886 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-15T22:11:18.166 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-01T22:06:34.123489+0000 '>' 2024-09-01T22:06:34.123489+0000 2024-09-15T22:11:18.166 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-15T22:11:19.167 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-15T22:11:19.167 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-15T22:11:19.168 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-15T22:11:19.168 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-15T22:11:19.168 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-15T22:11:19.169 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-15T22:11:19.169 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-15T22:11:19.449 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-01T22:06:34.123489+0000 '>' 2024-09-01T22:06:34.123489+0000 2024-09-15T22:11:19.449 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-15T22:11:20.450 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-15T22:11:20.450 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-15T22:11:20.451 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-15T22:11:20.451 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-15T22:11:20.451 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-15T22:11:20.451 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-15T22:11:20.451 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-15T22:11:20.730 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-01T22:06:34.123489+0000 '>' 2024-09-01T22:06:34.123489+0000 2024-09-15T22:11:20.730 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-15T22:11:21.733 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-15T22:11:21.733 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-15T22:11:21.733 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-15T22:11:21.733 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-15T22:11:21.733 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-15T22:11:21.733 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-15T22:11:21.733 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-15T22:11:22.012 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-01T22:06:34.123489+0000 '>' 2024-09-01T22:06:34.123489+0000 2024-09-15T22:11:22.012 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-15T22:11:23.013 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-15T22:11:23.013 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-15T22:11:23.014 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-15T22:11:23.014 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-15T22:11:23.014 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-15T22:11:23.014 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-15T22:11:23.014 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-15T22:11:23.287 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-01T22:06:34.123489+0000 '>' 2024-09-01T22:06:34.123489+0000 2024-09-15T22:11:23.287 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-15T22:11:24.287 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-15T22:11:24.288 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-15T22:11:24.289 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-15T22:11:24.289 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-15T22:11:24.289 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-15T22:11:24.289 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-15T22:11:24.289 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-15T22:11:24.573 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-01T22:06:34.123489+0000 '>' 2024-09-01T22:06:34.123489+0000 2024-09-15T22:11:24.573 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-15T22:11:25.574 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-15T22:11:25.574 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-15T22:11:25.575 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-15T22:11:25.575 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-15T22:11:25.575 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-15T22:11:25.575 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-15T22:11:25.576 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-15T22:11:25.852 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-01T22:06:34.123489+0000 '>' 2024-09-01T22:06:34.123489+0000 2024-09-15T22:11:25.852 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-15T22:11:26.853 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-15T22:11:26.853 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-15T22:11:26.854 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-15T22:11:26.854 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-15T22:11:26.854 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-15T22:11:26.854 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-15T22:11:26.854 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-15T22:11:27.130 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-01T22:06:34.123489+0000 '>' 2024-09-01T22:06:34.123489+0000 2024-09-15T22:11:27.131 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-15T22:11:28.132 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-15T22:11:28.132 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-15T22:11:28.132 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-15T22:11:28.132 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-15T22:11:28.133 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-15T22:11:28.133 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-15T22:11:28.134 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-15T22:11:28.418 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-01T22:06:34.123489+0000 '>' 2024-09-01T22:06:34.123489+0000 2024-09-15T22:11:28.418 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-15T22:11:29.419 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-15T22:11:29.419 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-15T22:11:29.420 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-15T22:11:29.420 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-15T22:11:29.420 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-15T22:11:29.420 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-15T22:11:29.421 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-15T22:11:29.700 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-01T22:06:34.123489+0000 '>' 2024-09-01T22:06:34.123489+0000 2024-09-15T22:11:29.701 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-15T22:11:30.701 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-15T22:11:30.702 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-15T22:11:30.702 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-15T22:11:30.703 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-15T22:11:30.703 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-15T22:11:30.703 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-15T22:11:30.703 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-15T22:11:30.978 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-01T22:06:34.123489+0000 '>' 2024-09-01T22:06:34.123489+0000 2024-09-15T22:11:30.978 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-15T22:11:31.979 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-15T22:11:31.979 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-15T22:11:31.980 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-15T22:11:31.980 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-15T22:11:31.980 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-15T22:11:31.980 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-15T22:11:31.980 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-15T22:11:32.264 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-01T22:06:34.123489+0000 '>' 2024-09-01T22:06:34.123489+0000 2024-09-15T22:11:32.264 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-15T22:11:33.265 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-15T22:11:33.265 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-15T22:11:33.266 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-15T22:11:33.266 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-15T22:11:33.266 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-15T22:11:33.266 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-15T22:11:33.266 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-15T22:11:33.566 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-01T22:06:34.123489+0000 '>' 2024-09-01T22:06:34.123489+0000 2024-09-15T22:11:33.566 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-15T22:11:34.567 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-15T22:11:34.568 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-15T22:11:34.568 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-15T22:11:34.568 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-15T22:11:34.569 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-15T22:11:34.569 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-15T22:11:34.569 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-15T22:11:34.850 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-01T22:06:34.123489+0000 '>' 2024-09-01T22:06:34.123489+0000 2024-09-15T22:11:34.850 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-15T22:11:35.851 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-15T22:11:35.852 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-15T22:11:35.852 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-15T22:11:35.852 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-15T22:11:35.852 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-15T22:11:35.853 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-15T22:11:35.853 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-15T22:11:36.132 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-01T22:06:34.123489+0000 '>' 2024-09-01T22:06:34.123489+0000 2024-09-15T22:11:36.132 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-15T22:11:37.133 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-15T22:11:37.133 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-15T22:11:37.134 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-15T22:11:37.134 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-15T22:11:37.134 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-15T22:11:37.134 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-15T22:11:37.134 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-15T22:11:37.413 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-01T22:06:34.123489+0000 '>' 2024-09-01T22:06:34.123489+0000 2024-09-15T22:11:37.413 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-15T22:11:38.414 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-15T22:11:38.414 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-15T22:11:38.415 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-15T22:11:38.415 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-15T22:11:38.415 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-15T22:11:38.415 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-15T22:11:38.415 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-15T22:11:38.694 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-01T22:06:34.123489+0000 '>' 2024-09-01T22:06:34.123489+0000 2024-09-15T22:11:38.694 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-15T22:11:39.695 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-15T22:11:39.696 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-15T22:11:39.696 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-15T22:11:39.696 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-15T22:11:39.696 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-15T22:11:39.697 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-15T22:11:39.697 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-15T22:11:39.976 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-01T22:06:34.123489+0000 '>' 2024-09-01T22:06:34.123489+0000 2024-09-15T22:11:39.976 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-15T22:11:40.977 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-15T22:11:40.977 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-15T22:11:40.978 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-15T22:11:40.978 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-15T22:11:40.978 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-15T22:11:40.978 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-15T22:11:40.978 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-15T22:11:41.263 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-01T22:06:34.123489+0000 '>' 2024-09-01T22:06:34.123489+0000 2024-09-15T22:11:41.264 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-15T22:11:42.264 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-15T22:11:42.265 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-15T22:11:42.265 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-15T22:11:42.265 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-15T22:11:42.265 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-15T22:11:42.266 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-15T22:11:42.266 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-15T22:11:42.592 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-01T22:06:34.123489+0000 '>' 2024-09-01T22:06:34.123489+0000 2024-09-15T22:11:42.592 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-15T22:11:43.593 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-15T22:11:43.593 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-15T22:11:43.594 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-15T22:11:43.594 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-15T22:11:43.594 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-15T22:11:43.594 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-15T22:11:43.594 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-15T22:11:43.875 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-01T22:06:34.123489+0000 '>' 2024-09-01T22:06:34.123489+0000 2024-09-15T22:11:43.875 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-15T22:11:44.876 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-15T22:11:44.876 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-15T22:11:44.876 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-15T22:11:44.876 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-15T22:11:44.877 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-15T22:11:44.877 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-15T22:11:44.877 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-15T22:11:45.156 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-01T22:06:34.123489+0000 '>' 2024-09-01T22:06:34.123489+0000 2024-09-15T22:11:45.156 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-15T22:11:46.157 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-15T22:11:46.157 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-15T22:11:46.158 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-15T22:11:46.158 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-15T22:11:46.158 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-15T22:11:46.158 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-15T22:11:46.158 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-15T22:11:46.445 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-01T22:06:34.123489+0000 '>' 2024-09-01T22:06:34.123489+0000 2024-09-15T22:11:46.445 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-15T22:11:47.446 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-15T22:11:47.446 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-15T22:11:47.447 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-15T22:11:47.447 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-15T22:11:47.447 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-15T22:11:47.447 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-15T22:11:47.448 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-15T22:11:47.729 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-01T22:06:34.123489+0000 '>' 2024-09-01T22:06:34.123489+0000 2024-09-15T22:11:47.729 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-15T22:11:48.730 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-15T22:11:48.730 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-15T22:11:48.730 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-15T22:11:48.730 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-15T22:11:48.730 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-15T22:11:48.731 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-15T22:11:48.731 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-15T22:11:49.008 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-01T22:06:34.123489+0000 '>' 2024-09-01T22:06:34.123489+0000 2024-09-15T22:11:49.009 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-15T22:11:50.010 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-15T22:11:50.010 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-15T22:11:50.011 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-15T22:11:50.011 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-15T22:11:50.011 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-15T22:11:50.011 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-15T22:11:50.011 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-15T22:11:50.286 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-01T22:06:34.123489+0000 '>' 2024-09-01T22:06:34.123489+0000 2024-09-15T22:11:50.287 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-15T22:11:51.287 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-15T22:11:51.288 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-15T22:11:51.288 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-15T22:11:51.288 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-15T22:11:51.288 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-15T22:11:51.289 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-15T22:11:51.289 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-15T22:11:51.578 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-01T22:06:34.123489+0000 '>' 2024-09-01T22:06:34.123489+0000 2024-09-15T22:11:51.578 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-15T22:11:52.579 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-15T22:11:52.579 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-15T22:11:52.580 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-15T22:11:52.580 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-15T22:11:52.580 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-15T22:11:52.580 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-15T22:11:52.580 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-15T22:11:52.856 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-01T22:06:34.123489+0000 '>' 2024-09-01T22:06:34.123489+0000 2024-09-15T22:11:52.856 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-15T22:11:53.857 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-15T22:11:53.857 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-15T22:11:53.858 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-15T22:11:53.858 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-15T22:11:53.858 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-15T22:11:53.858 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-15T22:11:53.858 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-15T22:11:54.142 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-01T22:06:34.123489+0000 '>' 2024-09-01T22:06:34.123489+0000 2024-09-15T22:11:54.142 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-15T22:11:55.143 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-15T22:11:55.143 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-15T22:11:55.144 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-15T22:11:55.144 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-15T22:11:55.144 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-15T22:11:55.144 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-15T22:11:55.144 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-15T22:11:55.424 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-01T22:06:34.123489+0000 '>' 2024-09-01T22:06:34.123489+0000 2024-09-15T22:11:55.425 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-15T22:11:56.425 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-15T22:11:56.425 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-15T22:11:56.426 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-15T22:11:56.426 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-15T22:11:56.426 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-15T22:11:56.427 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-15T22:11:56.427 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-15T22:11:56.708 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-01T22:06:34.123489+0000 '>' 2024-09-01T22:06:34.123489+0000 2024-09-15T22:11:56.708 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-15T22:11:57.709 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-15T22:11:57.709 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-15T22:11:57.710 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-15T22:11:57.710 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-15T22:11:57.710 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-15T22:11:57.710 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-15T22:11:57.710 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-15T22:11:57.985 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-01T22:06:34.123489+0000 '>' 2024-09-01T22:06:34.123489+0000 2024-09-15T22:11:57.985 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-15T22:11:58.986 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-15T22:11:58.986 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-15T22:11:58.987 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-15T22:11:58.987 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-15T22:11:58.987 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-15T22:11:58.987 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-15T22:11:58.987 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-15T22:11:59.261 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-01T22:06:34.123489+0000 '>' 2024-09-01T22:06:34.123489+0000 2024-09-15T22:11:59.261 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-15T22:12:00.262 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-15T22:12:00.263 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-15T22:12:00.263 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-15T22:12:00.263 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-15T22:12:00.264 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-15T22:12:00.264 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-15T22:12:00.264 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-15T22:12:00.543 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-01T22:06:34.123489+0000 '>' 2024-09-01T22:06:34.123489+0000 2024-09-15T22:12:00.543 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-15T22:12:01.544 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-15T22:12:01.544 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-15T22:12:01.546 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-15T22:12:01.546 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-15T22:12:01.546 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-15T22:12:01.546 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-15T22:12:01.546 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-15T22:12:01.821 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-01T22:06:34.123489+0000 '>' 2024-09-01T22:06:34.123489+0000 2024-09-15T22:12:01.821 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-15T22:12:02.822 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-15T22:12:02.823 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-15T22:12:02.823 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-15T22:12:02.823 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-15T22:12:02.823 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-15T22:12:02.823 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-15T22:12:02.824 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-15T22:12:03.104 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-01T22:06:34.123489+0000 '>' 2024-09-01T22:06:34.123489+0000 2024-09-15T22:12:03.104 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-15T22:12:04.104 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-15T22:12:04.104 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-15T22:12:04.105 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-15T22:12:04.105 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-15T22:12:04.105 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-15T22:12:04.105 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-15T22:12:04.106 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-15T22:12:04.389 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-01T22:06:34.123489+0000 '>' 2024-09-01T22:06:34.123489+0000 2024-09-15T22:12:04.389 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-15T22:12:05.390 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-15T22:12:05.390 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-15T22:12:05.391 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-15T22:12:05.391 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-15T22:12:05.391 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-15T22:12:05.391 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-15T22:12:05.391 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-15T22:12:05.669 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-01T22:06:34.123489+0000 '>' 2024-09-01T22:06:34.123489+0000 2024-09-15T22:12:05.670 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-15T22:12:06.671 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-15T22:12:06.671 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-15T22:12:06.671 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-15T22:12:06.671 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-15T22:12:06.672 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-15T22:12:06.672 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-15T22:12:06.672 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-15T22:12:06.949 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-01T22:06:34.123489+0000 '>' 2024-09-01T22:06:34.123489+0000 2024-09-15T22:12:06.949 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-15T22:12:07.950 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-15T22:12:07.950 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-15T22:12:07.951 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-15T22:12:07.951 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-15T22:12:07.951 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-15T22:12:07.951 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-15T22:12:07.951 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-15T22:12:08.236 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-01T22:06:34.123489+0000 '>' 2024-09-01T22:06:34.123489+0000 2024-09-15T22:12:08.236 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-15T22:12:09.237 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-15T22:12:09.237 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-15T22:12:09.238 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-15T22:12:09.238 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-15T22:12:09.238 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-15T22:12:09.238 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-15T22:12:09.238 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-15T22:12:09.515 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-01T22:06:34.123489+0000 '>' 2024-09-01T22:06:34.123489+0000 2024-09-15T22:12:09.515 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-15T22:12:10.517 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-15T22:12:10.517 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-15T22:12:10.517 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-15T22:12:10.517 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-15T22:12:10.517 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-15T22:12:10.518 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-15T22:12:10.518 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-15T22:12:10.800 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-01T22:06:34.123489+0000 '>' 2024-09-01T22:06:34.123489+0000 2024-09-15T22:12:10.800 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-15T22:12:11.801 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-15T22:12:11.801 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-15T22:12:11.802 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-15T22:12:11.802 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-15T22:12:11.802 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-15T22:12:11.803 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-15T22:12:11.803 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-15T22:12:12.084 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-01T22:06:34.123489+0000 '>' 2024-09-01T22:06:34.123489+0000 2024-09-15T22:12:12.084 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-15T22:12:13.085 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-15T22:12:13.085 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-15T22:12:13.086 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-15T22:12:13.086 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-15T22:12:13.086 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-15T22:12:13.086 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-15T22:12:13.086 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-15T22:12:13.359 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-01T22:06:34.123489+0000 '>' 2024-09-01T22:06:34.123489+0000 2024-09-15T22:12:13.359 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-15T22:12:14.360 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-15T22:12:14.360 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-15T22:12:14.361 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-15T22:12:14.361 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-15T22:12:14.361 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-15T22:12:14.361 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-15T22:12:14.361 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-15T22:12:14.643 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-01T22:06:34.123489+0000 '>' 2024-09-01T22:06:34.123489+0000 2024-09-15T22:12:14.643 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-15T22:12:15.643 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-15T22:12:15.644 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-15T22:12:15.644 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-15T22:12:15.644 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-15T22:12:15.644 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-15T22:12:15.645 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-15T22:12:15.645 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-15T22:12:15.924 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-01T22:06:34.123489+0000 '>' 2024-09-01T22:06:34.123489+0000 2024-09-15T22:12:15.924 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-15T22:12:16.925 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-15T22:12:16.926 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-15T22:12:16.926 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-15T22:12:16.926 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-15T22:12:16.926 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-15T22:12:16.927 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-15T22:12:16.927 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-15T22:12:17.208 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-01T22:06:34.123489+0000 '>' 2024-09-01T22:06:34.123489+0000 2024-09-15T22:12:17.208 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-15T22:12:18.209 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-15T22:12:18.210 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-15T22:12:18.210 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-15T22:12:18.211 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-15T22:12:18.211 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-15T22:12:18.211 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-15T22:12:18.211 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-15T22:12:18.493 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-01T22:06:34.123489+0000 '>' 2024-09-01T22:06:34.123489+0000 2024-09-15T22:12:18.493 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-15T22:12:19.493 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-15T22:12:19.494 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-15T22:12:19.494 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-15T22:12:19.494 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-15T22:12:19.494 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-15T22:12:19.495 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-15T22:12:19.495 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-15T22:12:19.773 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-01T22:06:34.123489+0000 '>' 2024-09-01T22:06:34.123489+0000 2024-09-15T22:12:19.773 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-15T22:12:20.774 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-15T22:12:20.774 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-15T22:12:20.775 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-15T22:12:20.775 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-15T22:12:20.775 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-15T22:12:20.775 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-15T22:12:20.775 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-15T22:12:21.055 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-01T22:06:34.123489+0000 '>' 2024-09-01T22:06:34.123489+0000 2024-09-15T22:12:21.055 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-15T22:12:22.056 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-15T22:12:22.056 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-15T22:12:22.057 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-15T22:12:22.057 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-15T22:12:22.057 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-15T22:12:22.057 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-15T22:12:22.057 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-15T22:12:22.342 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-01T22:06:34.123489+0000 '>' 2024-09-01T22:06:34.123489+0000 2024-09-15T22:12:22.342 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-15T22:12:23.343 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-15T22:12:23.343 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-15T22:12:23.344 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-15T22:12:23.344 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-15T22:12:23.344 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-15T22:12:23.344 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-15T22:12:23.344 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-15T22:12:23.621 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-01T22:06:34.123489+0000 '>' 2024-09-01T22:06:34.123489+0000 2024-09-15T22:12:23.621 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-15T22:12:24.622 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-15T22:12:24.623 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-15T22:12:24.623 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-15T22:12:24.623 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-15T22:12:24.623 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-15T22:12:24.624 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-15T22:12:24.624 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-15T22:12:24.902 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-01T22:06:34.123489+0000 '>' 2024-09-01T22:06:34.123489+0000 2024-09-15T22:12:24.902 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-15T22:12:25.903 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-15T22:12:25.903 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-15T22:12:25.903 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-15T22:12:25.904 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-15T22:12:25.904 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-15T22:12:25.904 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-15T22:12:25.904 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-15T22:12:26.183 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-01T22:06:34.123489+0000 '>' 2024-09-01T22:06:34.123489+0000 2024-09-15T22:12:26.183 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-15T22:12:27.184 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-15T22:12:27.184 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-15T22:12:27.185 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-15T22:12:27.185 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-15T22:12:27.185 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-15T22:12:27.185 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-15T22:12:27.186 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-15T22:12:27.470 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-01T22:06:34.123489+0000 '>' 2024-09-01T22:06:34.123489+0000 2024-09-15T22:12:27.470 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-15T22:12:28.471 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-15T22:12:28.471 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-15T22:12:28.472 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-15T22:12:28.472 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-15T22:12:28.472 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-15T22:12:28.473 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-15T22:12:28.473 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-15T22:12:28.753 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-01T22:06:34.123489+0000 '>' 2024-09-01T22:06:34.123489+0000 2024-09-15T22:12:28.753 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-15T22:12:29.755 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-15T22:12:29.755 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-15T22:12:29.756 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-15T22:12:29.756 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-15T22:12:29.756 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-15T22:12:29.756 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-15T22:12:29.756 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-15T22:12:30.036 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-01T22:06:34.123489+0000 '>' 2024-09-01T22:06:34.123489+0000 2024-09-15T22:12:30.036 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-15T22:12:31.037 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-15T22:12:31.037 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-15T22:12:31.038 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-15T22:12:31.038 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-15T22:12:31.038 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-15T22:12:31.038 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-15T22:12:31.038 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-15T22:12:31.318 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-01T22:06:34.123489+0000 '>' 2024-09-01T22:06:34.123489+0000 2024-09-15T22:12:31.318 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-15T22:12:32.319 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-15T22:12:32.320 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-15T22:12:32.320 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-15T22:12:32.320 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-15T22:12:32.320 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-15T22:12:32.321 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-15T22:12:32.321 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-15T22:12:32.601 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-01T22:06:34.123489+0000 '>' 2024-09-01T22:06:34.123489+0000 2024-09-15T22:12:32.601 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-15T22:12:33.602 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-15T22:12:33.602 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-15T22:12:33.603 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-15T22:12:33.603 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-15T22:12:33.603 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-15T22:12:33.603 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-15T22:12:33.604 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-15T22:12:33.882 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-01T22:06:34.123489+0000 '>' 2024-09-01T22:06:34.123489+0000 2024-09-15T22:12:33.882 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-15T22:12:34.883 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-15T22:12:34.883 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-15T22:12:34.884 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-15T22:12:34.884 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-15T22:12:34.884 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-15T22:12:34.884 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-15T22:12:34.885 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-15T22:12:35.164 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-01T22:06:34.123489+0000 '>' 2024-09-01T22:06:34.123489+0000 2024-09-15T22:12:35.165 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-15T22:12:36.166 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-15T22:12:36.166 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-15T22:12:36.166 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-15T22:12:36.166 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-15T22:12:36.166 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-15T22:12:36.167 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-15T22:12:36.167 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-15T22:12:36.462 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-01T22:06:34.123489+0000 '>' 2024-09-01T22:06:34.123489+0000 2024-09-15T22:12:36.463 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-15T22:12:37.463 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-15T22:12:37.463 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-15T22:12:37.464 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-15T22:12:37.464 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-15T22:12:37.464 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-15T22:12:37.464 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-15T22:12:37.464 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-15T22:12:37.746 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-01T22:06:34.123489+0000 '>' 2024-09-01T22:06:34.123489+0000 2024-09-15T22:12:37.746 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-15T22:12:38.747 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-15T22:12:38.747 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-15T22:12:38.748 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-15T22:12:38.748 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-15T22:12:38.748 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-15T22:12:38.748 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-15T22:12:38.748 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-15T22:12:39.021 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-01T22:06:34.123489+0000 '>' 2024-09-01T22:06:34.123489+0000 2024-09-15T22:12:39.021 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-15T22:12:40.022 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-15T22:12:40.022 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-15T22:12:40.023 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-15T22:12:40.023 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-15T22:12:40.023 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-15T22:12:40.023 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-15T22:12:40.023 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-15T22:12:40.303 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-01T22:06:34.123489+0000 '>' 2024-09-01T22:06:34.123489+0000 2024-09-15T22:12:40.303 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-15T22:12:41.304 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-15T22:12:41.304 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-15T22:12:41.305 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-15T22:12:41.305 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-15T22:12:41.305 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-15T22:12:41.305 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-15T22:12:41.305 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-15T22:12:41.578 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-01T22:06:34.123489+0000 '>' 2024-09-01T22:06:34.123489+0000 2024-09-15T22:12:41.578 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-15T22:12:42.579 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-15T22:12:42.579 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-15T22:12:42.580 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-15T22:12:42.580 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-15T22:12:42.580 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-15T22:12:42.580 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-15T22:12:42.581 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-15T22:12:42.857 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-01T22:06:34.123489+0000 '>' 2024-09-01T22:06:34.123489+0000 2024-09-15T22:12:42.857 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-15T22:12:43.858 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-15T22:12:43.858 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-15T22:12:43.859 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-15T22:12:43.859 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-15T22:12:43.859 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-15T22:12:43.859 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-15T22:12:43.859 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-15T22:12:44.133 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-01T22:06:34.123489+0000 '>' 2024-09-01T22:06:34.123489+0000 2024-09-15T22:12:44.133 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-15T22:12:45.134 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-15T22:12:45.134 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-15T22:12:45.135 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-15T22:12:45.135 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-15T22:12:45.135 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-15T22:12:45.136 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-15T22:12:45.136 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-15T22:12:45.406 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-01T22:06:34.123489+0000 '>' 2024-09-01T22:06:34.123489+0000 2024-09-15T22:12:45.407 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-15T22:12:46.407 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-15T22:12:46.407 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-15T22:12:46.408 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-15T22:12:46.408 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-15T22:12:46.408 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-15T22:12:46.409 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-15T22:12:46.409 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-15T22:12:46.686 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-01T22:06:34.123489+0000 '>' 2024-09-01T22:06:34.123489+0000 2024-09-15T22:12:46.686 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-15T22:12:47.687 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-15T22:12:47.687 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-15T22:12:47.687 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-15T22:12:47.688 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-15T22:12:47.688 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-15T22:12:47.688 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-15T22:12:47.688 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-15T22:12:47.970 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-01T22:06:34.123489+0000 '>' 2024-09-01T22:06:34.123489+0000 2024-09-15T22:12:47.970 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-15T22:12:48.971 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-15T22:12:48.971 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-15T22:12:48.972 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-15T22:12:48.972 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-15T22:12:48.972 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-15T22:12:48.972 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-15T22:12:48.972 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-15T22:12:49.248 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-01T22:06:34.123489+0000 '>' 2024-09-01T22:06:34.123489+0000 2024-09-15T22:12:49.248 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-15T22:12:50.249 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-15T22:12:50.249 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-15T22:12:50.250 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-15T22:12:50.250 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-15T22:12:50.250 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-15T22:12:50.250 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-15T22:12:50.250 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-15T22:12:50.537 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-01T22:06:34.123489+0000 '>' 2024-09-01T22:06:34.123489+0000 2024-09-15T22:12:50.537 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-15T22:12:51.538 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-15T22:12:51.539 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-15T22:12:51.539 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-15T22:12:51.539 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-15T22:12:51.539 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-15T22:12:51.540 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-15T22:12:51.540 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-15T22:12:51.818 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-01T22:06:34.123489+0000 '>' 2024-09-01T22:06:34.123489+0000 2024-09-15T22:12:51.819 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-15T22:12:52.820 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-15T22:12:52.820 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-15T22:12:52.821 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-15T22:12:52.821 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-15T22:12:52.821 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-15T22:12:52.821 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-15T22:12:52.821 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-15T22:12:53.096 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-01T22:06:34.123489+0000 '>' 2024-09-01T22:06:34.123489+0000 2024-09-15T22:12:53.096 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-15T22:12:54.097 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-15T22:12:54.097 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-15T22:12:54.098 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-15T22:12:54.098 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-15T22:12:54.098 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-15T22:12:54.098 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-15T22:12:54.099 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-15T22:12:54.379 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-01T22:06:34.123489+0000 '>' 2024-09-01T22:06:34.123489+0000 2024-09-15T22:12:54.379 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-15T22:12:55.380 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-15T22:12:55.380 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-15T22:12:55.381 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-15T22:12:55.381 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-15T22:12:55.381 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-15T22:12:55.381 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-15T22:12:55.381 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-15T22:12:55.658 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-01T22:06:34.123489+0000 '>' 2024-09-01T22:06:34.123489+0000 2024-09-15T22:12:55.659 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-15T22:12:56.659 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-15T22:12:56.659 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-15T22:12:56.660 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-15T22:12:56.660 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-15T22:12:56.660 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-15T22:12:56.660 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-15T22:12:56.660 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-15T22:12:56.936 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-01T22:06:34.123489+0000 '>' 2024-09-01T22:06:34.123489+0000 2024-09-15T22:12:56.936 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-15T22:12:57.937 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-15T22:12:57.937 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-15T22:12:57.938 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-15T22:12:57.938 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-15T22:12:57.938 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-15T22:12:57.938 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-15T22:12:57.938 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-15T22:12:58.218 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-01T22:06:34.123489+0000 '>' 2024-09-01T22:06:34.123489+0000 2024-09-15T22:12:58.218 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-15T22:12:59.219 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-15T22:12:59.219 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-15T22:12:59.220 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-15T22:12:59.220 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-15T22:12:59.220 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-15T22:12:59.220 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-15T22:12:59.220 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-15T22:12:59.499 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-01T22:06:34.123489+0000 '>' 2024-09-01T22:06:34.123489+0000 2024-09-15T22:12:59.499 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-15T22:13:00.500 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-15T22:13:00.500 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-15T22:13:00.501 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-15T22:13:00.501 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-15T22:13:00.501 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-15T22:13:00.502 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-15T22:13:00.502 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-15T22:13:00.780 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-01T22:06:34.123489+0000 '>' 2024-09-01T22:06:34.123489+0000 2024-09-15T22:13:00.781 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-15T22:13:01.781 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-15T22:13:01.782 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-15T22:13:01.783 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-15T22:13:01.783 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-15T22:13:01.783 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-15T22:13:01.783 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-15T22:13:01.783 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-15T22:13:02.063 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-01T22:06:34.123489+0000 '>' 2024-09-01T22:06:34.123489+0000 2024-09-15T22:13:02.063 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-15T22:13:03.064 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-15T22:13:03.064 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-15T22:13:03.065 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-15T22:13:03.065 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-15T22:13:03.065 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-15T22:13:03.066 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-15T22:13:03.066 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-15T22:13:03.346 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-01T22:06:34.123489+0000 '>' 2024-09-01T22:06:34.123489+0000 2024-09-15T22:13:03.346 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-15T22:13:04.347 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-15T22:13:04.347 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-15T22:13:04.349 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-15T22:13:04.349 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-15T22:13:04.349 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-15T22:13:04.349 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-15T22:13:04.349 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-15T22:13:04.628 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-01T22:06:34.123489+0000 '>' 2024-09-01T22:06:34.123489+0000 2024-09-15T22:13:04.628 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-15T22:13:05.629 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-15T22:13:05.629 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-15T22:13:05.630 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-15T22:13:05.630 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-15T22:13:05.630 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-15T22:13:05.630 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-15T22:13:05.630 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-15T22:13:05.908 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-01T22:06:34.123489+0000 '>' 2024-09-01T22:06:34.123489+0000 2024-09-15T22:13:05.908 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-15T22:13:06.909 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-15T22:13:06.909 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-15T22:13:06.910 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-15T22:13:06.910 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-15T22:13:06.910 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-15T22:13:06.910 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-15T22:13:06.910 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-15T22:13:07.192 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-01T22:06:34.123489+0000 '>' 2024-09-01T22:06:34.123489+0000 2024-09-15T22:13:07.192 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-15T22:13:08.193 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-15T22:13:08.194 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-15T22:13:08.195 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-15T22:13:08.195 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-15T22:13:08.195 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-15T22:13:08.195 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-15T22:13:08.195 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-15T22:13:08.481 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-01T22:06:34.123489+0000 '>' 2024-09-01T22:06:34.123489+0000 2024-09-15T22:13:08.481 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-15T22:13:09.482 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-15T22:13:09.482 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-15T22:13:09.483 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-15T22:13:09.484 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-15T22:13:09.484 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-15T22:13:09.484 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-15T22:13:09.484 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-15T22:13:09.763 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-01T22:06:34.123489+0000 '>' 2024-09-01T22:06:34.123489+0000 2024-09-15T22:13:09.764 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-15T22:13:10.764 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-15T22:13:10.765 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-15T22:13:10.765 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-15T22:13:10.766 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-15T22:13:10.766 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-15T22:13:10.766 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-15T22:13:10.766 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-15T22:13:11.046 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-01T22:06:34.123489+0000 '>' 2024-09-01T22:06:34.123489+0000 2024-09-15T22:13:11.047 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-15T22:13:12.048 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-15T22:13:12.048 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-15T22:13:12.049 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-15T22:13:12.049 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-15T22:13:12.049 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-15T22:13:12.049 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-15T22:13:12.049 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-15T22:13:12.332 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-01T22:06:34.123489+0000 '>' 2024-09-01T22:06:34.123489+0000 2024-09-15T22:13:12.332 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-15T22:13:13.333 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-15T22:13:13.333 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-15T22:13:13.334 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-15T22:13:13.334 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-15T22:13:13.334 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-15T22:13:13.334 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-15T22:13:13.334 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-15T22:13:13.615 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-01T22:06:34.123489+0000 '>' 2024-09-01T22:06:34.123489+0000 2024-09-15T22:13:13.616 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-15T22:13:14.617 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-15T22:13:14.617 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-15T22:13:14.618 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-15T22:13:14.618 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-15T22:13:14.618 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-15T22:13:14.618 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-15T22:13:14.618 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-15T22:13:14.895 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-01T22:06:34.123489+0000 '>' 2024-09-01T22:06:34.123489+0000 2024-09-15T22:13:14.895 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-15T22:13:15.896 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-15T22:13:15.897 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-15T22:13:15.897 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-15T22:13:15.898 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-15T22:13:15.898 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-15T22:13:15.898 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-15T22:13:15.898 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-15T22:13:16.179 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-01T22:06:34.123489+0000 '>' 2024-09-01T22:06:34.123489+0000 2024-09-15T22:13:16.179 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-15T22:13:17.180 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-15T22:13:17.181 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-15T22:13:17.181 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-15T22:13:17.182 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-15T22:13:17.182 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-15T22:13:17.182 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-15T22:13:17.182 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-15T22:13:17.460 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-01T22:06:34.123489+0000 '>' 2024-09-01T22:06:34.123489+0000 2024-09-15T22:13:17.460 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-15T22:13:18.461 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-15T22:13:18.461 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-15T22:13:18.462 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-15T22:13:18.462 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-15T22:13:18.462 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-15T22:13:18.462 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-15T22:13:18.463 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-15T22:13:18.745 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-01T22:06:34.123489+0000 '>' 2024-09-01T22:06:34.123489+0000 2024-09-15T22:13:18.745 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-15T22:13:19.746 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-15T22:13:19.746 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-15T22:13:19.747 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-15T22:13:19.747 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-15T22:13:19.747 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-15T22:13:19.748 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-15T22:13:19.748 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-15T22:13:20.023 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-01T22:06:34.123489+0000 '>' 2024-09-01T22:06:34.123489+0000 2024-09-15T22:13:20.023 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-15T22:13:21.024 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-15T22:13:21.024 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-15T22:13:21.025 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-15T22:13:21.025 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-15T22:13:21.025 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-15T22:13:21.025 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-15T22:13:21.025 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-15T22:13:21.298 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-01T22:06:34.123489+0000 '>' 2024-09-01T22:06:34.123489+0000 2024-09-15T22:13:21.298 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-15T22:13:22.299 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-15T22:13:22.299 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-15T22:13:22.300 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-15T22:13:22.300 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-15T22:13:22.300 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-15T22:13:22.300 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-15T22:13:22.300 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-15T22:13:22.604 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-01T22:06:34.123489+0000 '>' 2024-09-01T22:06:34.123489+0000 2024-09-15T22:13:22.604 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-15T22:13:23.605 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-15T22:13:23.605 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-15T22:13:23.606 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-15T22:13:23.606 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-15T22:13:23.606 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-15T22:13:23.606 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-15T22:13:23.606 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-15T22:13:23.887 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-01T22:06:34.123489+0000 '>' 2024-09-01T22:06:34.123489+0000 2024-09-15T22:13:23.887 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-15T22:13:24.888 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-15T22:13:24.888 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-15T22:13:24.889 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-15T22:13:24.889 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-15T22:13:24.889 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-15T22:13:24.889 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-15T22:13:24.889 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-15T22:13:25.172 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-01T22:06:34.123489+0000 '>' 2024-09-01T22:06:34.123489+0000 2024-09-15T22:13:25.172 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-15T22:13:26.173 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-15T22:13:26.173 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-15T22:13:26.174 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-15T22:13:26.174 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-15T22:13:26.174 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-15T22:13:26.174 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-15T22:13:26.175 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-15T22:13:26.466 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-01T22:06:34.123489+0000 '>' 2024-09-01T22:06:34.123489+0000 2024-09-15T22:13:26.466 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-15T22:13:27.467 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-15T22:13:27.467 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-15T22:13:27.468 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-15T22:13:27.468 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-15T22:13:27.468 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-15T22:13:27.468 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-15T22:13:27.468 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-15T22:13:27.747 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-01T22:06:34.123489+0000 '>' 2024-09-01T22:06:34.123489+0000 2024-09-15T22:13:27.747 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-15T22:13:28.748 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-15T22:13:28.748 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-15T22:13:28.749 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-15T22:13:28.749 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-15T22:13:28.749 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-15T22:13:28.749 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-15T22:13:28.750 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-15T22:13:29.033 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-01T22:06:34.123489+0000 '>' 2024-09-01T22:06:34.123489+0000 2024-09-15T22:13:29.033 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-15T22:13:30.033 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-15T22:13:30.034 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-15T22:13:30.035 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-15T22:13:30.035 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-15T22:13:30.035 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-15T22:13:30.035 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-15T22:13:30.035 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-15T22:13:30.317 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-01T22:06:34.123489+0000 '>' 2024-09-01T22:06:34.123489+0000 2024-09-15T22:13:30.317 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-15T22:13:31.318 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-15T22:13:31.318 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-15T22:13:31.319 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-15T22:13:31.319 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-15T22:13:31.319 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-15T22:13:31.319 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-15T22:13:31.319 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-15T22:13:31.599 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-01T22:06:34.123489+0000 '>' 2024-09-01T22:06:34.123489+0000 2024-09-15T22:13:31.599 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-15T22:13:32.600 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-15T22:13:32.600 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-15T22:13:32.601 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-15T22:13:32.601 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-15T22:13:32.601 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-15T22:13:32.601 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-15T22:13:32.601 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-15T22:13:32.880 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-01T22:06:34.123489+0000 '>' 2024-09-01T22:06:34.123489+0000 2024-09-15T22:13:32.881 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-15T22:13:33.882 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-15T22:13:33.882 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-15T22:13:33.882 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-15T22:13:33.883 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-15T22:13:33.883 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-15T22:13:33.883 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-15T22:13:33.883 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-15T22:13:34.165 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-01T22:06:34.123489+0000 '>' 2024-09-01T22:06:34.123489+0000 2024-09-15T22:13:34.166 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-15T22:13:35.167 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-15T22:13:35.167 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-15T22:13:35.167 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-15T22:13:35.168 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-15T22:13:35.168 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-15T22:13:35.168 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-15T22:13:35.168 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-15T22:13:35.449 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-01T22:06:34.123489+0000 '>' 2024-09-01T22:06:34.123489+0000 2024-09-15T22:13:35.449 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-15T22:13:36.450 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-15T22:13:36.451 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-15T22:13:36.451 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-15T22:13:36.452 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-15T22:13:36.452 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-15T22:13:36.452 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-15T22:13:36.452 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-15T22:13:36.736 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-01T22:06:34.123489+0000 '>' 2024-09-01T22:06:34.123489+0000 2024-09-15T22:13:36.736 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-15T22:13:37.737 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-15T22:13:37.737 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-15T22:13:37.738 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-15T22:13:37.738 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-15T22:13:37.738 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-15T22:13:37.738 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-15T22:13:37.738 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-15T22:13:38.018 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-01T22:06:34.123489+0000 '>' 2024-09-01T22:06:34.123489+0000 2024-09-15T22:13:38.018 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-15T22:13:39.019 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-15T22:13:39.019 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-15T22:13:39.020 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-15T22:13:39.020 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-15T22:13:39.020 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-15T22:13:39.020 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-15T22:13:39.020 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-15T22:13:39.298 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-01T22:06:34.123489+0000 '>' 2024-09-01T22:06:34.123489+0000 2024-09-15T22:13:39.298 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-15T22:13:40.299 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-15T22:13:40.299 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-15T22:13:40.300 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-15T22:13:40.300 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-15T22:13:40.300 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-15T22:13:40.300 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-15T22:13:40.300 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-15T22:13:40.578 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-01T22:06:34.123489+0000 '>' 2024-09-01T22:06:34.123489+0000 2024-09-15T22:13:40.578 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-15T22:13:41.579 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-15T22:13:41.579 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-15T22:13:41.580 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-15T22:13:41.580 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-15T22:13:41.580 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-15T22:13:41.580 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-15T22:13:41.580 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-15T22:13:41.852 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-01T22:06:34.123489+0000 '>' 2024-09-01T22:06:34.123489+0000 2024-09-15T22:13:41.852 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-15T22:13:42.853 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-15T22:13:42.853 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-15T22:13:42.854 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-15T22:13:42.854 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-15T22:13:42.854 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-15T22:13:42.854 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-15T22:13:42.854 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-15T22:13:43.131 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-01T22:06:34.123489+0000 '>' 2024-09-01T22:06:34.123489+0000 2024-09-15T22:13:43.131 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-15T22:13:44.132 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-15T22:13:44.132 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-15T22:13:44.133 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-15T22:13:44.133 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-15T22:13:44.133 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-15T22:13:44.133 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-15T22:13:44.133 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-15T22:13:44.411 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-01T22:06:34.123489+0000 '>' 2024-09-01T22:06:34.123489+0000 2024-09-15T22:13:44.411 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-15T22:13:45.412 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-15T22:13:45.412 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-15T22:13:45.413 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-15T22:13:45.413 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-15T22:13:45.413 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-15T22:13:45.413 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-15T22:13:45.413 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-15T22:13:45.689 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-01T22:06:34.123489+0000 '>' 2024-09-01T22:06:34.123489+0000 2024-09-15T22:13:45.689 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-15T22:13:46.690 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-15T22:13:46.690 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-15T22:13:46.691 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-15T22:13:46.691 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-15T22:13:46.692 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-15T22:13:46.692 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-15T22:13:46.692 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-15T22:13:46.971 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-01T22:06:34.123489+0000 '>' 2024-09-01T22:06:34.123489+0000 2024-09-15T22:13:46.971 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-15T22:13:47.972 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-15T22:13:47.973 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-15T22:13:47.973 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-15T22:13:47.973 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-15T22:13:47.973 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-15T22:13:47.974 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-15T22:13:47.974 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-15T22:13:48.249 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-01T22:06:34.123489+0000 '>' 2024-09-01T22:06:34.123489+0000 2024-09-15T22:13:48.249 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-15T22:13:49.250 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-15T22:13:49.250 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-15T22:13:49.251 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-15T22:13:49.251 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-15T22:13:49.251 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-15T22:13:49.251 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-15T22:13:49.251 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-15T22:13:49.534 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-01T22:06:34.123489+0000 '>' 2024-09-01T22:06:34.123489+0000 2024-09-15T22:13:49.534 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-15T22:13:50.535 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-15T22:13:50.535 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-15T22:13:50.536 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-15T22:13:50.536 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-15T22:13:50.536 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-15T22:13:50.536 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-15T22:13:50.536 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-15T22:13:50.813 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-01T22:06:34.123489+0000 '>' 2024-09-01T22:06:34.123489+0000 2024-09-15T22:13:50.814 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-15T22:13:51.815 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-15T22:13:51.815 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-15T22:13:51.815 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-15T22:13:51.815 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-15T22:13:51.816 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-15T22:13:51.816 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-15T22:13:51.816 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-15T22:13:52.093 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-01T22:06:34.123489+0000 '>' 2024-09-01T22:06:34.123489+0000 2024-09-15T22:13:52.094 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-15T22:13:53.095 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-15T22:13:53.095 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-15T22:13:53.095 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-15T22:13:53.096 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-15T22:13:53.096 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-15T22:13:53.096 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-15T22:13:53.096 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-15T22:13:53.374 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-01T22:06:34.123489+0000 '>' 2024-09-01T22:06:34.123489+0000 2024-09-15T22:13:53.374 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-15T22:13:54.375 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-15T22:13:54.375 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-15T22:13:54.376 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-15T22:13:54.376 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-15T22:13:54.376 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-15T22:13:54.377 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-15T22:13:54.377 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-15T22:13:54.659 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-01T22:06:34.123489+0000 '>' 2024-09-01T22:06:34.123489+0000 2024-09-15T22:13:54.659 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-15T22:13:55.660 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-15T22:13:55.660 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-15T22:13:55.661 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-15T22:13:55.661 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-15T22:13:55.661 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-15T22:13:55.662 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-15T22:13:55.662 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-15T22:13:55.942 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-01T22:06:34.123489+0000 '>' 2024-09-01T22:06:34.123489+0000 2024-09-15T22:13:55.943 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-15T22:13:56.944 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-15T22:13:56.944 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-15T22:13:56.944 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-15T22:13:56.944 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-15T22:13:56.944 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-15T22:13:56.945 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-15T22:13:56.945 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-15T22:13:57.224 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-01T22:06:34.123489+0000 '>' 2024-09-01T22:06:34.123489+0000 2024-09-15T22:13:57.224 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-15T22:13:58.225 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-15T22:13:58.225 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-15T22:13:58.226 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-15T22:13:58.226 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-15T22:13:58.226 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-15T22:13:58.226 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-15T22:13:58.226 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-15T22:13:58.523 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-01T22:06:34.123489+0000 '>' 2024-09-01T22:06:34.123489+0000 2024-09-15T22:13:58.523 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-15T22:13:59.524 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-15T22:13:59.524 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-15T22:13:59.525 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-15T22:13:59.525 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-15T22:13:59.525 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-15T22:13:59.525 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-15T22:13:59.525 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-15T22:13:59.803 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-01T22:06:34.123489+0000 '>' 2024-09-01T22:06:34.123489+0000 2024-09-15T22:13:59.803 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-15T22:14:00.804 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-15T22:14:00.804 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-15T22:14:00.805 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-15T22:14:00.805 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-15T22:14:00.805 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-15T22:14:00.805 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-15T22:14:00.806 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-15T22:14:01.089 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-01T22:06:34.123489+0000 '>' 2024-09-01T22:06:34.123489+0000 2024-09-15T22:14:01.089 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-15T22:14:02.090 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-15T22:14:02.090 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-15T22:14:02.091 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-15T22:14:02.091 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-15T22:14:02.091 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-15T22:14:02.091 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-15T22:14:02.091 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-15T22:14:02.376 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-01T22:06:34.123489+0000 '>' 2024-09-01T22:06:34.123489+0000 2024-09-15T22:14:02.559 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-15T22:14:03.376 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-15T22:14:03.376 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-15T22:14:03.377 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-15T22:14:03.377 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-15T22:14:03.377 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-15T22:14:03.377 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-15T22:14:03.377 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-15T22:14:03.658 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-01T22:06:34.123489+0000 '>' 2024-09-01T22:06:34.123489+0000 2024-09-15T22:14:03.658 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-15T22:14:04.659 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-15T22:14:04.660 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-15T22:14:04.660 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-15T22:14:04.660 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-15T22:14:04.660 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-15T22:14:04.660 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-15T22:14:04.661 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-15T22:14:04.941 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-01T22:06:34.123489+0000 '>' 2024-09-01T22:06:34.123489+0000 2024-09-15T22:14:04.941 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-15T22:14:05.942 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-15T22:14:05.943 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-15T22:14:05.943 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-15T22:14:05.944 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-15T22:14:05.944 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-15T22:14:05.944 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-15T22:14:05.944 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-15T22:14:06.226 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-01T22:06:34.123489+0000 '>' 2024-09-01T22:06:34.123489+0000 2024-09-15T22:14:06.226 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-15T22:14:07.227 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-15T22:14:07.227 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-15T22:14:07.228 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-15T22:14:07.228 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-15T22:14:07.228 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-15T22:14:07.228 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-15T22:14:07.228 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-15T22:14:07.513 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-01T22:06:34.123489+0000 '>' 2024-09-01T22:06:34.123489+0000 2024-09-15T22:14:07.513 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-15T22:14:08.514 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-15T22:14:08.514 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-15T22:14:08.515 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-15T22:14:08.515 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-15T22:14:08.515 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-15T22:14:08.515 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-15T22:14:08.515 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-15T22:14:08.792 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-01T22:06:34.123489+0000 '>' 2024-09-01T22:06:34.123489+0000 2024-09-15T22:14:08.793 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-15T22:14:09.793 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-15T22:14:09.794 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-15T22:14:09.794 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-15T22:14:09.795 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-15T22:14:09.795 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-15T22:14:09.795 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-15T22:14:09.795 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-15T22:14:10.075 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-01T22:06:34.123489+0000 '>' 2024-09-01T22:06:34.123489+0000 2024-09-15T22:14:10.075 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-15T22:14:11.076 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-15T22:14:11.076 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-15T22:14:11.077 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-15T22:14:11.077 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-15T22:14:11.077 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-15T22:14:11.077 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-15T22:14:11.077 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-15T22:14:11.354 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-01T22:06:34.123489+0000 '>' 2024-09-01T22:06:34.123489+0000 2024-09-15T22:14:11.354 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-15T22:14:12.355 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-15T22:14:12.355 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-15T22:14:12.355 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-15T22:14:12.355 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-15T22:14:12.356 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-15T22:14:12.356 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-15T22:14:12.356 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-15T22:14:12.634 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-01T22:06:34.123489+0000 '>' 2024-09-01T22:06:34.123489+0000 2024-09-15T22:14:12.635 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-15T22:14:13.636 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-15T22:14:13.636 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-15T22:14:13.637 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-15T22:14:13.637 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-15T22:14:13.637 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-15T22:14:13.637 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-15T22:14:13.637 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-15T22:14:13.921 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-01T22:06:34.123489+0000 '>' 2024-09-01T22:06:34.123489+0000 2024-09-15T22:14:13.921 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-15T22:14:14.920 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-15T22:14:14.920 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-15T22:14:14.920 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-15T22:14:14.920 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-15T22:14:14.920 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-15T22:14:14.921 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-15T22:14:14.921 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-15T22:14:15.202 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-01T22:06:34.123489+0000 '>' 2024-09-01T22:06:34.123489+0000 2024-09-15T22:14:15.202 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-15T22:14:16.203 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-15T22:14:16.203 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-15T22:14:16.203 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-15T22:14:16.204 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-15T22:14:16.204 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-15T22:14:16.204 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-15T22:14:16.204 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-15T22:14:16.483 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-01T22:06:34.123489+0000 '>' 2024-09-01T22:06:34.123489+0000 2024-09-15T22:14:16.484 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-15T22:14:17.485 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-15T22:14:17.485 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-15T22:14:17.486 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-15T22:14:17.486 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-15T22:14:17.486 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-15T22:14:17.486 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-15T22:14:17.486 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-15T22:14:17.767 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-01T22:06:34.123489+0000 '>' 2024-09-01T22:06:34.123489+0000 2024-09-15T22:14:17.767 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-15T22:14:18.768 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-15T22:14:18.768 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-15T22:14:18.769 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-15T22:14:18.769 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-15T22:14:18.769 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-15T22:14:18.769 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-15T22:14:18.769 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-15T22:14:19.047 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-01T22:06:34.123489+0000 '>' 2024-09-01T22:06:34.123489+0000 2024-09-15T22:14:19.047 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-15T22:14:20.048 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-15T22:14:20.049 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-15T22:14:20.052 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-15T22:14:20.052 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-15T22:14:20.052 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-15T22:14:20.052 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-15T22:14:20.052 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-15T22:14:20.329 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-01T22:06:34.123489+0000 '>' 2024-09-01T22:06:34.123489+0000 2024-09-15T22:14:20.329 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-15T22:14:21.330 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-15T22:14:21.330 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-15T22:14:21.331 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-15T22:14:21.331 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-15T22:14:21.331 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-15T22:14:21.331 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-15T22:14:21.331 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-15T22:14:21.611 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-01T22:06:34.123489+0000 '>' 2024-09-01T22:06:34.123489+0000 2024-09-15T22:14:21.611 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-15T22:14:22.613 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-15T22:14:22.613 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-15T22:14:22.614 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-15T22:14:22.614 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-15T22:14:22.614 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-15T22:14:22.614 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-15T22:14:22.614 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-15T22:14:22.891 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-01T22:06:34.123489+0000 '>' 2024-09-01T22:06:34.123489+0000 2024-09-15T22:14:22.891 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-15T22:14:23.891 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-15T22:14:23.892 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-15T22:14:23.892 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-15T22:14:23.892 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-15T22:14:23.892 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-15T22:14:23.893 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-15T22:14:23.893 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-15T22:14:24.174 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-01T22:06:34.123489+0000 '>' 2024-09-01T22:06:34.123489+0000 2024-09-15T22:14:24.174 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-15T22:14:25.175 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-15T22:14:25.175 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-15T22:14:25.176 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-15T22:14:25.176 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-15T22:14:25.176 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-15T22:14:25.176 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-15T22:14:25.177 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-15T22:14:25.454 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-01T22:06:34.123489+0000 '>' 2024-09-01T22:06:34.123489+0000 2024-09-15T22:14:25.455 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-15T22:14:26.456 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-15T22:14:26.456 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-15T22:14:26.457 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-15T22:14:26.457 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-15T22:14:26.457 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-15T22:14:26.457 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-15T22:14:26.457 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-15T22:14:26.736 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-01T22:06:34.123489+0000 '>' 2024-09-01T22:06:34.123489+0000 2024-09-15T22:14:26.736 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-15T22:14:27.737 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-15T22:14:27.737 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-15T22:14:27.738 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-15T22:14:27.738 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-15T22:14:27.738 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-15T22:14:27.739 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-15T22:14:27.739 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-15T22:14:28.019 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-01T22:06:34.123489+0000 '>' 2024-09-01T22:06:34.123489+0000 2024-09-15T22:14:28.020 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-15T22:14:29.021 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-15T22:14:29.021 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-15T22:14:29.022 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-15T22:14:29.022 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-15T22:14:29.022 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-15T22:14:29.022 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-15T22:14:29.022 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-15T22:14:29.299 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-01T22:06:34.123489+0000 '>' 2024-09-01T22:06:34.123489+0000 2024-09-15T22:14:29.299 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-15T22:14:30.301 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-15T22:14:30.301 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-15T22:14:30.302 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-15T22:14:30.302 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-15T22:14:30.302 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-15T22:14:30.302 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-15T22:14:30.302 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-15T22:14:30.598 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-01T22:06:34.123489+0000 '>' 2024-09-01T22:06:34.123489+0000 2024-09-15T22:14:30.599 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-15T22:14:31.599 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-15T22:14:31.600 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-15T22:14:31.600 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-15T22:14:31.600 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-15T22:14:31.600 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-15T22:14:31.601 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-15T22:14:31.601 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-15T22:14:31.879 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-01T22:06:34.123489+0000 '>' 2024-09-01T22:06:34.123489+0000 2024-09-15T22:14:31.880 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-15T22:14:32.880 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-15T22:14:32.881 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-15T22:14:32.881 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-15T22:14:32.881 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-15T22:14:32.881 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-15T22:14:32.882 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-15T22:14:32.882 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-15T22:14:33.160 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-01T22:06:34.123489+0000 '>' 2024-09-01T22:06:34.123489+0000 2024-09-15T22:14:33.160 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-15T22:14:34.161 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-15T22:14:34.162 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-15T22:14:34.162 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-15T22:14:34.162 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-15T22:14:34.163 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-15T22:14:34.163 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-15T22:14:34.163 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-15T22:14:34.445 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-01T22:06:34.123489+0000 '>' 2024-09-01T22:06:34.123489+0000 2024-09-15T22:14:34.445 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-15T22:14:35.446 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-15T22:14:35.446 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-15T22:14:35.447 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-15T22:14:35.447 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-15T22:14:35.447 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-15T22:14:35.447 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-15T22:14:35.448 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-15T22:14:35.733 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-01T22:06:34.123489+0000 '>' 2024-09-01T22:06:34.123489+0000 2024-09-15T22:14:35.733 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-15T22:14:36.734 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-15T22:14:36.734 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-15T22:14:36.735 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-15T22:14:36.735 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-15T22:14:36.735 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-15T22:14:36.735 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-15T22:14:36.735 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-15T22:14:37.014 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-01T22:06:34.123489+0000 '>' 2024-09-01T22:06:34.123489+0000 2024-09-15T22:14:37.014 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-15T22:14:38.015 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-15T22:14:38.015 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-15T22:14:38.016 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-15T22:14:38.016 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-15T22:14:38.016 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-15T22:14:38.017 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-15T22:14:38.017 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-15T22:14:38.299 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-01T22:06:34.123489+0000 '>' 2024-09-01T22:06:34.123489+0000 2024-09-15T22:14:38.299 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-15T22:14:39.300 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-15T22:14:39.300 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-15T22:14:39.302 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-15T22:14:39.302 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-15T22:14:39.302 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-15T22:14:39.302 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-15T22:14:39.302 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-15T22:14:39.588 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-01T22:06:34.123489+0000 '>' 2024-09-01T22:06:34.123489+0000 2024-09-15T22:14:39.589 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-15T22:14:40.589 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-15T22:14:40.589 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-15T22:14:40.590 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-15T22:14:40.590 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-15T22:14:40.590 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-15T22:14:40.590 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-15T22:14:40.591 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-15T22:14:40.867 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-01T22:06:34.123489+0000 '>' 2024-09-01T22:06:34.123489+0000 2024-09-15T22:14:40.868 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-15T22:14:41.869 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-15T22:14:41.869 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-15T22:14:41.869 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-15T22:14:41.870 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-15T22:14:41.870 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-15T22:14:41.870 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-15T22:14:41.870 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-15T22:14:42.153 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-01T22:06:34.123489+0000 '>' 2024-09-01T22:06:34.123489+0000 2024-09-15T22:14:42.154 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-15T22:14:43.154 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-15T22:14:43.155 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-15T22:14:43.155 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-15T22:14:43.155 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-15T22:14:43.155 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-15T22:14:43.156 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-15T22:14:43.156 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-15T22:14:43.437 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-01T22:06:34.123489+0000 '>' 2024-09-01T22:06:34.123489+0000 2024-09-15T22:14:43.437 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-15T22:14:44.438 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-15T22:14:44.438 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-15T22:14:44.439 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-15T22:14:44.439 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-15T22:14:44.439 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-15T22:14:44.440 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-15T22:14:44.440 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-15T22:14:44.720 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-01T22:06:34.123489+0000 '>' 2024-09-01T22:06:34.123489+0000 2024-09-15T22:14:44.720 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-15T22:14:45.721 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-15T22:14:45.721 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-15T22:14:45.722 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-15T22:14:45.722 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-15T22:14:45.722 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-15T22:14:45.722 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-15T22:14:45.722 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-15T22:14:46.000 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-01T22:06:34.123489+0000 '>' 2024-09-01T22:06:34.123489+0000 2024-09-15T22:14:46.000 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-15T22:14:47.001 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-15T22:14:47.001 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-15T22:14:47.001 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-15T22:14:47.001 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-15T22:14:47.002 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-15T22:14:47.002 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-15T22:14:47.002 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-15T22:14:47.274 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-01T22:06:34.123489+0000 '>' 2024-09-01T22:06:34.123489+0000 2024-09-15T22:14:47.274 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-15T22:14:48.275 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-15T22:14:48.275 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-15T22:14:48.276 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-15T22:14:48.276 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-15T22:14:48.276 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-15T22:14:48.276 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-15T22:14:48.277 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-15T22:14:48.608 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-01T22:06:34.123489+0000 '>' 2024-09-01T22:06:34.123489+0000 2024-09-15T22:14:48.609 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-15T22:14:49.610 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-15T22:14:49.610 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-15T22:14:49.610 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-15T22:14:49.611 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-15T22:14:49.611 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-15T22:14:49.611 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-15T22:14:49.611 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-15T22:14:49.891 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-01T22:06:34.123489+0000 '>' 2024-09-01T22:06:34.123489+0000 2024-09-15T22:14:49.891 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-15T22:14:50.892 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-15T22:14:50.892 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-15T22:14:50.893 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-15T22:14:50.893 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-15T22:14:50.893 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-15T22:14:50.893 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-15T22:14:50.893 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-15T22:14:51.171 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-01T22:06:34.123489+0000 '>' 2024-09-01T22:06:34.123489+0000 2024-09-15T22:14:51.171 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-15T22:14:52.172 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-15T22:14:52.172 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-15T22:14:52.173 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-15T22:14:52.173 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-15T22:14:52.173 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-15T22:14:52.173 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-15T22:14:52.173 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-15T22:14:52.475 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-01T22:06:34.123489+0000 '>' 2024-09-01T22:06:34.123489+0000 2024-09-15T22:14:52.475 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-15T22:14:53.476 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-15T22:14:53.476 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-15T22:14:53.477 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-15T22:14:53.477 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-15T22:14:53.477 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-15T22:14:53.477 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-15T22:14:53.477 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-15T22:14:53.755 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-01T22:06:34.123489+0000 '>' 2024-09-01T22:06:34.123489+0000 2024-09-15T22:14:53.755 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-15T22:14:54.756 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-15T22:14:54.756 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-15T22:14:54.757 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-15T22:14:54.757 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-15T22:14:54.757 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-15T22:14:54.757 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-15T22:14:54.757 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-15T22:14:55.031 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-01T22:06:34.123489+0000 '>' 2024-09-01T22:06:34.123489+0000 2024-09-15T22:14:55.032 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-15T22:14:56.032 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-15T22:14:56.033 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-15T22:14:56.033 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-15T22:14:56.033 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-15T22:14:56.034 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-15T22:14:56.034 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-15T22:14:56.034 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-15T22:14:56.314 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-01T22:06:34.123489+0000 '>' 2024-09-01T22:06:34.123489+0000 2024-09-15T22:14:56.315 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-15T22:14:57.315 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-15T22:14:57.316 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-15T22:14:57.316 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-15T22:14:57.316 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-15T22:14:57.316 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-15T22:14:57.317 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-15T22:14:57.317 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-15T22:14:57.606 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-01T22:06:34.123489+0000 '>' 2024-09-01T22:06:34.123489+0000 2024-09-15T22:14:57.606 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-15T22:14:58.607 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-15T22:14:58.608 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-15T22:14:58.608 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-15T22:14:58.608 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-15T22:14:58.609 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-15T22:14:58.609 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-15T22:14:58.609 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-15T22:14:58.888 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-01T22:06:34.123489+0000 '>' 2024-09-01T22:06:34.123489+0000 2024-09-15T22:14:58.889 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-15T22:14:59.889 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-15T22:14:59.890 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-15T22:14:59.890 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-15T22:14:59.890 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-15T22:14:59.890 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-15T22:14:59.891 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-15T22:14:59.891 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-15T22:15:00.171 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-01T22:06:34.123489+0000 '>' 2024-09-01T22:06:34.123489+0000 2024-09-15T22:15:00.171 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-15T22:15:01.172 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-15T22:15:01.172 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-15T22:15:01.173 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-15T22:15:01.173 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-15T22:15:01.173 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-15T22:15:01.173 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-15T22:15:01.174 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-15T22:15:01.446 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-01T22:06:34.123489+0000 '>' 2024-09-01T22:06:34.123489+0000 2024-09-15T22:15:01.447 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-15T22:15:02.448 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-15T22:15:02.448 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-15T22:15:02.449 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-15T22:15:02.449 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-15T22:15:02.449 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-15T22:15:02.449 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-15T22:15:02.449 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-15T22:15:02.730 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-01T22:06:34.123489+0000 '>' 2024-09-01T22:06:34.123489+0000 2024-09-15T22:15:02.730 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-15T22:15:03.731 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-15T22:15:03.731 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-15T22:15:03.732 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-15T22:15:03.732 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-15T22:15:03.732 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-15T22:15:03.732 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-15T22:15:03.732 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-15T22:15:04.011 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-01T22:06:34.123489+0000 '>' 2024-09-01T22:06:34.123489+0000 2024-09-15T22:15:04.011 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-15T22:15:05.012 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-15T22:15:05.012 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-15T22:15:05.013 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-15T22:15:05.013 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-15T22:15:05.013 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-15T22:15:05.013 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-15T22:15:05.013 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-15T22:15:05.293 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-01T22:06:34.123489+0000 '>' 2024-09-01T22:06:34.123489+0000 2024-09-15T22:15:05.293 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-15T22:15:06.295 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-15T22:15:06.295 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-15T22:15:06.295 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-15T22:15:06.296 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-15T22:15:06.296 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-15T22:15:06.296 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-15T22:15:06.296 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-15T22:15:06.622 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-01T22:06:34.123489+0000 '>' 2024-09-01T22:06:34.123489+0000 2024-09-15T22:15:06.622 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-15T22:15:07.623 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-15T22:15:07.623 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-15T22:15:07.623 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-15T22:15:07.624 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-15T22:15:07.624 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-15T22:15:07.624 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-15T22:15:07.624 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-15T22:15:07.909 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-01T22:06:34.123489+0000 '>' 2024-09-01T22:06:34.123489+0000 2024-09-15T22:15:07.909 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-15T22:15:08.910 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-15T22:15:08.910 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-15T22:15:08.911 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-15T22:15:08.911 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-15T22:15:08.911 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-15T22:15:08.912 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-15T22:15:08.912 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-15T22:15:09.194 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-01T22:06:34.123489+0000 '>' 2024-09-01T22:06:34.123489+0000 2024-09-15T22:15:09.329 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-15T22:15:10.195 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-15T22:15:10.195 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-15T22:15:10.195 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-15T22:15:10.196 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-15T22:15:10.196 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-15T22:15:10.196 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-15T22:15:10.196 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-15T22:15:10.486 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-01T22:06:34.123489+0000 '>' 2024-09-01T22:06:34.123489+0000 2024-09-15T22:15:10.486 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-15T22:15:11.487 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-15T22:15:11.487 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-15T22:15:11.488 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-15T22:15:11.488 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-15T22:15:11.488 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-15T22:15:11.488 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-15T22:15:11.488 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-15T22:15:11.760 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-01T22:06:34.123489+0000 '>' 2024-09-01T22:06:34.123489+0000 2024-09-15T22:15:11.760 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-15T22:15:12.761 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-15T22:15:12.761 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-15T22:15:12.762 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-15T22:15:12.763 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-15T22:15:12.763 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-15T22:15:12.763 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-15T22:15:12.763 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-15T22:15:13.039 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-01T22:06:34.123489+0000 '>' 2024-09-01T22:06:34.123489+0000 2024-09-15T22:15:13.039 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-15T22:15:14.041 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-15T22:15:14.041 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-15T22:15:14.041 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-15T22:15:14.041 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-15T22:15:14.041 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-15T22:15:14.041 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-15T22:15:14.042 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-15T22:15:14.318 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-01T22:06:34.123489+0000 '>' 2024-09-01T22:06:34.123489+0000 2024-09-15T22:15:14.318 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-15T22:15:15.319 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-15T22:15:15.319 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-15T22:15:15.320 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-15T22:15:15.320 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-15T22:15:15.320 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-15T22:15:15.320 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-15T22:15:15.320 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-15T22:15:15.598 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-01T22:06:34.123489+0000 '>' 2024-09-01T22:06:34.123489+0000 2024-09-15T22:15:15.598 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-15T22:15:16.599 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-15T22:15:16.599 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-15T22:15:16.600 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-15T22:15:16.600 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-15T22:15:16.600 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-15T22:15:16.601 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-15T22:15:16.601 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-15T22:15:16.880 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-01T22:06:34.123489+0000 '>' 2024-09-01T22:06:34.123489+0000 2024-09-15T22:15:16.880 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-15T22:15:17.881 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-15T22:15:17.881 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-15T22:15:17.881 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-15T22:15:17.882 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-15T22:15:17.882 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-15T22:15:17.882 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-15T22:15:17.882 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-15T22:15:18.162 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-01T22:06:34.123489+0000 '>' 2024-09-01T22:06:34.123489+0000 2024-09-15T22:15:18.162 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-15T22:15:19.163 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-15T22:15:19.163 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-15T22:15:19.164 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-15T22:15:19.164 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-15T22:15:19.164 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-15T22:15:19.164 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-15T22:15:19.164 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-15T22:15:19.441 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-01T22:06:34.123489+0000 '>' 2024-09-01T22:06:34.123489+0000 2024-09-15T22:15:19.441 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-15T22:15:20.442 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-15T22:15:20.442 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-15T22:15:20.443 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-15T22:15:20.443 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-15T22:15:20.443 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-15T22:15:20.443 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-15T22:15:20.443 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-15T22:15:20.719 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-01T22:06:34.123489+0000 '>' 2024-09-01T22:06:34.123489+0000 2024-09-15T22:15:20.719 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-15T22:15:21.720 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-15T22:15:21.720 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-15T22:15:21.721 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-15T22:15:21.721 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-15T22:15:21.721 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-15T22:15:21.721 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-15T22:15:21.721 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-15T22:15:21.998 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-01T22:06:34.123489+0000 '>' 2024-09-01T22:06:34.123489+0000 2024-09-15T22:15:21.999 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-15T22:15:22.999 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-15T22:15:23.000 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-15T22:15:23.000 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-15T22:15:23.000 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-15T22:15:23.001 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-15T22:15:23.001 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-15T22:15:23.001 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-15T22:15:23.272 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-01T22:06:34.123489+0000 '>' 2024-09-01T22:06:34.123489+0000 2024-09-15T22:15:23.272 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-15T22:15:24.273 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-15T22:15:24.274 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-15T22:15:24.274 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-15T22:15:24.275 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-15T22:15:24.275 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-15T22:15:24.275 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-15T22:15:24.275 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-15T22:15:24.555 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-01T22:06:34.123489+0000 '>' 2024-09-01T22:06:34.123489+0000 2024-09-15T22:15:24.556 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-15T22:15:25.556 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-15T22:15:25.556 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-15T22:15:25.557 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-15T22:15:25.557 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-15T22:15:25.557 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-15T22:15:25.557 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-15T22:15:25.557 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-15T22:15:25.837 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-01T22:06:34.123489+0000 '>' 2024-09-01T22:06:34.123489+0000 2024-09-15T22:15:25.838 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-15T22:15:26.839 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-15T22:15:26.839 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-15T22:15:26.840 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-15T22:15:26.840 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-15T22:15:26.840 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-15T22:15:26.840 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-15T22:15:26.840 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-15T22:15:27.123 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-01T22:06:34.123489+0000 '>' 2024-09-01T22:06:34.123489+0000 2024-09-15T22:15:27.123 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-15T22:15:28.124 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-15T22:15:28.124 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-15T22:15:28.126 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-15T22:15:28.126 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-15T22:15:28.126 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-15T22:15:28.126 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-15T22:15:28.126 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-15T22:15:28.402 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-01T22:06:34.123489+0000 '>' 2024-09-01T22:06:34.123489+0000 2024-09-15T22:15:28.402 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-15T22:15:29.403 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-15T22:15:29.403 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-15T22:15:29.404 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-15T22:15:29.404 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-15T22:15:29.404 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-15T22:15:29.404 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-15T22:15:29.404 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-15T22:15:29.684 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-01T22:06:34.123489+0000 '>' 2024-09-01T22:06:34.123489+0000 2024-09-15T22:15:29.685 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-15T22:15:30.685 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-15T22:15:30.686 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-15T22:15:30.686 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-15T22:15:30.687 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-15T22:15:30.687 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-15T22:15:30.687 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-15T22:15:30.687 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-15T22:15:30.968 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-01T22:06:34.123489+0000 '>' 2024-09-01T22:06:34.123489+0000 2024-09-15T22:15:30.969 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-15T22:15:31.969 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-15T22:15:31.970 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-15T22:15:31.970 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-15T22:15:31.970 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-15T22:15:31.970 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-15T22:15:31.971 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-15T22:15:31.971 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-15T22:15:32.252 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-01T22:06:34.123489+0000 '>' 2024-09-01T22:06:34.123489+0000 2024-09-15T22:15:32.252 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-15T22:15:33.253 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-15T22:15:33.253 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-15T22:15:33.254 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-15T22:15:33.254 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-15T22:15:33.254 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-15T22:15:33.254 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-15T22:15:33.254 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-15T22:15:33.530 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-01T22:06:34.123489+0000 '>' 2024-09-01T22:06:34.123489+0000 2024-09-15T22:15:33.531 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-15T22:15:34.531 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-15T22:15:34.532 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-15T22:15:34.532 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-15T22:15:34.533 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-15T22:15:34.533 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-15T22:15:34.533 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-15T22:15:34.533 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-15T22:15:34.812 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-01T22:06:34.123489+0000 '>' 2024-09-01T22:06:34.123489+0000 2024-09-15T22:15:34.812 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-15T22:15:35.813 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-15T22:15:35.813 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-15T22:15:35.814 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-15T22:15:35.814 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-15T22:15:35.814 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-15T22:15:35.814 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-15T22:15:35.814 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-15T22:15:36.097 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-01T22:06:34.123489+0000 '>' 2024-09-01T22:06:34.123489+0000 2024-09-15T22:15:36.097 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-15T22:15:37.098 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-15T22:15:37.098 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-15T22:15:37.098 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-15T22:15:37.099 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-15T22:15:37.099 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-15T22:15:37.099 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-15T22:15:37.099 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-15T22:15:37.378 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-01T22:06:34.123489+0000 '>' 2024-09-01T22:06:34.123489+0000 2024-09-15T22:15:37.378 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-15T22:15:38.379 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-15T22:15:38.379 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-15T22:15:38.380 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-15T22:15:38.380 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-15T22:15:38.380 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-15T22:15:38.380 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-15T22:15:38.381 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-15T22:15:38.661 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-01T22:06:34.123489+0000 '>' 2024-09-01T22:06:34.123489+0000 2024-09-15T22:15:38.661 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-15T22:15:39.662 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-15T22:15:39.663 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-15T22:15:39.663 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-15T22:15:39.663 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-15T22:15:39.663 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-15T22:15:39.663 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-15T22:15:39.664 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-15T22:15:39.945 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-01T22:06:34.123489+0000 '>' 2024-09-01T22:06:34.123489+0000 2024-09-15T22:15:40.300 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-15T22:15:40.946 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-15T22:15:40.946 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-15T22:15:40.947 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-15T22:15:40.947 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-15T22:15:40.947 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-15T22:15:40.947 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-15T22:15:40.947 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-15T22:15:41.224 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-01T22:06:34.123489+0000 '>' 2024-09-01T22:06:34.123489+0000 2024-09-15T22:15:41.224 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-15T22:15:42.225 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-15T22:15:42.225 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-15T22:15:42.226 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-15T22:15:42.226 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-15T22:15:42.226 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-15T22:15:42.226 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-15T22:15:42.226 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-15T22:15:42.500 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-01T22:06:34.123489+0000 '>' 2024-09-01T22:06:34.123489+0000 2024-09-15T22:15:42.500 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-15T22:15:43.501 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-15T22:15:43.501 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-15T22:15:43.502 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-15T22:15:43.502 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-15T22:15:43.502 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-15T22:15:43.502 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-15T22:15:43.502 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-15T22:15:43.783 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-01T22:06:34.123489+0000 '>' 2024-09-01T22:06:34.123489+0000 2024-09-15T22:15:43.783 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-15T22:15:44.784 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-15T22:15:44.784 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-15T22:15:44.785 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-15T22:15:44.785 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-15T22:15:44.785 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-15T22:15:44.785 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-15T22:15:44.785 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-15T22:15:45.064 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-01T22:06:34.123489+0000 '>' 2024-09-01T22:06:34.123489+0000 2024-09-15T22:15:45.064 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-15T22:15:46.066 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-15T22:15:46.066 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-15T22:15:46.067 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-15T22:15:46.067 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-15T22:15:46.067 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-15T22:15:46.067 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-15T22:15:46.067 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-15T22:15:46.345 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-01T22:06:34.123489+0000 '>' 2024-09-01T22:06:34.123489+0000 2024-09-15T22:15:46.345 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-15T22:15:47.347 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-15T22:15:47.347 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-15T22:15:47.347 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-15T22:15:47.347 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-15T22:15:47.347 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-15T22:15:47.348 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-15T22:15:47.348 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-15T22:15:47.628 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-01T22:06:34.123489+0000 '>' 2024-09-01T22:06:34.123489+0000 2024-09-15T22:15:47.628 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-15T22:15:48.629 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-15T22:15:48.629 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-15T22:15:48.630 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-15T22:15:48.630 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-15T22:15:48.630 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-15T22:15:48.630 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-15T22:15:48.630 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-15T22:15:48.911 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-01T22:06:34.123489+0000 '>' 2024-09-01T22:06:34.123489+0000 2024-09-15T22:15:48.911 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-15T22:15:49.912 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-15T22:15:49.913 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-15T22:15:49.913 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-15T22:15:49.913 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-15T22:15:49.913 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-15T22:15:49.914 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-15T22:15:49.914 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-15T22:15:50.198 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-01T22:06:34.123489+0000 '>' 2024-09-01T22:06:34.123489+0000 2024-09-15T22:15:50.198 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-15T22:15:51.188 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-15T22:15:51.188 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-15T22:15:51.189 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-15T22:15:51.189 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-15T22:15:51.189 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-15T22:15:51.189 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-15T22:15:51.189 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-15T22:15:51.467 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-01T22:06:34.123489+0000 '>' 2024-09-01T22:06:34.123489+0000 2024-09-15T22:15:51.467 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-15T22:15:52.469 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-15T22:15:52.469 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-15T22:15:52.470 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-15T22:15:52.470 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-15T22:15:52.470 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-15T22:15:52.470 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-15T22:15:52.470 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-15T22:15:52.744 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-01T22:06:34.123489+0000 '>' 2024-09-01T22:06:34.123489+0000 2024-09-15T22:15:52.744 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-15T22:15:53.745 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-15T22:15:53.745 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-15T22:15:53.746 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-15T22:15:53.746 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-15T22:15:53.746 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-15T22:15:53.746 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-15T22:15:53.746 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-15T22:15:54.026 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-01T22:06:34.123489+0000 '>' 2024-09-01T22:06:34.123489+0000 2024-09-15T22:15:54.027 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-15T22:15:55.027 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-15T22:15:55.028 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-15T22:15:55.028 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-15T22:15:55.029 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-15T22:15:55.029 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-15T22:15:55.029 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-15T22:15:55.029 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-15T22:15:55.309 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-01T22:06:34.123489+0000 '>' 2024-09-01T22:06:34.123489+0000 2024-09-15T22:15:55.309 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-15T22:15:56.310 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-15T22:15:56.310 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-15T22:15:56.311 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-15T22:15:56.311 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-15T22:15:56.311 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-15T22:15:56.312 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-15T22:15:56.312 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-15T22:15:56.624 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-01T22:06:34.123489+0000 '>' 2024-09-01T22:06:34.123489+0000 2024-09-15T22:15:56.624 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-15T22:15:57.625 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-15T22:15:57.625 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-15T22:15:57.626 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-15T22:15:57.626 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-15T22:15:57.626 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-15T22:15:57.626 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-15T22:15:57.626 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-15T22:15:57.909 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-01T22:06:34.123489+0000 '>' 2024-09-01T22:06:34.123489+0000 2024-09-15T22:15:57.910 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-15T22:15:58.911 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-15T22:15:58.911 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-15T22:15:58.912 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-15T22:15:58.912 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-15T22:15:58.912 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-15T22:15:58.912 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-15T22:15:58.912 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-15T22:15:59.191 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-01T22:06:34.123489+0000 '>' 2024-09-01T22:06:34.123489+0000 2024-09-15T22:15:59.191 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-15T22:16:00.192 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-15T22:16:00.192 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-15T22:16:00.193 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-15T22:16:00.193 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-15T22:16:00.193 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-15T22:16:00.193 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-15T22:16:00.193 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-15T22:16:00.493 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-01T22:06:34.123489+0000 '>' 2024-09-01T22:06:34.123489+0000 2024-09-15T22:16:00.493 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-15T22:16:01.494 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-15T22:16:01.494 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-15T22:16:01.495 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-15T22:16:01.495 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-15T22:16:01.495 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-15T22:16:01.495 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-15T22:16:01.495 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-15T22:16:01.772 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-01T22:06:34.123489+0000 '>' 2024-09-01T22:06:34.123489+0000 2024-09-15T22:16:01.773 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-15T22:16:02.774 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-15T22:16:02.774 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-15T22:16:02.774 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-15T22:16:02.775 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-15T22:16:02.775 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-15T22:16:02.775 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-15T22:16:02.775 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-15T22:16:03.057 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-01T22:06:34.123489+0000 '>' 2024-09-01T22:06:34.123489+0000 2024-09-15T22:16:03.057 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-15T22:16:04.058 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-15T22:16:04.058 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-15T22:16:04.059 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-15T22:16:04.059 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-15T22:16:04.059 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-15T22:16:04.059 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-15T22:16:04.059 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-15T22:16:04.343 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-01T22:06:34.123489+0000 '>' 2024-09-01T22:06:34.123489+0000 2024-09-15T22:16:04.343 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-15T22:16:05.344 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-15T22:16:05.345 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-15T22:16:05.345 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-15T22:16:05.346 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-15T22:16:05.346 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-15T22:16:05.346 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-15T22:16:05.346 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-15T22:16:05.628 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-01T22:06:34.123489+0000 '>' 2024-09-01T22:06:34.123489+0000 2024-09-15T22:16:05.628 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-15T22:16:06.629 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-15T22:16:06.629 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-15T22:16:06.630 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-15T22:16:06.630 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-15T22:16:06.630 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-15T22:16:06.630 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-15T22:16:06.630 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-15T22:16:06.909 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-01T22:06:34.123489+0000 '>' 2024-09-01T22:06:34.123489+0000 2024-09-15T22:16:06.909 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-15T22:16:07.910 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-15T22:16:07.910 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-15T22:16:07.911 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-15T22:16:07.911 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-15T22:16:07.911 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-15T22:16:07.911 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-15T22:16:07.911 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-15T22:16:08.192 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-01T22:06:34.123489+0000 '>' 2024-09-01T22:06:34.123489+0000 2024-09-15T22:16:08.193 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-15T22:16:09.193 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-15T22:16:09.194 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-15T22:16:09.194 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-15T22:16:09.195 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-15T22:16:09.195 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-15T22:16:09.195 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-15T22:16:09.195 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-15T22:16:09.473 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-01T22:06:34.123489+0000 '>' 2024-09-01T22:06:34.123489+0000 2024-09-15T22:16:09.473 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-15T22:16:10.474 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-15T22:16:10.475 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-15T22:16:10.476 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-15T22:16:10.476 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-15T22:16:10.476 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-15T22:16:10.476 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-15T22:16:10.476 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-15T22:16:10.754 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-01T22:06:34.123489+0000 '>' 2024-09-01T22:06:34.123489+0000 2024-09-15T22:16:10.754 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-15T22:16:11.754 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-15T22:16:11.754 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-15T22:16:11.755 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-15T22:16:11.755 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-15T22:16:11.755 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-15T22:16:11.756 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-15T22:16:11.756 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-15T22:16:12.035 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-01T22:06:34.123489+0000 '>' 2024-09-01T22:06:34.123489+0000 2024-09-15T22:16:12.035 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-15T22:16:13.036 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-15T22:16:13.036 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-15T22:16:13.037 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-15T22:16:13.037 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-15T22:16:13.037 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-15T22:16:13.037 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-15T22:16:13.037 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-15T22:16:13.318 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-01T22:06:34.123489+0000 '>' 2024-09-01T22:06:34.123489+0000 2024-09-15T22:16:13.318 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-15T22:16:14.319 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-15T22:16:14.319 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-15T22:16:14.320 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-15T22:16:14.320 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-15T22:16:14.320 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-15T22:16:14.321 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-15T22:16:14.321 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-15T22:16:14.622 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-01T22:06:34.123489+0000 '>' 2024-09-01T22:06:34.123489+0000 2024-09-15T22:16:14.622 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-15T22:16:15.623 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-15T22:16:15.623 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-15T22:16:15.624 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-15T22:16:15.624 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-15T22:16:15.624 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-15T22:16:15.624 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-15T22:16:15.624 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-15T22:16:15.903 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-01T22:06:34.123489+0000 '>' 2024-09-01T22:06:34.123489+0000 2024-09-15T22:16:15.903 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-15T22:16:16.904 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-15T22:16:16.904 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-15T22:16:16.906 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-15T22:16:16.906 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-15T22:16:16.906 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-15T22:16:16.906 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-15T22:16:16.906 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-15T22:16:17.184 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-01T22:06:34.123489+0000 '>' 2024-09-01T22:06:34.123489+0000 2024-09-15T22:16:17.184 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-15T22:16:18.185 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-15T22:16:18.185 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-15T22:16:18.186 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-15T22:16:18.186 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-15T22:16:18.186 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-15T22:16:18.186 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-15T22:16:18.187 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-15T22:16:18.471 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-01T22:06:34.123489+0000 '>' 2024-09-01T22:06:34.123489+0000 2024-09-15T22:16:18.471 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-15T22:16:19.472 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-15T22:16:19.472 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-15T22:16:19.473 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-15T22:16:19.473 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-15T22:16:19.473 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-15T22:16:19.473 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-15T22:16:19.473 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-15T22:16:19.756 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-01T22:06:34.123489+0000 '>' 2024-09-01T22:06:34.123489+0000 2024-09-15T22:16:19.756 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-15T22:16:20.757 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-15T22:16:20.757 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-15T22:16:20.758 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-15T22:16:20.758 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-15T22:16:20.758 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-15T22:16:20.758 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-15T22:16:20.759 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-15T22:16:21.037 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-01T22:06:34.123489+0000 '>' 2024-09-01T22:06:34.123489+0000 2024-09-15T22:16:21.037 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-15T22:16:22.038 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-15T22:16:22.038 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-15T22:16:22.039 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-15T22:16:22.039 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-15T22:16:22.039 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-15T22:16:22.039 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-15T22:16:22.040 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-15T22:16:22.320 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-01T22:06:34.123489+0000 '>' 2024-09-01T22:06:34.123489+0000 2024-09-15T22:16:22.320 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-15T22:16:23.321 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-15T22:16:23.322 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-15T22:16:23.323 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-15T22:16:23.323 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-15T22:16:23.323 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-15T22:16:23.323 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-15T22:16:23.323 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-15T22:16:23.622 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-01T22:06:34.123489+0000 '>' 2024-09-01T22:06:34.123489+0000 2024-09-15T22:16:23.622 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-15T22:16:24.623 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-15T22:16:24.623 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-15T22:16:24.624 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-15T22:16:24.624 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-15T22:16:24.624 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-15T22:16:24.625 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-15T22:16:24.625 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-15T22:16:24.908 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-01T22:06:34.123489+0000 '>' 2024-09-01T22:06:34.123489+0000 2024-09-15T22:16:24.908 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-15T22:16:25.909 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-15T22:16:25.909 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-15T22:16:25.910 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-15T22:16:25.910 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-15T22:16:25.910 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-15T22:16:25.910 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-15T22:16:25.910 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-15T22:16:26.193 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-01T22:06:34.123489+0000 '>' 2024-09-01T22:06:34.123489+0000 2024-09-15T22:16:26.193 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-15T22:16:27.194 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-15T22:16:27.194 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-15T22:16:27.195 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-15T22:16:27.195 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-15T22:16:27.195 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-15T22:16:27.195 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-15T22:16:27.195 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-15T22:16:27.467 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-01T22:06:34.123489+0000 '>' 2024-09-01T22:06:34.123489+0000 2024-09-15T22:16:27.467 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-15T22:16:28.468 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-15T22:16:28.468 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-15T22:16:28.469 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-15T22:16:28.469 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-15T22:16:28.469 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-15T22:16:28.469 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-15T22:16:28.469 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-15T22:16:28.749 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-01T22:06:34.123489+0000 '>' 2024-09-01T22:06:34.123489+0000 2024-09-15T22:16:28.749 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-15T22:16:29.750 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-15T22:16:29.750 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-15T22:16:29.751 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-15T22:16:29.751 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-15T22:16:29.751 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-15T22:16:29.751 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-15T22:16:29.751 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-15T22:16:30.032 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-01T22:06:34.123489+0000 '>' 2024-09-01T22:06:34.123489+0000 2024-09-15T22:16:30.032 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-15T22:16:31.033 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-15T22:16:31.033 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-15T22:16:31.034 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-15T22:16:31.034 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-15T22:16:31.034 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-15T22:16:31.034 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-15T22:16:31.034 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-15T22:16:31.314 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-01T22:06:34.123489+0000 '>' 2024-09-01T22:06:34.123489+0000 2024-09-15T22:16:31.314 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-15T22:16:32.315 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-15T22:16:32.315 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-15T22:16:32.316 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-15T22:16:32.316 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-15T22:16:32.316 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-15T22:16:32.316 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-15T22:16:32.316 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-15T22:16:32.620 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-01T22:06:34.123489+0000 '>' 2024-09-01T22:06:34.123489+0000 2024-09-15T22:16:32.621 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-15T22:16:33.622 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-15T22:16:33.622 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-15T22:16:33.623 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-15T22:16:33.623 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-15T22:16:33.623 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-15T22:16:33.623 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-15T22:16:33.623 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-15T22:16:33.903 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-01T22:06:34.123489+0000 '>' 2024-09-01T22:06:34.123489+0000 2024-09-15T22:16:33.903 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-15T22:16:34.904 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-15T22:16:34.905 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-15T22:16:34.906 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-15T22:16:34.906 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-15T22:16:34.906 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-15T22:16:34.906 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-15T22:16:34.906 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-15T22:16:35.181 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-01T22:06:34.123489+0000 '>' 2024-09-01T22:06:34.123489+0000 2024-09-15T22:16:35.181 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-15T22:16:36.182 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-15T22:16:36.182 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-15T22:16:36.182 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-15T22:16:36.183 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-15T22:16:36.183 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-15T22:16:36.183 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-15T22:16:36.183 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-15T22:16:36.466 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-01T22:06:34.123489+0000 '>' 2024-09-01T22:06:34.123489+0000 2024-09-15T22:16:36.466 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-15T22:16:37.467 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-15T22:16:37.467 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-15T22:16:37.468 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-15T22:16:37.468 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-15T22:16:37.468 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-15T22:16:37.468 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-15T22:16:37.468 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-15T22:16:37.749 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-01T22:06:34.123489+0000 '>' 2024-09-01T22:06:34.123489+0000 2024-09-15T22:16:37.750 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-15T22:16:38.751 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-15T22:16:38.751 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-15T22:16:38.751 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-15T22:16:38.752 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-15T22:16:38.752 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-15T22:16:38.752 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-15T22:16:38.752 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-15T22:16:39.022 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-01T22:06:34.123489+0000 '>' 2024-09-01T22:06:34.123489+0000 2024-09-15T22:16:39.022 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-15T22:16:40.023 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-15T22:16:40.023 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-15T22:16:40.024 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-15T22:16:40.024 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-15T22:16:40.024 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-15T22:16:40.024 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-15T22:16:40.024 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-15T22:16:40.307 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-15T22:16:38.168116+0000 '>' 2024-09-01T22:06:34.123489+0000 2024-09-15T22:16:40.307 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2023: wait_for_scrub: return 0 2024-09-15T22:16:40.307 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:407: _scrub_abort: perf_counters td/osd-scrub-test 3 2024-09-15T22:16:40.307 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:43: perf_counters: local dir=td/osd-scrub-test 2024-09-15T22:16:40.307 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:44: perf_counters: local OSDS=3 2024-09-15T22:16:40.308 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:45: perf_counters: expr 3 - 1 2024-09-15T22:16:40.308 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:45: perf_counters: seq 0 2 2024-09-15T22:16:40.309 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:45: perf_counters: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-09-15T22:16:40.309 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:47: perf_counters: ceph tell osd.0 counter dump 2024-09-15T22:16:40.309 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:47: perf_counters: jq 'with_entries(select(.key | startswith("osd_scrub")))' 2024-09-15T22:16:40.445 INFO:tasks.workunit.client.0.smithi136.stdout:{ 2024-09-15T22:16:40.445 INFO:tasks.workunit.client.0.smithi136.stdout: "osd_scrub_dp_ec": [ 2024-09-15T22:16:40.445 INFO:tasks.workunit.client.0.smithi136.stdout: { 2024-09-15T22:16:40.445 INFO:tasks.workunit.client.0.smithi136.stdout: "labels": { 2024-09-15T22:16:40.445 INFO:tasks.workunit.client.0.smithi136.stdout: "level": "deep", 2024-09-15T22:16:40.445 INFO:tasks.workunit.client.0.smithi136.stdout: "pooltype": "ec" 2024-09-15T22:16:40.445 INFO:tasks.workunit.client.0.smithi136.stdout: }, 2024-09-15T22:16:40.445 INFO:tasks.workunit.client.0.smithi136.stdout: "counters": { 2024-09-15T22:16:40.446 INFO:tasks.workunit.client.0.smithi136.stdout: "num_scrubs_started": 0, 2024-09-15T22:16:40.446 INFO:tasks.workunit.client.0.smithi136.stdout: "num_scrubs_past_reservation": 0, 2024-09-15T22:16:40.446 INFO:tasks.workunit.client.0.smithi136.stdout: "successful_scrubs": 0, 2024-09-15T22:16:40.446 INFO:tasks.workunit.client.0.smithi136.stdout: "successful_scrubs_elapsed": { 2024-09-15T22:16:40.446 INFO:tasks.workunit.client.0.smithi136.stdout: "avgcount": 0, 2024-09-15T22:16:40.446 INFO:tasks.workunit.client.0.smithi136.stdout: "sum": 0E-9, 2024-09-15T22:16:40.446 INFO:tasks.workunit.client.0.smithi136.stdout: "avgtime": 0E-9 2024-09-15T22:16:40.446 INFO:tasks.workunit.client.0.smithi136.stdout: }, 2024-09-15T22:16:40.446 INFO:tasks.workunit.client.0.smithi136.stdout: "failed_scrubs": 0, 2024-09-15T22:16:40.446 INFO:tasks.workunit.client.0.smithi136.stdout: "failed_scrubs_elapsed": { 2024-09-15T22:16:40.446 INFO:tasks.workunit.client.0.smithi136.stdout: "avgcount": 0, 2024-09-15T22:16:40.446 INFO:tasks.workunit.client.0.smithi136.stdout: "sum": 0E-9, 2024-09-15T22:16:40.446 INFO:tasks.workunit.client.0.smithi136.stdout: "avgtime": 0E-9 2024-09-15T22:16:40.447 INFO:tasks.workunit.client.0.smithi136.stdout: }, 2024-09-15T22:16:40.447 INFO:tasks.workunit.client.0.smithi136.stdout: "preemptions": 0, 2024-09-15T22:16:40.447 INFO:tasks.workunit.client.0.smithi136.stdout: "chunk_selected": 0, 2024-09-15T22:16:40.447 INFO:tasks.workunit.client.0.smithi136.stdout: "chunk_busy": 0, 2024-09-15T22:16:40.447 INFO:tasks.workunit.client.0.smithi136.stdout: "locked_object": 0, 2024-09-15T22:16:40.447 INFO:tasks.workunit.client.0.smithi136.stdout: "write_blocked_by_scrub": 0, 2024-09-15T22:16:40.447 INFO:tasks.workunit.client.0.smithi136.stdout: "scrub_reservations_completed": 0, 2024-09-15T22:16:40.447 INFO:tasks.workunit.client.0.smithi136.stdout: "successful_reservations_elapsed": { 2024-09-15T22:16:40.447 INFO:tasks.workunit.client.0.smithi136.stdout: "avgcount": 0, 2024-09-15T22:16:40.447 INFO:tasks.workunit.client.0.smithi136.stdout: "sum": 0E-9, 2024-09-15T22:16:40.447 INFO:tasks.workunit.client.0.smithi136.stdout: "avgtime": 0E-9 2024-09-15T22:16:40.447 INFO:tasks.workunit.client.0.smithi136.stdout: }, 2024-09-15T22:16:40.448 INFO:tasks.workunit.client.0.smithi136.stdout: "reservation_process_aborted": 0, 2024-09-15T22:16:40.448 INFO:tasks.workunit.client.0.smithi136.stdout: "reservation_process_failure": 0, 2024-09-15T22:16:40.448 INFO:tasks.workunit.client.0.smithi136.stdout: "reservation_process_skipped": 0, 2024-09-15T22:16:40.448 INFO:tasks.workunit.client.0.smithi136.stdout: "failed_reservations_elapsed": { 2024-09-15T22:16:40.448 INFO:tasks.workunit.client.0.smithi136.stdout: "avgcount": 0, 2024-09-15T22:16:40.448 INFO:tasks.workunit.client.0.smithi136.stdout: "sum": 0E-9, 2024-09-15T22:16:40.448 INFO:tasks.workunit.client.0.smithi136.stdout: "avgtime": 0E-9 2024-09-15T22:16:40.448 INFO:tasks.workunit.client.0.smithi136.stdout: }, 2024-09-15T22:16:40.448 INFO:tasks.workunit.client.0.smithi136.stdout: "replicas_in_reservation": 0 2024-09-15T22:16:40.448 INFO:tasks.workunit.client.0.smithi136.stdout: } 2024-09-15T22:16:40.448 INFO:tasks.workunit.client.0.smithi136.stdout: } 2024-09-15T22:16:40.448 INFO:tasks.workunit.client.0.smithi136.stdout: ], 2024-09-15T22:16:40.449 INFO:tasks.workunit.client.0.smithi136.stdout: "osd_scrub_dp_repl": [ 2024-09-15T22:16:40.449 INFO:tasks.workunit.client.0.smithi136.stdout: { 2024-09-15T22:16:40.449 INFO:tasks.workunit.client.0.smithi136.stdout: "labels": { 2024-09-15T22:16:40.449 INFO:tasks.workunit.client.0.smithi136.stdout: "level": "deep", 2024-09-15T22:16:40.449 INFO:tasks.workunit.client.0.smithi136.stdout: "pooltype": "replicated" 2024-09-15T22:16:40.450 INFO:tasks.workunit.client.0.smithi136.stdout: }, 2024-09-15T22:16:40.450 INFO:tasks.workunit.client.0.smithi136.stdout: "counters": { 2024-09-15T22:16:40.450 INFO:tasks.workunit.client.0.smithi136.stdout: "num_scrubs_started": 0, 2024-09-15T22:16:40.450 INFO:tasks.workunit.client.0.smithi136.stdout: "num_scrubs_past_reservation": 0, 2024-09-15T22:16:40.450 INFO:tasks.workunit.client.0.smithi136.stdout: "successful_scrubs": 0, 2024-09-15T22:16:40.450 INFO:tasks.workunit.client.0.smithi136.stdout: "successful_scrubs_elapsed": { 2024-09-15T22:16:40.450 INFO:tasks.workunit.client.0.smithi136.stdout: "avgcount": 0, 2024-09-15T22:16:40.450 INFO:tasks.workunit.client.0.smithi136.stdout: "sum": 0E-9, 2024-09-15T22:16:40.450 INFO:tasks.workunit.client.0.smithi136.stdout: "avgtime": 0E-9 2024-09-15T22:16:40.450 INFO:tasks.workunit.client.0.smithi136.stdout: }, 2024-09-15T22:16:40.450 INFO:tasks.workunit.client.0.smithi136.stdout: "failed_scrubs": 0, 2024-09-15T22:16:40.450 INFO:tasks.workunit.client.0.smithi136.stdout: "failed_scrubs_elapsed": { 2024-09-15T22:16:40.450 INFO:tasks.workunit.client.0.smithi136.stdout: "avgcount": 0, 2024-09-15T22:16:40.450 INFO:tasks.workunit.client.0.smithi136.stdout: "sum": 0E-9, 2024-09-15T22:16:40.451 INFO:tasks.workunit.client.0.smithi136.stdout: "avgtime": 0E-9 2024-09-15T22:16:40.451 INFO:tasks.workunit.client.0.smithi136.stdout: }, 2024-09-15T22:16:40.451 INFO:tasks.workunit.client.0.smithi136.stdout: "preemptions": 0, 2024-09-15T22:16:40.451 INFO:tasks.workunit.client.0.smithi136.stdout: "chunk_selected": 0, 2024-09-15T22:16:40.451 INFO:tasks.workunit.client.0.smithi136.stdout: "chunk_busy": 0, 2024-09-15T22:16:40.451 INFO:tasks.workunit.client.0.smithi136.stdout: "locked_object": 0, 2024-09-15T22:16:40.451 INFO:tasks.workunit.client.0.smithi136.stdout: "write_blocked_by_scrub": 0, 2024-09-15T22:16:40.451 INFO:tasks.workunit.client.0.smithi136.stdout: "scrub_reservations_completed": 0, 2024-09-15T22:16:40.451 INFO:tasks.workunit.client.0.smithi136.stdout: "successful_reservations_elapsed": { 2024-09-15T22:16:40.451 INFO:tasks.workunit.client.0.smithi136.stdout: "avgcount": 0, 2024-09-15T22:16:40.451 INFO:tasks.workunit.client.0.smithi136.stdout: "sum": 0E-9, 2024-09-15T22:16:40.451 INFO:tasks.workunit.client.0.smithi136.stdout: "avgtime": 0E-9 2024-09-15T22:16:40.451 INFO:tasks.workunit.client.0.smithi136.stdout: }, 2024-09-15T22:16:40.452 INFO:tasks.workunit.client.0.smithi136.stdout: "reservation_process_aborted": 0, 2024-09-15T22:16:40.452 INFO:tasks.workunit.client.0.smithi136.stdout: "reservation_process_failure": 0, 2024-09-15T22:16:40.452 INFO:tasks.workunit.client.0.smithi136.stdout: "reservation_process_skipped": 0, 2024-09-15T22:16:40.452 INFO:tasks.workunit.client.0.smithi136.stdout: "failed_reservations_elapsed": { 2024-09-15T22:16:40.452 INFO:tasks.workunit.client.0.smithi136.stdout: "avgcount": 0, 2024-09-15T22:16:40.452 INFO:tasks.workunit.client.0.smithi136.stdout: "sum": 0E-9, 2024-09-15T22:16:40.452 INFO:tasks.workunit.client.0.smithi136.stdout: "avgtime": 0E-9 2024-09-15T22:16:40.453 INFO:tasks.workunit.client.0.smithi136.stdout: }, 2024-09-15T22:16:40.453 INFO:tasks.workunit.client.0.smithi136.stdout: "replicas_in_reservation": 0 2024-09-15T22:16:40.453 INFO:tasks.workunit.client.0.smithi136.stdout: } 2024-09-15T22:16:40.453 INFO:tasks.workunit.client.0.smithi136.stdout: } 2024-09-15T22:16:40.453 INFO:tasks.workunit.client.0.smithi136.stdout: ], 2024-09-15T22:16:40.453 INFO:tasks.workunit.client.0.smithi136.stdout: "osd_scrub_sh_ec": [ 2024-09-15T22:16:40.453 INFO:tasks.workunit.client.0.smithi136.stdout: { 2024-09-15T22:16:40.453 INFO:tasks.workunit.client.0.smithi136.stdout: "labels": { 2024-09-15T22:16:40.453 INFO:tasks.workunit.client.0.smithi136.stdout: "level": "shallow", 2024-09-15T22:16:40.453 INFO:tasks.workunit.client.0.smithi136.stdout: "pooltype": "ec" 2024-09-15T22:16:40.453 INFO:tasks.workunit.client.0.smithi136.stdout: }, 2024-09-15T22:16:40.454 INFO:tasks.workunit.client.0.smithi136.stdout: "counters": { 2024-09-15T22:16:40.454 INFO:tasks.workunit.client.0.smithi136.stdout: "num_scrubs_started": 0, 2024-09-15T22:16:40.454 INFO:tasks.workunit.client.0.smithi136.stdout: "num_scrubs_past_reservation": 0, 2024-09-15T22:16:40.454 INFO:tasks.workunit.client.0.smithi136.stdout: "successful_scrubs": 0, 2024-09-15T22:16:40.454 INFO:tasks.workunit.client.0.smithi136.stdout: "successful_scrubs_elapsed": { 2024-09-15T22:16:40.454 INFO:tasks.workunit.client.0.smithi136.stdout: "avgcount": 0, 2024-09-15T22:16:40.454 INFO:tasks.workunit.client.0.smithi136.stdout: "sum": 0E-9, 2024-09-15T22:16:40.454 INFO:tasks.workunit.client.0.smithi136.stdout: "avgtime": 0E-9 2024-09-15T22:16:40.454 INFO:tasks.workunit.client.0.smithi136.stdout: }, 2024-09-15T22:16:40.454 INFO:tasks.workunit.client.0.smithi136.stdout: "failed_scrubs": 0, 2024-09-15T22:16:40.454 INFO:tasks.workunit.client.0.smithi136.stdout: "failed_scrubs_elapsed": { 2024-09-15T22:16:40.454 INFO:tasks.workunit.client.0.smithi136.stdout: "avgcount": 0, 2024-09-15T22:16:40.454 INFO:tasks.workunit.client.0.smithi136.stdout: "sum": 0E-9, 2024-09-15T22:16:40.455 INFO:tasks.workunit.client.0.smithi136.stdout: "avgtime": 0E-9 2024-09-15T22:16:40.455 INFO:tasks.workunit.client.0.smithi136.stdout: }, 2024-09-15T22:16:40.455 INFO:tasks.workunit.client.0.smithi136.stdout: "preemptions": 0, 2024-09-15T22:16:40.455 INFO:tasks.workunit.client.0.smithi136.stdout: "chunk_selected": 0, 2024-09-15T22:16:40.455 INFO:tasks.workunit.client.0.smithi136.stdout: "chunk_busy": 0, 2024-09-15T22:16:40.455 INFO:tasks.workunit.client.0.smithi136.stdout: "locked_object": 0, 2024-09-15T22:16:40.455 INFO:tasks.workunit.client.0.smithi136.stdout: "write_blocked_by_scrub": 0, 2024-09-15T22:16:40.455 INFO:tasks.workunit.client.0.smithi136.stdout: "scrub_reservations_completed": 0, 2024-09-15T22:16:40.455 INFO:tasks.workunit.client.0.smithi136.stdout: "successful_reservations_elapsed": { 2024-09-15T22:16:40.455 INFO:tasks.workunit.client.0.smithi136.stdout: "avgcount": 0, 2024-09-15T22:16:40.455 INFO:tasks.workunit.client.0.smithi136.stdout: "sum": 0E-9, 2024-09-15T22:16:40.455 INFO:tasks.workunit.client.0.smithi136.stdout: "avgtime": 0E-9 2024-09-15T22:16:40.455 INFO:tasks.workunit.client.0.smithi136.stdout: }, 2024-09-15T22:16:40.456 INFO:tasks.workunit.client.0.smithi136.stdout: "reservation_process_aborted": 0, 2024-09-15T22:16:40.456 INFO:tasks.workunit.client.0.smithi136.stdout: "reservation_process_failure": 0, 2024-09-15T22:16:40.456 INFO:tasks.workunit.client.0.smithi136.stdout: "reservation_process_skipped": 0, 2024-09-15T22:16:40.456 INFO:tasks.workunit.client.0.smithi136.stdout: "failed_reservations_elapsed": { 2024-09-15T22:16:40.456 INFO:tasks.workunit.client.0.smithi136.stdout: "avgcount": 0, 2024-09-15T22:16:40.456 INFO:tasks.workunit.client.0.smithi136.stdout: "sum": 0E-9, 2024-09-15T22:16:40.456 INFO:tasks.workunit.client.0.smithi136.stdout: "avgtime": 0E-9 2024-09-15T22:16:40.456 INFO:tasks.workunit.client.0.smithi136.stdout: }, 2024-09-15T22:16:40.456 INFO:tasks.workunit.client.0.smithi136.stdout: "replicas_in_reservation": 0 2024-09-15T22:16:40.456 INFO:tasks.workunit.client.0.smithi136.stdout: } 2024-09-15T22:16:40.456 INFO:tasks.workunit.client.0.smithi136.stdout: } 2024-09-15T22:16:40.456 INFO:tasks.workunit.client.0.smithi136.stdout: ], 2024-09-15T22:16:40.456 INFO:tasks.workunit.client.0.smithi136.stdout: "osd_scrub_sh_repl": [ 2024-09-15T22:16:40.457 INFO:tasks.workunit.client.0.smithi136.stdout: { 2024-09-15T22:16:40.457 INFO:tasks.workunit.client.0.smithi136.stdout: "labels": { 2024-09-15T22:16:40.457 INFO:tasks.workunit.client.0.smithi136.stdout: "level": "shallow", 2024-09-15T22:16:40.457 INFO:tasks.workunit.client.0.smithi136.stdout: "pooltype": "replicated" 2024-09-15T22:16:40.457 INFO:tasks.workunit.client.0.smithi136.stdout: }, 2024-09-15T22:16:40.457 INFO:tasks.workunit.client.0.smithi136.stdout: "counters": { 2024-09-15T22:16:40.457 INFO:tasks.workunit.client.0.smithi136.stdout: "num_scrubs_started": 0, 2024-09-15T22:16:40.457 INFO:tasks.workunit.client.0.smithi136.stdout: "num_scrubs_past_reservation": 0, 2024-09-15T22:16:40.457 INFO:tasks.workunit.client.0.smithi136.stdout: "successful_scrubs": 0, 2024-09-15T22:16:40.457 INFO:tasks.workunit.client.0.smithi136.stdout: "successful_scrubs_elapsed": { 2024-09-15T22:16:40.457 INFO:tasks.workunit.client.0.smithi136.stdout: "avgcount": 0, 2024-09-15T22:16:40.457 INFO:tasks.workunit.client.0.smithi136.stdout: "sum": 0E-9, 2024-09-15T22:16:40.457 INFO:tasks.workunit.client.0.smithi136.stdout: "avgtime": 0E-9 2024-09-15T22:16:40.458 INFO:tasks.workunit.client.0.smithi136.stdout: }, 2024-09-15T22:16:40.458 INFO:tasks.workunit.client.0.smithi136.stdout: "failed_scrubs": 0, 2024-09-15T22:16:40.458 INFO:tasks.workunit.client.0.smithi136.stdout: "failed_scrubs_elapsed": { 2024-09-15T22:16:40.458 INFO:tasks.workunit.client.0.smithi136.stdout: "avgcount": 0, 2024-09-15T22:16:40.458 INFO:tasks.workunit.client.0.smithi136.stdout: "sum": 0E-9, 2024-09-15T22:16:40.458 INFO:tasks.workunit.client.0.smithi136.stdout: "avgtime": 0E-9 2024-09-15T22:16:40.458 INFO:tasks.workunit.client.0.smithi136.stdout: }, 2024-09-15T22:16:40.458 INFO:tasks.workunit.client.0.smithi136.stdout: "preemptions": 0, 2024-09-15T22:16:40.458 INFO:tasks.workunit.client.0.smithi136.stdout: "chunk_selected": 0, 2024-09-15T22:16:40.458 INFO:tasks.workunit.client.0.smithi136.stdout: "chunk_busy": 0, 2024-09-15T22:16:40.458 INFO:tasks.workunit.client.0.smithi136.stdout: "locked_object": 0, 2024-09-15T22:16:40.458 INFO:tasks.workunit.client.0.smithi136.stdout: "write_blocked_by_scrub": 0, 2024-09-15T22:16:40.459 INFO:tasks.workunit.client.0.smithi136.stdout: "scrub_reservations_completed": 0, 2024-09-15T22:16:40.459 INFO:tasks.workunit.client.0.smithi136.stdout: "successful_reservations_elapsed": { 2024-09-15T22:16:40.459 INFO:tasks.workunit.client.0.smithi136.stdout: "avgcount": 0, 2024-09-15T22:16:40.459 INFO:tasks.workunit.client.0.smithi136.stdout: "sum": 0E-9, 2024-09-15T22:16:40.459 INFO:tasks.workunit.client.0.smithi136.stdout: "avgtime": 0E-9 2024-09-15T22:16:40.459 INFO:tasks.workunit.client.0.smithi136.stdout: }, 2024-09-15T22:16:40.459 INFO:tasks.workunit.client.0.smithi136.stdout: "reservation_process_aborted": 0, 2024-09-15T22:16:40.459 INFO:tasks.workunit.client.0.smithi136.stdout: "reservation_process_failure": 0, 2024-09-15T22:16:40.459 INFO:tasks.workunit.client.0.smithi136.stdout: "reservation_process_skipped": 0, 2024-09-15T22:16:40.459 INFO:tasks.workunit.client.0.smithi136.stdout: "failed_reservations_elapsed": { 2024-09-15T22:16:40.459 INFO:tasks.workunit.client.0.smithi136.stdout: "avgcount": 0, 2024-09-15T22:16:40.460 INFO:tasks.workunit.client.0.smithi136.stdout: "sum": 0E-9, 2024-09-15T22:16:40.460 INFO:tasks.workunit.client.0.smithi136.stdout: "avgtime": 0E-9 2024-09-15T22:16:40.460 INFO:tasks.workunit.client.0.smithi136.stdout: }, 2024-09-15T22:16:40.460 INFO:tasks.workunit.client.0.smithi136.stdout: "replicas_in_reservation": 0 2024-09-15T22:16:40.460 INFO:tasks.workunit.client.0.smithi136.stdout: } 2024-09-15T22:16:40.460 INFO:tasks.workunit.client.0.smithi136.stdout: } 2024-09-15T22:16:40.460 INFO:tasks.workunit.client.0.smithi136.stdout: ] 2024-09-15T22:16:40.460 INFO:tasks.workunit.client.0.smithi136.stdout:} 2024-09-15T22:16:40.461 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:45: perf_counters: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-09-15T22:16:40.461 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:47: perf_counters: ceph tell osd.1 counter dump 2024-09-15T22:16:40.461 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:47: perf_counters: jq 'with_entries(select(.key | startswith("osd_scrub")))' 2024-09-15T22:16:40.546 INFO:tasks.workunit.client.0.smithi136.stdout:{ 2024-09-15T22:16:40.546 INFO:tasks.workunit.client.0.smithi136.stdout: "osd_scrub_dp_ec": [ 2024-09-15T22:16:40.546 INFO:tasks.workunit.client.0.smithi136.stdout: { 2024-09-15T22:16:40.546 INFO:tasks.workunit.client.0.smithi136.stdout: "labels": { 2024-09-15T22:16:40.546 INFO:tasks.workunit.client.0.smithi136.stdout: "level": "deep", 2024-09-15T22:16:40.546 INFO:tasks.workunit.client.0.smithi136.stdout: "pooltype": "ec" 2024-09-15T22:16:40.547 INFO:tasks.workunit.client.0.smithi136.stdout: }, 2024-09-15T22:16:40.547 INFO:tasks.workunit.client.0.smithi136.stdout: "counters": { 2024-09-15T22:16:40.547 INFO:tasks.workunit.client.0.smithi136.stdout: "num_scrubs_started": 0, 2024-09-15T22:16:40.547 INFO:tasks.workunit.client.0.smithi136.stdout: "num_scrubs_past_reservation": 0, 2024-09-15T22:16:40.547 INFO:tasks.workunit.client.0.smithi136.stdout: "successful_scrubs": 0, 2024-09-15T22:16:40.547 INFO:tasks.workunit.client.0.smithi136.stdout: "successful_scrubs_elapsed": { 2024-09-15T22:16:40.547 INFO:tasks.workunit.client.0.smithi136.stdout: "avgcount": 0, 2024-09-15T22:16:40.547 INFO:tasks.workunit.client.0.smithi136.stdout: "sum": 0E-9, 2024-09-15T22:16:40.547 INFO:tasks.workunit.client.0.smithi136.stdout: "avgtime": 0E-9 2024-09-15T22:16:40.547 INFO:tasks.workunit.client.0.smithi136.stdout: }, 2024-09-15T22:16:40.547 INFO:tasks.workunit.client.0.smithi136.stdout: "failed_scrubs": 0, 2024-09-15T22:16:40.547 INFO:tasks.workunit.client.0.smithi136.stdout: "failed_scrubs_elapsed": { 2024-09-15T22:16:40.547 INFO:tasks.workunit.client.0.smithi136.stdout: "avgcount": 0, 2024-09-15T22:16:40.547 INFO:tasks.workunit.client.0.smithi136.stdout: "sum": 0E-9, 2024-09-15T22:16:40.548 INFO:tasks.workunit.client.0.smithi136.stdout: "avgtime": 0E-9 2024-09-15T22:16:40.548 INFO:tasks.workunit.client.0.smithi136.stdout: }, 2024-09-15T22:16:40.548 INFO:tasks.workunit.client.0.smithi136.stdout: "preemptions": 0, 2024-09-15T22:16:40.548 INFO:tasks.workunit.client.0.smithi136.stdout: "chunk_selected": 0, 2024-09-15T22:16:40.548 INFO:tasks.workunit.client.0.smithi136.stdout: "chunk_busy": 0, 2024-09-15T22:16:40.548 INFO:tasks.workunit.client.0.smithi136.stdout: "locked_object": 0, 2024-09-15T22:16:40.548 INFO:tasks.workunit.client.0.smithi136.stdout: "write_blocked_by_scrub": 0, 2024-09-15T22:16:40.548 INFO:tasks.workunit.client.0.smithi136.stdout: "scrub_reservations_completed": 0, 2024-09-15T22:16:40.548 INFO:tasks.workunit.client.0.smithi136.stdout: "successful_reservations_elapsed": { 2024-09-15T22:16:40.548 INFO:tasks.workunit.client.0.smithi136.stdout: "avgcount": 0, 2024-09-15T22:16:40.548 INFO:tasks.workunit.client.0.smithi136.stdout: "sum": 0E-9, 2024-09-15T22:16:40.548 INFO:tasks.workunit.client.0.smithi136.stdout: "avgtime": 0E-9 2024-09-15T22:16:40.548 INFO:tasks.workunit.client.0.smithi136.stdout: }, 2024-09-15T22:16:40.549 INFO:tasks.workunit.client.0.smithi136.stdout: "reservation_process_aborted": 0, 2024-09-15T22:16:40.549 INFO:tasks.workunit.client.0.smithi136.stdout: "reservation_process_failure": 0, 2024-09-15T22:16:40.549 INFO:tasks.workunit.client.0.smithi136.stdout: "reservation_process_skipped": 0, 2024-09-15T22:16:40.549 INFO:tasks.workunit.client.0.smithi136.stdout: "failed_reservations_elapsed": { 2024-09-15T22:16:40.549 INFO:tasks.workunit.client.0.smithi136.stdout: "avgcount": 0, 2024-09-15T22:16:40.549 INFO:tasks.workunit.client.0.smithi136.stdout: "sum": 0E-9, 2024-09-15T22:16:40.549 INFO:tasks.workunit.client.0.smithi136.stdout: "avgtime": 0E-9 2024-09-15T22:16:40.549 INFO:tasks.workunit.client.0.smithi136.stdout: }, 2024-09-15T22:16:40.549 INFO:tasks.workunit.client.0.smithi136.stdout: "replicas_in_reservation": 0 2024-09-15T22:16:40.549 INFO:tasks.workunit.client.0.smithi136.stdout: } 2024-09-15T22:16:40.549 INFO:tasks.workunit.client.0.smithi136.stdout: } 2024-09-15T22:16:40.550 INFO:tasks.workunit.client.0.smithi136.stdout: ], 2024-09-15T22:16:40.550 INFO:tasks.workunit.client.0.smithi136.stdout: "osd_scrub_dp_repl": [ 2024-09-15T22:16:40.550 INFO:tasks.workunit.client.0.smithi136.stdout: { 2024-09-15T22:16:40.550 INFO:tasks.workunit.client.0.smithi136.stdout: "labels": { 2024-09-15T22:16:40.550 INFO:tasks.workunit.client.0.smithi136.stdout: "level": "deep", 2024-09-15T22:16:40.550 INFO:tasks.workunit.client.0.smithi136.stdout: "pooltype": "replicated" 2024-09-15T22:16:40.550 INFO:tasks.workunit.client.0.smithi136.stdout: }, 2024-09-15T22:16:40.550 INFO:tasks.workunit.client.0.smithi136.stdout: "counters": { 2024-09-15T22:16:40.550 INFO:tasks.workunit.client.0.smithi136.stdout: "num_scrubs_started": 2, 2024-09-15T22:16:40.550 INFO:tasks.workunit.client.0.smithi136.stdout: "num_scrubs_past_reservation": 2, 2024-09-15T22:16:40.550 INFO:tasks.workunit.client.0.smithi136.stdout: "successful_scrubs": 1, 2024-09-15T22:16:40.550 INFO:tasks.workunit.client.0.smithi136.stdout: "successful_scrubs_elapsed": { 2024-09-15T22:16:40.550 INFO:tasks.workunit.client.0.smithi136.stdout: "avgcount": 1, 2024-09-15T22:16:40.551 INFO:tasks.workunit.client.0.smithi136.stdout: "sum": 337.084758534, 2024-09-15T22:16:40.551 INFO:tasks.workunit.client.0.smithi136.stdout: "avgtime": 337.084758534 2024-09-15T22:16:40.551 INFO:tasks.workunit.client.0.smithi136.stdout: }, 2024-09-15T22:16:40.551 INFO:tasks.workunit.client.0.smithi136.stdout: "failed_scrubs": 1, 2024-09-15T22:16:40.551 INFO:tasks.workunit.client.0.smithi136.stdout: "failed_scrubs_elapsed": { 2024-09-15T22:16:40.551 INFO:tasks.workunit.client.0.smithi136.stdout: "avgcount": 1, 2024-09-15T22:16:40.551 INFO:tasks.workunit.client.0.smithi136.stdout: "sum": 10.104783762, 2024-09-15T22:16:40.551 INFO:tasks.workunit.client.0.smithi136.stdout: "avgtime": 10.104783762 2024-09-15T22:16:40.551 INFO:tasks.workunit.client.0.smithi136.stdout: }, 2024-09-15T22:16:40.551 INFO:tasks.workunit.client.0.smithi136.stdout: "preemptions": 0, 2024-09-15T22:16:40.551 INFO:tasks.workunit.client.0.smithi136.stdout: "chunk_selected": 68, 2024-09-15T22:16:40.551 INFO:tasks.workunit.client.0.smithi136.stdout: "chunk_busy": 0, 2024-09-15T22:16:40.551 INFO:tasks.workunit.client.0.smithi136.stdout: "locked_object": 0, 2024-09-15T22:16:40.551 INFO:tasks.workunit.client.0.smithi136.stdout: "write_blocked_by_scrub": 0, 2024-09-15T22:16:40.552 INFO:tasks.workunit.client.0.smithi136.stdout: "scrub_reservations_completed": 2, 2024-09-15T22:16:40.552 INFO:tasks.workunit.client.0.smithi136.stdout: "successful_reservations_elapsed": { 2024-09-15T22:16:40.552 INFO:tasks.workunit.client.0.smithi136.stdout: "avgcount": 2, 2024-09-15T22:16:40.552 INFO:tasks.workunit.client.0.smithi136.stdout: "sum": 0.000999979, 2024-09-15T22:16:40.552 INFO:tasks.workunit.client.0.smithi136.stdout: "avgtime": 0.000499989 2024-09-15T22:16:40.552 INFO:tasks.workunit.client.0.smithi136.stdout: }, 2024-09-15T22:16:40.552 INFO:tasks.workunit.client.0.smithi136.stdout: "reservation_process_aborted": 0, 2024-09-15T22:16:40.552 INFO:tasks.workunit.client.0.smithi136.stdout: "reservation_process_failure": 0, 2024-09-15T22:16:40.552 INFO:tasks.workunit.client.0.smithi136.stdout: "reservation_process_skipped": 0, 2024-09-15T22:16:40.552 INFO:tasks.workunit.client.0.smithi136.stdout: "failed_reservations_elapsed": { 2024-09-15T22:16:40.552 INFO:tasks.workunit.client.0.smithi136.stdout: "avgcount": 0, 2024-09-15T22:16:40.552 INFO:tasks.workunit.client.0.smithi136.stdout: "sum": 0E-9, 2024-09-15T22:16:40.552 INFO:tasks.workunit.client.0.smithi136.stdout: "avgtime": 0E-9 2024-09-15T22:16:40.552 INFO:tasks.workunit.client.0.smithi136.stdout: }, 2024-09-15T22:16:40.553 INFO:tasks.workunit.client.0.smithi136.stdout: "replicas_in_reservation": 2 2024-09-15T22:16:40.553 INFO:tasks.workunit.client.0.smithi136.stdout: } 2024-09-15T22:16:40.553 INFO:tasks.workunit.client.0.smithi136.stdout: } 2024-09-15T22:16:40.553 INFO:tasks.workunit.client.0.smithi136.stdout: ], 2024-09-15T22:16:40.553 INFO:tasks.workunit.client.0.smithi136.stdout: "osd_scrub_sh_ec": [ 2024-09-15T22:16:40.553 INFO:tasks.workunit.client.0.smithi136.stdout: { 2024-09-15T22:16:40.553 INFO:tasks.workunit.client.0.smithi136.stdout: "labels": { 2024-09-15T22:16:40.553 INFO:tasks.workunit.client.0.smithi136.stdout: "level": "shallow", 2024-09-15T22:16:40.553 INFO:tasks.workunit.client.0.smithi136.stdout: "pooltype": "ec" 2024-09-15T22:16:40.553 INFO:tasks.workunit.client.0.smithi136.stdout: }, 2024-09-15T22:16:40.553 INFO:tasks.workunit.client.0.smithi136.stdout: "counters": { 2024-09-15T22:16:40.553 INFO:tasks.workunit.client.0.smithi136.stdout: "num_scrubs_started": 0, 2024-09-15T22:16:40.554 INFO:tasks.workunit.client.0.smithi136.stdout: "num_scrubs_past_reservation": 0, 2024-09-15T22:16:40.554 INFO:tasks.workunit.client.0.smithi136.stdout: "successful_scrubs": 0, 2024-09-15T22:16:40.554 INFO:tasks.workunit.client.0.smithi136.stdout: "successful_scrubs_elapsed": { 2024-09-15T22:16:40.554 INFO:tasks.workunit.client.0.smithi136.stdout: "avgcount": 0, 2024-09-15T22:16:40.554 INFO:tasks.workunit.client.0.smithi136.stdout: "sum": 0E-9, 2024-09-15T22:16:40.554 INFO:tasks.workunit.client.0.smithi136.stdout: "avgtime": 0E-9 2024-09-15T22:16:40.554 INFO:tasks.workunit.client.0.smithi136.stdout: }, 2024-09-15T22:16:40.554 INFO:tasks.workunit.client.0.smithi136.stdout: "failed_scrubs": 0, 2024-09-15T22:16:40.554 INFO:tasks.workunit.client.0.smithi136.stdout: "failed_scrubs_elapsed": { 2024-09-15T22:16:40.554 INFO:tasks.workunit.client.0.smithi136.stdout: "avgcount": 0, 2024-09-15T22:16:40.554 INFO:tasks.workunit.client.0.smithi136.stdout: "sum": 0E-9, 2024-09-15T22:16:40.554 INFO:tasks.workunit.client.0.smithi136.stdout: "avgtime": 0E-9 2024-09-15T22:16:40.554 INFO:tasks.workunit.client.0.smithi136.stdout: }, 2024-09-15T22:16:40.554 INFO:tasks.workunit.client.0.smithi136.stdout: "preemptions": 0, 2024-09-15T22:16:40.555 INFO:tasks.workunit.client.0.smithi136.stdout: "chunk_selected": 0, 2024-09-15T22:16:40.555 INFO:tasks.workunit.client.0.smithi136.stdout: "chunk_busy": 0, 2024-09-15T22:16:40.555 INFO:tasks.workunit.client.0.smithi136.stdout: "locked_object": 0, 2024-09-15T22:16:40.555 INFO:tasks.workunit.client.0.smithi136.stdout: "write_blocked_by_scrub": 0, 2024-09-15T22:16:40.555 INFO:tasks.workunit.client.0.smithi136.stdout: "scrub_reservations_completed": 0, 2024-09-15T22:16:40.555 INFO:tasks.workunit.client.0.smithi136.stdout: "successful_reservations_elapsed": { 2024-09-15T22:16:40.555 INFO:tasks.workunit.client.0.smithi136.stdout: "avgcount": 0, 2024-09-15T22:16:40.555 INFO:tasks.workunit.client.0.smithi136.stdout: "sum": 0E-9, 2024-09-15T22:16:40.555 INFO:tasks.workunit.client.0.smithi136.stdout: "avgtime": 0E-9 2024-09-15T22:16:40.555 INFO:tasks.workunit.client.0.smithi136.stdout: }, 2024-09-15T22:16:40.555 INFO:tasks.workunit.client.0.smithi136.stdout: "reservation_process_aborted": 0, 2024-09-15T22:16:40.555 INFO:tasks.workunit.client.0.smithi136.stdout: "reservation_process_failure": 0, 2024-09-15T22:16:40.555 INFO:tasks.workunit.client.0.smithi136.stdout: "reservation_process_skipped": 0, 2024-09-15T22:16:40.555 INFO:tasks.workunit.client.0.smithi136.stdout: "failed_reservations_elapsed": { 2024-09-15T22:16:40.556 INFO:tasks.workunit.client.0.smithi136.stdout: "avgcount": 0, 2024-09-15T22:16:40.556 INFO:tasks.workunit.client.0.smithi136.stdout: "sum": 0E-9, 2024-09-15T22:16:40.556 INFO:tasks.workunit.client.0.smithi136.stdout: "avgtime": 0E-9 2024-09-15T22:16:40.556 INFO:tasks.workunit.client.0.smithi136.stdout: }, 2024-09-15T22:16:40.556 INFO:tasks.workunit.client.0.smithi136.stdout: "replicas_in_reservation": 0 2024-09-15T22:16:40.556 INFO:tasks.workunit.client.0.smithi136.stdout: } 2024-09-15T22:16:40.556 INFO:tasks.workunit.client.0.smithi136.stdout: } 2024-09-15T22:16:40.556 INFO:tasks.workunit.client.0.smithi136.stdout: ], 2024-09-15T22:16:40.556 INFO:tasks.workunit.client.0.smithi136.stdout: "osd_scrub_sh_repl": [ 2024-09-15T22:16:40.556 INFO:tasks.workunit.client.0.smithi136.stdout: { 2024-09-15T22:16:40.556 INFO:tasks.workunit.client.0.smithi136.stdout: "labels": { 2024-09-15T22:16:40.557 INFO:tasks.workunit.client.0.smithi136.stdout: "level": "shallow", 2024-09-15T22:16:40.557 INFO:tasks.workunit.client.0.smithi136.stdout: "pooltype": "replicated" 2024-09-15T22:16:40.557 INFO:tasks.workunit.client.0.smithi136.stdout: }, 2024-09-15T22:16:40.557 INFO:tasks.workunit.client.0.smithi136.stdout: "counters": { 2024-09-15T22:16:40.557 INFO:tasks.workunit.client.0.smithi136.stdout: "num_scrubs_started": 0, 2024-09-15T22:16:40.557 INFO:tasks.workunit.client.0.smithi136.stdout: "num_scrubs_past_reservation": 0, 2024-09-15T22:16:40.557 INFO:tasks.workunit.client.0.smithi136.stdout: "successful_scrubs": 0, 2024-09-15T22:16:40.557 INFO:tasks.workunit.client.0.smithi136.stdout: "successful_scrubs_elapsed": { 2024-09-15T22:16:40.557 INFO:tasks.workunit.client.0.smithi136.stdout: "avgcount": 0, 2024-09-15T22:16:40.557 INFO:tasks.workunit.client.0.smithi136.stdout: "sum": 0E-9, 2024-09-15T22:16:40.557 INFO:tasks.workunit.client.0.smithi136.stdout: "avgtime": 0E-9 2024-09-15T22:16:40.557 INFO:tasks.workunit.client.0.smithi136.stdout: }, 2024-09-15T22:16:40.557 INFO:tasks.workunit.client.0.smithi136.stdout: "failed_scrubs": 0, 2024-09-15T22:16:40.557 INFO:tasks.workunit.client.0.smithi136.stdout: "failed_scrubs_elapsed": { 2024-09-15T22:16:40.558 INFO:tasks.workunit.client.0.smithi136.stdout: "avgcount": 0, 2024-09-15T22:16:40.558 INFO:tasks.workunit.client.0.smithi136.stdout: "sum": 0E-9, 2024-09-15T22:16:40.558 INFO:tasks.workunit.client.0.smithi136.stdout: "avgtime": 0E-9 2024-09-15T22:16:40.558 INFO:tasks.workunit.client.0.smithi136.stdout: }, 2024-09-15T22:16:40.558 INFO:tasks.workunit.client.0.smithi136.stdout: "preemptions": 0, 2024-09-15T22:16:40.558 INFO:tasks.workunit.client.0.smithi136.stdout: "chunk_selected": 0, 2024-09-15T22:16:40.558 INFO:tasks.workunit.client.0.smithi136.stdout: "chunk_busy": 0, 2024-09-15T22:16:40.558 INFO:tasks.workunit.client.0.smithi136.stdout: "locked_object": 0, 2024-09-15T22:16:40.559 INFO:tasks.workunit.client.0.smithi136.stdout: "write_blocked_by_scrub": 0, 2024-09-15T22:16:40.559 INFO:tasks.workunit.client.0.smithi136.stdout: "scrub_reservations_completed": 0, 2024-09-15T22:16:40.559 INFO:tasks.workunit.client.0.smithi136.stdout: "successful_reservations_elapsed": { 2024-09-15T22:16:40.559 INFO:tasks.workunit.client.0.smithi136.stdout: "avgcount": 0, 2024-09-15T22:16:40.559 INFO:tasks.workunit.client.0.smithi136.stdout: "sum": 0E-9, 2024-09-15T22:16:40.559 INFO:tasks.workunit.client.0.smithi136.stdout: "avgtime": 0E-9 2024-09-15T22:16:40.559 INFO:tasks.workunit.client.0.smithi136.stdout: }, 2024-09-15T22:16:40.559 INFO:tasks.workunit.client.0.smithi136.stdout: "reservation_process_aborted": 0, 2024-09-15T22:16:40.559 INFO:tasks.workunit.client.0.smithi136.stdout: "reservation_process_failure": 0, 2024-09-15T22:16:40.559 INFO:tasks.workunit.client.0.smithi136.stdout: "reservation_process_skipped": 0, 2024-09-15T22:16:40.559 INFO:tasks.workunit.client.0.smithi136.stdout: "failed_reservations_elapsed": { 2024-09-15T22:16:40.560 INFO:tasks.workunit.client.0.smithi136.stdout: "avgcount": 0, 2024-09-15T22:16:40.560 INFO:tasks.workunit.client.0.smithi136.stdout: "sum": 0E-9, 2024-09-15T22:16:40.560 INFO:tasks.workunit.client.0.smithi136.stdout: "avgtime": 0E-9 2024-09-15T22:16:40.560 INFO:tasks.workunit.client.0.smithi136.stdout: }, 2024-09-15T22:16:40.560 INFO:tasks.workunit.client.0.smithi136.stdout: "replicas_in_reservation": 0 2024-09-15T22:16:40.560 INFO:tasks.workunit.client.0.smithi136.stdout: } 2024-09-15T22:16:40.560 INFO:tasks.workunit.client.0.smithi136.stdout: } 2024-09-15T22:16:40.560 INFO:tasks.workunit.client.0.smithi136.stdout: ] 2024-09-15T22:16:40.560 INFO:tasks.workunit.client.0.smithi136.stdout:} 2024-09-15T22:16:40.561 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:45: perf_counters: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-09-15T22:16:40.561 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:47: perf_counters: ceph tell osd.2 counter dump 2024-09-15T22:16:40.561 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:47: perf_counters: jq 'with_entries(select(.key | startswith("osd_scrub")))' 2024-09-15T22:16:40.647 INFO:tasks.workunit.client.0.smithi136.stdout:{ 2024-09-15T22:16:40.648 INFO:tasks.workunit.client.0.smithi136.stdout: "osd_scrub_dp_ec": [ 2024-09-15T22:16:40.648 INFO:tasks.workunit.client.0.smithi136.stdout: { 2024-09-15T22:16:40.648 INFO:tasks.workunit.client.0.smithi136.stdout: "labels": { 2024-09-15T22:16:40.648 INFO:tasks.workunit.client.0.smithi136.stdout: "level": "deep", 2024-09-15T22:16:40.648 INFO:tasks.workunit.client.0.smithi136.stdout: "pooltype": "ec" 2024-09-15T22:16:40.648 INFO:tasks.workunit.client.0.smithi136.stdout: }, 2024-09-15T22:16:40.648 INFO:tasks.workunit.client.0.smithi136.stdout: "counters": { 2024-09-15T22:16:40.648 INFO:tasks.workunit.client.0.smithi136.stdout: "num_scrubs_started": 0, 2024-09-15T22:16:40.648 INFO:tasks.workunit.client.0.smithi136.stdout: "num_scrubs_past_reservation": 0, 2024-09-15T22:16:40.648 INFO:tasks.workunit.client.0.smithi136.stdout: "successful_scrubs": 0, 2024-09-15T22:16:40.648 INFO:tasks.workunit.client.0.smithi136.stdout: "successful_scrubs_elapsed": { 2024-09-15T22:16:40.648 INFO:tasks.workunit.client.0.smithi136.stdout: "avgcount": 0, 2024-09-15T22:16:40.648 INFO:tasks.workunit.client.0.smithi136.stdout: "sum": 0E-9, 2024-09-15T22:16:40.649 INFO:tasks.workunit.client.0.smithi136.stdout: "avgtime": 0E-9 2024-09-15T22:16:40.649 INFO:tasks.workunit.client.0.smithi136.stdout: }, 2024-09-15T22:16:40.649 INFO:tasks.workunit.client.0.smithi136.stdout: "failed_scrubs": 0, 2024-09-15T22:16:40.649 INFO:tasks.workunit.client.0.smithi136.stdout: "failed_scrubs_elapsed": { 2024-09-15T22:16:40.649 INFO:tasks.workunit.client.0.smithi136.stdout: "avgcount": 0, 2024-09-15T22:16:40.649 INFO:tasks.workunit.client.0.smithi136.stdout: "sum": 0E-9, 2024-09-15T22:16:40.649 INFO:tasks.workunit.client.0.smithi136.stdout: "avgtime": 0E-9 2024-09-15T22:16:40.649 INFO:tasks.workunit.client.0.smithi136.stdout: }, 2024-09-15T22:16:40.649 INFO:tasks.workunit.client.0.smithi136.stdout: "preemptions": 0, 2024-09-15T22:16:40.649 INFO:tasks.workunit.client.0.smithi136.stdout: "chunk_selected": 0, 2024-09-15T22:16:40.649 INFO:tasks.workunit.client.0.smithi136.stdout: "chunk_busy": 0, 2024-09-15T22:16:40.649 INFO:tasks.workunit.client.0.smithi136.stdout: "locked_object": 0, 2024-09-15T22:16:40.649 INFO:tasks.workunit.client.0.smithi136.stdout: "write_blocked_by_scrub": 0, 2024-09-15T22:16:40.650 INFO:tasks.workunit.client.0.smithi136.stdout: "scrub_reservations_completed": 0, 2024-09-15T22:16:40.650 INFO:tasks.workunit.client.0.smithi136.stdout: "successful_reservations_elapsed": { 2024-09-15T22:16:40.650 INFO:tasks.workunit.client.0.smithi136.stdout: "avgcount": 0, 2024-09-15T22:16:40.650 INFO:tasks.workunit.client.0.smithi136.stdout: "sum": 0E-9, 2024-09-15T22:16:40.650 INFO:tasks.workunit.client.0.smithi136.stdout: "avgtime": 0E-9 2024-09-15T22:16:40.650 INFO:tasks.workunit.client.0.smithi136.stdout: }, 2024-09-15T22:16:40.650 INFO:tasks.workunit.client.0.smithi136.stdout: "reservation_process_aborted": 0, 2024-09-15T22:16:40.650 INFO:tasks.workunit.client.0.smithi136.stdout: "reservation_process_failure": 0, 2024-09-15T22:16:40.650 INFO:tasks.workunit.client.0.smithi136.stdout: "reservation_process_skipped": 0, 2024-09-15T22:16:40.650 INFO:tasks.workunit.client.0.smithi136.stdout: "failed_reservations_elapsed": { 2024-09-15T22:16:40.650 INFO:tasks.workunit.client.0.smithi136.stdout: "avgcount": 0, 2024-09-15T22:16:40.650 INFO:tasks.workunit.client.0.smithi136.stdout: "sum": 0E-9, 2024-09-15T22:16:40.650 INFO:tasks.workunit.client.0.smithi136.stdout: "avgtime": 0E-9 2024-09-15T22:16:40.651 INFO:tasks.workunit.client.0.smithi136.stdout: }, 2024-09-15T22:16:40.651 INFO:tasks.workunit.client.0.smithi136.stdout: "replicas_in_reservation": 0 2024-09-15T22:16:40.651 INFO:tasks.workunit.client.0.smithi136.stdout: } 2024-09-15T22:16:40.651 INFO:tasks.workunit.client.0.smithi136.stdout: } 2024-09-15T22:16:40.651 INFO:tasks.workunit.client.0.smithi136.stdout: ], 2024-09-15T22:16:40.651 INFO:tasks.workunit.client.0.smithi136.stdout: "osd_scrub_dp_repl": [ 2024-09-15T22:16:40.651 INFO:tasks.workunit.client.0.smithi136.stdout: { 2024-09-15T22:16:40.651 INFO:tasks.workunit.client.0.smithi136.stdout: "labels": { 2024-09-15T22:16:40.651 INFO:tasks.workunit.client.0.smithi136.stdout: "level": "deep", 2024-09-15T22:16:40.651 INFO:tasks.workunit.client.0.smithi136.stdout: "pooltype": "replicated" 2024-09-15T22:16:40.651 INFO:tasks.workunit.client.0.smithi136.stdout: }, 2024-09-15T22:16:40.651 INFO:tasks.workunit.client.0.smithi136.stdout: "counters": { 2024-09-15T22:16:40.651 INFO:tasks.workunit.client.0.smithi136.stdout: "num_scrubs_started": 0, 2024-09-15T22:16:40.651 INFO:tasks.workunit.client.0.smithi136.stdout: "num_scrubs_past_reservation": 0, 2024-09-15T22:16:40.651 INFO:tasks.workunit.client.0.smithi136.stdout: "successful_scrubs": 0, 2024-09-15T22:16:40.652 INFO:tasks.workunit.client.0.smithi136.stdout: "successful_scrubs_elapsed": { 2024-09-15T22:16:40.652 INFO:tasks.workunit.client.0.smithi136.stdout: "avgcount": 0, 2024-09-15T22:16:40.652 INFO:tasks.workunit.client.0.smithi136.stdout: "sum": 0E-9, 2024-09-15T22:16:40.652 INFO:tasks.workunit.client.0.smithi136.stdout: "avgtime": 0E-9 2024-09-15T22:16:40.652 INFO:tasks.workunit.client.0.smithi136.stdout: }, 2024-09-15T22:16:40.652 INFO:tasks.workunit.client.0.smithi136.stdout: "failed_scrubs": 0, 2024-09-15T22:16:40.652 INFO:tasks.workunit.client.0.smithi136.stdout: "failed_scrubs_elapsed": { 2024-09-15T22:16:40.652 INFO:tasks.workunit.client.0.smithi136.stdout: "avgcount": 0, 2024-09-15T22:16:40.652 INFO:tasks.workunit.client.0.smithi136.stdout: "sum": 0E-9, 2024-09-15T22:16:40.652 INFO:tasks.workunit.client.0.smithi136.stdout: "avgtime": 0E-9 2024-09-15T22:16:40.652 INFO:tasks.workunit.client.0.smithi136.stdout: }, 2024-09-15T22:16:40.652 INFO:tasks.workunit.client.0.smithi136.stdout: "preemptions": 0, 2024-09-15T22:16:40.653 INFO:tasks.workunit.client.0.smithi136.stdout: "chunk_selected": 0, 2024-09-15T22:16:40.653 INFO:tasks.workunit.client.0.smithi136.stdout: "chunk_busy": 0, 2024-09-15T22:16:40.653 INFO:tasks.workunit.client.0.smithi136.stdout: "locked_object": 0, 2024-09-15T22:16:40.653 INFO:tasks.workunit.client.0.smithi136.stdout: "write_blocked_by_scrub": 0, 2024-09-15T22:16:40.653 INFO:tasks.workunit.client.0.smithi136.stdout: "scrub_reservations_completed": 0, 2024-09-15T22:16:40.653 INFO:tasks.workunit.client.0.smithi136.stdout: "successful_reservations_elapsed": { 2024-09-15T22:16:40.653 INFO:tasks.workunit.client.0.smithi136.stdout: "avgcount": 0, 2024-09-15T22:16:40.653 INFO:tasks.workunit.client.0.smithi136.stdout: "sum": 0E-9, 2024-09-15T22:16:40.653 INFO:tasks.workunit.client.0.smithi136.stdout: "avgtime": 0E-9 2024-09-15T22:16:40.653 INFO:tasks.workunit.client.0.smithi136.stdout: }, 2024-09-15T22:16:40.653 INFO:tasks.workunit.client.0.smithi136.stdout: "reservation_process_aborted": 0, 2024-09-15T22:16:40.653 INFO:tasks.workunit.client.0.smithi136.stdout: "reservation_process_failure": 0, 2024-09-15T22:16:40.653 INFO:tasks.workunit.client.0.smithi136.stdout: "reservation_process_skipped": 0, 2024-09-15T22:16:40.653 INFO:tasks.workunit.client.0.smithi136.stdout: "failed_reservations_elapsed": { 2024-09-15T22:16:40.654 INFO:tasks.workunit.client.0.smithi136.stdout: "avgcount": 0, 2024-09-15T22:16:40.654 INFO:tasks.workunit.client.0.smithi136.stdout: "sum": 0E-9, 2024-09-15T22:16:40.654 INFO:tasks.workunit.client.0.smithi136.stdout: "avgtime": 0E-9 2024-09-15T22:16:40.654 INFO:tasks.workunit.client.0.smithi136.stdout: }, 2024-09-15T22:16:40.654 INFO:tasks.workunit.client.0.smithi136.stdout: "replicas_in_reservation": 0 2024-09-15T22:16:40.654 INFO:tasks.workunit.client.0.smithi136.stdout: } 2024-09-15T22:16:40.654 INFO:tasks.workunit.client.0.smithi136.stdout: } 2024-09-15T22:16:40.654 INFO:tasks.workunit.client.0.smithi136.stdout: ], 2024-09-15T22:16:40.654 INFO:tasks.workunit.client.0.smithi136.stdout: "osd_scrub_sh_ec": [ 2024-09-15T22:16:40.654 INFO:tasks.workunit.client.0.smithi136.stdout: { 2024-09-15T22:16:40.654 INFO:tasks.workunit.client.0.smithi136.stdout: "labels": { 2024-09-15T22:16:40.654 INFO:tasks.workunit.client.0.smithi136.stdout: "level": "shallow", 2024-09-15T22:16:40.654 INFO:tasks.workunit.client.0.smithi136.stdout: "pooltype": "ec" 2024-09-15T22:16:40.654 INFO:tasks.workunit.client.0.smithi136.stdout: }, 2024-09-15T22:16:40.654 INFO:tasks.workunit.client.0.smithi136.stdout: "counters": { 2024-09-15T22:16:40.655 INFO:tasks.workunit.client.0.smithi136.stdout: "num_scrubs_started": 0, 2024-09-15T22:16:40.655 INFO:tasks.workunit.client.0.smithi136.stdout: "num_scrubs_past_reservation": 0, 2024-09-15T22:16:40.655 INFO:tasks.workunit.client.0.smithi136.stdout: "successful_scrubs": 0, 2024-09-15T22:16:40.655 INFO:tasks.workunit.client.0.smithi136.stdout: "successful_scrubs_elapsed": { 2024-09-15T22:16:40.655 INFO:tasks.workunit.client.0.smithi136.stdout: "avgcount": 0, 2024-09-15T22:16:40.655 INFO:tasks.workunit.client.0.smithi136.stdout: "sum": 0E-9, 2024-09-15T22:16:40.655 INFO:tasks.workunit.client.0.smithi136.stdout: "avgtime": 0E-9 2024-09-15T22:16:40.655 INFO:tasks.workunit.client.0.smithi136.stdout: }, 2024-09-15T22:16:40.655 INFO:tasks.workunit.client.0.smithi136.stdout: "failed_scrubs": 0, 2024-09-15T22:16:40.655 INFO:tasks.workunit.client.0.smithi136.stdout: "failed_scrubs_elapsed": { 2024-09-15T22:16:40.656 INFO:tasks.workunit.client.0.smithi136.stdout: "avgcount": 0, 2024-09-15T22:16:40.656 INFO:tasks.workunit.client.0.smithi136.stdout: "sum": 0E-9, 2024-09-15T22:16:40.656 INFO:tasks.workunit.client.0.smithi136.stdout: "avgtime": 0E-9 2024-09-15T22:16:40.656 INFO:tasks.workunit.client.0.smithi136.stdout: }, 2024-09-15T22:16:40.656 INFO:tasks.workunit.client.0.smithi136.stdout: "preemptions": 0, 2024-09-15T22:16:40.656 INFO:tasks.workunit.client.0.smithi136.stdout: "chunk_selected": 0, 2024-09-15T22:16:40.656 INFO:tasks.workunit.client.0.smithi136.stdout: "chunk_busy": 0, 2024-09-15T22:16:40.656 INFO:tasks.workunit.client.0.smithi136.stdout: "locked_object": 0, 2024-09-15T22:16:40.656 INFO:tasks.workunit.client.0.smithi136.stdout: "write_blocked_by_scrub": 0, 2024-09-15T22:16:40.656 INFO:tasks.workunit.client.0.smithi136.stdout: "scrub_reservations_completed": 0, 2024-09-15T22:16:40.656 INFO:tasks.workunit.client.0.smithi136.stdout: "successful_reservations_elapsed": { 2024-09-15T22:16:40.657 INFO:tasks.workunit.client.0.smithi136.stdout: "avgcount": 0, 2024-09-15T22:16:40.657 INFO:tasks.workunit.client.0.smithi136.stdout: "sum": 0E-9, 2024-09-15T22:16:40.657 INFO:tasks.workunit.client.0.smithi136.stdout: "avgtime": 0E-9 2024-09-15T22:16:40.657 INFO:tasks.workunit.client.0.smithi136.stdout: }, 2024-09-15T22:16:40.657 INFO:tasks.workunit.client.0.smithi136.stdout: "reservation_process_aborted": 0, 2024-09-15T22:16:40.657 INFO:tasks.workunit.client.0.smithi136.stdout: "reservation_process_failure": 0, 2024-09-15T22:16:40.657 INFO:tasks.workunit.client.0.smithi136.stdout: "reservation_process_skipped": 0, 2024-09-15T22:16:40.657 INFO:tasks.workunit.client.0.smithi136.stdout: "failed_reservations_elapsed": { 2024-09-15T22:16:40.657 INFO:tasks.workunit.client.0.smithi136.stdout: "avgcount": 0, 2024-09-15T22:16:40.657 INFO:tasks.workunit.client.0.smithi136.stdout: "sum": 0E-9, 2024-09-15T22:16:40.657 INFO:tasks.workunit.client.0.smithi136.stdout: "avgtime": 0E-9 2024-09-15T22:16:40.658 INFO:tasks.workunit.client.0.smithi136.stdout: }, 2024-09-15T22:16:40.658 INFO:tasks.workunit.client.0.smithi136.stdout: "replicas_in_reservation": 0 2024-09-15T22:16:40.658 INFO:tasks.workunit.client.0.smithi136.stdout: } 2024-09-15T22:16:40.658 INFO:tasks.workunit.client.0.smithi136.stdout: } 2024-09-15T22:16:40.658 INFO:tasks.workunit.client.0.smithi136.stdout: ], 2024-09-15T22:16:40.658 INFO:tasks.workunit.client.0.smithi136.stdout: "osd_scrub_sh_repl": [ 2024-09-15T22:16:40.658 INFO:tasks.workunit.client.0.smithi136.stdout: { 2024-09-15T22:16:40.658 INFO:tasks.workunit.client.0.smithi136.stdout: "labels": { 2024-09-15T22:16:40.658 INFO:tasks.workunit.client.0.smithi136.stdout: "level": "shallow", 2024-09-15T22:16:40.658 INFO:tasks.workunit.client.0.smithi136.stdout: "pooltype": "replicated" 2024-09-15T22:16:40.658 INFO:tasks.workunit.client.0.smithi136.stdout: }, 2024-09-15T22:16:40.658 INFO:tasks.workunit.client.0.smithi136.stdout: "counters": { 2024-09-15T22:16:40.658 INFO:tasks.workunit.client.0.smithi136.stdout: "num_scrubs_started": 0, 2024-09-15T22:16:40.658 INFO:tasks.workunit.client.0.smithi136.stdout: "num_scrubs_past_reservation": 0, 2024-09-15T22:16:40.659 INFO:tasks.workunit.client.0.smithi136.stdout: "successful_scrubs": 0, 2024-09-15T22:16:40.659 INFO:tasks.workunit.client.0.smithi136.stdout: "successful_scrubs_elapsed": { 2024-09-15T22:16:40.659 INFO:tasks.workunit.client.0.smithi136.stdout: "avgcount": 0, 2024-09-15T22:16:40.659 INFO:tasks.workunit.client.0.smithi136.stdout: "sum": 0E-9, 2024-09-15T22:16:40.659 INFO:tasks.workunit.client.0.smithi136.stdout: "avgtime": 0E-9 2024-09-15T22:16:40.659 INFO:tasks.workunit.client.0.smithi136.stdout: }, 2024-09-15T22:16:40.659 INFO:tasks.workunit.client.0.smithi136.stdout: "failed_scrubs": 0, 2024-09-15T22:16:40.659 INFO:tasks.workunit.client.0.smithi136.stdout: "failed_scrubs_elapsed": { 2024-09-15T22:16:40.659 INFO:tasks.workunit.client.0.smithi136.stdout: "avgcount": 0, 2024-09-15T22:16:40.659 INFO:tasks.workunit.client.0.smithi136.stdout: "sum": 0E-9, 2024-09-15T22:16:40.659 INFO:tasks.workunit.client.0.smithi136.stdout: "avgtime": 0E-9 2024-09-15T22:16:40.659 INFO:tasks.workunit.client.0.smithi136.stdout: }, 2024-09-15T22:16:40.659 INFO:tasks.workunit.client.0.smithi136.stdout: "preemptions": 0, 2024-09-15T22:16:40.659 INFO:tasks.workunit.client.0.smithi136.stdout: "chunk_selected": 0, 2024-09-15T22:16:40.660 INFO:tasks.workunit.client.0.smithi136.stdout: "chunk_busy": 0, 2024-09-15T22:16:40.660 INFO:tasks.workunit.client.0.smithi136.stdout: "locked_object": 0, 2024-09-15T22:16:40.660 INFO:tasks.workunit.client.0.smithi136.stdout: "write_blocked_by_scrub": 0, 2024-09-15T22:16:40.660 INFO:tasks.workunit.client.0.smithi136.stdout: "scrub_reservations_completed": 0, 2024-09-15T22:16:40.660 INFO:tasks.workunit.client.0.smithi136.stdout: "successful_reservations_elapsed": { 2024-09-15T22:16:40.660 INFO:tasks.workunit.client.0.smithi136.stdout: "avgcount": 0, 2024-09-15T22:16:40.660 INFO:tasks.workunit.client.0.smithi136.stdout: "sum": 0E-9, 2024-09-15T22:16:40.660 INFO:tasks.workunit.client.0.smithi136.stdout: "avgtime": 0E-9 2024-09-15T22:16:40.661 INFO:tasks.workunit.client.0.smithi136.stdout: }, 2024-09-15T22:16:40.661 INFO:tasks.workunit.client.0.smithi136.stdout: "reservation_process_aborted": 0, 2024-09-15T22:16:40.661 INFO:tasks.workunit.client.0.smithi136.stdout: "reservation_process_failure": 0, 2024-09-15T22:16:40.661 INFO:tasks.workunit.client.0.smithi136.stdout: "reservation_process_skipped": 0, 2024-09-15T22:16:40.661 INFO:tasks.workunit.client.0.smithi136.stdout: "failed_reservations_elapsed": { 2024-09-15T22:16:40.661 INFO:tasks.workunit.client.0.smithi136.stdout: "avgcount": 0, 2024-09-15T22:16:40.661 INFO:tasks.workunit.client.0.smithi136.stdout: "sum": 0E-9, 2024-09-15T22:16:40.661 INFO:tasks.workunit.client.0.smithi136.stdout: "avgtime": 0E-9 2024-09-15T22:16:40.661 INFO:tasks.workunit.client.0.smithi136.stdout: }, 2024-09-15T22:16:40.661 INFO:tasks.workunit.client.0.smithi136.stdout: "replicas_in_reservation": 0 2024-09-15T22:16:40.661 INFO:tasks.workunit.client.0.smithi136.stdout: } 2024-09-15T22:16:40.661 INFO:tasks.workunit.client.0.smithi136.stdout: } 2024-09-15T22:16:40.661 INFO:tasks.workunit.client.0.smithi136.stdout: ] 2024-09-15T22:16:40.661 INFO:tasks.workunit.client.0.smithi136.stdout:} 2024-09-15T22:16:40.662 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:36: run: echo '-------------- Teardown Test TEST_deep_scrub_abort ------------------' 2024-09-15T22:16:40.662 INFO:tasks.workunit.client.0.smithi136.stdout:-------------- Teardown Test TEST_deep_scrub_abort ------------------ 2024-09-15T22:16:40.663 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:37: run: teardown td/osd-scrub-test 2024-09-15T22:16:40.663 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/osd-scrub-test 2024-09-15T22:16:40.663 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-15T22:16:40.663 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/osd-scrub-test KILL 2024-09-15T22:16:40.663 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-15T22:16:40.663 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-15T22:16:40.663 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-15T22:16:40.663 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-15T22:16:40.663 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-15T22:16:40.763 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-15T22:16:40.764 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-15T22:16:40.764 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-15T22:16:40.765 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-15T22:16:40.765 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-15T22:16:40.766 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-15T22:16:40.766 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-15T22:16:40.766 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T22:16:40.767 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-15T22:16:40.767 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-15T22:16:40.767 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T22:16:40.767 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-15T22:16:40.769 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-15T22:16:40.769 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/osd-scrub-test 2024-09-15T22:16:40.825 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-15T22:16:40.825 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T22:16:40.826 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.519518 2024-09-15T22:16:40.826 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.519518 2024-09-15T22:16:40.826 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-15T22:16:40.826 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-15T22:16:40.827 INFO:tasks.workunit.client.0.smithi136.stdout:-------------- Complete Test TEST_deep_scrub_abort ------------------ 2024-09-15T22:16:40.827 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:38: run: echo '-------------- Complete Test TEST_deep_scrub_abort ------------------' 2024-09-15T22:16:40.827 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:31: run: for func in $funcs 2024-09-15T22:16:40.828 INFO:tasks.workunit.client.0.smithi136.stdout:-------------- Prepare Test TEST_dump_scrub_schedule ------------------- 2024-09-15T22:16:40.829 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:32: run: echo '-------------- Prepare Test TEST_dump_scrub_schedule -------------------' 2024-09-15T22:16:40.829 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:33: run: setup td/osd-scrub-test 2024-09-15T22:16:40.830 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/osd-scrub-test 2024-09-15T22:16:40.830 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/osd-scrub-test 2024-09-15T22:16:40.830 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/osd-scrub-test 2024-09-15T22:16:40.830 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-15T22:16:40.830 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/osd-scrub-test KILL 2024-09-15T22:16:40.830 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-15T22:16:40.830 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-15T22:16:40.830 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-15T22:16:40.830 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-15T22:16:40.830 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-15T22:16:40.831 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-15T22:16:40.831 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-15T22:16:40.831 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-15T22:16:40.831 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-15T22:16:40.832 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-15T22:16:40.832 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-15T22:16:40.832 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-15T22:16:40.832 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T22:16:40.832 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-15T22:16:40.832 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-15T22:16:40.832 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T22:16:40.833 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-15T22:16:40.833 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-15T22:16:40.834 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/osd-scrub-test 2024-09-15T22:16:40.835 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-15T22:16:40.835 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T22:16:40.835 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.519518 2024-09-15T22:16:40.835 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.519518 2024-09-15T22:16:40.836 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-15T22:16:40.836 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-15T22:16:40.836 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/osd-scrub-test 2024-09-15T22:16:40.837 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2024-09-15T22:16:40.837 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T22:16:40.837 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.519518 2024-09-15T22:16:40.837 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.519518 2024-09-15T22:16:40.839 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2024-09-15T22:16:40.839 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 4096 -le 1024 ']' 2024-09-15T22:16:40.839 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2024-09-15T22:16:40.839 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/osd-scrub-test 1' TERM HUP INT 2024-09-15T22:16:40.839 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:34: run: echo '-------------- Run Test TEST_dump_scrub_schedule -----------------------' 2024-09-15T22:16:40.839 INFO:tasks.workunit.client.0.smithi136.stdout:-------------- Run Test TEST_dump_scrub_schedule ----------------------- 2024-09-15T22:16:40.841 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:35: run: TEST_dump_scrub_schedule td/osd-scrub-test 2024-09-15T22:16:40.841 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:528: TEST_dump_scrub_schedule: local dir=td/osd-scrub-test 2024-09-15T22:16:40.841 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:529: TEST_dump_scrub_schedule: local poolname=test 2024-09-15T22:16:40.841 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:530: TEST_dump_scrub_schedule: local OSDS=3 2024-09-15T22:16:40.841 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:531: TEST_dump_scrub_schedule: local objects=15 2024-09-15T22:16:40.841 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:533: TEST_dump_scrub_schedule: TESTDATA=testdata.519518 2024-09-15T22:16:40.841 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:535: TEST_dump_scrub_schedule: run_mon td/osd-scrub-test a --osd_pool_default_size=3 2024-09-15T22:16:40.841 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/osd-scrub-test 2024-09-15T22:16:40.842 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2024-09-15T22:16:40.842 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2024-09-15T22:16:40.842 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2024-09-15T22:16:40.842 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/osd-scrub-test/a 2024-09-15T22:16:40.842 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-scrub-test/a --run-dir=td/osd-scrub-test --osd_pool_default_size=3 2024-09-15T22:16:41.313 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2024-09-15T22:16:41.313 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-15T22:16:41.313 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-15T22:16:41.313 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-15T22:16:41.313 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T22:16:41.313 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.519518 2024-09-15T22:16:41.314 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.519518/$cluster-$name.asok' 2024-09-15T22:16:41.314 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-scrub-test/a '--log-file=td/osd-scrub-test/$name.log' '--admin-socket=/tmp/ceph-asok.519518/$cluster-$name.asok' --mon-cluster-log-file=td/osd-scrub-test/log --run-dir=td/osd-scrub-test '--pid-file=td/osd-scrub-test/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=3 2024-09-15T22:16:41.493 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2024-09-15T22:16:41.495 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2024-09-15T22:16:41.495 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-15T22:16:41.495 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-15T22:16:41.495 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2024-09-15T22:16:41.495 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2024-09-15T22:16:41.495 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-15T22:16:41.495 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-15T22:16:41.495 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-15T22:16:41.495 INFO:tasks.workunit.client.0.smithi136.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-15T22:16:41.495 INFO:tasks.workunit.client.0.smithi136.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T22:16:41.495 INFO:tasks.workunit.client.0.smithi136.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.519518 2024-09-15T22:16:41.495 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.519518/ceph-mon.a.asok 2024-09-15T22:16:41.496 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-15T22:16:41.496 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.519518/ceph-mon.a.asok config get fsid 2024-09-15T22:16:41.573 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2024-09-15T22:16:41.928 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-15T22:16:41.928 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-15T22:16:41.928 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2024-09-15T22:16:41.928 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2024-09-15T22:16:41.929 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-15T22:16:41.929 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-15T22:16:41.930 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-15T22:16:41.930 INFO:tasks.workunit.client.0.smithi136.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-15T22:16:41.930 INFO:tasks.workunit.client.0.smithi136.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T22:16:41.930 INFO:tasks.workunit.client.0.smithi136.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.519518 2024-09-15T22:16:41.930 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.519518/ceph-mon.a.asok 2024-09-15T22:16:41.930 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-15T22:16:41.930 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.519518/ceph-mon.a.asok config get mon_host 2024-09-15T22:16:41.930 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:536: TEST_dump_scrub_schedule: run_mgr td/osd-scrub-test x 2024-09-15T22:16:41.930 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: run_mgr: local dir=td/osd-scrub-test 2024-09-15T22:16:41.930 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: run_mgr: shift 2024-09-15T22:16:41.930 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: run_mgr: local id=x 2024-09-15T22:16:41.930 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: run_mgr: shift 2024-09-15T22:16:41.931 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:548: run_mgr: local data=td/osd-scrub-test/x 2024-09-15T22:16:41.931 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:550: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-15T22:16:42.453 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: get_asok_path 2024-09-15T22:16:42.453 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-15T22:16:42.453 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-15T22:16:42.453 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-15T22:16:42.453 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T22:16:42.454 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.519518 2024-09-15T22:16:42.454 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.519518/$cluster-$name.asok' 2024-09-15T22:16:42.454 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-15T22:16:42.454 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-scrub-test/x '--log-file=td/osd-scrub-test/$name.log' '--admin-socket=/tmp/ceph-asok.519518/$cluster-$name.asok' --run-dir=td/osd-scrub-test '--pid-file=td/osd-scrub-test/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-15T22:16:42.480 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:547: TEST_dump_scrub_schedule: local 'ceph_osd_args=--osd_deep_scrub_randomize_ratio=0 --osd_scrub_interval_randomize_ratio=0 --osd_scrub_backoff_ratio=0.0 --osd_op_queue=wpq --osd_stats_update_period_not_scrubbing=1 --osd_stats_update_period_scrubbing=1 --osd_scrub_sleep=0.2' 2024-09-15T22:16:42.481 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:549: TEST_dump_scrub_schedule: expr 3 - 1 2024-09-15T22:16:42.482 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:549: TEST_dump_scrub_schedule: seq 0 2 2024-09-15T22:16:42.485 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:549: TEST_dump_scrub_schedule: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-09-15T22:16:42.485 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:551: TEST_dump_scrub_schedule: run_osd td/osd-scrub-test 0 --osd_deep_scrub_randomize_ratio=0 --osd_scrub_interval_randomize_ratio=0 --osd_scrub_backoff_ratio=0.0 --osd_op_queue=wpq --osd_stats_update_period_not_scrubbing=1 --osd_stats_update_period_scrubbing=1 --osd_scrub_sleep=0.2 2024-09-15T22:16:42.486 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-scrub-test 2024-09-15T22:16:42.486 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-15T22:16:42.486 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=0 2024-09-15T22:16:42.486 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-15T22:16:42.486 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-scrub-test/0 2024-09-15T22:16:42.486 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=79d14839-1c9a-4907-b0ca-211bfb85130f --auth-supported=none --mon-host=127.0.0.1:7138 ' 2024-09-15T22:16:42.486 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T22:16:42.486 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T22:16:42.486 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T22:16:42.486 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-scrub-test/0' 2024-09-15T22:16:42.486 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-test/0/journal' 2024-09-15T22:16:42.486 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-15T22:16:42.486 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-15T22:16:42.486 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-scrub-test' 2024-09-15T22:16:42.487 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-15T22:16:42.487 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-15T22:16:42.487 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-15T22:16:42.487 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-15T22:16:42.487 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T22:16:42.487 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.519518 2024-09-15T22:16:42.487 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.519518/$cluster-$name.asok' 2024-09-15T22:16:42.487 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.519518/$cluster-$name.asok' 2024-09-15T22:16:42.487 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-15T22:16:42.487 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-15T22:16:42.487 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-15T22:16:42.487 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-scrub-test/$name.log' 2024-09-15T22:16:42.487 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-scrub-test/$name.pid' 2024-09-15T22:16:42.487 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T22:16:42.488 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T22:16:42.488 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T22:16:42.488 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T22:16:42.488 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-15T22:16:42.488 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+='--osd_deep_scrub_randomize_ratio=0 --osd_scrub_interval_randomize_ratio=0 --osd_scrub_backoff_ratio=0.0 --osd_op_queue=wpq --osd_stats_update_period_not_scrubbing=1 --osd_stats_update_period_scrubbing=1 --osd_scrub_sleep=0.2' 2024-09-15T22:16:42.488 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-scrub-test/0 2024-09-15T22:16:42.488 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-15T22:16:42.488 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=45d5cbbf-a9f0-4372-a41a-7edcf757e793 2024-09-15T22:16:42.489 INFO:tasks.workunit.client.0.smithi136.stdout:add osd0 45d5cbbf-a9f0-4372-a41a-7edcf757e793 2024-09-15T22:16:42.489 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd0 45d5cbbf-a9f0-4372-a41a-7edcf757e793' 2024-09-15T22:16:42.489 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-15T22:16:42.503 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQDKXOdmgD3rHRAAkX/PwPDOP3Y7xwliIO8dxA== 2024-09-15T22:16:42.504 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQDKXOdmgD3rHRAAkX/PwPDOP3Y7xwliIO8dxA=="}' 2024-09-15T22:16:42.504 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 45d5cbbf-a9f0-4372-a41a-7edcf757e793 -i td/osd-scrub-test/0/new.json 2024-09-15T22:16:42.743 INFO:tasks.workunit.client.0.smithi136.stdout:0 2024-09-15T22:16:42.753 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-scrub-test/0/new.json 2024-09-15T22:16:42.754 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 0 --fsid=79d14839-1c9a-4907-b0ca-211bfb85130f --auth-supported=none --mon-host=127.0.0.1:7138 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-test/0 --osd-journal=td/osd-scrub-test/0/journal --chdir= --run-dir=td/osd-scrub-test '--admin-socket=/tmp/ceph-asok.519518/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-test/$name.log' '--pid-file=td/osd-scrub-test/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_deep_scrub_randomize_ratio=0 --osd_scrub_interval_randomize_ratio=0 --osd_scrub_backoff_ratio=0.0 --osd_op_queue=wpq --osd_stats_update_period_not_scrubbing=1 --osd_stats_update_period_scrubbing=1 --osd_scrub_sleep=0.2 --mkfs --key AQDKXOdmgD3rHRAAkX/PwPDOP3Y7xwliIO8dxA== --osd-uuid 45d5cbbf-a9f0-4372-a41a-7edcf757e793 2024-09-15T22:16:42.785 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T22:16:42.784+0000 7fe5268f1640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T22:16:42.794 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T22:16:42.792+0000 7fe5268f1640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T22:16:42.795 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T22:16:42.793+0000 7fe5268f1640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T22:16:42.795 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T22:16:42.794+0000 7fe5268f1640 -1 bdev(0x56159039fc00 td/osd-scrub-test/0/block) open stat got: (1) Operation not permitted 2024-09-15T22:16:42.795 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T22:16:42.794+0000 7fe5268f1640 -1 bluestore(td/osd-scrub-test/0) _read_fsid unparsable uuid 2024-09-15T22:16:48.551 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-scrub-test/0/keyring 2024-09-15T22:16:48.551 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-15T22:16:48.552 INFO:tasks.workunit.client.0.smithi136.stdout:adding osd0 key to auth repository 2024-09-15T22:16:48.552 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd0 key to auth repository 2024-09-15T22:16:48.552 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-scrub-test/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-15T22:16:48.913 INFO:tasks.workunit.client.0.smithi136.stdout:start osd.0 2024-09-15T22:16:48.914 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.0 2024-09-15T22:16:48.914 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 0 --fsid=79d14839-1c9a-4907-b0ca-211bfb85130f --auth-supported=none --mon-host=127.0.0.1:7138 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-test/0 --osd-journal=td/osd-scrub-test/0/journal --chdir= --run-dir=td/osd-scrub-test '--admin-socket=/tmp/ceph-asok.519518/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-test/$name.log' '--pid-file=td/osd-scrub-test/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_deep_scrub_randomize_ratio=0 --osd_scrub_interval_randomize_ratio=0 --osd_scrub_backoff_ratio=0.0 --osd_op_queue=wpq --osd_stats_update_period_not_scrubbing=1 --osd_stats_update_period_scrubbing=1 --osd_scrub_sleep=0.2 2024-09-15T22:16:48.914 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-15T22:16:48.914 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-15T22:16:48.915 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-15T22:16:48.944 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T22:16:48.942+0000 7f620c8b2640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T22:16:48.946 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T22:16:48.944+0000 7f620c8b2640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T22:16:48.947 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T22:16:48.945+0000 7f620c8b2640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T22:16:49.187 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 0 2024-09-15T22:16:49.187 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-15T22:16:49.187 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2024-09-15T22:16:49.187 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-15T22:16:49.187 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-15T22:16:49.188 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 500 )) 2024-09-15T22:16:49.188 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-15T22:16:49.188 INFO:tasks.workunit.client.0.smithi136.stdout:0 2024-09-15T22:16:49.188 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:16:49.188 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T22:16:49.462 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:16:50.463 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:16:50.463 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 500 )) 2024-09-15T22:16:50.463 INFO:tasks.workunit.client.0.smithi136.stdout:1 2024-09-15T22:16:50.464 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-15T22:16:50.464 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:16:50.464 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T22:16:50.562 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T22:16:50.561+0000 7f620c8b2640 -1 Falling back to public interface 2024-09-15T22:16:50.740 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:16:51.741 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:16:51.741 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 500 )) 2024-09-15T22:16:51.742 INFO:tasks.workunit.client.0.smithi136.stdout:2 2024-09-15T22:16:51.742 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-15T22:16:51.742 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:16:51.742 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T22:16:52.027 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:16:53.028 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:16:53.029 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 500 )) 2024-09-15T22:16:53.029 INFO:tasks.workunit.client.0.smithi136.stdout:3 2024-09-15T22:16:53.029 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-15T22:16:53.029 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:16:53.029 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T22:16:53.083 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T22:16:53.081+0000 7f620c8b2640 -1 osd.0 0 log_to_monitors true 2024-09-15T22:16:53.349 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:16:54.351 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:16:54.351 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 500 )) 2024-09-15T22:16:54.351 INFO:tasks.workunit.client.0.smithi136.stdout:4 2024-09-15T22:16:54.352 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-15T22:16:54.352 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:16:54.352 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T22:16:54.653 INFO:tasks.workunit.client.0.smithi136.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/2157098434,v1:127.0.0.1:6803/2157098434] [v2:127.0.0.1:6804/2157098434,v1:127.0.0.1:6805/2157098434] exists,up 45d5cbbf-a9f0-4372-a41a-7edcf757e793 2024-09-15T22:16:54.657 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-15T22:16:54.657 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-15T22:16:54.657 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-15T22:16:54.657 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:549: TEST_dump_scrub_schedule: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-09-15T22:16:54.657 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:551: TEST_dump_scrub_schedule: run_osd td/osd-scrub-test 1 --osd_deep_scrub_randomize_ratio=0 --osd_scrub_interval_randomize_ratio=0 --osd_scrub_backoff_ratio=0.0 --osd_op_queue=wpq --osd_stats_update_period_not_scrubbing=1 --osd_stats_update_period_scrubbing=1 --osd_scrub_sleep=0.2 2024-09-15T22:16:54.657 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-scrub-test 2024-09-15T22:16:54.657 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-15T22:16:54.658 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=1 2024-09-15T22:16:54.658 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-15T22:16:54.658 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-scrub-test/1 2024-09-15T22:16:54.658 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=79d14839-1c9a-4907-b0ca-211bfb85130f --auth-supported=none --mon-host=127.0.0.1:7138 ' 2024-09-15T22:16:54.658 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T22:16:54.658 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T22:16:54.658 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T22:16:54.658 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-scrub-test/1' 2024-09-15T22:16:54.658 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-test/1/journal' 2024-09-15T22:16:54.658 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-15T22:16:54.658 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-15T22:16:54.658 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-scrub-test' 2024-09-15T22:16:54.658 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-15T22:16:54.659 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-15T22:16:54.659 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-15T22:16:54.659 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-15T22:16:54.659 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T22:16:54.659 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.519518 2024-09-15T22:16:54.659 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.519518/$cluster-$name.asok' 2024-09-15T22:16:54.659 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.519518/$cluster-$name.asok' 2024-09-15T22:16:54.659 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-15T22:16:54.659 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-15T22:16:54.659 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-15T22:16:54.659 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-scrub-test/$name.log' 2024-09-15T22:16:54.659 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-scrub-test/$name.pid' 2024-09-15T22:16:54.660 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T22:16:54.660 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T22:16:54.660 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T22:16:54.660 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T22:16:54.660 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-15T22:16:54.660 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+='--osd_deep_scrub_randomize_ratio=0 --osd_scrub_interval_randomize_ratio=0 --osd_scrub_backoff_ratio=0.0 --osd_op_queue=wpq --osd_stats_update_period_not_scrubbing=1 --osd_stats_update_period_scrubbing=1 --osd_scrub_sleep=0.2' 2024-09-15T22:16:54.660 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-scrub-test/1 2024-09-15T22:16:54.660 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-15T22:16:54.660 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=84195623-08cc-4978-a258-c2bce95a3867 2024-09-15T22:16:54.660 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd1 84195623-08cc-4978-a258-c2bce95a3867' 2024-09-15T22:16:54.660 INFO:tasks.workunit.client.0.smithi136.stdout:add osd1 84195623-08cc-4978-a258-c2bce95a3867 2024-09-15T22:16:54.661 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-15T22:16:54.674 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQDWXOdmkJ4QKBAAOM1w6AAASMqif2GI1ix2kA== 2024-09-15T22:16:54.674 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQDWXOdmkJ4QKBAAOM1w6AAASMqif2GI1ix2kA=="}' 2024-09-15T22:16:54.674 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 84195623-08cc-4978-a258-c2bce95a3867 -i td/osd-scrub-test/1/new.json 2024-09-15T22:16:55.002 INFO:tasks.workunit.client.0.smithi136.stdout:1 2024-09-15T22:16:55.013 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-scrub-test/1/new.json 2024-09-15T22:16:55.014 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 1 --fsid=79d14839-1c9a-4907-b0ca-211bfb85130f --auth-supported=none --mon-host=127.0.0.1:7138 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-test/1 --osd-journal=td/osd-scrub-test/1/journal --chdir= --run-dir=td/osd-scrub-test '--admin-socket=/tmp/ceph-asok.519518/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-test/$name.log' '--pid-file=td/osd-scrub-test/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_deep_scrub_randomize_ratio=0 --osd_scrub_interval_randomize_ratio=0 --osd_scrub_backoff_ratio=0.0 --osd_op_queue=wpq --osd_stats_update_period_not_scrubbing=1 --osd_stats_update_period_scrubbing=1 --osd_scrub_sleep=0.2 --mkfs --key AQDWXOdmkJ4QKBAAOM1w6AAASMqif2GI1ix2kA== --osd-uuid 84195623-08cc-4978-a258-c2bce95a3867 2024-09-15T22:16:55.046 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T22:16:55.045+0000 7f73887c5640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T22:16:55.078 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T22:16:55.077+0000 7f73887c5640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T22:16:55.079 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T22:16:55.078+0000 7f73887c5640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T22:16:55.079 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T22:16:55.078+0000 7f73887c5640 -1 bdev(0x556c915e7c00 td/osd-scrub-test/1/block) open stat got: (1) Operation not permitted 2024-09-15T22:16:55.080 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T22:16:55.078+0000 7f73887c5640 -1 bluestore(td/osd-scrub-test/1) _read_fsid unparsable uuid 2024-09-15T22:17:00.952 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-scrub-test/1/keyring 2024-09-15T22:17:00.952 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-15T22:17:00.953 INFO:tasks.workunit.client.0.smithi136.stdout:adding osd1 key to auth repository 2024-09-15T22:17:00.953 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd1 key to auth repository 2024-09-15T22:17:00.953 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-scrub-test/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-15T22:17:01.306 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.1 2024-09-15T22:17:01.307 INFO:tasks.workunit.client.0.smithi136.stdout:start osd.1 2024-09-15T22:17:01.307 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 1 --fsid=79d14839-1c9a-4907-b0ca-211bfb85130f --auth-supported=none --mon-host=127.0.0.1:7138 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-test/1 --osd-journal=td/osd-scrub-test/1/journal --chdir= --run-dir=td/osd-scrub-test '--admin-socket=/tmp/ceph-asok.519518/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-test/$name.log' '--pid-file=td/osd-scrub-test/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_deep_scrub_randomize_ratio=0 --osd_scrub_interval_randomize_ratio=0 --osd_scrub_backoff_ratio=0.0 --osd_op_queue=wpq --osd_stats_update_period_not_scrubbing=1 --osd_stats_update_period_scrubbing=1 --osd_scrub_sleep=0.2 2024-09-15T22:17:01.307 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-15T22:17:01.307 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-15T22:17:01.308 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-15T22:17:01.337 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T22:17:01.335+0000 7f2a78a0d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T22:17:01.340 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T22:17:01.337+0000 7f2a78a0d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T22:17:01.341 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T22:17:01.338+0000 7f2a78a0d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T22:17:01.609 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 1 2024-09-15T22:17:01.609 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-15T22:17:01.609 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2024-09-15T22:17:01.610 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-15T22:17:01.610 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-15T22:17:01.610 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 500 )) 2024-09-15T22:17:01.610 INFO:tasks.workunit.client.0.smithi136.stdout:0 2024-09-15T22:17:01.610 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-15T22:17:01.611 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:17:01.611 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T22:17:01.889 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:17:02.233 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T22:17:02.231+0000 7f2a78a0d640 -1 Falling back to public interface 2024-09-15T22:17:02.890 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:17:02.890 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 500 )) 2024-09-15T22:17:02.891 INFO:tasks.workunit.client.0.smithi136.stdout:1 2024-09-15T22:17:02.891 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-15T22:17:02.891 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:17:02.891 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T22:17:03.175 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:17:04.176 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:17:04.176 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 500 )) 2024-09-15T22:17:04.177 INFO:tasks.workunit.client.0.smithi136.stdout:2 2024-09-15T22:17:04.177 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-15T22:17:04.177 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:17:04.177 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T22:17:04.466 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:17:05.124 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T22:17:05.123+0000 7f2a78a0d640 -1 osd.1 0 log_to_monitors true 2024-09-15T22:17:05.467 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:17:05.467 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 500 )) 2024-09-15T22:17:05.467 INFO:tasks.workunit.client.0.smithi136.stdout:3 2024-09-15T22:17:05.468 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-15T22:17:05.468 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:17:05.468 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T22:17:05.758 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:17:06.759 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:17:06.759 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 500 )) 2024-09-15T22:17:06.760 INFO:tasks.workunit.client.0.smithi136.stdout:4 2024-09-15T22:17:06.760 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-15T22:17:06.760 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:17:06.761 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T22:17:07.043 INFO:tasks.workunit.client.0.smithi136.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/3598209714,v1:127.0.0.1:6811/3598209714] [v2:127.0.0.1:6812/3598209714,v1:127.0.0.1:6813/3598209714] exists,up 84195623-08cc-4978-a258-c2bce95a3867 2024-09-15T22:17:07.047 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-15T22:17:07.047 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-15T22:17:07.048 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-15T22:17:07.048 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:549: TEST_dump_scrub_schedule: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-09-15T22:17:07.048 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:551: TEST_dump_scrub_schedule: run_osd td/osd-scrub-test 2 --osd_deep_scrub_randomize_ratio=0 --osd_scrub_interval_randomize_ratio=0 --osd_scrub_backoff_ratio=0.0 --osd_op_queue=wpq --osd_stats_update_period_not_scrubbing=1 --osd_stats_update_period_scrubbing=1 --osd_scrub_sleep=0.2 2024-09-15T22:17:07.048 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-scrub-test 2024-09-15T22:17:07.048 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-15T22:17:07.048 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=2 2024-09-15T22:17:07.048 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-15T22:17:07.048 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-scrub-test/2 2024-09-15T22:17:07.048 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=79d14839-1c9a-4907-b0ca-211bfb85130f --auth-supported=none --mon-host=127.0.0.1:7138 ' 2024-09-15T22:17:07.048 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T22:17:07.048 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T22:17:07.048 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T22:17:07.049 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-scrub-test/2' 2024-09-15T22:17:07.049 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-test/2/journal' 2024-09-15T22:17:07.049 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-15T22:17:07.049 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-15T22:17:07.049 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-scrub-test' 2024-09-15T22:17:07.049 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-15T22:17:07.049 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-15T22:17:07.049 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-15T22:17:07.049 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-15T22:17:07.049 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T22:17:07.049 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.519518 2024-09-15T22:17:07.050 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.519518/$cluster-$name.asok' 2024-09-15T22:17:07.050 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.519518/$cluster-$name.asok' 2024-09-15T22:17:07.050 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-15T22:17:07.050 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-15T22:17:07.050 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-15T22:17:07.050 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-scrub-test/$name.log' 2024-09-15T22:17:07.050 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-scrub-test/$name.pid' 2024-09-15T22:17:07.050 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T22:17:07.050 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T22:17:07.050 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T22:17:07.050 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T22:17:07.050 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-15T22:17:07.050 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+='--osd_deep_scrub_randomize_ratio=0 --osd_scrub_interval_randomize_ratio=0 --osd_scrub_backoff_ratio=0.0 --osd_op_queue=wpq --osd_stats_update_period_not_scrubbing=1 --osd_stats_update_period_scrubbing=1 --osd_scrub_sleep=0.2' 2024-09-15T22:17:07.051 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-scrub-test/2 2024-09-15T22:17:07.051 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-15T22:17:07.051 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=d18724e3-c0e5-4909-aa38-4a6763399134 2024-09-15T22:17:07.051 INFO:tasks.workunit.client.0.smithi136.stdout:add osd2 d18724e3-c0e5-4909-aa38-4a6763399134 2024-09-15T22:17:07.051 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd2 d18724e3-c0e5-4909-aa38-4a6763399134' 2024-09-15T22:17:07.052 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-15T22:17:07.064 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQDjXOdmb8q3AxAAkg0pYeVVP675aRJmi5Ofsg== 2024-09-15T22:17:07.064 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQDjXOdmb8q3AxAAkg0pYeVVP675aRJmi5Ofsg=="}' 2024-09-15T22:17:07.064 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new d18724e3-c0e5-4909-aa38-4a6763399134 -i td/osd-scrub-test/2/new.json 2024-09-15T22:17:07.427 INFO:tasks.workunit.client.0.smithi136.stdout:2 2024-09-15T22:17:07.438 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-scrub-test/2/new.json 2024-09-15T22:17:07.439 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 2 --fsid=79d14839-1c9a-4907-b0ca-211bfb85130f --auth-supported=none --mon-host=127.0.0.1:7138 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-test/2 --osd-journal=td/osd-scrub-test/2/journal --chdir= --run-dir=td/osd-scrub-test '--admin-socket=/tmp/ceph-asok.519518/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-test/$name.log' '--pid-file=td/osd-scrub-test/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_deep_scrub_randomize_ratio=0 --osd_scrub_interval_randomize_ratio=0 --osd_scrub_backoff_ratio=0.0 --osd_op_queue=wpq --osd_stats_update_period_not_scrubbing=1 --osd_stats_update_period_scrubbing=1 --osd_scrub_sleep=0.2 --mkfs --key AQDjXOdmb8q3AxAAkg0pYeVVP675aRJmi5Ofsg== --osd-uuid d18724e3-c0e5-4909-aa38-4a6763399134 2024-09-15T22:17:07.472 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T22:17:07.470+0000 7f59b13bd640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T22:17:07.511 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T22:17:07.510+0000 7f59b13bd640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T22:17:07.512 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T22:17:07.511+0000 7f59b13bd640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T22:17:07.512 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T22:17:07.511+0000 7f59b13bd640 -1 bdev(0x56376af9dc00 td/osd-scrub-test/2/block) open stat got: (1) Operation not permitted 2024-09-15T22:17:07.513 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T22:17:07.511+0000 7f59b13bd640 -1 bluestore(td/osd-scrub-test/2) _read_fsid unparsable uuid 2024-09-15T22:17:13.965 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-scrub-test/2/keyring 2024-09-15T22:17:13.965 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-15T22:17:13.966 INFO:tasks.workunit.client.0.smithi136.stdout:adding osd2 key to auth repository 2024-09-15T22:17:13.966 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd2 key to auth repository 2024-09-15T22:17:13.966 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-scrub-test/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-15T22:17:14.330 INFO:tasks.workunit.client.0.smithi136.stdout:start osd.2 2024-09-15T22:17:14.331 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.2 2024-09-15T22:17:14.331 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 2 --fsid=79d14839-1c9a-4907-b0ca-211bfb85130f --auth-supported=none --mon-host=127.0.0.1:7138 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-test/2 --osd-journal=td/osd-scrub-test/2/journal --chdir= --run-dir=td/osd-scrub-test '--admin-socket=/tmp/ceph-asok.519518/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-test/$name.log' '--pid-file=td/osd-scrub-test/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_deep_scrub_randomize_ratio=0 --osd_scrub_interval_randomize_ratio=0 --osd_scrub_backoff_ratio=0.0 --osd_op_queue=wpq --osd_stats_update_period_not_scrubbing=1 --osd_stats_update_period_scrubbing=1 --osd_scrub_sleep=0.2 2024-09-15T22:17:14.331 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-15T22:17:14.331 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-15T22:17:14.331 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-15T22:17:14.360 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T22:17:14.359+0000 7fc3a3218640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T22:17:14.362 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T22:17:14.360+0000 7fc3a3218640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T22:17:14.363 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T22:17:14.362+0000 7fc3a3218640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T22:17:14.608 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 2 2024-09-15T22:17:14.609 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-15T22:17:14.609 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=2 2024-09-15T22:17:14.609 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-15T22:17:14.609 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-15T22:17:14.609 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 500 )) 2024-09-15T22:17:14.609 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-15T22:17:14.609 INFO:tasks.workunit.client.0.smithi136.stdout:0 2024-09-15T22:17:14.610 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:17:14.610 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T22:17:14.884 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:17:15.885 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:17:15.886 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 500 )) 2024-09-15T22:17:15.886 INFO:tasks.workunit.client.0.smithi136.stdout:1 2024-09-15T22:17:15.886 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-15T22:17:15.886 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:17:15.887 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T22:17:16.169 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:17:16.221 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T22:17:16.219+0000 7fc3a3218640 -1 Falling back to public interface 2024-09-15T22:17:17.170 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:17:17.170 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 500 )) 2024-09-15T22:17:17.171 INFO:tasks.workunit.client.0.smithi136.stdout:2 2024-09-15T22:17:17.171 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-15T22:17:17.171 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:17:17.171 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T22:17:17.451 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:17:18.453 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:17:18.453 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 500 )) 2024-09-15T22:17:18.454 INFO:tasks.workunit.client.0.smithi136.stdout:3 2024-09-15T22:17:18.454 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-15T22:17:18.454 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:17:18.454 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T22:17:18.749 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T22:17:19.234 INFO:tasks.workunit.client.0.smithi136.stderr:2024-09-15T22:17:19.232+0000 7fc3a3218640 -1 osd.2 0 log_to_monitors true 2024-09-15T22:17:19.750 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T22:17:19.750 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 500 )) 2024-09-15T22:17:19.750 INFO:tasks.workunit.client.0.smithi136.stdout:4 2024-09-15T22:17:19.751 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-15T22:17:19.751 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T22:17:19.751 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T22:17:20.041 INFO:tasks.workunit.client.0.smithi136.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/2767855600,v1:127.0.0.1:6819/2767855600] [v2:127.0.0.1:6820/2767855600,v1:127.0.0.1:6821/2767855600] exists,up d18724e3-c0e5-4909-aa38-4a6763399134 2024-09-15T22:17:20.042 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-15T22:17:20.042 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-15T22:17:20.042 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-15T22:17:20.042 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:555: TEST_dump_scrub_schedule: create_pool test 1 1 2024-09-15T22:17:20.042 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create test 1 1 2024-09-15T22:17:20.594 INFO:tasks.workunit.client.0.smithi136.stderr:pool 'test' created 2024-09-15T22:17:20.606 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2024-09-15T22:17:21.608 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:556: TEST_dump_scrub_schedule: wait_for_clean 2024-09-15T22:17:21.608 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-15T22:17:21.609 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-15T22:17:21.609 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-15T22:17:21.609 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-15T22:17:21.609 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-15T22:17:21.610 INFO:tasks.workunit.client.0.smithi136.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-15T22:17:21.610 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-15T22:17:21.610 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-15T22:17:21.610 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-15T22:17:21.707 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-15T22:17:21.707 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-15T22:17:21.707 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-15T22:17:21.707 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-15T22:17:21.707 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=500 2024-09-15T22:17:21.707 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-15T22:17:21.985 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-15T22:17:21.985 INFO:tasks.workunit.client.0.smithi136.stderr:1 2024-09-15T22:17:21.985 INFO:tasks.workunit.client.0.smithi136.stderr:2' 2024-09-15T22:17:21.985 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-15T22:17:21.985 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T22:17:21.985 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-15T22:17:22.083 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836487 2024-09-15T22:17:22.083 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836487 2024-09-15T22:17:22.083 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836487' 2024-09-15T22:17:22.083 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T22:17:22.083 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-15T22:17:22.184 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=42949672965 2024-09-15T22:17:22.184 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 42949672965 2024-09-15T22:17:22.184 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836487 1-42949672965' 2024-09-15T22:17:22.184 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T22:17:22.184 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-15T22:17:22.285 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=64424509442 2024-09-15T22:17:22.285 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 64424509442 2024-09-15T22:17:22.285 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836487 1-42949672965 2-64424509442' 2024-09-15T22:17:22.285 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T22:17:22.286 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836487 2024-09-15T22:17:22.286 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T22:17:22.286 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-15T22:17:22.287 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836487 2024-09-15T22:17:22.287 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T22:17:22.288 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836487 2024-09-15T22:17:22.288 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836487' 2024-09-15T22:17:22.288 INFO:tasks.workunit.client.0.smithi136.stdout:waiting osd.0 seq 21474836487 2024-09-15T22:17:22.288 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T22:17:22.662 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836487 -lt 21474836487 2024-09-15T22:17:22.662 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T22:17:22.663 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-42949672965 2024-09-15T22:17:22.663 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T22:17:22.663 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-15T22:17:22.664 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-42949672965 2024-09-15T22:17:22.664 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T22:17:22.665 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=42949672965 2024-09-15T22:17:22.665 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 42949672965' 2024-09-15T22:17:22.665 INFO:tasks.workunit.client.0.smithi136.stdout:waiting osd.1 seq 42949672965 2024-09-15T22:17:22.665 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-15T22:17:23.033 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 42949672965 -lt 42949672965 2024-09-15T22:17:23.033 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T22:17:23.033 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-64424509442 2024-09-15T22:17:23.033 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T22:17:23.034 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-15T22:17:23.034 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-64424509442 2024-09-15T22:17:23.035 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T22:17:23.036 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=64424509442 2024-09-15T22:17:23.036 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 64424509442' 2024-09-15T22:17:23.036 INFO:tasks.workunit.client.0.smithi136.stdout:waiting osd.2 seq 64424509442 2024-09-15T22:17:23.036 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-15T22:17:23.310 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 64424509442 -lt 64424509442 2024-09-15T22:17:23.311 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-15T22:17:23.311 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T22:17:23.311 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T22:17:23.652 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 1 == 0 2024-09-15T22:17:23.652 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-15T22:17:23.652 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-15T22:17:23.652 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-15T22:17:23.652 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T22:17:23.652 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T22:17:23.653 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T22:17:23.653 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T22:17:23.923 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=0 2024-09-15T22:17:23.923 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-15T22:17:23.923 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T22:17:23.923 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T22:17:24.263 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 0 = 1 2024-09-15T22:17:24.263 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 0 '!=' -1 2024-09-15T22:17:24.263 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: loop=0 2024-09-15T22:17:24.263 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1652: wait_for_clean: num_active_clean=0 2024-09-15T22:17:24.263 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-15T22:17:24.263 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 0.1 2024-09-15T22:17:24.364 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-15T22:17:24.365 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-15T22:17:24.365 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-15T22:17:24.365 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-15T22:17:24.365 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T22:17:24.365 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T22:17:24.365 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T22:17:24.365 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T22:17:24.636 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=0 2024-09-15T22:17:24.636 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-15T22:17:24.636 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T22:17:24.636 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T22:17:24.977 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 0 = 1 2024-09-15T22:17:24.977 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 0 '!=' 0 2024-09-15T22:17:24.977 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: get_is_making_recovery_progress 2024-09-15T22:17:24.977 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2024-09-15T22:17:24.977 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-15T22:17:24.977 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-15T22:17:24.977 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-15T22:17:24.977 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2024-09-15T22:17:24.977 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-15T22:17:25.320 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=null 2024-09-15T22:17:25.320 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test null '!=' null 2024-09-15T22:17:25.320 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1655: wait_for_clean: (( 1 >= 13 )) 2024-09-15T22:17:25.320 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-15T22:17:25.320 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 0.2 2024-09-15T22:17:25.521 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-15T22:17:25.521 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-15T22:17:25.521 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-15T22:17:25.522 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-15T22:17:25.522 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T22:17:25.522 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T22:17:25.522 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T22:17:25.522 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T22:17:25.790 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=0 2024-09-15T22:17:25.790 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-15T22:17:25.791 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T22:17:25.791 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T22:17:26.130 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 0 = 1 2024-09-15T22:17:26.130 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 0 '!=' 0 2024-09-15T22:17:26.130 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: get_is_making_recovery_progress 2024-09-15T22:17:26.130 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2024-09-15T22:17:26.130 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-15T22:17:26.130 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-15T22:17:26.130 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-15T22:17:26.130 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2024-09-15T22:17:26.130 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-15T22:17:26.486 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=null 2024-09-15T22:17:26.486 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test null '!=' null 2024-09-15T22:17:26.487 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1655: wait_for_clean: (( 2 >= 13 )) 2024-09-15T22:17:26.487 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-15T22:17:26.487 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 0.4 2024-09-15T22:17:26.887 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-15T22:17:26.888 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-15T22:17:26.888 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-15T22:17:26.888 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-15T22:17:26.888 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T22:17:26.888 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T22:17:26.888 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T22:17:26.888 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T22:17:27.156 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=0 2024-09-15T22:17:27.156 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-15T22:17:27.157 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T22:17:27.157 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T22:17:27.502 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 0 = 1 2024-09-15T22:17:27.502 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 0 '!=' 0 2024-09-15T22:17:27.502 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: get_is_making_recovery_progress 2024-09-15T22:17:27.502 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2024-09-15T22:17:27.502 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-15T22:17:27.503 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-15T22:17:27.503 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-15T22:17:27.503 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2024-09-15T22:17:27.503 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-15T22:17:27.837 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=null 2024-09-15T22:17:27.838 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test null '!=' null 2024-09-15T22:17:27.838 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1655: wait_for_clean: (( 3 >= 13 )) 2024-09-15T22:17:27.838 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-15T22:17:27.838 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 0.8 2024-09-15T22:17:28.638 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-15T22:17:28.639 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-15T22:17:28.639 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-15T22:17:28.639 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-15T22:17:28.640 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T22:17:28.640 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T22:17:28.640 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T22:17:28.640 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T22:17:28.912 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=1 2024-09-15T22:17:28.912 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-15T22:17:28.913 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T22:17:28.913 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T22:17:29.253 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 = 1 2024-09-15T22:17:29.253 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-15T22:17:29.253 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-15T22:17:29.253 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:557: TEST_dump_scrub_schedule: ceph osd dump 2024-09-15T22:17:29.253 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:557: TEST_dump_scrub_schedule: grep '^pool.*['\'']test['\'']' 2024-09-15T22:17:29.253 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:557: TEST_dump_scrub_schedule: awk '{ print $2 }' 2024-09-15T22:17:29.526 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:557: TEST_dump_scrub_schedule: poolid=1 2024-09-15T22:17:29.526 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:559: TEST_dump_scrub_schedule: dd if=/dev/urandom of=testdata.519518 bs=1032 count=1 2024-09-15T22:17:29.526 INFO:tasks.workunit.client.0.smithi136.stderr:1+0 records in 2024-09-15T22:17:29.527 INFO:tasks.workunit.client.0.smithi136.stderr:1+0 records out 2024-09-15T22:17:29.527 INFO:tasks.workunit.client.0.smithi136.stderr:1032 bytes (1.0 kB, 1.0 KiB) copied, 4.56e-05 s, 22.6 MB/s 2024-09-15T22:17:29.527 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:560: TEST_dump_scrub_schedule: seq 1 15 2024-09-15T22:17:29.528 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:560: TEST_dump_scrub_schedule: for i in `seq 1 $objects` 2024-09-15T22:17:29.528 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:562: TEST_dump_scrub_schedule: rados -p test put obj1 testdata.519518 2024-09-15T22:17:29.702 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:560: TEST_dump_scrub_schedule: for i in `seq 1 $objects` 2024-09-15T22:17:29.702 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:562: TEST_dump_scrub_schedule: rados -p test put obj2 testdata.519518 2024-09-15T22:17:29.814 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:560: TEST_dump_scrub_schedule: for i in `seq 1 $objects` 2024-09-15T22:17:29.814 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:562: TEST_dump_scrub_schedule: rados -p test put obj3 testdata.519518 2024-09-15T22:17:29.948 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:560: TEST_dump_scrub_schedule: for i in `seq 1 $objects` 2024-09-15T22:17:29.949 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:562: TEST_dump_scrub_schedule: rados -p test put obj4 testdata.519518 2024-09-15T22:17:30.074 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:560: TEST_dump_scrub_schedule: for i in `seq 1 $objects` 2024-09-15T22:17:30.074 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:562: TEST_dump_scrub_schedule: rados -p test put obj5 testdata.519518 2024-09-15T22:17:30.218 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:560: TEST_dump_scrub_schedule: for i in `seq 1 $objects` 2024-09-15T22:17:30.218 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:562: TEST_dump_scrub_schedule: rados -p test put obj6 testdata.519518 2024-09-15T22:17:30.361 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:560: TEST_dump_scrub_schedule: for i in `seq 1 $objects` 2024-09-15T22:17:30.361 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:562: TEST_dump_scrub_schedule: rados -p test put obj7 testdata.519518 2024-09-15T22:17:30.484 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:560: TEST_dump_scrub_schedule: for i in `seq 1 $objects` 2024-09-15T22:17:30.485 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:562: TEST_dump_scrub_schedule: rados -p test put obj8 testdata.519518 2024-09-15T22:17:30.644 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:560: TEST_dump_scrub_schedule: for i in `seq 1 $objects` 2024-09-15T22:17:30.644 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:562: TEST_dump_scrub_schedule: rados -p test put obj9 testdata.519518 2024-09-15T22:17:30.803 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:560: TEST_dump_scrub_schedule: for i in `seq 1 $objects` 2024-09-15T22:17:30.803 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:562: TEST_dump_scrub_schedule: rados -p test put obj10 testdata.519518 2024-09-15T22:17:30.938 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:560: TEST_dump_scrub_schedule: for i in `seq 1 $objects` 2024-09-15T22:17:30.938 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:562: TEST_dump_scrub_schedule: rados -p test put obj11 testdata.519518 2024-09-15T22:17:31.071 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:560: TEST_dump_scrub_schedule: for i in `seq 1 $objects` 2024-09-15T22:17:31.071 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:562: TEST_dump_scrub_schedule: rados -p test put obj12 testdata.519518 2024-09-15T22:17:31.189 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:560: TEST_dump_scrub_schedule: for i in `seq 1 $objects` 2024-09-15T22:17:31.189 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:562: TEST_dump_scrub_schedule: rados -p test put obj13 testdata.519518 2024-09-15T22:17:31.323 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:560: TEST_dump_scrub_schedule: for i in `seq 1 $objects` 2024-09-15T22:17:31.323 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:562: TEST_dump_scrub_schedule: rados -p test put obj14 testdata.519518 2024-09-15T22:17:31.457 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:560: TEST_dump_scrub_schedule: for i in `seq 1 $objects` 2024-09-15T22:17:31.457 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:562: TEST_dump_scrub_schedule: rados -p test put obj15 testdata.519518 2024-09-15T22:17:31.582 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:564: TEST_dump_scrub_schedule: rm -f testdata.519518 2024-09-15T22:17:31.583 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:566: TEST_dump_scrub_schedule: local pgid=1.0 2024-09-15T22:17:31.583 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:568: TEST_dump_scrub_schedule: date +%Y-%m-%dT%H:%M:%S.%N%:z 2024-09-15T22:17:31.585 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:568: TEST_dump_scrub_schedule: local now_is=2024-09-15T22:17:31.584036435+00:00 2024-09-15T22:17:31.585 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:576: TEST_dump_scrub_schedule: expct_starting=(['query_active']='false' ['query_is_future']='true' ['query_schedule']='scrub scheduled') 2024-09-15T22:17:31.585 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:576: TEST_dump_scrub_schedule: declare -A expct_starting 2024-09-15T22:17:31.585 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:577: TEST_dump_scrub_schedule: declare -A sched_data 2024-09-15T22:17:31.585 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:578: TEST_dump_scrub_schedule: extract_published_sch 1.0 2024-09-15T22:17:31.584036435+00:00 2019-10-12T20:32:43.645168+0000 sched_data 2024-09-15T22:17:31.586 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:17: extract_published_sch: local pgn=1.0 2024-09-15T22:17:31.586 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:18: extract_published_sch: local -n dict=sched_data 2024-09-15T22:17:31.586 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:19: extract_published_sch: local current_time=2024-09-15T22:17:31.584036435+00:00 2024-09-15T22:17:31.586 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:20: extract_published_sch: local extra_time=2019-10-12T20:32:43.645168+0000 2024-09-15T22:17:31.586 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:21: extract_published_sch: local extr_dbg=2 2024-09-15T22:17:31.586 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:24: extract_published_sch: local saved_echo_flag=x 2024-09-15T22:17:31.586 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:25: extract_published_sch: set +x 2024-09-15T22:17:31.843 INFO:tasks.workunit.client.0.smithi136.stderr:dumped pgs 2024-09-15T22:17:31.870 INFO:tasks.workunit.client.0.smithi136.stdout:from pg dump pg: { 2024-09-15T22:17:31.870 INFO:tasks.workunit.client.0.smithi136.stdout: "dmp_pg_state": "active+clean", 2024-09-15T22:17:31.870 INFO:tasks.workunit.client.0.smithi136.stdout: "dmp_state_has_scrubbing": false, 2024-09-15T22:17:31.870 INFO:tasks.workunit.client.0.smithi136.stdout: "dmp_last_duration": 0, 2024-09-15T22:17:31.870 INFO:tasks.workunit.client.0.smithi136.stdout: "dmp_schedule": "periodic scrub scheduled", 2024-09-15T22:17:31.870 INFO:tasks.workunit.client.0.smithi136.stdout: "dmp_schedule_at": "2024-09-16T22:17:20.476037+0000", 2024-09-15T22:17:31.870 INFO:tasks.workunit.client.0.smithi136.stdout: "dmp_is_future": true, 2024-09-15T22:17:31.870 INFO:tasks.workunit.client.0.smithi136.stdout: "dmp_vs_date": true, 2024-09-15T22:17:31.870 INFO:tasks.workunit.client.0.smithi136.stdout: "dmp_reported_epoch": 19, 2024-09-15T22:17:31.871 INFO:tasks.workunit.client.0.smithi136.stdout: "dmp_seq": 15 2024-09-15T22:17:31.871 INFO:tasks.workunit.client.0.smithi136.stdout:} 2024-09-15T22:17:31.871 INFO:tasks.workunit.client.0.smithi136.stdout:query output: 2024-09-15T22:17:31.973 INFO:tasks.workunit.client.0.smithi136.stdout: "scrubber": { 2024-09-15T22:17:31.973 INFO:tasks.workunit.client.0.smithi136.stdout: "active": false, 2024-09-15T22:17:31.973 INFO:tasks.workunit.client.0.smithi136.stdout: "must_scrub": false, 2024-09-15T22:17:31.973 INFO:tasks.workunit.client.0.smithi136.stdout: "must_deep_scrub": false, 2024-09-15T22:17:31.973 INFO:tasks.workunit.client.0.smithi136.stdout: "must_repair": false, 2024-09-15T22:17:31.973 INFO:tasks.workunit.client.0.smithi136.stdout: "need_auto": false, 2024-09-15T22:17:31.973 INFO:tasks.workunit.client.0.smithi136.stdout: "scrub_reg_stamp": "2024-09-16T22:17:20.476037+0000", 2024-09-15T22:17:31.974 INFO:tasks.workunit.client.0.smithi136.stdout: "schedule": "scrub scheduled @ 2024-09-16T22:17:20.476 (2024-09-16T22:17:20.476)" 2024-09-15T22:17:31.974 INFO:tasks.workunit.client.0.smithi136.stdout: }, 2024-09-15T22:17:31.974 INFO:tasks.workunit.client.0.smithi136.stdout: "agent_state": {} 2024-09-15T22:17:32.080 INFO:tasks.workunit.client.0.smithi136.stdout:( 2024-09-15T22:17:32.081 INFO:tasks.workunit.client.0.smithi136.stdout:[query_epoch]=19 2024-09-15T22:17:32.081 INFO:tasks.workunit.client.0.smithi136.stdout:[query_seq]=32 2024-09-15T22:17:32.081 INFO:tasks.workunit.client.0.smithi136.stdout:[query_active]=false 2024-09-15T22:17:32.081 INFO:tasks.workunit.client.0.smithi136.stdout:[query_schedule]='scrub scheduled' 2024-09-15T22:17:32.081 INFO:tasks.workunit.client.0.smithi136.stdout:[query_schedule_at]='2024-09-16T22:17:20.476' 2024-09-15T22:17:32.081 INFO:tasks.workunit.client.0.smithi136.stdout:[query_target_at]='2024-09-16T22:17:20.476' 2024-09-15T22:17:32.081 INFO:tasks.workunit.client.0.smithi136.stdout:[query_last_duration]=0 2024-09-15T22:17:32.081 INFO:tasks.workunit.client.0.smithi136.stdout:[query_last_stamp]='2024-09-15T22:17:20.476037+0000' 2024-09-15T22:17:32.081 INFO:tasks.workunit.client.0.smithi136.stdout:[query_last_scrub]='0x0' 2024-09-15T22:17:32.081 INFO:tasks.workunit.client.0.smithi136.stdout:[query_is_future]=true 2024-09-15T22:17:32.081 INFO:tasks.workunit.client.0.smithi136.stdout:[query_vs_date]=true 2024-09-15T22:17:32.081 INFO:tasks.workunit.client.0.smithi136.stdout:[query_scrub_seq]=null 2024-09-15T22:17:32.081 INFO:tasks.workunit.client.0.smithi136.stdout:[dmp_pg_state]='active+clean' 2024-09-15T22:17:32.081 INFO:tasks.workunit.client.0.smithi136.stdout:[dmp_state_has_scrubbing]=false 2024-09-15T22:17:32.082 INFO:tasks.workunit.client.0.smithi136.stdout:[dmp_last_duration]=0 2024-09-15T22:17:32.082 INFO:tasks.workunit.client.0.smithi136.stdout:[dmp_schedule]='periodic scrub scheduled' 2024-09-15T22:17:32.082 INFO:tasks.workunit.client.0.smithi136.stdout:[dmp_schedule_at]='2024-09-16T22:17:20.476037+0000' 2024-09-15T22:17:32.082 INFO:tasks.workunit.client.0.smithi136.stdout:[dmp_is_future]=true 2024-09-15T22:17:32.082 INFO:tasks.workunit.client.0.smithi136.stdout:[dmp_vs_date]=true 2024-09-15T22:17:32.082 INFO:tasks.workunit.client.0.smithi136.stdout:[dmp_reported_epoch]=19 2024-09-15T22:17:32.082 INFO:tasks.workunit.client.0.smithi136.stdout:[dmp_seq]=15 2024-09-15T22:17:32.082 INFO:tasks.workunit.client.0.smithi136.stdout:) 2024-09-15T22:17:32.084 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:579: TEST_dump_scrub_schedule: schedule_against_expected sched_data expct_starting initial 2024-09-15T22:17:32.084 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:178: schedule_against_expected: local -n dict=sched_data 2024-09-15T22:17:32.084 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:179: schedule_against_expected: local -n ep=expct_starting 2024-09-15T22:17:32.085 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:180: schedule_against_expected: local extr_dbg=1 2024-09-15T22:17:32.085 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:183: schedule_against_expected: local saved_echo_flag=x 2024-09-15T22:17:32.085 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:184: schedule_against_expected: set +x 2024-09-15T22:17:32.085 INFO:tasks.workunit.client.0.smithi136.stdout:-- - comparing: 2024-09-15T22:17:32.085 INFO:tasks.workunit.client.0.smithi136.stdout:key is query_schedule expected: scrub scheduled in actual: scrub scheduled 2024-09-15T22:17:32.085 INFO:tasks.workunit.client.0.smithi136.stdout:key is query_is_future expected: true in actual: true 2024-09-15T22:17:32.086 INFO:tasks.workunit.client.0.smithi136.stdout:key is query_active expected: false in actual: false 2024-09-15T22:17:32.086 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:203: schedule_against_expected: return 0 2024-09-15T22:17:32.086 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:580: TEST_dump_scrub_schedule: (( 0 == 0 )) 2024-09-15T22:17:32.086 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:581: TEST_dump_scrub_schedule: echo 'last-scrub --- ' 0x0 2024-09-15T22:17:32.086 INFO:tasks.workunit.client.0.smithi136.stdout:last-scrub --- 0x0 2024-09-15T22:17:32.087 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:587: TEST_dump_scrub_schedule: saved_last_stamp=2024-09-15T22:17:20.476037+0000 2024-09-15T22:17:32.087 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:588: TEST_dump_scrub_schedule: ceph tell 'osd.*' config set osd_scrub_sleep 0 2024-09-15T22:17:32.177 INFO:tasks.workunit.client.0.smithi136.stdout:osd.0: { 2024-09-15T22:17:32.177 INFO:tasks.workunit.client.0.smithi136.stdout: "success": "osd_scrub_sleep = '' " 2024-09-15T22:17:32.177 INFO:tasks.workunit.client.0.smithi136.stdout:} 2024-09-15T22:17:32.191 INFO:tasks.workunit.client.0.smithi136.stdout:osd.1: { 2024-09-15T22:17:32.191 INFO:tasks.workunit.client.0.smithi136.stdout: "success": "osd_scrub_sleep = '' " 2024-09-15T22:17:32.191 INFO:tasks.workunit.client.0.smithi136.stdout:} 2024-09-15T22:17:32.203 INFO:tasks.workunit.client.0.smithi136.stdout:osd.2: { 2024-09-15T22:17:32.203 INFO:tasks.workunit.client.0.smithi136.stdout: "success": "osd_scrub_sleep = '' " 2024-09-15T22:17:32.203 INFO:tasks.workunit.client.0.smithi136.stdout:} 2024-09-15T22:17:32.212 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:589: TEST_dump_scrub_schedule: ceph tell 1.0 deep-scrub 2024-09-15T22:17:32.324 INFO:tasks.workunit.client.0.smithi136.stdout:{ 2024-09-15T22:17:32.324 INFO:tasks.workunit.client.0.smithi136.stdout: "deep": true, 2024-09-15T22:17:32.324 INFO:tasks.workunit.client.0.smithi136.stdout: "must": true, 2024-09-15T22:17:32.325 INFO:tasks.workunit.client.0.smithi136.stdout: "stamp": "0.000000" 2024-09-15T22:17:32.325 INFO:tasks.workunit.client.0.smithi136.stdout:} 2024-09-15T22:17:32.333 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:594: TEST_dump_scrub_schedule: sleep 5 2024-09-15T22:17:37.336 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:595: TEST_dump_scrub_schedule: sched_data=() 2024-09-15T22:17:37.336 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:596: TEST_dump_scrub_schedule: expct_qry_duration=(['query_last_duration']='0' ['query_last_duration_neg']='not0') 2024-09-15T22:17:37.337 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:596: TEST_dump_scrub_schedule: declare -A expct_qry_duration 2024-09-15T22:17:37.337 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:597: TEST_dump_scrub_schedule: wait_any_cond 1.0 10 2024-09-15T22:17:20.476037+0000 expct_qry_duration 'WaitingAfterScrub ' sched_data 2024-09-15T22:17:37.337 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:106: wait_any_cond: local pgid=1.0 2024-09-15T22:17:37.337 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:107: wait_any_cond: local retries=10 2024-09-15T22:17:37.337 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:108: wait_any_cond: local cmp_date=2024-09-15T22:17:20.476037+0000 2024-09-15T22:17:37.337 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:109: wait_any_cond: local -n ep=expct_qry_duration 2024-09-15T22:17:37.337 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:110: wait_any_cond: local -n out_array=sched_data 2024-09-15T22:17:37.337 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:111: wait_any_cond: local -A sc_data 2024-09-15T22:17:37.337 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:112: wait_any_cond: local extr_dbg=2 2024-09-15T22:17:37.338 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:115: wait_any_cond: local saved_echo_flag=x 2024-09-15T22:17:37.338 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:116: wait_any_cond: set +x 2024-09-15T22:17:37.338 INFO:tasks.workunit.client.0.smithi136.stdout:waiting for any condition (WaitingAfterScrub ): pg:1.0 dt:2024-09-15T22:17:20.476037+0000 (10 retries) 2024-09-15T22:17:38.112 INFO:tasks.workunit.client.0.smithi136.stderr:dumped pgs 2024-09-15T22:17:38.124 INFO:tasks.workunit.client.0.smithi136.stdout:from pg dump pg: { 2024-09-15T22:17:38.124 INFO:tasks.workunit.client.0.smithi136.stdout: "dmp_pg_state": "active+clean", 2024-09-15T22:17:38.124 INFO:tasks.workunit.client.0.smithi136.stdout: "dmp_state_has_scrubbing": false, 2024-09-15T22:17:38.125 INFO:tasks.workunit.client.0.smithi136.stdout: "dmp_last_duration": 0, 2024-09-15T22:17:38.125 INFO:tasks.workunit.client.0.smithi136.stdout: "dmp_schedule": "periodic scrub scheduled", 2024-09-15T22:17:38.125 INFO:tasks.workunit.client.0.smithi136.stdout: "dmp_schedule_at": "2024-09-16T22:17:20.476037+0000", 2024-09-15T22:17:38.125 INFO:tasks.workunit.client.0.smithi136.stdout: "dmp_is_future": true, 2024-09-15T22:17:38.125 INFO:tasks.workunit.client.0.smithi136.stdout: "dmp_vs_date": true, 2024-09-15T22:17:38.125 INFO:tasks.workunit.client.0.smithi136.stdout: "dmp_reported_epoch": 19, 2024-09-15T22:17:38.125 INFO:tasks.workunit.client.0.smithi136.stdout: "dmp_seq": 30 2024-09-15T22:17:38.125 INFO:tasks.workunit.client.0.smithi136.stdout:} 2024-09-15T22:17:38.125 INFO:tasks.workunit.client.0.smithi136.stdout:query output: 2024-09-15T22:17:38.228 INFO:tasks.workunit.client.0.smithi136.stdout: "scrubber": { 2024-09-15T22:17:38.228 INFO:tasks.workunit.client.0.smithi136.stdout: "active": true, 2024-09-15T22:17:38.228 INFO:tasks.workunit.client.0.smithi136.stdout: "epoch_start": "17", 2024-09-15T22:17:38.228 INFO:tasks.workunit.client.0.smithi136.stdout: "start": "MIN", 2024-09-15T22:17:38.229 INFO:tasks.workunit.client.0.smithi136.stdout: "end": "MIN", 2024-09-15T22:17:38.229 INFO:tasks.workunit.client.0.smithi136.stdout: "max_end": "MIN", 2024-09-15T22:17:38.229 INFO:tasks.workunit.client.0.smithi136.stdout: "subset_last_update": "0'0", 2024-09-15T22:17:38.229 INFO:tasks.workunit.client.0.smithi136.stdout: "deep": true, 2024-09-15T22:17:38.229 INFO:tasks.workunit.client.0.smithi136.stdout: "req_scrub": false, 2024-09-15T22:17:38.229 INFO:tasks.workunit.client.0.smithi136.stdout: "auto_repair": false, 2024-09-15T22:17:38.229 INFO:tasks.workunit.client.0.smithi136.stdout: "check_repair": false, 2024-09-15T22:17:38.229 INFO:tasks.workunit.client.0.smithi136.stdout: "deep_scrub_on_error": false, 2024-09-15T22:17:38.229 INFO:tasks.workunit.client.0.smithi136.stdout: "priority": 0, 2024-09-15T22:17:38.229 INFO:tasks.workunit.client.0.smithi136.stdout: "shallow_errors": 0, 2024-09-15T22:17:38.229 INFO:tasks.workunit.client.0.smithi136.stdout: "deep_errors": 0, 2024-09-15T22:17:38.229 INFO:tasks.workunit.client.0.smithi136.stdout: "fixed": 0, 2024-09-15T22:17:38.229 INFO:tasks.workunit.client.0.smithi136.stdout: "waiting_on_whom": [], 2024-09-15T22:17:38.229 INFO:tasks.workunit.client.0.smithi136.stdout: "schedule": "scrubbing" 2024-09-15T22:17:38.229 INFO:tasks.workunit.client.0.smithi136.stdout: }, 2024-09-15T22:17:38.230 INFO:tasks.workunit.client.0.smithi136.stdout: "agent_state": {} 2024-09-15T22:17:38.335 INFO:tasks.workunit.client.0.smithi136.stdout:( 2024-09-15T22:17:38.335 INFO:tasks.workunit.client.0.smithi136.stdout:[query_epoch]=19 2024-09-15T22:17:38.335 INFO:tasks.workunit.client.0.smithi136.stdout:[query_seq]=42 2024-09-15T22:17:38.336 INFO:tasks.workunit.client.0.smithi136.stdout:[query_active]=true 2024-09-15T22:17:38.336 INFO:tasks.workunit.client.0.smithi136.stdout:[query_schedule]='scrubbing' 2024-09-15T22:17:38.336 INFO:tasks.workunit.client.0.smithi136.stdout:[query_schedule_at]='0' 2024-09-15T22:17:38.336 INFO:tasks.workunit.client.0.smithi136.stdout:[query_target_at]='0' 2024-09-15T22:17:38.336 INFO:tasks.workunit.client.0.smithi136.stdout:[query_last_duration]=1 2024-09-15T22:17:38.336 INFO:tasks.workunit.client.0.smithi136.stdout:[query_last_stamp]='2024-09-15T22:17:33.277327+0000' 2024-09-15T22:17:38.336 INFO:tasks.workunit.client.0.smithi136.stdout:[query_last_scrub]='19x15' 2024-09-15T22:17:38.336 INFO:tasks.workunit.client.0.smithi136.stdout:[query_is_future]=false 2024-09-15T22:17:38.336 INFO:tasks.workunit.client.0.smithi136.stdout:[query_vs_date]=false 2024-09-15T22:17:38.336 INFO:tasks.workunit.client.0.smithi136.stdout:[query_scrub_seq]=null 2024-09-15T22:17:38.336 INFO:tasks.workunit.client.0.smithi136.stdout:[dmp_pg_state]='active+clean' 2024-09-15T22:17:38.336 INFO:tasks.workunit.client.0.smithi136.stdout:[dmp_state_has_scrubbing]=false 2024-09-15T22:17:38.336 INFO:tasks.workunit.client.0.smithi136.stdout:[dmp_last_duration]=0 2024-09-15T22:17:38.336 INFO:tasks.workunit.client.0.smithi136.stdout:[dmp_schedule]='periodic scrub scheduled' 2024-09-15T22:17:38.337 INFO:tasks.workunit.client.0.smithi136.stdout:[dmp_schedule_at]='2024-09-16T22:17:20.476037+0000' 2024-09-15T22:17:38.337 INFO:tasks.workunit.client.0.smithi136.stdout:[dmp_is_future]=true 2024-09-15T22:17:38.337 INFO:tasks.workunit.client.0.smithi136.stdout:[dmp_vs_date]=true 2024-09-15T22:17:38.337 INFO:tasks.workunit.client.0.smithi136.stdout:[dmp_reported_epoch]=19 2024-09-15T22:17:38.338 INFO:tasks.workunit.client.0.smithi136.stdout:[dmp_seq]=30 2024-09-15T22:17:38.338 INFO:tasks.workunit.client.0.smithi136.stdout:) 2024-09-15T22:17:38.340 INFO:tasks.workunit.client.0.smithi136.stdout:--> loop: 1 ~ true / 42 / 30 / false / 2024-09-15T22:17:33.277327+0000 / scrubbing %%% query_last_duration_neg query_last_duration 2024-09-15T22:17:38.340 INFO:tasks.workunit.client.0.smithi136.stdout:key is query_last_duration: negation:1 # expected: 0 # in actual: 1 2024-09-15T22:17:38.341 INFO:tasks.workunit.client.0.smithi136.stdout:WaitingAfterScrub - 'query_last_duration' actual value (1) matches expected (0) (negation: 1) 2024-09-15T22:17:38.342 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:157: wait_any_cond: return 0 2024-09-15T22:17:38.342 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:599: TEST_dump_scrub_schedule: sched_data=() 2024-09-15T22:17:38.342 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:600: TEST_dump_scrub_schedule: expct_dmp_duration=(['dmp_last_duration']='0' ['dmp_last_duration_neg']='not0') 2024-09-15T22:17:38.342 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:600: TEST_dump_scrub_schedule: declare -A expct_dmp_duration 2024-09-15T22:17:38.342 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:601: TEST_dump_scrub_schedule: wait_any_cond 1.0 10 2024-09-15T22:17:20.476037+0000 expct_dmp_duration 'WaitingAfterScrub_dmp ' sched_data 2024-09-15T22:17:38.342 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:106: wait_any_cond: local pgid=1.0 2024-09-15T22:17:38.342 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:107: wait_any_cond: local retries=10 2024-09-15T22:17:38.342 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:108: wait_any_cond: local cmp_date=2024-09-15T22:17:20.476037+0000 2024-09-15T22:17:38.343 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:109: wait_any_cond: local -n ep=expct_dmp_duration 2024-09-15T22:17:38.343 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:110: wait_any_cond: local -n out_array=sched_data 2024-09-15T22:17:38.343 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:111: wait_any_cond: local -A sc_data 2024-09-15T22:17:38.343 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:112: wait_any_cond: local extr_dbg=2 2024-09-15T22:17:38.343 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:115: wait_any_cond: local saved_echo_flag=x 2024-09-15T22:17:38.343 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:116: wait_any_cond: set +x 2024-09-15T22:17:38.343 INFO:tasks.workunit.client.0.smithi136.stdout:waiting for any condition (WaitingAfterScrub_dmp ): pg:1.0 dt:2024-09-15T22:17:20.476037+0000 (10 retries) 2024-09-15T22:17:39.100 INFO:tasks.workunit.client.0.smithi136.stderr:dumped pgs 2024-09-15T22:17:39.111 INFO:tasks.workunit.client.0.smithi136.stdout:from pg dump pg: { 2024-09-15T22:17:39.112 INFO:tasks.workunit.client.0.smithi136.stdout: "dmp_pg_state": "active+clean", 2024-09-15T22:17:39.112 INFO:tasks.workunit.client.0.smithi136.stdout: "dmp_state_has_scrubbing": false, 2024-09-15T22:17:39.112 INFO:tasks.workunit.client.0.smithi136.stdout: "dmp_last_duration": 1, 2024-09-15T22:17:39.112 INFO:tasks.workunit.client.0.smithi136.stdout: "dmp_schedule": "periodic scrub scheduled", 2024-09-15T22:17:39.112 INFO:tasks.workunit.client.0.smithi136.stdout: "dmp_schedule_at": "2024-09-16T22:17:33.277327+0000", 2024-09-15T22:17:39.112 INFO:tasks.workunit.client.0.smithi136.stdout: "dmp_is_future": true, 2024-09-15T22:17:39.112 INFO:tasks.workunit.client.0.smithi136.stdout: "dmp_vs_date": true, 2024-09-15T22:17:39.113 INFO:tasks.workunit.client.0.smithi136.stdout: "dmp_reported_epoch": 19, 2024-09-15T22:17:39.113 INFO:tasks.workunit.client.0.smithi136.stdout: "dmp_seq": 41 2024-09-15T22:17:39.113 INFO:tasks.workunit.client.0.smithi136.stdout:} 2024-09-15T22:17:39.113 INFO:tasks.workunit.client.0.smithi136.stdout:query output: 2024-09-15T22:17:39.215 INFO:tasks.workunit.client.0.smithi136.stdout: "scrubber": { 2024-09-15T22:17:39.215 INFO:tasks.workunit.client.0.smithi136.stdout: "active": true, 2024-09-15T22:17:39.215 INFO:tasks.workunit.client.0.smithi136.stdout: "epoch_start": "17", 2024-09-15T22:17:39.215 INFO:tasks.workunit.client.0.smithi136.stdout: "start": "MIN", 2024-09-15T22:17:39.215 INFO:tasks.workunit.client.0.smithi136.stdout: "end": "MIN", 2024-09-15T22:17:39.215 INFO:tasks.workunit.client.0.smithi136.stdout: "max_end": "MIN", 2024-09-15T22:17:39.215 INFO:tasks.workunit.client.0.smithi136.stdout: "subset_last_update": "0'0", 2024-09-15T22:17:39.215 INFO:tasks.workunit.client.0.smithi136.stdout: "deep": true, 2024-09-15T22:17:39.215 INFO:tasks.workunit.client.0.smithi136.stdout: "req_scrub": false, 2024-09-15T22:17:39.215 INFO:tasks.workunit.client.0.smithi136.stdout: "auto_repair": false, 2024-09-15T22:17:39.215 INFO:tasks.workunit.client.0.smithi136.stdout: "check_repair": false, 2024-09-15T22:17:39.215 INFO:tasks.workunit.client.0.smithi136.stdout: "deep_scrub_on_error": false, 2024-09-15T22:17:39.216 INFO:tasks.workunit.client.0.smithi136.stdout: "priority": 0, 2024-09-15T22:17:39.216 INFO:tasks.workunit.client.0.smithi136.stdout: "shallow_errors": 0, 2024-09-15T22:17:39.216 INFO:tasks.workunit.client.0.smithi136.stdout: "deep_errors": 0, 2024-09-15T22:17:39.216 INFO:tasks.workunit.client.0.smithi136.stdout: "fixed": 0, 2024-09-15T22:17:39.216 INFO:tasks.workunit.client.0.smithi136.stdout: "waiting_on_whom": [], 2024-09-15T22:17:39.216 INFO:tasks.workunit.client.0.smithi136.stdout: "schedule": "scrubbing" 2024-09-15T22:17:39.216 INFO:tasks.workunit.client.0.smithi136.stdout: }, 2024-09-15T22:17:39.216 INFO:tasks.workunit.client.0.smithi136.stdout: "agent_state": {} 2024-09-15T22:17:39.321 INFO:tasks.workunit.client.0.smithi136.stdout:( 2024-09-15T22:17:39.321 INFO:tasks.workunit.client.0.smithi136.stdout:[query_epoch]=19 2024-09-15T22:17:39.321 INFO:tasks.workunit.client.0.smithi136.stdout:[query_seq]=42 2024-09-15T22:17:39.321 INFO:tasks.workunit.client.0.smithi136.stdout:[query_active]=true 2024-09-15T22:17:39.321 INFO:tasks.workunit.client.0.smithi136.stdout:[query_schedule]='scrubbing' 2024-09-15T22:17:39.321 INFO:tasks.workunit.client.0.smithi136.stdout:[query_schedule_at]='0' 2024-09-15T22:17:39.321 INFO:tasks.workunit.client.0.smithi136.stdout:[query_target_at]='0' 2024-09-15T22:17:39.321 INFO:tasks.workunit.client.0.smithi136.stdout:[query_last_duration]=1 2024-09-15T22:17:39.322 INFO:tasks.workunit.client.0.smithi136.stdout:[query_last_stamp]='2024-09-15T22:17:33.277327+0000' 2024-09-15T22:17:39.322 INFO:tasks.workunit.client.0.smithi136.stdout:[query_last_scrub]='19x15' 2024-09-15T22:17:39.322 INFO:tasks.workunit.client.0.smithi136.stdout:[query_is_future]=false 2024-09-15T22:17:39.322 INFO:tasks.workunit.client.0.smithi136.stdout:[query_vs_date]=false 2024-09-15T22:17:39.322 INFO:tasks.workunit.client.0.smithi136.stdout:[query_scrub_seq]=null 2024-09-15T22:17:39.322 INFO:tasks.workunit.client.0.smithi136.stdout:[dmp_pg_state]='active+clean' 2024-09-15T22:17:39.322 INFO:tasks.workunit.client.0.smithi136.stdout:[dmp_state_has_scrubbing]=false 2024-09-15T22:17:39.322 INFO:tasks.workunit.client.0.smithi136.stdout:[dmp_last_duration]=1 2024-09-15T22:17:39.322 INFO:tasks.workunit.client.0.smithi136.stdout:[dmp_schedule]='periodic scrub scheduled' 2024-09-15T22:17:39.322 INFO:tasks.workunit.client.0.smithi136.stdout:[dmp_schedule_at]='2024-09-16T22:17:33.277327+0000' 2024-09-15T22:17:39.322 INFO:tasks.workunit.client.0.smithi136.stdout:[dmp_is_future]=true 2024-09-15T22:17:39.322 INFO:tasks.workunit.client.0.smithi136.stdout:[dmp_vs_date]=true 2024-09-15T22:17:39.322 INFO:tasks.workunit.client.0.smithi136.stdout:[dmp_reported_epoch]=19 2024-09-15T22:17:39.322 INFO:tasks.workunit.client.0.smithi136.stdout:[dmp_seq]=41 2024-09-15T22:17:39.322 INFO:tasks.workunit.client.0.smithi136.stdout:) 2024-09-15T22:17:39.324 INFO:tasks.workunit.client.0.smithi136.stdout:--> loop: 1 ~ true / 42 / 41 / false / 2024-09-15T22:17:33.277327+0000 / scrubbing %%% dmp_last_duration dmp_last_duration_neg 2024-09-15T22:17:39.325 INFO:tasks.workunit.client.0.smithi136.stdout:key is dmp_last_duration: negation:1 # expected: 0 # in actual: 1 2024-09-15T22:17:39.325 INFO:tasks.workunit.client.0.smithi136.stdout:WaitingAfterScrub_dmp - 'dmp_last_duration' actual value (1) matches expected (0) (negation: 1) 2024-09-15T22:17:39.325 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:157: wait_any_cond: return 0 2024-09-15T22:17:39.325 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:603: TEST_dump_scrub_schedule: sleep 2 2024-09-15T22:17:41.326 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:610: TEST_dump_scrub_schedule: ceph tell 'osd.*' config set osd_shallow_scrub_chunk_max 3 2024-09-15T22:17:41.431 INFO:tasks.workunit.client.0.smithi136.stdout:osd.0: { 2024-09-15T22:17:41.431 INFO:tasks.workunit.client.0.smithi136.stdout: "success": "osd_shallow_scrub_chunk_max = '' (not observed, change may require restart) " 2024-09-15T22:17:41.431 INFO:tasks.workunit.client.0.smithi136.stdout:} 2024-09-15T22:17:41.445 INFO:tasks.workunit.client.0.smithi136.stdout:osd.1: { 2024-09-15T22:17:41.445 INFO:tasks.workunit.client.0.smithi136.stdout: "success": "osd_shallow_scrub_chunk_max = '' (not observed, change may require restart) " 2024-09-15T22:17:41.445 INFO:tasks.workunit.client.0.smithi136.stdout:} 2024-09-15T22:17:41.457 INFO:tasks.workunit.client.0.smithi136.stdout:osd.2: { 2024-09-15T22:17:41.457 INFO:tasks.workunit.client.0.smithi136.stdout: "success": "osd_shallow_scrub_chunk_max = '' (not observed, change may require restart) " 2024-09-15T22:17:41.458 INFO:tasks.workunit.client.0.smithi136.stdout:} 2024-09-15T22:17:41.467 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:611: TEST_dump_scrub_schedule: ceph tell 'osd.*' config set osd_scrub_sleep 2.0 2024-09-15T22:17:41.570 INFO:tasks.workunit.client.0.smithi136.stdout:osd.0: { 2024-09-15T22:17:41.570 INFO:tasks.workunit.client.0.smithi136.stdout: "success": "osd_scrub_sleep = '' " 2024-09-15T22:17:41.570 INFO:tasks.workunit.client.0.smithi136.stdout:} 2024-09-15T22:17:41.583 INFO:tasks.workunit.client.0.smithi136.stdout:osd.1: { 2024-09-15T22:17:41.583 INFO:tasks.workunit.client.0.smithi136.stdout: "success": "osd_scrub_sleep = '' " 2024-09-15T22:17:41.583 INFO:tasks.workunit.client.0.smithi136.stdout:} 2024-09-15T22:17:41.596 INFO:tasks.workunit.client.0.smithi136.stdout:osd.2: { 2024-09-15T22:17:41.596 INFO:tasks.workunit.client.0.smithi136.stdout: "success": "osd_scrub_sleep = '' " 2024-09-15T22:17:41.596 INFO:tasks.workunit.client.0.smithi136.stdout:} 2024-09-15T22:17:41.605 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:612: TEST_dump_scrub_schedule: ceph osd set noscrub 2024-09-15T22:17:41.986 INFO:tasks.workunit.client.0.smithi136.stderr:noscrub is set 2024-09-15T22:17:41.997 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:613: TEST_dump_scrub_schedule: sleep 2 2024-09-15T22:17:43.998 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:614: TEST_dump_scrub_schedule: saved_last_stamp=2024-09-15T22:17:33.277327+0000 2024-09-15T22:17:43.999 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:616: TEST_dump_scrub_schedule: ceph tell 1.0 schedule-scrub 2024-09-15T22:17:44.104 INFO:tasks.workunit.client.0.smithi136.stdout:{ 2024-09-15T22:17:44.104 INFO:tasks.workunit.client.0.smithi136.stdout: "deep": false, 2024-09-15T22:17:44.104 INFO:tasks.workunit.client.0.smithi136.stdout: "must": false, 2024-09-15T22:17:44.104 INFO:tasks.workunit.client.0.smithi136.stdout: "stamp": "2024-09-08T22:16:04.103344+0000" 2024-09-15T22:17:44.104 INFO:tasks.workunit.client.0.smithi136.stdout:} 2024-09-15T22:17:44.113 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:617: TEST_dump_scrub_schedule: sleep 1 2024-09-15T22:17:45.116 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:618: TEST_dump_scrub_schedule: sched_data=() 2024-09-15T22:17:45.116 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:619: TEST_dump_scrub_schedule: expct_scrub_peri_sched=(['query_is_future']='false') 2024-09-15T22:17:45.117 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:619: TEST_dump_scrub_schedule: declare -A expct_scrub_peri_sched 2024-09-15T22:17:45.117 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:620: TEST_dump_scrub_schedule: wait_any_cond 1.0 10 2024-09-15T22:17:33.277327+0000 expct_scrub_peri_sched waitingBeingScheduled sched_data 2024-09-15T22:17:45.117 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:106: wait_any_cond: local pgid=1.0 2024-09-15T22:17:45.117 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:107: wait_any_cond: local retries=10 2024-09-15T22:17:45.117 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:108: wait_any_cond: local cmp_date=2024-09-15T22:17:33.277327+0000 2024-09-15T22:17:45.117 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:109: wait_any_cond: local -n ep=expct_scrub_peri_sched 2024-09-15T22:17:45.117 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:110: wait_any_cond: local -n out_array=sched_data 2024-09-15T22:17:45.117 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:111: wait_any_cond: local -A sc_data 2024-09-15T22:17:45.117 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:112: wait_any_cond: local extr_dbg=2 2024-09-15T22:17:45.117 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:115: wait_any_cond: local saved_echo_flag=x 2024-09-15T22:17:45.117 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:116: wait_any_cond: set +x 2024-09-15T22:17:45.118 INFO:tasks.workunit.client.0.smithi136.stdout:waiting for any condition (waitingBeingScheduled): pg:1.0 dt:2024-09-15T22:17:33.277327+0000 (10 retries) 2024-09-15T22:17:45.886 INFO:tasks.workunit.client.0.smithi136.stderr:dumped pgs 2024-09-15T22:17:45.898 INFO:tasks.workunit.client.0.smithi136.stdout:from pg dump pg: { 2024-09-15T22:17:45.898 INFO:tasks.workunit.client.0.smithi136.stdout: "dmp_pg_state": "active+clean", 2024-09-15T22:17:45.898 INFO:tasks.workunit.client.0.smithi136.stdout: "dmp_state_has_scrubbing": false, 2024-09-15T22:17:45.898 INFO:tasks.workunit.client.0.smithi136.stdout: "dmp_last_duration": 1, 2024-09-15T22:17:45.898 INFO:tasks.workunit.client.0.smithi136.stdout: "dmp_schedule": "periodic scrub scheduled", 2024-09-15T22:17:45.898 INFO:tasks.workunit.client.0.smithi136.stdout: "dmp_schedule_at": "2024-09-16T22:17:33.277327+0000", 2024-09-15T22:17:45.898 INFO:tasks.workunit.client.0.smithi136.stdout: "dmp_is_future": true, 2024-09-15T22:17:45.898 INFO:tasks.workunit.client.0.smithi136.stdout: "dmp_vs_date": true, 2024-09-15T22:17:45.899 INFO:tasks.workunit.client.0.smithi136.stdout: "dmp_reported_epoch": 19, 2024-09-15T22:17:45.899 INFO:tasks.workunit.client.0.smithi136.stdout: "dmp_seq": 42 2024-09-15T22:17:45.899 INFO:tasks.workunit.client.0.smithi136.stdout:} 2024-09-15T22:17:45.899 INFO:tasks.workunit.client.0.smithi136.stdout:query output: 2024-09-15T22:17:46.001 INFO:tasks.workunit.client.0.smithi136.stdout: "scrubber": { 2024-09-15T22:17:46.001 INFO:tasks.workunit.client.0.smithi136.stdout: "active": true, 2024-09-15T22:17:46.001 INFO:tasks.workunit.client.0.smithi136.stdout: "epoch_start": "17", 2024-09-15T22:17:46.001 INFO:tasks.workunit.client.0.smithi136.stdout: "start": "MIN", 2024-09-15T22:17:46.001 INFO:tasks.workunit.client.0.smithi136.stdout: "end": "MIN", 2024-09-15T22:17:46.001 INFO:tasks.workunit.client.0.smithi136.stdout: "max_end": "MIN", 2024-09-15T22:17:46.001 INFO:tasks.workunit.client.0.smithi136.stdout: "subset_last_update": "0'0", 2024-09-15T22:17:46.001 INFO:tasks.workunit.client.0.smithi136.stdout: "deep": true, 2024-09-15T22:17:46.001 INFO:tasks.workunit.client.0.smithi136.stdout: "req_scrub": false, 2024-09-15T22:17:46.001 INFO:tasks.workunit.client.0.smithi136.stdout: "auto_repair": false, 2024-09-15T22:17:46.001 INFO:tasks.workunit.client.0.smithi136.stdout: "check_repair": false, 2024-09-15T22:17:46.001 INFO:tasks.workunit.client.0.smithi136.stdout: "deep_scrub_on_error": false, 2024-09-15T22:17:46.002 INFO:tasks.workunit.client.0.smithi136.stdout: "priority": 0, 2024-09-15T22:17:46.002 INFO:tasks.workunit.client.0.smithi136.stdout: "shallow_errors": 0, 2024-09-15T22:17:46.002 INFO:tasks.workunit.client.0.smithi136.stdout: "deep_errors": 0, 2024-09-15T22:17:46.002 INFO:tasks.workunit.client.0.smithi136.stdout: "fixed": 0, 2024-09-15T22:17:46.002 INFO:tasks.workunit.client.0.smithi136.stdout: "waiting_on_whom": [], 2024-09-15T22:17:46.002 INFO:tasks.workunit.client.0.smithi136.stdout: "schedule": "scrubbing" 2024-09-15T22:17:46.002 INFO:tasks.workunit.client.0.smithi136.stdout: }, 2024-09-15T22:17:46.002 INFO:tasks.workunit.client.0.smithi136.stdout: "agent_state": {} 2024-09-15T22:17:46.107 INFO:tasks.workunit.client.0.smithi136.stdout:( 2024-09-15T22:17:46.107 INFO:tasks.workunit.client.0.smithi136.stdout:[query_epoch]=20 2024-09-15T22:17:46.107 INFO:tasks.workunit.client.0.smithi136.stdout:[query_seq]=47 2024-09-15T22:17:46.107 INFO:tasks.workunit.client.0.smithi136.stdout:[query_active]=true 2024-09-15T22:17:46.107 INFO:tasks.workunit.client.0.smithi136.stdout:[query_schedule]='scrubbing' 2024-09-15T22:17:46.107 INFO:tasks.workunit.client.0.smithi136.stdout:[query_schedule_at]='0' 2024-09-15T22:17:46.107 INFO:tasks.workunit.client.0.smithi136.stdout:[query_target_at]='0' 2024-09-15T22:17:46.107 INFO:tasks.workunit.client.0.smithi136.stdout:[query_last_duration]=1 2024-09-15T22:17:46.108 INFO:tasks.workunit.client.0.smithi136.stdout:[query_last_stamp]='2024-09-08T22:16:04.103344+0000' 2024-09-15T22:17:46.108 INFO:tasks.workunit.client.0.smithi136.stdout:[query_last_scrub]='19x15' 2024-09-15T22:17:46.108 INFO:tasks.workunit.client.0.smithi136.stdout:[query_is_future]=false 2024-09-15T22:17:46.108 INFO:tasks.workunit.client.0.smithi136.stdout:[query_vs_date]=false 2024-09-15T22:17:46.108 INFO:tasks.workunit.client.0.smithi136.stdout:[query_scrub_seq]=null 2024-09-15T22:17:46.108 INFO:tasks.workunit.client.0.smithi136.stdout:[dmp_pg_state]='active+clean' 2024-09-15T22:17:46.108 INFO:tasks.workunit.client.0.smithi136.stdout:[dmp_state_has_scrubbing]=false 2024-09-15T22:17:46.108 INFO:tasks.workunit.client.0.smithi136.stdout:[dmp_last_duration]=1 2024-09-15T22:17:46.108 INFO:tasks.workunit.client.0.smithi136.stdout:[dmp_schedule]='periodic scrub scheduled' 2024-09-15T22:17:46.108 INFO:tasks.workunit.client.0.smithi136.stdout:[dmp_schedule_at]='2024-09-16T22:17:33.277327+0000' 2024-09-15T22:17:46.108 INFO:tasks.workunit.client.0.smithi136.stdout:[dmp_is_future]=true 2024-09-15T22:17:46.108 INFO:tasks.workunit.client.0.smithi136.stdout:[dmp_vs_date]=true 2024-09-15T22:17:46.108 INFO:tasks.workunit.client.0.smithi136.stdout:[dmp_reported_epoch]=19 2024-09-15T22:17:46.108 INFO:tasks.workunit.client.0.smithi136.stdout:[dmp_seq]=42 2024-09-15T22:17:46.109 INFO:tasks.workunit.client.0.smithi136.stdout:) 2024-09-15T22:17:46.110 INFO:tasks.workunit.client.0.smithi136.stdout:--> loop: 1 ~ true / 47 / 42 / false / 2024-09-08T22:16:04.103344+0000 / scrubbing %%% query_is_future 2024-09-15T22:17:46.110 INFO:tasks.workunit.client.0.smithi136.stdout:key is query_is_future: negation:0 # expected: false # in actual: false 2024-09-15T22:17:46.110 INFO:tasks.workunit.client.0.smithi136.stdout:waitingBeingScheduled - 'query_is_future' actual value (false) matches expected (false) (negation: 0) 2024-09-15T22:17:46.111 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:157: wait_any_cond: return 0 2024-09-15T22:17:46.111 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:632: TEST_dump_scrub_schedule: saved_last_stamp=2024-09-08T22:16:04.103344+0000 2024-09-15T22:17:46.111 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:633: TEST_dump_scrub_schedule: ceph osd unset noscrub 2024-09-15T22:17:46.489 INFO:tasks.workunit.client.0.smithi136.stderr:noscrub is unset 2024-09-15T22:17:46.502 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:635: TEST_dump_scrub_schedule: cond_active=(['query_active']='true') 2024-09-15T22:17:46.502 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:635: TEST_dump_scrub_schedule: declare -A cond_active 2024-09-15T22:17:46.502 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:636: TEST_dump_scrub_schedule: sched_data=() 2024-09-15T22:17:46.502 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:637: TEST_dump_scrub_schedule: wait_any_cond 1.0 10 2024-09-08T22:16:04.103344+0000 cond_active 'WaitingActive ' sched_data 2024-09-15T22:17:46.502 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:106: wait_any_cond: local pgid=1.0 2024-09-15T22:17:46.502 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:107: wait_any_cond: local retries=10 2024-09-15T22:17:46.502 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:108: wait_any_cond: local cmp_date=2024-09-08T22:16:04.103344+0000 2024-09-15T22:17:46.502 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:109: wait_any_cond: local -n ep=cond_active 2024-09-15T22:17:46.502 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:110: wait_any_cond: local -n out_array=sched_data 2024-09-15T22:17:46.502 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:111: wait_any_cond: local -A sc_data 2024-09-15T22:17:46.502 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:112: wait_any_cond: local extr_dbg=2 2024-09-15T22:17:46.503 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:115: wait_any_cond: local saved_echo_flag=x 2024-09-15T22:17:46.503 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:116: wait_any_cond: set +x 2024-09-15T22:17:46.503 INFO:tasks.workunit.client.0.smithi136.stdout:waiting for any condition (WaitingActive ): pg:1.0 dt:2024-09-08T22:16:04.103344+0000 (10 retries) 2024-09-15T22:17:47.265 INFO:tasks.workunit.client.0.smithi136.stderr:dumped pgs 2024-09-15T22:17:47.277 INFO:tasks.workunit.client.0.smithi136.stdout:from pg dump pg: { 2024-09-15T22:17:47.277 INFO:tasks.workunit.client.0.smithi136.stdout: "dmp_pg_state": "active+clean", 2024-09-15T22:17:47.277 INFO:tasks.workunit.client.0.smithi136.stdout: "dmp_state_has_scrubbing": false, 2024-09-15T22:17:47.277 INFO:tasks.workunit.client.0.smithi136.stdout: "dmp_last_duration": 1, 2024-09-15T22:17:47.277 INFO:tasks.workunit.client.0.smithi136.stdout: "dmp_schedule": "periodic scrub scheduled", 2024-09-15T22:17:47.277 INFO:tasks.workunit.client.0.smithi136.stdout: "dmp_schedule_at": "2024-09-16T22:17:33.277327+0000", 2024-09-15T22:17:47.277 INFO:tasks.workunit.client.0.smithi136.stdout: "dmp_is_future": true, 2024-09-15T22:17:47.277 INFO:tasks.workunit.client.0.smithi136.stdout: "dmp_vs_date": true, 2024-09-15T22:17:47.277 INFO:tasks.workunit.client.0.smithi136.stdout: "dmp_reported_epoch": 19, 2024-09-15T22:17:47.277 INFO:tasks.workunit.client.0.smithi136.stdout: "dmp_seq": 42 2024-09-15T22:17:47.278 INFO:tasks.workunit.client.0.smithi136.stdout:} 2024-09-15T22:17:47.278 INFO:tasks.workunit.client.0.smithi136.stdout:query output: 2024-09-15T22:17:47.381 INFO:tasks.workunit.client.0.smithi136.stdout: "scrubber": { 2024-09-15T22:17:47.381 INFO:tasks.workunit.client.0.smithi136.stdout: "active": true, 2024-09-15T22:17:47.381 INFO:tasks.workunit.client.0.smithi136.stdout: "epoch_start": "17", 2024-09-15T22:17:47.381 INFO:tasks.workunit.client.0.smithi136.stdout: "start": "MIN", 2024-09-15T22:17:47.381 INFO:tasks.workunit.client.0.smithi136.stdout: "end": "MIN", 2024-09-15T22:17:47.381 INFO:tasks.workunit.client.0.smithi136.stdout: "max_end": "MIN", 2024-09-15T22:17:47.381 INFO:tasks.workunit.client.0.smithi136.stdout: "subset_last_update": "0'0", 2024-09-15T22:17:47.381 INFO:tasks.workunit.client.0.smithi136.stdout: "deep": true, 2024-09-15T22:17:47.381 INFO:tasks.workunit.client.0.smithi136.stdout: "req_scrub": false, 2024-09-15T22:17:47.381 INFO:tasks.workunit.client.0.smithi136.stdout: "auto_repair": false, 2024-09-15T22:17:47.381 INFO:tasks.workunit.client.0.smithi136.stdout: "check_repair": false, 2024-09-15T22:17:47.382 INFO:tasks.workunit.client.0.smithi136.stdout: "deep_scrub_on_error": false, 2024-09-15T22:17:47.382 INFO:tasks.workunit.client.0.smithi136.stdout: "priority": 0, 2024-09-15T22:17:47.382 INFO:tasks.workunit.client.0.smithi136.stdout: "shallow_errors": 0, 2024-09-15T22:17:47.382 INFO:tasks.workunit.client.0.smithi136.stdout: "deep_errors": 0, 2024-09-15T22:17:47.382 INFO:tasks.workunit.client.0.smithi136.stdout: "fixed": 0, 2024-09-15T22:17:47.382 INFO:tasks.workunit.client.0.smithi136.stdout: "waiting_on_whom": [], 2024-09-15T22:17:47.382 INFO:tasks.workunit.client.0.smithi136.stdout: "schedule": "scrubbing" 2024-09-15T22:17:47.382 INFO:tasks.workunit.client.0.smithi136.stdout: }, 2024-09-15T22:17:47.382 INFO:tasks.workunit.client.0.smithi136.stdout: "agent_state": {} 2024-09-15T22:17:47.487 INFO:tasks.workunit.client.0.smithi136.stdout:( 2024-09-15T22:17:47.487 INFO:tasks.workunit.client.0.smithi136.stdout:[query_epoch]=20 2024-09-15T22:17:47.487 INFO:tasks.workunit.client.0.smithi136.stdout:[query_seq]=48 2024-09-15T22:17:47.487 INFO:tasks.workunit.client.0.smithi136.stdout:[query_active]=true 2024-09-15T22:17:47.487 INFO:tasks.workunit.client.0.smithi136.stdout:[query_schedule]='scrubbing' 2024-09-15T22:17:47.487 INFO:tasks.workunit.client.0.smithi136.stdout:[query_schedule_at]='0' 2024-09-15T22:17:47.488 INFO:tasks.workunit.client.0.smithi136.stdout:[query_target_at]='0' 2024-09-15T22:17:47.488 INFO:tasks.workunit.client.0.smithi136.stdout:[query_last_duration]=1 2024-09-15T22:17:47.488 INFO:tasks.workunit.client.0.smithi136.stdout:[query_last_stamp]='2024-09-08T22:16:04.103344+0000' 2024-09-15T22:17:47.488 INFO:tasks.workunit.client.0.smithi136.stdout:[query_last_scrub]='19x15' 2024-09-15T22:17:47.488 INFO:tasks.workunit.client.0.smithi136.stdout:[query_is_future]=false 2024-09-15T22:17:47.488 INFO:tasks.workunit.client.0.smithi136.stdout:[query_vs_date]=false 2024-09-15T22:17:47.488 INFO:tasks.workunit.client.0.smithi136.stdout:[query_scrub_seq]=null 2024-09-15T22:17:47.488 INFO:tasks.workunit.client.0.smithi136.stdout:[dmp_pg_state]='active+clean' 2024-09-15T22:17:47.488 INFO:tasks.workunit.client.0.smithi136.stdout:[dmp_state_has_scrubbing]=false 2024-09-15T22:17:47.488 INFO:tasks.workunit.client.0.smithi136.stdout:[dmp_last_duration]=1 2024-09-15T22:17:47.488 INFO:tasks.workunit.client.0.smithi136.stdout:[dmp_schedule]='periodic scrub scheduled' 2024-09-15T22:17:47.488 INFO:tasks.workunit.client.0.smithi136.stdout:[dmp_schedule_at]='2024-09-16T22:17:33.277327+0000' 2024-09-15T22:17:47.488 INFO:tasks.workunit.client.0.smithi136.stdout:[dmp_is_future]=true 2024-09-15T22:17:47.488 INFO:tasks.workunit.client.0.smithi136.stdout:[dmp_vs_date]=true 2024-09-15T22:17:47.489 INFO:tasks.workunit.client.0.smithi136.stdout:[dmp_reported_epoch]=19 2024-09-15T22:17:47.489 INFO:tasks.workunit.client.0.smithi136.stdout:[dmp_seq]=42 2024-09-15T22:17:47.489 INFO:tasks.workunit.client.0.smithi136.stdout:) 2024-09-15T22:17:47.490 INFO:tasks.workunit.client.0.smithi136.stdout:--> loop: 1 ~ true / 48 / 42 / false / 2024-09-08T22:16:04.103344+0000 / scrubbing %%% query_active 2024-09-15T22:17:47.490 INFO:tasks.workunit.client.0.smithi136.stdout:key is query_active: negation:0 # expected: true # in actual: true 2024-09-15T22:17:47.490 INFO:tasks.workunit.client.0.smithi136.stdout:WaitingActive - 'query_active' actual value (true) matches expected (true) (negation: 0) 2024-09-15T22:17:47.492 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:157: wait_any_cond: return 0 2024-09-15T22:17:47.492 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:641: TEST_dump_scrub_schedule: cond_active_dmp=(['dmp_state_has_scrubbing']='true' ['query_active']='false') 2024-09-15T22:17:47.492 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:641: TEST_dump_scrub_schedule: declare -A cond_active_dmp 2024-09-15T22:17:47.492 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:642: TEST_dump_scrub_schedule: sched_data=() 2024-09-15T22:17:47.492 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:643: TEST_dump_scrub_schedule: wait_any_cond 1.0 10 2024-09-08T22:16:04.103344+0000 cond_active_dmp 'WaitingActive ' sched_data 2024-09-15T22:17:47.492 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:106: wait_any_cond: local pgid=1.0 2024-09-15T22:17:47.492 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:107: wait_any_cond: local retries=10 2024-09-15T22:17:47.492 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:108: wait_any_cond: local cmp_date=2024-09-08T22:16:04.103344+0000 2024-09-15T22:17:47.492 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:109: wait_any_cond: local -n ep=cond_active_dmp 2024-09-15T22:17:47.492 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:110: wait_any_cond: local -n out_array=sched_data 2024-09-15T22:17:47.492 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:111: wait_any_cond: local -A sc_data 2024-09-15T22:17:47.493 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:112: wait_any_cond: local extr_dbg=2 2024-09-15T22:17:47.493 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:115: wait_any_cond: local saved_echo_flag=x 2024-09-15T22:17:47.493 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:116: wait_any_cond: set +x 2024-09-15T22:17:47.493 INFO:tasks.workunit.client.0.smithi136.stdout:waiting for any condition (WaitingActive ): pg:1.0 dt:2024-09-08T22:16:04.103344+0000 (10 retries) 2024-09-15T22:17:48.258 INFO:tasks.workunit.client.0.smithi136.stderr:dumped pgs 2024-09-15T22:17:48.270 INFO:tasks.workunit.client.0.smithi136.stdout:from pg dump pg: { 2024-09-15T22:17:48.270 INFO:tasks.workunit.client.0.smithi136.stdout: "dmp_pg_state": "active+clean", 2024-09-15T22:17:48.270 INFO:tasks.workunit.client.0.smithi136.stdout: "dmp_state_has_scrubbing": false, 2024-09-15T22:17:48.270 INFO:tasks.workunit.client.0.smithi136.stdout: "dmp_last_duration": 1, 2024-09-15T22:17:48.270 INFO:tasks.workunit.client.0.smithi136.stdout: "dmp_schedule": "periodic scrub scheduled", 2024-09-15T22:17:48.271 INFO:tasks.workunit.client.0.smithi136.stdout: "dmp_schedule_at": "2024-09-16T22:17:33.277327+0000", 2024-09-15T22:17:48.271 INFO:tasks.workunit.client.0.smithi136.stdout: "dmp_is_future": true, 2024-09-15T22:17:48.271 INFO:tasks.workunit.client.0.smithi136.stdout: "dmp_vs_date": true, 2024-09-15T22:17:48.271 INFO:tasks.workunit.client.0.smithi136.stdout: "dmp_reported_epoch": 19, 2024-09-15T22:17:48.271 INFO:tasks.workunit.client.0.smithi136.stdout: "dmp_seq": 42 2024-09-15T22:17:48.271 INFO:tasks.workunit.client.0.smithi136.stdout:} 2024-09-15T22:17:48.271 INFO:tasks.workunit.client.0.smithi136.stdout:query output: 2024-09-15T22:17:48.374 INFO:tasks.workunit.client.0.smithi136.stdout: "scrubber": { 2024-09-15T22:17:48.374 INFO:tasks.workunit.client.0.smithi136.stdout: "active": true, 2024-09-15T22:17:48.374 INFO:tasks.workunit.client.0.smithi136.stdout: "epoch_start": "17", 2024-09-15T22:17:48.374 INFO:tasks.workunit.client.0.smithi136.stdout: "start": "MIN", 2024-09-15T22:17:48.374 INFO:tasks.workunit.client.0.smithi136.stdout: "end": "MIN", 2024-09-15T22:17:48.374 INFO:tasks.workunit.client.0.smithi136.stdout: "max_end": "MIN", 2024-09-15T22:17:48.375 INFO:tasks.workunit.client.0.smithi136.stdout: "subset_last_update": "0'0", 2024-09-15T22:17:48.375 INFO:tasks.workunit.client.0.smithi136.stdout: "deep": true, 2024-09-15T22:17:48.375 INFO:tasks.workunit.client.0.smithi136.stdout: "req_scrub": false, 2024-09-15T22:17:48.375 INFO:tasks.workunit.client.0.smithi136.stdout: "auto_repair": false, 2024-09-15T22:17:48.375 INFO:tasks.workunit.client.0.smithi136.stdout: "check_repair": false, 2024-09-15T22:17:48.375 INFO:tasks.workunit.client.0.smithi136.stdout: "deep_scrub_on_error": false, 2024-09-15T22:17:48.375 INFO:tasks.workunit.client.0.smithi136.stdout: "priority": 0, 2024-09-15T22:17:48.375 INFO:tasks.workunit.client.0.smithi136.stdout: "shallow_errors": 0, 2024-09-15T22:17:48.375 INFO:tasks.workunit.client.0.smithi136.stdout: "deep_errors": 0, 2024-09-15T22:17:48.375 INFO:tasks.workunit.client.0.smithi136.stdout: "fixed": 0, 2024-09-15T22:17:48.375 INFO:tasks.workunit.client.0.smithi136.stdout: "waiting_on_whom": [], 2024-09-15T22:17:48.375 INFO:tasks.workunit.client.0.smithi136.stdout: "schedule": "scrubbing" 2024-09-15T22:17:48.375 INFO:tasks.workunit.client.0.smithi136.stdout: }, 2024-09-15T22:17:48.375 INFO:tasks.workunit.client.0.smithi136.stdout: "agent_state": {} 2024-09-15T22:17:48.480 INFO:tasks.workunit.client.0.smithi136.stdout:( 2024-09-15T22:17:48.480 INFO:tasks.workunit.client.0.smithi136.stdout:[query_epoch]=21 2024-09-15T22:17:48.481 INFO:tasks.workunit.client.0.smithi136.stdout:[query_seq]=50 2024-09-15T22:17:48.481 INFO:tasks.workunit.client.0.smithi136.stdout:[query_active]=true 2024-09-15T22:17:48.481 INFO:tasks.workunit.client.0.smithi136.stdout:[query_schedule]='scrubbing' 2024-09-15T22:17:48.481 INFO:tasks.workunit.client.0.smithi136.stdout:[query_schedule_at]='0' 2024-09-15T22:17:48.481 INFO:tasks.workunit.client.0.smithi136.stdout:[query_target_at]='0' 2024-09-15T22:17:48.481 INFO:tasks.workunit.client.0.smithi136.stdout:[query_last_duration]=1 2024-09-15T22:17:48.481 INFO:tasks.workunit.client.0.smithi136.stdout:[query_last_stamp]='2024-09-08T22:16:04.103344+0000' 2024-09-15T22:17:48.481 INFO:tasks.workunit.client.0.smithi136.stdout:[query_last_scrub]='19x15' 2024-09-15T22:17:48.481 INFO:tasks.workunit.client.0.smithi136.stdout:[query_is_future]=false 2024-09-15T22:17:48.481 INFO:tasks.workunit.client.0.smithi136.stdout:[query_vs_date]=false 2024-09-15T22:17:48.481 INFO:tasks.workunit.client.0.smithi136.stdout:[query_scrub_seq]=null 2024-09-15T22:17:48.481 INFO:tasks.workunit.client.0.smithi136.stdout:[dmp_pg_state]='active+clean' 2024-09-15T22:17:48.481 INFO:tasks.workunit.client.0.smithi136.stdout:[dmp_state_has_scrubbing]=false 2024-09-15T22:17:48.481 INFO:tasks.workunit.client.0.smithi136.stdout:[dmp_last_duration]=1 2024-09-15T22:17:48.482 INFO:tasks.workunit.client.0.smithi136.stdout:[dmp_schedule]='periodic scrub scheduled' 2024-09-15T22:17:48.482 INFO:tasks.workunit.client.0.smithi136.stdout:[dmp_schedule_at]='2024-09-16T22:17:33.277327+0000' 2024-09-15T22:17:48.482 INFO:tasks.workunit.client.0.smithi136.stdout:[dmp_is_future]=true 2024-09-15T22:17:48.482 INFO:tasks.workunit.client.0.smithi136.stdout:[dmp_vs_date]=true 2024-09-15T22:17:48.482 INFO:tasks.workunit.client.0.smithi136.stdout:[dmp_reported_epoch]=19 2024-09-15T22:17:48.482 INFO:tasks.workunit.client.0.smithi136.stdout:[dmp_seq]=42 2024-09-15T22:17:48.482 INFO:tasks.workunit.client.0.smithi136.stdout:) 2024-09-15T22:17:48.484 INFO:tasks.workunit.client.0.smithi136.stdout:--> loop: 1 ~ true / 50 / 42 / false / 2024-09-08T22:16:04.103344+0000 / scrubbing %%% dmp_state_has_scrubbing query_active 2024-09-15T22:17:48.484 INFO:tasks.workunit.client.0.smithi136.stdout:key is dmp_state_has_scrubbing: negation:0 # expected: true # in actual: false 2024-09-15T22:17:48.484 INFO:tasks.workunit.client.0.smithi136.stdout:key is query_active: negation:0 # expected: false # in actual: true 2024-09-15T22:17:49.243 INFO:tasks.workunit.client.0.smithi136.stderr:dumped pgs 2024-09-15T22:17:49.255 INFO:tasks.workunit.client.0.smithi136.stdout:from pg dump pg: { 2024-09-15T22:17:49.255 INFO:tasks.workunit.client.0.smithi136.stdout: "dmp_pg_state": "active+clean", 2024-09-15T22:17:49.255 INFO:tasks.workunit.client.0.smithi136.stdout: "dmp_state_has_scrubbing": false, 2024-09-15T22:17:49.255 INFO:tasks.workunit.client.0.smithi136.stdout: "dmp_last_duration": 1, 2024-09-15T22:17:49.255 INFO:tasks.workunit.client.0.smithi136.stdout: "dmp_schedule": "periodic scrub scheduled", 2024-09-15T22:17:49.255 INFO:tasks.workunit.client.0.smithi136.stdout: "dmp_schedule_at": "2024-09-15T22:18:44.290846+0000", 2024-09-15T22:17:49.256 INFO:tasks.workunit.client.0.smithi136.stdout: "dmp_is_future": true, 2024-09-15T22:17:49.256 INFO:tasks.workunit.client.0.smithi136.stdout: "dmp_vs_date": true, 2024-09-15T22:17:49.256 INFO:tasks.workunit.client.0.smithi136.stdout: "dmp_reported_epoch": 20, 2024-09-15T22:17:49.256 INFO:tasks.workunit.client.0.smithi136.stdout: "dmp_seq": 47 2024-09-15T22:17:49.256 INFO:tasks.workunit.client.0.smithi136.stdout:} 2024-09-15T22:17:49.256 INFO:tasks.workunit.client.0.smithi136.stdout:query output: 2024-09-15T22:17:49.358 INFO:tasks.workunit.client.0.smithi136.stdout: "scrubber": { 2024-09-15T22:17:49.358 INFO:tasks.workunit.client.0.smithi136.stdout: "active": true, 2024-09-15T22:17:49.358 INFO:tasks.workunit.client.0.smithi136.stdout: "epoch_start": "17", 2024-09-15T22:17:49.358 INFO:tasks.workunit.client.0.smithi136.stdout: "start": "MIN", 2024-09-15T22:17:49.359 INFO:tasks.workunit.client.0.smithi136.stdout: "end": "MIN", 2024-09-15T22:17:49.359 INFO:tasks.workunit.client.0.smithi136.stdout: "max_end": "MIN", 2024-09-15T22:17:49.359 INFO:tasks.workunit.client.0.smithi136.stdout: "subset_last_update": "0'0", 2024-09-15T22:17:49.359 INFO:tasks.workunit.client.0.smithi136.stdout: "deep": true, 2024-09-15T22:17:49.359 INFO:tasks.workunit.client.0.smithi136.stdout: "req_scrub": false, 2024-09-15T22:17:49.359 INFO:tasks.workunit.client.0.smithi136.stdout: "auto_repair": false, 2024-09-15T22:17:49.359 INFO:tasks.workunit.client.0.smithi136.stdout: "check_repair": false, 2024-09-15T22:17:49.359 INFO:tasks.workunit.client.0.smithi136.stdout: "deep_scrub_on_error": false, 2024-09-15T22:17:49.359 INFO:tasks.workunit.client.0.smithi136.stdout: "priority": 0, 2024-09-15T22:17:49.359 INFO:tasks.workunit.client.0.smithi136.stdout: "shallow_errors": 0, 2024-09-15T22:17:49.359 INFO:tasks.workunit.client.0.smithi136.stdout: "deep_errors": 0, 2024-09-15T22:17:49.359 INFO:tasks.workunit.client.0.smithi136.stdout: "fixed": 0, 2024-09-15T22:17:49.359 INFO:tasks.workunit.client.0.smithi136.stdout: "waiting_on_whom": [], 2024-09-15T22:17:49.359 INFO:tasks.workunit.client.0.smithi136.stdout: "schedule": "scrubbing" 2024-09-15T22:17:49.359 INFO:tasks.workunit.client.0.smithi136.stdout: }, 2024-09-15T22:17:49.360 INFO:tasks.workunit.client.0.smithi136.stdout: "agent_state": {} 2024-09-15T22:17:49.464 INFO:tasks.workunit.client.0.smithi136.stdout:( 2024-09-15T22:17:49.464 INFO:tasks.workunit.client.0.smithi136.stdout:[query_epoch]=21 2024-09-15T22:17:49.464 INFO:tasks.workunit.client.0.smithi136.stdout:[query_seq]=50 2024-09-15T22:17:49.464 INFO:tasks.workunit.client.0.smithi136.stdout:[query_active]=true 2024-09-15T22:17:49.465 INFO:tasks.workunit.client.0.smithi136.stdout:[query_schedule]='scrubbing' 2024-09-15T22:17:49.465 INFO:tasks.workunit.client.0.smithi136.stdout:[query_schedule_at]='0' 2024-09-15T22:17:49.465 INFO:tasks.workunit.client.0.smithi136.stdout:[query_target_at]='0' 2024-09-15T22:17:49.465 INFO:tasks.workunit.client.0.smithi136.stdout:[query_last_duration]=1 2024-09-15T22:17:49.465 INFO:tasks.workunit.client.0.smithi136.stdout:[query_last_stamp]='2024-09-08T22:16:04.103344+0000' 2024-09-15T22:17:49.465 INFO:tasks.workunit.client.0.smithi136.stdout:[query_last_scrub]='19x15' 2024-09-15T22:17:49.465 INFO:tasks.workunit.client.0.smithi136.stdout:[query_is_future]=false 2024-09-15T22:17:49.465 INFO:tasks.workunit.client.0.smithi136.stdout:[query_vs_date]=false 2024-09-15T22:17:49.465 INFO:tasks.workunit.client.0.smithi136.stdout:[query_scrub_seq]=null 2024-09-15T22:17:49.465 INFO:tasks.workunit.client.0.smithi136.stdout:[dmp_pg_state]='active+clean' 2024-09-15T22:17:49.465 INFO:tasks.workunit.client.0.smithi136.stdout:[dmp_state_has_scrubbing]=false 2024-09-15T22:17:49.465 INFO:tasks.workunit.client.0.smithi136.stdout:[dmp_last_duration]=1 2024-09-15T22:17:49.465 INFO:tasks.workunit.client.0.smithi136.stdout:[dmp_schedule]='periodic scrub scheduled' 2024-09-15T22:17:49.465 INFO:tasks.workunit.client.0.smithi136.stdout:[dmp_schedule_at]='2024-09-15T22:18:44.290846+0000' 2024-09-15T22:17:49.466 INFO:tasks.workunit.client.0.smithi136.stdout:[dmp_is_future]=true 2024-09-15T22:17:49.466 INFO:tasks.workunit.client.0.smithi136.stdout:[dmp_vs_date]=true 2024-09-15T22:17:49.466 INFO:tasks.workunit.client.0.smithi136.stdout:[dmp_reported_epoch]=20 2024-09-15T22:17:49.466 INFO:tasks.workunit.client.0.smithi136.stdout:[dmp_seq]=47 2024-09-15T22:17:49.466 INFO:tasks.workunit.client.0.smithi136.stdout:) 2024-09-15T22:17:49.468 INFO:tasks.workunit.client.0.smithi136.stdout:--> loop: 2 ~ true / 50 / 47 / false / 2024-09-08T22:16:04.103344+0000 / scrubbing %%% dmp_state_has_scrubbing query_active 2024-09-15T22:17:49.468 INFO:tasks.workunit.client.0.smithi136.stdout:key is dmp_state_has_scrubbing: negation:0 # expected: true # in actual: false 2024-09-15T22:17:49.468 INFO:tasks.workunit.client.0.smithi136.stdout:key is query_active: negation:0 # expected: false # in actual: true 2024-09-15T22:17:50.232 INFO:tasks.workunit.client.0.smithi136.stderr:dumped pgs 2024-09-15T22:17:50.244 INFO:tasks.workunit.client.0.smithi136.stdout:from pg dump pg: { 2024-09-15T22:17:50.244 INFO:tasks.workunit.client.0.smithi136.stdout: "dmp_pg_state": "active+clean", 2024-09-15T22:17:50.244 INFO:tasks.workunit.client.0.smithi136.stdout: "dmp_state_has_scrubbing": false, 2024-09-15T22:17:50.244 INFO:tasks.workunit.client.0.smithi136.stdout: "dmp_last_duration": 1, 2024-09-15T22:17:50.244 INFO:tasks.workunit.client.0.smithi136.stdout: "dmp_schedule": "periodic scrub scheduled", 2024-09-15T22:17:50.244 INFO:tasks.workunit.client.0.smithi136.stdout: "dmp_schedule_at": "2024-09-15T22:18:44.290846+0000", 2024-09-15T22:17:50.244 INFO:tasks.workunit.client.0.smithi136.stdout: "dmp_is_future": true, 2024-09-15T22:17:50.244 INFO:tasks.workunit.client.0.smithi136.stdout: "dmp_vs_date": true, 2024-09-15T22:17:50.245 INFO:tasks.workunit.client.0.smithi136.stdout: "dmp_reported_epoch": 20, 2024-09-15T22:17:50.245 INFO:tasks.workunit.client.0.smithi136.stdout: "dmp_seq": 47 2024-09-15T22:17:50.245 INFO:tasks.workunit.client.0.smithi136.stdout:} 2024-09-15T22:17:50.245 INFO:tasks.workunit.client.0.smithi136.stdout:query output: 2024-09-15T22:17:50.346 INFO:tasks.workunit.client.0.smithi136.stdout: "scrubber": { 2024-09-15T22:17:50.347 INFO:tasks.workunit.client.0.smithi136.stdout: "active": true, 2024-09-15T22:17:50.347 INFO:tasks.workunit.client.0.smithi136.stdout: "epoch_start": "17", 2024-09-15T22:17:50.347 INFO:tasks.workunit.client.0.smithi136.stdout: "start": "MIN", 2024-09-15T22:17:50.347 INFO:tasks.workunit.client.0.smithi136.stdout: "end": "MIN", 2024-09-15T22:17:50.347 INFO:tasks.workunit.client.0.smithi136.stdout: "max_end": "MIN", 2024-09-15T22:17:50.347 INFO:tasks.workunit.client.0.smithi136.stdout: "subset_last_update": "0'0", 2024-09-15T22:17:50.347 INFO:tasks.workunit.client.0.smithi136.stdout: "deep": true, 2024-09-15T22:17:50.347 INFO:tasks.workunit.client.0.smithi136.stdout: "req_scrub": false, 2024-09-15T22:17:50.347 INFO:tasks.workunit.client.0.smithi136.stdout: "auto_repair": false, 2024-09-15T22:17:50.347 INFO:tasks.workunit.client.0.smithi136.stdout: "check_repair": false, 2024-09-15T22:17:50.347 INFO:tasks.workunit.client.0.smithi136.stdout: "deep_scrub_on_error": false, 2024-09-15T22:17:50.347 INFO:tasks.workunit.client.0.smithi136.stdout: "priority": 0, 2024-09-15T22:17:50.347 INFO:tasks.workunit.client.0.smithi136.stdout: "shallow_errors": 0, 2024-09-15T22:17:50.347 INFO:tasks.workunit.client.0.smithi136.stdout: "deep_errors": 0, 2024-09-15T22:17:50.348 INFO:tasks.workunit.client.0.smithi136.stdout: "fixed": 0, 2024-09-15T22:17:50.348 INFO:tasks.workunit.client.0.smithi136.stdout: "waiting_on_whom": [], 2024-09-15T22:17:50.348 INFO:tasks.workunit.client.0.smithi136.stdout: "schedule": "scrubbing" 2024-09-15T22:17:50.348 INFO:tasks.workunit.client.0.smithi136.stdout: }, 2024-09-15T22:17:50.348 INFO:tasks.workunit.client.0.smithi136.stdout: "agent_state": {} 2024-09-15T22:17:50.454 INFO:tasks.workunit.client.0.smithi136.stdout:( 2024-09-15T22:17:50.454 INFO:tasks.workunit.client.0.smithi136.stdout:[query_epoch]=21 2024-09-15T22:17:50.454 INFO:tasks.workunit.client.0.smithi136.stdout:[query_seq]=50 2024-09-15T22:17:50.454 INFO:tasks.workunit.client.0.smithi136.stdout:[query_active]=true 2024-09-15T22:17:50.454 INFO:tasks.workunit.client.0.smithi136.stdout:[query_schedule]='scrubbing' 2024-09-15T22:17:50.455 INFO:tasks.workunit.client.0.smithi136.stdout:[query_schedule_at]='0' 2024-09-15T22:17:50.455 INFO:tasks.workunit.client.0.smithi136.stdout:[query_target_at]='0' 2024-09-15T22:17:50.455 INFO:tasks.workunit.client.0.smithi136.stdout:[query_last_duration]=1 2024-09-15T22:17:50.455 INFO:tasks.workunit.client.0.smithi136.stdout:[query_last_stamp]='2024-09-08T22:16:04.103344+0000' 2024-09-15T22:17:50.455 INFO:tasks.workunit.client.0.smithi136.stdout:[query_last_scrub]='19x15' 2024-09-15T22:17:50.455 INFO:tasks.workunit.client.0.smithi136.stdout:[query_is_future]=false 2024-09-15T22:17:50.455 INFO:tasks.workunit.client.0.smithi136.stdout:[query_vs_date]=false 2024-09-15T22:17:50.455 INFO:tasks.workunit.client.0.smithi136.stdout:[query_scrub_seq]=null 2024-09-15T22:17:50.455 INFO:tasks.workunit.client.0.smithi136.stdout:[dmp_pg_state]='active+clean' 2024-09-15T22:17:50.455 INFO:tasks.workunit.client.0.smithi136.stdout:[dmp_state_has_scrubbing]=false 2024-09-15T22:17:50.455 INFO:tasks.workunit.client.0.smithi136.stdout:[dmp_last_duration]=1 2024-09-15T22:17:50.455 INFO:tasks.workunit.client.0.smithi136.stdout:[dmp_schedule]='periodic scrub scheduled' 2024-09-15T22:17:50.455 INFO:tasks.workunit.client.0.smithi136.stdout:[dmp_schedule_at]='2024-09-15T22:18:44.290846+0000' 2024-09-15T22:17:50.455 INFO:tasks.workunit.client.0.smithi136.stdout:[dmp_is_future]=true 2024-09-15T22:17:50.455 INFO:tasks.workunit.client.0.smithi136.stdout:[dmp_vs_date]=true 2024-09-15T22:17:50.456 INFO:tasks.workunit.client.0.smithi136.stdout:[dmp_reported_epoch]=20 2024-09-15T22:17:50.456 INFO:tasks.workunit.client.0.smithi136.stdout:[dmp_seq]=47 2024-09-15T22:17:50.456 INFO:tasks.workunit.client.0.smithi136.stdout:) 2024-09-15T22:17:50.458 INFO:tasks.workunit.client.0.smithi136.stdout:--> loop: 3 ~ true / 50 / 47 / false / 2024-09-08T22:16:04.103344+0000 / scrubbing %%% dmp_state_has_scrubbing query_active 2024-09-15T22:17:50.458 INFO:tasks.workunit.client.0.smithi136.stdout:key is dmp_state_has_scrubbing: negation:0 # expected: true # in actual: false 2024-09-15T22:17:50.458 INFO:tasks.workunit.client.0.smithi136.stdout:key is query_active: negation:0 # expected: false # in actual: true 2024-09-15T22:17:51.220 INFO:tasks.workunit.client.0.smithi136.stderr:dumped pgs 2024-09-15T22:17:51.231 INFO:tasks.workunit.client.0.smithi136.stdout:from pg dump pg: { 2024-09-15T22:17:51.231 INFO:tasks.workunit.client.0.smithi136.stdout: "dmp_pg_state": "active+clean", 2024-09-15T22:17:51.231 INFO:tasks.workunit.client.0.smithi136.stdout: "dmp_state_has_scrubbing": false, 2024-09-15T22:17:51.232 INFO:tasks.workunit.client.0.smithi136.stdout: "dmp_last_duration": 1, 2024-09-15T22:17:51.232 INFO:tasks.workunit.client.0.smithi136.stdout: "dmp_schedule": "periodic scrub scheduled", 2024-09-15T22:17:51.232 INFO:tasks.workunit.client.0.smithi136.stdout: "dmp_schedule_at": "2024-09-15T22:18:44.290846+0000", 2024-09-15T22:17:51.232 INFO:tasks.workunit.client.0.smithi136.stdout: "dmp_is_future": true, 2024-09-15T22:17:51.232 INFO:tasks.workunit.client.0.smithi136.stdout: "dmp_vs_date": true, 2024-09-15T22:17:51.232 INFO:tasks.workunit.client.0.smithi136.stdout: "dmp_reported_epoch": 20, 2024-09-15T22:17:51.232 INFO:tasks.workunit.client.0.smithi136.stdout: "dmp_seq": 47 2024-09-15T22:17:51.232 INFO:tasks.workunit.client.0.smithi136.stdout:} 2024-09-15T22:17:51.232 INFO:tasks.workunit.client.0.smithi136.stdout:query output: 2024-09-15T22:17:51.334 INFO:tasks.workunit.client.0.smithi136.stdout: "scrubber": { 2024-09-15T22:17:51.334 INFO:tasks.workunit.client.0.smithi136.stdout: "active": true, 2024-09-15T22:17:51.334 INFO:tasks.workunit.client.0.smithi136.stdout: "epoch_start": "17", 2024-09-15T22:17:51.334 INFO:tasks.workunit.client.0.smithi136.stdout: "start": "MIN", 2024-09-15T22:17:51.334 INFO:tasks.workunit.client.0.smithi136.stdout: "end": "MIN", 2024-09-15T22:17:51.334 INFO:tasks.workunit.client.0.smithi136.stdout: "max_end": "MIN", 2024-09-15T22:17:51.334 INFO:tasks.workunit.client.0.smithi136.stdout: "subset_last_update": "0'0", 2024-09-15T22:17:51.334 INFO:tasks.workunit.client.0.smithi136.stdout: "deep": true, 2024-09-15T22:17:51.335 INFO:tasks.workunit.client.0.smithi136.stdout: "req_scrub": false, 2024-09-15T22:17:51.335 INFO:tasks.workunit.client.0.smithi136.stdout: "auto_repair": false, 2024-09-15T22:17:51.335 INFO:tasks.workunit.client.0.smithi136.stdout: "check_repair": false, 2024-09-15T22:17:51.335 INFO:tasks.workunit.client.0.smithi136.stdout: "deep_scrub_on_error": false, 2024-09-15T22:17:51.335 INFO:tasks.workunit.client.0.smithi136.stdout: "priority": 0, 2024-09-15T22:17:51.335 INFO:tasks.workunit.client.0.smithi136.stdout: "shallow_errors": 0, 2024-09-15T22:17:51.335 INFO:tasks.workunit.client.0.smithi136.stdout: "deep_errors": 0, 2024-09-15T22:17:51.335 INFO:tasks.workunit.client.0.smithi136.stdout: "fixed": 0, 2024-09-15T22:17:51.335 INFO:tasks.workunit.client.0.smithi136.stdout: "waiting_on_whom": [], 2024-09-15T22:17:51.335 INFO:tasks.workunit.client.0.smithi136.stdout: "schedule": "scrubbing" 2024-09-15T22:17:51.335 INFO:tasks.workunit.client.0.smithi136.stdout: }, 2024-09-15T22:17:51.335 INFO:tasks.workunit.client.0.smithi136.stdout: "agent_state": {} 2024-09-15T22:17:51.441 INFO:tasks.workunit.client.0.smithi136.stdout:( 2024-09-15T22:17:51.441 INFO:tasks.workunit.client.0.smithi136.stdout:[query_epoch]=21 2024-09-15T22:17:51.441 INFO:tasks.workunit.client.0.smithi136.stdout:[query_seq]=50 2024-09-15T22:17:51.441 INFO:tasks.workunit.client.0.smithi136.stdout:[query_active]=true 2024-09-15T22:17:51.441 INFO:tasks.workunit.client.0.smithi136.stdout:[query_schedule]='scrubbing' 2024-09-15T22:17:51.441 INFO:tasks.workunit.client.0.smithi136.stdout:[query_schedule_at]='0' 2024-09-15T22:17:51.442 INFO:tasks.workunit.client.0.smithi136.stdout:[query_target_at]='0' 2024-09-15T22:17:51.442 INFO:tasks.workunit.client.0.smithi136.stdout:[query_last_duration]=1 2024-09-15T22:17:51.442 INFO:tasks.workunit.client.0.smithi136.stdout:[query_last_stamp]='2024-09-08T22:16:04.103344+0000' 2024-09-15T22:17:51.442 INFO:tasks.workunit.client.0.smithi136.stdout:[query_last_scrub]='19x15' 2024-09-15T22:17:51.442 INFO:tasks.workunit.client.0.smithi136.stdout:[query_is_future]=false 2024-09-15T22:17:51.442 INFO:tasks.workunit.client.0.smithi136.stdout:[query_vs_date]=false 2024-09-15T22:17:51.442 INFO:tasks.workunit.client.0.smithi136.stdout:[query_scrub_seq]=null 2024-09-15T22:17:51.442 INFO:tasks.workunit.client.0.smithi136.stdout:[dmp_pg_state]='active+clean' 2024-09-15T22:17:51.443 INFO:tasks.workunit.client.0.smithi136.stdout:[dmp_state_has_scrubbing]=false 2024-09-15T22:17:51.443 INFO:tasks.workunit.client.0.smithi136.stdout:[dmp_last_duration]=1 2024-09-15T22:17:51.443 INFO:tasks.workunit.client.0.smithi136.stdout:[dmp_schedule]='periodic scrub scheduled' 2024-09-15T22:17:51.443 INFO:tasks.workunit.client.0.smithi136.stdout:[dmp_schedule_at]='2024-09-15T22:18:44.290846+0000' 2024-09-15T22:17:51.443 INFO:tasks.workunit.client.0.smithi136.stdout:[dmp_is_future]=true 2024-09-15T22:17:51.443 INFO:tasks.workunit.client.0.smithi136.stdout:[dmp_vs_date]=true 2024-09-15T22:17:51.443 INFO:tasks.workunit.client.0.smithi136.stdout:[dmp_reported_epoch]=20 2024-09-15T22:17:51.443 INFO:tasks.workunit.client.0.smithi136.stdout:[dmp_seq]=47 2024-09-15T22:17:51.443 INFO:tasks.workunit.client.0.smithi136.stdout:) 2024-09-15T22:17:51.444 INFO:tasks.workunit.client.0.smithi136.stdout:--> loop: 4 ~ true / 50 / 47 / false / 2024-09-08T22:16:04.103344+0000 / scrubbing %%% dmp_state_has_scrubbing query_active 2024-09-15T22:17:51.444 INFO:tasks.workunit.client.0.smithi136.stdout:key is dmp_state_has_scrubbing: negation:0 # expected: true # in actual: false 2024-09-15T22:17:51.444 INFO:tasks.workunit.client.0.smithi136.stdout:key is query_active: negation:0 # expected: false # in actual: true 2024-09-15T22:17:52.205 INFO:tasks.workunit.client.0.smithi136.stderr:dumped pgs 2024-09-15T22:17:52.216 INFO:tasks.workunit.client.0.smithi136.stdout:from pg dump pg: { 2024-09-15T22:17:52.217 INFO:tasks.workunit.client.0.smithi136.stdout: "dmp_pg_state": "active+clean", 2024-09-15T22:17:52.217 INFO:tasks.workunit.client.0.smithi136.stdout: "dmp_state_has_scrubbing": false, 2024-09-15T22:17:52.217 INFO:tasks.workunit.client.0.smithi136.stdout: "dmp_last_duration": 1, 2024-09-15T22:17:52.217 INFO:tasks.workunit.client.0.smithi136.stdout: "dmp_schedule": "periodic scrub scheduled", 2024-09-15T22:17:52.217 INFO:tasks.workunit.client.0.smithi136.stdout: "dmp_schedule_at": "2024-09-15T22:18:44.290846+0000", 2024-09-15T22:17:52.217 INFO:tasks.workunit.client.0.smithi136.stdout: "dmp_is_future": true, 2024-09-15T22:17:52.217 INFO:tasks.workunit.client.0.smithi136.stdout: "dmp_vs_date": true, 2024-09-15T22:17:52.217 INFO:tasks.workunit.client.0.smithi136.stdout: "dmp_reported_epoch": 20, 2024-09-15T22:17:52.217 INFO:tasks.workunit.client.0.smithi136.stdout: "dmp_seq": 47 2024-09-15T22:17:52.217 INFO:tasks.workunit.client.0.smithi136.stdout:} 2024-09-15T22:17:52.217 INFO:tasks.workunit.client.0.smithi136.stdout:query output: 2024-09-15T22:17:52.319 INFO:tasks.workunit.client.0.smithi136.stdout: "scrubber": { 2024-09-15T22:17:52.319 INFO:tasks.workunit.client.0.smithi136.stdout: "active": true, 2024-09-15T22:17:52.319 INFO:tasks.workunit.client.0.smithi136.stdout: "epoch_start": "17", 2024-09-15T22:17:52.319 INFO:tasks.workunit.client.0.smithi136.stdout: "start": "MIN", 2024-09-15T22:17:52.319 INFO:tasks.workunit.client.0.smithi136.stdout: "end": "MIN", 2024-09-15T22:17:52.320 INFO:tasks.workunit.client.0.smithi136.stdout: "max_end": "MIN", 2024-09-15T22:17:52.320 INFO:tasks.workunit.client.0.smithi136.stdout: "subset_last_update": "0'0", 2024-09-15T22:17:52.320 INFO:tasks.workunit.client.0.smithi136.stdout: "deep": true, 2024-09-15T22:17:52.320 INFO:tasks.workunit.client.0.smithi136.stdout: "req_scrub": false, 2024-09-15T22:17:52.320 INFO:tasks.workunit.client.0.smithi136.stdout: "auto_repair": false, 2024-09-15T22:17:52.320 INFO:tasks.workunit.client.0.smithi136.stdout: "check_repair": false, 2024-09-15T22:17:52.320 INFO:tasks.workunit.client.0.smithi136.stdout: "deep_scrub_on_error": false, 2024-09-15T22:17:52.320 INFO:tasks.workunit.client.0.smithi136.stdout: "priority": 0, 2024-09-15T22:17:52.320 INFO:tasks.workunit.client.0.smithi136.stdout: "shallow_errors": 0, 2024-09-15T22:17:52.320 INFO:tasks.workunit.client.0.smithi136.stdout: "deep_errors": 0, 2024-09-15T22:17:52.320 INFO:tasks.workunit.client.0.smithi136.stdout: "fixed": 0, 2024-09-15T22:17:52.320 INFO:tasks.workunit.client.0.smithi136.stdout: "waiting_on_whom": [], 2024-09-15T22:17:52.321 INFO:tasks.workunit.client.0.smithi136.stdout: "schedule": "scrubbing" 2024-09-15T22:17:52.321 INFO:tasks.workunit.client.0.smithi136.stdout: }, 2024-09-15T22:17:52.321 INFO:tasks.workunit.client.0.smithi136.stdout: "agent_state": {} 2024-09-15T22:17:52.426 INFO:tasks.workunit.client.0.smithi136.stdout:( 2024-09-15T22:17:52.426 INFO:tasks.workunit.client.0.smithi136.stdout:[query_epoch]=21 2024-09-15T22:17:52.427 INFO:tasks.workunit.client.0.smithi136.stdout:[query_seq]=51 2024-09-15T22:17:52.427 INFO:tasks.workunit.client.0.smithi136.stdout:[query_active]=true 2024-09-15T22:17:52.427 INFO:tasks.workunit.client.0.smithi136.stdout:[query_schedule]='scrubbing' 2024-09-15T22:17:52.427 INFO:tasks.workunit.client.0.smithi136.stdout:[query_schedule_at]='0' 2024-09-15T22:17:52.427 INFO:tasks.workunit.client.0.smithi136.stdout:[query_target_at]='0' 2024-09-15T22:17:52.427 INFO:tasks.workunit.client.0.smithi136.stdout:[query_last_duration]=1 2024-09-15T22:17:52.427 INFO:tasks.workunit.client.0.smithi136.stdout:[query_last_stamp]='2024-09-08T22:16:04.103344+0000' 2024-09-15T22:17:52.427 INFO:tasks.workunit.client.0.smithi136.stdout:[query_last_scrub]='19x15' 2024-09-15T22:17:52.427 INFO:tasks.workunit.client.0.smithi136.stdout:[query_is_future]=false 2024-09-15T22:17:52.427 INFO:tasks.workunit.client.0.smithi136.stdout:[query_vs_date]=false 2024-09-15T22:17:52.427 INFO:tasks.workunit.client.0.smithi136.stdout:[query_scrub_seq]=null 2024-09-15T22:17:52.427 INFO:tasks.workunit.client.0.smithi136.stdout:[dmp_pg_state]='active+clean' 2024-09-15T22:17:52.427 INFO:tasks.workunit.client.0.smithi136.stdout:[dmp_state_has_scrubbing]=false 2024-09-15T22:17:52.427 INFO:tasks.workunit.client.0.smithi136.stdout:[dmp_last_duration]=1 2024-09-15T22:17:52.428 INFO:tasks.workunit.client.0.smithi136.stdout:[dmp_schedule]='periodic scrub scheduled' 2024-09-15T22:17:52.428 INFO:tasks.workunit.client.0.smithi136.stdout:[dmp_schedule_at]='2024-09-15T22:18:44.290846+0000' 2024-09-15T22:17:52.428 INFO:tasks.workunit.client.0.smithi136.stdout:[dmp_is_future]=true 2024-09-15T22:17:52.428 INFO:tasks.workunit.client.0.smithi136.stdout:[dmp_vs_date]=true 2024-09-15T22:17:52.428 INFO:tasks.workunit.client.0.smithi136.stdout:[dmp_reported_epoch]=20 2024-09-15T22:17:52.428 INFO:tasks.workunit.client.0.smithi136.stdout:[dmp_seq]=47 2024-09-15T22:17:52.428 INFO:tasks.workunit.client.0.smithi136.stdout:) 2024-09-15T22:17:52.430 INFO:tasks.workunit.client.0.smithi136.stdout:--> loop: 5 ~ true / 51 / 47 / false / 2024-09-08T22:16:04.103344+0000 / scrubbing %%% dmp_state_has_scrubbing query_active 2024-09-15T22:17:52.430 INFO:tasks.workunit.client.0.smithi136.stdout:key is dmp_state_has_scrubbing: negation:0 # expected: true # in actual: false 2024-09-15T22:17:52.430 INFO:tasks.workunit.client.0.smithi136.stdout:key is query_active: negation:0 # expected: false # in actual: true 2024-09-15T22:17:53.188 INFO:tasks.workunit.client.0.smithi136.stderr:dumped pgs 2024-09-15T22:17:53.200 INFO:tasks.workunit.client.0.smithi136.stdout:from pg dump pg: { 2024-09-15T22:17:53.200 INFO:tasks.workunit.client.0.smithi136.stdout: "dmp_pg_state": "active+clean", 2024-09-15T22:17:53.200 INFO:tasks.workunit.client.0.smithi136.stdout: "dmp_state_has_scrubbing": false, 2024-09-15T22:17:53.200 INFO:tasks.workunit.client.0.smithi136.stdout: "dmp_last_duration": 1, 2024-09-15T22:17:53.200 INFO:tasks.workunit.client.0.smithi136.stdout: "dmp_schedule": "periodic scrub scheduled", 2024-09-15T22:17:53.201 INFO:tasks.workunit.client.0.smithi136.stdout: "dmp_schedule_at": "2024-09-15T22:18:44.290846+0000", 2024-09-15T22:17:53.201 INFO:tasks.workunit.client.0.smithi136.stdout: "dmp_is_future": true, 2024-09-15T22:17:53.201 INFO:tasks.workunit.client.0.smithi136.stdout: "dmp_vs_date": true, 2024-09-15T22:17:53.201 INFO:tasks.workunit.client.0.smithi136.stdout: "dmp_reported_epoch": 21, 2024-09-15T22:17:53.201 INFO:tasks.workunit.client.0.smithi136.stdout: "dmp_seq": 50 2024-09-15T22:17:53.201 INFO:tasks.workunit.client.0.smithi136.stdout:} 2024-09-15T22:17:53.201 INFO:tasks.workunit.client.0.smithi136.stdout:query output: 2024-09-15T22:17:53.302 INFO:tasks.workunit.client.0.smithi136.stdout: "scrubber": { 2024-09-15T22:17:53.302 INFO:tasks.workunit.client.0.smithi136.stdout: "active": true, 2024-09-15T22:17:53.302 INFO:tasks.workunit.client.0.smithi136.stdout: "epoch_start": "17", 2024-09-15T22:17:53.303 INFO:tasks.workunit.client.0.smithi136.stdout: "start": "MIN", 2024-09-15T22:17:53.303 INFO:tasks.workunit.client.0.smithi136.stdout: "end": "MIN", 2024-09-15T22:17:53.303 INFO:tasks.workunit.client.0.smithi136.stdout: "max_end": "MIN", 2024-09-15T22:17:53.303 INFO:tasks.workunit.client.0.smithi136.stdout: "subset_last_update": "0'0", 2024-09-15T22:17:53.303 INFO:tasks.workunit.client.0.smithi136.stdout: "deep": true, 2024-09-15T22:17:53.303 INFO:tasks.workunit.client.0.smithi136.stdout: "req_scrub": false, 2024-09-15T22:17:53.303 INFO:tasks.workunit.client.0.smithi136.stdout: "auto_repair": false, 2024-09-15T22:17:53.303 INFO:tasks.workunit.client.0.smithi136.stdout: "check_repair": false, 2024-09-15T22:17:53.303 INFO:tasks.workunit.client.0.smithi136.stdout: "deep_scrub_on_error": false, 2024-09-15T22:17:53.303 INFO:tasks.workunit.client.0.smithi136.stdout: "priority": 0, 2024-09-15T22:17:53.303 INFO:tasks.workunit.client.0.smithi136.stdout: "shallow_errors": 0, 2024-09-15T22:17:53.303 INFO:tasks.workunit.client.0.smithi136.stdout: "deep_errors": 0, 2024-09-15T22:17:53.303 INFO:tasks.workunit.client.0.smithi136.stdout: "fixed": 0, 2024-09-15T22:17:53.303 INFO:tasks.workunit.client.0.smithi136.stdout: "waiting_on_whom": [], 2024-09-15T22:17:53.303 INFO:tasks.workunit.client.0.smithi136.stdout: "schedule": "scrubbing" 2024-09-15T22:17:53.304 INFO:tasks.workunit.client.0.smithi136.stdout: }, 2024-09-15T22:17:53.304 INFO:tasks.workunit.client.0.smithi136.stdout: "agent_state": {} 2024-09-15T22:17:53.409 INFO:tasks.workunit.client.0.smithi136.stdout:( 2024-09-15T22:17:53.409 INFO:tasks.workunit.client.0.smithi136.stdout:[query_epoch]=21 2024-09-15T22:17:53.409 INFO:tasks.workunit.client.0.smithi136.stdout:[query_seq]=51 2024-09-15T22:17:53.409 INFO:tasks.workunit.client.0.smithi136.stdout:[query_active]=true 2024-09-15T22:17:53.409 INFO:tasks.workunit.client.0.smithi136.stdout:[query_schedule]='scrubbing' 2024-09-15T22:17:53.409 INFO:tasks.workunit.client.0.smithi136.stdout:[query_schedule_at]='0' 2024-09-15T22:17:53.409 INFO:tasks.workunit.client.0.smithi136.stdout:[query_target_at]='0' 2024-09-15T22:17:53.409 INFO:tasks.workunit.client.0.smithi136.stdout:[query_last_duration]=1 2024-09-15T22:17:53.409 INFO:tasks.workunit.client.0.smithi136.stdout:[query_last_stamp]='2024-09-08T22:16:04.103344+0000' 2024-09-15T22:17:53.409 INFO:tasks.workunit.client.0.smithi136.stdout:[query_last_scrub]='19x15' 2024-09-15T22:17:53.410 INFO:tasks.workunit.client.0.smithi136.stdout:[query_is_future]=false 2024-09-15T22:17:53.410 INFO:tasks.workunit.client.0.smithi136.stdout:[query_vs_date]=false 2024-09-15T22:17:53.410 INFO:tasks.workunit.client.0.smithi136.stdout:[query_scrub_seq]=null 2024-09-15T22:17:53.410 INFO:tasks.workunit.client.0.smithi136.stdout:[dmp_pg_state]='active+clean' 2024-09-15T22:17:53.410 INFO:tasks.workunit.client.0.smithi136.stdout:[dmp_state_has_scrubbing]=false 2024-09-15T22:17:53.410 INFO:tasks.workunit.client.0.smithi136.stdout:[dmp_last_duration]=1 2024-09-15T22:17:53.410 INFO:tasks.workunit.client.0.smithi136.stdout:[dmp_schedule]='periodic scrub scheduled' 2024-09-15T22:17:53.410 INFO:tasks.workunit.client.0.smithi136.stdout:[dmp_schedule_at]='2024-09-15T22:18:44.290846+0000' 2024-09-15T22:17:53.410 INFO:tasks.workunit.client.0.smithi136.stdout:[dmp_is_future]=true 2024-09-15T22:17:53.410 INFO:tasks.workunit.client.0.smithi136.stdout:[dmp_vs_date]=true 2024-09-15T22:17:53.410 INFO:tasks.workunit.client.0.smithi136.stdout:[dmp_reported_epoch]=21 2024-09-15T22:17:53.410 INFO:tasks.workunit.client.0.smithi136.stdout:[dmp_seq]=50 2024-09-15T22:17:53.411 INFO:tasks.workunit.client.0.smithi136.stdout:) 2024-09-15T22:17:53.412 INFO:tasks.workunit.client.0.smithi136.stdout:--> loop: 6 ~ true / 51 / 50 / false / 2024-09-08T22:16:04.103344+0000 / scrubbing %%% dmp_state_has_scrubbing query_active 2024-09-15T22:17:53.412 INFO:tasks.workunit.client.0.smithi136.stdout:key is dmp_state_has_scrubbing: negation:0 # expected: true # in actual: false 2024-09-15T22:17:53.412 INFO:tasks.workunit.client.0.smithi136.stdout:key is query_active: negation:0 # expected: false # in actual: true 2024-09-15T22:17:54.176 INFO:tasks.workunit.client.0.smithi136.stderr:dumped pgs 2024-09-15T22:17:54.188 INFO:tasks.workunit.client.0.smithi136.stdout:from pg dump pg: { 2024-09-15T22:17:54.188 INFO:tasks.workunit.client.0.smithi136.stdout: "dmp_pg_state": "active+clean", 2024-09-15T22:17:54.188 INFO:tasks.workunit.client.0.smithi136.stdout: "dmp_state_has_scrubbing": false, 2024-09-15T22:17:54.188 INFO:tasks.workunit.client.0.smithi136.stdout: "dmp_last_duration": 1, 2024-09-15T22:17:54.188 INFO:tasks.workunit.client.0.smithi136.stdout: "dmp_schedule": "periodic scrub scheduled", 2024-09-15T22:17:54.188 INFO:tasks.workunit.client.0.smithi136.stdout: "dmp_schedule_at": "2024-09-15T22:18:44.290846+0000", 2024-09-15T22:17:54.188 INFO:tasks.workunit.client.0.smithi136.stdout: "dmp_is_future": true, 2024-09-15T22:17:54.189 INFO:tasks.workunit.client.0.smithi136.stdout: "dmp_vs_date": true, 2024-09-15T22:17:54.189 INFO:tasks.workunit.client.0.smithi136.stdout: "dmp_reported_epoch": 21, 2024-09-15T22:17:54.189 INFO:tasks.workunit.client.0.smithi136.stdout: "dmp_seq": 50 2024-09-15T22:17:54.189 INFO:tasks.workunit.client.0.smithi136.stdout:} 2024-09-15T22:17:54.189 INFO:tasks.workunit.client.0.smithi136.stdout:query output: 2024-09-15T22:17:54.290 INFO:tasks.workunit.client.0.smithi136.stdout: "scrubber": { 2024-09-15T22:17:54.290 INFO:tasks.workunit.client.0.smithi136.stdout: "active": true, 2024-09-15T22:17:54.290 INFO:tasks.workunit.client.0.smithi136.stdout: "epoch_start": "17", 2024-09-15T22:17:54.290 INFO:tasks.workunit.client.0.smithi136.stdout: "start": "MIN", 2024-09-15T22:17:54.290 INFO:tasks.workunit.client.0.smithi136.stdout: "end": "MIN", 2024-09-15T22:17:54.290 INFO:tasks.workunit.client.0.smithi136.stdout: "max_end": "MIN", 2024-09-15T22:17:54.291 INFO:tasks.workunit.client.0.smithi136.stdout: "subset_last_update": "0'0", 2024-09-15T22:17:54.291 INFO:tasks.workunit.client.0.smithi136.stdout: "deep": true, 2024-09-15T22:17:54.291 INFO:tasks.workunit.client.0.smithi136.stdout: "req_scrub": false, 2024-09-15T22:17:54.291 INFO:tasks.workunit.client.0.smithi136.stdout: "auto_repair": false, 2024-09-15T22:17:54.291 INFO:tasks.workunit.client.0.smithi136.stdout: "check_repair": false, 2024-09-15T22:17:54.291 INFO:tasks.workunit.client.0.smithi136.stdout: "deep_scrub_on_error": false, 2024-09-15T22:17:54.291 INFO:tasks.workunit.client.0.smithi136.stdout: "priority": 0, 2024-09-15T22:17:54.291 INFO:tasks.workunit.client.0.smithi136.stdout: "shallow_errors": 0, 2024-09-15T22:17:54.291 INFO:tasks.workunit.client.0.smithi136.stdout: "deep_errors": 0, 2024-09-15T22:17:54.291 INFO:tasks.workunit.client.0.smithi136.stdout: "fixed": 0, 2024-09-15T22:17:54.291 INFO:tasks.workunit.client.0.smithi136.stdout: "waiting_on_whom": [], 2024-09-15T22:17:54.291 INFO:tasks.workunit.client.0.smithi136.stdout: "schedule": "scrubbing" 2024-09-15T22:17:54.291 INFO:tasks.workunit.client.0.smithi136.stdout: }, 2024-09-15T22:17:54.291 INFO:tasks.workunit.client.0.smithi136.stdout: "agent_state": {} 2024-09-15T22:17:54.397 INFO:tasks.workunit.client.0.smithi136.stdout:( 2024-09-15T22:17:54.397 INFO:tasks.workunit.client.0.smithi136.stdout:[query_epoch]=21 2024-09-15T22:17:54.398 INFO:tasks.workunit.client.0.smithi136.stdout:[query_seq]=51 2024-09-15T22:17:54.398 INFO:tasks.workunit.client.0.smithi136.stdout:[query_active]=true 2024-09-15T22:17:54.398 INFO:tasks.workunit.client.0.smithi136.stdout:[query_schedule]='scrubbing' 2024-09-15T22:17:54.398 INFO:tasks.workunit.client.0.smithi136.stdout:[query_schedule_at]='0' 2024-09-15T22:17:54.398 INFO:tasks.workunit.client.0.smithi136.stdout:[query_target_at]='0' 2024-09-15T22:17:54.399 INFO:tasks.workunit.client.0.smithi136.stdout:[query_last_duration]=1 2024-09-15T22:17:54.399 INFO:tasks.workunit.client.0.smithi136.stdout:[query_last_stamp]='2024-09-08T22:16:04.103344+0000' 2024-09-15T22:17:54.399 INFO:tasks.workunit.client.0.smithi136.stdout:[query_last_scrub]='19x15' 2024-09-15T22:17:54.399 INFO:tasks.workunit.client.0.smithi136.stdout:[query_is_future]=false 2024-09-15T22:17:54.399 INFO:tasks.workunit.client.0.smithi136.stdout:[query_vs_date]=false 2024-09-15T22:17:54.399 INFO:tasks.workunit.client.0.smithi136.stdout:[query_scrub_seq]=null 2024-09-15T22:17:54.400 INFO:tasks.workunit.client.0.smithi136.stdout:[dmp_pg_state]='active+clean' 2024-09-15T22:17:54.400 INFO:tasks.workunit.client.0.smithi136.stdout:[dmp_state_has_scrubbing]=false 2024-09-15T22:17:54.400 INFO:tasks.workunit.client.0.smithi136.stdout:[dmp_last_duration]=1 2024-09-15T22:17:54.400 INFO:tasks.workunit.client.0.smithi136.stdout:[dmp_schedule]='periodic scrub scheduled' 2024-09-15T22:17:54.400 INFO:tasks.workunit.client.0.smithi136.stdout:[dmp_schedule_at]='2024-09-15T22:18:44.290846+0000' 2024-09-15T22:17:54.400 INFO:tasks.workunit.client.0.smithi136.stdout:[dmp_is_future]=true 2024-09-15T22:17:54.400 INFO:tasks.workunit.client.0.smithi136.stdout:[dmp_vs_date]=true 2024-09-15T22:17:54.401 INFO:tasks.workunit.client.0.smithi136.stdout:[dmp_reported_epoch]=21 2024-09-15T22:17:54.401 INFO:tasks.workunit.client.0.smithi136.stdout:[dmp_seq]=50 2024-09-15T22:17:54.401 INFO:tasks.workunit.client.0.smithi136.stdout:) 2024-09-15T22:17:54.402 INFO:tasks.workunit.client.0.smithi136.stdout:--> loop: 7 ~ true / 51 / 50 / false / 2024-09-08T22:16:04.103344+0000 / scrubbing %%% dmp_state_has_scrubbing query_active 2024-09-15T22:17:54.402 INFO:tasks.workunit.client.0.smithi136.stdout:key is dmp_state_has_scrubbing: negation:0 # expected: true # in actual: false 2024-09-15T22:17:54.402 INFO:tasks.workunit.client.0.smithi136.stdout:key is query_active: negation:0 # expected: false # in actual: true 2024-09-15T22:17:55.162 INFO:tasks.workunit.client.0.smithi136.stderr:dumped pgs 2024-09-15T22:17:55.174 INFO:tasks.workunit.client.0.smithi136.stdout:from pg dump pg: { 2024-09-15T22:17:55.174 INFO:tasks.workunit.client.0.smithi136.stdout: "dmp_pg_state": "active+clean", 2024-09-15T22:17:55.174 INFO:tasks.workunit.client.0.smithi136.stdout: "dmp_state_has_scrubbing": false, 2024-09-15T22:17:55.174 INFO:tasks.workunit.client.0.smithi136.stdout: "dmp_last_duration": 1, 2024-09-15T22:17:55.174 INFO:tasks.workunit.client.0.smithi136.stdout: "dmp_schedule": "periodic scrub scheduled", 2024-09-15T22:17:55.174 INFO:tasks.workunit.client.0.smithi136.stdout: "dmp_schedule_at": "2024-09-15T22:18:44.290846+0000", 2024-09-15T22:17:55.174 INFO:tasks.workunit.client.0.smithi136.stdout: "dmp_is_future": true, 2024-09-15T22:17:55.174 INFO:tasks.workunit.client.0.smithi136.stdout: "dmp_vs_date": true, 2024-09-15T22:17:55.175 INFO:tasks.workunit.client.0.smithi136.stdout: "dmp_reported_epoch": 21, 2024-09-15T22:17:55.175 INFO:tasks.workunit.client.0.smithi136.stdout: "dmp_seq": 50 2024-09-15T22:17:55.175 INFO:tasks.workunit.client.0.smithi136.stdout:} 2024-09-15T22:17:55.175 INFO:tasks.workunit.client.0.smithi136.stdout:query output: 2024-09-15T22:17:55.276 INFO:tasks.workunit.client.0.smithi136.stdout: "scrubber": { 2024-09-15T22:17:55.276 INFO:tasks.workunit.client.0.smithi136.stdout: "active": true, 2024-09-15T22:17:55.276 INFO:tasks.workunit.client.0.smithi136.stdout: "epoch_start": "17", 2024-09-15T22:17:55.276 INFO:tasks.workunit.client.0.smithi136.stdout: "start": "MIN", 2024-09-15T22:17:55.277 INFO:tasks.workunit.client.0.smithi136.stdout: "end": "MIN", 2024-09-15T22:17:55.277 INFO:tasks.workunit.client.0.smithi136.stdout: "max_end": "MIN", 2024-09-15T22:17:55.277 INFO:tasks.workunit.client.0.smithi136.stdout: "subset_last_update": "0'0", 2024-09-15T22:17:55.277 INFO:tasks.workunit.client.0.smithi136.stdout: "deep": true, 2024-09-15T22:17:55.277 INFO:tasks.workunit.client.0.smithi136.stdout: "req_scrub": false, 2024-09-15T22:17:55.277 INFO:tasks.workunit.client.0.smithi136.stdout: "auto_repair": false, 2024-09-15T22:17:55.277 INFO:tasks.workunit.client.0.smithi136.stdout: "check_repair": false, 2024-09-15T22:17:55.277 INFO:tasks.workunit.client.0.smithi136.stdout: "deep_scrub_on_error": false, 2024-09-15T22:17:55.277 INFO:tasks.workunit.client.0.smithi136.stdout: "priority": 0, 2024-09-15T22:17:55.277 INFO:tasks.workunit.client.0.smithi136.stdout: "shallow_errors": 0, 2024-09-15T22:17:55.277 INFO:tasks.workunit.client.0.smithi136.stdout: "deep_errors": 0, 2024-09-15T22:17:55.277 INFO:tasks.workunit.client.0.smithi136.stdout: "fixed": 0, 2024-09-15T22:17:55.277 INFO:tasks.workunit.client.0.smithi136.stdout: "waiting_on_whom": [], 2024-09-15T22:17:55.277 INFO:tasks.workunit.client.0.smithi136.stdout: "schedule": "scrubbing" 2024-09-15T22:17:55.277 INFO:tasks.workunit.client.0.smithi136.stdout: }, 2024-09-15T22:17:55.278 INFO:tasks.workunit.client.0.smithi136.stdout: "agent_state": {} 2024-09-15T22:17:55.384 INFO:tasks.workunit.client.0.smithi136.stdout:( 2024-09-15T22:17:55.384 INFO:tasks.workunit.client.0.smithi136.stdout:[query_epoch]=21 2024-09-15T22:17:55.384 INFO:tasks.workunit.client.0.smithi136.stdout:[query_seq]=51 2024-09-15T22:17:55.384 INFO:tasks.workunit.client.0.smithi136.stdout:[query_active]=true 2024-09-15T22:17:55.384 INFO:tasks.workunit.client.0.smithi136.stdout:[query_schedule]='scrubbing' 2024-09-15T22:17:55.384 INFO:tasks.workunit.client.0.smithi136.stdout:[query_schedule_at]='0' 2024-09-15T22:17:55.384 INFO:tasks.workunit.client.0.smithi136.stdout:[query_target_at]='0' 2024-09-15T22:17:55.385 INFO:tasks.workunit.client.0.smithi136.stdout:[query_last_duration]=1 2024-09-15T22:17:55.385 INFO:tasks.workunit.client.0.smithi136.stdout:[query_last_stamp]='2024-09-08T22:16:04.103344+0000' 2024-09-15T22:17:55.385 INFO:tasks.workunit.client.0.smithi136.stdout:[query_last_scrub]='19x15' 2024-09-15T22:17:55.385 INFO:tasks.workunit.client.0.smithi136.stdout:[query_is_future]=false 2024-09-15T22:17:55.385 INFO:tasks.workunit.client.0.smithi136.stdout:[query_vs_date]=false 2024-09-15T22:17:55.385 INFO:tasks.workunit.client.0.smithi136.stdout:[query_scrub_seq]=null 2024-09-15T22:17:55.385 INFO:tasks.workunit.client.0.smithi136.stdout:[dmp_pg_state]='active+clean' 2024-09-15T22:17:55.385 INFO:tasks.workunit.client.0.smithi136.stdout:[dmp_state_has_scrubbing]=false 2024-09-15T22:17:55.385 INFO:tasks.workunit.client.0.smithi136.stdout:[dmp_last_duration]=1 2024-09-15T22:17:55.385 INFO:tasks.workunit.client.0.smithi136.stdout:[dmp_schedule]='periodic scrub scheduled' 2024-09-15T22:17:55.385 INFO:tasks.workunit.client.0.smithi136.stdout:[dmp_schedule_at]='2024-09-15T22:18:44.290846+0000' 2024-09-15T22:17:55.385 INFO:tasks.workunit.client.0.smithi136.stdout:[dmp_is_future]=true 2024-09-15T22:17:55.385 INFO:tasks.workunit.client.0.smithi136.stdout:[dmp_vs_date]=true 2024-09-15T22:17:55.385 INFO:tasks.workunit.client.0.smithi136.stdout:[dmp_reported_epoch]=21 2024-09-15T22:17:55.385 INFO:tasks.workunit.client.0.smithi136.stdout:[dmp_seq]=50 2024-09-15T22:17:55.386 INFO:tasks.workunit.client.0.smithi136.stdout:) 2024-09-15T22:17:55.388 INFO:tasks.workunit.client.0.smithi136.stdout:--> loop: 8 ~ true / 51 / 50 / false / 2024-09-08T22:16:04.103344+0000 / scrubbing %%% dmp_state_has_scrubbing query_active 2024-09-15T22:17:55.388 INFO:tasks.workunit.client.0.smithi136.stdout:key is dmp_state_has_scrubbing: negation:0 # expected: true # in actual: false 2024-09-15T22:17:55.388 INFO:tasks.workunit.client.0.smithi136.stdout:key is query_active: negation:0 # expected: false # in actual: true 2024-09-15T22:17:56.152 INFO:tasks.workunit.client.0.smithi136.stderr:dumped pgs 2024-09-15T22:17:56.164 INFO:tasks.workunit.client.0.smithi136.stdout:from pg dump pg: { 2024-09-15T22:17:56.164 INFO:tasks.workunit.client.0.smithi136.stdout: "dmp_pg_state": "active+clean", 2024-09-15T22:17:56.164 INFO:tasks.workunit.client.0.smithi136.stdout: "dmp_state_has_scrubbing": false, 2024-09-15T22:17:56.164 INFO:tasks.workunit.client.0.smithi136.stdout: "dmp_last_duration": 1, 2024-09-15T22:17:56.164 INFO:tasks.workunit.client.0.smithi136.stdout: "dmp_schedule": "periodic scrub scheduled", 2024-09-15T22:17:56.164 INFO:tasks.workunit.client.0.smithi136.stdout: "dmp_schedule_at": "2024-09-15T22:18:44.290846+0000", 2024-09-15T22:17:56.164 INFO:tasks.workunit.client.0.smithi136.stdout: "dmp_is_future": true, 2024-09-15T22:17:56.164 INFO:tasks.workunit.client.0.smithi136.stdout: "dmp_vs_date": true, 2024-09-15T22:17:56.164 INFO:tasks.workunit.client.0.smithi136.stdout: "dmp_reported_epoch": 21, 2024-09-15T22:17:56.164 INFO:tasks.workunit.client.0.smithi136.stdout: "dmp_seq": 50 2024-09-15T22:17:56.165 INFO:tasks.workunit.client.0.smithi136.stdout:} 2024-09-15T22:17:56.165 INFO:tasks.workunit.client.0.smithi136.stdout:query output: 2024-09-15T22:17:56.266 INFO:tasks.workunit.client.0.smithi136.stdout: "scrubber": { 2024-09-15T22:17:56.267 INFO:tasks.workunit.client.0.smithi136.stdout: "active": true, 2024-09-15T22:17:56.267 INFO:tasks.workunit.client.0.smithi136.stdout: "epoch_start": "17", 2024-09-15T22:17:56.267 INFO:tasks.workunit.client.0.smithi136.stdout: "start": "MIN", 2024-09-15T22:17:56.267 INFO:tasks.workunit.client.0.smithi136.stdout: "end": "MIN", 2024-09-15T22:17:56.267 INFO:tasks.workunit.client.0.smithi136.stdout: "max_end": "MIN", 2024-09-15T22:17:56.267 INFO:tasks.workunit.client.0.smithi136.stdout: "subset_last_update": "0'0", 2024-09-15T22:17:56.267 INFO:tasks.workunit.client.0.smithi136.stdout: "deep": true, 2024-09-15T22:17:56.267 INFO:tasks.workunit.client.0.smithi136.stdout: "req_scrub": false, 2024-09-15T22:17:56.267 INFO:tasks.workunit.client.0.smithi136.stdout: "auto_repair": false, 2024-09-15T22:17:56.267 INFO:tasks.workunit.client.0.smithi136.stdout: "check_repair": false, 2024-09-15T22:17:56.267 INFO:tasks.workunit.client.0.smithi136.stdout: "deep_scrub_on_error": false, 2024-09-15T22:17:56.267 INFO:tasks.workunit.client.0.smithi136.stdout: "priority": 0, 2024-09-15T22:17:56.267 INFO:tasks.workunit.client.0.smithi136.stdout: "shallow_errors": 0, 2024-09-15T22:17:56.267 INFO:tasks.workunit.client.0.smithi136.stdout: "deep_errors": 0, 2024-09-15T22:17:56.268 INFO:tasks.workunit.client.0.smithi136.stdout: "fixed": 0, 2024-09-15T22:17:56.268 INFO:tasks.workunit.client.0.smithi136.stdout: "waiting_on_whom": [], 2024-09-15T22:17:56.268 INFO:tasks.workunit.client.0.smithi136.stdout: "schedule": "scrubbing" 2024-09-15T22:17:56.268 INFO:tasks.workunit.client.0.smithi136.stdout: }, 2024-09-15T22:17:56.268 INFO:tasks.workunit.client.0.smithi136.stdout: "agent_state": {} 2024-09-15T22:17:56.374 INFO:tasks.workunit.client.0.smithi136.stdout:( 2024-09-15T22:17:56.374 INFO:tasks.workunit.client.0.smithi136.stdout:[query_epoch]=21 2024-09-15T22:17:56.374 INFO:tasks.workunit.client.0.smithi136.stdout:[query_seq]=51 2024-09-15T22:17:56.374 INFO:tasks.workunit.client.0.smithi136.stdout:[query_active]=true 2024-09-15T22:17:56.374 INFO:tasks.workunit.client.0.smithi136.stdout:[query_schedule]='scrubbing' 2024-09-15T22:17:56.375 INFO:tasks.workunit.client.0.smithi136.stdout:[query_schedule_at]='0' 2024-09-15T22:17:56.375 INFO:tasks.workunit.client.0.smithi136.stdout:[query_target_at]='0' 2024-09-15T22:17:56.375 INFO:tasks.workunit.client.0.smithi136.stdout:[query_last_duration]=1 2024-09-15T22:17:56.375 INFO:tasks.workunit.client.0.smithi136.stdout:[query_last_stamp]='2024-09-08T22:16:04.103344+0000' 2024-09-15T22:17:56.375 INFO:tasks.workunit.client.0.smithi136.stdout:[query_last_scrub]='19x15' 2024-09-15T22:17:56.375 INFO:tasks.workunit.client.0.smithi136.stdout:[query_is_future]=false 2024-09-15T22:17:56.375 INFO:tasks.workunit.client.0.smithi136.stdout:[query_vs_date]=false 2024-09-15T22:17:56.375 INFO:tasks.workunit.client.0.smithi136.stdout:[query_scrub_seq]=null 2024-09-15T22:17:56.375 INFO:tasks.workunit.client.0.smithi136.stdout:[dmp_pg_state]='active+clean' 2024-09-15T22:17:56.375 INFO:tasks.workunit.client.0.smithi136.stdout:[dmp_state_has_scrubbing]=false 2024-09-15T22:17:56.375 INFO:tasks.workunit.client.0.smithi136.stdout:[dmp_last_duration]=1 2024-09-15T22:17:56.375 INFO:tasks.workunit.client.0.smithi136.stdout:[dmp_schedule]='periodic scrub scheduled' 2024-09-15T22:17:56.375 INFO:tasks.workunit.client.0.smithi136.stdout:[dmp_schedule_at]='2024-09-15T22:18:44.290846+0000' 2024-09-15T22:17:56.375 INFO:tasks.workunit.client.0.smithi136.stdout:[dmp_is_future]=true 2024-09-15T22:17:56.375 INFO:tasks.workunit.client.0.smithi136.stdout:[dmp_vs_date]=true 2024-09-15T22:17:56.376 INFO:tasks.workunit.client.0.smithi136.stdout:[dmp_reported_epoch]=21 2024-09-15T22:17:56.376 INFO:tasks.workunit.client.0.smithi136.stdout:[dmp_seq]=50 2024-09-15T22:17:56.376 INFO:tasks.workunit.client.0.smithi136.stdout:) 2024-09-15T22:17:56.378 INFO:tasks.workunit.client.0.smithi136.stdout:--> loop: 9 ~ true / 51 / 50 / false / 2024-09-08T22:16:04.103344+0000 / scrubbing %%% dmp_state_has_scrubbing query_active 2024-09-15T22:17:56.378 INFO:tasks.workunit.client.0.smithi136.stdout:key is dmp_state_has_scrubbing: negation:0 # expected: true # in actual: false 2024-09-15T22:17:56.378 INFO:tasks.workunit.client.0.smithi136.stdout:key is query_active: negation:0 # expected: false # in actual: true 2024-09-15T22:17:57.144 INFO:tasks.workunit.client.0.smithi136.stderr:dumped pgs 2024-09-15T22:17:57.154 INFO:tasks.workunit.client.0.smithi136.stdout:from pg dump pg: { 2024-09-15T22:17:57.154 INFO:tasks.workunit.client.0.smithi136.stdout: "dmp_pg_state": "active+clean", 2024-09-15T22:17:57.154 INFO:tasks.workunit.client.0.smithi136.stdout: "dmp_state_has_scrubbing": false, 2024-09-15T22:17:57.154 INFO:tasks.workunit.client.0.smithi136.stdout: "dmp_last_duration": 1, 2024-09-15T22:17:57.154 INFO:tasks.workunit.client.0.smithi136.stdout: "dmp_schedule": "periodic scrub scheduled", 2024-09-15T22:17:57.154 INFO:tasks.workunit.client.0.smithi136.stdout: "dmp_schedule_at": "2024-09-15T22:18:44.290846+0000", 2024-09-15T22:17:57.154 INFO:tasks.workunit.client.0.smithi136.stdout: "dmp_is_future": true, 2024-09-15T22:17:57.154 INFO:tasks.workunit.client.0.smithi136.stdout: "dmp_vs_date": true, 2024-09-15T22:17:57.155 INFO:tasks.workunit.client.0.smithi136.stdout: "dmp_reported_epoch": 21, 2024-09-15T22:17:57.155 INFO:tasks.workunit.client.0.smithi136.stdout: "dmp_seq": 50 2024-09-15T22:17:57.155 INFO:tasks.workunit.client.0.smithi136.stdout:} 2024-09-15T22:17:57.155 INFO:tasks.workunit.client.0.smithi136.stdout:query output: 2024-09-15T22:17:57.256 INFO:tasks.workunit.client.0.smithi136.stdout: "scrubber": { 2024-09-15T22:17:57.256 INFO:tasks.workunit.client.0.smithi136.stdout: "active": true, 2024-09-15T22:17:57.256 INFO:tasks.workunit.client.0.smithi136.stdout: "epoch_start": "17", 2024-09-15T22:17:57.257 INFO:tasks.workunit.client.0.smithi136.stdout: "start": "MIN", 2024-09-15T22:17:57.257 INFO:tasks.workunit.client.0.smithi136.stdout: "end": "MIN", 2024-09-15T22:17:57.257 INFO:tasks.workunit.client.0.smithi136.stdout: "max_end": "MIN", 2024-09-15T22:17:57.257 INFO:tasks.workunit.client.0.smithi136.stdout: "subset_last_update": "0'0", 2024-09-15T22:17:57.257 INFO:tasks.workunit.client.0.smithi136.stdout: "deep": true, 2024-09-15T22:17:57.257 INFO:tasks.workunit.client.0.smithi136.stdout: "req_scrub": false, 2024-09-15T22:17:57.257 INFO:tasks.workunit.client.0.smithi136.stdout: "auto_repair": false, 2024-09-15T22:17:57.257 INFO:tasks.workunit.client.0.smithi136.stdout: "check_repair": false, 2024-09-15T22:17:57.257 INFO:tasks.workunit.client.0.smithi136.stdout: "deep_scrub_on_error": false, 2024-09-15T22:17:57.257 INFO:tasks.workunit.client.0.smithi136.stdout: "priority": 0, 2024-09-15T22:17:57.257 INFO:tasks.workunit.client.0.smithi136.stdout: "shallow_errors": 0, 2024-09-15T22:17:57.257 INFO:tasks.workunit.client.0.smithi136.stdout: "deep_errors": 0, 2024-09-15T22:17:57.257 INFO:tasks.workunit.client.0.smithi136.stdout: "fixed": 0, 2024-09-15T22:17:57.257 INFO:tasks.workunit.client.0.smithi136.stdout: "waiting_on_whom": [], 2024-09-15T22:17:57.258 INFO:tasks.workunit.client.0.smithi136.stdout: "schedule": "scrubbing" 2024-09-15T22:17:57.258 INFO:tasks.workunit.client.0.smithi136.stdout: }, 2024-09-15T22:17:57.258 INFO:tasks.workunit.client.0.smithi136.stdout: "agent_state": {} 2024-09-15T22:17:57.363 INFO:tasks.workunit.client.0.smithi136.stdout:( 2024-09-15T22:17:57.364 INFO:tasks.workunit.client.0.smithi136.stdout:[query_epoch]=21 2024-09-15T22:17:57.364 INFO:tasks.workunit.client.0.smithi136.stdout:[query_seq]=52 2024-09-15T22:17:57.364 INFO:tasks.workunit.client.0.smithi136.stdout:[query_active]=true 2024-09-15T22:17:57.364 INFO:tasks.workunit.client.0.smithi136.stdout:[query_schedule]='scrubbing' 2024-09-15T22:17:57.364 INFO:tasks.workunit.client.0.smithi136.stdout:[query_schedule_at]='0' 2024-09-15T22:17:57.364 INFO:tasks.workunit.client.0.smithi136.stdout:[query_target_at]='0' 2024-09-15T22:17:57.364 INFO:tasks.workunit.client.0.smithi136.stdout:[query_last_duration]=1 2024-09-15T22:17:57.364 INFO:tasks.workunit.client.0.smithi136.stdout:[query_last_stamp]='2024-09-08T22:16:04.103344+0000' 2024-09-15T22:17:57.364 INFO:tasks.workunit.client.0.smithi136.stdout:[query_last_scrub]='19x15' 2024-09-15T22:17:57.364 INFO:tasks.workunit.client.0.smithi136.stdout:[query_is_future]=false 2024-09-15T22:17:57.364 INFO:tasks.workunit.client.0.smithi136.stdout:[query_vs_date]=false 2024-09-15T22:17:57.364 INFO:tasks.workunit.client.0.smithi136.stdout:[query_scrub_seq]=null 2024-09-15T22:17:57.365 INFO:tasks.workunit.client.0.smithi136.stdout:[dmp_pg_state]='active+clean' 2024-09-15T22:17:57.365 INFO:tasks.workunit.client.0.smithi136.stdout:[dmp_state_has_scrubbing]=false 2024-09-15T22:17:57.365 INFO:tasks.workunit.client.0.smithi136.stdout:[dmp_last_duration]=1 2024-09-15T22:17:57.365 INFO:tasks.workunit.client.0.smithi136.stdout:[dmp_schedule]='periodic scrub scheduled' 2024-09-15T22:17:57.365 INFO:tasks.workunit.client.0.smithi136.stdout:[dmp_schedule_at]='2024-09-15T22:18:44.290846+0000' 2024-09-15T22:17:57.365 INFO:tasks.workunit.client.0.smithi136.stdout:[dmp_is_future]=true 2024-09-15T22:17:57.365 INFO:tasks.workunit.client.0.smithi136.stdout:[dmp_vs_date]=true 2024-09-15T22:17:57.365 INFO:tasks.workunit.client.0.smithi136.stdout:[dmp_reported_epoch]=21 2024-09-15T22:17:57.365 INFO:tasks.workunit.client.0.smithi136.stdout:[dmp_seq]=50 2024-09-15T22:17:57.365 INFO:tasks.workunit.client.0.smithi136.stdout:) 2024-09-15T22:17:57.367 INFO:tasks.workunit.client.0.smithi136.stdout:--> loop: 10 ~ true / 52 / 50 / false / 2024-09-08T22:16:04.103344+0000 / scrubbing %%% dmp_state_has_scrubbing query_active 2024-09-15T22:17:57.367 INFO:tasks.workunit.client.0.smithi136.stdout:key is dmp_state_has_scrubbing: negation:0 # expected: true # in actual: false 2024-09-15T22:17:57.367 INFO:tasks.workunit.client.0.smithi136.stdout:key is query_active: negation:0 # expected: false # in actual: true 2024-09-15T22:17:57.367 INFO:tasks.workunit.client.0.smithi136.stdout:WaitingActive : wait_any_cond(): failure. Note: query-active=true 2024-09-15T22:17:57.368 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:164: wait_any_cond: return 1 2024-09-15T22:17:57.369 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:643: TEST_dump_scrub_schedule: return 1 2024-09-15T22:17:57.369 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:35: run: return 1 2024-09-15T22:17:57.369 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2324: main: code=1 2024-09-15T22:17:57.369 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2326: main: teardown td/osd-scrub-test 1 2024-09-15T22:17:57.369 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/osd-scrub-test 2024-09-15T22:17:57.369 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs=1 2024-09-15T22:17:57.369 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/osd-scrub-test KILL 2024-09-15T22:17:57.369 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-15T22:17:57.369 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-15T22:17:57.369 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-15T22:17:57.369 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-15T22:17:57.369 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-15T22:17:57.475 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-15T22:17:57.476 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-15T22:17:57.476 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-15T22:17:57.477 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-15T22:17:57.478 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-15T22:17:57.478 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-15T22:17:57.478 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-15T22:17:57.479 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T22:17:57.479 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-15T22:17:57.479 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-15T22:17:57.479 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T22:17:57.480 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-15T22:17:57.481 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o 1 = 1 ']' 2024-09-15T22:17:57.481 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:181: teardown: '[' -n '' ']' 2024-09-15T22:17:57.481 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:185: teardown: mkdir -p /home/ubuntu/cephtest/archive/log 2024-09-15T22:17:57.482 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:186: teardown: mv td/osd-scrub-test/mgr.x.log td/osd-scrub-test/mon.a.log td/osd-scrub-test/osd.0.log td/osd-scrub-test/osd.1.log td/osd-scrub-test/osd.2.log /home/ubuntu/cephtest/archive/log 2024-09-15T22:17:57.483 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/osd-scrub-test 2024-09-15T22:17:57.486 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-15T22:17:57.486 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T22:17:57.486 INFO:tasks.workunit.client.0.smithi136.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.519518 2024-09-15T22:17:57.486 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.519518 2024-09-15T22:17:57.487 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-15T22:17:57.487 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-15T22:17:57.487 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2327: main: return 1 2024-09-15T22:17:57.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-15T22:17:57.488 INFO:tasks.workunit:Stopping ['scrub'] on client.0... 2024-09-15T22:17:57.488 DEBUG:teuthology.orchestra.run.smithi136:> sudo rm -rf -- /home/ubuntu/cephtest/workunits.list.client.0 /home/ubuntu/cephtest/clone.client.0 2024-09-15T22:17:57.824 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/run_tasks.py", line 105, in run_tasks manager = run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/git.ceph.com_ceph_7498493279982dfe87f00616198a5967475f6169/qa/tasks/workunit.py", line 145, in task _spawn_on_all_clients(ctx, refspec, all_tasks, config.get('env'), File "/home/teuthworker/src/git.ceph.com_ceph_7498493279982dfe87f00616198a5967475f6169/qa/tasks/workunit.py", line 293, in _spawn_on_all_clients with parallel() as p: File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/parallel.py", line 84, in __exit__ for result in self: File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/parallel.py", line 98, in __next__ resurrect_traceback(result) File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/parallel.py", line 30, in resurrect_traceback raise exc.exc_info[1] File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/parallel.py", line 23, in capture_traceback return func(*args, **kwargs) File "/home/teuthworker/src/git.ceph.com_ceph_7498493279982dfe87f00616198a5967475f6169/qa/tasks/workunit.py", line 434, in _run_tests remote.run( File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed (workunit test scrub/osd-scrub-test.sh) on smithi136 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=7498493279982dfe87f00616198a5967475f6169 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh' 2024-09-15T22:17:58.000 ERROR:teuthology.util.sentry: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=82ef50f0957041d3b08b3ef90caf6f4d Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/run_tasks.py", line 105, in run_tasks manager = run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/git.ceph.com_ceph_7498493279982dfe87f00616198a5967475f6169/qa/tasks/workunit.py", line 145, in task _spawn_on_all_clients(ctx, refspec, all_tasks, config.get('env'), File "/home/teuthworker/src/git.ceph.com_ceph_7498493279982dfe87f00616198a5967475f6169/qa/tasks/workunit.py", line 293, in _spawn_on_all_clients with parallel() as p: File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/parallel.py", line 84, in __exit__ for result in self: File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/parallel.py", line 98, in __next__ resurrect_traceback(result) File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/parallel.py", line 30, in resurrect_traceback raise exc.exc_info[1] File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/parallel.py", line 23, in capture_traceback return func(*args, **kwargs) File "/home/teuthworker/src/git.ceph.com_ceph_7498493279982dfe87f00616198a5967475f6169/qa/tasks/workunit.py", line 434, in _run_tests remote.run( File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed (workunit test scrub/osd-scrub-test.sh) on smithi136 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=7498493279982dfe87f00616198a5967475f6169 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh' 2024-09-15T22:17:58.003 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-09-15T22:17:58.012 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-09-15T22:17:58.012 DEBUG:teuthology.orchestra.run.smithi136:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-09-15T22:17:58.066 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-09-15T22:17:58.074 INFO:teuthology.task.clock:Checking final clock skew... 2024-09-15T22:17:58.074 DEBUG:teuthology.orchestra.run.smithi136:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-15T22:17:58.094 INFO:teuthology.orchestra.run.smithi136.stderr:bash: line 1: ntpq: command not found 2024-09-15T22:17:58.099 INFO:teuthology.orchestra.run.smithi136.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-09-15T22:17:58.099 INFO:teuthology.orchestra.run.smithi136.stdout:=============================================================================== 2024-09-15T22:17:58.100 INFO:teuthology.orchestra.run.smithi136.stdout:^* hv01.front.sepia.ceph.com 3 6 377 23 +1077us[+1049us] +/- 44ms 2024-09-15T22:17:58.100 INFO:teuthology.orchestra.run.smithi136.stdout:^+ hv02.front.sepia.ceph.com 3 6 377 32 -840us[ -869us] +/- 56ms 2024-09-15T22:17:58.100 INFO:teuthology.orchestra.run.smithi136.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 31 -953us[ -981us] +/- 80ms 2024-09-15T22:17:58.100 INFO:teuthology.orchestra.run.smithi136.stdout:^? hv04.front.sepia.ceph.com 0 10 0 - +0ns[ +0ns] +/- 0ns 2024-09-15T22:17:58.100 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-09-15T22:17:58.110 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-09-15T22:17:58.111 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-09-15T22:17:58.119 DEBUG:teuthology.orchestra.run.smithi136:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-09-15T22:17:58.198 DEBUG:teuthology.orchestra.run.smithi136:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 2024-09-15T22:17:58.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-15T22:17:58.225 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-09-15T22:17:58.235 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-09-15T22:17:58.243 INFO:teuthology.task.internal:Duration was 7504.494915 seconds 2024-09-15T22:17:58.243 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-09-15T22:17:58.251 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-09-15T22:17:58.252 DEBUG:teuthology.orchestra.run.smithi136:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-09-15T22:17:58.304 INFO:teuthology.orchestra.run.smithi136.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-09-15T22:17:58.504 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-09-15T22:17:58.504 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi136.front.sepia.ceph.com 2024-09-15T22:17:58.505 DEBUG:teuthology.orchestra.run.smithi136:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-09-15T22:17:58.564 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-09-15T22:17:58.564 DEBUG:teuthology.orchestra.run.smithi136:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-09-15T22:17:58.630 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-09-15T22:17:58.631 DEBUG:teuthology.orchestra.run.smithi136:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-09-15T22:17:58.771 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-09-15T22:17:58.781 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-09-15T22:17:58.782 DEBUG:teuthology.orchestra.run.smithi136:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-09-15T22:17:58.806 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-09-15T22:17:58.816 DEBUG:teuthology.orchestra.run.smithi136:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-09-15T22:17:58.869 INFO:teuthology.orchestra.run.smithi136.stdout:kernel.core_pattern = core 2024-09-15T22:17:58.886 DEBUG:teuthology.orchestra.run.smithi136:> test -e /home/ubuntu/cephtest/archive/coredump 2024-09-15T22:17:58.939 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-15T22:17:58.939 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-09-15T22:17:58.948 INFO:teuthology.task.internal:Transferring archived files... 2024-09-15T22:17:58.950 DEBUG:teuthology.misc:Transferring archived files from smithi136:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/teuthology-2024-09-15_20:00:16-rados-main-distro-default-smithi/7905590/remote/smithi136 2024-09-15T22:17:58.951 DEBUG:teuthology.orchestra.run.smithi136:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-09-15T22:17:59.637 INFO:teuthology.task.internal:Removing archive directory... 2024-09-15T22:17:59.637 DEBUG:teuthology.orchestra.run.smithi136:> rm -rf -- /home/ubuntu/cephtest/archive 2024-09-15T22:17:59.680 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-09-15T22:17:59.690 INFO:teuthology.task.internal:Not uploading archives. 2024-09-15T22:17:59.690 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-09-15T22:17:59.702 INFO:teuthology.task.internal:Tidying up after the test... 2024-09-15T22:17:59.702 DEBUG:teuthology.orchestra.run.smithi136:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-09-15T22:17:59.736 INFO:teuthology.orchestra.run.smithi136.stdout: 266063 4 drwxr-xr-x 3 ubuntu ubuntu 4096 Sep 15 22:17 /home/ubuntu/cephtest 2024-09-15T22:17:59.736 INFO:teuthology.orchestra.run.smithi136.stdout: 405775 4 drwxr-xr-x 3 ubuntu ubuntu 4096 Sep 15 20:18 /home/ubuntu/cephtest/mnt.0 2024-09-15T22:17:59.736 INFO:teuthology.orchestra.run.smithi136.stdout: 405776 4 drwxr-xr-x 3 ubuntu ubuntu 4096 Sep 15 22:05 /home/ubuntu/cephtest/mnt.0/client.0 2024-09-15T22:17:59.737 INFO:teuthology.orchestra.run.smithi136.stdout: 405820 4 drwxr-xr-x 3 ubuntu ubuntu 4096 Sep 15 22:17 /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-09-15T22:17:59.737 INFO:teuthology.orchestra.run.smithi136.stdout: 420635 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Sep 15 22:17 /home/ubuntu/cephtest/mnt.0/client.0/tmp/td 2024-09-15T22:17:59.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-15T22:17:59.738 INFO:teuthology.orchestra.run.smithi136.stderr:rmdir: failed to remove '/home/ubuntu/cephtest': Directory not empty 2024-09-15T22:17:59.738 ERROR:teuthology.run_tasks:Manager failed: internal.base Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/run_tasks.py", line 160, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.10/contextlib.py", line 142, in __exit__ next(self.gen) File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/task/internal/__init__.py", line 53, in base run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi136 with status 1: 'find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest' 2024-09-15T22:17:59.742 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-09-15T22:17:59.752 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-09-15T22:17:59.761 DEBUG:teuthology.run_tasks:Exception was not quenched, exiting: CommandFailedError: Command failed on smithi136 with status 1: 'find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest' 2024-09-15T22:17:59.765 INFO:teuthology.run:Summary data: description: rados/standalone/{supported-random-distro$/{centos_latest} workloads/scrub} duration: 7504.494914531708 failure_reason: 'Command failed (workunit test scrub/osd-scrub-test.sh) on smithi136 with status 1: ''mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=7498493279982dfe87f00616198a5967475f6169 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh''' flavor: default owner: scheduled_teuthology@teuthology sentry_event: https://sentry.ceph.com/organizations/ceph/?query=82ef50f0957041d3b08b3ef90caf6f4d status: fail success: false 2024-09-15T22:17:59.765 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-09-15T22:17:59.844 INFO:teuthology.run:FAIL